Slave has disconnected errors

We also have same issues of slave disconnection happened few times.  The "disconnect job" issue always happened at end of 2D classfication or Ab initio 3D model process.  When the error message pops up, the calculation kind of paused forever.  We ran cisTEM on a local GPU workstation (16 Xeon cores, 128 GB DDR4 ram, 25TB local HDD, 1080 Ti cards x4).

Best,

Kuen

Hi Daniel,

We routinely ran 33 processes (16 cores, 32 threads). The HDD is configured by RAID 0 (10TB x2, 4TB x4, Hitachi enterprise SAS HDD), filesystem is ZFS.

When seeing the slave disconnection error, one CPU in htop was idling ( I usually got 1 slave disconnected). The werid thing is we reran same process without changing settings, it often finished normally (sometimes need to rerun 2 times). I am sure no other disc I/O process except cisTEM during calcuation. 

Dear Tim and cisTEM users,

I run into similar issues with my (first) 2d classification job.

It goes through to round5.

If I run it from the cluster, I get a series of Error: A slave has disconnected before all jobs are finished. It then remains idle and the job never progresses past round 5.

If I run it interactively from a multicore workstation I get no error at all, but still the job gets stuck at round 5 and never goes forward.

In both cases if I look at the results I notice that for some rounds the classes are nonsensical: either totally grey, or weird pixelated boxes. Still, round 1,2 and 4 show what expected.

My first guess was that there must be some corrupted particles that when picked up in the randomly selected subset cause trouble, but I manually visualised all of them and couldn't see anything obvious. I got no warning or anything while extracting them.

Not quite sure where to start from to troubleshoot, any help would be much appreciated.

Thanks

Giulia

Hi Giulia,

Does it reproducibly stop at round 5?  If so, would be able to share the stack with me, so that I can look into it?

Cheers!

Tim

Hi Tim,

It does reproducibly get stuck at round5.

This is a groel test dataset that was collected on our new microscope for a test. I was handed the pre-motioncorrected sums and pre-picked .box files.
I imported DW and noDW, run ctffind through cisTEM (on noDW), then converted .box to .txt with a script I found on the forum and imported that too.
Then I followed the tutorial to extract particles (from DW), and run 2D class.

(FYI same .box and .mrc files imported in relion run just fine through the end)

What is the best way to share the stack with you?

Thanks a lot!!!

Giulia

Hi,

I just want to report that I was also getting "Error: A slave has disconnected before all jobs are finished" errors and the jobs would hang indefinitely during Movie Alignment. This would happen both when submitting jobs to SLURM or when running locally on a computing node ("Default local" profile). This was strange because normally cisTEM runs fine on our cluster. Turns out for this particular dataset I was using the option "Include All Frames in Sum?" to select only a subset of the aligned frames. When disabling this option (i.e. including all frames, which is default), the job ran normally to completion.

I'm not sure how using this option would cause such an error but this could be a useful hint for debugging.

Cheers,

Ricardo

Hi Ricardo,

Thanks for the info - i'm sure it will be helpful!

Tim

I ran into exactly same issue as rdrighetto mentioned. When I only select subset of frames for movie alignment, I got multiple "Error: A slave has disconnected before all jobs are finished". Everything is OK when I include all the frames.

Hi Tim,

I am having the same issue as rdrighetto, but the issue isn't solved by switching back to including all frames. Was this ever solved more thoroghly?

Thanks in advance,

Chris

If you split into groups as detailed above, you can pick in groups, and all the results will sum in the particle position assests as (all positions)

Cheers,

Tim