Cryogenic Electron Microscopy (Cryo-EM) Data Processing on McCleary
Below is a work in progress collection of general hints, tips and tricks for running your work on McCleary. As always, if anything below is unclear or could use updating, please let us know during office hours, via email or through our web ticketing system.
Storage
Be wary of you and your group's storage quotas. Run getquota from time to time to make sure there isn't usage you aren't expecting. We strongly recommend that you archive raw data off-cluster, as only home directories are backed up. Let us know if you need extra space and we can work with you to find a solution that is right for your project and your group.
On most GPU nodes there is a fast SSD mounted at /tmp. You can use this as a fast local cache if your program can take advantage of it.
Schedule Jobs
Many Cryo-EM applications can make use of GPUs as co-processors. In order to use a GPU on McCleary you must allocate a job on a partition with GPUs available and explicitly request GPU(s). Make sure to familiarize yourself with our documentation on scheduling jobs and requesting specific resources.
In addition to public partitions that give you access to GPUs, there are pi_cryoem and pi_tomo partitions which are limited to users of the Cryo-EM resources on campus. Please coordinate with the staff from West Campus and CCMI (See here for contact info) for access.
Software
Many Cryo-EM applications are meant to be viewed and interacted with in real-time. This mode of working is not ideal for the way most HPC clusters are set up, so where possible try to prototype a job you would like to run with a smaller dataset or subset of your data. Then develop a script to submit with sbatch.
RELION
The RELION pipeline operates in two modes. You can use it as a more familiar and beginner-friendly graphical interface, or call the programs involved directly. Once you are comfortable, using the commands directly in scripts submitted with sbatch will allow you to get the most work done the fastest.
The authors provide up-to-date hints about performance on their Benchmarks page. If you need technical help (jobs submit fine but having other issues) you should search and submit to their mailing list.
Module
We have GPU-enabled versions of RELION available on McCleary as software modules. To check witch versions are available, run module avail relion. To see specific notes about a particular install, you can use module help, e.g. module help RELION/4.0.0-fosscuda-2020b .
Example Job Parameters
RELION reserves one worker (slurm task) for orchestrating an MPI-based job, which they call the "master". This can lead to inefficient jobs where there are tasks that could be using a GPU but are stuck being the master process. You can request a better layout for your job with a heterogenous job, allocating CPUs on a cpu-only compute node for the task that will not use GPUs. Here is an example 3D refinement job submission script (replace choose_a_version with the version you want to use):
#!/bin/bash
#SBATCH --partition=general --ntasks 1 -c2 --job-name=class3D_hetero_01 --mem=10G --output="class3D_hetero_01-%j.out"
#SBATCH hetjob
#SBATCH --partition=gpu --ntasks 4 -c2 -N1 --mem-per-cpu=16G --gpus-per-task=1
module load RELION/choose_a_version
srun --pack-group=0,1 relion_refine_mpi --o hetero/refine3D/job0001 ... --dont_combine_weights_via_disc --j ${SLURM_CPUS_PER_TASK} --gpu
This job submission request will result in RELION using a single task/worker on a general purpose CPU node, and efficiently find four GPUs even if they aren't all available on the same compute node. Each GPU node task/worker will have a dedicated GPU, two CPU cores, and 30GiB total memory.
EMAN2
EMAN2 has always been a bit of a struggle for us to install properly on the clusters. Below are a few options
Conda Install
The EMAN2 authors offer some instructions on how to get EMAN2 running in a cluster environment on their install page. The default install may work as well if you avoid using MPI.
Container
At present, we have a mostly working apptainer container for EMAN2.3 available here:
/gpfs/ysm/datasets/cryoem/eman2.3_ubuntu18.04.sif
To run a program from EMAN2 using this container you would use a command like:
apptainer exec /gpfs/ysm/datasets/cryoem/eman2.3_ubuntu18.04.sif e2projectmanager.py
Cryosparc
We have a whole separate page about this one, it is a bit involved.
Other Software
We have CCP4, Phenix and some other software modules of interest installed. Run module avail and the software name to search for them. If you can't find one you need, please contact us.