Run Jobs with Slurm
Submitting a job involves specifying a resource request then running one or more commands or applications. These requests take the form of options to the command-line programs
sbatch or those same options as directives inside submission scripts. Requests are made of groups of compute nodes (servers) called partitions. Partitions, their defaults, limits, and purposes are listed on each cluster page. Once submitted, jobs wait in a queue and are subject to several factors affecting scheduling priority. When your scheduled job begins, the commands or applications you specify are run on compute nodes the scheduler found to satisfy your resource request. If the job was submitted as a batch job, output normally printed to the screen will be saved to file.
Please be a good cluster citizen.
- Do not run heavy computation on login nodes (e.g.
farnam2). Doing so negatively impacts everyone's ability to interact with the cluster.
- Make resource requests for your jobs that reflect what they will use. Wasteful job allocations slow down everyone's work on the clusters. See our documentation on Monitoring CPU and Memory Usage for how to measure job resource usage.
- If you plan to run many similar jobs, use our Dead Simple Queue tool or job arrays - we enforce limits on job submission rates on all clusters.
If you find yourself wondering how best to schedule a job, please contact us for some help.
Common Slurm Commands
For an exhaustive list of commands and their official manuals, see the SchedMD Man Pages. Below are some of the most common commands used to interact with the scheduler.
Submit a script called
my_job.sh as a job (see below for details):
List your queued and running jobs:
Cancel a queued job or kill a running job, e.g. a job with ID 12345:
Check status of a job, e.g. a job with ID 12345:
sacct -j 12345
Check how efficiently a job ran, e.g. a job with ID 12345:
Common Job Request Options
These options modify the size, length and behavior of jobs you submit. They can be specified when calling
sbatch, or saved to a batch script. Options specified on the command line to
sbatch will override those in a batch script. See our Request Compute Resources page for discussion on the differences between
--cpus-per-task, constraints, GPUs, etc. If options are left unspecified defaults are used.
|Long Option||Short Option||Default||Description|
||Name of script||Custom job name.|
||Where to save
||Varies by cluster||Partition to run on. See individual cluster pages for details.|
||Your group name||Specify if you have access to multiple private partitions.|
||Varies by partition||Time limit for the job in D-HH:MM:SS, e.g.
||Total number of nodes.|
||Number of tasks (MPI workers).|
||Scheduler decides||Number of tasks per node.|
||Number of CPUs for each task. Use this for threads/cores in single-node jobs.|
||Memory requested per CPU in MiB. Add
||Memory requested per node in MiB. Add
||Used to request GPUs|
||Constraints on node features. To limit kinds of nodes to run on.|
||Your Yale email||Mail address (alternatively, put your email address in ~/.forward).|
||None||Send email when jobs change state. Use
Interactive jobs can be used for testing and troubleshooting code. Requesting an interactive job will allocate resources and log you into a shell on a compute node.
You can start an interactive job using the
salloc command. Unless specified otherwise using the
-p flag (see above), all
salloc requests will go to the
interactive partition on the cluster.
For example, to request an interactive job with 8GB of RAM for 2 hours:
salloc -t 2:00:00 --mem=8G
This will assign one CPU and 8GiB of RAM to you for two hours. You can run commands in this shell as needed. To exit, you can type
exit or Ctrl+d
tmux with Interactive Sessions
Remote sessions are vulnerable to being killed if you lose your network connection. We recommend using
tmux alleviate this. When using
tmux with interactive jobs, please take extra care to stop jobs that are no longer needed.
Many graphical applications are well served with the Open OnDemand Remote Desktop app. If you would like to use X11 forwarding, first make sure it is installed and configured. Then, add the
--x11 flag to an interactive job request:
You can submit a script as a batch job, i.e. one that can be run non-interactively in batches. These submission scripts are comprised of three parts:
- A hashbang line specifying the program that runs the script. This is normally
- Directives that list job request options. These lines must appear before any other commands or definitions, otherwise they will be ignored.
- The commands or applications you want executed during your job.
#!/bin/bash #SBATCH --job-name=example_job #SBATCH --time=2:00:00 #SBATCH --mail-type=ALL module purge module load MATLAB/2021a matlab -b "your_script"
Save this file as
example_job.sh, then submit it with:
When the job finishes the output should be stored in a file called
jobid is the submitted job's ID. If you find yourself writing loops to submit jobs, instead use our Dead Simple Queue tool or job arrays.