Working with the SLURM Queuing System

The SLURM job scheduling system fairly and efficiently schedules computational tasks across the cluster’s worker nodes according to the resources requested by the job. Users interact with SLURM via a set of control commands, most commonly via sbatch for submitting jobs and squeue and sinfo for monitoring cluster status (check out this SLURM-provided cheatsheet for the commonly used options/resource specifiers). The most common use case is submitting “batch” jobs (i.e., a set of commands that run without user intervention), but you can also run “interactive” jobs.

See the pages below for details on submitting batch, array and interactive jobs.

Batch Compute Jobs

To submit a job, you typically write a job BASH script that specifies the command(s) to run and the resources required. An example job script, created as the file slurm_serial.sh, is shown below. Note the #SBATCH comments at the top that specify the resources (memory, time) required for this job (using the same command line options as sbatch). As described on the Getting Started page you would submit this job script with the command: sbatch slurm_serial.sh. SLURM will respond with the job ID.

#!/usr/bin/env bash

# SLURM template for serial jobs

# Set SLURM options
#SBATCH --job-name=serial_test      # Job name
#SBATCH --output=serial_test-%j.out # Output file incorporating job ID
#SBATCH --partition=standard        # Partition (queue)
#SBATCH --time=00:05:00             # Time limit hrs:min:sec
#SBATCH --mem=100mb                 # Job memory request

# Print SLURM environment variables
echo "Job ID: ${SLURM_JOB_ID}"
echo "Node: ${SLURMD_NODENAME}"

# Start of job info
echo "Starting: "`date +"%D %T"`

# Your calculations here
printf "\nHello world from ${SLURMD_NODENAME}!\n\n"

# End of job info
echo "Ending: "`date +"%D %T"`

Correctly configuring your job will require determining the relevant partition (queue) based on execution time and need for specialized resources (high memory, GPUs), number of nodes and cores, and memory per node. The table below shows the relevant partition for different workload types and time requirements. The requested partition and time, memory and other resource requirements need to match. For example, submitting a job requesting 2 days to the short partition, or more memory than is available on nodes in the standard partition, will never execute (and you may or may not get a corresponding error message). Review the Resources page for the number of nodes, cores and memory available for each class of machine.

Workload type Jobs ≤ 2 hours Jobs ≤ 2 days Jobs ≤ 7 days
CPU-based workloads needing < 256 GB of memory short standard long
GPU-based workloads gpu-short gpu-standard gpu-long
CPU-based workloads needing ≥ 256 GB of memory himem-short himem-standard himem-long

If your workload can take advantage of multiple CPU cores on a single node you can request additional cores with the --cpus-per-task option. For example, adding the following to the SLURM options at the top of your job script would request 8 CPU cores for the job. You can request up to as many cores as the nodes have (see the Resources page). All cores will be on the same node and share main memory, as if the job was running on a single stand-alone workstation.

#SBATCH --cpus-per-task=8  # Number of CPU cores for this job

Your code must be able to take advantage of the multiple CPU cores. If you request multiple cores for a purely serial program (which can only use 1 CPU core), such as most Python or R programs that don’t use specialized libraries like Torch, the additional CPU cores will remain idle (wasting resources that could be used by others).

To maximize throughput we suggest submitting your job to the shortest partition that allows enough execution time (with a buffer if the job takes longer than expected) and only requesting the number of cores and the amount of memory (with a small buffer for overage) you will actually use. If you don’t specify the required memory or time (mem or time), SLURM will use defaults of 4G (per requested core) and 2 hours. The “shorter” queues have higher priority. SLURM doesn’t over-allocate the worker nodes, even if jobs use less than the requested resources, so requesting just the needed resources will enable more of your (and others’) jobs to run at the same time.

If your workload can take advantage of GPUs, e.g., machine learning libraries, see Working with GPUs for details on submitting jobs that use GPUs.

Submitting Many Batch Jobs at Once with Array Jobs

If your workload consists of many independent parts, e.g., running the same program on different input files, you can increase throughput by running each instance as a separate job (that can execute in parallel). You can do so with multiple sbatch invocations, or more efficiently as an “array” job that will launch multiple instances of the same job script with unique identifiers.

  • To submit an array job, use the SLURM option --array. For example, --array=0-4 will run 5 independent tasks, labeled 0-4 by the environment variable SLURM_ARRAY_TASK_ID.
  • To allow each array task to perform a different calculation, you can use SLURM_ARRAY_TASK_ID as an input parameter to your calculation.
  • Each array task will appear as an independent job in the queue and execute independently.
  • An entire array job can be canceled at once or each task can be canceled individually.

An example array job script is shown below. Assuming this script is saved as slurm_array.sh, you can submit the job as sbatch slurm_array.sh.

#!/usr/bin/env bash

# SLURM template for array jobs

# Set SLURM options
#SBATCH --job-name=array_test         # Job name
#SBATCH --output=array_test-%A-%a.out # Output file incorporating job and array ID
#SBATCH --mem=100mb                   # Job memory request
#SBATCH --partition=standard          # Partition (queue)
#SBATCH --time=00:05:00               # Time limit hrs:min:sec
#SBATCH --array=0-4                   # Array range

# Print SLURM environment variables
echo "Job ID: ${SLURM_JOB_ID}"
echo "Array ID: ${SLURM_ARRAY_TASK_ID}"
echo "Node: ${SLURMD_NODENAME}"

# Start of job info
echo "Starting: "`date +"%D %T"`

# Your calculations here
printf "\nHello world from array task ${SLURM_ARRAY_TASK_ID}!\n\n"

# End of job info
echo "Ending: "`date +"%D %T"`

Interactive Jobs

If you have computationally intensive tasks that should run on the cluster’s worker nodes, but are still interactive in some way (e.g., you are working with large datasets, but are testing out commands or don’t yet know what you want to run), you can run an “interactive job”. For example, the following would run a 48 hour interactive job with 8G of memory. The srun command will wait until the job is successfully scheduled, at which point you will automatically be connected to a shell on the worker node. Use the exit command to end that shell and “return” to the head node.

srun --time=48:00:00 --partition=standard --mem=8G --pty bash -i