Working with GPUs

There are two steps to successfully launching jobs to utilize one or more GPUs (assuming your program can do so):

  1. Submit your job to one of the GPU-enabled partitions (queues), e.g., gpu-standard, described previously
  2. Request one or more GPUs as a SLURM “generic resource”, e.g., in your script or on the job submission command line specify --gres=gpu:1. The 1 specifies the number of GPUs requested. You can further refine the “gres” specification with the specific model of GPU if needed for your application, e.g., --gres=gpu:rtxa6000:1

Once the job is running you can monitor GPU utilization with the following command, replacing <job id> with the id of your GPU job (as reported by squeue):

srun --overlap --jobid=<job id> --pty nvidia-smi

This runs the NVIDIA system management tool to report GPU memory and other usage statistics within the allocation of your already running job. The --overlap argument is critical to sharing the resources of that existing job. An example output is:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 565.57.01              Driver Version: 565.57.01      CUDA Version: 12.7     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX A6000               Off |   00000000:5E:00.0 Off |                  Off |
| 35%   63C    P2             88W /  300W |     918MiB /  49140MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A     40747      C   python                                        908MiB |
+-----------------------------------------------------------------------------------------+