Using GPUs with Slurm/fr

Pour l'information générale sur l'ordonnancement des tâches, consultez Exécuter des tâches.

Nœuds disponibles

Le tableau suivant décrit les nœuds avec GPUs présentement disponibles sur Cedar et Graham.

# de nœuds	Type de nœud	Cœurs CPU	Mémoire CPU	# de GPUs	Type de GPU	Topologie du bus PCIe
114	GPU base, Cedar	24	128Go	4	NVIDIA P100-PCIE-12GB	deux GPUs par socket CPU
32	GPU large, Cedar	24	256Go	4	NVIDIA P100-PCIE-16GB	tous les GPUs sous le même socket CPU
160	GPU base, Graham	32	128Go	2	NVIDIA P100-PCIE-12GB	un GPU par socket CPU

Tâche avec cœur unique

Pour une tâche qui nécessite un seul cœur CPU et un GPU,

File : gpu_serial_job.sh

#!/bin/bash
#SBATCH --account=def-someuser
#SBATCH --gres=gpu:1              # Number of GPUs (per node)
#SBATCH --mem=4000M               # memory (per node)
#SBATCH --time=0-03:00            # time (DD-HH:MM)
./program

Tâches multifils

Pour une tâche GPU qui nécessite plusieurs CPU dans un seul nœud,

File : gpu_threaded_job.sh

#!/bin/bash
#SBATCH --account=def-someuser
#SBATCH --gres=gpu:1              # Number of GPU(s) per node
#SBATCH --cpus-per-task=6         # CPU cores/threads
#SBATCH --mem=4000M               # memory per node
#SBATCH --time=0-03:00            # time (DD-HH:MM)
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
./program

Les tâches multifils ne devraient pas dépasser

• avec Cedar, 6 cœurs CPU pour chaque GPU;
• avec Graham, 16 cœurs pour chaque GPU.

Tâches MPI

File : gpu_mpi_job.sh

#!/bin/bash
#SBATCH --account=def-someuser
#SBATCH --gres=gpu:4              # Number of GPUs per node
#SBATCH --nodes=2                 # Number of nodes
#SBATCH --ntask=48                # Number of MPI process
#SBATCH --cpus-per-task=1         # CPU cores per MPI process
#SBATCH --mem=120G                # memory per node
#SBATCH --time=0-03:00            # time (DD-HH:MM)
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
srun ./program

Nœuds entiers

If your application can efficiently use an entire node and its associated GPUs, you will probably experience shorter wait times if you ask Slurm for a whole node. Use one of the following job scripts as a template.

Ordonnancement d'un nœud GPU pour Graham

File : graham_gpu_node_job.sh

#!/bin/bash
#SBATCH --nodes=1
#SBATCH --gres=gpu:2
#SBATCH --ntasks-per-node=32
#SBATCH --mem=128000M
#SBATCH --time=3:00
#SBATCH --account=def-someuser
nvidia-smi

Ordonnancement d'un nœud GPU pour Cedar

File : cedar_gpu_node_job.sh

#!/bin/bash
#SBATCH --nodes=1
#SBATCH --gres=gpu:4
#SBATCH --exclusive
#SBATCH --mem=125G
#SBATCH --time=3:00
#SBATCH --account=def-someuser
nvidia-smi

Scheduling a Large GPU node at Cedar

There is a special group of large-memory GPU nodes at Cedar which have four Tesla P100 16GB cards each. (Other GPUs in the cluster have 12GB.) These GPUs all use the same PCI switch so the inter-GPU communication latency is lower, but bandwidth between CPU and GPU is lower than on the regular GPU nodes. The nodes also have 256 GB RAM instead of 128GB. In order to use these nodes you must specify lgpu. By-gpu requests can only run up to 24 hours.

File : large_gpu_job.sh

#!/bin/bash
#SBATCH --nodes=1 
#SBATCH --gres=gpu:lgpu:4   
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=24    # There are 24 CPU cores on Cedar GPU nodes
#SBATCH --time=3:00
#SBATCH --account=def-someuser
hostname
nvidia-smi

Packing single-GPU jobs within one SLURM job

If user needs to run 4 x single GPU codes or 2 x 2-GPU codes in a node for longer than 24 hours, GNU Parallel is recommended. A simple example is given below:

cat params.input | parallel -j4 'CUDA_VISIBLE_DEVICES=$(({%} - 1)) python {} &> {#}.out'

GPU id will be calculated by slot id {%} minus 1. {#} is the job id, starting from 1.

A params.input file should include input parameters in each line like:

code1.py
code2.py
code3.py
code4.py
...

With this method, users can run multiple codes in one submission. In this case, GNU Parallel will run a maximum of 4 jobs at a time. It will launch the next job when one job is finished. CUDA_VISIBLE_DEVICES is used to force using only 1 GPU for each code.

Message Passing Interface