RAPIDS
This is not a complete article: This is a draft, a work in progress that is intended to be published into an article, which may or may not be ready for inclusion in the main wiki. It should not necessarily be considered factual or authoritative.
Overview
RAPIDS is a suite of open source software libraries from NVIDIA, mainly for executing data science and analytics pipelines on GPUs. It relies on NVIDIA CUDA primitives for low level compute optimization and provides users with friendly Python APIs, similar to those in Pandas, Scikit-learn, etc.
Since RAPIDS is available as Conda packages which require having Anaconda for the installation, however Anaconda is not advised to use on the Compute Canada clusters. Instead, a container solution of using Singularity is recommended. As RAPIDS is also available as Docker container images from NVIDIA, and a Singularity image for RAPIDS can be built based from a Docker image.
This page provides the instructions for working with RAPIDS on Compute Canada clusters based from a Singularity container.
Build a Singularity image for RAPIDS
To build a Singularity image for RAPIDS the first thing to do is to find a Docker image for RAPIDS.
Where to look for a Docker image for RAPIDS
There are three types of RAPIDS Docker images, i.e. base, runtime, and devel types, and they are available at two major sites. For each type of Docker images, multiple images are provided with different combinations of RAPIDS versions and CUDA versions either in Ubuntu base or in CentOS base. You can find the Docker pull command of a selected image via the Tag tab on each given site:
- NVIDIA GPU Cloud (NGC): this site provides two types of RAPIDS images, i.e. base type and runtime type.
- base - contains a RAPIDS environment ready to use. Use this type of image if you want to submit a job to the Slurm scheduler.
- runtime - extends the base image by adding a Jupyter notebook server and example notebooks. Use this type of image if you want to interactively work with RAPIDS through notebooks and examples.
- Docker Hub: this site provides RAPIDS images in devel type.
- devel - contains the full RAPIDS source tree, the compiler toolchain, the debugging tools, the headers and the static libraries for RAPIDS development. Use this type of image if you want to implement any customized operations with low-level access to cuda-based processes.
Build a RAPIDS Singularity image
For example, if a docker pull command for a selected image is given as:
docker pull nvcr.io/nvidia/rapidsai/rapidsai:cuda11.0-runtime-centos7
On a computer that has Singularity supported, you can build a Singularity image, e.g. called rapids.sif, with following command based on the given pull tag:
[name@server ~]$ singularity build rapids.sif docker://nvcr.io/nvidia/rapidsai/rapidsai:cuda11.0-runtime-centos7
It usually takes half to one hour to complete the image building process. Since the image size is relatively large, you need to have enough memory and disk spaces on the server for building such an image.
Work on Clusters with a RAPIDS Singularity image
Once you have a Singularity image for RAPIDS located on Compute Canada clusters, you can work interactively by requesting an interactive session on a GPU node or submit a batch job to the Slurm queue if you have your RAPIDS code ready.
Explore the contents in RAPIDS
If simply exploring the contents without doing any computations, you can use following commands to access the container shell of the Singularity image, e.g. called rapids.sif on any node without requesting any GPUs.
Load the Singularity module first:
[name@server ~]$ module load singularity
Then access the container shell:
[name@server ~]$ singularity shell rapids.sif
The shell prompt is then changed to:
Singularity>
Inside the singularity shell initiate Conda and activate RAPIDS environment:
Singularity> source /opt/conda/etc/profile.d/conda.sh
Singularity> conda activate rapids
The shell prompt in the rapids env is then changed to:
(rapids) Singularity>
Then you can list available packages in the rapids env:
(rapids) Singularity> conda list
To deactivate rapids env and exit from the container:
(rapids) Singularity> conda deactivate
Singularity> exit
You are then back to the host shell.
Work interactively on a GPU node
If a Singularity image was built based on a runtime or a devel type of Docker image, it includes a Jupyter Notebook server and can be used to explore RAPIDS interactively on a compute node with GPU.
To request an interactive session on a compute node with a single GPU, e.g. a T4 type of GPU on Graham:
[name@gra-login ~]$ salloc --ntasks=1 --cpus-per-task=2 --mem=10G --gres=gpu:t4:1 --time=1:0:0 --account=def-someuser
Once the request is granted, start RAPIDS shell on the GPU node:
[name@gra#### ~]$ module load singularity
[name@gra#### ~]$ singularity shell --nv -B /home -B /project -B /scratch rapids.sif
Where --nv option is to bind mount the GPU driver on the host to the container, so the GPU device can be accessed from inside the singularity container. Use -B option to bind mound any file system that you would like to access in the container.
Submit a RAPIDS job to Slurm scheduler
You would need to prepare two script files to submit your RAPIDS code execution to the Slurm queue, i.e. one for job submission script and one for job execution script.
Here is an example of a job submission script, e.g. submit.sh:
#!/bin/bash
#SBATCH --ntasks=1
#SBATCH --gres=gpu:t4:1
#SBATCH --cpus-per-task=2
#SBATCH --mem=10G
#SBATCH --time=dd:hh:mm
#SBATCH --account=def-someuser
module load singularity
singularity run --nv -B /home -B /scratch rapids.sif /path/to/run_script.sh
Where --nv is to bind mount the GPU driver on the host to the container, so the GPU device can be accessed from inside the singularity container.
Here is an example of a job execution script, e.g. run_script.sh, which you would like to run in the container to start the execution of the python code that has been programed with RAPIDS:
#!/bin/bash
source /opt/conda/etc/profile.d/conda.sh
conda activate rapids
nvidia-smi
python /path/to/my_rapids_code.py
Helpful Links
- RAPIDS Docs: a collection of all the documentation for RAPIDS, how to stay connected and report issues.
- RAPIDS Notebooks: a collection of example notebooks on GitHub for getting started quickly.
- RAPIDS AI on Medium: a collection of use cases and blogs for RAPIDS applications.