RAPIDS
This is not a complete article: This is a draft, a work in progress that is intended to be published into an article, which may or may not be ready for inclusion in the main wiki. It should not necessarily be considered factual or authoritative.
Overview
RAPIDS is a suite of open source software libraries from NVIDIA, mainly for executing data science and analytics pipelines on GPUs. It relies on NVIDIA CUDA primitives for low level compute optimization and provides users with friendly Python APIs, similar to those in Pandas, Scikit-learn, etc.
Since RAPIDS is available as Conda packages which require having Anaconda for the installation, however Anaconda is not advised to use on the Compute Canada clusters. Instead, a container solution of using Singularity is recommended. As RAPIDS is also available as Docker container images from NVIDIA, and a Singularity image for RAPIDS can be built based from a Docker image.
This page provides the instructions for working with RAPIDS on Compute Canada clusters based from a Singularity container.
Build a Singularity image for RAPIDS
To build a Singularity image for RAPIDS the first thing to do is to find a Docker image for RAPIDS.
Where to look for a Docker image for RAPIDS
There are two major sites where to look for a Docker image for RAPIDS.
- NVIDIA GPU Cloud (NGC): The RAPIDS Docker images are available on NGC in two types, i.e.
- base type - contains a RAPIDS environment ready to use. Use this type of image if you want to submit a job to the Slurm scheduler.
- runtime type - extends the base image by adding a Jupyter notebook server and example notebooks. Use this type of images if you want to interactively work with RAPIDS through notebooks and examples.
There are multiple images with different combinations of RAPIDS versions and CUDA versions with Ubuntu or CentOS base images for both types. You can find the Docker pull command of a selected image via the Tags tab on that page.
- Docker Hub: The RAPIDS Docker images on the rapidsai-dev repo on Docker Hub provides RAPIDS Docker images as devel type.
- devel - contains the full RAPIDS source tree, the compiler toolchain, the debugging tools, the headers and the static libraries for RAPIDS development. Use this type of images if you want to implement any customized operations with low-level access to cuda-based processes.