Singularity

Overview

Singularity is open source software created by Berkeley Lab:

• as a secure way to use Linux containers on Linux multi-user clusters,
• as a way to enable users to have full control of their environment, and,
• as a way to package scientific software and deploy such to different clusters having the same architecture.

i.e., it provides operating-system-level virtualization commonly called containers.

A container is different from a virtual machine in that a container:

• likely has less overhead, and,
• can only run programs capable of running in the same operating system kernel (i.e., Linux when using Singularity) for the same hardware architecture.

(Virtual machines can run different operating systems and sometimes support running software designed for foreign CPU architectures.)

Containers use Linux control groups (cgroups), kernel namespaces, and an overlay filesystem where:

• cgroups limit, control, and isolate resource usage (e.g., RAM, disk I/O, CPU access)
• kernel namespaces virtualize and isolate operating system resources of a group of processes, e.g., process and user IDs, filesystems, network access; and,
• overlay filesystems can be used to enable the appearance of writing to otherwise read-only filesystems.

Singularity is similar to other container solutions such as Docker except Singularity was specifically designed to enable containers to be used securely without requiring any special permissions especially on multi-user compute clusters.

Availability

Singularity is available on the Compute Canada clusters.

Should you wish to use Singularity on your own computer, you will need to download and install it per its documentation. You should be using a relatively recent version of some Linux distribution (e.g., ideally your kernel is v3.10.0 or newer).

Singularity on Compute Canada clusters

Loading a module

To use Singularity, first load the specific module you would like to use, e.g.,

$ module load singularity/2.5

Should you need to see all versions of Singularity modules that are available then run:

$ module spider singularity

Creating images

Before using Singularity, you will first need to create a (container) image. A Singularity image is either a file or a directory containing an installation of Linux. One can create a Singularity image using the singularity build command, e.g.,

• singularity build WHAT-TO-WRITE SPECIAL-URI-OR-PATH

where WHAT-TO-WRITE is:

• a filename of the singularity image file (*.sif) where the built image will be written
• a directory if one is building a sandbox using the --sandbox option typically on one's own (Linux) computer (requiring root account access)

and SPECIAL-URI-OR-PATH is:

• a URI starting with library:// to build from a Container Library,
• a URI starting with docker:// to build from Docker Hub,
• a URI starting with shub:// to build from Singularity Hub,
• a path to an existing container,
• a path to a directory (to build from a sandbox), or
• a path to a Singularity image file (which is a recipe on how to build the image).

Warning

You are strongly advised to create Singularity images using a computer or virtual machine

• that runs Linux
• with Singularity installed
• with Internet access,
• ideally where you have root, e.g., sudo, permissions.

If you do not have root permissions to create your images, be aware that all permissions (user and group) inside the image will be set to those of the account the image is made under. In general, it is not easy to re-establish proper permissions, thus, depending on the image content, you may not be able to upgrade it later.

For example, Debian and Ubuntu images' dpkg, apt-get, and apt commands all require root to upgrade/install packages. Thus, if the ability to install and upgrade software in the future is important, then create the image on a Linux system where you have root permissions.

NOTE: Any image you create on your own computer needs to be uploaded to the cluster before you can use that image.

Build error when building on a Linux distribution

When building on certain Linux distributions, you may get this error despite having enough memory on your device.

$ WARNING: 'nodev' mount option set on /tmp, it could be a source of failure during build process
$FATAL: no memory left on device

Many modern Linux distributions, use an in-memory tmpfs filesystem, which may limit the size of the container you can build. The Singularity Documentation gives a more detailed explanation.

If that is the case, you can solve the problem by setting SINGULARITY_TMPDIR to a disk location.

$ SINGULARITY_TMPDIR="disk/location" singularity build IMAGE_NAME.sif docker://DOCKER-IMAGE-NAME

Creating images on Compute Canada clusters

If you decide to create an image on a Compute Canada cluster, be aware of the fact that you will never have sudo access and so the caveats of the previous section apply. Images can be created on any of the Compute Canada clusters or visualization computers, e.g., gra-vdi.computecanada.ca. Our image creation advice differs depending on which machine you use:

• beluga.computecanada.ca: Connect using SSH. Use a login node to create the image.
• cedar.computecanada.ca: Connect using SSH Create the image in an interactive job. Do not use a login node.
• graham.computecanada.ca: Connect using SSH Use a login node to create the image.
• gra-vdi.computecanada.ca: Connect using VNC. Use a terminal window to create the image.
• niagara.computecanada.ca: Connect using SSH Use a login node to create the image.
  • IMPORTANT: Do not bind to /localscratch on Niagara as it does not exist!

Among these options, gra-vdi.computecanada.ca is the best choice for creating images. The others have issues such as:

• beluga, graham, and niagara:
  • fix the maximum amount of RAM that can be used on login nodes, and,
  • there is no Internet access on compute nodes.
• cedar:
  • login nodes cannot be used and,
  • compute node jobs require specifying the amount of RAM needed but this is very difficult to know what is required in advance, thus if an error occurs, exit the interactive job and try again requesting more memory.

Creating an image using Docker Hub and Dockerfile

Docker Hub provides an interface to search for images. Docker also requires authentication at times to download images. If, when using Singularity, you get errors similar to these:

• requested access to the resource is denied,
• unauthorized: authentication required,

then this might have occurred for either or both of these reasons:

• your Docker URL is incorrect, and/or,
  • NOTE: You can verify the Docker URL by using Docker Hub and searching for the specific image you want.
• you need to authenticate with Docker in order to download the image.
  • e.g., you need to add --docker-login after the Singularity build or pull command.

Suppose the Docker Hub URL for a container you want is docker://ubuntu, then you would download the container by running:

$ singularity build myubuntuimage.sif docker://ubuntu

This command will work on a Cedar compute node but not on Béluga or Graham, where compute nodes can not access the internet. We don't recommend building large Singularity images on login nodes because this process will affect other users and likely fail due to the high CPU and RAM usage. The following two-step process can solve the problem.

1. Download a Docker image without converting it to Singularity. The download-frozen-image tool from the Moby Project can be used to do this. This step is not computationally intensive and can be done on any data transfer or login node.
2. Convert the downloaded Docker image to Singularity image. This step is computationally intensive and so should be done on a compute node with a suitable allocation of memory and CPU.

As an example, let's download and build a Singularity image for the package GQCP.

On a data transfer node or a login node download the Docker image:

$ cd ~/scratch
$ wget https://raw.githubusercontent.com/moby/moby/master/contrib/download-frozen-image-v2.sh
$ sh download-frozen-image-v2.sh gqcp gqcg/gqcp:latest
$ cd gqcp && tar cf ../gqcp.tar * && cd ..

Start an interactive job:

$ cd ~/scratch
$ salloc --mem-per-cpu=2000 --cpus-per-task=4 --time=2:0:0

On the allocated compute node, build the Singularity image:

$ module load singularity
$ singularity build gqcp.sif docker-archive://gqcp.tar

One can also use a Dockerfile to create Singularity images, but this is a two step process.

• The first step is to create a Docker image with the Dockerfile. This step should be executed on your VM or Linux machine, where you have sudo access. Go to the directory which contains the Dockerfile and run the command sudo docker build -t CC:latest .. By passing '.' as a parameter we are asking Docker to search for the Dockerfile in the current directory. This command will create a Docker image with the name CC and assign the image a tag with the value latest. Remember that latest is the default tag that Docker assigns to an image, so if you don't provide a tag, the image will use the default tag value latest.
• The second step is to use the Docker image created above as the source for Singularity. The following command will create a Singularity image cc_image_latest.sif from the Docker image CC:latest,

sudo singularity build cc_image_latest.sif docker-daemon://CC:latest

You don't need to provide a path to the Docker image as the Docker daemon will automatically pick the image specified though the Docker registry, but we have to ensure that we use the correct image name and tag value.

Creating a custom image with sandbox

The --sandbox flag allows you to create a container within a writable directory. this allows for easy modifications of existing containers.

$ singularity build --sandbox IMAGE_NAME/ docker://DOCKER-IMAGE-NAME

Changes can be made within the container with the --writable flag.

$ singularity shell --writable IMAGE_NAME/

A sandbox directory can be converted into a sif file with the build command.

$ singularity build production.sif IMAGE_NAME/

Is sudo needed or not needed?

The Singularity documentation on building a container uses the sudo command. This is because, in general, many uses of the build command requires root, i.e., superuser, access on the system it is run. On Compute Canada clusters, regular users do not have root account access so the sudo command cannot be used. If you are building a pre-built image from Singularity or Docker Hub, you typically will not need sudo access. If you do need root access to build an image, then you will either need to ask Compute Canada persons for help, or, install Linux and Singularity on your own computer to have root account access.

Many users don't need to use the sudo command to build their images from Singularity or Docker Hub. If sudo is not used, then the following will happen when you build the image:

• Singularity will output a warning that such may result in an image that does not work. This message is only a warning though --the image will still be created.
• All filesystem permissions will be collapsed to be the permissions of the Linux user and group that is running singularity build. (This is normally the user and group you are logged in as.)

Typically one will not need to be concerned with retaining all filesystem permissions unless:

• one needs to regularly update/reconfigure the contents of the image, and,
• tools used to update/reconfigure the contents of the image require those permissions to be retained.

For example, many Linux distributions make it easy to update or install new software using commands such as:

• apt-get update && apt-get upgrade
• apt-get install some-software-package
• yum install some-software-package
• dnf install some-software-package
• etc.

It is possible that these and other commands may not run successfully unless filesystem permissions are retained.

Sometimes image creation will fail due to various user restrictions placed on the node you are using. The login nodes, in particular, have a number of restrictions which may prevent one from successfully building an image. If this is an issue, then request assistance to create the Singularity image by contacting Technical support.

Using Singularity

NOTE: The discussion below does not describe how to use Slurm to run interactive or batch jobs --it only describes how to use Singularity. For interactive and batch job information see the Running jobs page.

Unlike perhaps when you created your Singularity image, you will never use, don't need to use, and cannot use sudo to run programs in your image on Compute Canada clusters. There are a number of ways to run programs in your image:

1. Running commands interactively in one Singularity session.
2. Run a single command which executes and then stops running.
3. Run a container instance in order to run daemons which may have backgrounded processes.

Running commands interactively

Singularity can be used interactively by using its shell command, e.g.,

$ singularity shell --help

will give help on shell command usage. The following:

$ singularity shell -B /home -B /project -B /scratch -B /localscratch myimage.simg

will do the following within the container image myimage.simg:

• bind mount /home so that all home directories can be accessed (subject to your account's permissions)
• bind mount /project so that project directories can be accessed (subject to your account's permissions)
• bind mount /scratch so that the scratch directory can be accessed (subject to your account's permissions)
• bind mount /localscratch so that the localscratch directory can be accessed (subject to your account's permissions)
• run a shell (e.g., /bin/bash)

If this command is successful, you can interactively run commands from within your container while still being able to access your files in home, project, scratch, and localscratch. :-)

• NOTE: When done, type "exit" to exit the shell.

In some cases, you will not want the pollution of shell environment variables from your Compute Canada shell. You can run a "clean environment" shell by adding a -e option, e.g.,

$ singularity shell -e -B /home -B /project -B /scratch -B /localscratch myimage.simg

but know you may need to define some shell environment variables such as $USER.

Finally, if you are using Singularity interactively on your own machine, in order for your changes to the image to be written to the disk, you must:

• be using a Singularity "sandbox" image (i.e., be using a directory not the read-only .simg file)
• be using the -w option, and,
• be using sudo

e.g., first create your sandbox image:

$ sudo singularity build -s myimage-dir myimage.simg

and then engage with Singularity interactively:

$ sudo singularity shell -w myimage-dir

When done, you can build a new/updated simg file, with the command:

$ sudo singularity build myimage-new.simg myimage-dir/

and upload myimage-new.simg to a cluster in order to use it.

Running a single command

When submitting jobs that invoke commands in Singularity containers, one will either use Singularity's exec or run commands.

• The exec command does not require any configuration.
• The run command requires configuring an application within a Singularity recipe file and this is not discussed on this page.

The Singularity exec command's options are almost identical to the shell command's options, e.g.,

$ singularity exec --help

When not asking for help, the exec command runs the command you specify within the container and then leaves the container, e.g.,

$ singularity exec -B /home -B /project -B /scratch -B /localscratch myimage.simg ls /

which will output the contents of the root directory within the container. The version of ls is the one installed within the container! For example, should GCC's gcc be installed in the myimage.simg container, then this command:

$ singularity exec -B /home -B /project -B /scratch -B /localscratch myimage.simg gcc -v

will output the version information of what is installed within the container whereas running at the normal Compute Canada shell prompt:

$ gcc -v

will output the version of GCC currently loaded on Compute Canada clusters.

If you need to run a single command from within your Singularity container in a job, then the exec command will suffice. Remember to bind mount the directories you will need access to in order for your job to run successfully.

Singularity can also accept user defined environment variables. This can be achieved via --env-file flag. We pass path of the file that define all the variables which we want to use inside the singularity context as an argument to the --env-file flag. The --env-file flag is especially useful for the scenarios when we want to use the -e flag with singularity exec which forces a "clean environment".

For example, if we want to use the PYSPARK_SUBMIT_ARGS environment variable inside the singularity context, we will create a file, for example envfile, and define the variable PYSPARK_SUBMIT_ARGS in it. We can define more than one variable inside the file.

PYSPARK_SUBMIT_ARGS='--driver-memory 96g  --driver-java-options "-verbose:gc  -XX:+UseSerialGC  -XX:-UseGCOverheadLimit" pyspark-shell'

And then we pass the path of file to the singularity command via --env-file flag.

$ singularity exec --env-file envfile  myimage.simg gcc -v

In the above example we assume that envfile is present at the same location from where we execute the singularity command and thus we passed just the filename rather than the path to the file.

Running container instances

Should you need to run daemons and backgrounded processes within your container, then do not use the Singularity exec command! Instead you want to use Singularity's instance.start and instance.stop commands to create and destroy sessions (i.e., container instances). By using sessions, Singularity will ensure that all programs running within the instance are terminated when your job ends, unexpectedly dies, is killed, etc.

To start a Singularity session instance, decide on a name for this session, e.g., quadrat5run, and run the instance.start command specifying the image name, e.g., myimage.simg, and your session name:

$ singularity instance.start myimage.simg quadrat5run

A session (and all associated programs that are running) can be stopped (i.e., destroyed/killed) by running the instance.stop command, e.g.,

$ singularity instance.stop myimage.simg quadrat5run

At any time you can obtain a list of all sessions you currently have running by running:

$ singularity instance.list

which will list the daemon name, its PID, and the path to the container's image.

With a session started, programs can be run using Singularity's shell, exec, or run commands by specifying the name of the session immediately after the image name prefixed with instance://, e.g.,

$ singularity instance.start mysessionname
$ singularity exec myimage.simg instance://mysessionname ps -eaf
$ singularity shell myimage.simg instance://mysessionname 
nohup find / -type d >dump.txt
exit
$ singularity exec myimage.simg instance://mysessionname ps -eaf
$ singularity instance.stop mysessionname

Bind mounts

When running a program within a Singularity container, by default, it can only see the files within the container image and the current directory. Realistically your Singularity jobs will need to mount the various filesystems where your files are. This is done using the -B option to the Singularity shell, exec, or run commands, e.g.,

$ singularity shell -B /home -B /project -B /scratch -B /localscratch myimage.simg

$ singularity exec -B /home -B /project -B /scratch -B /localscratch myimage.simg ls /

$ singularity run -B /home -B /project -B /scratch -B /localscratch:/temp myimage.simg some-program

The previous three commands show how to bind mount the various filesystems on Compute Canada's clusters, i.e., within the container image myimage.simg these commands bind mount:

• /home so that all home directories can be accessed (subject to your account's permissions)
• /project so that project directories can be accessed (subject to your account's permissions)
• /scratch so that the scratch directory can be accessed (subject to your account's permissions)
• /localscratch so that the localscratch directory can be accessed (subject to your account's permissions)

In the final item we demonstrate how we can change the mount name that is visible within the Singularity container, so that /localscratch will be seen as /temp inside the container. This can be useful when you want to use storage like $SLURM_TMPDIR which is attached directly to the compute node and available inside a job. The program itself can abstract away the details by always using /temp.

In most cases, it is not recommended to directly mount each directory you need as this can cause access issues. Instead, mount the top directory of the filesystem as shown above.

HPC issues

Running MPI programs from within a container

If you are running MPI programs nothing special needs to be done for jobs running on a single node.

To run jobs across nodes with MPI requires:

• Ensuring your MPI program is compiled using the OpenMPI installed inside your Singularity container.
  • Ideally the version of OpenMPI inside the container is version 3 or 4. Version 2 may or may not work. Version 1 will not work.
• Ensuring your SLURM job script uses srun to run the MPI program. Do not use mpirun or mpiexec, e.g.,

srun singularity exec /path/to/your/singularity/image.sif /path/to/your-program

• Ensure there are no module load commands in your job script.
• Install the slurm-client package from your distribution within your container to enable interaction(s) with the Slurm scheduler.
• Before submitting the job using sbatch, in the CC shell environment, module load the following:
  • singularity
  • openmpi (This does not need to match the OpenMPI version installed inside the container. Ideally use version 4 or version 3; version 2 may or may not work; version 1 will not work.)

Finally, ensure that a high performance interconnect package is also installed in your image, i.e.,

• clusters using OmniPath need libpsm2, and,
• clusters using Infiniband need UCX.

Using CUDA on Compute Canada clusters

If you are running programs which use the CUDA library, you must make sure that your Singularity container has CUDA installed.

The --nv flag needs to be added to make CUDA work.

$ srun singularity run --nv container-name.sif [job to do]

Changing Singularity default directories

You can override Singularity's default temporary and cache directories by setting these environment variables before running singularity:

• SINGULARITY_CACHEDIR: the directory where singularity will download (and cache) files
• SINGULARITY_TMPDIR: the directory where singularity will write temporary files including when building (squashfs) images

For example, to tell singularity to use your scratch space for its cache and temporary files, one might run:

$ mkdir -p /scratch/$USER/singularity/{cache,tmp}
$ export SINGULARITY_CACHEDIR="/scratch/$USER/singularity/cache"
$ export SINGULARITY_TMPDIR="/scratch/$USER/singularity/tmp"

before running singularity.

See also

• SHARCNET General Interest Webinar, "Singularity", presented by Paul Preney on Feb. 14, 2018. See this YouTube Video as well as the SHARCNET Online Seminars page for slides.

GNU Compiler Collection, an open source compiler collection

Message Passing Interface