Singularity

This article is a draft

This is not a complete article: This is a draft, a work in progress that is intended to be published into an article, which may or may not be ready for inclusion in the main wiki. It should not necessarily be considered factual or authoritative.

Overview

Singularity is open source software created by Berkeley Lab:

• as a secure way to use Linux containers on Linux multi-user clusters,
• as a way to enable users to have full control of their environment, and,
• as a way to package scientific software and deploy such to different clusters having the same architecture.

i.e., it provides operating-system-level virtualization commonly called containers.

A container is different from a virtual machine in that a container:

• likely has less overhead, and,
• can only run programs capable of running in the same operating system (i.e., Linux when using Singularity) for the same hardware architecture.

(Virtual machines can run different operating systems and sometimes support running software designed for foreign CPU architectures.)

Containers use Linux control groups (cgroups), kernel namespaces, and an overlay filesystem where:

• cgroups limit, control, and isolate resource usage (e.g., RAM, disk I/O, CPU access)
• kernel namespaces virtualize and isolate operating system resources of a group of processes, e.g., process and user IDs, filesystems, network access; and,
• overlay filesystems can be used to enable the appearance of writing to otherwise read-only filesystems.

Singularity is similar to other container solutions such as Docker except Singularity was specifically designed to enable containers to be used securely without requiring any special permissions especially on multi-user compute clusters.

Singularity Availability

Singularity is available on Compute Canada clusters (e.g., Cedar and Graham) and some legacy cluster systems run by various Compute Canada involved members/consortia across Canada.

Should you wish to use Singularity on your own computer systems, you will need to download and install Singularity per its documentation. You should be using a relatively recent version of some Linux distribution (e.g., ideally your kernel is v3.10.0 or newer).

Using Singularity On Compute Canada Systems

Module Loading

To use Singularity, first load the specific module you would like to use, e.g.,

$ module load singularity/2.4

Should you need to see all versions of Singularity modules that are available then run:

$ module spider singularity

Creating Images

Before using Singularity, you will first need to create a (container) image. A Singularity image is either a file or a directory containing an installation of Linux. One can create a Singularity image by any of the following:

• downloading a container from Singularity Hub
• downloading a container from Docker Hub
• from a container you already have,
• from a tarball or a directory containing an installation of Linux, or,
• from a Singularity recipe file.

Creating an Image Using Singularity Hub

Singularity Hub provides a search interface for pre-built images. Suppose you find one you want to use, for instance Ubuntu, then you would download the image by running:

$ singularity pull shub://singularityhub/ubuntu

Creating an Image Using Docker Hub

Docker Hub provides an interface to search for images.

Suppose the Docker Hub URL for a container you want is docker://ubuntu, then you would download the container by running:

$ singularity pull docker://ubuntu

Creating a Tarball of Your Own Linux System

If you already have a configured Intel-CPU-based 64-bit version of Linux installed, then you can create a tarball of your system using the tar similar to this:

$ sudo tar -cvpf -C / my-system.tar --exclude=/dev --exclude=/proc --exclude=/sys

although you may probably want to exclude additional directories.

The created tarball will need to be converted into a Singularity image which is discussed later on this page.

Creating an Image From a Tarball

If you have a tarball or a gzip-compressed tarball, a Singularity image can be made from it by using the Singularity build command:

$ sudo singularity build my-image.simg my-system.tar

if you are using your own system, or,

$ singularity build my-image.simg my-system.tar

if you are using a Compute Canada system.

The structure of the build command used to build an image from a tarball can be any one of the following:

singularity build IMAGE_FILE_NAME TARBALL_FILE_NAME
singularity build [OPTIONS] IMAGE_FILE_NAME TARBALL_FILE_NAME

The full syntax of the build command can be obtained by running:

$ singularity build --help

Singularity single-file images filenames typically have a .simg extension.

Creating an Image From a Singularity Recipe

NOTE: Singularity recipes require root permissions, thus, recipes can only be run on a computer where you can be the root user, e.g., your own Linux computer.

Recipe: Creating a Singularity Image of the Local Filesystem

If the following:

Bootstrap: self
Exclude: /boot /dev /home /lost+found /media /mnt /opt /proc /run /sys

is placed in a file, e.g., copy-drive-into-container-recipe then it can be used to copy one's Linux system directly into a container (except for the excluded directories listed) by running:

$ sudo singularity build self.simg copy-drive-into-container-recipe

(Clearly such has to be run on your own Linux system and Singularity must already be installed on that system.)

If you had the need to periodically re-generate your Singularity image from a script, then you might write a Singularity recipe such as this:

Bootstrap: localimage
From: ubuntu-16.04-x86_64.simg

%help
This is a modified Ubuntu 16.06 x86_64 Singularity container image.

%post
  sudo apt-get -y update
  sudo apt-get -y upgrade
  sudo apt-get -y install build-essential git
  sudo apt-get -y install python-dev python-pip python-virtualenv python-numpy python-matplotlib
  sudo apt-get -y install vim
  sudo apt-get clean

The above recipe allows one to update-regenerate a Singularity image from an existing Singularity image. In the above example, the recipe ensures all security updates are applied and that certain software programs are installed. If this script was in a file called update-existing-container-recipe and the image ubuntu-16.04-x86_64.simg already exists in the current directory, then the image can be updated by running:

$ sudo singularity build new-ubuntu-image.simg update-existing-container-recipe

Recipe: Creating a Singularity Image From a Docker URL

The following Singularity recipe will download the latest FEniCS docker image and then run a series of installation commands to install a number of Python packages:

File : FEniCS-From-Docker-With-Python-Tools-Singularity-Recipe

Bootstrap: docker
From: quay.io/fenicsproject/stable:latest

%post
  sudo apt-get -qq update
  sudo apt-get -y upgrade
  sudo apt-get -y install python-bitstring python3-bitstring
  sudo apt-get -y install python-certifi python3-certifi 
  sudo apt-get -y install python-cryptography python3-cryptography 
  sudo apt-get -y install python-cycler python3-cycler 
  sudo apt-get -y install cython cython3 
  sudo apt-get -y install python-dateutil python3-dateutil 
  sudo apt-get -y install python-deap python3-deap
  sudo apt-get -y install python-decorator python3-decorator
  sudo apt-get -y install python-ecdsa python3-ecdsa
  sudo apt-get -y install python-ecdsa python3-ecdsa
  sudo apt-get -y install python-enum34
  sudo apt-get -y install python-funcsigs python3-funcsigs
  sudo apt-get -y install ipython ipython3 python-ipython-genutils python3-ipython-genutils
  sudo apt-get -y install python-jinja2 python3-jinja2
  sudo apt-get -y install python-jsonschema python3-jsonschema
  sudo apt-get -y install python-lockfile python3-lockfile
  sudo apt-get -y install python-markupsafe python3-markupsafe
  sudo apt-get -y install python-matplotlib python3-matplotlib
  sudo apt-get -y install python-mistune python3-mistune
  sudo apt-get -y install python-mock python3-mock
  sudo apt-get -y install python-mpmath python3-mpmath
  sudo apt-get -y install python-netaddr python3-netaddr
  sudo apt-get -y install python-netifaces python3-netifaces
  sudo apt-get -y install python-nose python3-nose
  sudo apt-get -y install ipython-notebook ipython3-notebook
  sudo apt-get -y install python-numpy python3-numpy
  sudo apt-get -y install python-pandas python3-pandas
  sudo apt-get -y install python-paramiko python3-paramiko
  sudo apt-get -y install python-path python3-path
  sudo apt-get -y install python-pathlib
  sudo apt-get -y install python-pbr python3-pbr
  sudo apt-get -y install python-pexpect python3-pexpect
  sudo apt-get -y install python-pickleshare python3-pickleshare
  sudo apt-get -y install python-prompt-toolkit python3-prompt-toolkit
  sudo apt-get -y install python-ptyprocess python3-ptyprocess
  sudo apt-get -y install python-pycryptopp
  sudo apt-get -y install python-pygments python3-pygments
  sudo apt-get -y install python-pyparsing python3-pyparsing
  sudo apt-get -y install python-zmq python3-zmq
  sudo apt-get -y install python-requests python3-requests
  sudo apt-get -y install python-scipy python3-scipy
  sudo apt-get -y install python-setuptools python3-setuptools
  sudo apt-get -y install python-simplegeneric python3-simplegeneric
  sudo apt-get -y install python-singledispatch python3-singledispatch
  sudo apt-get -y install python-six python3-six
  sudo apt-get -y install python-sympy python3-sympy
  sudo apt-get -y install python-terminado python3-terminado
  sudo apt-get -y install python-tornado python3-tornado
  sudo apt-get -y install python-traitlets python3-traitlets
  sudo apt-get clean
  sudo rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

This recipe would be executed by running:

sudo singularity build an-image-name.simg FEniCS-From-Docker-With-Python-Tools-Singularity-Recipe

and illustrates how one can easily make new images at later points-in-time.

Creating/Updating an Image Interactively and Manually

This section is incomplete and will be completed later.

Is sudo Needed or Not Needed?

Notice the different between the two commands is whether or not sudo appears. The sudo command runs the command after it as the root user (i.e., superuser) of that system. On Compute Canada systems, no users have such access so the sudo command cannot be used there. Presumably you do have root access on your own computer so you can use sudo on it.

It is entirely possible that you will not need to use the sudo command with your image. If sudo is not used, then the following will happen when you build the image:

• Singularity will output a warning that such may result in an image that does not work. This message is only a warning though --the image will still be created.
• All filesystem permissions will be collapsed to be the permissions of the Linux user and group that is running singularity build. (This is normally the user and group you are logged in as.)

If sudo is used, then all filesystem permissions will be kept as they are in the tarball.

Typically one will not need to be concerned with retaining all filesystem permissions unless:

• one needs to regularly update/reconfigure the contents of the image, and,
• tools used to update/reconfigure the contents of the image require those permissions to be retained.

For example, many Linux distributions make it easy to update or install new software using commands such as:

• apt-get update && apt-get upgrade
• apt-get install some-software-package
• yum install some-software-package
• dnf install some-software-package
• etc.

It is possible that these and other commands may not run successfully unless filesystem permissions are retained. If this is of concern, then:

1. Install Singularity on your own computer.
2. Always build the Singularity image on your own computer using sudo.

If this is not a concern, then you may be able to build the Singularity image on a Compute Canada system without sudo, however, be aware that such might fail for any of the following reasons:

• When using Lustre filesystems, e.g., /project, you may run out of quota. If this occurs, it is likely because there are too many small files causing all of your quota to be used. (Lustre is excellent for large files but stores small files very inefficiently.)
• Sometimes image creation will fail due to various user restrictions placed on the node you are using. The login nodes, in particular, have a number of restrictions which may prevent one from successfully building an image.

If such occurs, then you will need to create your image using your own computer. If this is an issue, then request assistance to create the Singularity image you want by creating a Compute Canada ticket by sending an email to [1].

Using Singularity

NOTE: The discussion below does not describe how to use Slurm to run interactive or batch jobs --it only describes how to use Singularity. For interactive and batch job information see the Running jobs page.

Unlike perhaps when you created your Singularity image, you will never use, don't need to use, and cannot use sudo to run programs in your image on Compute Canada systems. There are a number of ways to run programs in your image:

1. Running commands interactively in one Singularity session.
2. Run a single command which executes and then stops running.
3. Run a container instance in order to run daemons which may have backgrounded processes.

Running Commands Interactively

This section is incomplete and will be completed later.

Running a Single Command

This section is incomplete and will be completed later.

Running Container Instances

This section is incomplete and will be completed later.

Bind Mounts

This section is incomplete and will be completed later.

HPC Issues With Singularity

This section is incomplete and will be completed later.

See Also

• SHARCNET General Interest Webinar, "Singularity", presented by Paul Preney on Feb. 14, 2018. See this YouTube Video as well as the SHARCNET Online Seminars page for slides.