Apptainer

This article is a draft

This is not a complete article: This is a draft, a work in progress that is intended to be published into an article, which may or may not be ready for inclusion in the main wiki. It should not necessarily be considered factual or authoritative.

Notice

This page is draft work-in-progress. At the time of writing (August 2022) Apptainer version 1.1 is in release candidate status and the full release may well induce further updates to this page. (This page's current content is correct for Apptainer version 1.0.)

If Currently Using Singularity

We strongly recommend that you start using Apptainer instead of Singularity. SingularityCE (up to v3.9.5) was adopted by The Linux Foundation and renamed to Apptainer with these changes:

• added experimental support for DMTCP checkpointing,
  • NOTE: Support for such is not in Singularity.
• removed support for the --nvccli command line option,
• removed support for apptainer build --remote,
• removed support the SylabsCloud remote endpoint replacing it with a DefaultRemote endpoint with no defined server for library://,
  • NOTE: If the SylabsCloud remote is needed, the previous default can be restored.
• renamed all executable names, paths, etc. having singularity in their names to have apptainer in them,
  • e.g., instead of using the singularity command one uses the apptainer command
  • e.g., the ~/.singularity directory is now ~/.apptainer
• renamed all environment variables having SINGULARITY in their names to have APPTAINER in them,

Should you need to port scripts, etc. to Apptainer, know Apptainer version 1 is backwards compatible with Singularity so switching to Apptainer can be done incrementally.

Using Apptainer

In order to use Apptainer one must first already have a container image, e.g., a .sif file or a "sandbox" directory created previously. If you don't already have a container image, see the section on building an image below.

Loading an Apptainer Module

In order to use the default version of Apptainer available run:

$ module load apptainer

To see the available versions of Apptainer that can be loaded run:

$ module spider apptainer

Running Programs Within a Container

Important Items

sudo Access With Apptainer is NOT Available

Many users ask about this since documentation and web sites do discuss using using sudo with Apptainer/Singularity. A major design aspect of Apptainer is security and know that sudo access is NOT available on our clusters. Should you require using sudo, consider the following:

• Install Linux, Apptainer, and sudo in a virtual machine on a system you control where so you will be able to have sudo access within such. (Some images cannot be successfully built without using sudo.)
• If appropriate, submit a ticket asking if Alliance staff would be able to help build the image(s), etc. required needing sudo. (Understand that this may or may not be possible --but ask in a ticket if what you wish to achieve is beyond your means: there may be other ways to achieve those needs.)
• Apptainer version 1.1 (which has not yet been released as of this writing (August 2022)), may well allow users to use the --fakeroot option with various Apptainer commands. After such has been released, this web page will be updated after the feature has been explored more, tested, and installed on our clusters.

Important Command Line Options

Containers contain software (very likely) installed with a different environment, libraries, and tools than what is installed on a given system. It is, therefore, wise to run software within containers by not importing any environment settings or software defined outside of the container. By default Apptainer will run adopting the shell environment of the host which can create issues when executing programs within a container. When running Apptainer commands that execute programs installed inside the container, e.g., apptainer run, apptainer shell, apptainer exec, and apptainer instance, do consider using one of these options (with more preference to those options listed above other options):

Apptainer Environment Command Line Options

Option	Description
-C	Isolates the running container from all file systems as well as the parent PID, IPC, and environment.
-c	Isolates the running container from most file systems only using a minimal /dev, an empty /tmp directory, and an empty /home directory.
-e	Cleans (some) shell environment variables before running container commands and applies settings for increased OCI/Docker compatibility. Using this option also implies the use of these options: --containall, --no-init, --no-umask, --writable-tmpfs.

Not using any of these options will have the convenience of file systems being mounted and shell variables existing inside your container but be aware such can be the source of issues/errors when running software inside your container.

Additionally there is another option, -W or --workdir, one should consider using. On our clusters (and on most Linux systems), the /tmp and similar filesystems use RAM --not disk space-- and jobs are run on our clusters typically with limited RAM requests. This can result in jobs getting killed because they consume too much RAM (relative to what was requested). A suitable work-around for this is to tell Apptainer to use a "workdir" for such filesystems, $HOME (if -c is also used), etc. This is done by passing the -W option followed by a path to a disk space location where Apptainer can read/write temporary files, etc.

Using GPUs

When running software inside a container that requires the use of GPUs it is important to do the following:

• Ensure that you pass the --nv (for NVIDIA hardware) and --rocm (for AMD hardware) to Apptainer commands.
  • These options will ensure the appropriate /dev entries are bind mounted inside the container.
  • These options will locate and bind GPU-related libraries on the host (e.g., so such becomes bind-mounted inside the container) as well as setting the LD_LIBRARY_PATH environment variable to enable the aforementioned libraries will work inside the container.
• Ensure the application using the GPU inside the container was properly compiled to use the GPU and its libraries.
• When needing to use OpenCL inside the container, besides using the aforementioned options use the following bind mount: --bind /etc/OpenCL.

Using MPI Programs

If you want to run MPI programs inside a container there are things that need to be done in the host environment in order for such to work. Please see the Running MPI Programs section below for more detail.

Container-Specific Help: apptainer run-help

Text to come.

Running Software (Preferred): apptainer run

Text to come.

Interactively Running Software: apptainer shell

Text to come.

Running Software (Basic): apptainer exec

Text to come.

Running Daemons: apptainer instance

Text to come.

Bind Mounts and Persistent Overlays

Text to come.

Bind Mounts

Text to come.

Persistent Overlays

Text to come.

Building an Apptainer Container/Image

Apptainer images can be created in the following formats:

• as an SIF file, or,
• as a "sandbox" directory.

SIF files can contain multiple sections where each section is normally a squashfs, i.e., read-only and compressed, filesystem although read-write filesystems and overlay images can also be added to the SIF file. That said, unless such additional sections are present, one need not be concerned about the sections contained within an SIF file, e.g., the Apptainer build command produces a read-only SIF file when building images.

A "sandbox" directory is a normal directory in the filesystem that starts out as empty and as Apptainer builds the image it adds the files, etc. needed in the image to that directory. The contents of a "sandbox" directory should only be accessed, updated, etc. through the use of Apptainer. One might need to use a "sandbox" directory in situations where one needs read-write access in order to be able to update the container image. (By default SIF files are read-only.)

If the container image will not need to be updated, one should create an SIF file image as disk performance with such is much faster, e.g.,

• an SIF file, which can be many gigabytes in size, is a single file which will result in excellent disk access performance and inode use on underlying filesystems,
• an SIF file, is (typically) read-only and compressed further improving disk access speeds, and,
• an SIF file that is read-only can be used with bind-mounted directories and/or overlay images that are not read-only should such be needed.

Building a Sandbox Directory

In order to build a "sandbox" directory instead of an SIF file instead of providing an SIF file name, instead provide --sandbox DIR_NAME or -s DIR_NAME where DIR_NAME is the name of the to-be-created-directory where you want your "sandbox" image. For example, if the apptainer build command to create an SIF file was:

$ apptainer build bb.sif docker://busybox

then change bb.sif to a directory name, e.g., bb.dir, and prefix such with --sandbox:

$ apptainer build --sandbox bb.dir docker://busybox

Differences between building a "sandbox" image and a (normal) SIF file are:

• the SIF file's image will be contained in a single file, compressed, and read-only,
• the "sandbox" image will be placed in a directory, uncompressed, may contain thousands of files (depending on what exactly is in the image), and will be able to be read-write.

Within an account, using a "sandbox" directory will consume significant amounts of both disk space and file count quotas, thus, if read-write access to the underlying image is not normally required, you are advised to use an SIF instead. Additionally, using an SIF file will have higher disk access speeds to content contained within the SIF file.

Building an SIF File

NOTE: This section only discusses some basics of creating a simple compressed, read-only SIF file container image. See the Apptainer documentation for more advanced aspects of building images.

Text to come.

Example Use Cases

Using Docker Images Within an Apptainer Container

Text to come.

Using Conda Within an Apptainer Container

Text to come.

Using Spack Within an Apptainer Container

Text to come.

Using NVIDIA GPUs Within an Apptainer Container

Text to come.

Running MPI Programs Inside an Apptainer Container

Text to come.

FAQ

Text to come.

Message Passing Interface