Apptainer: Difference between revisions

<languages />

<translate>

This page does not describe all features of Apptainer and does not replace [http://apptainer.org/docs Apptainer's official documentation].  It summarizes basic use, documents some aspects of using Apptainer on Alliance systems, and provides some relevant examples. We recommend you read the official Apptainer documentation concerning the features of Apptainer you are using.

Should you wish to install Apptainer on your own system, [http://apptainer.org/docs/user/main/quick_start.html#quick-installation instructions appear here]. If you are using a recent Windows system, install [https://learn.microsoft.com/en-us/windows/wsl/install|Windows Subsystem for Linux] first, then within such install Apptainer. If you are using a Mac, install a Linux distribution in a virtual machine on your computer first, then install Apptainer within such.

We strongly recommend that you use Apptainer instead of Singularity. SingularityCE (up to v3.9.5) was adopted by The Linux Foundation and renamed to Apptainer with these changes:

* Added support for [https://dmtcp.sourceforge.io/ DMTCP checkpointing].

* Removed support for the <code>--nvccli</code> command line option.

* Renamed all environment variables having <code>SINGULARITY</code> in their names to have <code>APPTAINER</code> in them.

Should you need to port scripts, etc. to Apptainer, know Apptainer version 1 is backwards compatible with Singularity so switching to Apptainer can be done incrementally.

HPC clusters typically use Apptainer. Many users ask about other Linux container technologies so here are some with some comments:

* [https://podman.io/ Podman]  

** You can install and use Docker on your own computer and use it to create an Apptainer image, which can then be uploaded to an HPC cluster as outlined in '''[[#Creating_an_Apptainer_Container_From_a_Dockerfile|this section]]''' later on this page.

===General===

* In order to use Apptainer you must have a container '''image''', e.g., a <code>.sif</code> file or a "sandbox" directory created previously. If you don't already have an image or a sandbox, see the section on '''[[#Building_an_Apptainer_Container/Image|building an image]]''' below.

* While Apptainer is installed and available for use, using Apptainer will require you to install and/or build all software you will need to make use of in your container. In many instances, '''[[Available_software|we already have such software installed on our clusters]]''' so there is often no need to create a container with the same installed in it.

Many users ask about <code>sudo</code> since documentation and web sites often discuss using <code>sudo</code>. Know the ability to use <code>sudo</code> to obtain superuser/root permissions is not available on our clusters. Should you require using <code>sudo</code>, consider the following options:

* Install Linux, Apptainer, and <code>sudo</code> in a virtual machine on a system you control so you will be able to have <code>sudo</code> access within such. Build your image(s) on that machine and upload them in order to use them on Alliance systems.

* If appropriate, [[Technical Support|submit a ticket]] asking if Alliance staff would be able to help build the image(s), etc. required needing <code>sudo</code>. This may or may not be possible, but feel free to ask in a ticket if what you wish to achieve is beyond your means. Additionally, we may respond with other ways to achieve such which may or may not involve Apptainer.

* Apptainer version 1.1.x and newer has improved support for users using <code>--fakeroot</code> implicitly and explicitly so some things may be possible that were not with Apptainer version 1.0 and Singularity. This includes being able to build some images from <code>.def</code> definition files and building some images without needing to use <code>sudo</code>. That said, not all images will be able to be built without needing to use <code>sudo</code> or superuser/root.

Should you need to build your own container image(s) or overlay(s), be aware of the following:

* Avoid building a sandbox image using <code>--fakeroot</code> on networked filesystem(s): [https://apptainer.org/docs/admin/main/installation.html#fakeroot-with-uid-gid-mapping-on-network-filesystems link to Apptainer documentation].

* Avoid using Lustre/GPFS filesystems as they don't have the feature set required to properly support building Apptainer containers (including <code>--fakeroot</code>): [https://apptainer.org/docs/admin/main/installation.html#lustre-gpfs link to Apptainer documentation].

In order to use the default version of Apptainer available run:

<source lang="console">$ module load apptainer</source>

To see the available versions of Apptainer that can be loaded run:

<source lang="console">$ module spider apptainer</source>

Software that is run inside a container runs in a different environment using different libraries and tools than what is installed on the host system. It is, therefore, important to run programs within containers by '''not''' using any environment settings or software defined outside of the container. Unfortunately, by default, Apptainer will run adopting the shell environment of the host and this can result in issues when running programs. To avoid such issues, when using <code>apptainer run</code>, <code>apptainer shell</code>, <code>apptainer exec</code>, and/or </code>apptainer instance</code>, use one of these options (with more preference to those options listed earlier in the table below):

{| class="wikitable"

|+Apptainer Environment Command Line Options

|}

Another important option is the <code>-W</code> or <code>--workdir</code> option. On Alliance clusters and on most Linux systems, <code>/tmp</code> and similar filesystems use RAM, not disk space. Since jobs typically run on our clusters with limited RAM amounts, this can result in jobs getting killed because they consume too much RAM relative to what was requested for the job. A suitable work-around for this is to tell Apptainer to use a real disk location for its working directory ("workdir"). This is done by passing the <code>-W</code> option followed by a path to a disk location where Apptainer can read/write temporary files, etc. For example, suppose one wanted to run a command called <code>myprogram</code> in a using an Apptainer container image called <code>myimage.sif</code> with its "workdir" set to <code>/path/to/a/workdir</code> in the filesystem:

<source lang="console">$ mkdir -p $HOME/aworkdir

$ apptainer run -C -B /home -W /path/to/a/workdir myimage.sif myprogram</source>

where:

* The workdir can be removed if there are no live containers using it.

* When using bind mounts, see the [[#Bind_Mounts|section on bind mounts]] below since not all Alliance clusters are the same concerning the exact bind mounts needed to access <code>/home</code>, <code>/project</code>, and <code>/scratch</code>.

When running software inside a container that requires the use of GPUs it is important to do the following:

* Ensure that you pass the <code>--nv</code> (for NVIDIA hardware) and <code>--rocm</code> (for AMD hardware) to Apptainer commands.

* When needing to use OpenCL inside the container, besides using the aforementioned options use the following bind mount: <code>-B /etc/OpenCL</code>.

An example of [[#Using_NVIDIA_GPUs_Within_an_Apptainer_Container|using NVIDIA GPUs within an apptainer container]] appears later on this page.

If you need to run MPI programs inside a container there are things that need to be done in the host environment in order for such to work. Please see the [[#Running_MPI_Programs_Inside_an_Apptainer Container|Running MPI Programs section below]] for an example of how to run MPI programs inside a container. The [http://apptainer.org/docs/user/main/mpi.html official Apptainer documentation] has more information concerning how MPI programs can be run inside a container.

Apptainer containers built from [http://apptainer.org/docs/user/main/definition_files.html Definition files] often will have a <code>%help</code> section. To see this section run:

where:

* <code>your-container-name.sif</code> is the name of your container

It is possible your container also has "apps" defined in it, you can get help for those apps by running:

where:

* <code>appname</code> is the name of the app

* <code>your-container-name.sif</code> is the name of your container

To see a list of apps installed in your container (if there are any), run:

where:

* <code>your-container-name.sif</code> is the name of your container

The <code>apptainer run</code> command will launch an Apptainer container, runs the <code>%runscript</code> defined for that container (if one is defined), and then runs the specific command (subject to the code in the <code>%runscript</code> script).  

* Using the <code>apptainer run</code> command is preferred over using the <code>apptainer exec</code> command (which directly runs a command within the specified container).

For example, suppose you want to run the <code>g++</code> compiler inside your container to compile a C++ program called <code>myprog.cpp</code> and then run that program. To this this you might use this command:

  apptainer run your-container-name.sif ./a.out

where:

* <code>your-container-name.sif</code> is the name of your SIF file

* <code>g++ -O2 -march=broadwell ./myprog.cpp</code> is the command you want to run inside the container

On our clusters, you will want to use a number of additional options (that appear after <code>run</code> but before <code>your-container-name.sif</code>). These options will include <code>-C</code>, <code>-c</code>, <code>-e</code>, <code>-W</code> as well as various bind mount options to make your disk space available to the programs that run in your container. For example:

   apptainer run -C -W $SLURM_TMPDIR -B /home -B /project -B /scratch ./a.out

For more information on these options see the following sections on this page:

* [[#Important_Command_Line_Options|Important Command Line Options]]

* [[#Using_GPUs|Using GPUs]]

* [[#Bind_Mounts_and_Persistent_Overlays|Bind Mounts and Persistent Overlays]]

as well as the [http://apptainer.org/docs/user/main/index.html official Apptainer documentation].

The <code>apptainer run</code>, <code>apptainer exec</code>, and <code>apptainer instance</code> commands run the programs provided immediately which makes them excellent for use in BASH and SLURM job scripts. There are times when one needs to interactively do work inside a container. To run commands interactively while remaining inside a container, use the <code>apptainer shell</code> command instead.

For example, to run commands interactively in a container one first invokes the <code>apptainer shell</code> command, e.g.,

where:

* <code>your-container-name.sif</code> is the name of your SIF file

Once the container starts, you will see an <code>Apptainer&gt;</code> prompt (or <code>Singularity&gt;</code> prompt if using Singularity). At this prompt you can run desired shell commands in the container. When done, type <code>exit</code> and hit the Enter/Return key to exit the container.

On our clusters, you will want to use a number of additional options (that appear after <code>run</code> and before <code>your-container-name.sif</code>). These options will include <code>-C</code>, <code>-c</code>, <code>-e</code>, <code>-W</code> as well as various bind mount options to make your disk space available to the programs that run in your container. For example:

For more information on these options see the following sections on this page:

* [[#Important_Command_Line_Options|Important Command Line Options]]

* [[#Using_GPUs|Using GPUs]]

* [[#Bind_Mounts_and_Persistent_Overlays|Bind Mounts and Persistent Overlays]]

as well as the [http://apptainer.org/docs/user/main/index.html official Apptainer documentation].

'''IMPORTANT:''' In addition to choose to use the above options, if you are making use of a persistent overlay image (as a separate file or contained within the SIF file) and want changes to be written to that image, it is extremely important to pass the <code>-w</code> or <code>--writable</code> option to your container. If this option is not passed to it, any changes you make to the image in the <code>apptainer shell</code> session will not be saved!

Apptainer has been designed to be able to properly run daemons within compute jobs on clusters. Running daemons is achieved, in part, by using <code>apptainer instance</code>. See the [http://apptainer.org/docs/user/main/running_services.html official Apptainer documentation on Running Services] for the details.

'''NOTE 1:''' Don't run daemons manually without using <code>apptainer instance</code> and related commands. Apptainer works properly with other tools such as the Slurm scheduler that run on our clusters. When a job is cancelled, killed, crashes, or is otherwise finished, daemons run using <code>apptainer instance</code> will not hang or result in defunct processes. Additionally by using the <code>apptainer instance</code> command you will be able to control the daemons and programs running in the same container.

'''NOTE 2:''' Running daemons in your job will only run those daemons while your job runs. The scheduler will kill all processes attached to a job when the granted job time expires. If you need to run daemons continuously for longer than a job, submit a ticket asking to discuss such with staff persons. Such may require creating a cloud virtual machine to achieve such.

Running MPI programs within an Apptainer container across nodes likely will require special configuration. MPI exploits cluster interconnection hardware to communicate amongst nodes much more efficiently. Normally one does not need to worry about this since it is automatically done --except when running MPI programs across cluster nodes.

'''NOTE:''' When all MPI processes are running on a single shared-memory node, there is no need to use interconnection hardware and there will be no issues running MPI programs within an Apptainer container when all MPI processes run on a single cluster node, e.g., when the slurm option <code>--nodes=1</code> is used with an <code>sbatch</code> script. Unless one '''explicitly''' sets the maximum number of cluster nodes used to <code>1</code>, the scheduler can choose to run an MPI program over multiple nodes. If such will run from within an Apptainer container and has not been set up to properly run, then it is possible it will fail to run.

Text to come.

Often one will want to use either or both of these features in Apptainer:

* "bind mounts" are used to access disk space originating outside of the container, and,

* "persistent overlays" are used to overlay a writable filesystem on an otherwise immutable (i.e., read-only) container image.

When Apptainer is used with the <code>-C</code> or <code>-c</code> options, one will notice that they cannot access their disk space when inside the container. The remedy for this is to explicitly bind mount the disk space they wish to access. For example, suppose a user was using <code>-C</code> like this in an sbatch job to use apptainer:

where <code>./my_data_file.txt</code> is is a file in the current directory on the host, i.e., the file is not stored in the container at all. Because of the <code>-C</code> option, this file will not be accessible to the <code>wc</code> program inside the container --and so an access error will result. The fix is to bind mount the current directory, e.g.,

where <code>-B .</code> will bind mount the current directory, <code>.</code>.

While one can have multiple bind mounts specified, often it is easier to specify the top directories of the filesystems one wishes to access. For example, on our clusters one might want to use:

where:

* <code>-B /home</code> mounts the home filesystem

* <code>-B /scratch</code> mounts the scratch filesystem

Doing this is especially useful when:

* you need to access others' files on your research time in other locations, and/or,

* you need to access files/directories some of which are symlinks to different locations that would/might otherwise be broken if you did not mount the entire filesystem.

If using these bind mounts do not work on the cluster you are using, then run this script to obtain the bind mount options you need to pass to Apptainer for /home, /project, and /scratch on that cluster:

</translate>

<translate>

It should be mentioned that a bind mount does not need to be in the same location inside the container: one can bind mount any file or directory to be at a different location, e.g.,

i.e., <code>-B ./my_data_file.txt:/special/input.dat</code> bind mount maps the file <code>./my_data_file.txt</code> to be the file <code>/special/input.dat</code> inside the container and the <code>wc</code> command now processes that file. This feature can be useful when programs/scripts inside the container have hard-coded paths to files and directories that must be located in certain locations.

Finally, '''don't mount our CVMFS paths''' inside your containers as this is fraught with perils and defeats many reasons to use a container. The programs needed to be run inside a container need to be completely inside the container --don't introduce even more programs inside the container that don't need to be inside the container.

Please refer to Apptainer documentation page about [https://apptainer.org/docs/user/main/persistent_overlays.html persistent overlays].

==Overview==

'''NOTE:''' Please note and heed the advice concerning building images/overlays given '''[[#Building_Images.2FOverlays|earlier on this page]]'''.

Apptainer "images" can be created in the following formats:

* as an <code>SIF</code> file, or,

* as a "sandbox" directory.

<code>SIF</code> files internally can contain multiple parts where each part is typically a squashfs filesystem (which are read-only and compressed). It is possible for <code>SIF</code> files to contain read-write filesystems and overlay images as well --but such is beyond the scope of this page: see the official Apptainer documentation on how to do such. Unless more advanced methods of building an "image" were used, the Apptainer <code>build</code> command produces a <code>SIF</code> file with a read-only squashfs filesystem when building images. (This is the preferred option since the resulting image remains as-is since it is read-only, and, the image is much smaller since it is compressed. Know that disk reads from that image are done very quickly.)

A "sandbox" directory is a normal directory in the filesystem that starts out as empty and as Apptainer builds the image it adds the files, etc. needed in the image to that directory. The contents of a "sandbox" directory should only be accessed, updated, etc. through the use of Apptainer. One might need to use a "sandbox" directory in situations where one needs to have read-write access to the image itself in order to be able to update the container image. That said, if updates are infrequent, it is typically easier and better to use an <code>SIF</code> and when updates need to be done, build a sandbox image from the <code>SIF</code> file, make the required changes, and then build a new <code>SIF</code> file, e.g.,

<source lang="console">$ cd $HOME

$ mkdir mynewimage.dir

$ rm -rf mynewimage.dir</source>

Using an <code>SIF</code> image is recommended as disk performance (from the container image) will be faster than storing each file, etc. separately on Alliance cluster filesystems (which are set up to handle large files and parallel I/O). Using an <code>SIF</code> file instead of a sandbox image will also only use a quota file count amount of 1 instead of thousands (e.g., images will typically contain thousands of files and directories).

Many Linux distributions' package managers require root in order to use them. This implies that Apptainer version 1.0.x and the older Singularity cannot be used on compute clusters to build images as a normal user. Should such occur, submit a ticket asking for help to create that image or use a computer with Apptainer installed and you have root permissions.

Apptainer has a <code>--fakeroot</code> feature that can be used to build and manipulate images. Before Apptainer version 1.1, one wanting to use this feature on a cluster requires submitting a ticket requesting a system administrator to consider adding that person so Apptainer's <code>--fakeroot</code> for a specific cluster. (This may or may not be done --such will be responded to in that ticket.) With Apptainer version 1.1, <code>--fakeroot</code> can be used without being formally added.

Know that some containers will not build successfully without using a <code>root</code> account to build them. These images cannot be built on our clusters.

If all you need is to use a Docker image as-is with Apptainer, often those images can be built and run without issues, e.g.,  without any need to have additional permissions or explicitly use <code>--fakeroot</code>. Should you need to modify the image after creating it, such may require elevated permissions to successfully do this, e.g., if the image's Linux distribution's package manager requires such and you need to install a package using it. For this reason, the examples shown below assume one only needs to use a Docker image as-is.

'''NOTE:''' Please note and heed the advice concerning building images/overlays given '''[[#Building_Images.2FOverlays|earlier on this page]]'''.

To build an Apptainer SIF file image from Docker's latest available busybox image, use the <code>apptainer build</code> command:

<source lang="console">$ apptainer build bb.sif docker://busybox</source>

See the [https://apptainer.org/docs Apptainer documentation] for more advanced aspects of building images.

'''NOTE:''' Please note and heed the advice concerning building images/overlays given '''[[#Building_Images.2FOverlays|earlier on this page]]'''.

In order to build a "sandbox" directory instead of an <code>SIF</code> file instead of providing an <code>SIF</code> file name, instead provide <code>--sandbox DIR_NAME</code> or <code>-s DIR_NAME</code> where <code>DIR_NAME</code> is the name of the to-be-created-directory where you want your "sandbox" image. For example, if the <code>apptainer build</code> command to create an <code>SIF</code> file was:

<source lang="console">$ apptainer build bb.sif docker://busybox</source>

<source lang="console">$ apptainer build --sandbox bb.dir docker://busybox</source>

Differences between building a "sandbox" image and a (normal) <code>SIF</code> file are:

* the <code>SIF</code> file's image will be contained in a single file, compressed, and read-only,

* the "sandbox" image will be placed in a directory, uncompressed, may contain thousands of files (depending on what exactly is in the image), and will be able to be read-write.

Within an account, using a "sandbox" directory will consume significant amounts of both disk space and file count quotas, thus, if read-write access to the underlying image is not normally required, you are advised to use an <code>SIF</code> instead. Additionally, using an <code>SIF</code> file will have higher disk access speeds to content contained within the <code>SIF</code> file.

Text to come.

Text to come.

Text to come.

Text to come.

'''NOTE: This section requires you to install and use Docker and Apptainer on a system where you have appropriate privileges. These instructions will ''not'' work on our compute clusters.'''

Unfortunately some instructions for packages only provide a <code>Dockerfile</code> without a container image. A <code>Dockerfile</code> contains the instructions necessary for the Docker software to build that container. Our clusters do not have the Docker software installed. That said, if you've access to a system with both Docker and Apptainer installed, and, sufficient access to Docker (e.g., <code>sudo</code> or root access, or, you are in that system's <code>docker</code> group) and if needed Apptainer (e.g., <code>sudo</code> or root access, or, you have <code>--fakeroot</code> access), then you can follow the instructions below to use Docker and then Apptainer to build an Apptainer image on that system.

'''NOTE:''' Using Docker may fail if you are not in the <code>docker</code> group. Similarly, building some containers may fail with Apptainer without appropriate <code>sudo</code>, root, or <code>--fakeroot</code> permissions. It is your responsibility to ensure you've such access on the system you are running the commands below.

If one only has a Dockerfile and wishes to create an Apptainer image, run the following on a computer with Docker and Apptainer installed (where you've sufficient permissions, etc.):

  docker save your-tag-name -o your-tarball-name.tar

  docker image rm your-tag-name

  rm your-tarball-name.tar

where:

* <code>your-tag-name</code> is a name you make up that will identify the container created in Docker

* <code>your-tarball-name.tar</code> is a filename you create that Docker will save the generated content of the container to

* <code>your-sif-name.sif</code> is the name of the Apptainer SIF file for the Apptainer container

After this is done, the SIF file is an Apptainer container for the <code>Dockerfile</code>. Transfer the SIF to the approprate cluster(s) in order to use such.

'''NOTE:''' It is possible that the Dockerfile pulled in more layers which means you will have to manually delete those additional layers by running:

</translate>

<translate>

followed by runninng <code>docker image rm ID</code> (where ID is the image ID output from the <code>docker images</code> command) in order to free up the disk space associated with those other image layers on the system you are using.

Over time Apptainer's file cache will grow. To see where these files are run:

</translate>

<translate>

and to remove those files, run:

</translate>

<translate>

You can override Apptainer's default temporary and cache directories by setting these environment variables before running <code>apptainer</code>:

* <code>APPTAINER_CACHEDIR</code>: the directory where Apptainer will download and cache files

* <code>APPTAINER_TMPDIR</code>: the directory where Apptainer will write temporary files including when building (squashfs) images

For example, to tell Apptainer to use your scratch space for its cache and temporary files (which is might be a better location), one might run:

</translate>

<translate>

before running <code>apptainer</code>.

</translate>