Bureaucrats, cc_docs_admin, cc_staff
2,879
edits
Apptainer: Difference between revisions
stylistic changes, mostly upper->lower case in headers
(stylistic changes, mostly upper->lower case in headers) |
|||
| Line 4: | Line 4: | ||
This page is current a '''draft work-in-progress'''. | This page is current a '''draft work-in-progress'''. | ||
==Official Apptainer | ==Official Apptainer documentation== | ||
This page is neither exhaustive nor all-features complete and does not replace [http://apptainer.org/docs Apptainer's official documentation], rather, it summarizes basic use, documents some aspects of using Apptainer on Alliance systems, and provides some examples relevant in using Apptainer on Alliance systems. We recommend all users read the official Apptainer documentation concerning the features of Apptainer they are using. | This page is neither exhaustive nor all-features complete and does not replace [http://apptainer.org/docs Apptainer's official documentation], rather, it summarizes basic use, documents some aspects of using Apptainer on Alliance systems, and provides some examples relevant in using Apptainer on Alliance systems. We recommend all users read the official Apptainer documentation concerning the features of Apptainer they are using. | ||
| Line 10: | Line 10: | ||
Should you wish to install Apptainer on your own system, [http://apptainer.org/docs/user/main/quick_start.html#quick-installation instructions appear here]. If you are using a recent Windows system, install [https://learn.microsoft.com/en-us/windows/wsl/install|Windows Subsystem for Linux] first, then within such install Apptainer. If you are using a Mac, install a Linux distribution in a virtual machine on your computer first, then install Apptainer within such. | Should you wish to install Apptainer on your own system, [http://apptainer.org/docs/user/main/quick_start.html#quick-installation instructions appear here]. If you are using a recent Windows system, install [https://learn.microsoft.com/en-us/windows/wsl/install|Windows Subsystem for Linux] first, then within such install Apptainer. If you are using a Mac, install a Linux distribution in a virtual machine on your computer first, then install Apptainer within such. | ||
==If | ==If you are currently using Singularity== | ||
We strongly recommend that you use Apptainer instead of Singularity. SingularityCE (up to v3.9.5) was adopted by The Linux Foundation and renamed to Apptainer with these changes: | We strongly recommend that you use Apptainer instead of Singularity. SingularityCE (up to v3.9.5) was adopted by The Linux Foundation and renamed to Apptainer with these changes: | ||
* | * Added support for [https://dmtcp.sourceforge.io/ DMTCP checkpointing]. | ||
** | ** Singularity does not support this. | ||
* | * Removed support for the <code>--nvccli</code> command line option. | ||
* | * Removed support for <code>apptainer build --remote</code>. | ||
* | * Removed support the SylabsCloud remote endpoint replacing it with a DefaultRemote endpoint with no defined server for <code>library://</code>. | ||
** | ** If the SylabsCloud remote is needed, the [https://apptainer.org/docs/user/1.0/endpoint.html#restoring-pre-apptainer-library-behavior previous default can be restored]. | ||
* | * Renamed all executable names, paths, etc. having <code>singularity</code> in their names to have <code>apptainer</code> in them. | ||
** | ** E.g., instead of using the <code>singularity</code> command one uses the <code>apptainer</code> command. | ||
** | ** E.g., the <code>~/.singularity</code> directory is now <code>~/.apptainer</code>. | ||
* | * Renamed all environment variables having <code>SINGULARITY</code> in their names to have <code>APPTAINER</code> in them. | ||
Should you need to port scripts, etc. to Apptainer, know Apptainer version 1 is backwards compatible with Singularity so switching to Apptainer can be done incrementally. | Should you need to port scripts, etc. to Apptainer, know Apptainer version 1 is backwards compatible with Singularity so switching to Apptainer can be done incrementally. | ||
==Other Linux | ==Other Linux container technologies== | ||
HPC clusters typically use Apptainer. Many users | HPC clusters typically use Apptainer. Many users ask about other Linux container technologies so here are some with some comments: | ||
* [https://podman.io/ Podman] | * [https://podman.io/ Podman] | ||
** Supports rootless (i.e., normal) container use, etc. similar to Apptainer. | ** Supports rootless (i.e., normal) container use, etc. similar to Apptainer. | ||
| Line 40: | Line 40: | ||
NOTE: Apptainer used to be called Singularity. | NOTE: Apptainer used to be called Singularity. | ||
==Other | ==Other items== | ||
===General=== | ===General=== | ||
* In order to use Apptainer one must | * In order to use Apptainer one must have a container image, e.g., a <code>.sif</code> file or a "sandbox" directory created previously. If you don't already have a container image/sandbox, see the section on '''[[#Building_an_Apptainer_Container/Image|building an image]]''' below. | ||
* While Apptainer is installed and available for use, using Apptainer will require you to install and/or build all software you will need to make use of in your container. In many instances, '''[[Available_software|we already have such software installed on our clusters]]''' so there is often no need to create a container with the same installed in it. | * While Apptainer is installed and available for use, using Apptainer will require you to install and/or build all software you will need to make use of in your container. In many instances, '''[[Available_software|we already have such software installed on our clusters]]''' so there is often no need to create a container with the same installed in it. | ||
| Line 49: | Line 49: | ||
* Install Linux, Apptainer, and <code>sudo</code> in a virtual machine on a system you control so you will be able to have <code>sudo</code> access within such. Build your image(s) on that machine and upload them in order to use them on Alliance systems. | * Install Linux, Apptainer, and <code>sudo</code> in a virtual machine on a system you control so you will be able to have <code>sudo</code> access within such. Build your image(s) on that machine and upload them in order to use them on Alliance systems. | ||
* If appropriate, [[Technical Support|submit a ticket]] asking if Alliance staff would be able to help build the image(s), etc. required needing <code>sudo</code>. | * If appropriate, [[Technical Support|submit a ticket]] asking if Alliance staff would be able to help build the image(s), etc. required needing <code>sudo</code>. This may or may not be possible, but feel free to ask in a ticket if what you wish to achieve is beyond your means. Additionally, we may respond with other ways to achieve such which may or may not involve Apptainer. | ||
* Apptainer version 1.1.x and newer has improved support for users using <code>--fakeroot</code> implicitly and explicitly so some things may be possible that were not with Apptainer version 1.0 and Singularity. This includes being able to build some images from <code>.def</code> definition files and building some images without needing to use <code>sudo</code>. That said, not all images will be able to be built without needing to use <code>sudo</code> or superuser/root. | * Apptainer version 1.1.x and newer has improved support for users using <code>--fakeroot</code> implicitly and explicitly so some things may be possible that were not with Apptainer version 1.0 and Singularity. This includes being able to build some images from <code>.def</code> definition files and building some images without needing to use <code>sudo</code>. That said, not all images will be able to be built without needing to use <code>sudo</code> or superuser/root. | ||
===Building | ===Building images or overlays=== | ||
Should you need to build your own container image(s) or overlay(s), be aware of the following: | Should you need to build your own container image(s) or overlay(s), be aware of the following: | ||
* | * Avoid building a sandbox image using <code>--fakeroot</code> on networked filesystem(s): [https://apptainer.org/docs/admin/main/installation.html#fakeroot-with-uid-gid-mapping-on-network-filesystems link to Apptainer documentation]. | ||
* | * Explicitly set <code>APPTAINER_CACHEDIR</code> to a non-networked filesystem location before using Apptainer: [https://apptainer.org/docs/admin/main/installation.html#apptainer-cache-atomic-rename link to Apptainer documentation]. | ||
* | * Explicitly set <code>APPTAINER_TMPDIR</code> to a non-Lustre/GPFS filesystem location before using Apptainer: [https://apptainer.org/docs/admin/main/installation.html#lustre-gpfs link to Apptainer documentation]. | ||
* | * Avoid using Lustre/GPFS filesystems as they don't have the feature set required to properly support building Apptainer containers (including <code>--fakeroot</code>): [https://apptainer.org/docs/admin/main/installation.html#lustre-gpfs link to Apptainer documentation]. | ||
=Loading an Apptainer | =Loading an Apptainer module= | ||
In order to use the default version of Apptainer available run: | In order to use the default version of Apptainer available run: | ||
<source lang="console">$ module load apptainer</source> | <source lang="console">$ module load apptainer</source> | ||
| Line 66: | Line 66: | ||
<source lang="console">$ module spider apptainer</source> | <source lang="console">$ module spider apptainer</source> | ||
=Running | =Running programs within a container= | ||
==Important | ==Important command line options== | ||
Software that is run inside a container runs in a different environment using different libraries and tools than what is installed on the host system. It is, therefore, important to run programs within containers by '''not''' using any environment settings or software defined outside of the container. Unfortunately, by default, Apptainer will run adopting the shell environment of the host and this can result in issues when running programs. To avoid such issues, when using <code>apptainer run</code>, <code>apptainer shell</code>, <code>apptainer exec</code>, and/or </code>apptainer instance</code>, use one of these options (with more preference to those options listed earlier in the table below): | Software that is run inside a container runs in a different environment using different libraries and tools than what is installed on the host system. It is, therefore, important to run programs within containers by '''not''' using any environment settings or software defined outside of the container. Unfortunately, by default, Apptainer will run adopting the shell environment of the host and this can result in issues when running programs. To avoid such issues, when using <code>apptainer run</code>, <code>apptainer shell</code>, <code>apptainer exec</code>, and/or </code>apptainer instance</code>, use one of these options (with more preference to those options listed earlier in the table below): | ||
| Line 84: | Line 84: | ||
|} | |} | ||
Another important option one should consider and may need to use in order to use Apptainer successfully is the <code>-W</code> or <code>--workdir</code> option. On Alliance clusters and on most Linux systems, <code>/tmp</code> and similar filesystems use RAM | Another important option one should consider and may need to use in order to use Apptainer successfully is the <code>-W</code> or <code>--workdir</code> option. On Alliance clusters and on most Linux systems, <code>/tmp</code> and similar filesystems use RAM, not disk space. Since jobs typically run on our clusters with limited RAM amounts, this can result in jobs getting killed because they consume too much RAM relative to what was requested for the job. A suitable work-around for this is to tell Apptainer to use a real disk location for its working directory ("workdir"). This is done by passing the <code>-W</code> option followed by a path to a disk location where Apptainer can read/write temporary files, etc. For example, suppose one wanted to run a command called <code>myprogram</code> in a using an Apptainer container image called <code>myimage.sif</code> with its "workdir" set to <code>/path/to/a/workdir</code> in the filesystem: | ||
<source lang="console">$ mkdir -p $HOME/aworkdir | <source lang="console">$ mkdir -p $HOME/aworkdir | ||
| Line 90: | Line 90: | ||
where: | where: | ||
* The workdir | * The workdir can be removed if there are no live containers using it. | ||
* When using Apptainer in an <code>salloc</code>, in an <code>sbatch</code> job, or when using [JupyterHub] on our clusters, use <code>${SLURM_TMPDIR}</code> for the "workdir" location, e.g., <code>-W ${SLURM_TMPDIR}</code>. | * When using Apptainer in an <code>salloc</code>, in an <code>sbatch</code> job, or when using [JupyterHub] on our clusters, use <code>${SLURM_TMPDIR}</code> for the "workdir" location, e.g., <code>-W ${SLURM_TMPDIR}</code>. | ||
** ASIDE: One should '''not''' be running programs (including Apptainer) on a login node | ** ASIDE: One should '''not''' be running programs (including Apptainer) on a login node. Use an interactive <code>salloc</code> job. | ||
* When using bind mounts, see the [[#Bind_Mounts|section on bind mounts]] below since not all Alliance clusters are the same concerning the exact bind mounts needed to access <code>/home</code>, <code>/project</code>, and <code>/scratch</code>. | * When using bind mounts, see the [[#Bind_Mounts|section on bind mounts]] below since not all Alliance clusters are the same concerning the exact bind mounts needed to access <code>/home</code>, <code>/project</code>, and <code>/scratch</code>. | ||
| Line 106: | Line 106: | ||
An example of [[#Using_NVIDIA_GPUs_Within_an_Apptainer_Container|using NVIDIA GPUs within an apptainer container]] appears later on this page. | An example of [[#Using_NVIDIA_GPUs_Within_an_Apptainer_Container|using NVIDIA GPUs within an apptainer container]] appears later on this page. | ||
==Using MPI | ==Using MPI programs== | ||
If you need to run MPI programs inside a container there are things that need to be done in the host environment in order for such to work. Please see the [[#Running_MPI_Programs_Inside_an_Apptainer Container|Running MPI Programs section below]] for an example of how to run MPI programs inside a container. The [http://apptainer.org/docs/user/main/mpi.html official Apptainer documentation] has more information concerning how MPI programs can be run inside a container. | If you need to run MPI programs inside a container there are things that need to be done in the host environment in order for such to work. Please see the [[#Running_MPI_Programs_Inside_an_Apptainer Container|Running MPI Programs section below]] for an example of how to run MPI programs inside a container. The [http://apptainer.org/docs/user/main/mpi.html official Apptainer documentation] has more information concerning how MPI programs can be run inside a container. | ||
==Container- | ==Container-specific help: <code>apptainer run-help</code>== | ||
Apptainer containers built from [http://apptainer.org/docs/user/main/definition_files.html Definition files] often will have a <code>%help</code> section. To see this section run: | Apptainer containers built from [http://apptainer.org/docs/user/main/definition_files.html Definition files] often will have a <code>%help</code> section. To see this section run: | ||
| Line 134: | Line 134: | ||
* <code>your-container-name.sif</code> is the name of your container | * <code>your-container-name.sif</code> is the name of your container | ||
==Running | ==Running software: <code>apptainer run</code> or <code>apptainer exec</code>== | ||
The <code>apptainer run</code> command will launch an Apptainer container, runs the <code>%runscript</code> defined for that container (if one is defined), and then runs the specific command (subject to the code in the <code>%runscript</code> script). | The <code>apptainer run</code> command will launch an Apptainer container, runs the <code>%runscript</code> defined for that container (if one is defined), and then runs the specific command (subject to the code in the <code>%runscript</code> script). | ||
| Line 162: | Line 162: | ||
as well as the [http://apptainer.org/docs/user/main/index.html official Apptainer documentation]. | as well as the [http://apptainer.org/docs/user/main/index.html official Apptainer documentation]. | ||
==Interactively | ==Interactively running software: <code>apptainer shell</code>== | ||
The <code>apptainer run</code>, <code>apptainer exec</code>, and <code>apptainer instance</code> commands run the programs provided immediately which makes them excellent for use in BASH and SLURM job scripts. There are times when one needs to interactively do work inside a container. To run commands interactively while remaining inside a container, use the <code>apptainer shell</code> command instead. | The <code>apptainer run</code>, <code>apptainer exec</code>, and <code>apptainer instance</code> commands run the programs provided immediately which makes them excellent for use in BASH and SLURM job scripts. There are times when one needs to interactively do work inside a container. To run commands interactively while remaining inside a container, use the <code>apptainer shell</code> command instead. | ||
| Line 189: | Line 189: | ||
'''IMPORTANT:''' In addition to choose to use the above options, if you are making use of a persistent overlay image (as a separate file or contained within the SIF file) and want changes to be written to that image, it is extremely important to pass the <code>-w</code> or <code>--writable</code> option to your container. If this option is not passed to it, any changes you make to the image in the <code>apptainer shell</code> session will not be saved! | '''IMPORTANT:''' In addition to choose to use the above options, if you are making use of a persistent overlay image (as a separate file or contained within the SIF file) and want changes to be written to that image, it is extremely important to pass the <code>-w</code> or <code>--writable</code> option to your container. If this option is not passed to it, any changes you make to the image in the <code>apptainer shell</code> session will not be saved! | ||
==Running | ==Running daemons: <code>apptainer instance</code>== | ||
Apptainer has been designed to be able to properly run daemons within compute jobs on clusters. Running daemons is achieved, in part, by using <code>apptainer instance</code>. See the [http://apptainer.org/docs/user/main/running_services.html official Apptainer documentation on Running Services] for the details. | Apptainer has been designed to be able to properly run daemons within compute jobs on clusters. Running daemons is achieved, in part, by using <code>apptainer instance</code>. See the [http://apptainer.org/docs/user/main/running_services.html official Apptainer documentation on Running Services] for the details. | ||
| Line 197: | Line 197: | ||
'''NOTE 2:''' Running daemons in your job will only run those daemons while your job runs. The scheduler will kill all processes attached to a job when the granted job time expires. If you need to run daemons continuously for longer than a job, submit a ticket asking to discuss such with staff persons. Such may require creating a cloud virtual machine to achieve such. | '''NOTE 2:''' Running daemons in your job will only run those daemons while your job runs. The scheduler will kill all processes attached to a job when the granted job time expires. If you need to run daemons continuously for longer than a job, submit a ticket asking to discuss such with staff persons. Such may require creating a cloud virtual machine to achieve such. | ||
==Running MPI | ==Running MPI programs== | ||
Running MPI programs within an Apptainer container across nodes likely will require special configuration. MPI exploits cluster interconnection hardware to communicate amongst nodes much more efficiently. Normally one does not need to worry about this since it is automatically done --except when running MPI programs across cluster nodes. | Running MPI programs within an Apptainer container across nodes likely will require special configuration. MPI exploits cluster interconnection hardware to communicate amongst nodes much more efficiently. Normally one does not need to worry about this since it is automatically done --except when running MPI programs across cluster nodes. | ||
| Line 205: | Line 205: | ||
Text to come. | Text to come. | ||
=Bind | =Bind mounts and persistent overlays= | ||
Often one will want to use either or both of these features in Apptainer: | Often one will want to use either or both of these features in Apptainer: | ||
| Line 211: | Line 211: | ||
* "persistent overlays" are used to overlay a writable filesystem on an otherwise immutable (i.e., read-only) container image. | * "persistent overlays" are used to overlay a writable filesystem on an otherwise immutable (i.e., read-only) container image. | ||
==Bind | ==Bind mounts== | ||
When Apptainer is used with the <code>-C</code> or <code>-c</code> options, one will notice that they cannot access their disk space when inside the container. The remedy for this is to explicitly bind mount the disk space they wish to access. For example, suppose a user was using <code>-C</code> like this in an sbatch job to use apptainer: | When Apptainer is used with the <code>-C</code> or <code>-c</code> options, one will notice that they cannot access their disk space when inside the container. The remedy for this is to explicitly bind mount the disk space they wish to access. For example, suppose a user was using <code>-C</code> like this in an sbatch job to use apptainer: | ||
| Line 248: | Line 248: | ||
Finally, '''don't mount our CVMFS paths''' inside your containers as this is fraught with perils and defeats many reasons to use a container. The programs needed to be run inside a container need to be completely inside the container --don't introduce even more programs inside the container that don't need to be inside the container. | Finally, '''don't mount our CVMFS paths''' inside your containers as this is fraught with perils and defeats many reasons to use a container. The programs needed to be run inside a container need to be completely inside the container --don't introduce even more programs inside the container that don't need to be inside the container. | ||
==Persistent | ==Persistent overlays== | ||
Please refer to Apptainer documentation page about [https://apptainer.org/docs/user/main/persistent_overlays.html persistent overlays]. | Please refer to Apptainer documentation page about [https://apptainer.org/docs/user/main/persistent_overlays.html persistent overlays]. | ||
=Building an Apptainer | =Building an Apptainer image= | ||
==Overview== | ==Overview== | ||
| Line 285: | Line 285: | ||
If all you need is to use a Docker image as-is with Apptainer, often those images can be built and run without issues, e.g., without any need to have additional permissions or explicitly use <code>--fakeroot</code>. Should you need to modify the image after creating it, such may require elevated permissions to successfully do this, e.g., if the image's Linux distribution's package manager requires such and you need to install a package using it. For this reason, the examples shown below assume one only needs to use a Docker image as-is. | If all you need is to use a Docker image as-is with Apptainer, often those images can be built and run without issues, e.g., without any need to have additional permissions or explicitly use <code>--fakeroot</code>. Should you need to modify the image after creating it, such may require elevated permissions to successfully do this, e.g., if the image's Linux distribution's package manager requires such and you need to install a package using it. For this reason, the examples shown below assume one only needs to use a Docker image as-is. | ||
==Building | ==Building a SIF image== | ||
'''NOTE:''' Please note and heed the advice concerning building images/overlays given '''[[#Building_Images.2FOverlays|earlier on this page]]'''. | '''NOTE:''' Please note and heed the advice concerning building images/overlays given '''[[#Building_Images.2FOverlays|earlier on this page]]'''. | ||
| Line 294: | Line 294: | ||
See the [https://apptainer.org/docs Apptainer documentation] for more advanced aspects of building images. | See the [https://apptainer.org/docs Apptainer documentation] for more advanced aspects of building images. | ||
==Building a | ==Building a sandbox image== | ||
'''NOTE:''' Please note and heed the advice concerning building images/overlays given '''[[#Building_Images.2FOverlays|earlier on this page]]'''. | '''NOTE:''' Please note and heed the advice concerning building images/overlays given '''[[#Building_Images.2FOverlays|earlier on this page]]'''. | ||
| Line 310: | Line 310: | ||
Within an account, using a "sandbox" directory will consume significant amounts of both disk space and file count quotas, thus, if read-write access to the underlying image is not normally required, you are advised to use an <code>SIF</code> instead. Additionally, using an <code>SIF</code> file will have higher disk access speeds to content contained within the <code>SIF</code> file. | Within an account, using a "sandbox" directory will consume significant amounts of both disk space and file count quotas, thus, if read-write access to the underlying image is not normally required, you are advised to use an <code>SIF</code> instead. Additionally, using an <code>SIF</code> file will have higher disk access speeds to content contained within the <code>SIF</code> file. | ||
=Example | =Example use cases= | ||
==Using Conda | ==Using Conda in Apptainer == | ||
Text to come. | Text to come. | ||
==Using Spack | ==Using Spack in Apptainer == | ||
Text to come. | Text to come. | ||
==Using NVIDIA GPUs | ==Using NVIDIA GPUs in Apptainer == | ||
Text to come. | Text to come. | ||
==Running MPI | ==Running MPI programs in Apptainer == | ||
Text to come. | Text to come. | ||
==Creating an Apptainer | ==Creating an Apptainer container from a Dockerfile== | ||
'''NOTE: This section requires you to install and use Docker and Apptainer on a system where you have appropriate privileges. These instructions will ''not'' work on our compute clusters.''' | '''NOTE: This section requires you to install and use Docker and Apptainer on a system where you have appropriate privileges. These instructions will ''not'' work on our compute clusters.''' | ||
| Line 361: | Line 361: | ||
=Miscellaneous Items= | =Miscellaneous Items= | ||
==Cleaning Apptainer's | ==Cleaning Apptainer's cache directory== | ||
Over time Apptainer's file cache will grow. To see where these files are run: | Over time Apptainer's file cache will grow. To see where these files are run: | ||
| Line 371: | Line 371: | ||
apptainer cache clean | apptainer cache clean | ||
==Changing Apptainer's | ==Changing Apptainer's default directories== | ||
You can override Apptainer's default temporary and cache directories by setting these environment variables before running <code>apptainer</code>: | You can override Apptainer's default temporary and cache directories by setting these environment variables before running <code>apptainer</code>: | ||