# Containers¶

Warning

The ARCHER2 Service is not yet available. This documentation is in development.

This page was originally based on the documentation at the University of Sheffield HPC service <http://docs.hpc.shef.ac.uk/en/latest/sharc/software/apps/singularity.html>.

Designed around the notion of mobility of compute and reproducible science, Singularity enables users to have full control of their operating system environment. This means that a non-privileged user can “swap out” the Linux operating system and environment on the host for a Linux OS and environment that they control. So if the host system is running CentOS Linux but your application runs in Ubuntu Linux with a particular software stack; you can create an Ubuntu image, install your software into that image, copy the image to another host (e.g. ARCHER2), and run your application on that host in it’s native Ubuntu environment.

Singularity also allows you to leverage the resources of whatever host you are on. This includes high-speed interconnects (i.e. Infinband on ARCHER2), file systems (i.e. /lustre on ARCHER2) and potentially other resources (e.g. the licensed Intel compilers on ARCHER2).

Note: Singularity only supports Linux containers. You cannot create images that use Windows or macOS (this is a restriction of the containerisation model rather than Singularity).

Similar to Docker, a Singularity container (or, more commonly, image) is a self-contained software stack. As Singularity does not require a root-level daemon to run its images (as is required by Docker) it is suitable for use on multi-user HPC systems such as ARCHER2. Within the container/image, you have exactly the same permissions as you do in a standard login session on the system.

In practice, this means that an image created on your local machine with all your research software installed for local development will also run on ARCHER2.

Pre-built images (such as those on DockerHub or SingularityHub) can simply be downloaded and used on ARCHER2 (or anywhere else Singularity is installed); see Using Singularity Images on ARCHER2).

Creating and modifying images requires root permission and so must be done on a system where you have such access (in practice, this is usually within a virtual machine on your laptop/workstation); see Creating Your Own Singularity Images.

## Using Singularity Images on ARCHER2¶

Singularity images can be used on ARCHER2 in a number of ways, including:

• Interactively on the login nodes
• Interactively on compute nodes
• As serial processes within a non-interactive batch script
• As parallel processes within a non-interactive batch script (not yet documented)

We provide information on each of these scenarios (apart from the parallel use where we are still preparing the documentation) below. First, we describe briefly how to get existing images onto ARCHER2 so you can use them.

### Getting existing images onto ARCHER2¶

Singularity images are simply files, so, if you already have an image file, you can use scp to copy the file to ARCHER2 as you would with any other file.

If you wish to get a file from one of the container image repositories then Singularity allows you to do this from ARCHER2 itself.

This functionality requires tools that are not part of the standard OS on ARCHER2 so we have provided a Singularity image that allows you to build images from remote repositories (i.e. you use a Singularity image to build Singularity images!).

For example, to retrieve an image from DockerHub on ARCHER2 we fist need to enter an interactive session in the image we provide for building Singularity images:

[user@archer2-login0 ~]$module load singularity [user@archer2-login0 ~]$ singularity exec $CIRRUS_SIMG/archer2-sbuild.simg /bin/bash --login Singularity>  This invokes a login bash shell within the $CIRRUS_SIMG/archer2-sbuild.simg image as indicated by our prompt change. (We need a login shell to allow module commands to work within the image.)

Now we are in the image we can load the singularity module (to get access to the Singularity commands) and pull an image from DockerHub:

Singularity> module load singularity
Singularity> singularity build lolcow.simg docker://godlovedc/lolcow
Docker image path: index.docker.io/godlovedc/lolcow:latest
Cache folder set to /lustre/home/t01/user/.singularity/docker
Importing: base Singularity environment
Importing: /lustre/home/t01/user/.singularity/docker/sha256:9fb6c798fa41e509b58bccc5c29654c3ff4648b608f5daa67c1aab6a7d02c118.tar.gz
Importing: /lustre/home/t01/user/.singularity/docker/sha256:3b61febd4aefe982e0cb9c696d415137384d1a01052b50a85aae46439e15e49a.tar.gz
Importing: /lustre/home/t01/user/.singularity/docker/sha256:9d99b9777eb02b8943c0e72d7a7baec5c782f8fd976825c9d3fb48b3101aacc2.tar.gz
Importing: /lustre/home/t01/user/.singularity/docker/sha256:d010c8cf75d7eb5d2504d5ffa0d19696e8d745a457dd8d28ec6dd41d3763617e.tar.gz
Importing: /lustre/home/t01/user/.singularity/docker/sha256:7fac07fb303e0589b9c23e6f49d5dc1ff9d6f3c8c88cabe768b430bdb47f03a9.tar.gz
Importing: /lustre/home/t01/user/.singularity/docker/sha256:8e860504ff1ee5dc7953672d128ce1e4aa4d8e3716eb39fe710b849c64b20945.tar.gz
WARNING: Building container as an unprivileged user. If you run this container as root
WARNING: it may be missing some functionality.
Building Singularity image...
Singularity container built: lolcow.simg
Cleaning up...


The first argument to the singularity build command (lolcow.simg) specifies a path and name for your container. The second argument (docker://godlovedc/lolcow) gives the DockerHub URI from which to download the image.

Now we can exit the archer2-sbuild image and run the new lolcow image that we have just built on the ARCHER2 login node:

[user@archer2-login0 ~]$singularity run lolcow.simg  This image contains a runscript that tells Singularity what to do if we run the image. We demonstrate different ways to use images below. Similar syntax can be used for Singularity Hub. For more information see the Singularity documentation: ### Interactive use on the login nodes¶ Once you have an image file, using it on the login nodes in an interactive way is extremely simple: you use the singularity shell command. Using the image we built in the example above: [user@archer2-login0 ~]$ module load singularity
[user@archer2-login0 ~]$singularity shell lolcow.simg Singularity: Invoking an interactive shell within container... Singularity lolcow.simg:~>  Within a Singularity image your home directory will be available. The directory with centrally-installed software (/lustre/sw) is also available in images by default. Note that the module command will not work in images unless you have installed the required software and configured the environment correctly; we describe how to do this below. Once you have finished using your image, you can return to the ARCHER2 login node command line with the exit command: Singularity lolcow.simg:~> exit exit [user@archer2-login0 ~]$


### Interactive use on the compute nodes¶

The process for using an image interactively on the compute nodes is very similar to that for using them on the login nodes. The only difference is that you have to submit an interactive serial job to get interactive access to the compute node first.

For example, to reserve a full node for you to work on interactively you would use:

[user@archer2-login0 ~]$qsub -IVl select=1:ncpus=36,walltime=0:20:0,place=scatter:excl -A t01 qsub: waiting for job 234192.indy2-login0 to start ...wait until job starts... qsub: job 234192.indy2-login0 ready [user@r1i2n13 ~]$


Note that the prompt has changed to show you are on a compute node. Now you can use the image in the same way as on the login node.

[user@r1i2n13 ~]$module load singularity [user@r1i2n13 ~]$ singularity shell lolcow.simg
Singularity: Invoking an interactive shell within container...

Singularity lolcow.simg:~> exit
exit
[user@r1i2n13 ~]$exit [user@archer2-login0 ~]$


Note how we used exit to leave the interactive image shell and then exit again to leave the interactive job on the compute node.

### Serial processes within a non-interactive batch script¶

You can also use Singularity images within a non-interactive batch script as you would any other command. If your image contains a runscript then you can use singularity run to execute the runscript in the job. You can also use singularity exec to execute arbitrary commands (or scripts) within the image.

An example job submission script to run a serial job that executes the runscript within the lolcow.simg image that we built previously on an ARCHER2 login node would be as follows.

#!/bin/bash --login

# Slurm job options (name, compute nodes, job time)

#SBATCH -J simgtest
#SBATCH -o simgtest.o%j
#SBATCH -e simgtest.o%j
#SBATCH -p [partition]
#SBATCH -A [budget code]
#SBATCH --nodes=1
#SBATCH --time=00:10:00

# Change to the directory that the job was submitted from
cd $HOME # Load any required modules module load singularity # Run the serial executable singularity run$HOME/lolcow.simg


You submit this in the usual way and the standard output and error should be written to simgtest.o..., where the output filename ends with the job number.

### Parallel processes within a non-interactive batch script¶

Running a Singularity image within a parallel batch script is somewhat more involved. Let’s assume that the lolcow image contains an executable of the same name whose path is /opt/apps/lolcow. And we will also assume that the image contains an installation of a specific MPI library, openmpi v4.0.3 in this case.

Please note, the MPI library contained in the image must match the MPI library on the host. In practice, the MPI library used on the host and within the container must have the same vendor (e.g., MPICH, openmpi, Intel MPI) and the same version number (although, in some situations it might be possible to have different minor version numbers).

Below is an example job submission script that runs a Singularity container over four nodes.

#!/bin/bash --login

# Slurm job options (name, compute nodes, job time)

#SBATCH -J simgtest
#SBATCH -o simgtest.o%j
#SBATCH -e simgtest.o%j
#SBATCH -p [partition]
#SBATCH -A [budget code]
#SBATCH --nodes=4
#SBATCH --time=01:10:00

# setup resource-related environment
NNODES=$SLURM_JOB_NUM_NODES NCORESPN=$SLURM_CPUS_ON_NODE
NCORES=expr ${NNODES} \*${NCORESPN}

# setup local openmpi installation (env.sh exports OPENMPI_ROOT)
. $HOME/opt/openmpi-4.0.3/dist/env.sh # Change to the directory that contains lolcow.simg and lolcow.in cd$HOME

RUN_START=$(date +%s.%N) MPIRUN_PREFIX_OPT="--prefix${OPENMPI_ROOT}"
MPIRUN_RES_OPTS="-N ${NCORESPN} -n${NCORES} --hostfile ${HOME}/hosts --bind-to core" MPIRUN_MCA_OPTS="--mca btl ^sm --mca btl_openib_allow_ib true" MPIRUN_OPTS="${MPIRUN_PREFIX_OPT} ${MPIRUN_RES_OPTS}${MPIRUN_MCA_OPTS}"
SINGULARITY_OPTS="exec -B /etc/libibverbs.d"

mpirun $MPIRUN_OPTS singularity$SINGULARITY_OPTS $HOME/lolcow.simg /opt/apps/lolcow$HOME/lolcow.in &> $HOME/lolcow.out RUN_STOP=$(date +%s.%N)