Containers

This page was originally based on the documentation at the University of Sheffield HPC service

Designed around the notion of mobility of compute and reproducible science, Singularity enables users to have full control of their operating system environment. This means that a non-privileged user can "swap out" the Linux operating system and environment on the host for a Linux OS and environment that they control. So if the host system is running CentOS Linux but your application runs in Ubuntu Linux with a particular software stack; you can create an Ubuntu image, install your software into that image, copy the image to another host (e.g. ARCHER2), and run your application on that host in it’s native Ubuntu environment.

Singularity also allows you to leverage the resources of whatever host you are on. This includes high-speed interconnects (i.e. Slingshot on ARCHER2), file systems (i.e. /home and /work on ARCHER2) and potentially other resources.

Note

Singularity only supports Linux containers. You cannot create images that use Windows or macOS (this is a restriction of the containerisation model rather than Singularity).

Useful Links

About Singularity Containers (Images)

Similar to Docker, a Singularity container (or, more commonly, image) is a self-contained software stack. As Singularity does not require a root-level daemon to run its images (as is required by Docker) it is suitable for use on multi-user HPC systems such as ARCHER2. Within the container/image, you have exactly the same permissions as you do in a standard login session on the system.

In practice, this means that an image created on your local machine with all your research software installed for local development will also run on ARCHER2.

Pre-built images (such as those on DockerHub or SingularityHub) can simply be downloaded and used on ARCHER2 (or anywhere else Singularity is installed).

Creating and modifying images requires root permission and so must be done on a system where you have such access (in practice, this is usually within a virtual machine on your laptop/workstation).

Using Singularity Images on ARCHER2

Singularity images can be used on ARCHER2 in a number of ways, including:

Interactively on the login nodes
Interactively on compute nodes
As serial processes within a non-interactive batch script
As parallel processes within a non-interactive batch script

We provide information on each of these scenarios below. First, we describe briefly how to get existing images onto ARCHER2 so that you can use them.

Getting existing images onto ARCHER2

Singularity images are files, so, if you already have an image file, you can use scp to copy the file to ARCHER2 as you would with any other file.

If you wish to get a file from one of the container image repositories then Singularity allows you to do this from ARCHER2 itself.

For example, to retrieve an image from SingularityHub on Cirrus we can simply issue a Singularity command to pull the image.

[auser@uan01 ~]$ singularity pull hello-world.sif shub://vsoch/hello-world

The image located at the shub URI is written to a Singularity Image File (SIF) called hello-world.sif.

Once you have an image file, using it on the login nodes in an interactive way is extremely simple: you use the singularity shell command. Using the image we built in the example above:

[auser@uan01 ~]$ singularity shell hello-world.sif
Singularity>

Within a Singularity image your home directory will be available.

Once you have finished using your image, you can return to the ARCHER2 login node command line with the exit command:

Singularity> exit
exit
[auser@uan01 ~]$

Interactive use on the compute nodes

The process for using an image interactively on the compute nodes is very similar to that for using them on the login nodes. The only difference is that you have to submit an interactive serial job to get interactive access to the compute node first.

For example, to reserve a full node for you to work on interactively you would use:

auser@uan01:/work/t01/t01/auser> srun --nodes=1 --exclusive --time=00:20:00 --account=[budget code] \
                                      --partition=standard --qos=standard --pty /bin/bash

...wait until job starts...

auser@nid00001:/work/t01/t01/auser>

Note that the prompt has changed to show you are on a compute node. Now you can use the image in the same way as on the login node.

auser@nid00001:/work/t01/t01/auser> singularity shell hello-world.sif
Singularity> exit
exit
auser@nid00001:/work/t01/t01/auser> exit
auser@uan01:/work/t01/t01/auser>

Note

We used exit to leave the interactive image shell and then exit again to leave the interactive job on the compute node.

Serial processes within a non-interactive batch script

You can also use Singularity images within a non-interactive batch script as you would any other command. If your image contains a runscript then you can use singularity run to execute the runscript in the job. You can also use singularity exec to execute arbitrary commands (or scripts) within the image.

An example job submission script to run a serial job that executes the runscript within the hello-world.sif image that we downloaded previously to an ARCHER2 login node would be as follows.

#!/bin/bash --login

# Slurm job options (name, compute nodes, job time)

#SBATCH --job-name=helloworld
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=1
#SBATCH --time=00:10:00

#SBATCH --account=[budget code]
#SBATCH --partition=standard
#SBATCH --qos=standard

# Setup the batch environment
module load epcc-job-env

# Run the serial executable
singularity run $SLURM_SUBMIT_DIR/hello-world.sif

You submit this in the usual way and the standard output and error should be written to slurm-..., where the output filename ends with the job number.

Creating Your Own Singularity Images

As we saw above, you can create Singularity images by importing from DockerHub or Singularity Hub on ARCHER2 itself. If you wish to create your own custom image using Singularity then you must install Singularity on a system where you have root (or administrator) privileges - often your own laptop or workstation.

There are three different options to install Singularity on your local system: install Docker and use the Docker Singularity image to build Singularity containers, install a virtual machine that you can use to build Singularity images, or install Singularity on your local system.

For macOS and Windows users we recommend installing Docker Desktop and using the official Singularity image to build your own images. For Linux users, we recommend installing Singularity directly on your local system.

We cover the mechanism that uses Docker in more detail below for macOS and Windows users. If your local system is Linux, you can find information on installing Singularity on Linux distribution at:

Installing Singularity on Linux

Building Singularity images using Docker (macOS/Windows)

Tip

You should install Docker Desktop to allow you to build Singularity images using Docker. Instructions can be found at:

Once you have installed Docker, you can build Singularity images with a command similar to:

docker run -it --privileged --rm -v ${PWD}:/home/singularity quay.io/singularity/singularity:v3.7.0-slim build /home/singularity/my_image.sif /home/singularity/my_image.def

Tip

You can setup an alias to the long docker run command above to make it easier to use. e.g. If you are using bash or zsh you could use a command like alias dsingularity="docker run -it --privileged --rm -v ${PWD}:/home/singularity quay.io/singularity/singularity:v3.7.0-slim"

For information on how to write Singularity image definition files, see the Singularity documentation.

Tip

You can, of course, also get to a Singularity image via a Docker image (as Singularity can use images directly from the DockerHub). In this workflow you would create a Docker image, upload it to the DockerHub and then pull it using Singularity on ARCHER2.

Using Singularity with MPI on ARCHER2

MPI on ARCHER2 is provided by the Cray MPICH libraries with the interface to the high-performance Slingshot interconnect provided via the OFI interface. This means that your Singularity image must contain a version of the MPICH MPI library compiled with support for OFI to use MPI on ARCHER2 in a Singularity container. We provide instructions on creating a container with a version of MPICH compiled in the right way below but, for convenience, we also provide a base image file with MPICH compiled correctly built in so you can skip this step if you wish. Instructions on how to use this image as the base for your own images are also provided below. Finally, we provide an example of how to run a Singularity container with MPI over multiple ARCHER2 compute nodes.

Building an image with MPI from scratch

Note

These instructions are based on those used in the Reproducible computational environments using containers: Introduction to Singularity.

Warning

Remember, all these steps should be executed on your local system where you have administrator privileges, not on ARCHER2.

We will illustrate the process of building a Singularity image with MPI from scratch by building an image that contains MPI provided by MPICH and the OSU MPI benchmarks. In order to do this, we first need to download the source code for both MPICH and the OSU benchmarks. At the time of writing, the stable MPICH release is 3.4.1 and the stable OSU benchmark release is 5.7 - this may have changed by the time you are following these instructions.

Go ahead and download the source code:

wget http://www.mpich.org/static/downloads/3.4.1/mpich-3.4.1.tar.gz
wget https://mvapich.cse.ohio-state.edu/download/mvapich/osu-micro-benchmarks-5.7.tar.gz

Now create a Singularity image definition file that describes how to build the image:

Bootstrap: docker
From: ubuntu:20.04

%files
    /home/singularity/osu-micro-benchmarks-5.7.tar.gz /root/
    /home/singularity/mpich-3.4.1.tar.gz /root/

%environment
    export SINGULARITY_MPICH_DIR=/usr

%post
    apt-get -y update && DEBIAN_FRONTEND=noninteractive apt-get -y install build-essential libfabric-dev libibverbs-dev gfortran
    cd /root
    tar zxvf mpich-3.4.1.tar.gz && cd mpich-3.4.1
    echo "Configuring and building MPICH..."
    ./configure --prefix=/usr --with-device=ch3:nemesis:ofi && make -j2 && make install
    cd /root
    tar zxvf osu-micro-benchmarks-5.7.tar.gz
    cd osu-micro-benchmarks-5.7/
    echo "Configuring and building OSU Micro-Benchmarks..."
    ./configure --prefix=/usr/local/osu CC=/usr/bin/mpicc CXX=/usr/bin/mpicxx
    make -j2 && make install

%runscript
    exec /usr/local/osu/libexec/osu-micro-benchmarks/mpi/$*

A quick overview of what the above definition file is doing:

The image is being bootstrapped from the ubuntu:20.04 Docker image.
In the %files section: The OSU Micro-Benchmarks and MPICH tar files are copied from the current directory into the /root directory in the image.
In the %environment section: Set an environment variable that will be available within all containers run from the generated image.
In the %post section:
Ubuntu's apt-get package manager is used to update the package directory and then install the compilers and other libraries required for the MPICH build.
The MPICH .tar.gz file is extracted and the configure, build and install steps are run. Note the use of the --with-device option to configure MPICH to use the correct driver to support improved communication performance on a high performance cluster.
The OSU Micro-Benchmarks tar.gz file is extracted and the configure, build and install steps are run to build the benchmark code from source.
In the %runscript section: A runscript is set up that will echo the rank number of the current process and then run the command provided as a command line argument.

Note that base path of the the executable to run is hardcoded in the run script so the command line parameter to provide when running a container based on this image is relative to this base path, for example, startup/osu_hello, collective/osu_allgather, pt2pt/osu_latency, one-sided/osu_put_latency.

Info

You can find more information on Singularity definition file syntax in the Singularity documentation.

Now go ahead and build the Singularity image using Singularity via Docker:

docker run -it --privileged --rm -v ${PWD}:/home/singularity quay.io/singularity/singularity:v3.7.0-slim build /home/singularity/osu_benchmarks.sif /home/singularity/osu_benchmarks.def

Once you have successfully created your Singularity image file, osu_benchmarks.sif, use scp to copy it to ARCHER2 and you can test as described in the section below.

Tip

You can setup an alias to the long docker run command above to make it easier to use. e.g. If you are using bash or zsh you could use a command like alias dsingularity="docker run -it --privileged --rm -v ${PWD}:/home/singularity quay.io/singularity/singularity:v3.7.0-slim"

Tip

You can find a copy of the osu_benchmarks.sif image on ARCHER2 in the directory $EPCC_SINGULARITY_DIR if you do not want to build it yourself but still want to test.

Creating an image based on the base ARCHER2 MPI Singularity image

We have built an image with MPICH 3.4.1 built against OFI that you can use as a base image to install further software and create your own images. The image can be found on ARCHER2 at $EPCC_SINGULARITY_DIR/mpich_base.sif.

To use this image in your own image builds you should download it from ARCHER2 to the system where you are building your images. You can then add the following lines to your image definition file to start from this base image:

Bootstrap: localimage
From: /path/to/container/mpich_base.sif

(Remember to put the actual path to the mpich_base.sif image file that you have downloaded to your system.)

Running parallel MPI jobs using Singularity containers

Tip

These instructions assume you have a Singularity image uploaded to ARCHER2 that includes MPI provided by MPICH with the OFI interface. See the sections above for how to create such images.

Once you have uploaded your Singularity image that includes MPICH built with OFI to ARCHER2 then you can use it to run parallel jobs in a similar way to non-Singularity jobs. The example job submission script below uses the image we built above with MPICH and the OSU benchmarks to run the Allreduce benchmark on two nodes where all 128 cores on each node are used for MPI processes (so, 256 MPI processes in total).

#!/bin/bash

# Slurm job options (name, compute nodes, job time)
#SBATCH --job-name=singularity_parallel
#SBATCH --time=0:10:0
#SBATCH --nodes=2
#SBATCH --tasks-per-node=128
#SBATCH --cpus-per-task=1

# Replace [budget code] below with your budget code (e.g. t01)
#SBATCH --partition=standard
#SBATCH --qos=standard
#SBATCH --account=[budget code]

# Setup the job environment (this module needs to be loaded before any other modules)
module load epcc-job-env

# Set the number of threads to 1
#   This prevents any threaded system libraries from automatically 
#   using threading.
export OMP_NUM_THREADS=1

# Set the LD_LIBRARY_PATH environment variable within the Singularity container
# to ensure that it used the correct MPI libraries
export SINGULARITYENV_LD_LIBRARY_PATH=/opt/cray/pe/mpich/8.0.16/ofi/gnu/9.1/lib-abi-mpich:/usr/lib/x86_64-linux-gnu/libibverbs:/opt/cray/pe/pmi/6.0.7/lib:/opt/cray/libfabric/1.11.0.0.233/lib64:/usr/lib64/host:/.singularity.d/libs

# Set the options for the Singularity executable
#   This makes sure the locations with Cray Slingshot interconnect libraries are available
singopts="-B /opt/cray,/usr/lib64:/usr/lib64/host,/usr/lib64/tcl,/var/spool/slurmd/mpi_cray_shasta"

# Launch the parallel job
srun --hint=nomultithread --distribution=block:block singularity run $singopts osu_benchmarks.sif collective/osu_allreduce

The only changes from a standard submission script are:

We set the environment variable SINGULARITY_LD_LIBRARY_PATH to ensure the correct libraries are available within the container to be able to use Cray MPICH.
srun calls the singularity software with the image file we created rather than the parallel program directly.

Important

Remember that the image file must be located on /work to run jobs on the compute nodes.

If the job runs correctly, you should see output similar to the following in your slurm-*.out file:

# OSU MPI Allreduce Latency Test v5.7
# Size       Avg Latency(us)
4                       6.81
8                       6.86
16                      6.90
32                      7.09
64                     10.79
128                    10.97
256                    13.07
512                    15.50
1024                   17.35
2048                   22.49
4096                   41.13
8192                   54.00
16384                  75.09
32768                 120.40
65536                 243.26
131072                614.60
262144                618.11
524288                877.92
1048576              2981.61

Containers

Useful Links

About Singularity Containers (Images)

Using Singularity Images on ARCHER2

Getting existing images onto ARCHER2

Interactive use on the login nodes

Interactive use on the compute nodes

Serial processes within a non-interactive batch script

Creating Your Own Singularity Images

Building Singularity images using Docker (macOS/Windows)

Using Singularity with MPI on ARCHER2

Building an image with MPI from scratch

Creating an image based on the base ARCHER2 MPI Singularity image

Running parallel MPI jobs using Singularity containers