Singularity

Overview

Singularity is a container platform. It allows you to create and run containers that package up pieces of software in a way that is portable and reproducible. You can build a container using Singularity on your laptop, and then run it on many of the largest HPC clusters in the world, local university or company clusters, a single server, in the cloud, or on a workstation down the hall. Your container is a single file, and you don’t have to worry about how to install all the software you need on each different operating system.

Details at: https://apptainer.org/user-docs/3.8/introduction.html

Availability

Singularity is available on Gadi by loading the singularity  module. This module will always point to the latest installed version; previous versions will not be made available under any circumstances due to the highly-privileged nature of the software. If you would like to know the currently available version, you can run

$ module load singularity
$ singularity version

Usage Notes

Please note that we consider the use of Singularity to be a "measure of last resort", for when native builds (i.e. those against Gadi's core system image and anything provided via modules) are impossible. For example, if a particular package is only available via a pre-built Singularity image. While "portability" and "reliability" are often touted as reasons for using containers, our experience is actually the opposite: forcing the use of containers often makes it less reliable and/or performant – especially when you're trying to use the "high performance" features of a HPC cluster.

For example, if you just download an image from a repository, chances are the binaries contained within have not been optimised with any specific CPU architecture. The default architectural options in most compilers only enable the SSE2 instruction set – which was introduced over two decades ago, in 2000. By only allowing SSE2 instructions, you've artificially crippled the performance of your program, with a theoretical maximum of only an eighth that of what Gadi's CPUs are capable of. In order to obtain access to this extra performance, you would need to modify the build recipe for your container to include the appropriate compiler flags to enable the extra instruction sets available on Gadi – but this would then mean that you've created an image that would only work on Gadi (or other systems that have those instruction sets), the exact opposite of the "portability" touted by using the container in the first place.

Another thing to consider is how your container will interface with the system as a whole. As soon as you want to move beyond isolated, single-node jobs your container needs to be aware of how to access other nodes and communicate between them. On Gadi, launching processes on remote nodes in a job is accomplished through PBS's TM interface; this means that you either need to launch the remote processes outside of the container, or you'll need to include the appropriate libraries and utilities inside the container (the latter again making the container Gadi-specific).

Using MPI

The canonical way for processes to communicate in a HPC environment is via MPI. However, in order to get good performance from an MPI library it needs to know how to use the high-performance interconnect – something that the MPI libraries that are provided in pre-built containers generally lack. However you solve this, you're then creating a custom container specifically for Gadi, and you're needing to import a lot of Gadi's operating system into the container – at which point, you might as well just do a native build. Moreover, if you're not using the drivers and MPI libraries that we provide then you will probably need to rebuild your container every time we update the driver stack (generally each quarterly maintenance period). Please see Singularity's documentation for more information on running MPI applications via Singularity. Note that only limited guidance may be given by NCI if your MPI application doesn't run as expected using Singularity.

Creating containers

Build your own container

You should create new singularity images on a machine where you have the needed permissions (such as your desktop machine, if running a compatible operating system) and then upload the image to Gadi from there. You can use singularity build command to build your image from a definition file.

Example:

$ sudo singularity build /path/to/mycontainer.sif /path/to/mydefinition.def

Please see the Singularity User Guide for more information on creating new images from scratch

Pulling pre-built images

You can download a pre-built container on Gadi from using the normal singularity pull  command (from Docker hub, Biocontainers etc). Example:

$ singularity pull --dir /path/to/my/dir docker://alpine:latest

Please see the Singularity User Guide for more information on pulling a pre-built container.

Filesystem mounts

When launching a container on Gadi, the following file systems will be automatically mounted along with the user's home directory.

• /apps
• /g
• /opt/nci
• /scratch
• /jobfs

Similarly, the NVIDIA driver libraries needed to utilise Gadi's GPUs are automatically presented inside the container at .singularity.d/libs.

User bind mounts

User bind mounts allow you to place a directory from Gadi into a specific place inside the container. This may be required by some containers which, for example, expect input data to be present at a specific path.

The --bind option can be used at runtime to specify a user-defined bind path.

Example:

$ singularity exec --bind /scratch/XXX/YYY/input:/data my_container.sif ls /data
# where /scratch/XXX/YYY/input is the path in the host to your input directory and /data is a path in the container.

Checking containers for issues

You can use the NCI provided singularity-check-container tool to check your container for any issues which may prevent it from running as expected on Gadi.

Please note that we can generally only provide limited support if your container isn't working the way you expect – while it's still running on the same physical Gadi hardware, you're effectively using your own operating system and libraries, and so in essence you're not really using the Gadi environment we provide.

Launching containers

Containers can be launched using singularity run or singularity exec commands

• run

  If a command is associated with the run action, it is executed when you execute singularity run on a sif file.

  $ singularity run mycontainer.sif

• exec

  This allows you to execute any command inside the container

  $ singularity exec mycontainer.sif cat /etc/os-release

Executing inside a PBS Job

These can be used in a PBS jobscript similar to running any other program

#!/bin/bash
 
#PBS -l ncpus=X
#PBS -l mem=XGB
#PBS -q normal
#PBS -P <proj>
#PBS -l walltime=00:0X:00
#PBS -l wd

# Load module, always specify version number.
module load singularity

# Must include `#PBS -l storage=scratch/ab12+gdata/yz98` if the job
# needs access to `/scratch/ab12/` and `/g/data/yz98/`. Details on:
# https://opus.nci.org.au/display/Help/PBS+Directives+Explained

# Run program
singularity exec mycontainer.sif /path/to/my/program