Singularity Jobs

3 minute read

What is Singularity

In short, Singularity is a program that will allow a user to run code or command, within a customized environment. We will refer to this customized environment as a container. This type of container system is common, the more popular one being Docker. Since Docker requires root access and HPC users are not typically granted these permissions, we use Singularity instead. Docker containers can be used via Singularity, with varying compatibility.

Singularity is forking into 2 branches:

• Singularity-CE - Community Edition from Sylabs.io
• Apptainer - Original Sinularity open source project

We will be using Apptainer when it is ready for production use. However, in the meantime, singularity-ce is currently availble on the cluster.

Limitations

Currently we are not supporting Slurm jobs bening submitted from within a container. If you load the container centos/7.9 and try to submit a job from within it will fail. Please contact support in order to work around this issue.

How to use Singularity

Singularity is easy to use. You can run singularity in an interactive mode by calling a shell, or you can run singularity in a non-interactive mode and just pass it a script. This 2 modes are very similar to job submission on the cluster; srun is used for interactive, while sbatch is used for non-interactive.

Interactive Singularity

When running singularity you need to provide the path to a singularity image file. For example, this would be the most basic way to get a shell within your container:

module load singularity
singularity shell /path/to/singularity/image

Typically when we install singularity enabled software, we will also create an environment variable which holds the location of the singularity image file. For example, this would be how we would use the centos module:

module load centos
singularity shell $CENTOS7_SING

There is a special shortcut for the centos module that allows us to run the above more simply, as:

module load centos
centos.sh

Here is a another example that utilizes an interactive job:

module load centos
srun -p batch --mem=1g -c 4 --time=2:00:00 --pty centos.sh

Non-Interactive Singularity

When running singularity as non-interactive, the same basic rules apply, we need a path to our singularity image file as well as a command.

Basics

For example, here is the basic syntax:

module load singularity
singularity exec /path/to/singularity/image someCommand

Using centos as an example, you can execute an abitraty command like so:

module load singularity
singularity exec $CENTOS7_SING cat /etc/redhat-release

Shortcuts

Now using the centos shortcut:

module load centos
centos.sh "cat /etc/redhat-release"

Here is a more complex example with modules:

module load centos
centos.sh "module load samtools; samtools --help"

Jobs

Here is an example submitted as a job:

module load centos
sbatch -p batch --wrap="centos.sh 'module load samtools; samtools --help'"

Variables

Here is an example with passing environment variables:

export SINGULARITYENV_SOMETHING='stuff'
centos.sh 'echo $SOMETHING'

Notice: Just add the SINGULARITYENV_ prefix to pass any varibales to the centos container.

Enable GPUs

First review how to submit a GPU job from here. Then request an interactive GPU job, or embed one of the following within your submission script.

In order to enable GPUs within your container you need to add the --nv option to the singularity command:

module load centos
singularity exec -nv $CENTOS7_SING cat /etc/redhat-release

However, when using the centos shortcut it is easier to just set the following environment variable then run centos.sh as usual:

module load centos
export SINGULARITY_NV=1
centos.sh