Containers on ponyland

Nowadays, software is often being distributed in the form of containers. Containers ship software along with all their dependencies and in doing so facilitate installation and maintenance of complex software, the only thing that is shared with the host system is the Linux kernel itself.

The most popular and ubiquitous container platform is Docker. On shared high-performance computer clusters such as Ponyland, however, docker is not the best solution due to security concerns. When we run containers on ponyland, we want everything to run under normal user permissions.

We have chosen Apptainer (formally known as Singularity) as container platform for Ponyland. This solution is specifically tailored for High Performance Computing clusters like ours. Furthermore, it interoperates fine with Docker/OCI containers.

Install apptainer

You have to install apptainer on every server you want to use it. Apptainer and all containers you use with it will be held locally on the server in your /scratch/$USER directory so will not be shared on the entire cluster. To build and install apptainer from source, simply run:

$ /vol/customopt/apptainer/install.sh

This script may take a while to finish. It will also adapt your ~/.bashrc and to add the APPTAINER_CACHEDIR variable and update the PATH variable. So after running this script, make sure to refresh your shell before running apptainer.

Using apptainer

You can obtain container images from popular registries such as Docker Hub and run them right away. For example:

$ apptainer run docker://proycon/frog

This will download the container image from Docker Hub, convert it the SIF format for apptainer, and finally run it, all in one command. Containers may either present you with an interactive shell (you will usually see the Apptainer> prompt in that case) or they often run an application directly (like Frog does).

If running applications via containers is too much typing for you, you can always consider setting an alias in your ~/.bashrc:

alias frog="apptainer run docker://proycon/frog"

Depending on the container image, parameters to the software can be often specified directly as command-line parameters. Suppose you are in a directory where you have a file text.txt, then you can simply do the following:

$ apptainer run docker://proycon/frog test.txt

However, something like this would not work:

$ apptainer run docker://proycon/frog /vol/tensusers/proycon/test.txt

When you're in a container it sees only its own filesystem. The current working directly and your home directory are automatically mounted into the container. To make your data accessible you need to explicitly make it available to the container upon starting it. This is done by binding or mounting your data directory as follows:

$ apptainer run --bind /vol/tensusers/proycon:/data docker://proycon/frog /data/test.txt

This says, make the /vol/tensusers/proycon directory (and everything under it), available inside the container under /data. It corresponds to the -v parameter you may be used to from Docker.

It is important to understand that containers are non-persistent, and by default even immutable. That means you can't write inside the container's file system by default. The persistence means that even if you could write, the changes would be gone the moment the container ends. It is therefore important to ensure all data lives on external mounts.

Some container images need write access for themselves, or you may want to have write access yourself so you have an environment to play around in and install whatever you want as if you were root.

In the following example we run an Alpine Linux container where we can be (fake) root and write:

$ apptainer run --fakeroot --writable docker://alpine

If the --fakeroot option is not yet supported, there's a chance things may also work with just --writable.

For further documentation, please see the Apptainer user guide.

Troubleshooting

If you have trouble getting docker containers to run (some containers may fail to due the differences between docker and apptainer), try adding the --compat flag. This will make apptainer behave a bit more like docker (it also means you won't automatically get mounts like your home directory and current working directory). Check https://apptainer.org/docs/user/main/docker_and_oci.html#troubleshooting for further troubleshooting options.

If you get errors like No space on device when running a container, then it's most likely the /scratch disk on the server you are running on is full or low on space, please check and clean up as needed. Caches are kept under /scratch/$USERNAME/apptainer.

You may also run into issues with the normal /tmp/ directory not having enough space. You can set your own custom temp directory with export APPTAINER_TMPDIR=/path/to/tmp.