Custom Applications with Containers
If you are new to using containers on Torch, start with our Apptainer tutorial. It covers running containers, accessing files, pulling images, and customizing container environments. Below is a condensed refresher on the same topic.
What Is Apptainer?
Apptainer is a container based Linux kernel workspace that works just like Docker. You can run pre-built programs in containers without having to worry about the pre-install environment. You can even run Docker containers with Apptainer. Please see the Apptainer and Docker documentation by Syslabs for details about all the ways Apptainer supports Docker. For a detailed introduction on Apptainer, visit their official site.
Apptainer is the continuation of the Singularity project. When we transition to the new Torch cluster you will only see reference to Apptainer. The reason for this is that there were two container projects with the name Singularity. The original free and open-source (FOSS) project and a closed-source corporate fork. The FOSS version has renamed itself to Apptainer to avoid confusion and this is the version we use.
Why Do We Use Apptainer
There are multiple reasons to use Apptainer on the HPC clusters:
- Security: Apptainer provides a layer of security as it does not require any root access on our clusters. This makes it safer against malware and bad scripts that might jeopardize the outer system. Thus we only support Apptainer on our clusters (there are not other options such as Kubernetes or Docker on our clusters right now).
- Containerization: Apptainer will run all your images (packaged and prebuilt programs) inside of its containers, each container works like a small vm. They contain all the required environment and files of a single Linux kernel and you don't have to worry about any pre-installation issues.
- Inter-connectivity: Containers are able to talk to each other, as well as the home system, so while each container has its own small space, they are still a part of a big interconnected structure. Thus enabling you to connect your programs.
- Accessibility: Probably the most important feature of all, Apptainer allows you to run your program in 2 to 3 simple steps.
How to Create an Apptainer Container
So what if you want to create an image from your container and save it for a rainy day?
Apptainer documentation has instructions for building containers for your convenience. Please read through them to create your own Apptainer container and package it into an image!
Similarly, you can build docker containers using the information from Docker's documentation. You can then upload them onto docker hub and pull them using Apptainer. Apptainer supports all Docker images!
Apptainer vs Docker
Why are there so many mentions of Docker? The reason is that Apptainer is essentially compatible with Docker and you don't need to relearn Apptainer if you already have experience with Docker. Now let's get into some pros and cons between the two programs.
- Docker is more accepted commercially than Apptainer. You can download and run Docker on your own computer with any operating system and build containers with ease while Apptainer is used in a more academic setting. Apptainer primarily supports Linux environments and is designed for HPC use cases.
- However, Docker requires root or admin access for the operating system it deploys on, and our clusters do not offer that access to any software that requires this criteria. Thus Docker is not available on the clusters and Apptainer is.
- A silver lining in all of this is that Apptainer fully supports Docker images and you can do everything in Docker and push your image to Docker Hub and pull them on the clusters. Thus making sure that you don't need to relearn workflows and can just use it through the simplest of commands in this documentation.
Getting Started
Common Commands
Pull an image from Docker Hub:
apptainer pull docker://<image>
Run a container:
apptainer run <image>.sif
Execute a command inside a container:
apptainer exec <image>.sif <command>
Start an interactive shell:
apptainer shell <image>.sif
For full information and documentation please visit Apptainer. Good luck with Apptainer, and have fun!