When you want to run Docker containers on the clusters.
Authentication
You can use public images without being authenticated (shifterimg login) in the following cases:
| Image source | Public image | Login required |
|---|---|---|
| Dockerhub | Yes | No |
| Dockerhub | No | Yes, dockerhub |
| C4science registry | Yes | No |
| C4science registry | No | Yes, c4science registry |
When using authentication, it's relative to the source you want to use (dockerhub and c4science registry supported) and the authentication is saved per cluster.
You need to ask for an account on the c4science registry
Running a Docker image with Shifter - Step by step
Prerequisite
You need to have Docker installed on your machine
Get a docker image from dockerhub for instance
$ docker pull alpine:latest $ docker images
- Set up your machine
Login on the registry from your local Docker installation (you need an account, see #RunningDockerimagesusingShifter-Accountonthec4scienceregistry)
$ docker login registry.c4science.ch Username (username): username Password: Login Succeeded
Upload a Docker image to the registry
- On the web interface, create a Project on the registry (private or public)
Tag the image you want to upload on your local machine and push it to the registry
NOTE: Do not use the `-` character in the tag name, only letters, numbers and underscore$ docker tag alpine:latest registry.c4science.ch/yourproject/alpine:latest $ docker push registry.c4science.ch/yourproject/alpine:latest
Pull an image on Shifter and specify a user or group ACL
From each cluster frontend (i.e.: fidis.epfl.ch), login to the registry, pull the image and check it's was pulled with success.
$ shifterimg login default username: <username> default password: $ shifterimg pull yourproject/alpine:latest $ shifterimg images tcm docker READY 9797e5e798 2018-03-15T16:00:59 yourproject/alpine:latest
You can specify one or multiple (separated by a comma) LDAP username and/or group so the image is only available to those people
$ id $ shifterimg --group scitas-ge --user aubort,user2 pull yourproject/alpin
- To update the user/group ACL you can re-run the pull command
- The images are unique for each cluster (deneb, fidis, helvetios, izar)
To view the full info about the images (warning: JSON):
- Run the image
You can submit the following Slurm script with the sbatch command
#!/bin/bash -l #SBATCH --nodes 1 #SBATCH --ntasks 1 #SBATCH --cpus-per-task 1 #SBATCH --mem 1024 srun shifter --image yourproject/alpine ls /etc
Interactive Shell (Bash)
To have an interactive shell within your image, simply use this:
$ srun --pty shifter --image yourproject/alpine bash
Using GPUs
On Deneb shifter runtime is installed on the GPU nodes. You need prior access to the GPUs nodes, see FAQ
[aubort@deneb1 ~]$ srun --gres gpu:1 --partition gpu --qos gpu shifter --image library/debian:stable-slim nvidia-smi -L GPU 0: Tesla K40m (UUID: GPU-21730043-7144-85e7-d251-7834adb2d1ee) [aubort@deneb1 ~]$ srun --gres gpu:1 --partition gpu --qos gpu shifter --image library/nvidia-cuda:9.1-runtime /home/aubort/gpu/cuda-samples/bin/x86_64/linux/release/simpleCUFFT [simpleCUFFT] is starting... GPU Device 0: "Tesla K40m" with compute capability 3.5 Temporary buffer size 448 bytes Transforming signal cufftExecC2C Launching ComplexPointwiseMulAndScale<<< >>> Transforming signal back cufftExecC2C
FEEDBACK is welcome as this feature is experimental.
Account on the c4science registry
- Request an account
- Change your password on https://registry.c4science.ch
Related articles