/
Podman

Podman

Owned by Dustin Lo
Last updated: Jan 19, 2022
6 min read

podman is:

… a daemon-less container engine for developing, managing, and running OCI Containers on your Linux System

For the most part podman behaves very similarly to docker, with much of the commands being identical, ie:

  • podman images - lists all images in host repository

  • podman ps - lists all running containers

  • podman run - run image in container (all flags work as well, ie. --rm -it --net=host -v, etc.)

  • etc.

 

Enabling the Podman service

Podman comes with a optional service/API that can be enabled:

user@ubuntu-20-04:~$ systemctl --user enable podman.socket user@ubuntu-20-04:~$ systemctl --user start podman.socket user@ubuntu-20-04:~$ systemctl --user status podman.socket ● podman.socket - Podman API Socket Loaded: loaded (/usr/lib/systemd/user/podman.socket; enabled; vendor preset: enabled) Active: inactive (dead) since Thu 2022-01-06 15:59:44 PST; 16min ago Triggers: ● podman.service Docs: man:podman-system-service(1) Listen: /run/user/1000/podman/podman.sock (Stream) Jan 06 14:32:11 dustin-ThinkPad-T420 systemd[1042]: Listening on Podman API Socket. Jan 06 15:59:44 dustin-ThinkPad-T420 systemd[1042]: podman.socket: Succeeded. Jan 06 15:59:44 dustin-ThinkPad-T420 systemd[1042]: Closed Podman API Socket.

Makes your Podman service accessible by adding the --remote flag to your Podman command. ex. podman --remote run <image> ...

Not sure if it behaves similarly to the docker daemon b/c podman touts itself as a daemon-less tool

Mounting /run/user/1000/podman/podman.sock or /run/podman/podman.sock doesn’t seem to do anything, but further research may be needed

Podman Registry

We run a local docker registry on http://localhost:5050 for internal usage

Podman stores the list of registries in registries.conf:

  • /etc/containers/registries.conf for system wide config

  • $HOME/.config/containers/registries.conf for a single user

Podman will enable these registries by default to allow for short name usage when pulling images

unqualified-search-registries = ['registry.fedoraproject.org', 'registry.access.redhat.com', 'registry.centos.org', 'docker.io']

To enable the local Podman image registry (after creating it):

[[registry]] location="localhost:5000" insecure=true

Running podman in a container

Docker has a feature where you can start a sibling container while in a container simply by mounting -v /var/run/docker.sock:/var/run/docker.sock beforehand

  • It’s useful b/c chimera runs it workflows (or jobs) from within a container itself

Podman doesn’t quite have that feature, but it can be achieved through a workaround:

  • podman isn’t ran as a daemon so mounting a docker.sock file wouldn’t work

  • The user has to be root (but doing more research to see if it can be done without)

 

First activate the root user:

user@ubuntu-20-04:~$ sudo su - root@ubuntu-20-04:~#

Pull down some images:

root@ubuntu-20-04:~# podman pull podman:v3.3.1 Resolved "podman" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf) Trying to pull quay.io/podman/stable:v3.3.1... podman Getting image source signatures Copying blob bf213f2c84f8 done Copying blob bf213f2c84f8 done Copying blob 7c013e7a24b3 done Copying blob 5f41ed556d45 done Copying blob ecfb9899f4ce done Copying blob 2447ce384253 done Copying blob 8ba0306866b4 done Copying blob c2749003bd4e done Copying blob 2c7890a2c392 done Copying blob 9d19764e964e done Copying config 803fcf0206 done Writing manifest to image destination Storing signatures 803fcf0206f7751cb3a9e5b91ad53b833b649681ee51841c4deb56a855791beb root@ubuntu-20-04:~# podman pull python:3.9.7-slim Resolved "python" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf) Trying to pull docker.io/library/python:3.9.7-slim... Getting image source signatures Copying blob b489ab31cb6f done Copying blob 76cee30eba29 done Copying blob f8416d8bac72 done Copying blob 567b57d5964a done Copying blob 25db5d180001 done Copying config 66f4843b72 done Writing manifest to image destination Storing signatures 66f4843b721f81413e67156177981ee800736c895d44a25c22bea05eb744baed

Images are now in the repository:

root@ubuntu-20-04:~# podman images REPOSITORY TAG IMAGE ID CREATED SIZE quay.io/podman/stable v3.3.1 803fcf0206f7 14 hours ago 437 MB docker.io/library/python 3.9.7-slim 66f4843b721f 13 days ago 128 MB

According to oracle’s documentation:

 … images are stored in the /var/lib/containers directory when Podman is run by the root user. For standard users, images are typically stored in $HOME/.local/share/containers/storage/.

So by mounting /var/lib/containers to the container it will give it access to the host (or parent) images

  • for some reason we need to mount /run/netns as well

  • make sure to add the --privileged flag to let your container run podman commands (docs)

root@ubuntu-20-04:~# podman run --privileged \ -v /var/lib/containers:/var/lib/containers \ -v /run/netns:/run/netns \ --rm -it quay.io/podman/stable:v3.3.1 bash [root@bcedb6a1962e /]# podman images REPOSITORY TAG IMAGE ID CREATED SIZE quay.io/podman/stable v3.3.1 803fcf0206f7 14 hours ago 437 MB docker.io/library/python 3.9.7-slim 66f4843b721f 13 days ago 128 MB

If you’re receiving an error similar to this:

ERRO[0000]... error acquiring lock N for volume <image>: file exists

It can be resolved by running podman system renumber (docs)

And the container can now spawn new podman containers:

[root@755d60f4707a /]# podman run 66f4843b721f echo '{"foo": "bar"}' | python -m json.tool { "foo": "bar" }

 

Running podman in a container (rootless)

Method 1:

If you mount the -v $HOME/.local/share/containers/storage:/var/lib/shared to the container it will have access to the host images

Was unable to run a sibling container using this method, please look at method 2

  • more research needed to see if it behaves the same as a rootful user

user@ubuntu-20-04:~$ podman images REPOSITORY TAG IMAGE ID CREATED SIZE docker.io/library/python 3.9.7-slim 66f4843b721f 2 weeks ago 128 MB quay.io/podman/stable v3.3.0 b648fbc5870f 3 weeks ago 387 MB user@ubuntu-20-04:~$ podman run --rm -it --privileged \ -v $HOME/.local/share/containers/storage:/var/lib/shared \ quay.io/podman/stable:v3.3.0 bash [root@feb5564e8f5a /]# podman images REPOSITORY TAG IMAGE ID CREATED SIZE R/O docker.io/library/python 3.9.7-slim 66f4843b721f 2 weeks ago 128 MB true quay.io/podman/stable v3.3.0 b648fbc5870f 3 weeks ago 387 MB true

Method 2:

According to this Github issue, mount the storage directory twice:

user@ubuntu-20-04:~$ podman run -it --privileged \ -v /run/user/$UID:/run/user/$UID \ -v $HOME/.local/share/containers:$HOME/.local/share/containers \ -v $HOME/.local/share/containers:/var/lib/containers \ podman:v3.3.1 bash

inside the docker container able to see all the images (from host) and run a second container from inside

[root@9fd12376edc6 /]# podman images REPOSITORY TAG IMAGE ID CREATED SIZE quay.io/podman/stable v3.3.1 3758669a89c2 3 months ago 374 MB docker.io/library/python 3.9.7-slim 66f4843b721f 4 months ago 128 MB quay.io/podman/stable v3.3.0 b648fbc5870f 4 months ago 387 MB [root@9fd12376edc6 /]# podman run -it podman:v3.3.1 bash [root@9fd12376edc6 /]#

in a new tab was able to see 2 sibling containers (from host) running simultaneously:

user@ubuntu-20-04:~$ podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 9fd12376edc6 quay.io/podman/stable:v3.3.1 bash 19 seconds ago Up 20 seconds ago confident_noyce 27f952b0854b quay.io/podman/stable:v3.3.1 bash 3 seconds ago Up 3 seconds ago elegant_northcutt

Mounting work directories through multiple layers of podman containers

created a directory in /tmp/test with .txt files and when mounting the directory through multiple layers of podman containers the volume can still be accessed

root@ubuntu-20-04:~# podman run --rm -it --privileged \ > -v /tmp/test:/tmp/test \ > -v /var/lib/containers/storage:/var/lib/containers/storage \ > -v /run/libpod:/run/libpod \ > -v /run/containers/storage:/run/containers/storage \ > quay.io/podman/stable:v3.3.1 bash [root@6163ad73434f /]# podman run --rm -it --privileged \ -v /tmp/test:/tmp/test \ -v /var/lib/containers/storage:/var/lib/containers/storage \ -v /run/libpod:/run/libpod \ -v /run/containers/storage:/run/containers/storage \ quay.io/podman/stable:v3.3.1 bash [root@6163ad73434f /]# ls -l /tmp/test/ total 0 -rw-rw-r-- 1 podman podman 0 Sep 27 18:20 1.txt -rw-rw-r-- 1 podman podman 0 Sep 27 18:20 2.txt -rw-rw-r-- 1 podman podman 0 Sep 27 18:20 3.txt -rw-rw-r-- 1 podman podman 0 Sep 27 18:20 4.txt -rw-rw-r-- 1 podman podman 0 Sep 27 18:20 5.txt -rw-rw-r-- 1 podman podman 0 Sep 27 18:20 6.txt -rw-rw-r-- 1 podman podman 0 Sep 27 18:20 7.txt -rw-r--r-- 1 podman podman 0 Sep 27 18:23 8.txt

According to podman documentation:

… people intend to use rootless Podman - they want their UID inside and outside the container to match. Thus, we provide the --userns=keep-id flag, which ensures that your user is mapped to its own UID and GID inside the container.

It is also helpful to distinguish between running Podman as a rootless user, and a container which is built to run rootless. If the container you're trying to run has a USER which is not root, then when mounting volumes you must use --userns=keep-id. This is because the container user would not be able to become root and access the mounted volumes.

Changes to HySDS

With HySDS needing the option of supporting both docker and podman (and also singularity) there will be a large refactor required in job_worker.py (source code)

  • instead of separating it docker podman and singularity through different branches it should be config based

    • this will avoid having to keep each branch up-to-date and potential merge conflicts

  • The container engine logic can be split into separate classes: Docker, Podman & Singularity

  • gosu

 

Issues:

  • Unable to load the .tar.gz image into podman

    • $ podman load -i /data/work/cache/docker_image.tar.gz Error: payload does not match any of the supported image formats (oci, oci-archive, dir, docker-archive)

    • podman load < /data/work/cache/docker_image.tar.gz works but the < character doesn’t work with subprocess

  • Unable to map user 1011 to into pge-base image (or any image based off of pge-base)

    • same with using the --privileged flag; both do nothing

    • ~$ podman run -it -u 1011:1011 pge-base:latest bash ~$

  • By default if podman runs a container without --user it will run as root

    • was able to run a random docker image w/ mounting /data/work/jobs and able to edit the directory

    • $ podman run -it --rm \ -w /data/work/jobs/2022/01/12/22/43/foo/bar/directory \ -v /data/work/jobs:/data/work/jobs:Z golang:latest bash

    • How to debug issues with volumes mounted on rootless containers

      • the third solution listed (adding the --userns=keep-id flag) seemed to work well without compromising the host directories

      • second solution changes the ownership of directories in host, makes the celery worker unable to write/create directories

Related content

Note: JPL employees can also get answers to HySDS questions at Stack Overflow Enterprise:

https://stackoverflow.jpl.nasa.gov/