Adding Support for Podman
- Michael Cayanan
Overview
With the resolution of HC-522: Update Core such that either podman or docker is supported by a single Verdi containerClosed , this added the capability of HySDS Core to optionally be able to support the use of Podman instead of Docker, depending on project needs. The implementation was largely influenced by the analysis that was done on the use of Podman within HySDS core: Analysis of Podman Integration into HySDS core | Testing/Experimenting with podman using vagrant (Oracle Linux 8 VM)
This wiki is intended to provide a guide on how projects can set up their cluster so that Podman can be used.
Setup
Update SDS Config
In the SDS config file (~/.sds/config), add the following settings:
CONTAINER_ENGINE - set it to ‘podman’
HOST_VERDI_HOME
VERDI_HOME
VERDI_SHELL
The following table describes these settings in a bit more detail. Note that if you do not specify these in the SDS config, the default values specified in this table will be used.
SETTING | DESCRIPTION | DEFAULT VALUE |
|---|---|---|
CONTAINER_ENGINE | Tells the PCM system which container engine to use. Valid values are either ‘docker’ or ‘podman’ | docker |
HOST_VERDI_HOME | Tells the PCM system the base directory of where the code bundle will be deployed to on the workers. Default is $HOME. This value gets used when the celery worker starts up via supervisord. See the ‘celery worker queue’ service configuration in the supervisord.conf in a worker. This is also used to properly set the volume mounts in podman as podman seems to only recognize volume mounts at the host level than the container level. | $HOME |
VERDI_HOME | Tells the PCM system where the HySDS code resides inside the Verdi container. Default is ‘/home/ops’. This ultimately gets used when the --passwd-entry flag option is set in the podman run commands. This is also used to properly set the volume mounts in podman as podman seems to only recognize volume mounts at the host level than the container level. | /home/ops |
VERDI_SHELL | Tells the PCM system which shell to default to when the container is in use. Default is ‘/bin/bash’. This ultimately gets used when the --passwd-entry flag option is set in the podman run commands. | /bin/bash |
CeleryConfig and install.sh Updates
Make sure to pull in the latest CeleryConfig.py and install.sh updates into your project repository:
https://github.com/hysds/hysds/blob/develop/configs/celery/celeryconfig.py.tmpl
https://github.com/sdskit/sdscli/blob/develop/sdscli/adapters/hysds/files/install.sh
CeleryConfig - Podman Settings
There are settings in the celeryconfig specific to Podman:
PODMAN_CFG = {
"set_uid_gid": True,
"set_passwd_entry": True,
"cmd_base": {
"userns": "keep-id",
"security-opt": "label=disable"
},
"environment": [
"HOST_VERDI_HOME",
"HOST_USER"
]
}Basically, the settings here is used to build out the podman run commands that HySDS core uses when they need to run a container.
By default, if you leave this alone, the following podman run command gets built out as the following:
$ podman --remote --url unix:/run/user/<user id>/podman/podman.sock run --init --rm \
-e HOST_VERDI_HOME -e HOST_USER \
-u $UID:$(id -g) \
--passwd-entry=<HOST_USER>:*:$UID:$(id -g)::<VERDI_HOME>:<VERDI_SHELL> \
--userns=keep-id \
--security-opt label=disable
Below is a table that describes these settings in more detail:
Setting | Description | Default Value |
|---|---|---|
set_uid_gid | Sets the -u flag | -u $UID:$(id -g) |
set_passwd_entry | Sets the --passwd-entry flag | --passwd-entry=ops:*:$UID:$(id -g)::/home/ops:/bin/bash |
cmd_base | Contains a list of flag options that can optionally be additionally modified based on desired need |
|
environment | Contains a list of environment variables to be persisted onto the container. Assumes that the environment exists on the host machine. |
|
At the time of this writing, projects can just use the default settings and modify over time depending on circumstances.
Update PGE Wrapper Code
Note that this section may be project-specific, but documenting it here because this is what SWOT and NISAR have currently implemented in the adaptation layer when running PGE software.
The PGE Wrapper code assists the PCM Sciflo jobs in running the PGE software. It does so by calling a podman run command on the PGE container.
To use Podman, the following should be added to the PGE wrapper code:
engine_value = app.conf.get("CONTAINER_ENGINE", "docker")
container_engine = container_engine_factory(engine=engine_value)
if isinstance(container_engine, Podman):
container_engine.set_passwd_entry(False)
cmd = container_engine.get_container_cmd(docker_params[dep_img_name], [rc_file])
What this code snippet does is that when running a PGE container through podman, it is found that we do not need to run it with the --passwd-entry flag option. Therefore what this code snippet does is:
Checks that the celeryconfig has CONTAINER_ENGINE=podman
If true, then do not set the --passwd-entry flag option when building out the podman run container for the PGE
Customize Podman Start Up Script
For projects that need to do additional mounting of files/directories when running their container through podman, you’ll need to make sure that the podman start up script is updated accordingly. Some use-cases, for example, would be mounting the settings.yaml and adaptation code just like what is done on swot/nisar to do post-processing at the Verdi level (example: streaming job logs to AWS CloudWatch). Continue down this section if this applies to you as well.
Copy the following podman start up script to your conf/sds/files directory in your repository:
https://github.com/hysds/hysds-dockerfiles/blob/develop/verdi/run_verdi_podman.sh.tmpl
At this point, you can update this script to add the additional mounts if needed. Here’s an example:
Next, ensure that this modified version of the podman script is pushed out to your cluster. Nominally, this is done through the cluster.py tool. As an example, swot added the following piece of code in its update_swot_packages cluster function:
You want to make sure that however it is that you push this out to the cluster, that ordering is very important. Specifically, this update should occur prior to running sds ship.