integration with podman-hpc

[shahzebsiddiqui]

Is your feature request related to a problem? Please describe.
We want to integrate shpc with podman-hpc which is a wrapper to podman with additional commands that make it suitable for us to run containers properly.

Describe the solution you'd like
Some changes to shpc would be new template for podman-hpc under https://github.com/singularityhub/singularity-hpc/tree/main/shpc/main/wrappers/templates

Describe alternatives you've considered
Currently we have tried using podman as the container tech and would like to have a new container tech so we can do shpc config set podman-hpc

Note i am creating this issue and will try to add this feature in near future.

For more details on podman-hpc see https://docs.nersc.gov/development/podman-hpc/overview/

[vsoch]

@shahzebsiddiqui we already support Podman without an additional Python dependency. Could you be specific about what features or integrations you are looking for?

[shahzebsiddiqui]

I am not entirely sure but talking with @scanon I think we want to have support with podman-hpc as container-tech in the configuration file since I realize the template scripts depend on the container. I think we had issue running podman at NERSC due to uid/gid mapping and I saw the template is hard-coded with the podman options I think we wanted to try using --userns keep-id in order to run container as user as pose to root.

[vsoch]

We have an environment variable that will add those options to any podman executions, PODMAN_OPTS

And see in the code.
I'm still not understanding why a custom wrapper would be needed, but naybe @scanon can better explain?

[marcodelapierre]

Hi @vsoch , I can probably comment as I was talking about this with Shane recently.

Podman-HPC is a wrapper that NERSC developed to render Podman viable for HPC users (MPI and GPU hooks, and other useful tunings). I would consider adding direct support within SHPC, as there are good chances it is going to become a new standard for containers in HPC.

@shahzebsiddiqui other than the executable name and the option --userns keep-id, are there other notable differences in CLI between Podman-HPC and Podman?

[vsoch]

If it's just adding flags / otherwise still wrapping podman, my preference is still to not add it as another dependency. If/when it becomes "a new standard for containers in HPC" I would definitely consider adding it, but we're not there yet (and it likely would become an extra dependency that wasn't really needed in the first place).

What we likely want to do is what you suggested - get the flags or flag sets that are desired, and then we should ensure it's easy to either have a site, container, or user define (and use) a flag set.

[marcodelapierre]

Sure!

Shall we have a settings in the SHPC yaml, to allow defining the OPTS variable at the site level?
This would then be defined within each module as generated by shpc install.

What do you think?

[vsoch]

Would it not already work with the environment variable? I’m not great with module software so let me know what you think is best - I generally really like this idea!

But I’m off to bed now - feel free to spec out a design for me to work on - will be back after some sleep!

[marcodelapierre]

Left Helsinki...Greetings from Italy!

What I meant is the following. As we know that all modules will need specific PODMAN_OPTS flags, should we have a way to provide them via the shpc settings, and/or the view settings? E.g. it could be a way to provide a default value for PODMAN_OPTS (or other OPTS variables), as in (bash syntax just to give the idea):

PODMAN_OPTS=${PODMAN_OPTS:-DEFAULT_OPTS}

and then the DEFAULT_OPTS are controlled either via settings.yml or the view file.

Have I explained myself better?

[vsoch]

The view settings would work - my worry about adding to global settings proper is that then it would be globally applied without an easy way to take them off. But for specific containers it would make sense (this is already supported) or views (not supported but could be!)

[vsoch]

And that’s also the reason it’s currently specified as an environment variable - they can be one off defined by a user or for a use case without global enforcement. And why they are supported “singularity_opts” or “docker_opts” for a container because we assume some containers have very specific requirements.

[marcodelapierre]

make sense for me, thanks for clarifying Vanessa.

@shahzebsiddiqui so essentially, right now you could use SHPC out of the box with Podman HPC with just a couple of caveats:

1. SHPC looks for a podman executable, not podman-hpc. Can you define a symlink?
2. You can specify any extra flags using PODMAN_OPTS (see SHPC docs); what you could do is have a podman-hpc module that defines them ; you can then tell SHPC to always load that module (setting podman_module I think)

Would this work for you?