socket sharing between containers denied by selinux when using "--systemd=always"

[ah83]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

A container named "server" runs for example a memcached which is listening on a unix domain socket.
The socket is stored in a bind mounted directory (-v /var/local/container-volumes/data:/data:z).
A second container named "client" starts with exactly same bind mount to access the memcached socket as a client.

When the client container starts with the option "--systemd=always" and then tries to access the socket
/data/sockets/memcached.socket, the access will be denied by SELinux.
Without "--systemd=always" everything works as expected.

Steps to reproduce the issue:

1. Run server container

mkdir -p /var/local/container-volumes/data/sockets
chmod 700 /var/local/container-volumes
chmod 700 /var/local/container-volumes/data/sockets
podman run --name server -it --rm -v /var/local/container-volumes/data:/data:z centos:8 /bin/bash
yum install memcached
memcached -u root -s /data/sockets/memcached.socket

2. Run client container with "--systemd always" and try to connect to socket

podman run --name client -it --rm --systemd always -v /var/local/container-volumes/data:/data:z centos:8 /bin/bash
yum install socat
socat -v UNIX-CONNECT:/data/sockets/memcached.socket STDIN
2020/11/02 10:26:46 socat[56] E connect(5, AF=1 "/data/sockets/memcached.socket", 32): Permission denied

3. Run client container without "--systemd always" and try to connect to socket

podman run --name client -it --rm  -v /var/local/container-volumes/data:/data:z centos:8 /bin/bash
yum install socat
socat -v UNIX-CONNECT:/data/sockets/memcached.socket STDIN

Works as expected!

Describe the results you received:

SELinux error msg on the container host

type=AVC msg=audit(1604312588.452:67274): avc:  denied  { connectto } for  pid=148723 comm="socat" path="/data/sockets/memcached.socket" scontext=system_u:system_r:container_init_t:s0:c603,c1008 tcontext=system_u:system_r:container_t:s0:c175,c654 tclass=unix_stream_socket permissive=-1

socat error msg

2020/11/02 10:26:46 socat[56] E connect(5, AF=1 "/data/sockets/memcached.socket", 32): Permission denied

Describe the results you expected:

Access to the unix socket.

Additional information you deem important (e.g. issue happens only occasionally):

memcached is only used to show the issue and reproduce it.
In my production setup i share the sockets of a sssd container to provide nss and pam services for 200 containers per host.
There is no viable alternative to sharing the socket this way and the "client" container must use systemd,
because there are running multiple services inside the container.
This setup worked in previous versions of podman.

Output of podman version:

Version:      2.1.1
API Version:  2.0.0
Go Version:   go1.13.15
Built:        Wed Oct 28 13:38:47 2020
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.16.1
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.21-1.el8.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: fa5f92225c4c95759d10846106c1ebd325966f91-dirty'
  cpus: 8
  distribution:
    distribution: '"centos"'
    version: "8"
  eventLogger: journald
  hostname: xxxx
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.18.0-193.19.1.el8_2.x86_64
  linkmode: dynamic
  memFree: 9583423488
  memTotal: 16643891200
  ociRuntime:
    name: runc
    package: runc-1.0.0-145.rc91.git24a3cf8.el8.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.2-dev'
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  rootless: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 2147479552
  swapTotal: 2147479552
  uptime: 74h 7m 51.92s (Approximately 3.08 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 30
    paused: 0
    running: 8
    stopped: 22
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /data/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 1796
  runRoot: /var/run/containers/storage
  volumePath: /data/containers/storage/volumes
version:
  APIVersion: 2.0.0
  Built: 1603888727
  BuiltTime: Wed Oct 28 13:38:47 2020
  GitCommit: ""
  GoVersion: go1.13.15
  OsArch: linux/amd64
  Version: 2.1.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman-2.1.1-10.el8.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

[ah83]

The socket access works if both containers are startet with "--systemd always".
In my setup sssd container does not run systemd but just sssd.

[mheon]

This smells like SELinux. Can you check the journal for any AVCs?

…

On Mon, Nov 2, 2020, 08:39 ah83 ***@***.***> wrote: The socket access works if both containers are startet with "--systemd always". In my setup sssd container does not run systemd but just sssd. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#8216 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB3AOCGLXXLIDMCRRVBUV43SN2Y7RANCNFSM4THOBLFQ> .

[ah83]

this was in the audit.log of the container host:

type=AVC msg=audit(1604312588.452:67274): avc: denied { connectto } for pid=148723 comm="socat" path="/data/sockets/memcached.socket" scontext=system_u:system_r:container_init_t:s0:c603,c1008 tcontext=system_u:system_r:container_t:s0:c175,c654 tclass=unix_stream_socket permissive=-1

[mheon]

@rhatdan PTAL

[rhatdan]

Looks like you are attempting to have to containers talk to each other via a unix domain socket. This is naturally blocked by SELinux. Easiest thing to do would be have both container share IPC Namespace, since is really what you want both containers talking between inter processes. Podman automatically sets the SELinux labels the same when containers share the IPC namespace.

Other options would be to hard code the SELinux labels for both containers to match. Or disable SELinux labels support.

[ah83]

Hi Dan!

First, thank you for your help and advice!

My setup consist of about 200 webhosting containers (running systemd inside) per container host and one sssd container (no systemd).
These 200 containers should be able to access the sockets under /var/lib/sss/pipes and /var/lib/sss/mc to use the pam and nss services. Sharing the ipc namespace between all these containers used by different customers ist not really my goal.
Also using the same static selinux labels is not useful for this setup, because i try to isolate these container from each other.

Shouldn't be the labels identical just for these specific mounts when using "z" or does it not apply in the context of unix domains sockets?

-v /data/srv/sssd/volumes/sss/pipes:/var/lib/sss/pipes:z
-v /data/srv/sssd/volumes/sss/mc:/var/lib/sss/mc:ro,z

This setup worked before and it is still working, just not if one container uses systemd and the other container not.
As a workaround i started the sssd container with systemd=always, but this changes the stop signal of the container.

In my opinion the sharing of the sssd sockets between different containers running systemd or not is a quite important feature - maybe also for enterprise setups.

Thanks,

Alex

[rhatdan]

it is not that simple. You need to turn off confinement from at least the containers running sssd for you, and then see what AVCs are produced.

If you do this, the sssd container that needs to communicate with all of the containers will be running as spc_t:s0, and the containers should be allowed to connectto that process.

[rhatdan]

podman run --security-opt label=disabled SSDIMAGE ...

[ah83]

yes this works for my setup - thank you for your help!

system_u:system_r:spc_t:s0      root      218814  0.0  0.0 221024 13936 pts/0    Ss+  11:35   0:00 /usr/sbin/sssd

and the web containers can access the sssd sockets. there are also no avc msgs in the audit log.

One last question:
Currently i can share sockets between containers with the volume option "z", when all containers run systemd or none of them.
Will this behavior stay the same in the future or are planning to disallow the socket sharing generally?

Thanks,
Alex

[rhatdan]

On 11/4/20 07:39, ah83 wrote: yes this works for my setup - thank you for your help! |system_u:system_r:spc_t:s0 root 218814 0.0 0.0 221024 13936 pts/0 Ss+ 11:35 0:00 /usr/sbin/sssd | and the web containers can access the sssd sockets. there are also no avc msgs in the audit log. One last question: Currently i can share sockets between containers with the volume option "z", when all containers run systemd or none of them. Will this behavior stay the same in the future or are planning to disallow the socket sharing generally? Thanks, Alex — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8216 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAPIPQ5WWYXFZCE65KUPRQTSOFDRZANCNFSM4THOBLFQ>.

You can not share any sockets between container unless one of two things is happening.  The containers match the SELinux MCS Levels. IE IPC is shared causing both containers to run with the same SELinux labels. Or one of the containers is not running with SELInux confinement, which means that it is running as spc_t, and all other containers can communicate over the socket.

[ah83]

ok, thank you for your clear explanation!

[daiaji]

Looks like you are attempting to have to containers talk to each other via a unix domain socket. This is naturally blocked by SELinux. Easiest thing to do would be have both container share IPC Namespace, since is really what you want both containers talking between inter processes. Podman automatically sets the SELinux labels the same when containers share the IPC namespace.

Other options would be to hard code the SELinux labels for both containers to match. Or disable SELinux labels support.

If I want two containers in a pod to share a Unix socket, are there any other operations required? Or can I directly use the Unix socket of one container in another?

[rhatdan]

Two containers within a Pod should have the same SELinux label, so SELinux will not get in the way of the containers talking over a unix domain socket.