Kata 2.0.0 doesn't play well with Docker

[kholia]

The combination of Kata 1.12.0-rc0 with Docker 19.03.13 on Ubuntu 20.10 works well.

Today, I removed this old Kata + Docker setup to try out Kata Containers 2.0.0 on the same Ubuntu 20.10.

I used the following documentation to setup Kata Containers 2.0.0:

https://github.com/kata-containers/packaging/blob/master/snap/README.md#configure-kata-containers

$ docker --version
Docker version 19.03.13, build 4484c46

$ cat /etc/docker/daemon.json
{
  "default-runtime": "runc",
  "runtimes": {
    "kata-fc": {
      "path": "/snap/kata-containers/current/usr/bin/kata-runtime"
    }
  },
  "storage-driver": "devicemapper"
}

Docker says that kata-fc is available.

$ docker info --format '{{json .Runtimes}}'                           
{"kata-fc":{"path":"/snap/kata-containers/current/usr/bin/kata-runtime"},"runc":{"path":"runc"}}

However, the problems start when trying to run a container.

$ sudo docker run --rm -it --runtime=kata-fc --name=oh-sweet alpine sh
docker: Error response from daemon: OCI runtime create failed: Invalid command "create": unknown.

Problem: The container doesn't launch.

The containers launch just fine if I omit the --runtime=kata-fc parameter. So Docker + runC are working fine as usual.

Debugging this a bit using sudo execsnoop-perf:

504560 504007 docker run --rm -it --runtime=kata-fc --name=oh-sweet alpine sh
504576 504575 /usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/dm-4
504582 504575 /sbin/dmsetup udevflags 4229307
504584 504575 /sbin/dmsetup udevcomplete 4229307
504594   1129 xfs_info /mnt/data/docker/devicemapper/mnt/ec5a3d87ea554752254b2bff0a3cac8c7557615dddc9dfd953a5c89cd943bbc2-init
504597 504596 losetup -O NAME -j /mnt/data/docker/devicemapper/mnt/ec5a3d87ea554752254b2bff0a3cac8c7557615dddc9dfd953a5c89cd943bbc2-init
504598 504596 tail -n 1
504599 504594 findmnt -t xfs -f -n -o TARGET /mnt/data/docker/devicemapper/mnt/ec5a3d87ea554752254b2bff0a3cac8c7557615dddc9dfd953a5c89cd943bbc2-init
504600 504594 xfs_db -p xfs_info -c info /mnt/data/docker/devicemapper/mnt/ec5a3d87ea554752254b2bff0a3cac8c7557615dddc9dfd953a5c89cd943bbc2-init
504601 504575 /sbin/dmsetup udevcomplete 6333861
504603 504574 /usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/dm-4
504609 504575 /sbin/dmsetup udevflags 4208471
504610 504575 /sbin/dmsetup udevcomplete 4208471
504618 504575 /sbin/dmsetup udevcomplete 6320314
504620 504574 /usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/dm-4
504626 504575 /sbin/dmsetup udevflags 4215309
504627 504575 /sbin/dmsetup udevcomplete 4215309
504635 504574 /lib/udev/bridge-network-interface
504636 504574 /lib/udev/ifupdown-hotplug
504638 504637 /sbin/ifquery --allow hotplug -l veth1e15df3
504640 504575 /lib/udev/bridge-network-interface
504641 504639 /sbin/ifquery --allow auto -l veth1e15df3
504642 504575 /lib/udev/ifupdown-hotplug
504643 504574 /lib/systemd/systemd-sysctl --prefix=/net/ipv4/conf/veth1e15df3 --prefix=/net/ipv4/neigh/veth1e15df3 --prefix=/net/ipv6/conf/veth1e15df3 --prefix=/net/ipv6/neigh/veth1e15df3
504645 504644 /sbin/ifquery --allow hotplug -l veth64386b5
504647 504646 /sbin/ifquery --allow auto -l veth64386b5
504648 504575 /lib/systemd/systemd-sysctl --prefix=/net/ipv4/conf/veth64386b5 --prefix=/net/ipv4/neigh/veth64386b5 --prefix=/net/ipv6/conf/veth64386b5 --prefix=/net/ipv6/neigh/veth64386b5
504650   1129 xfs_info /mnt/data/docker/devicemapper/mnt/ec5a3d87ea554752254b2bff0a3cac8c7557615dddc9dfd953a5c89cd943bbc2
504653 504652 losetup -O NAME -j /mnt/data/docker/devicemapper/mnt/ec5a3d87ea554752254b2bff0a3cac8c7557615dddc9dfd953a5c89cd943bbc2
504654 504652 tail -n 1
504655 504650 findmnt -t xfs -f -n -o TARGET /mnt/data/docker/devicemapper/mnt/ec5a3d87ea554752254b2bff0a3cac8c7557615dddc9dfd953a5c89cd943bbc2
504656 504650 xfs_spaceman -p xfs_info -c info /mnt/data/docker/devicemapper/mnt/ec5a3d87ea554752254b2bff0a3cac8c7557615dddc9dfd953a5c89cd943bbc2
504657 483768 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/6e83624d705610d0acbc86d46b73a91feb560015307b81bc1f1006a25b26046b -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd [...]
504665 504663 /snap/kata-containers/current/usr/bin/kata-runtime --root /var/run/docker/runtime-kata-fc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/6e83624d705610d0acbc86d46b73a91feb560015307b81bc1f1006a25b26046b/log.json --log-format json create --bundle [...]
504674 504575 /lib/udev/ifupdown-hotplug
504675 504575 /lib/udev/ifupdown-hotplug

Last two lines (full version):

509498 483541 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/ca9b46c7f2a0cec81d00ae9e20bce2a5f411090ade19baaa0a668b4a79fa48fc -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-kata-fc
509507 509502 /snap/kata-containers/current/usr/bin/kata-runtime --root /var/run/docker/runtime-kata-fc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/ca9b46c7f2a0cec81d00ae9e20bce2a5f411090ade19baaa0a668b4a79fa48fc/log.json --log-format json create --bundle /run/containerd/io.containerd.runtime.v1.linux/moby/ca9b46c7f2a0cec81d00ae9e20bce2a5f411090ade19baaa0a668b4a79fa48fc --pid-file /run/containerd/io.containerd.runtime.v1.linux/moby/ca9b46c7f2a0cec81d00ae9e20bce2a5f411090ade19baaa0a668b4a79fa48fc/init.pid --console-socket /tmp/pty431286728/pty.sock ca9b46c7f2a0cec81d00ae9e20bce2a5f411090ade19baaa0a668b4a79fa48fc

I will be around to help with debugging this. Thank you!

[caoruidong]

Kata 2.0 only has no CLI kata-containers/kata-containers#332 , so it cannot be used with docker now.

[kholia]

https://github.com/kata-containers/packaging/blob/master/snap/README.md#integration-with-docker-and-kubernetes

So will we need to change this doc to say that Kata 2.0 is no longer compatible with Docker?

Many folks are still running Docker, and Kata was a nice way to seamlessly play with Firecracker via kata-fc. Is the plan to no longer support this?

[kholia]

After setting up new kata-fc:

$ sudo tar -xvf kata-static-2.0.0-x86_64.tar.xz -C / 

$ cat /etc/docker/daemon.json
{
  "default-runtime": "runc",
  "runtimes": {
    "kata-fc": {
      "path": "/opt/kata/bin/kata-fc"
    }
  },
  "storage-driver": "devicemapper"
}

$ sudo systemctl restart docker

VMs fail to launch:

$ docker run --rm -it --runtime=kata-fc --name=oh-sweet alpine sh
docker: Error response from daemon: OCI runtime create failed: Near line 27 (last key parsed 'hypervisor.firecracker.valid_hypervisor_paths'): expected value but found '\n' instead: unknown.

After disabling the valid_hypervisor_paths in /opt/kata/share/defaults/kata-containers/configuration-fc.toml file.

$ docker run --rm -it --runtime=kata-fc --name=oh-sweet alpine sh
docker: Error response from daemon: OCI runtime create failed: Invalid command "create": unknown.

Yep - it does seem that support for Docker was removed.

Edit: Hoping for kata-containers/kata-containers#722 to come through.

[jodh-intel]

Hi @kholia - Thanks for raising. However, unfortunately since Docker doesn't yet support the containerd shimv2 architecture directly, Kata 2.0.0 won't work with Docker.

The Kata 1.x runtime (in this repo) does work with docker and as such we provide Docker-based install guides:

• https://github.com/kata-containers/documentation/tree/master/install/docker

However, the newer Kata 2.x runtime (which lives in https://github.com/kata-containers/kata-containers/tree/2.0-dev/src/runtime) is "shim v2 only". Due to this, we provide containerd-based install guides for Kata 2.x:

• https://github.com/kata-containers/kata-containers/blob/2.0-dev/utils/README.md

For a comparison of the two architectures, see:

• Kata 1.x architecture document.
• Kata 2.x architecture document.

[jodh-intel]

Closing as invalid since docker doesn't yet support the shimv2 architecture.