[CVE-2019-5736]: Server uses more memory if start many runc process at one time

[lifubang]

Different from #1980 , If we start 100 runc processes at one time, the server will use about more 900M memory than before, it may cause failure. I don't know whether this is a problem or not?

root@iZ2ze1o61blvco5p5ducnnZ:/opt/busybox# cat runc100.sh 
#!/bin/bash
for i in {1..100}
do
	name="test$i"
	runc create $name &
done
echo "100 runc processes started."

Before fix:

root@iZ2ze1o61blvco5p5ducnnZ:/opt/busybox# free -m
              total        used        free      shared  buff/cache   available
Mem:           7983        3896        2256          39        1830        3618
Swap:             0           0           0
root@iZ2ze1o61blvco5p5ducnnZ:/opt/busybox# ./runc100.sh 
100 runc processes started.
root@iZ2ze1o61blvco5p5ducnnZ:/opt/busybox# free -m
              total        used        free      shared  buff/cache   available
Mem:           7983        4936        1178          40        1869        2545
Swap:             0           0           0

After fix:

root@iZ2ze1o61blvco5p5ducnnZ:/opt/busybox# free -m
              total        used        free      shared  buff/cache   available
Mem:           7983        3896        1980          39        2107        3621
Swap:             0           0           0
root@iZ2ze1o61blvco5p5ducnnZ:/opt/busybox# ./runc100.sh 
100 runc processes started.
root@iZ2ze1o61blvco5p5ducnnZ:/opt/busybox# free -m
              total        used        free      shared  buff/cache   available
Mem:           7983        4803         352        1107        2827        1611
Swap:             0           0           0

[cyphar]

It's because you're just doing runc create -- which results in a created container but not a started container. A created container has pid1 almost entirely setup and is just waiting for the command to actually start (which is done with runc start) and is blocking within runc init.

Once a container is started there is no memory overhead from the copying because the runc process is no longer around (if you're not using detached mode there is still a runc process around but it's not the copied one).

[lifubang]

Because runc start and runc create is called separately, so in a big server, there will be 100 runc create at a time before runc start? I don't know whether there is a lock when call runc create in docker run or kubernetes.

[cyphar]

Yeah, I just looked and containerd uses separate create and start operations. The problem is that even with a tmpfs fallback you would still use that memory -- you can try to write to a filesystem (which is what my current #1984 does) but then you're using storage space. The only solution is to have a smaller binary and that would require an enormous amount of work (effectively a full rewrite of runc init).

EDIT: Sorry, there is another solution which is to create a temporary overlayfs mount. I might actually just implement this because it will solve other problems too (and we can use memfd_create as a separate fallback with a tmpfs as the final fallback).

[cyphar]

#1984 also fixed this.