exec: failed to write to cgroup.procs

[chavafg]

Description of problem

There have been some random failures with the hot plug CPUs and cri-o tests. Digging a little bit more into the failures, both are failing when running an exec command as follows:

cri-o failure:

05:34:59 # time="2019-03-11 11:34:57.715914943Z" level=debug msg="Received container exit code: -1, message: rpc error: code = Internal desc = Could not run process: container_linux.go:348: starting container process caused "process_linux.go:90: adding pid 169 to cgroups caused \"failed to write 169 to cgroup.procs: write /sys/fs/cgroup/cpu,cpuacct/Burstable/pod_123-456/crio-1e4648d38f30013bbdba2249f0eb8b7c043ef40d904fb5276c9ac71b4ba054f9/cgroup.procs: invalid argument\""
05:34:59 # " 
05:34:59 # time="2019-03-11T11:34:57Z" level=fatal msg="execing command in container synchronously failed: rpc error: code = Unknown desc = command error: rpc error: code = Internal desc = Could not run process: container_linux.go:348: starting container process caused "process_linux.go:90: adding pid 169 to cgroups caused \"failed to write 169 to cgroup.procs: write /sys/fs/cgroup/cpu,cpuacct/Burstable/pod_123-456/crio-1e4648d38f30013bbdba2249f0eb8b7c043ef40d904fb5276c9ac71b4ba054f9/cgroup.procs: invalid argument\""
05:34:59 # , stdout: , stderr: , exit code -1" 

docker failure:

Running command '/usr/bin/docker [docker exec TRpJ2dq55fITz0QLTuH49SPporH9nE cat /sys/fs/cgroup/cpuset/cpuset.cpus]'
command failed error 'exit status 126'
[docker exec TRpJ2dq55fITz0QLTuH49SPporH9nE cat /sys/fs/cgroup/cpuset/cpuset.cpus]
Timeout: 60 seconds
Exit Code: 126
Stdout: OCI runtime exec failed: rpc error: code = Internal desc = Could not run process: container_linux.go:348: starting container process caused "process_linux.go:90: adding pid 128 to cgroups caused \"failed to write 128 to cgroup.procs: write /sys/fs/cgroup/cpu,cpuacct/docker/c81e621e3f3c96e7c1c05af8b814f9c507fbbe4fb017d50c8a3fa3edf2499b73/cgroup.procs: invalid argument\"": unknown

Some jobs affected by this issue:
http://jenkins.katacontainers.io/job/kata-containers-tests-fedora-PR/1236/consoleText
http://jenkins.katacontainers.io/job/kata-containers-tests-ubuntu-18-04-PR/126/consoleText
http://jenkins.katacontainers.io/job/kata-containers-tests-ubuntu-nemu/79/consoleText

[devimc]

seems like this is a known issue that was already fixed opencontainers/runc#1916

[jodh-intel]

Does someone want to pick up re-vendoring runc with this issue?

[chavafg]

I can do it, but to what revision should I revendor?
ccb5efd37fb7c86364786e9137e22948751de7ed is v1.0.0-rc6, which is latest release and has the fix in it.
or do we want a newer revision?

[devimc]

@chavafg use f414f497b50a61750ea3af9fccf998a3db687cea that is one before the cve

[sboeuf]

@devimc could you investigate this issue as it seems to cause a bunch of instabilities in our CI. Also, from what I gathered from @chavafg feedback, #477 might not be enough.
@chavafg @devimc please let us know what is the status on this issue as we want to fix it asap to get a more reliable CI, as we're delaying PRs right now because the CI never gets green on the first pass.

[sboeuf]

@devimc @chavafg I've opened #490 to fix the revendoring of libcontainer. PTAL