baremetal: Ironic bootstrap services should run as a staticpod

[hardys]

It has been noted in #2249 that the ironic services currently enabled via a systemd unit and start/stop scripts should probably be migrated to a staticpod definition.

Raising this to keep track of this, and I'll investigate then follow up with a PR to address this.

[hardys]

/label platform/baremetal

[hardys]

/assign

[derekhiggins]

This came about when we had problems with ironic containers openshift-metal3/dev-scripts#753

digging into it a little the podman containers are getting deleted by the crio service

$  systemctl stop crio                                           
$  podman inspect 3c182da7d9e1 | grep Running
            "Running": true,
$  ls /var/lib/containers/storage/overlay-containers/3c182da7d9e149dc75c6f1aa58ae3669659623162acaac98d165e341de8f22de/userdata/config.json
/var/lib/containers/storage/overlay-containers/3c182da7d9e149dc75c6f1aa58ae3669659623162acaac98d165e341de8f22de/userdata/config.json
$  /usr/bin/crio --enable-metrics=true --metrics-port=9537 --log-level debug |& grep "could not restore con"
time="2019-08-22 10:48:06.618623731Z" level=warning msg="could not restore container 3c182da7d9e149dc75c6f1aa58ae3669659623162acaac98d165e341de8f22de: unexpected end of JSON input"
^C
$  podman inspect 3c182da7d9e1 | grep Running
Error: error getting libpod container inspect data 3c182da7d9e149dc75c6f1aa58ae3669659623162acaac98d165e341de8f22de: error getting container from store "3c182da7d9e149dc75c6f1aa58ae3669659623162acaac98d165e341de8f22de": container not known
$  ls /var/lib/containers/storage/overlay-containers/3c182da7d9e149dc75c6f1aa58ae3669659623162acaac98d165e341de8f22de/userdata/config.json
ls: cannot access '/var/lib/containers/storage/overlay-containers/3c182da7d9e149dc75c6f1aa58ae3669659623162acaac98d165e341de8f22de/userdata/config.json': No such file or directory

crio attempts to load the containers started by podman and fails, it then deletes them
https://github.com/cri-o/cri-o/blob/278995c/server/server.go#L208

crio is getting restarted by a failing bootkube.sh (failure to contact the DNS server)

installer/data/data/bootstrap/files/usr/local/bin/bootkube.sh.template

Line 62 in 35d1169

systemctl restart cri-o.service

Aug 22 10:43:42 localhost bootkube.sh[1624]: Error: error pulling image "registry.svc.ci.openshift.org/ocp/release:4.2": unable to pull registry.svc.ci.openshift.org/ocp/release:4.2: unable to pull image: Error initializing source docker://registry.svc.ci.openshift.org/ocp/release:4.2: pinging docker registry returned: Get https://registry.svc.ci.openshift.org/v2/: dial tcp: lookup registry.svc.ci.openshift.org on [::1]:53: read udp [::1]:38894->[::1]:53: read: connection refused

[rhatdan]

@mrunalp @umohnani8 @haircommander Probably should not do that...

[haircommander]

@derekhiggins, the line you tagged in server.go will not effect podman containers (they've been filtered out by that point). However, the systemctl restart cri-o.service call will crio-wipe.service and wipe the whole storage (podman and cri-o). I have a fix in flight, such that crio-wipe will only clear the storage used by cri-o.

[derekhiggins]

@derekhiggins, the line you tagged in server.go will not effect podman containers (they've been filtered out by that point).

The crio command I've pasted above (with --log-level debug) shows that the podman container hasn't been filtered out, see
could not restore container 3c182da7d9e149dc75c6f1aa58ae3669659623162acaac98d165e341de8f22de

However, the systemctl restart cri-o.service call will crio-wipe.service and wipe the whole storage (podman and cri-o). I have a fix in flight, such that crio-wipe will only clear the storage used by cri-o.

I've run this while watching crio-wipe, the service is restarted when crio is restarted but I've only observed it effecting the storage the first time its run,

[haircommander]

crio-wipe will only run the first time. It expects a version file at /usr/libexec/crio/version, and will clear storage if that version is out of date (or doesn't exist). when cri-o starts up, it lays down a version file, so further restarts won't clear.

[derekhiggins]

crio-wipe will only run the first time. It expects a version file at /usr/libexec/crio/version, and will clear storage if that version is out of date (or doesn't exist). when cri-o starts up, it lays down a version file, so further restarts won't clear.

So just to clarify, your agreeing with me? Its not crio-wipe that is causing this problem, cri-o is what is removing the podman containers?

[haircommander]

I wasn't agreeing with you earlier, but I've since thought about it more and it does seem like cri-o is deleting podman containers. I'll look into it, what container are you using?

[haircommander]

and what versions of cri-o and podman?

[derekhiggins]

podman-1.4.2-1.module+el8.1.0+3423+f0eda5e0.x86_64
cri-o-1.14.10-0.3.dev.rhaos4.2.gitb5c74ef.el8.x86_64

Thanks, the containers where we noticed this are started by startironic.sh (we intend to move these to a static pod in future)
https://github.com/openshift/installer/blob/master/data/data/bootstrap/baremetal/files/usr/local/bin/startironic.sh.template