aarch64: crun checkpoint test failures

[lsm5]

A couple of podman e2e checkpoint tests are failing only on aarch64 envs (from fedora rawhide to centos stream 9), like so:

Podman checkpoint [It] podman checkpoint container with --pre-checkpoint
/var/tmp/tmp.utA7CvAqkm/podman-5.6.0-dev/test/e2e/checkpoint_test.go:979

  Timeline >>
  Running: /usr/bin/podman --storage-opt overlay.imagestore=/var/tmp/podman-e2e-1117732790/imagecachedir --root /var/tmp/podman-e2e-1117732790/subtest-3590578882/p/root --runroot /var/tmp/podman-e2e-1117732790/subtest-3590578882/p/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/containers/networks --network-backend netavark --cgroup-manager systemd --tmpdir /var/tmp/podman-e2e-1117732790/subtest-3590578882/p --events-backend file --db-backend sqlite --storage-driver overlay network create
  podman1
  Running: /usr/bin/podman --storage-opt overlay.imagestore=/var/tmp/podman-e2e-1117732790/imagecachedir --root /var/tmp/podman-e2e-1117732790/subtest-3590578882/p/root --runroot /var/tmp/podman-e2e-1117732790/subtest-3590578882/p/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/containers/networks --network-backend netavark --cgroup-manager systemd --tmpdir /var/tmp/podman-e2e-1117732790/subtest-3590578882/p --events-backend file --db-backend sqlite --storage-driver overlay run -d --network podman1 quay.io/libpod/alpine:latest top
  7787be3cfe4879d510c110aa1616a28746e44998a36d887664ff8fa29c5e2abe
  Running: /usr/bin/podman --storage-opt overlay.imagestore=/var/tmp/podman-e2e-1117732790/imagecachedir --root /var/tmp/podman-e2e-1117732790/subtest-3590578882/p/root --runroot /var/tmp/podman-e2e-1117732790/subtest-3590578882/p/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/containers/networks --network-backend netavark --cgroup-manager systemd --tmpdir /var/tmp/podman-e2e-1117732790/subtest-3590578882/p --events-backend file --db-backend sqlite --storage-driver overlay container checkpoint -P 7787be3cfe4879d510c110aa1616a28746e44998a36d887664ff8fa29c5e2abe
  memory tracking not supported. Please check CRIU logfile /var/tmp/podman-e2e-1117732790/subtest-3590578882/p/root/overlay-containers/7787be3cfe4879d510c110aa1616a28746e44998a36d887664ff8fa29c5e2abe/userdata/dump.log
  Error: `/usr/bin/crun checkpoint --image-path /var/tmp/podman-e2e-1117732790/subtest-3590578882/p/root/overlay-containers/7787be3cfe4879d510c110aa1616a28746e44998a36d887664ff8fa29c5e2abe/userdata/pre-checkpoint --work-path /var/tmp/podman-e2e-1117732790/subtest-3590578882/p/root/overlay-containers/7787be3cfe4879d510c110aa1616a28746e44998a36d887664ff8fa29c5e2abe/userdata --pre-dump 7787be3cfe4879d510c110aa1616a28746e44998a36d887664ff8fa29c5e2abe` failed: exit status 1
  [FAILED] in [It] - /var/tmp/tmp.utA7CvAqkm/podman-5.6.0-dev/test/e2e/checkpoint_test.go:993 @ 05/28/25 15:37:29.763
  Running: /usr/bin/podman --storage-opt overlay.imagestore=/var/tmp/podman-e2e-1117732790/imagecachedir --root /var/tmp/podman-e2e-1117732790/subtest-3590578882/p/root --runroot /var/tmp/podman-e2e-1117732790/subtest-3590578882/p/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/containers/networks --network-backend netavark --cgroup-manager systemd --tmpdir /var/tmp/podman-e2e-1117732790/subtest-3590578882/p --events-backend file --db-backend sqlite --storage-driver overlay network rm -f podman1
  podman1
  Running: /usr/bin/podman --storage-opt overlay.imagestore=/var/tmp/podman-e2e-1117732790/imagecachedir --root /var/tmp/podman-e2e-1117732790/subtest-3590578882/p/root --runroot /var/tmp/podman-e2e-1117732790/subtest-3590578882/p/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/containers/networks --network-backend netavark --cgroup-manager systemd --tmpdir /var/tmp/podman-e2e-1117732790/subtest-3590578882/p --events-backend file --db-backend sqlite --storage-driver overlay stop --all -t 0
  Running: /usr/bin/podman --storage-opt overlay.imagestore=/var/tmp/podman-e2e-1117732790/imagecachedir --root /var/tmp/podman-e2e-1117732790/subtest-3590578882/p/root --runroot /var/tmp/podman-e2e-1117732790/subtest-3590578882/p/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/containers/networks --network-backend netavark --cgroup-manager systemd --tmpdir /var/tmp/podman-e2e-1117732790/subtest-3590578882/p --events-backend file --db-backend sqlite --storage-driver overlay pod rm -fa -t 0
  Running: /usr/bin/podman --storage-opt overlay.imagestore=/var/tmp/podman-e2e-1117732790/imagecachedir --root /var/tmp/podman-e2e-1117732790/subtest-3590578882/p/root --runroot /var/tmp/podman-e2e-1117732790/subtest-3590578882/p/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/containers/networks --network-backend netavark --cgroup-manager systemd --tmpdir /var/tmp/podman-e2e-1117732790/subtest-3590578882/p --events-backend file --db-backend sqlite --storage-driver overlay rm -fa -t 0
  << Timeline

  [FAILED] Command failed with exit status 125. See above for error message.
  In [It] at: /var/tmp/tmp.utA7CvAqkm/podman-5.6.0-dev/test/e2e/checkpoint_test.go:993 @ 05/28/25 15:37:29.763
------------------------------
• [FAILED] [0.797 seconds]
Podman checkpoint [It] podman checkpoint container with --pre-checkpoint and export (migration)
/var/tmp/tmp.utA7CvAqkm/podman-5.6.0-dev/test/e2e/checkpoint_test.go:1012

  Timeline >>
  Running: /usr/bin/podman --storage-opt overlay.imagestore=/var/tmp/podman-e2e-1117732790/imagecachedir --root /var/tmp/podman-e2e-1117732790/subtest-3081939929/p/root --runroot /var/tmp/podman-e2e-1117732790/subtest-3081939929/p/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/containers/networks --network-backend netavark --cgroup-manager systemd --tmpdir /var/tmp/podman-e2e-1117732790/subtest-3081939929/p --events-backend file --db-backend sqlite --storage-driver overlay network create
  podman1
  Running: /usr/bin/podman --storage-opt overlay.imagestore=/var/tmp/podman-e2e-1117732790/imagecachedir --root /var/tmp/podman-e2e-1117732790/subtest-3081939929/p/root --runroot /var/tmp/podman-e2e-1117732790/subtest-3081939929/p/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/containers/networks --network-backend netavark --cgroup-manager systemd --tmpdir /var/tmp/podman-e2e-1117732790/subtest-3081939929/p --events-backend file --db-backend sqlite --storage-driver overlay run -d --network podman1 quay.io/libpod/alpine:latest top
  39f3315882382cf1dc88d601978d163a2f887583ff3cc23ad86cfbd612764276
  Running: /usr/bin/podman --storage-opt overlay.imagestore=/var/tmp/podman-e2e-1117732790/imagecachedir --root /var/tmp/podman-e2e-1117732790/subtest-3081939929/p/root --runroot /var/tmp/podman-e2e-1117732790/subtest-3081939929/p/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/containers/networks --network-backend netavark --cgroup-manager systemd --tmpdir /var/tmp/podman-e2e-1117732790/subtest-3081939929/p --events-backend file --db-backend sqlite --storage-driver overlay container checkpoint -P -e /var/tmp/podman-e2e-1117732790/subtest-3081939929/p/pre-checkpoint-39f3315882382cf1dc88d601978d163a2f887583ff3cc23ad86cfbd612764276.tar.gz 39f3315882382cf1dc88d601978d163a2f887583ff3cc23ad86cfbd612764276
  memory tracking not supported. Please check CRIU logfile /var/tmp/podman-e2e-1117732790/subtest-3081939929/p/root/overlay-containers/39f3315882382cf1dc88d601978d163a2f887583ff3cc23ad86cfbd612764276/userdata/dump.log
  Error: `/usr/bin/crun checkpoint --image-path /var/tmp/podman-e2e-1117732790/subtest-3081939929/p/root/overlay-containers/39f3315882382cf1dc88d601978d163a2f887583ff3cc23ad86cfbd612764276/userdata/pre-checkpoint --work-path /var/tmp/podman-e2e-1117732790/subtest-3081939929/p/root/overlay-containers/39f3315882382cf1dc88d601978d163a2f887583ff3cc23ad86cfbd612764276/userdata --pre-dump 39f3315882382cf1dc88d601978d163a2f887583ff3cc23ad86cfbd612764276` failed: exit status 1
  [FAILED] in [It] - /var/tmp/tmp.utA7CvAqkm/podman-5.6.0-dev/test/e2e/checkpoint_test.go:1029 @ 05/28/25 15:37:30.556
  Running: /usr/bin/podman --storage-opt overlay.imagestore=/var/tmp/podman-e2e-1117732790/imagecachedir --root /var/tmp/podman-e2e-1117732790/subtest-3081939929/p/root --runroot /var/tmp/podman-e2e-1117732790/subtest-3081939929/p/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/containers/networks --network-backend netavark --cgroup-manager systemd --tmpdir /var/tmp/podman-e2e-1117732790/subtest-3081939929/p --events-backend file --db-backend sqlite --storage-driver overlay network rm -f podman1
  podman1
  Running: /usr/bin/podman --storage-opt overlay.imagestore=/var/tmp/podman-e2e-1117732790/imagecachedir --root /var/tmp/podman-e2e-1117732790/subtest-3081939929/p/root --runroot /var/tmp/podman-e2e-1117732790/subtest-3081939929/p/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/containers/networks --network-backend netavark --cgroup-manager systemd --tmpdir /var/tmp/podman-e2e-1117732790/subtest-3081939929/p --events-backend file --db-backend sqlite --storage-driver overlay stop --all -t 0
  Running: /usr/bin/podman --storage-opt overlay.imagestore=/var/tmp/podman-e2e-1117732790/imagecachedir --root /var/tmp/podman-e2e-1117732790/subtest-3081939929/p/root --runroot /var/tmp/podman-e2e-1117732790/subtest-3081939929/p/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/containers/networks --network-backend netavark --cgroup-manager systemd --tmpdir /var/tmp/podman-e2e-1117732790/subtest-3081939929/p --events-backend file --db-backend sqlite --storage-driver overlay pod rm -fa -t 0
  Running: /usr/bin/podman --storage-opt overlay.imagestore=/var/tmp/podman-e2e-1117732790/imagecachedir --root /var/tmp/podman-e2e-1117732790/subtest-3081939929/p/root --runroot /var/tmp/podman-e2e-1117732790/subtest-3081939929/p/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/containers/networks --network-backend netavark --cgroup-manager systemd --tmpdir /var/tmp/podman-e2e-1117732790/subtest-3081939929/p --events-backend file --db-backend sqlite --storage-driver overlay rm -fa -t 0
  << Timeline

  [FAILED] Command failed with exit status 125. See above for error message.
  In [It] at: /var/tmp/tmp.utA7CvAqkm/podman-5.6.0-dev/test/e2e/checkpoint_test.go:1029 @ 05/28/25 15:37:30.556
------------------------------
•SSSSSS•••••SS••

Summarizing 2 Failures:
  [FAIL] Podman checkpoint [It] podman checkpoint container with --pre-checkpoint
  /var/tmp/tmp.utA7CvAqkm/podman-5.6.0-dev/test/e2e/checkpoint_test.go:993
  [FAIL] Podman checkpoint [It] podman checkpoint container with --pre-checkpoint and export (migration)
  /var/tmp/tmp.utA7CvAqkm/podman-5.6.0-dev/test/e2e/checkpoint_test.go:1029

Observed in podman e2e test runs on container-selinux.

The podman and crun rpms used are from rhcontainerbot/podman-next copr .

I don't notice any inconsistencies in rpm versions of podman, crun and criu for x86_64 v/s aarch64. You can see the rpm info in the e2e test logs, including snippet here:

# FAIL on aarch64
criu-4.1-2.fc43.aarch64
crun-1.21-1.20250527135755523677.main.139.gb1a71e7b.fc43.aarch64
golang-1.24.3-4.fc43.aarch64
podman-5.6.0~dev-1.20250528143701899717.main.930.a484f78c44.fc43.aarch64

# PASS on x86_64
criu-4.1-2.fc43.x86_64
crun-1.21-1.20250527135755523677.main.139.gb1a71e7b.fc43.x86_64
golang-1.24.3-4.fc43.x86_64
podman-5.6.0~dev-1.20250528143701899717.main.930.a484f78c44.fc43.x86_64

Note: It's only the podman checkpoint e2e tests failing on aarch64. The checkpoint tests included in system test suite pass on both arches.

[lsm5]

Btw, these don't appear to be container-selinux failures as there aren't any avc denials.

[giuseppe]

there is no arch specific code in crun for that, so it is likely a criu failure, can we just skip these tests?

[lsm5]

there is no arch specific code in crun for that, so it is likely a criu failure, can we just skip these tests?

I guess I could skip aarch64 for now, I'll report this to criu as well. Thanks.