Elemental upgrades miss files due to rsync’s only size/mtime comparison

[manfuin]

I had quite a troubleshooting with a systems not booting after elemental upgrade.

The root cause: the files in systemd packages in updated image of openSUSE Leap 16 have same size and mtime but actually different and NOT synced to the new snapshot by elemental upgrade.

Elemental runs the rsync, which by default only relies on metadata (compare size and mtime on source vs destination) to speedup process. But this leads to inconsistent state of the snapshot content in such a sensitive thing as OS files.

To improve reliability IMHO it is better to spend some CPU cycles on computing checksums, but guarantee the reliable copy from container image to btrfs snapshot.

More details from my troubleshooting of upgrading from image with systemd-257.13-160000.2.1.x86_64 to image with systemd-257.13-160000.3.1.x86_64 (output is omitted to show only relevant parts):

Baseline - snapshot content after elemental upgrade vs extracted pulled image content in the pulled directory:

p-x-dqd-cp-0:~ # elemental upgrade --system harbor.maxiv.lu.se/elemental/flavor/control-plane:master-0ba99e58 --debug
DEBU[2026-07-22T09:21:44Z] Running cmd: 'btrfs subvolume list -a --sort=path /'
DEBU[2026-07-22T09:21:44Z] Running cmd: 'btrfs subvolume get-default /'
DEBU[2026-07-22T09:21:44Z] Creating snapshot 3
DEBU[2026-07-22T09:21:44Z] Running cmd: 'btrfs subvolume snapshot /.snapshots/2/snapshot /.snapshots/3/snapshot'
INFO[2026-07-22T09:21:44Z] Copying harbor.maxiv.lu.se/elemental/flavor/control-plane:master-0ba99e58 source...
INFO[2026-07-22T09:21:55Z] Finished copying harbor.maxiv.lu.se/elemental/flavor/control-plane:master-0ba99e58 into /.snapshots/3/snapshot.workDir
INFO[2026-07-22T09:21:55Z] Starting rsync...
DEBU[2026-07-22T09:21:55Z] Running cmd: 'rsync --progress --partial --human-readable --archive --xattrs --acls --delete --filter=-x security.selinux /.snapshots/3/snapshot.workDir/ /.snapshots/3/snapshot/'
INFO[2026-07-22T09:21:57Z] Finished syncing

p-x-dqd-cp-0:~ # elemental pull-image harbor.maxiv.lu.se/elemental/flavor/control-plane:master-0ba99e58 pulled
INFO[2026-07-22T09:11:14Z] Pulling image harbor.maxiv.lu.se/elemental/flavor/control-plane:master-0ba99e58 platform linux/amd64

Comparing systemctl:

p-x-dqd-cp-0:~ # ll pulled/usr/bin/systemctl
-rwxr-xr-x 1 root root 297704 jul  1 08:28 pulled/usr/bin/systemctl

p-x-dqd-cp-0:~ # ll /.snapshots/3/snapshot/usr/bin/systemctl
-rwxr-xr-x 1 root root 297704 jul  1 08:28 /.snapshots/3/snapshot/usr/bin/systemctl

p-x-dqd-cp-0:~ # diff /.snapshots/3/snapshot/usr/bin/systemctl pulled/usr/bin/systemctl
Binary files /.snapshots/3/snapshot/usr/bin/systemctl and pulled/usr/bin/systemctl differ

p-x-dqd-cp-0:~ # ldd /.snapshots/3/snapshot/usr/bin/systemctl
	linux-vdso.so.1 (0x00007f39a32bf000)
	libsystemd-shared-257.13-160000.2.1.so => /usr/lib64/systemd/libsystemd-shared-257.13-160000.2.1.so (0x00007f39a2e00000)

p-x-dqd-cp-0:~ # ldd pulled/usr/bin/systemctl
	linux-vdso.so.1 (0x00007f497ea57000)
	libsystemd-shared-257.13-160000.3.1.so => not found

Testing checksum vs default behavior of rsync:

p-x-dqd-cp-0:~ # rsync --progress --partial --human-readable --archive --xattrs --acls --verbose --dry-run pulled/ /.snapshots/3/snapshot/
sending incremental file list
./
run/

sent 705,36K bytes  received 4,63K bytes  284,00K bytes/sec
total size is 1,94G  speedup is 2.730,22 (DRY RUN)
p-x-dqd-cp-0:~ # rsync --progress --partial --human-readable --archive --xattrs --acls --verbose --checksum --dry-run pulled/ /.snapshots/3/snapshot/
sending incremental file list
./
run/
usr/bin/bootctl
usr/bin/busctl
usr/bin/coredumpctl
usr/bin/hostnamectl
usr/bin/journalctl
usr/bin/kernel-install
usr/bin/localectl
usr/bin/loginctl
usr/bin/systemctl
usr/bin/systemd-ac-power
...

After elemental-toolkit was rebuilt with --checksum the system was able to receive correct update and boot successfully.

Note, that the newly build elemental binary needs to be used, not the one already on the affected system (for example, updating via operator will rely on elemental binary inside container image, making it non-issue).

P.S. systemd is at least noticeable, as system crashes. But any file can be out of sync without checksums and left unnoticed with small bugs here and there.