Backup integrity failure

[aminhashemi1994]

Hi,

I am encountering an issue while creating backups. I have two physical servers, and on one of them, every time I create a backup and attempt to verify it, I get an error for an MD5 checksum mismatch. I have tried converting my QCOW2 image to RAW and then back to QCOW2, using qemu-img check -r all to repair the QCOW2 image, but the issue persists.

Most of the time, when I create a backup and verify it using the command virtnbdrestore -i -o verify, I get the mismatch error. Occasionally, the full backup verifies successfully, but when I create a new incremental backup, it fails. This problem does not occur on my other server; it is only present on this specific one.

I have tried creating backups on different disks to rule out disk issues. I also moved my VM disk to another disk and created a backup, but none of these attempts resolved the issue. My next step is to upgrade all my Ubuntu 22 packages via a full distribution upgrade or potentially reinstall the OS to see if that resolves the problem.

There are only two VMs on this server: one is a VOIP VM running Debian 10, and the other is a Windows Server. Both VMs have the qemu-guest-agent active.

My server OS is Ubuntu 22 with virtnbd version 2.21 and my libvirt version is 8.0.0.
There is an screenshot of the situation occur in my server.

[abbbi]

cant reproduce. Have you tried another storage to store your backups to?
filesystem in /opt is OK? Have you executed fsck?

 ../virtnbdrestore -a verify -i /tmp/data/ --output /tmp
[2025-03-06 09:45:22] INFO lib common - printVersion [main]:  Version: 2.22 Arguments: ../virtnbdrestore -a verify -i /tmp/data/ --output /tmp
[2025-03-06 09:45:22] INFO root files - verify [main]:  Computing checksum for: /tmp/data/sda.full.data
[2025-03-06 09:45:22] INFO root files - verify [main]:  Checksum result: 4046526109
[2025-03-06 09:45:22] INFO root files - verify [main]:  Comparing checksum with stored information
[2025-03-06 09:45:22] INFO root files - verify [main]:  OK

i suspect:

1. filesystem issues in /opt/
2. memory issues on server
3. third party software messing with the data.

[abbbi]

unfortunately your screenshot of the log output doesn't include the checksum reported during backup.
You could also check if the checksum that is reported by the backup was stored in the apropriate .chksum file on the filesystem. Maybe the .chksum file was altered or doesnt match the backup.

[aminhashemi1994]

I haven't run fsck on /opt yet. However, I have several cron job scripts in place that back up the disk as a secondary method (I simply use cp to copy the running disk to another directory) in case the virtnbdbackup fails. Additionally, I create a snapshot of the disk every day and remove older snapshots, keeping only 3 days' worth.

"My script creates a lock file when virtnbdbackup is active, preventing another backup from starting."

I found a file named sda.copy.data.chksum with the value 2074616923. At the time I created the backup, no snapshots were being removed or created. However, I can remove all my snapshots and test again if you think that might be the cause.

What do you recommend I do next?

@abbbi

[abbbi]

run fsck on your disk. I have no other users complaining about such situations and its not reproducible for me.
Also, the CI test for this utility run the verify tests aswell, without any issues.
You speak about a custom script, which i dont know of, or your workflow.
Maybe the workflow is flawed so that .chksum files contain wrong checksum.
Comopare the cheksum in the .chksum file against the checksum reported during backup, checksum is logged there too.

[aminhashemi1994]

I ran fsck and got no any problem and the checksum that was reported by the virtnbdbackup during the backup was the checksum that was stored in sda.copy.data.chksum.
Where is the .checksum file? Is it somewhere else from my backup directory?

I will test some other things and let you know if i could fix the problem.

@abbbi

[rmk177]

Good day! I also encountered this problem. With qcow2 disks I have correct checksum, but with raw disks I have checksum failure. Maybe because we store qcow2 and raw disks in different formats and metadata matters?

[aminhashemi1994]

Hi
Creating backup for raw disk is not a good solution. It is not recoverable as far as i can remember. You can recover the disk but the disk won't be containing any data.
Best way is to convert that disk to qcow2.
But what is the problem that you are facing exactly?
Running checksum check on the backup directory will cause to a checksum missmatch? And does this happens on your qcow2 disks or your raw disk or both?

@rmk177

[abbbi]

checksums for disks with --raw option included are not calculated properly

#269

the raw images are treated differently during backup, no metadata format is written.