[Question] How to guarantee the data integrity of the images and the blocks?

[allencloud]

Question

today, the block format does not appear to include a checksum and I think that this could potentially lead to a bad block (either through corruption or alternatively via a malicious actor) being distributed through the P2P network and potentially poisoning the cache.   

You mentioned in the SIG presentation that there were some protections for this but I could not understand the specifics.

Would it be possible to explain how the data integrity of the images and the blocks are maintained please?

[allencloud]

the block format does not appear to include a checksum.

No, the block format does not include a checksum.

However, for each distributing file, we record all blocks' checksum into a separate file, which is named by MD5 file.

What is a MD5 file for a block?

Each MD5 file maps each distributing file. The MD5 file includes the following three parts:

• every block's MD5 info;
• the whole distributing file's MD5 info;
• the SHA1 info of the above data;

How is a MD5 file produced?

Here is the procedure of how to produce a MD5 file in Dragonfly:

1. When dfget agents trigger to distribute a file, Supernode makes the CDN manager to download the target file.
2. As downloading the file, CDN manager will process to generate blocks for target file according to raw of data .
3. Each time finishing to produce a block, the CDN manager would calculate MD5 for the block, and the store the MD5 in memory.
4. When finishing to calculate all MD5 for all blocks, the CDN manager will continue to calculate MD5 for the downloading file.
5. The CDN manager continues to calculate SHA1 of all calculated MD5 values.
6. The CDN managers write each block's MD5, the whole file's MD5 and SHA1 of all MD5 values into the block metadata file.

The following is the brief example of a MD5 file:

$ ll -h /home/admin/supernode/repo/download/1a0
-rw-r--r-- 1 root root  22M Feb  4 09:23 1a09b5b0c0bb42c2f87b217b71a637c766ee8c598ba3790025d83212f2e7dce1
-rw-r--r-- 1 root root  319 Feb  4 09:23 1a09b5b0c0bb42c2f87b217b71a637c766ee8c598ba3790025d83212f2e7dce1.md5
-rw-r--r-- 1 root root  376 Feb  4 09:23 1a09b5b0c0bb42c2f87b217b71a637c766ee8c598ba3790025d83212f2e7dce1.meta

$ cat 1a09b5b0c0bb42c2f87b217b71a637c766ee8c598ba3790025d83212f2e7dce1.md5
ce5584163a368f2856c0a28cdac1a731:4194304 // MD5 of 1st block and the length
73ac985ba8d37fbc99c3c113c9170e84:4194304 // MD5 of 2nd block and the length
60df935374c85b208c4cb43d1959f331:4194304 // MD5 of 3rd block and the length
35d319f56c4895daa5a9f8427de4340e:4194304 // MD5 of 4th block and the length
b0a07657a0b1b0e133c79785c63e8724:4194304 // MD5 of 5th block and the length
2362dcbf5d5294be0b1744f40f0e423a:1048606 // MD5 of 6th block and the length
4aae22e14d5a70eaa769d3ee50804427         // MD5 of the whole distributing file
418fe37595d7d3f9731a6d6b335b275605bdb791 // SHA1 of all above things

How is a MD5 file consumed?

Here is the procedure of how to consume a MD5 file in Dragonfly:

1. When distributing a block to a peer, supernode will fetch the MD5 of this block and send this MD5 to peer along with block;
2. After finishing to download the block and MD5, dfget will generate another MD5 of the downloaded block, and check the newly-generated MD5 value with downloaded MD5 value.
3. If they match, we can trust the data integrity. If they don't match, dfget will report this mismatch to supernode including block info and incorrect peer, and then dfget will try to download the block from other peers;
4. Supernode gets aware of which peer provided incorrect data integrity, and then supernode isolates this peer from the whole peer network.

[chira001]

Thank you for the comprehensive response - this clarifies the process well.

[allencloud]

I think we could add this into the FAQ or a separate documentation.