Upgrade trie-db from 0.28.0 to 0.29.0

[ffarall]

Description

• What does this PR do?

1. Upgrades trie-db's version to the latest release. This release includes, among others, an implementation of DoubleEndedIterator for the TrieDB struct, allowing to iterate both backwards and forwards within the leaves of a trie.
2. Upgrades trie-bench to 0.39.0 for compatibility.
3. Upgrades criterion to 0.5.1 for compatibility.

• Why are these changes needed?
  Besides keeping up with the upgrade of trie-db, this specifically adds the functionality of iterating back on the leafs of a trie, with sp-trie. In a project we're currently working on, this comes very handy to verify a Merkle proof that is the response to a challenge. The challenge is a random hash that (most likely) will not be an existing leaf in the trie. So the challenged user, has to provide a Merkle proof of the previous and next existing leafs in the trie, that surround the random challenged hash.

Without having DoubleEnded iterators, we're forced to iterate until we find the first existing leaf, like so:

        // ************* VERIFIER (RUNTIME) *************
        // Verify proof. This generates a partial trie based on the proof and
        // checks that the root hash matches the `expected_root`.
        let (memdb, root) = proof.to_memory_db(Some(&root)).unwrap();
        let trie = TrieDBBuilder::<LayoutV1<RefHasher>>::new(&memdb, &root).build();

        // Print all leaf node keys and values.
        println!("\nPrinting leaf nodes of partial tree...");
        for key in trie.key_iter().unwrap() {
            if key.is_ok() {
                println!("Leaf node key: {:?}", key.clone().unwrap());

                let val = trie.get(&key.unwrap());

                if val.is_ok() {
                    println!("Leaf node value: {:?}", val.unwrap());
                } else {
                    println!("Leaf node value: None");
                }
            }
        }

        println!("RECONSTRUCTED TRIE {:#?}", trie);

        // Create an iterator over the leaf nodes.
        let mut iter = trie.iter().unwrap();

        // First element with a value should be the previous existing leaf to the challenged hash.
        let mut prev_key = None;
        for element in &mut iter {
            if element.is_ok() {
                let (key, _) = element.unwrap();
                prev_key = Some(key);
                break;
            }
        }
        assert!(prev_key.is_some());

        // Since hashes are `Vec<u8>` ordered in big-endian, we can compare them directly.
        assert!(prev_key.unwrap() <= challenge_hash.to_vec());

        // The next element should exist (meaning there is no other existing leaf between the
        // previous and next leaf) and it should be greater than the challenged hash.
        let next_key = iter.next().unwrap().unwrap().0;
        assert!(next_key >= challenge_hash.to_vec());

With DoubleEnded iterators, we can avoid that, like this:

        // ************* VERIFIER (RUNTIME) *************
        // Verify proof. This generates a partial trie based on the proof and
        // checks that the root hash matches the `expected_root`.
        let (memdb, root) = proof.to_memory_db(Some(&root)).unwrap();
        let trie = TrieDBBuilder::<LayoutV1<RefHasher>>::new(&memdb, &root).build();

        // Print all leaf node keys and values.
        println!("\nPrinting leaf nodes of partial tree...");
        for key in trie.key_iter().unwrap() {
            if key.is_ok() {
                println!("Leaf node key: {:?}", key.clone().unwrap());

                let val = trie.get(&key.unwrap());

                if val.is_ok() {
                    println!("Leaf node value: {:?}", val.unwrap());
                } else {
                    println!("Leaf node value: None");
                }
            }
        }

        // println!("RECONSTRUCTED TRIE {:#?}", trie);
        println!("\nChallenged key: {:?}", challenge_hash);

        // Create an iterator over the leaf nodes.
        let mut double_ended_iter = trie.into_double_ended_iter().unwrap();

        // First element with a value should be the previous existing leaf to the challenged hash.
        double_ended_iter.seek(&challenge_hash.to_vec()).unwrap();
        let next_key = double_ended_iter.next_back().unwrap().unwrap().0;
        let prev_key = double_ended_iter.next_back().unwrap().unwrap().0;

        // Since hashes are `Vec<u8>` ordered in big-endian, we can compare them directly.
        println!("Prev key: {:?}", prev_key);
        assert!(prev_key <= challenge_hash.to_vec());

        println!("Next key: {:?}", next_key);
        assert!(next_key >= challenge_hash.to_vec());

• How were these changes implemented and what do they affect?
  All that is needed for this functionality to be exposed is changing the version number of trie-db in all the Cargo.tomls applicable, and re-exporting some additional structs from trie-db in sp-trie.

[cla-bot-2021]

User @ffarall, please sign the CLA here.

[snowmead]

@ffarall in lib.rs, can you re-export the double ended iterators.

pub use trie_db::{
  triedb::{TrieDBDoubleEndedIterator, TrieDBKeyDoubleEndedIterator},
  TrieDBNodeDoubleEndedIterator,
};

[bkchr]

@ffarall CI is not happy btw.

[ffarall]

@ffarall CI is not happy btw.

Yes, thanks @bkchr! I'll look into the failures and fix them.

[paritytech-cicd-pr]

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-clippy
Logs: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/5792020

[cheme]

In this file trie-bench need to be upgraded to 0.39 (forgot to tell you :)

[ffarall]

Thank you! Solved here: build: ⬆️ Upgrade trie-bench from 0.38.0 to 0.39.0

[cheme]

should have only trie-db here (here both 0.28 and 0.29 are living together in the repo).

[ffarall]

Also solved here: build: ⬆️ Upgrade trie-bench from 0.38.0 to 0.39.0

[ffarall]

@ffarall CI is not happy btw.

@bkchr CI seems to be happy now (at least the required ones). Only thing missing is another review apparently.