Add trie cache warmup by AndreiEres · Pull Request #7556 · paritytech/polkadot-sdk

AndreiEres · 2025-02-12T18:38:52Z

Description

Part of launching smart contracts on AssetHub. The current task involves warming up the state and performing sanity checks to ensure that memory usage remains within reasonable limits; otherwise, it should abort gracefully.

Here is a breakdown of the task by parts:

Configuration. We use a CLI flag to trigger the warm up. To increase the cache size, we use an existing flag.
Warm-up. We populate the trie cache with values from the database using a best hash as a storage root. For normal use we spawn a background task for it, for testing purposes we warm up in a blocking manner.
Sanity checks. If we find that the configured cache size is too large, we warn.

Variants of the added flag:

Flag	Meaning
[no flag]	no warm-up
warm-up-trie-cache	non-blocking warmup
warm-up-trie-cache non-blocking	non-blocking warmup (same as just the flag)
—warm-up-trie-cache blocking	blocking warmup

Integration

Tested on a snapshot from polkadot-asset-hub: 5426064 keys, ~900MIB state. Populating Trie cache takes 1.5–2 minutes on Macbook Pro M2.

Aditional Notes

Initially, we thought about putting the entire state into memory using a CLI flag or a runtime parameter, but decided against it. That’s up to collators to decide on which setup run their machines, they can use the current cache size setting to increase the throughput.

Todos

Testing moved to #8644

substrate/client/db/src/lib.rs

AndreiEres · 2025-02-28T14:03:18Z

/cmd prdoc --bump patch --audience node_dev

… node_dev'

# Description Working with #7556 I encountered an internal compiler error on polkadot-omni-node-lib: see bellow or [in CI](https://github.com/paritytech/polkadot-sdk/actions/runs/13521547633/job/37781894640). ``` error: internal compiler error: compiler/rustc_traits/src/codegen.rs:45:13: Encountered error `SignatureMismatch(SignatureMismatchData { found_trait_ref: <{closure@sp_state_machine::trie_backend_essence::TrieBackendEssence<sc_service::Arc<dyn sp_state_machine::trie_backend_essence::Storage<<<<NodeSpec as common::spec::BaseNodeSpec>::Block as common::NodeBlock>::BoundedHeader as sp_runtime::traits::Header>::Hashing>>, <<<NodeSpec as common::spec::BaseNodeSpec>::Block as common::NodeBlock>::BoundedHeader as sp_runtime::traits::Header>::Hashing, sp_trie::cache::LocalTrieCache<<<<NodeSpec as common::spec::BaseNodeSpec>::Block as common::NodeBlock>::BoundedHeader as sp_runtime::traits::Header>::Hashing>, sp_trie::recorder::Recorder<<<<NodeSpec as common::spec::BaseNodeSpec>::Block as common::NodeBlock>::BoundedHeader as sp_runtime::traits::Header>::Hashing>>::storage::{closure#1}} as std::ops::FnOnce<(std::option::Option<&mut dyn trie_db::TrieRecorder<sp_core::H256>>, std::option::Option<&mut dyn trie_db::TrieCache<sp_trie::node_codec::NodeCodec<<<<NodeSpec as common::spec::BaseNodeSpec>::Block as common::NodeBlock>::BoundedHeader as sp_runtime::traits::Header>::Hashing>>>)>>, expected_trait_ref: <{closure@sp_state_machine::trie_backend_essence::TrieBackendEssence<sc_service::Arc<dyn sp_state_machine::trie_backend_essence::Storage<<<<NodeSpec as common::spec::BaseNodeSpec>::Block as common::NodeBlock>::BoundedHeader as sp_runtime::traits::Header>::Hashing>>, <<<NodeSpec as common::spec::BaseNodeSpec>::Block as common::NodeBlock>::BoundedHeader as sp_runtime::traits::Header>::Hashing, sp_trie::cache::LocalTrieCache<<<<NodeSpec as common::spec::BaseNodeSpec>::Block as common::NodeBlock>::BoundedHeader as sp_runtime::traits::Header>::Hashing>, sp_trie::recorder::Recorder<<<<NodeSpec as common::spec::BaseNodeSpec>::Block as common::NodeBlock>::BoundedHeader as sp_runtime::traits::Header>::Hashing>>::storage::{closure#1}} as std::ops::FnOnce<(std::option::Option<&mut dyn trie_db::TrieRecorder<<<<NodeSpec as common::spec::BaseNodeSpec>::Block as common::NodeBlock>::BoundedHeader as sp_runtime::traits::Header>::Hash>>, std::option::Option<&mut dyn trie_db::TrieCache<sp_trie::node_codec::NodeCodec<<<<NodeSpec as common::spec::BaseNodeSpec>::Block as common::NodeBlock>::BoundedHeader as sp_runtime::traits::Header>::Hashing>>>)>>, terr: Sorts(ExpectedFound { expected: Alias(Projection, AliasTy { args: [Alias(Projection, AliasTy { args: [Alias(Projection, AliasTy { args: [NodeSpec/#0], def_id: DefId(0:410 ~ polkadot_omni_node_lib[7cce]::common::spec::BaseNodeSpec::Block), .. })], def_id: DefId(0:507 ~ polkadot_omni_node_lib[7cce]::common::NodeBlock::BoundedHeader), .. })], def_id: DefId(229:1706 ~ sp_runtime[5da1]::traits::Header::Hash), .. }), found: sp_core::H256 }) })` selecting `<{closure@sp_state_machine::trie_backend_essence::TrieBackendEssence<sc_service::Arc<dyn sp_state_machine::trie_backend_essence::Storage<<<<NodeSpec as common::spec::BaseNodeSpec>::Block as common::NodeBlock>::BoundedHeader as sp_runtime::traits::Header>::Hashing>>, <<<NodeSpec as common::spec::BaseNodeSpec>::Block as common::NodeBlock>::BoundedHeader as sp_runtime::traits::Header>::Hashing, sp_trie::cache::LocalTrieCache<<<<NodeSpec as common::spec::BaseNodeSpec>::Block as common::NodeBlock>::BoundedHeader as sp_runtime::traits::Header>::Hashing>, sp_trie::recorder::Recorder<<<<NodeSpec as common::spec::BaseNodeSpec>::Block as common::NodeBlock>::BoundedHeader as sp_runtime::traits::Header>::Hashing>>::storage::{closure#1}} as std::ops::FnOnce<(std::option::Option<&mut dyn trie_db::TrieRecorder<<<<NodeSpec as common::spec::BaseNodeSpec>::Block as common::NodeBlock>::BoundedHeader as sp_runtime::traits::Header>::Hash>>, std::option::Option<&mut dyn trie_db::TrieCache<sp_trie::node_codec::NodeCodec<<<<NodeSpec as common::spec::BaseNodeSpec>::Block as common::NodeBlock>::BoundedHeader as sp_runtime::traits::Header>::Hashing>>>)>>` during codegen ``` Trying to parse the error I found that TrieRecorder was not supposed to work with H256: - Expected: `&mut dyn TrieRecorder<<<<NodeSpec as common::spec::BaseNodeSpec>::Block as common::NodeBlock>::BoundedHeader as Header>::Hash>>` - Found: `&mut dyn TrieRecorder<sp_core::H256>>` The error happened because I added to `new_full_parts_with_genesis_builder` interaction with Trie Cache which eventually uses `TrieRecorder<H256>`. Here is the path: - In polkadot-omni-node-lib trait BaseNodeSpec defined with Block as `NodeBlock: BlockT<Hash = DbHash>`, where DbHash is H256. - BaseNodeSpec calls [new_full_parts_record_import::<Self::Block, … >](https://github.com/paritytech/polkadot-sdk/blob/75726c65d40d3631c441b946a26b2f0b30d40c26/cumulus/polkadot-omni-node/lib/src/common/spec.rs#L184-L189) and eventually it goes to [new_full_parts_with_genesis_builder](https://github.com/paritytech/polkadot-sdk/blob/08b302464a6ccaacf391516ff7b058fe07b5cca3/substrate/client/service/src/builder.rs#L195). - In `new_full_parts_with_genesis_builder` we accessed storage, initiating TrieRecorder with H256 what never happened before. I believe the compiler found a mismatch checking types for TrieRecorder: NodeBlock inherits from the trait `Block<Hash = DbHash>`, but it uses BoundedHeader, which inherits from the trait Header with the default Hash. --------- Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com>

substrate/client/db/src/lib.rs

bkchr

Looks good, mainly left nitpicks.

substrate/client/db/src/lib.rs

bkchr · 2025-05-23T15:40:53Z

substrate/client/service/src/builder.rs

+	use sc_client_api::backend::Backend;
+	use sp_state_machine::Backend as StateBackend;
+
+	let state = || backend.state_at(storage_root, TrieCacheContext::Untrusted);


Suggested change

let state = || backend.state_at(storage_root, TrieCacheContext::Untrusted);

let state = || backend.state_at(storage_root, TrieCacheContext::Trusted);

Or why is this untrusted? Also why are we not keeping the state alive the entire time? So, directly write back to the global cache?

Trusted is meant to be used when we are in a bounded context, otherwise it might balloon beyond the shared_trie_cache configured size.

So, using untrusted here and destroying it after every key access is the right call here, because that will end up populating the shared trie cache.

But untrusted means that we don't cache things like the :code (which is bigger than 2MiB in most of the cases already)

Really good point. Then I would go with UnTrusted for the iterator because it stays alive for the entire duration of the warmup and Trusted for each key access since we use a different instance for each iteration.

substrate/client/service/src/builder.rs

polkadot/zombienet-sdk-tests/tests/parachains/warm_up_trie_cache.rs

paritytech-workflow-stopper · 2025-05-26T09:13:08Z

All GitHub workflows were cancelled due to failure one of the required jobs.
Failed workflow url: https://github.com/paritytech/polkadot-sdk/actions/runs/15250050789
Failed job name: build-test-collators

paritytech-workflow-stopper · 2025-05-26T09:13:09Z

All GitHub workflows were cancelled due to failure one of the required jobs.
Failed workflow url: https://github.com/paritytech/polkadot-sdk/actions/runs/15250050775
Failed job name: test-linux-stable

bkchr

Besides this, it looks good to me.

This #7556 (comment) made me think that we can have some miss-uses of the trusted cache, so it needs some reasonable bounds for the size. Since there is no point in letting it grow past the shared cache limits, because the items will be discarded anyways when we propagate them back to the share cache so I decided to bound it to those limits. --------- Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io> Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com>

# Description Fixes #7533. Part of launching smart contracts on AssetHub. The current task involves warming up the state and performing sanity checks to ensure that memory usage remains within reasonable limits; otherwise, it should abort gracefully. Here is a breakdown of the task by parts: - **Configuration.** We use a CLI flag to trigger the warm up. To increase the cache size, we use an existing flag. - **Warm-up.** We populate the trie cache with values from the database using a best hash as a storage root. For normal use we spawn a background task for it, for testing purposes we warm up in a blocking manner. - **Sanity checks.** If we find that the configured cache size is too large, we warn. Variants of the added flag: | Flag | Meaning | |---------------------------------|---------------------------------------------| | [no flag] | no warm-up | | warm-up-trie-cache | non-blocking warmup | | warm-up-trie-cache non-blocking | non-blocking warmup (same as just the flag) | | —warm-up-trie-cache blocking | blocking warmup | ## Integration Tested on a snapshot from polkadot-asset-hub: 5426064 keys, ~900MIB state. Populating Trie cache takes 1.5–2 minutes on Macbook Pro M2. ## Aditional Notes Initially, we thought about putting the entire state into memory using a CLI flag or a runtime parameter, but decided against it. That’s up to collators to decide on which setup run their machines, they can use the current cache size setting to increase the throughput. ## Todos Testing moved to #8644 --------- Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Alexandru Gheorghe <49718502+alexggh@users.noreply.github.com> Co-authored-by: Alexander Theißen <alex.theissen@me.com>

This #7556 (comment) made me think that we can have some miss-uses of the trusted cache, so it needs some reasonable bounds for the size. Since there is no point in letting it grow past the shared cache limits, because the items will be discarded anyways when we propagate them back to the share cache so I decided to bound it to those limits. --------- Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io> Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com>

# Description Fixes #7533. Part of launching smart contracts on AssetHub. The current task involves warming up the state and performing sanity checks to ensure that memory usage remains within reasonable limits; otherwise, it should abort gracefully. Here is a breakdown of the task by parts: - **Configuration.** We use a CLI flag to trigger the warm up. To increase the cache size, we use an existing flag. - **Warm-up.** We populate the trie cache with values from the database using a best hash as a storage root. For normal use we spawn a background task for it, for testing purposes we warm up in a blocking manner. - **Sanity checks.** If we find that the configured cache size is too large, we warn. Variants of the added flag: | Flag | Meaning | |---------------------------------|---------------------------------------------| | [no flag] | no warm-up | | warm-up-trie-cache | non-blocking warmup | | warm-up-trie-cache non-blocking | non-blocking warmup (same as just the flag) | | —warm-up-trie-cache blocking | blocking warmup | ## Integration Tested on a snapshot from polkadot-asset-hub: 5426064 keys, ~900MIB state. Populating Trie cache takes 1.5–2 minutes on Macbook Pro M2. ## Aditional Notes Initially, we thought about putting the entire state into memory using a CLI flag or a runtime parameter, but decided against it. That’s up to collators to decide on which setup run their machines, they can use the current cache size setting to increase the throughput. ## Todos Testing moved to #8644 --------- Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Alexandru Gheorghe <49718502+alexggh@users.noreply.github.com> Co-authored-by: Alexander Theißen <alex.theissen@me.com>

This #7556 (comment) made me think that we can have some miss-uses of the trusted cache, so it needs some reasonable bounds for the size. Since there is no point in letting it grow past the shared cache limits, because the items will be discarded anyways when we propagate them back to the share cache so I decided to bound it to those limits. --------- Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io> Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com>

- Replace `<T as pallet::Config>::RuntimeEvent` with `<T as frame_system::Config>::RuntimeEvent` in all benchmark files, since RuntimeEvent was removed from pallet Config traits (paritytech/polkadot-sdk#7229) - Add missing 5th argument (SharedTrieCache) to storage benchmark `cmd.run()` calls in node/src/command.rs (paritytech/polkadot-sdk#7556) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

AndreiEres added 5 commits February 12, 2025 19:36

Bootstrap

1de5d45

Warm up trie cache

aa9fd47

Skip the warm-up if we shouldn’t fit the cache in memory

2a5984b

Define memory usage

ffe5f33

Fix DatabaseSettings

c520c63

AndreiEres added the T0-node This PR/Issue is related to the topic “node”. label Feb 19, 2025

AndreiEres added 5 commits February 19, 2025 13:10

Fix Configuration

59a1e61

Fix Configuration

c36916b

Fix Configuration

269f87a

Merge branch 'master' into AndreiEres/trie-warmup

9f708b5

Process child keys

d3e83db

bkchr reviewed Feb 19, 2025

View reviewed changes

substrate/client/db/src/lib.rs Outdated Show resolved Hide resolved

AndreiEres added 7 commits February 24, 2025 17:30

Fix compilation

1dc4de1

Add progress to key population

9bbb397

Fix clippy warning

ed89fb1

Fix missing force_in_memory_trie_cache

4c90ff2

Merge branch 'master' into AndreiEres/trie-warmup

75a1ba4

Add missing force_in_memory_trie_cache

3053fe0

Update ImportParams

75726c6

alexggh self-requested a review February 28, 2025 09:32

Specify DbHash in NodeBlock

08b3024

AndreiEres mentioned this pull request Feb 28, 2025

Fix unspecified Hash in NodeBlock #7756

Merged

github-actions bot and others added 2 commits February 28, 2025 14:05

Update from AndreiEres running command 'prdoc --bump patch --audience…

e8611eb

… node_dev'

Merge branch 'master' into AndreiEres/trie-warmup

07fb16e

AndreiEres added 3 commits March 4, 2025 17:28

Merge branch 'master' into AndreiEres/trie-warmup

1ae78b3

Add initial zombienet test

47ac732

Update test

428f606

alexggh reviewed Mar 7, 2025

View reviewed changes

substrate/client/db/src/lib.rs Outdated Show resolved Hide resolved

bkchr reviewed May 23, 2025

View reviewed changes

AndreiEres added 3 commits May 26, 2025 10:33

Decrease log level

ed7dd95

Don't error on lack of memory

be589e3

Remove zombie test

314818c

AndreiEres mentioned this pull request May 26, 2025

[WIP] Add Trie Cache warm-up test #8644

Closed

Merge branch 'master' into AndreiEres/trie-warmup

d7fb87b

AndreiEres added 2 commits May 26, 2025 11:20

Fix types

1981dda

Update pr doc

8fd35e6

bkchr approved these changes May 26, 2025

View reviewed changes

Use Trusted and Untrusted states

04f6823

alexggh approved these changes May 28, 2025

View reviewed changes

alexggh added this pull request to the merge queue May 28, 2025

Merged via the queue into master with commit 10cde1f May 28, 2025
182 checks passed

alexggh deleted the AndreiEres/trie-warmup branch May 28, 2025 08:01

alexggh mentioned this pull request May 28, 2025

bound trusted local cache to shared limits sizes #8688

Merged

AndreiEres mentioned this pull request Jun 10, 2025

Enable Trie Cache warm-up for AssetHub parachains #8810

Closed

11 tasks

github-actions bot mentioned this pull request Oct 24, 2025

Update polkadot-sdk from stable2503 to stable2506 moondance-labs/tanssi#1340

Closed

pgherveou added this to [preview] release tracker Mar 20, 2026

github-project-automation bot moved this to Todo in [preview] release tracker Mar 20, 2026

github-project-automation bot moved this from Todo to Done in [preview] release tracker Mar 20, 2026

pgherveou removed this from [preview] release tracker Mar 20, 2026

	let state = \|\| backend.state_at(storage_root, TrieCacheContext::Untrusted);
	let state = \|\| backend.state_at(storage_root, TrieCacheContext::Trusted);

Conversation

AndreiEres commented Feb 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Integration

Aditional Notes

Todos

Uh oh!

Uh oh!

AndreiEres commented Feb 28, 2025

Uh oh!

Uh oh!

bkchr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bkchr May 23, 2025

Choose a reason for hiding this comment

Uh oh!

alexggh May 26, 2025

Choose a reason for hiding this comment

Uh oh!

bkchr May 26, 2025

Choose a reason for hiding this comment

Uh oh!

alexggh May 26, 2025

Choose a reason for hiding this comment

Uh oh!

bkchr May 26, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

paritytech-workflow-stopper bot commented May 26, 2025

Uh oh!

paritytech-workflow-stopper bot commented May 26, 2025

Uh oh!

bkchr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

AndreiEres commented Feb 12, 2025 •

edited

Loading