Experimental RuntimeDebug -> Debug substitutions.#3107
Closed
hirschenberger wants to merge 3 commits intoparitytech:masterfrom
Closed
Experimental RuntimeDebug -> Debug substitutions.#3107hirschenberger wants to merge 3 commits intoparitytech:masterfrom
hirschenberger wants to merge 3 commits intoparitytech:masterfrom
Conversation
Member
|
I think we can also deprecate the |
Member
Yes. |
Member
|
@hirschenberger I think we can go ahead with this? |
Contributor
Author
Ok, so I'll continue. The next steps would be to deprecate RuntimeDebug and fix all resulting deprecation warnings, correct? |
Member
Yes! |
kianenigma
added
the
T1-FRAME
Member
|
@hirschenberger are you interested on continue working on this? |
Contributor
Author
Yes, just got a new notebook which makes development on that issue much snappier. |
Member
|
Ty! I'm quite interesting in this and would be happy to merge this in the near future. |
Contributor
Author
I try to deliver ASAP |
Contributor
Author
|
@bkchr What about |
Member
Yes! |
Contributor
Author
|
@bkchr Hey Basti, after subsituting all So there are many generic type parameters missing a |
|
The CI pipeline was cancelled due to failure one of the required jobs. |
Member
You can not just replace this with |
Contributor
Author
|
Still working on this, I tried to build my own refactoring tool using I also found the nice ast-grep tool which I'm giving a quick try. |
Member
|
@hirschenberger ping. |
1 task
17 tasks
github-merge-queue bot
pushed a commit
that referenced
this pull request
Apr 21, 2025
Moved from: #7601. Follow ups to: #7282. Closes: #8146 --- This PR is the final outcome of a multi-month development period, with a lot of background work since 2022. Its main aim is to make pallet-staking, alongside its `type ElectionProvider` compatible to be used in a parachain, and report back the validator set to a relay-chain. This setup is intended to be used for Polkadot, Kusama and Westend relay-chains, with the corresponding AssetHubs hosting the staking system. While this PR is quite big, a lot of the diffs are due to adding a relay and parachain runtime for testing. The following is a guide to help reviewers/auditors distinguish what has actually changed in this PR. > Additional reading: See polkadot-js/apps#11401, and the hackmd shared in there, which contains more in-depth explanation of how RC <> AH communicate. ## Added > [This shows the partial diff](ankn/diff-staking-async...ankn/diff-staking-async-1) introduced in pallet-staking-async and election-provider-multi-block relative to the existing (in master) pallet-staking and election-provider-multi-phase. This PR adds the following new pallets, all of which are not used anywhere yet, with the exception of one (see `westend-runtime` changes below). #### `pallet-election-provider-multi-block` This is a set of 4 pallets, capable of implementing an async, multi-page `ElectionProvider`. This pallet is not used in any real runtime yet, and is intended to be used in `AssetHub`, next to `pallet-staking-async`. #### `pallet-staking-async` A fork of the old `pallet-staking`, with a number of key differences, making it suitable to be used in a parachain: 1. It no longer has access to a secure timestamp, previously used to calculate the duration of an era. 2. It no longer has access to a `pallet-session`. 3. It no longer has access to a `pallet-authorship`. 5. It is capable of working with a multi-page `ElectionProvider`, aka. `pallet-election-provider-multi-block`. To compensate for the above, this pallet relies on XCM messages coming from the relay-chain, informing the pallet of: * When a new era should be activated, and how long its duration was * When an offence has happened on the relay relay-chain * When a session ends on the relay-chain, and how many reward points were accumulated for each validators during that period. #### `pallet-staking-async-ah-client` and `pallet-staking-async-rc-client` Are the two new pallets that facilitate the above communication. #### `pallet-ahm-test` A test-only crate that contains e2e rust-based unit test for all of the above. #### `pallet-staking-async-rc-runtime` and `pallet-staking-async-parachain-runtime` Forks of westend and westend-asset-hub, customized to be used for testing all of the above with Zombienet. It contains a lot of unrelated code as well. ## Changed > [This shows the partial diff](ankn/8127-diff-changed-base...ankn/8127-diff-changed-compare) that shows the changes to existing pallets used in prod runtimes as well as westend runtime changes. #### `Identification` This mechanism, which lives on the relay-chain, is expressed by `type FullIdentification` and `type FullIdentificationOf` in runtimes. It is a way to identify the full data needed to slash a validator. Historically, it was pointing to a validator, and their `struct Exposure`. With the move to Asset-Hub, this is no longer possible for two reasons: 1. Relay chain no longer knows the full exposures 2. Even if, the full exposures are getting bigger and bigger and relying the entirety of it is not scalable. Instead, runtimes now move to a new `type FullIdentificationOf = DefaultExposureOf`, which will identify a validator with a `Exposure::default()`. This is suboptimal, as it forces us to still store a number of bytes. Yet, it allows any old `FullIdentification`, pertaining to an old slash, to be decoded. This compromise is only needed to cater for slashes that happen around the time of AHM. #### `westend-runtime` This runtime already has the `pallet-staking-async-ah-client`, integrated into all the places such that: 1. It handles the validator reward points 2. It handles offences 5. It is the `SessionManager` Yet, it is delegating all of the above to its `type Fallback`, which is the old `pallet-staking`. This is a preparatory step for AHM, and should not be any logical change. #### `pallet-election-provider-multi-phase` This is the old single-page `ElectionProvider`. It has been updated to work with multi-page traits, yet it only supports `page-size = 1` for now. It should not have seen any logical changes. #### `pallet-bags-list` Now has two new features. 1. It can be `Locked`, in which case all updates to it fail with an `Err(_)`, even deletion of a node. This is needed because we cannot alter any nodes in this pallet during a multi-page iteration, aka. multi-page snapshot. 2. To combat this, the same `rebag` transaction can be also be used to remove a node from the list, or add a node to the list. This is done through the `score_of` api. See the file changes and tests under `./substrate/frame/bags-list` for more info. #### RuntimeDebug -> Debug To facilitate debugging, a number of types' `RuntimeDebug` impl has been changed to `Debug`. See #3107 ## Weights Below is a summary of the weights. These are generated using `staking-async/runtimes/parachain`, which assumes 22_500 nominators divided by `32` pages for Polkadot, and 12_500 nominators divided by `16` pages in Kusama, both leading to ~700 nominators snapshotted and exported per page. Doubling these parameters would easily slash the PoV weights by half, but with 10MB PoV, these numbers should be good. Also noting that with PoV clawback, we migth get even more proof_size weight back in the runtime. Although, afaik this reclaimed value does not take compression into account. ``` #### new: polkadot/pallet_election_provider_multi_block.rs old: kusama +-----------------------------------------+--------------------------------------+---------+---------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +======================================================================================================================+ | pallet_election_provider_multi_block.rs | on_initialize_into_snapshot_msp | 2.41MiB | 2.41MiB | -0.03 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_snapshot_rest | 3.24MiB | 3.06MiB | -5.53 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_signed | 3.36MiB | 3.12MiB | -7.12 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | export_non_terminal | 2.12MiB | 1.32MiB | -37.60 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | export_terminal | 4.08MiB | 2.25MiB | -44.82 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_nothing | 3.53KiB | 3.53KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_unsigned | 3.71KiB | 3.71KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_signed_validation | 3.71KiB | 3.71KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | manage | 0B | 0B | Unchanged | +-----------------------------------------+--------------------------------------+---------+---------+-----------------+ #### new: polkadot/pallet_election_provider_multi_block_signed.rs old: kusama +------------------------------------------------+----------------------+----------+----------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +===============================================================================================================+ | pallet_election_provider_multi_block_signed.rs | bail | 43.61KiB | 82.74KiB | +89.72 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | register_eject | 46.54KiB | 85.80KiB | +84.35 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | clear_old_round_data | 85.23KiB | 85.17KiB | -0.06 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | submit_page | 6.95KiB | 6.90KiB | -0.70 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | register_not_full | 6.45KiB | 6.39KiB | -1.00 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | unset_page | 20.76KiB | 18.55KiB | -10.67 | +------------------------------------------------+----------------------+----------+----------+-----------------+ #### new: polkadot/pallet_election_provider_multi_block_unsigned.rs old: kusama +--------------------------------------------------+-------------------+----------+-----------+------------------+ | File | Extrinsic | Old | New | Change [%] | +================================================================================================================+ | pallet_election_provider_multi_block_unsigned.rs | submit_unsigned | 63.56KiB | 696.00KiB | +995.01 | |--------------------------------------------------+-------------------+----------+-----------+------------------| | pallet_election_provider_multi_block_unsigned.rs | validate_unsigned | 1.81KiB | 3.66KiB | +102.65 | +--------------------------------------------------+-------------------+----------+-----------+------------------+ #### new: polkadot/pallet_election_provider_multi_block_verifier.rs old: kusama +--------------------------------------------------+------------------------------------+-----------+-----------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +=================================================================================================================================+ | pallet_election_provider_multi_block_verifier.rs | on_initialize_invalid_terminal | 1.18MiB | 1.69MiB | +42.87 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_valid_terminal | 1.18MiB | 1.69MiB | +42.71 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_invalid_non_terminal | 1.30MiB | 450.82KiB | -66.08 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_valid_non_terminal | 279.93KiB | 62.22KiB | -77.77 | +--------------------------------------------------+------------------------------------+-----------+-----------+-----------------+ ``` <summary> note for PR authors <details> <br> ## TODO - [x] Finalize weights - [x] Lock voter list when snapshot being taken - [x] push based election - [x] OffchainWorker miner can now run on multiple pages - [x] Trimming is improved, all bounds are respected. - [x] clients pallets: add ID - [x] make election prolonged - [x] bring westend-next and ah-next to staking-next - [x] Test pre-migration to post-migration state in ahm-test. - [x] Offence reporting works without exposure info on RC (done but recheck). - [ ] staking-async fix tests - [ ] root offence testing (minimally done in migration test) - [ ] Run benchmarking - [x] ~~Add custom decoder for OffenceDetails~~. ## TODO before finalizing PR - [x] Go over again and ensure no interaction with staking-classic except by AhClient (and pallets that are going away) in Westend. Make any non used apis private. - [ ] Create diff with changes from staking-classic. ## Migration Notes - At the start of the AHM migration, trigger: `RC::pallet_staking_async_ah_client::on_migration_start()` - At the start of the AHM migration, trigger the following: - definitely filter `staking::bond` - RC: set `staking::Forcing` to `ForceNone`. - At the end of the AHM migration, trigger the following - `RC::pallet_staking_async_ah_client::on_migration_end()` - Set `AH::pallet_staking_async::ForceEra` to `Forcing::NotForcing`. - Set RC staking and pool min bond to be u32::max. ## Follow-up - [ ] Offence generation e2e test (zombienet) </details> </summary> --------- Co-authored-by: kianenigma <kian@parity.io> Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
github-merge-queue bot
pushed a commit
that referenced
this pull request
Apr 22, 2025
Moved from: #7601. Follow ups to: #7282. Closes: #8146 --- This PR is the final outcome of a multi-month development period, with a lot of background work since 2022. Its main aim is to make pallet-staking, alongside its `type ElectionProvider` compatible to be used in a parachain, and report back the validator set to a relay-chain. This setup is intended to be used for Polkadot, Kusama and Westend relay-chains, with the corresponding AssetHubs hosting the staking system. While this PR is quite big, a lot of the diffs are due to adding a relay and parachain runtime for testing. The following is a guide to help reviewers/auditors distinguish what has actually changed in this PR. > Additional reading: See polkadot-js/apps#11401, and the hackmd shared in there, which contains more in-depth explanation of how RC <> AH communicate. ## Added > [This shows the partial diff](ankn/diff-staking-async...ankn/diff-staking-async-1) introduced in pallet-staking-async and election-provider-multi-block relative to the existing (in master) pallet-staking and election-provider-multi-phase. This PR adds the following new pallets, all of which are not used anywhere yet, with the exception of one (see `westend-runtime` changes below). #### `pallet-election-provider-multi-block` This is a set of 4 pallets, capable of implementing an async, multi-page `ElectionProvider`. This pallet is not used in any real runtime yet, and is intended to be used in `AssetHub`, next to `pallet-staking-async`. #### `pallet-staking-async` A fork of the old `pallet-staking`, with a number of key differences, making it suitable to be used in a parachain: 1. It no longer has access to a secure timestamp, previously used to calculate the duration of an era. 2. It no longer has access to a `pallet-session`. 3. It no longer has access to a `pallet-authorship`. 5. It is capable of working with a multi-page `ElectionProvider`, aka. `pallet-election-provider-multi-block`. To compensate for the above, this pallet relies on XCM messages coming from the relay-chain, informing the pallet of: * When a new era should be activated, and how long its duration was * When an offence has happened on the relay relay-chain * When a session ends on the relay-chain, and how many reward points were accumulated for each validators during that period. #### `pallet-staking-async-ah-client` and `pallet-staking-async-rc-client` Are the two new pallets that facilitate the above communication. #### `pallet-ahm-test` A test-only crate that contains e2e rust-based unit test for all of the above. #### `pallet-staking-async-rc-runtime` and `pallet-staking-async-parachain-runtime` Forks of westend and westend-asset-hub, customized to be used for testing all of the above with Zombienet. It contains a lot of unrelated code as well. ## Changed > [This shows the partial diff](ankn/8127-diff-changed-base...ankn/8127-diff-changed-compare) that shows the changes to existing pallets used in prod runtimes as well as westend runtime changes. #### `Identification` This mechanism, which lives on the relay-chain, is expressed by `type FullIdentification` and `type FullIdentificationOf` in runtimes. It is a way to identify the full data needed to slash a validator. Historically, it was pointing to a validator, and their `struct Exposure`. With the move to Asset-Hub, this is no longer possible for two reasons: 1. Relay chain no longer knows the full exposures 2. Even if, the full exposures are getting bigger and bigger and relying the entirety of it is not scalable. Instead, runtimes now move to a new `type FullIdentificationOf = DefaultExposureOf`, which will identify a validator with a `Exposure::default()`. This is suboptimal, as it forces us to still store a number of bytes. Yet, it allows any old `FullIdentification`, pertaining to an old slash, to be decoded. This compromise is only needed to cater for slashes that happen around the time of AHM. #### `westend-runtime` This runtime already has the `pallet-staking-async-ah-client`, integrated into all the places such that: 1. It handles the validator reward points 2. It handles offences 5. It is the `SessionManager` Yet, it is delegating all of the above to its `type Fallback`, which is the old `pallet-staking`. This is a preparatory step for AHM, and should not be any logical change. #### `pallet-election-provider-multi-phase` This is the old single-page `ElectionProvider`. It has been updated to work with multi-page traits, yet it only supports `page-size = 1` for now. It should not have seen any logical changes. #### `pallet-bags-list` Now has two new features. 1. It can be `Locked`, in which case all updates to it fail with an `Err(_)`, even deletion of a node. This is needed because we cannot alter any nodes in this pallet during a multi-page iteration, aka. multi-page snapshot. 2. To combat this, the same `rebag` transaction can be also be used to remove a node from the list, or add a node to the list. This is done through the `score_of` api. See the file changes and tests under `./substrate/frame/bags-list` for more info. #### RuntimeDebug -> Debug To facilitate debugging, a number of types' `RuntimeDebug` impl has been changed to `Debug`. See #3107 ## Weights Below is a summary of the weights. These are generated using `staking-async/runtimes/parachain`, which assumes 22_500 nominators divided by `32` pages for Polkadot, and 12_500 nominators divided by `16` pages in Kusama, both leading to ~700 nominators snapshotted and exported per page. Doubling these parameters would easily slash the PoV weights by half, but with 10MB PoV, these numbers should be good. Also noting that with PoV clawback, we migth get even more proof_size weight back in the runtime. Although, afaik this reclaimed value does not take compression into account. ``` #### new: polkadot/pallet_election_provider_multi_block.rs old: kusama +-----------------------------------------+--------------------------------------+---------+---------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +======================================================================================================================+ | pallet_election_provider_multi_block.rs | on_initialize_into_snapshot_msp | 2.41MiB | 2.41MiB | -0.03 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_snapshot_rest | 3.24MiB | 3.06MiB | -5.53 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_signed | 3.36MiB | 3.12MiB | -7.12 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | export_non_terminal | 2.12MiB | 1.32MiB | -37.60 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | export_terminal | 4.08MiB | 2.25MiB | -44.82 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_nothing | 3.53KiB | 3.53KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_unsigned | 3.71KiB | 3.71KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_signed_validation | 3.71KiB | 3.71KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | manage | 0B | 0B | Unchanged | +-----------------------------------------+--------------------------------------+---------+---------+-----------------+ #### new: polkadot/pallet_election_provider_multi_block_signed.rs old: kusama +------------------------------------------------+----------------------+----------+----------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +===============================================================================================================+ | pallet_election_provider_multi_block_signed.rs | bail | 43.61KiB | 82.74KiB | +89.72 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | register_eject | 46.54KiB | 85.80KiB | +84.35 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | clear_old_round_data | 85.23KiB | 85.17KiB | -0.06 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | submit_page | 6.95KiB | 6.90KiB | -0.70 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | register_not_full | 6.45KiB | 6.39KiB | -1.00 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | unset_page | 20.76KiB | 18.55KiB | -10.67 | +------------------------------------------------+----------------------+----------+----------+-----------------+ #### new: polkadot/pallet_election_provider_multi_block_unsigned.rs old: kusama +--------------------------------------------------+-------------------+----------+-----------+------------------+ | File | Extrinsic | Old | New | Change [%] | +================================================================================================================+ | pallet_election_provider_multi_block_unsigned.rs | submit_unsigned | 63.56KiB | 696.00KiB | +995.01 | |--------------------------------------------------+-------------------+----------+-----------+------------------| | pallet_election_provider_multi_block_unsigned.rs | validate_unsigned | 1.81KiB | 3.66KiB | +102.65 | +--------------------------------------------------+-------------------+----------+-----------+------------------+ #### new: polkadot/pallet_election_provider_multi_block_verifier.rs old: kusama +--------------------------------------------------+------------------------------------+-----------+-----------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +=================================================================================================================================+ | pallet_election_provider_multi_block_verifier.rs | on_initialize_invalid_terminal | 1.18MiB | 1.69MiB | +42.87 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_valid_terminal | 1.18MiB | 1.69MiB | +42.71 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_invalid_non_terminal | 1.30MiB | 450.82KiB | -66.08 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_valid_non_terminal | 279.93KiB | 62.22KiB | -77.77 | +--------------------------------------------------+------------------------------------+-----------+-----------+-----------------+ ``` <summary> note for PR authors <details> <br> ## TODO - [x] Finalize weights - [x] Lock voter list when snapshot being taken - [x] push based election - [x] OffchainWorker miner can now run on multiple pages - [x] Trimming is improved, all bounds are respected. - [x] clients pallets: add ID - [x] make election prolonged - [x] bring westend-next and ah-next to staking-next - [x] Test pre-migration to post-migration state in ahm-test. - [x] Offence reporting works without exposure info on RC (done but recheck). - [ ] staking-async fix tests - [ ] root offence testing (minimally done in migration test) - [ ] Run benchmarking - [x] ~~Add custom decoder for OffenceDetails~~. ## TODO before finalizing PR - [x] Go over again and ensure no interaction with staking-classic except by AhClient (and pallets that are going away) in Westend. Make any non used apis private. - [ ] Create diff with changes from staking-classic. ## Migration Notes - At the start of the AHM migration, trigger: `RC::pallet_staking_async_ah_client::on_migration_start()` - At the start of the AHM migration, trigger the following: - definitely filter `staking::bond` - RC: set `staking::Forcing` to `ForceNone`. - At the end of the AHM migration, trigger the following - `RC::pallet_staking_async_ah_client::on_migration_end()` - Set `AH::pallet_staking_async::ForceEra` to `Forcing::NotForcing`. - Set RC staking and pool min bond to be u32::max. ## Follow-up - [ ] Offence generation e2e test (zombienet) </details> </summary> --------- Co-authored-by: kianenigma <kian@parity.io> Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
castillax
pushed a commit
that referenced
this pull request
May 12, 2025
Moved from: #7601. Follow ups to: #7282. Closes: #8146 --- This PR is the final outcome of a multi-month development period, with a lot of background work since 2022. Its main aim is to make pallet-staking, alongside its `type ElectionProvider` compatible to be used in a parachain, and report back the validator set to a relay-chain. This setup is intended to be used for Polkadot, Kusama and Westend relay-chains, with the corresponding AssetHubs hosting the staking system. While this PR is quite big, a lot of the diffs are due to adding a relay and parachain runtime for testing. The following is a guide to help reviewers/auditors distinguish what has actually changed in this PR. > Additional reading: See polkadot-js/apps#11401, and the hackmd shared in there, which contains more in-depth explanation of how RC <> AH communicate. ## Added > [This shows the partial diff](ankn/diff-staking-async...ankn/diff-staking-async-1) introduced in pallet-staking-async and election-provider-multi-block relative to the existing (in master) pallet-staking and election-provider-multi-phase. This PR adds the following new pallets, all of which are not used anywhere yet, with the exception of one (see `westend-runtime` changes below). #### `pallet-election-provider-multi-block` This is a set of 4 pallets, capable of implementing an async, multi-page `ElectionProvider`. This pallet is not used in any real runtime yet, and is intended to be used in `AssetHub`, next to `pallet-staking-async`. #### `pallet-staking-async` A fork of the old `pallet-staking`, with a number of key differences, making it suitable to be used in a parachain: 1. It no longer has access to a secure timestamp, previously used to calculate the duration of an era. 2. It no longer has access to a `pallet-session`. 3. It no longer has access to a `pallet-authorship`. 5. It is capable of working with a multi-page `ElectionProvider`, aka. `pallet-election-provider-multi-block`. To compensate for the above, this pallet relies on XCM messages coming from the relay-chain, informing the pallet of: * When a new era should be activated, and how long its duration was * When an offence has happened on the relay relay-chain * When a session ends on the relay-chain, and how many reward points were accumulated for each validators during that period. #### `pallet-staking-async-ah-client` and `pallet-staking-async-rc-client` Are the two new pallets that facilitate the above communication. #### `pallet-ahm-test` A test-only crate that contains e2e rust-based unit test for all of the above. #### `pallet-staking-async-rc-runtime` and `pallet-staking-async-parachain-runtime` Forks of westend and westend-asset-hub, customized to be used for testing all of the above with Zombienet. It contains a lot of unrelated code as well. ## Changed > [This shows the partial diff](ankn/8127-diff-changed-base...ankn/8127-diff-changed-compare) that shows the changes to existing pallets used in prod runtimes as well as westend runtime changes. #### `Identification` This mechanism, which lives on the relay-chain, is expressed by `type FullIdentification` and `type FullIdentificationOf` in runtimes. It is a way to identify the full data needed to slash a validator. Historically, it was pointing to a validator, and their `struct Exposure`. With the move to Asset-Hub, this is no longer possible for two reasons: 1. Relay chain no longer knows the full exposures 2. Even if, the full exposures are getting bigger and bigger and relying the entirety of it is not scalable. Instead, runtimes now move to a new `type FullIdentificationOf = DefaultExposureOf`, which will identify a validator with a `Exposure::default()`. This is suboptimal, as it forces us to still store a number of bytes. Yet, it allows any old `FullIdentification`, pertaining to an old slash, to be decoded. This compromise is only needed to cater for slashes that happen around the time of AHM. #### `westend-runtime` This runtime already has the `pallet-staking-async-ah-client`, integrated into all the places such that: 1. It handles the validator reward points 2. It handles offences 5. It is the `SessionManager` Yet, it is delegating all of the above to its `type Fallback`, which is the old `pallet-staking`. This is a preparatory step for AHM, and should not be any logical change. #### `pallet-election-provider-multi-phase` This is the old single-page `ElectionProvider`. It has been updated to work with multi-page traits, yet it only supports `page-size = 1` for now. It should not have seen any logical changes. #### `pallet-bags-list` Now has two new features. 1. It can be `Locked`, in which case all updates to it fail with an `Err(_)`, even deletion of a node. This is needed because we cannot alter any nodes in this pallet during a multi-page iteration, aka. multi-page snapshot. 2. To combat this, the same `rebag` transaction can be also be used to remove a node from the list, or add a node to the list. This is done through the `score_of` api. See the file changes and tests under `./substrate/frame/bags-list` for more info. #### RuntimeDebug -> Debug To facilitate debugging, a number of types' `RuntimeDebug` impl has been changed to `Debug`. See #3107 ## Weights Below is a summary of the weights. These are generated using `staking-async/runtimes/parachain`, which assumes 22_500 nominators divided by `32` pages for Polkadot, and 12_500 nominators divided by `16` pages in Kusama, both leading to ~700 nominators snapshotted and exported per page. Doubling these parameters would easily slash the PoV weights by half, but with 10MB PoV, these numbers should be good. Also noting that with PoV clawback, we migth get even more proof_size weight back in the runtime. Although, afaik this reclaimed value does not take compression into account. ``` #### new: polkadot/pallet_election_provider_multi_block.rs old: kusama +-----------------------------------------+--------------------------------------+---------+---------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +======================================================================================================================+ | pallet_election_provider_multi_block.rs | on_initialize_into_snapshot_msp | 2.41MiB | 2.41MiB | -0.03 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_snapshot_rest | 3.24MiB | 3.06MiB | -5.53 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_signed | 3.36MiB | 3.12MiB | -7.12 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | export_non_terminal | 2.12MiB | 1.32MiB | -37.60 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | export_terminal | 4.08MiB | 2.25MiB | -44.82 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_nothing | 3.53KiB | 3.53KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_unsigned | 3.71KiB | 3.71KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_signed_validation | 3.71KiB | 3.71KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | manage | 0B | 0B | Unchanged | +-----------------------------------------+--------------------------------------+---------+---------+-----------------+ #### new: polkadot/pallet_election_provider_multi_block_signed.rs old: kusama +------------------------------------------------+----------------------+----------+----------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +===============================================================================================================+ | pallet_election_provider_multi_block_signed.rs | bail | 43.61KiB | 82.74KiB | +89.72 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | register_eject | 46.54KiB | 85.80KiB | +84.35 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | clear_old_round_data | 85.23KiB | 85.17KiB | -0.06 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | submit_page | 6.95KiB | 6.90KiB | -0.70 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | register_not_full | 6.45KiB | 6.39KiB | -1.00 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | unset_page | 20.76KiB | 18.55KiB | -10.67 | +------------------------------------------------+----------------------+----------+----------+-----------------+ #### new: polkadot/pallet_election_provider_multi_block_unsigned.rs old: kusama +--------------------------------------------------+-------------------+----------+-----------+------------------+ | File | Extrinsic | Old | New | Change [%] | +================================================================================================================+ | pallet_election_provider_multi_block_unsigned.rs | submit_unsigned | 63.56KiB | 696.00KiB | +995.01 | |--------------------------------------------------+-------------------+----------+-----------+------------------| | pallet_election_provider_multi_block_unsigned.rs | validate_unsigned | 1.81KiB | 3.66KiB | +102.65 | +--------------------------------------------------+-------------------+----------+-----------+------------------+ #### new: polkadot/pallet_election_provider_multi_block_verifier.rs old: kusama +--------------------------------------------------+------------------------------------+-----------+-----------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +=================================================================================================================================+ | pallet_election_provider_multi_block_verifier.rs | on_initialize_invalid_terminal | 1.18MiB | 1.69MiB | +42.87 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_valid_terminal | 1.18MiB | 1.69MiB | +42.71 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_invalid_non_terminal | 1.30MiB | 450.82KiB | -66.08 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_valid_non_terminal | 279.93KiB | 62.22KiB | -77.77 | +--------------------------------------------------+------------------------------------+-----------+-----------+-----------------+ ``` <summary> note for PR authors <details> <br> ## TODO - [x] Finalize weights - [x] Lock voter list when snapshot being taken - [x] push based election - [x] OffchainWorker miner can now run on multiple pages - [x] Trimming is improved, all bounds are respected. - [x] clients pallets: add ID - [x] make election prolonged - [x] bring westend-next and ah-next to staking-next - [x] Test pre-migration to post-migration state in ahm-test. - [x] Offence reporting works without exposure info on RC (done but recheck). - [ ] staking-async fix tests - [ ] root offence testing (minimally done in migration test) - [ ] Run benchmarking - [x] ~~Add custom decoder for OffenceDetails~~. ## TODO before finalizing PR - [x] Go over again and ensure no interaction with staking-classic except by AhClient (and pallets that are going away) in Westend. Make any non used apis private. - [ ] Create diff with changes from staking-classic. ## Migration Notes - At the start of the AHM migration, trigger: `RC::pallet_staking_async_ah_client::on_migration_start()` - At the start of the AHM migration, trigger the following: - definitely filter `staking::bond` - RC: set `staking::Forcing` to `ForceNone`. - At the end of the AHM migration, trigger the following - `RC::pallet_staking_async_ah_client::on_migration_end()` - Set `AH::pallet_staking_async::ForceEra` to `Forcing::NotForcing`. - Set RC staking and pool min bond to be u32::max. ## Follow-up - [ ] Offence generation e2e test (zombienet) </details> </summary> --------- Co-authored-by: kianenigma <kian@parity.io> Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
21 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
see #3005
My local results are:
compiled with
cargo build --profile=productionEDIT: I removed the kitchensink as it was a relic from previous runs
RuntimeDebugandRuntimeDebugNoBoundtoDebugRuntimeDebugandRuntimeDebugNoBoundRuntimeDebugandRuntimeDebugNoBoundRuntimeDebugandRuntimeDebugNoBoundreferences on docs and doctests