Add storage bounds for pallet `staking` and clean up deprecated non paged exposure storages by re-gius · Pull Request #6445 · paritytech/polkadot-sdk

re-gius · 2024-11-11T15:08:49Z

This is part of #6289 and necessary for the Asset Hub migration.

Building on the observations and suggestions from #255 .

Changes

Add MaxInvulnerables to bound Invulnerables Vec -> BoundedVec.
- Set to constant 20 in the pallet (must be >= 17 for backward compatibility of runtime westend).
Add MaxDisabledValidators to bound DisabledValidators Vec -> BoundedVec
- Set to constant 100 in the pallet (it should be <= 1/3 * MaxValidatorsCount according to the current disabling strategy)
Remove ErasStakers and ErasStakersClipped (see Tracker issue for cleaning up old non-paged exposure logic in staking pallet #433 )
- They were deprecated in v14 and could have been removed since staking era 1504 (now it's > 1600)
- Completing the task from [pallet-staking] Clean up stale non paged exposure storages in the pallet #5986
Use MaxExposurePageSize to bound ErasStakersPaged mapping to exposure pages: each ExposurePage.others Vec is turned into a WeakBoundedVec to allow easy and quick changes to this bound
Add MaxBondedEras to bound BondedEras Vec -> BoundedVec
- Set to BondingDuration::get() + 1 everywhere to include both time interval endpoints in [current_era - BondingDuration::get(), current_era]. Notice that this was done manually in every test and runtime, so I wonder if there is a better way to ensure that MaxBondedEras::get() == BondingDuration::get() + 1 everywhere.
Add MaxRewardPagesPerValidator to bound ClaimedRewards Vec of pages -> WeakBoundedVec
- Set to constant 20 in the pallet. The vector of pages is now a WeakBoundedVec to allow easy and quick changes to this parameter
Remove MaxValidatorsCount optional storage item to add MaxValidatorsCount mandatory config parameter
- Using it to to bound EraRewardPoints.individual BTreeMap -> BoundedBTreeMap;
- Set to dynamic parameter in runtime westend so that changing it should not require migrations for it

TO DO
Slashing storage items will be bounded in another PR.

UnappliedSlashes
SlashingSpans

substrate/frame/staking/src/pallet/mod.rs

substrate/primitives/staking/src/lib.rs

seadanda · 2025-01-16T16:50:41Z

polkadot/runtime/test-runtime/src/lib.rs

 	type MaxInvulnerables = ConstU32<20>;
 	type MaxRewardPagesPerValidator = ConstU32<20>;
-	type MaxValidatorsCount = ConstU32<300>;
+	type MaxValidatorsCount = MaxAuthorities;


This is a huge jump, is there a reason why the MaxAuthorities is so big in the test runtime?

Oh, right! Then it's probably better to revert the change in that case, adding also a comment: 6c9806c

gui1117

Overall looks good, but there is still some comments to resolve.

gui1117 · 2025-01-17T05:31:31Z

substrate/frame/staking/src/lib.rs

+	pub fn from_clipped(exposure: Exposure<AccountId, Balance>) -> Result<Self, ()> {
+		let old_exposures = exposure.others.len();
+		let others = WeakBoundedVec::try_from(exposure.others).unwrap_or_default();
+		defensive_assert!(old_exposures == others.len(), "Too many exposures for a page");


This function is not used, we can make it better or remove it.

Suggested change

pub fn from_clipped(exposure: Exposure<AccountId, Balance>) -> Result<Self, ()> {

let old_exposures = exposure.others.len();

let others = WeakBoundedVec::try_from(exposure.others).unwrap_or_default();

defensive_assert!(old_exposures == others.len(), "Too many exposures for a page");

pub fn try_from_clipped(exposure: Exposure<AccountId, Balance>) -> Result<Self, ()> {

let others = WeakBoundedVec::try_from(exposure.others).map_err(|_| ())?;

Changed in 3b7bc95

substrate/frame/staking/src/lib.rs

gui1117 · 2025-01-17T05:39:36Z

substrate/frame/staking/src/lib.rs

-		claimed_pages.push(page);
+		// try to add page to claimed entries
+		if claimed_pages.try_push(page).is_err() {
+			defensive!("Limit reached for maximum number of pages.");


The proof should be more precise. In what circumstance can this limit be reached, why is it impossible in practice.

What do you mean by "the proof"?

gui1117 · 2025-01-17T05:54:16Z

substrate/frame/staking/src/migrations.rs

+			let mut eras_stakers_keys =
+				v16::ErasStakers::<T>::iter_keys().map(|(k1, _k2)| k1).collect::<Vec<_>>();
+			eras_stakers_keys.dedup();
+			for k in eras_stakers_keys {
+				let mut removal_result =
+					v16::ErasStakers::<T>::clear_prefix(k, u32::max_value(), None);
+				while let Some(next_cursor) = removal_result.maybe_cursor {
+					removal_result = v16::ErasStakers::<T>::clear_prefix(
+						k,
+						u32::max_value(),
+						Some(&next_cursor[..]),
+					);
+				}
+			}


This seems to try to remove all keys in one time, if we don't need multi-block migration and we are sure it is ok then we can do:

Suggested change

let mut eras_stakers_keys =

v16::ErasStakers::<T>::iter_keys().map(|(k1, _k2)| k1).collect::<Vec<_>>();

eras_stakers_keys.dedup();

for k in eras_stakers_keys {

let mut removal_result =

v16::ErasStakers::<T>::clear_prefix(k, u32::max_value(), None);

while let Some(next_cursor) = removal_result.maybe_cursor {

removal_result = v16::ErasStakers::<T>::clear_prefix(

k,

u32::max_value(),

Some(&next_cursor[..]),

);

}

}

v16::ErasStakers::<T>::clear(u32::max_value(), None);

We can be sure for Polkadot or Kusama but how can we be sure for every other chain? Maybe there are some limits I'm not aware of

These storage items are already empty in Polkadot and Kusama (you can validate in an UI), and this pallet is not parachian-ready, so it should be fine.

gui1117 · 2025-01-17T05:54:48Z

substrate/frame/staking/src/migrations.rs

+			let mut eras_stakers_clipped_keys = v16::ErasStakersClipped::<T>::iter_keys()
+				.map(|(k1, _k2)| k1)
+				.collect::<Vec<_>>();
+			eras_stakers_clipped_keys.dedup();
+			for k in eras_stakers_clipped_keys {
+				let mut removal_result =
+					v16::ErasStakersClipped::<T>::clear_prefix(k, u32::max_value(), None);
+				while let Some(next_cursor) = removal_result.maybe_cursor {
+					removal_result = v16::ErasStakersClipped::<T>::clear_prefix(
+						k,
+						u32::max_value(),
+						Some(&next_cursor[..]),
+					);
+				}
+			}


Suggested change

let mut eras_stakers_clipped_keys = v16::ErasStakersClipped::<T>::iter_keys()

.map(|(k1, _k2)| k1)

.collect::<Vec<_>>();

eras_stakers_clipped_keys.dedup();

for k in eras_stakers_clipped_keys {

let mut removal_result =

v16::ErasStakersClipped::<T>::clear_prefix(k, u32::max_value(), None);

while let Some(next_cursor) = removal_result.maybe_cursor {

removal_result = v16::ErasStakersClipped::<T>::clear_prefix(

k,

u32::max_value(),

Some(&next_cursor[..]),

);

}

}

v16::ErasStakersClipped::<T>::clear(u32::max_value(), None);

Same here, are we assuming that this is safe?

gui1117 · 2025-01-17T05:58:20Z

substrate/frame/staking/src/migrations.rs

+			} else {
+				log!(info, "v17 applied successfully.");
+			}
+			T::DbWeight::get().reads_writes(1, 1)


we can count the number of operation as we do them.

How can we do that?

As you iterate and remove/set any storage item, bump a counter (let mut x = 0, y) in this code, and use that as the final reads_writes(x, y)

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

paritytech-workflow-stopper · 2025-01-21T17:04:18Z

All GitHub workflows were cancelled due to failure one of the required jobs.
Failed workflow url: https://github.com/paritytech/polkadot-sdk/actions/runs/12891905705
Failed job name: fmt

kianenigma · 2025-01-27T15:19:03Z

@re-gius can you please target #7282, and once the pallet compiles incl benchmarks, merge it over?

kianenigma · 2025-01-27T15:26:40Z

substrate/frame/staking/src/pallet/mod.rs

-	///
-	/// When this value is not set, no limits are enforced.
-	#[pallet::storage]
-	pub type MaxValidatorsCount<T> = StorageValue<_, u32, OptionQuery>;


Why is this removed? it should not be. This counter is still used, and is a moving target for maximum number of validators that the system can have.

In my branch, there is a static target declared as the maximum number of validators:

type MaxValidatorSet

And MaxValidatorsCount should still be dynamic, but always less thanMaxValidatorsSet

The names suck, I know :(

re-gius · 2025-02-05T15:24:25Z

This PR is in the process of being closed to be re-built as #7483 on top of #7282

… exposures (#7483) Building from #6445 on top of #7282 **Changes** - [x] Bound `Invulnerables`, vector of validators invulnerable to slashing. - Add `MaxInvulnerables` to bound `Invulnerables` Vec -> `BoundedVec`. - Set to constant 20 in the pallet (must be >= 17 for backward compatibility with runtime `westend`). - [x] Bound `Disabled Validators`, vector of validators that have offended in a given era and have been disabled. - Add `MaxDisabledValidators` to bound `DisabledValidators` Vec -> `BoundedVec`. - Set to constant 100 in the pallet (it should be <= 1/3 * `MaxValidatorsCount` according to the current disabling strategy). - [x] Remove `ErasStakers` and `ErasStakersClipped` (see #433 ), non-paged validators exposures. - They were deprecated in v14 and could have been removed since staking era 1504 (now it's > 1700). - They are already empty on Polkadot and Kusama. - Completing the task from #5986. Migrating pallet `staking` storage to v17 to apply all changes. **TO DO** (in a follow-up PR) - [ ] Bound `ErasStakersPaged` - this needs bounding `ExposurePage.others` vector - [ ] Bound `BondedEras` vector - [ ] Bound `ClaimedRewards` pages vector - [ ] Bound `ErasRewardPoints` - this needs bounding `EraRewardPoints.individual` BTreeMap - [ ] Bound `UnappliedSlashes` - [ ] Bound `SlashingSpans` - this needs bounding `SlashingSpans.prior` vector --------- Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: kianenigma <kian@parity.io>

add MaxInvulnerables

6c808b6

re-gius added the T2-pallets This PR/Issue is related to a particular pallet. label Nov 11, 2024

re-gius self-assigned this Nov 11, 2024

This was referenced Nov 11, 2024

[pallet-staking] Bound all storage #5987

Closed

Tracking issue for bounding pallet-staking storage items #255

Closed

[tracking] remove without_storage_info for pallets migrating to AH #6289

Closed

re-gius added 23 commits November 13, 2024 14:51

add MaxActiveValidators

1b18bb1

Merge branch 'master' into re-gius/staking/add-storage-bounds

0ac60af

fix + start working on migrations

4edb8e1

nit (typo)

65c7de6

fix migrations for BoundedVec

35069c9

Merge branch 'master' into re-gius/staking/add-storage-bounds

d4b15af

Migrate ErasRewardPoints

eb93a70

small fixes - BoundedBTreeMap

58cf45d

Merge branch 'master' into re-gius/staking/add-storage-bounds

292c5d0

fix

74f2055

Merge branch 'master' into re-gius/staking/add-storage-bounds

63f2c5e

Merge branch 'master' into re-gius/staking/add-storage-bounds

b249a67

Adapt to new recent DisabledValidators change

362f505

Change MaxActiveValidators to MaxWinners

4f21d53

undo change from another PR

e258baf

Merge branch 'master' into re-gius/staking/add-storage-bounds

4b7d896

small fix for MaxWinners

01e2233

small fixes

243bec8

Merge branch 'master' into re-gius/staking/add-storage-bounds

a599afc

Remove deprecated ErasStakers and ErasStakersClipped

9a25fbc

Fixes from #5986

98daabc

nit

aec5087

remove unused import

36204da

paritytech deleted a comment from paritytech-workflow-stopper bot Nov 25, 2024

re-gius added 4 commits December 20, 2024 13:21

fix comment

c51d09b

fix

1a9be6b

Update pr_6445.prdoc from semver CI

235a260

Merge branch 'master' into re-gius/staking/add-storage-bounds

83a888b

paritytech deleted a comment from paritytech-workflow-stopper bot Jan 2, 2025

Merge branch 'master' into re-gius/staking/add-storage-bounds

b2efa18

gui1117 reviewed Jan 16, 2025

View reviewed changes

substrate/frame/staking/src/pallet/mod.rs Show resolved Hide resolved

substrate/frame/staking/src/pallet/mod.rs Outdated Show resolved Hide resolved

substrate/primitives/staking/src/lib.rs Show resolved Hide resolved

seadanda reviewed Jan 16, 2025

View reviewed changes

gui1117 reviewed Jan 17, 2025

View reviewed changes

Merge branch 'master' into re-gius/staking/add-storage-bounds

78539f0

paritytech-review-bot bot requested a review from a team January 21, 2025 16:22

re-gius and others added 8 commits January 21, 2025 17:29

add more info on deleted storage items to prdoc

1a39de4

add error when processing non-paged exposures

3b7bc95

Update substrate/frame/staking/src/pallet/mod.rs

dc93f0f

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

Update substrate/primitives/staking/src/lib.rs

e6c9942

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

Update substrate/frame/staking/src/lib.rs

d91414b

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

Update substrate/frame/staking/src/lib.rs

9d2e835

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

Update substrate/frame/staking/src/lib.rs

88d20a3

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

fmt

649b0f9

re-gius added 2 commits January 21, 2025 18:07

Change MaxValidatorsCount in test runtime

6c9806c

Merge branch 'master' into re-gius/staking/add-storage-bounds

6ca4347

kianenigma reviewed Jan 27, 2025

View reviewed changes

kianenigma mentioned this pull request Jan 28, 2025

[AHM] Multi-block staking election pallet #7282

Merged

26 tasks

re-gius mentioned this pull request Feb 5, 2025

Bound some storage items for pallet staking and clean up deprecated exposures #7483

Merged

9 tasks

kianenigma closed this Feb 12, 2025

re-gius deleted the re-gius/staking/add-storage-bounds branch February 17, 2025 12:08

Conversation

re-gius commented Nov 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gui1117 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

paritytech-workflow-stopper bot commented Jan 21, 2025

Uh oh!

kianenigma commented Jan 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

re-gius commented Feb 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

re-gius commented Nov 11, 2024 •

edited

Loading