v3.1: runtime: Collect stake delegations only once during epoch activation (backport of #8065)

[mergify]

Problem

Processing new epoch (Bank::process_new_epoch) involves collecting stake delegations twice:

1. In Stakes::activate_epoch, to create a stake history entry and refresh vote accounts.
2. In Bank::filter_stake_delegations, which is then used in Bank::calculate_stake_vote_rewards to calculate rewards for stakers and voters.

The overall time of crossing the epoch boundary is ~519ms:

update_epoch_us=519953i

Where the two heaviest operations are collect() calls on stake delegations, each of them taking ~200-220ms:

Summary of Changes

Reduce that to just one collect to a Vec<(&Pubkey, &StakeAccount)> done on the beginning of Bank::process_new_epoch and passing the stake delegations to the other methods.

The new time of crossing the epoch boundary is ~337ms:

update_epoch_us=337371i

There is only one heavy collect() done on stake delegations, which still takes the most of main thread's time. But that's the best we can do while still using im::HashMap.

Making that change possible required several refactors:

• Tale &PointValue in Bank::create_epoch_rewards_sysvar. That makes it easier to operate on references of PartitionedRewardsCalculation. Copying integers from PointValue is cheap and has no visible
  performance impact.
• Split Stakes::activate_epoch, that was performing calculations and mutating the cache at the same time. The calculations got split to Stakes::calculate_activated_stake that takes &self.
• Add Stakes::stake_delegations_ves method. Stake delegations are stored as hash array mapped trie (HAMT)[0], which means that inserts, deletions and lookups are average-case O(1) and worst-case O(log n). However, the performance of iterations is poor due to depth-first traversal and jumps. Currently it's also impossible to iterate over it with rayon. That issue is known and handled by converting the HAMT to a vector with stakes.stake_delegations.iter().collect(). Move that trick to a dedicated method that describes the performance consequences.
• Add FilteredStakeDelegation wrapper type, that wraps a vector of stake delegations and acts as a lazy iterator that filters out ones with insufficient stake.
• Split the code dealing with rewards calculation and vote rewards distribution into separate methods:
  • Bank::calculate_rewards that takes &self and does not acquire any locks.
  • Bank::begin_partitioned_rewards that takes &mut self, sets calculation status and creates a sysvar.
  • Bank::distribute_vote_rewards that stores partitioned rewards and increases capitalization.

[0] https://en.wikipedia.org/wiki/Hash_array_mapped_trie

Fixes: #8282

---

This is an automatic backport of pull request #8065 done by [Mergify](https://mergify.com).

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.2%. Comparing base (c103244) to head (79d115e).
⚠️ Report is 1 commits behind head on v3.1.

Additional details and impacted files

@@           Coverage Diff            @@
##             v3.1    #9321    +/-   ##
========================================
  Coverage    83.2%    83.2%            
========================================
  Files         865      865            
  Lines      375603   375938   +335     
========================================
+ Hits       312632   313003   +371     
+ Misses      62971    62935    -36     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.