v3.1: streamer/sigverify: use bounded channels between streamers and sigver (backport of #9732)

[mergify]

Problem

Streamer and sigverify exchange information using unbounded channels.
This leads to unnecessary allocations:

Summary of Changes

Instead, use bounded channel between streamer and sigverify for TPU transactions. Votes and Fwd will be addressed separately. To determine the size of the channel I propose to use metric data. It should be noted that the version 3.1 and earlier do have separate batching stage between sigverify and streamer which is removed in 4.0. In this batching stage, we used bounded channel of size 250k packets. During the high load event 10-10-25, it was never saturated which means that this might be considered as upper bound. From the other hand, there is metric tpu-verifier.max_packets which indicates how many packets we consume at once from the input channel for the sigverify. The peak on the same date was 20k. Hence, I think that 50k is a more reasonable upper bound.

The results in profiler are the following:

---

This is an automatic backport of pull request #9732 done by [Mergify](https://mergify.com).

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.2%. Comparing base (01a77c7) to head (2b623c6).

Additional details and impacted files

@@            Coverage Diff            @@
##             v3.1   #10551     +/-   ##
=========================================
- Coverage    83.2%    83.2%   -0.1%     
=========================================
  Files         865      865             
  Lines      376601   376601             
=========================================
- Hits       313609   313557     -52     
- Misses      62992    63044     +52     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[t-nelson]

please ensure that #10552 goes in before this one to avoid introducing the channel full bug

[KirillLykov]

please ensure that #10552 goes in before this one to avoid introducing the channel full bug

do you mean revert this one from branch v3.1 and add it back?

[steviez]

do you mean revert this one from branch v3.1 and add it back?

No, merges were done in a proper order so no action required (newest commits at top in below output):

$ git status
On branch v3.1
Your branch is up to date with 'anza/v3.1'.
...
$ git log --pretty=oneline
3cbc9d1e24e4c4d603f4c23b71f027ab7cba3d3d (HEAD -> v3.1, anza/v3.1) Bump version to v3.1.10 (#10701)
765ee54adc4f574b1cd4f03a5500bf46c0af0817 (tag: v3.1.9) v3.1: consensus: axe the intermediate accumulation pathway for OC (backport of #10594) (#10697)
ba0d1f8b108f22b0375de1eec9e03ba43575237a v3.1: Switch from `UnsafeCell<Box<[ITEM]>>` to `Box<[UnsafeCell<ITEM>]>` in transaction context (backport of #10524) (#10556)
7018f4aabae3122200322544f6dc6305b658eedd v3.1: accounts-db: Use `SmallRng` for LRU eviction sampling (backport of #10640) (#10644)
e9b827f57b32dae406a1d5a548fcc58801cf87cc v3.1: accounts-db: Reduce read-only cache lock contention (backport of #10641) (#10645)
eed17d618ca17597c0a06c7f3c102447fc2a0d7b v3.1: Sigverify - receive loop up to packet limit (backport of #10428) (#10521)
46a35ec1b6cb9b55c20dcefc4beea25fdce8927f v3.1: streamer/sigverify: use bounded channels between streamers and sigver (backport of #9732) (#10551)
29412570476055a531f51ce8b02dcf7379d29359 v3.1: bug: use try_send in handle_forwarded_packets (backport of #10498) (#10552)
...