Tweak open connection limits for unstaked connections

[lijunwangs]

Problem

There are only 500 unstaked connection while we allow per address 8 concurrent connections. This leads to large number of connections from unstaked clients to contend for 500/8 for about 62 slots and cause churning.s

Summary of Changes

Reduce the per-connection from unstaked clients to 4 per ip address
Increase total connection count for unstaked connections to 2000.
Classifying eviction metrics between staked vs non-staked.

Fixes #

[codecov-commenter]

Codecov Report

❌ Patch coverage is 87.09677% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.0%. Comparing base (7015e9c) to head (6c35787).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #6953   +/-   ##
=======================================
  Coverage    83.0%    83.0%           
=======================================
  Files         827      827           
  Lines      362695   362694    -1     
=======================================
+ Hits       301233   301249   +16     
+ Misses      61462    61445   -17     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[lijunwangs]

Memory usage: observed using the bench-vote tools.

Connections:
32: 54.828 (MB)

500： 241.844 (MB)

2000: 645.684 (MB)

[alexpyattaev]

Memory usage: observed using the bench-vote tools.

Connections: 32: 54.828 (MB)

500： 241.844 (MB)

2000: 645.684 (MB)

This is for how much RX window and max streams per connection?

[alexpyattaev]

Overall the direction is good, but I think this needs to get narrowed down to only address the unstaked connection limit, rather than everything at once. Especially we must avoid changes to SWQOS quotas.

[alexpyattaev]

We really should not be changing this here. This is completely unrelated change and will not have the impact you intend.

[lijunwangs]

We are using the connection count to distribute the streams/s to different connections out of the total streams/s allowed. With increased connection count quadrupled, the per connection TPS for unstaked, can be reduced to 1/4. The increased percentage reduces that effect.

Our allocation of staked vs. unstaked does not reflect the real usage in MB:

[alexpyattaev]

Yes, it does, but it also cuts massively into the staked TPU bandwidth allocations, reducing them. Not sure that is desirable.

[lijunwangs]

The staked is not using the allocated bandwidth. See the max. Even 50% --> 250K/s should be more than enough for them to use realistically.

[alexpyattaev]

Should this change be in a separate PR?

[t-nelson]

this seems to be making several changes that are unrelated, but on the path to a singular goal. please split this up to isolate them