Lower default connection limits (Swarm.ConnMgr)

[lidel]

The default limits defined in Swarm.ConnMgr are very old, come from the time when network was very small and DHT was not the best (<0.5), and we had no QUIC transports.

We keep hundreds of connections around:

kubo/config/init.go

Lines 97 to 103 in e34c0da

	// DefaultConnMgrHighWater is the default value for the connection managers
	// 'high water' mark
	const DefaultConnMgrHighWater = 900
	// DefaultConnMgrLowWater is the default value for the connection managers 'low
	// water' mark
	const DefaultConnMgrLowWater = 600

Times changed, and these are, arguably, no longer good defaults. Keeping hundreds of connections around introduces various problems such as ambient bandwidth cost due to bitswap gossip. Every peer asking all peers it is connected to if they have every CID it is tryting to fetch

Proposed change

Lower the implicit defaults for Swarm.ConnMgr.LowWater and Swarm.ConnMgr.HighWater.

What should be the new values?

Brave and IPFS Desktop both run with lowered limits:

Swarm.ConnMgr.LowWater = 20
Swarm.ConnMgr.HighWater = 40

Unless there is a good reason to keep these connections around, I would set default to the same values.

TODO

• decide on new limits (use 20 / 40 dogfooded in IPFS Desktop in no comments)
• change implicit defaults in config/init.go
• create config migration for legacy nodes
  • we've recently removed defaults from user config (IPFS config: lower the ConnMgr.HighWater and LowWater limits brave/brave-browser#13525) but preexisting nodes have values hardcoded in the config
  • we should run one-time migration that inspects Swarm.ConnMgr and removes all fields from the object if they all match legacy defaults:

    {
      "GracePeriod": "20s",
      "HighWater": 900,
      "LowWater": 600,
      "Type": "basic"
    }

    This way old nodes will benefit from less chatty bitswap too.

[BigLep]

2022-12-06 conversation:

• We're going to take this on to help with [Tracking issue] go-libp2p resource manager critical post release fixes #9442
• We'd ideally like to have these values based on Swarm.ResourceMgr.System.ConnsInbound (which is usually a calculated value)
• For this release though we're going to do 50/100.