Memory leak under high bitswap load

[Jorropo]

Checklist

• This is a bug report, not a question. Ask questions on discuss.ipfs.io.
• I have searched on the issue tracker for my bug.
• I am running the latest go-ipfs version or have an issue updating.

Installation method

built from source

Version

go-ipfs version: 0.13.0-dev
Repo version: 12
System version: amd64/linux
Golang version: devel go1.19-4e26ab0ed8 Tue Mar 15 17:29:54 2022 +0000

Config

N/A

Description

Bitswap is a big memory hog when you start having 10k+ wantlists.

I belive this is because it use a map for things like the wantlist, ...
This is sad because due to the way we use the maps there, we quickly reach a high level of fragmentation and the buckets wont get freed, taking lots of space for nothing.
Linked: golang/go#20135

Edit: I don't know how I missed that, but if I understand correctly go-bitswap creates a copy of the wantlist per partner (the map shrinking stuff might still be there, but have far less of an impact than I first thought (likely the same small impact I mesure in synthetic tests)).
10MiB times 600 easly explains such memory usage.

Profile done with ~50k blocks in the wantlist.

Profile: https://jorropo.net/ipfs/Qmf5J2dyvpBgc6RoKDUTaSzgsSVYqdCJFnFgb5puFRoXEY/profile.zip

FYI a synthetically created wantlist, need tens to hundred millions of elements to reach thoses sizes.

Reproduce

In go-ipfs

Spam the gateway with lots of different files to download concurrently

Everything is in the name, just find a way to request many blocks

Use the #8795 branch

Right now (d086aef) it doesn't rate limit, so you can ipfs dag stat -s=false Qmfoo where Qmfoo is any big dag (tens of gigs in my experience).

Tip, use watch ipfs stats bitswap to ensure your wantlist don't go past 100k, you likely don't have enough ram for that.

Solutions

• From time to time, make a new map and copy the data from the old one into the new one.
• Use a custom map implementation that aggressively chunk
• Refactor the map out (have I talked about rewriting the bitswap client into a workstealing meta / overlay scheduler yet ?)
• ... ?

[guseggert]

This is a well-known problem with bitswap, we experience this often on the gateways, here is the go-bitswap tracking issue: ipfs/boxo#79

[BigLep]

@Jorropo : yeah, this definitely needs to get solved. It's on the roadmap to get to "soon". Help is likely going to be needed here, at least on reviewing.

I'm not sure your bitswap knowledge, but maybe consider reviewing the updated spec (which needs review anyways) ipfs/specs#269 to ramp up to be able to help more here?

[BigLep]

We're going to close and let ipfs/boxo#79 be the master issue.