Technical deep-dive: Retrofitting "backpressure" in yamux

[thomaseizinger]

tl;dr: Write a technical blogpost on how we configure yamux to retroactively fit "backpressure" on new streams.

Why

I know that the focus is on rolling out QUIC but all of Ethereum for example still runs on yamux and lighthouse is afaik the client with the largest deployment share. From that perspective, I don't see yamux going away in the next 2 years. Pretty much any production deployment of rust-libp2p still runs on yamux today.

Backwards-compatibility is a super interesting topic and I'd assume that most experienced engineers will appreciate the complexity involved in retrofitting a feature without breaking existing clients.

What

In a nutshell, libp2p/specs#547 is adjusting the spec to what we ended up doing. I think there are at least two interesting things to write about here:

• A description of the actual solution: Definition of an ACK backlog on the "opening" end and a buffer of the same size on the receiving end.
• How we implemented it in rust-libp2p: Poll-based functions that suspend once we hit the ack backlog and wake up the corresponding task as soon as we can open more streams. This integrates seamlessly into Rust's async/await architecture: A task trying to open a new stream will not only suspend until the underlying IO socket is ready but also until the remote is ready to accept a stream (i.e. our ACK backlog is < 256).

---

I'd expect the Rust part of this blogpost to be the more interesting bit but the overall solution is IMO still interesting for the wider libp2p community.

Any vetos on posting this on the libp2p blog?

[marten-seemann]

Same objections as on libp2p/specs#547 apply here. I'd prefer to keep the discussion on the specs issue. That said, given the hacky nature of the "backpressure" in yamux, I don't think this is a very interesting topic for a blog post to begin with.

[thomaseizinger]

Same objections as on libp2p/specs#547 apply here. I'd prefer to keep the discussion on the specs issue.

Sure! I am currently waiting for a reply there :)

That said, given the hacky nature of the "backpressure" in yamux, I don't think this is a very interesting topic for a blog post to begin with.

It is not ideal, sure but given the constraints (backwards-compatibility), I think the outcome is quite nice.

Let's debate this once we have the specs issue resolved.

[mxinden]

I think "retrofitting backpressure" is a difficult challenge worth talking about. That said, I think our time is better spent on e.g. getting libp2p/rust-libp2p#3454 merged or writing about rust-libp2p's WASM capabilities (libp2p/rust-libp2p#4015).

I am afraid our human resources are a zero-sum-game, i.e. the time we spent on this blog post we won't spend on higher impact items.

[thomaseizinger]

Thanks for the input!

Closing as not desired.