Backpressure in Gossipsub

[mxinden]

Description

See #3078 for tracking issue on backpressure in rust-libp2p in general.

Today EnabledHandler::send_queue can grow unboundedly, where the Gossipsub NetworkBehaviour sends messages faster than the Gossipsub EnabledHandler can process them.

rust-libp2p/protocols/gossipsub/src/handler.rs

Lines 99 to 100 in ecdd0ff

	/// Queue of values that we want to send to the remote.
	send_queue: SmallVec<[proto::RPC; 16]>,

I see two solutions:

1. Front or tail drop in EnabledHandler once send_queue reaches a certain size.
2. Implement backpressure between Gossipsubs NetworkBehaviour and ConnectionHandler and only forward from NetworkBehaviour to ConnectionHandler when the latter can handle another message. I.e. drop messages for a specific peer in NetworkBehaviour instead of as suggested in (1) in ConnectionHandler.

Motivation

See #3078. Prevents past failures like #4572.

Requirements

Bound send_queue, i.e. make it impossible for it to grow unboundedly.

Open questions

No response

Are you planning to do it yourself in a pull request ?

No

[0xcrust]

Hello. I'd love to take this on

[thomaseizinger]

Hello. I'd love to take this on

That would be awesome! I'll assign you :)
Let us know if you have any questions!

[mxinden]

#4756 implements solution (1), more specifically tail dropping in the ConnectionHandler.

@divagant-martian raised concerns with this approach. After syncing with @divagant-martian, @jxs and @thomaseizinger, here is a rough summary.

Concerns with tail drop implemented in #4756

• Not all messages are created equal.
  • E.g. ephemeral heart beats vs. publish messages. The former are fine to drop. Dropping the latter can cause trouble.
  • Subscription and control messages are likely very small, thus not worth dropping in the first place.
  • Dropping subscription and control messages can lead to an inconsistent view of the world between two directly connected peers.
  • Dropping forward messages is less severe than dropping publish messages. The former are likely propagated by other nodes as well, for the latter, the local node might (currently) be the only source.
• Dropping a publish message should at least surface as an error to the user, ideally through Behaviour::publish.

(Above a publish message is authored by the local node whereas a forward message is published by another node but forwarded by the local node.)

Alternative solutions

There are many solutions to fulfill these requirements. Above all, I would like to keep added complexity low.

Smarter dropping in ConnectionHandler

• Differentiate the ConnectionHandler HandlerIn by the various types of messages. Roughly along the lines of:

  @@ -61,8 +61,10 @@ pub enum HandlerEvent {
  /// A message sent from the behaviour to the handler.
  #[derive(Debug)]
  pub enum HandlerIn {
  -    /// A gossipsub message to send.
  -    Message(proto::RPC),
  +    Publish(proto::Message),
  +    Forward(proto::Message),
  +    Subscriptions(proto::SubOpts),
  +    Control(proto::ControlMessage),
      /// The peer has joined the mesh.
      JoinedMesh,
      /// The peer has left the mesh.

• Have a queue for Subscriptions, Control and Publish. For now, this can be unbounded. At least memory footprint wise, Subscription and Control likely don't play a big role.
• Have a separate queue for Forward. This queue is bounded. Forward messages are delivered on a best-effort basis. Tail dropping messages on reaching the limit.
• When sending out messages, the first queue would be prioritized over the second queue.

Concerns with this solution:

• Publish, forward, control and subscribe would end up in separate Protobuf messages. I doubt this will have a performance impact though.
• Order between messages of the two queues is no longer guaranteed. E.g. the local node might unsubscribe from a topic and then forward a messages for that same topic.
• Publish messages might still grow unboundedly in the send queue.

Backpressure between ConnectionHandler and NetworkBehaviour, dropping in NetworkBehaviour.

Builds on the initially proposed solution (2) in the GitHub issue description.

Implement backpressure between Gossipsubs NetworkBehaviour and ConnectionHandler and only forward from NetworkBehaviour to ConnectionHandler when the latter can handle another message. I.e. drop messages for a specific peer in NetworkBehaviour instead of as suggested in (1) in ConnectionHandler.

• Add a bounded channel between NetworkBehaviour and ConnectionHandler.
• In Behaviour::publish check whether enough channels to the many ConnectionHandlers have capacity. If not, return an error to the user, ideally taking a Context to wake-up the user once there is more capacity.
• In Behaviour::foward_msg only forward to ConnectionHandlers that have capacity in the channel.

---

Curious to hear other people's thoughts. @divagant-martian, @jxs, @thomaseizinger, @0xcrust

[thomaseizinger]

Thanks for writing this up @mxinden !

I recorded a related issue in #4781 in how we would go about reducing memory usage when encoding protobuf messages.

There is some overlap in the solutions. Most importantly, both require us to pass down different messages from the behaviour to the handler.

I think a good starting point would be to do that first. We could then fan-out (no pun intended) the work and tackle message dropping and memory-efficient encoding separately.

[AgeManning]

I agree with the current solution. I think we shouldn't be dropping some control messages like GRAFT, PRUNE, SUBSCRIBE,UNSUBSCRIBE for the reasons that have been listed.

I know that we want to keep the complexity simple here, but my initial thoughts were to time-bound the forward messages and maybe the publish messages. I think if there are forward messages hanging around for > 300ms its prob not worth forwarding them anyway (and we should prioritize newer messages rather than trying to get old ones out the door which are now likely duplicates/waste)

There is also probably an optimisation we could do, but this one will add prob too much complexity. When we forward messages, we check to make sure we haven't seen this message from peers we want to forward to (this happens in the behaviour) because if we send it to these peers, it's a waste because it just gets registered as a duplicate. Often there is a race between us receiving the message from multiple peers and trying to forward that message on. If a node waits a bit in the validation, it often doesn't have to forward it on, because it receives it from all its mesh peers. If we have the situation where a forward message is sitting in the ConnectionHandler its very likely the case we've already seen it from the peer and we could just remove it from the send_queue.

One solution could be that in the ConnectionHandler if we receive a message then we check the send_queue and remove it if it exists from the forward or publish queues.

Anyway. This has become a pretty high priority for us, so I'm happy to help out and contribute here, but don't want to step on anyone's toes.

Is someone already planning on implementing the proposed soln by @mxinden ?

[thomaseizinger]

Great, thanks for weighing in @AgeManning !

Regardless of which optimisations we implement, I think it is safe to say that we will need to change the interface between gossipsub's NetworkBehaviour and ConnectionHandler from currently a single proto::RPC to differently typed messages for control, publish etc.

That allows the ConnectionHandler a more fine-grained reasoning in how to handle different messages. Do we agree on this? If yes, then that could be a safe first step whilst we work out more details on what we want to optimise.

(and we should prioritize newer messages rather than trying to get old ones out the door which are now likely duplicates/waste)

This should be easy to do once we have separate queues. I'd probably hold off the time-bound optimisation for now and see what effects the others have.

One solution could be that in the ConnectionHandler if we receive a message then we check the send_queue and remove it if it exists from the forward or publish queues.

This also seems very reasonable and easy to implement because we just have to check local state.

Is someone already planning on implementing the proposed soln by @mxinden ?

I don't know of anybody actively working on. There is a related effort to reduce memory allocations in gossipsub which I will need a similar refactoring in terms of changing the interface between behaviour and handler. cc @joshuef for visibility here.

Would it be safe to say that we should always prioritize control messages over publishes? Then we could do the following: Use the regular channel between behaviour and handler for control messages (i.e. ToSwarm::NotifyHandler). Those then go into a separate queue and are always processed first inside ConnectionHandler::poll.

Then, in a separate channel (per ConnectionHandler), we can send the publish messages from behaviour to handler. Separating these would allow control messages to always "jump the queue".

[AgeManning]

Yeah nice.

Others may want to weigh in here, but its not clear cut which messages should get prioritised. There is a control message called IHAVE and I think that one can be prioritised lower than anything else. As we are not going to be this fine grained (i.e individual control messages) my gut feeling would be the following prioritisation:

1. Publish messages
2. Subscription
3. Control messages
4. Forward messages

However, if we split up the GRAFT/PRUNE from the IHAVE/IWANT. I think it would go like:

1. Publish
2. Subscribe/Unsubscribe
3. Graft/Prune
4. Forward Message
5. IHAVE/IWANT