Skip to content

Gossipsub extension for Epidemic Meshes (v1.2.0) #413

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
AgeManning wants to merge 4 commits into
base: master
Choose a base branch
from
Draft

Gossipsub extension for Epidemic Meshes (v1.2.0) #413

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
587aacd
Initial draft of gossipsub v2
AgeManning May 17, 2022
b19bcb8
Rename to v1.2
AgeManning Jun 3, 2022
847c56b
Spec improvement
AgeManning Nov 9, 2022
c03418c
Reviewers comments
AgeManning Nov 9, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
170 changes: 145 additions & 25 deletions pubsub/gossipsub/gossipsub-v1.2.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,22 +3,42 @@

# Overview

This document aims to provide a minimal extension to the [gossipsub v1.1](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md) protocol, that supersedes the previous [episub](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/episub.md) proposal.

The proposed extensions are backwards-compatible and aim to enhance the efficiency (minimize amplification/duplicates and decrease message latency) of the gossip mesh networks by dynamically adjusting the messages sent by mesh peers based on a local view of message duplication and latency.

In more specific terms, two new control messages are introduced, `CHOKE` and `UNCHOKE`. When a Gossipsub router is receiving many duplicates on a particular mesh, it can send a `CHOKE` message to it's mesh peers that are sending duplicates slower than its fellow mesh peers. Upon receiving a `CHOKE` message, a peer is informed to no longer propagate mesh messages to the sender of the `CHOKE` message, rather lazily (in every heartbeat) send it's gossip.

A Gossipsub router may notice that it is receiving messages via gossip from it's `CHOKE`'d peers faster than it receives them via the mesh. In this case it may send an `UNCHOKE` message to the peer to inform the peer to resume propagating messages in the mesh.
The router may also notice that it is receiving messages from gossip from peers not in the mesh (fanout) faster than it receives messages in the mesh. It may then add these peers into the mesh.

The modifications outlined above intend to optimize the Gossipsub mesh to receive minimal duplicates from peers with the lowest latency.
This document aims to provide a minimal extension to the [gossipsub
v1.1](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md)
protocol, that supersedes the previous
[episub](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/episub.md)
proposal.

The proposed extensions are backwards-compatible and aim to enhance the
efficiency (minimize amplification/duplicates and decrease message latency) of
the gossip mesh networks by dynamically adjusting the messages sent by mesh
peers based on a local view of message duplication and latency.

In more specific terms, two new control messages are introduced, `CHOKE` and
`UNCHOKE`. When a Gossipsub router is receiving many duplicates on a particular
mesh, it can send a `CHOKE` message to it's mesh peers that are sending
duplicates slower than its fellow mesh peers. Upon receiving a `CHOKE` message,
a peer is informed to no longer propagate mesh messages to the sender of the
`CHOKE` message, rather lazily (in every heartbeat) send it's gossip.

A Gossipsub router may notice that it is receiving messages via gossip from
it's `CHOKE`'d peers faster than it receives them via the mesh. In this case it
may send an `UNCHOKE` message to the peer to inform the peer to resume
propagating messages in the mesh. The router may also notice that it is
receiving messages from gossip from peers not in the mesh faster than it
receives messages from it's mesh peers. It may then add these peers into the
mesh.

The modifications outlined above intend to optimize the Gossipsub mesh to
receive minimal duplicates from peers with the lowest latency.

# Specification

## Protocol Id

Nodes that support this Gossipsub extension should additionally advertise the version number `1.2.0`. Gossipsub nodes can advertise their own protocol-id prefix, by default this is `meshsub` giving the default protocol id:
Nodes that support this Gossipsub extension should additionally advertise the
version number `1.2.0`. Gossipsub nodes can advertise their own protocol-id
prefix, by default this is `meshsub` giving the default protocol id:
- `/meshsub/1.2.0`

## Parameters
Expand All @@ -29,32 +49,95 @@ This section lists the configuration parameters that control the behaviour of th
| Parameter | Description | Reasonable Default |
| -------- | -------- | -------- |
| `D_non_choke` | The minimum number of peers in a mesh that must remain unchoked. | `D_lo` |
| `choke_heartbeat_interval` | The number of heartbeats before assessing and applying `CHOKE`/`UNCHOKE` control messages and adding `D_max_add`. | 20 |
| `D_max_add` | The maximum number of peers to add into the mesh (from fanout) if they are performing well per `choke_heartbeat_interval`. | 1 |
| `choke_duplicates_threshold` | The minimum number of duplicates as a percentage of received messages a peer must send before being eligible of being `CHOKE`'d. | 60 |
| `choke_heartbeat_interval` | The number of heartbeats before assessing and applying `CHOKE`/`UNCHOKE` control messages and adding peers to the mesh. | 20 |
| `choke_churn` | The maximum number of peers that can be `CHOKE`'d or `UNCHOKE`'d in any `choke_heartbeat_interval`. | 2 |
|` unchoke_threshold` | Determines how aggressively we unchoke peers. The percentage of messages that we receive in the `choke_heartbeat_interval` that were received by gossip from a choked peer. | 50 |
| `fanout_addition_threshold` | How aggressively we add peers from the fanout into the mesh. The percentage of messages that we receive in the `choke_heartbeat_interval` that were received from a fanout peer. | 10 |
|` unchoke_churn` | Determines how aggressively we unchoke peers. The number of peers per `choke_heartbeat_interval` that can be unchoked on an individual mesh. | 2 |
| `mesh_addition_churn` | How aggressively we add peers from into the mesh. The number of peers per `choke_heartbeat_interval` that can be added to an individual mesh. | 1 |

## Implementation Notes

The actual strategy for choking/unchoking peers is left to each implementation
and potentially application user. Although the strategies for choking/unchoking
can be generic, a few useful strategies are listed in the appendix for implementers.

## The CHOKE Message

Every `choke_heartbeat_interval` the router applies its choking strategy to a
set of collected metrics of recent messages, in order to decide if any of its
mesh peers should be choked. The router should send no more
than `choke_churn` `CHOKE` messages to peers per mesh topic. The router should
also ensure that `D_non_choke` peers remain unchoked in each mesh topic.

Upon receiving a `CHOKE` message, the router MUST no longer forward messages to
the peer that sent the `CHOKE` message, while it is still in the mesh. Instead
it MUST always send an IHAVE message (provided it does not hit the IHAVE
message limit) in the next gossipsub heartbeat to the peer.
Copy link
Contributor

@Menduist Menduist Nov 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we could be more aggressive, and send IHAVE (or a similar message) instantly?
This way, the worst delay induced by episub would be 1 RTT, instead of 700ms + 1 RTT (in the case of eth2)

It would also be easier to judge if a chocked peer is faster than unchocked ones

Copy link
Contributor

@Menduist Menduist Nov 9, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thinking about it, we would even let the "topic score" handle chocking / unchoking:

  • When you are fast, you will get "first deliveries" point, and have a big score
  • So we choke peers with the lowest scores for this topic (n peers or peers with less than n points)
  • If one start to catch up, he will get "first deliveries" points thanks to the IWANT replies
  • Up until the point where he isn't amongst the slowest one, and gets unchocked

Copy link
Contributor

@vyzo vyzo Nov 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would rather not conflate scoring with this; scoring is a defense mechanism, while this is a bandwidth optimization mechanism.

Copy link
Contributor Author

@AgeManning AgeManning Nov 9, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, I've tried to leave the scoring out, because its optional. I'd like 1.2 to work for those users who have opted out of scoring.

In regards to instantly sending the IHAVE, I'm not sure about the best approach.

I guess if we do it in the heartbeat we can group messages into a single IHAVE, rather than sending an IHAVE per message, but the downside is we can't identify peers that are sending messages faster than the mesh but within this timeframe.

Happy to go with either path here. Maybe we try simulate both and see how it goes?

Copy link
Contributor

@Menduist Menduist Nov 10, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The basic overhead is quite easy to compute:

Here is a RPC message with an IHAVE:
1a0e0a0c0a04626262621204deadbeef
"deadbeef" is the message id, "26262626" is the topic
So we have 8 bytes of useful data, 8 bytes of headers & co

Even if we add the muxer headers etc, it's still <32 bytes overhead per message, which depending on the use case, might be a lot, or nothing :)

For instance, for eth2 block topic, that's nothing (at most 32 bytes every 12 seconds)
For an attestation topic, chocked peers will send:

  • Currently (not chocked): attestation (233 bytes) + topic (~43 bytes) + headers (8) = 284 bytes
  • Instant IHAVE: message id (20 bytes) + topic (~43 bytes) + headers (8) = 71 bytes = 75% reduction
  • Batched IHAVE: message id (20 bytes) = 20 bytes = 93% reduction
    (not counting topic & headers in the batched ihave, as they are enough attestation in one heartbeat that it doesn't matter)

So we loose 18% reduction on attestations, an about 0% on blocks
(sorry, I'm being eth2 centric here, but I think it highlights two very different use cases)

And of course, if we receive an IHAVE with an unknown block, we will request it, which will cost more bandwidth. That's seems harder to quantify, and even simulate, since it depends heavily on networking jitter, mesh topology, etc

Remains to see if these 18% are worth it :) I would say yes as it limit the latency cost quite drastically


A peer MUST NOT send a `CHOKE` message to another peer that is not currently
grafted into it's mesh.

A peer MUST NOT send a `CHOKE` message to another peer that is already choked
on a given mesh topic.

#### Pruning

If a mesh peer sends a `PRUNE`, the local router should consider itself also
unchoked by this peer. If that peer was choked by the local router, as it is no
longer in the mesh, it should also be considered unchoked.

Therefore, when pruning a choked peer from the mesh, an `UNCHOKE` message is
not required to be sent.

#### Publishing

Messages that are published to mesh peers MUST only be published to non-choked
peers. If flood-publishing, messages can be sent to non-mesh peers, which are
unchoked by definition.

## The UNCHOKE Message

## Choking
Every `choke_heartbeat_interval` the router applies its unchoking strategy to a
set of collected metrics of recent messages, in order to decide whether to
unchoke any of it's choked peers. The router should send no more
than `unchoke_churn` `UNCHOKE` messages to peers per mesh topic.

Every `choke_heartbeat_interval` the router counts the number of valid (or not invalid) duplicate messages (note that the first message of its kind received is not a duplicate) and the time it took to receive each duplicate for each peer in each mesh that are not `CHOKE`'d. The router then filters which peers have sent duplicates over the `choke_duplicates_threshold` and sends `CHOKE` messages to at most `choke_churn` peers ordered by largest average latency. A router should ensure that at least `D_non_choke` peers remain in the mesh (and should not send `CHOKE` messages if this limit is to be violated) and should perform this check every heartbeat with the mesh maintenance.
Upon receiving an `UNCHOKE` message, the router MUST resume forwarding messages to
the peer that sent the `UNCHOKE` message and resume the normal lazy stochastic
gossiping operation in each heartbeat.

## UnChoking
A peer MUST NOT send an `UNCHOKE` message to any peer that is not currently
grafted into it's mesh.

Every `choke_heartbeat_interval` the router counts the number of received valid messages obtained via `IWANT` (and hence gossip) from a `CHOKE`'d peer. If the percentage of received valid messages is greater then `unchoke_threshold` we send an `UNCHOKE` to randomly selected peers up to the `choke_churn` limit.
A peer MUST NOT send an `UNCHOKE` message to a peer that is already unchoked on
a given mesh topic.

## Fanout Addition
## Mesh Addition

Every `choke_heartbeat_interval` the router counts the number of received valid messages obtained via `IWANT` (and hence gossip) from `fanout` peers. If the percentage of received messages is greater than `fanout_addition_threshold` a random selection of these peers up to `D_max_add` are added to the mesh (provided the mesh bounds remain valid, i.e `D_high`).
Every `choke_heartbeat_interval` the local router may use a strategy to decide
if it wishes to add peers into its meshes. Peers may be added to fill up to
`mesh_n_high` but should be limited to at most `mesh_addition_churn` per
`choke_heartbeat_interval`.

## Handling Gossipsub Scoring For Choked Peers

TODO
## Scoring for Episub

Peers have an incentive to be choked by their mesh neighbours. Being choked
means less bandwidth the node is required to send to support the mesh network.
Malicious nodes may then intentionally attempt to game various choking
strategies in order to get choked by the router.

Choked peers are inherently less valuable mesh peers than unchoked peers. As
Copy link
Contributor

@vyzo vyzo Nov 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, this needs some more thought.

Copy link
Contributor Author

@AgeManning AgeManning Nov 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah

such, a slight scoring penalty should be added to each choked-peer which grows
the longer they are choked. This will disfavour them when a mesh gets pruned.
As Episub introduces mesh additions, peer pruning should be more common.

The exact scoring penalty is currently left as a TODO.

## Protobuf Extension

The protobuf messages are identical to those specified in the [gossipsub v1.0.0 specification](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.0.md) with the following control message modifications:
The protobuf messages are identical to those specified in the [gossipsub v1.0.0
specification](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.0.md)
with the following control message modifications:

```protobuf
message RPC {
Expand All @@ -78,3 +161,40 @@ message ControlUnChoke {
optional string topicID = 1;
}
```

# Appendix

### Optional Choking Strategies

#### Latency Cutoff

A mesh peer can get choked if it sends duplicates that arrive beyond a cut-off
latency. A threshold may be added such that if more than this duplicate
threshold (as a percentage) is sent over the latency threshold the peer is
eligible to be choked.

#### Percentile Latency

Duplicate messages can be collected over a topic and ordered by the latency
received. Peers can be choked if they send over a threshold amount of
duplicates that lie in a specific percentile.

#### Order of Arrival

Messages can be grouped based on the order that they arrive. If a peers average
order of messages is greater than a specified number, that peer is eligible to
be choked.

### Optional UnChoking Strategies

#### IHAVE Message Percentage

If a choked peer has a sent an IHAVE message prior to mesh message for more
than a specified percent of the total mesh messages received, that peer is
eligible to be unchoked.

### Optional Mesh Addition Strategies

Similar to the unchoking strategy mentioned above, a router may wish to add
peers that are frequently and consistently sending IHAVE messages prior to
receiving the referenced message on the mesh.