Phase 0 Wire Protocol

[mslipper]

Follow-on to #593.

This specification describes Ethereum 2.0's networking wire protocol.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL", NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

Use of libp2p

This protocol uses the libp2p networking stack. libp2p provides a composable wrapper around common networking primitives, including:

1. Transport.
2. Encryption.
3. Stream multiplexing.

Clients MUST be compliant with the corresponding libp2p specification whenever libp2p specific protocols are mentioned. This document will link to those specifications when applicable.

Client Identity

Identification

Upon first startup, clients MUST generate an RSA key pair in order to identify the client on the network. The SHA-256 multihash of the public key is the clients's Peer ID, which is used to look up the client in libp2p's peer book and allows a client's identity to remain constant across network changes.

Addressing

Clients on the Ethereum 2.0 network are identified bymultiaddrs. multiaddrs are self-describing network addresses that include the client's location on the network, the transport protocols it supports, and its peer ID. For example, the human-readable multiaddr for a client located at example.com, available via TCP on port 8080, and with peer ID QmUWmZnpZb6xFryNDeNU7KcJ1Af5oHy7fB9npU67sseEjR would look like this:

/dns4/example.com/tcp/8080/p2p/QmUWmZnpZb6xFryNDeNU7KcJ1Af5oHy7fB9npU67sseEjR

We refer to the /dns4/example.com part as the 'lookup protocol', the /tcp/8080 part as the networking protocol, and the /p2p/<peer ID> part as the 'identity protocol.'

Clients MAY use either dns4 or ip4 lookup protocols. Clients MUST set the identity protocol to /tcp followed by a port of their choosing. It is RECOMMENDED to use the default port of 9000. Clients MUST set the identity protocol to /p2p/ followed by their peer ID.

Relevant libp2p Specifications

• multiaddr
• multihash
• peer-id

Transport

Clients communicate with one another over a TCP stream. Through that TCP stream, clients receive messages either as a result of a 1-1 RPC request/response between peers or via pubsub broadcasts.

Weak-Subjectivity Period

Some of the message types below depend on a calculated value called the 'weak subjectivity period' to be processed correctly. The weak subjectivity period is a function of the size of the validator set at the last finalized epoch. The goal of the weak-subjectivity period is to define the maximum number of validator set changes a client can tolerate before requiring out-of-band information to resync.

The definition of this function will be added to the 0-beacon-chain specification in the coming days.

Messaging

All ETH 2.0 messages conform to the following structure:

+--------------------------+
|       protocol path      |
+--------------------------+
|      compression ID      |
+--------------------------+
|                          |
|    compressed body       |
|    (SSZ encoded)         |
|                          |
+--------------------------+

The protocol path is a human-readable prefix that identifies the message's contents. It is compliant with the libp2p multistream specification. For example, the protocol path for libp2p's internal ping message is /p2p/ping/1.0.0. All protocol paths include a version for future upgradeability. In practice, client implementors will not have to manually prepend the protocol path since libp2p implements this as part of the libp2p library.

The compression ID is a single-byte sigil that denotes which compression algorithm is used to compress the message body. Currently, the following compression algorithms are supported:

1. ID 0x00: no compression
2. ID 0x01: Snappy compression

We suggest starting with Snappy because of its high throughput (~250MB/s without needing assembler), permissive license, and availability in a variety of different languages.

Finally, the compressed body is the SSZ-encoded message body after being compressed by the algorithm denoted by the compression ID.

Relevant Specifications

• multistream
• SSZ

Messages

The schema of message bodies is notated like this:

(
    field_name_1: type
    field_name_2: type
)

SSZ serialization is field-order dependent. Therefore, fields MUST be encoded and decoded according to the order described in this document. The encoded values of each field are concatenated to form the final encoded message body. Embedded structs are serialized as Containers unless otherwise noted.

All ETH 2.0 RPC messages prefix their protocol path with /eth/serenity.

Handshake

Hello

Protocol Path: /eth/serenity/hello/1.0.0

Body:

(
    network_id: uint8
    latest_finalized_root: bytes32
    latest_finalized_epoch: uint64
    best_root: bytes32
    best_slot: uint64
)

Clients exchange hello messages upon connection, forming a two-phase handshake. The first message the initiating client sends MUST be the hello message. In response, the receiving client MUST respond with its own hello message.

Clients SHOULD immediately disconnect from one another following the handshake above under the following conditions:

1. If network_id belongs to a different chain, since the client definitionally cannot sync with this client.
2. If the time between each peer's latest_finalized_epoch exceeds the weak-subjectivity period, since syncing with this client would be unsafe.
3. If the latest_finalized_root shared by the peer is not in the client's chain at the expected epoch. For example, if Peer 1 in the diagram below has (root, epoch) of (A, 5) and Peer 2 has (B, 3), Peer 1 would disconnect because it knows that B is not the root in their chain at epoch 3:

              Root A

              +---+
              |xxx|  +----+ Epoch 5
              +-+-+
                ^
                |
              +-+-+
              |   |  +----+ Epoch 4
              +-+-+
Root B          ^
                |
+---+         +-+-+
|xxx+<---+--->+   |  +----+ Epoch 3
+---+    |    +---+
         |
       +-+-+
       |   |  +-----------+ Epoch 2
       +-+-+
         ^
         |
       +-+-+
       |   |  +-----------+ Epoch 1
       +---+

Once the handshake completes, the client with the higher latest_finalized_epoch or best_slot (if the clients have equal latest_finalized_epochs) SHOULD send beacon block roots to its counterparty via beacon_block_roots.

RPC

These protocols represent RPC-like request/response interactions between two clients. Clients send serialized request objects to streams at the protocol paths described below, and wait for a response. If no response is received within a reasonable amount of time, clients MAY disconnect.

Beacon Block Roots

Protocol Path: /eth/serenity/rpc/beacon_block_roots/1.0.0

Body:

# BlockRootSlot
(
    block_root: HashTreeRoot
    slot: uint64
)

(
    roots: []BlockRootSlot
)

Send a list of block roots and slots to the peer.

Beacon Block Headers

Protocol Path: /eth/serenity/rpc/beacon_block_headers/1.0.0

Request Body

(
    start_root: HashTreeRoot
    start_slot: uint64
    max_headers: uint64
    skip_slots: uint64
)

Response Body:

# Eth1Data
(
    deposit_root: bytes32
    block_hash: bytes32
)

# BlockHeader
(
    slot: uint64
    parent_root: bytes32
    state_root: bytes32
    randao_reveal: bytes96
    eth1_data: Eth1Data
    body_root: HashTreeRoot
    signature: bytes96
)

(
    headers: []BlockHeader
)

Requests beacon block headers from the peer starting from (start_root, start_slot). The response MUST contain fewer than max_headers headers. skip_slots defines the maximum number of slots to skip between blocks. For example, requesting blocks starting at slots 2 a skip_slots value of 2 would return the blocks at [2, 4, 6, 8, 10]. In cases where a slot is undefined for a given slot number, the closest previous block MUST be returned returned. For example, if slot 4 were undefined in the previous example, the returned array would contain [2, 3, 6, 8, 10]. If slot three were further undefined, the array would contain [2, 6, 8, 10] - i.e., duplicate blocks MUST be collapsed.

The function of the skip_slots parameter helps facilitate light client sync - for example, in #459 - and allows clients to balance the peers from whom they request headers. Client could, for instance, request every 10th block from a set of peers where each per has a different starting block in order to populate block data.

Beacon Block Bodies

Protocol Path: /eth/serenity/rpc/beacon_block_bodies/1.0.0

Request Body:

(
    block_roots: []HashTreeRoot
)

Requests the block_bodies associated with the provided block_roots from the peer. Responses MUST return block_roots in the order provided in the request. If the receiver does not have a particular block_root, it must return a zero-value block_body (i.e., a zero-filled bytes32).

Response Body:

For type definitions of the below objects, see the 0-beacon-chain specification.

# BlockRoot
(
    proposer_slashings: []ProposerSlashing
    attester_slashings: []AttesterSlashing
    attestations: []Attestation
    deposits: []Deposit
    voluntary_exits: []VoluntaryExit
    transfers: []Transfer
)

(
    block_roots: BlockRoot[]
)

Beacon Chain State

Note: This section is preliminary, pending the definition of the data structures to be transferred over the wire during fast sync operations.

Protocol Path: /eth/serenity/rpc/beacon_chain_state/1.0.0

Request Body:

(
    hashes: []HashTreeRoot
)

Requests contain the hashes of Merkle tree nodes that when merkelized yield the block's state_root.

Response Body: TBD

The response will contain the values that, when hashed, yield the hashes inside the request body.

Broadcast

These protocols represent 'topics' that clients can subscribe to via GossipSub.

Beacon Blocks

The response bodies of each topic below map to the response bodies of the Beacon RPC methods above. Note that since broadcasts have no concept of a request, any limitations to the RPC response bodies do not apply to broadcast messages.

Topics:

• beacon/block_roots
• beacon/block_headers
• beacon/block_bodies

Voluntary Exits

Topic: beacon/exits

Body:

See the 0-beacon-chain spec for the definition of the VoluntaryExit type.

(
    exit: VoluntaryExit
)

Transfers

Topic: beacon/transfer

Body:

See the 0-beacon-chain spec for the definition of the Transfer type.

(
    transfer: Transfer
)

Clients MUST ignore transfer messages if transfer.slot < current_slot - GRACE_PERIOD, where GRACE_PERIOD is an integer that represents the number of slots that a remote peer is allowed to drift from current_slot in order to take potential network time differences into account.

Shard Attestations

Topics: shard-{number}, where number is an integer in [0, SHARD_SUBNET_COUNT), and beacon/attestations.

The Attestation object below includes fully serialized AttestationData in its data field. See the 0-beacon-chain for the definition of the Attestation type.

Body:

(
    attestations: []Attestation
)

Only aggregate attestations are broadcast to the beacon/attestations topic.

Clients SHOULD NOT send attestations for shards that the recipient is not interested in. Clients receiving uninteresting attestations MAY disconnect from senders.

Relevant Specifications

• gossipsub

Client Synchronization

When a client joins the network, or has otherwise fallen behind the latest_finalized_root or latest_finalized_epoch, the client MUST perform a sync in order to catch up with the head of the chain. This specification defines two sync methods:

1. Standard: Used when clients already have state at latest_finalized_root or latest_finalized_epoch. In a standard sync, clients process per-block state transitions until they reach the head of the chain.
2. Fast: Used when clients do not have state at latest_finalized_root or latest_finalized_epoch. In a fast sync, clients use RPC methods to download nodes in the state tree for a given state_root via the /eth/serenity/rpc/beacon_chain_state/1.0.0 endpoint. The basic algorithm is as follows:
   1. Peer 1 and Peer 2 connect. Peer 1 has (C, 1) and Peer 2 has (A, 5). Peer 1 validates that this new head is within the weak subjectivity period.
   2. If the head is within the weak subjectivity period, Peer 2 checks the validity of the new chain by verifying that all children point to valid parent roots.
   3. Peer 2 then takes the state root of (A, 5) and sends /eth/serenity/rpc/beacon_chain_state/1.0.0 requests recursively to its peers in order to build its SSZ BeaconState.

Note that nodes MUST perform a fast sync if they do not have state at their starting finalized root. For example, if Peer 1 in the example above did not have the state at (C, 1), Peer 1 would have to perform a fast sync because it would have no base state to compute transitions from.

Open Questions

Encryption

This specification does not currently define an encrypted transport mechanism because the set of libp2p-native encryption libraries is limited. libp2p currently supports an encryption scheme called SecIO, which is a variant of TLSv1.3 that uses a peer's public key for authentication rather than a certificate authority. While SecIO is available for public use, it has not been audited and is going to be deprecated when TLSv1.3 ships.

Another potential solution would be to support an encryption scheme such as Noise. The Lightning Network team has successfully deployed Noise in production to secure inter-node communications.

Granularity of Topics

This specification defines granular GossipSub topics - i.e., beacon/block_headers vs. simply beacon. The goal of using granular topics is to simplify client development by defining a single payload type for each topic. For example, beacon/block_headers will only ever contain block headers, so clients know the content type of without needing to read the message body itself. This may have drawbacks. For example, having too many topics may hinder peer discovery speed. If this is the case, this specification will be updated to use less granular topics.

Block Structure Changes

The structure of blocks may change due to #649. Changes that affect this specification will be incorporated here once the PR is merged.

[Mikerah]

Great writeup!

A few notes:

There has been discussions on whether to use RLP for discovery and SSZ for messages.

There has also been some discussion on using QUIC instead of UDP for discovery and TCP for all other peer communication. By using QUIC, we no longer need a 3-way handshake as in TCP and we get built-in encryption. The main issue is that it hasn't been implemented in all programming languages.

[vbuterin]

If the time between each peer's latest_finalized_epoch exceeds the weak-subjectivity period, since syncing with this client would be unsafe.

What do we mean by this? Note that especially in cases where the validator set is very small, it is absolutely possible for the latest finalized block to be more than a weak subjectivity period behind the current block.

3. If the latest_finalized_root shared by the peer is not in the client's chain at the expected epoch. For example, if Peer 1 in the diagram below has (root, epoch) of (A, 5) and Peer 2 has (B, 3), Peer 1 would disconnect because it knows that B is not the root in their chain at epoch 3:

This feels backward. It's not the peer's latest finalized root that should be checked against the client head, it's the peer's head that should be checked against the client's latest finalized root. As an example of how the current approach is insufficient, consider the case where the client head is H with a latest finalized root R with parent P, and a peer says hello with a chain going through some different child R' of P that doesn't finalize any blocks. The peer's latest finalized root would be some ancestor of P; so this is clearly invalid from the point of view of the client, but it would still be accepted.

Only aggregate attestations are broadcast to the beacon/attestations topic.

Is there a need for this particular wire protocol spec to specify where unaggregated attestations get sent?

There has been discussions on whether to use RLP for discovery and SSZ for messages.

As the inventor of RLP, I'm inclined to prefer SSZ (or SOS if we end up replacing SSZ with SOS or something similar) 😆

The "simplicity over efficiency" desideratum would imply that we should ideally only use one algorithm per task (in this case the task being "serialization"), even if they have slightly different tradeoffs. I see the overhead of a few extra zero bytes in SSZ being tiny relative to the absolute size of the objects that would be transferred, so the gains wouldn't be large.

[djrtwo]

What do we mean by this? Note that especially in cases where the validator set is very small, it is absolutely possible for the latest finalized block to be more than a weak subjectivity period behind the current block.

The point here is to make clear that you may think you are safe to sync but realize on the wire that you don't have recent enough information to safely sync. I suppose this information can (and maybe should) be known before you even contact peers. An epoch # is an absolute time so you can know the expected head of chain epoch before you sync and know you are in the danger zone without knocking on a peer's door. In this case, maybe this is not a wire protocol note, but a client implementer safety note that goes in another doc.

This feels backward

The peer with the lower finalized epoch (peer-1) cannot immediately tell just by shaking hands with the peer with the higher finalized epoch (peer-2) if the peer-1's finalized epoch is in peer-2's chain or not. Peer-2 on the other hand can easily check if peer-1's finalized epoch is in their chain. If not, no use in sending blocks to them because to sync peer-2's blocks, peer-1 would have to revert.

I'm not sure if what we currently have in (3) is backward, but it is does not sufficiently cover all the cases in which two chains might be out of sync wrt finality

Is there a need for this particular wire protocol spec to specify where unaggregated attestations get sent?

Not 100% sure. The expectation is that single signer attestations are sent to the shard subnet. Only rarely would a single signer attestation be broadcast to the beacon net as a "best effort" aggregation.

The "simplicity over efficiency" desideratum would imply that we should ideally only use one algorithm per task

+1

[vbuterin]

Not 100% sure. The expectation is that single signer attestations are sent to the shard subnet. Only rarely would a single signer attestation be broadcast to the beacon net as a "best effort" aggregation.

Agree! So do we need a "shard subnet wire protocol" section?

[CarlBeek]

Re-raising this point, but is there any particular reason why beacon_block_roots is built in as a part of hello instead of being exposed as its own RPC request? It improves the modularity of hello and passes the onus of getting up to date to the peer that is behind. Under the above proposal, if a peer is behind and conducts several hello exchanges with its peers (perhaps due to being offline temporarily) would receive beacon block roots several times. This obviously represents sevral redundant messages. Furthermore, in the above protocol, the only means of obtaining beacon block roots is via new hello exchanges.

Also, if we are going to make topics plural, please can we make beacon/transfer plural too.

[jannikluhn]

Nice writeup! This is quite a big document, so maybe opening a PR would allow for easier review (not sure what the best place for this would be though). I'll try giving some feedback anyway:

SSZ/SOS vs RLP debate

I'm leaning towards RLP for message encoding. Not because of efficiency, but because with SSZ we at all times need to know the exact message type that we're expecting. RLP is more flexible in that regard which would make it easier to support versioning, multiple possible response message types, and using the same gossip topic for multiple object types. Also, if we go with RLP at the discovery level anyway, then using it at the "content" level would be more consistent (but this is a different debate I guess).

Message format

I think whenever we send objects that are defined in the spec, we should put them in the message either as a (list of) SSZ blob(s) (if we go with non-SSZ as message format) or as a container field (for SSZ messages). Right now, it seems like we're redefining some object types (e.g. block header and block body).

Handshake

• network id: It seems like we don't actually use this to distinguish between networks, but rather between chains. Replacing it with something like "hard fork choices" would make this more explicit: It would contain slot or epoch and block hash pairs for relevant blocks (genesis and all hard forks, or at least the latest one)

• Shall we add the latest justified head as well? If I'm not making a mistake, the last finalized epoch will be 1.5 epochs or ~10 minutes old on average, even in optimal conditions, and adding the justified epoch should be quite cheap.

• Shall we rename best to head to follow the terminology of the spec and be more neutral?

Disconnect

Maybe we should have an explicit "Disconnect" message. This would enable peers to do some clean ups and also stop responding to any potential outstanding requests.

Request/Responses

We should add a request_id to each request/response message. This way handling replies gets much easier and also allows for parallel requests.

Beacon Block Roots

I think the corresponding request is missing, no? We can probably just use the same format as we do for requesting beacon block headers.

Beacon Block Headers

The request contains both slot and root. Why? The slot seems to be redundant so I think removing it is better for simplicity.

Scope

I think we should try to keep this as simple as possible for now and don't specify any fast or light sync etc.

Protocol paths

I think we should add beacon/shard to the protocol path, i.e.

/eth/serenity/rpc/beacon_block_roots/1.0.0 --> /eth/serenity/beacon/rpc/beacon_block_roots/1.0.0

Agree! So do we need a "shard subnet wire protocol" section?

👍

[djrtwo]

Agree! So do we need a "shard subnet wire protocol" section?

The "Shard Attestations" section includes both shard and beacon topics as valid topics to broadcast on. It just makes the note that only aggregated are expected to be passed to the beacon topic. Is this what you mean? We can pull these things out so they are more clearly distinct.

[djrtwo]

Re-raising this point, but is there any particular reason why beacon_block_roots is built in as a part of hello instead of being exposed as its own RPC request?

This is similar to how block hashes are exchanged on the ethereum 1.0 wire protocol. The expectation is that you are only gathering information to "catch up" when initially finding peers and that beyond that you generally stay synced by listening to broadcasts on the wire. If you get a recent block and don't have it's parent, you can ask for it's parent via the root embedded in the block and walk the chain back until you find a common ancestor. If fallen way out of sync, you can reconnect to peers to get a list of roots.

I see the potential utility of what you describe but am not sure needing to request the list after initial hello is generally necessary. I'd like to look into how geth handles this and will ask a couple of 1.0 sync masters about it today.

Also, if we are going to make topics plural, please can we make beacon/transfer plural too.

👍

[djrtwo]

Nice writeup! This is quite a big document, so maybe opening a PR would allow for easier review (not sure what the best place for this would be though). I'll try giving some feedback anyway:

Will open up a PR after these initial comments. Thanks!

Right now, it seems like we're redefining some object types (e.g. block header and block body).

The current format of BlockHeader here is just the BeaconBlock with the body served as a hash root. I hear you on making the formats just easily reference the SSZ objects for clarity. We have a couple of header/body changes coming in #649 (firm separation between header/body) and then will clean up accordingly.

network id: It seems like we don't actually use this to distinguish between networks, but rather between chains.

Interesting. In the 1.0 protocol we use it for both distinguishing between entirely different chains (mainnet vs ropsten vs ..) as well as forks within a single chain. and contentious forks become different networks. I see there could be some use in coming up with a succinct format to describe forks independent of base "network". Nick Johnson had an interesting proposal on twitter. Something like this might be worth pursuing.

If I'm not making a mistake, the last finalized epoch will be 1.5 epochs or ~10 minutes old on average, even in optimal conditions, and adding the justified epoch should be quite cheap.

The latest_finalized_epoch/root is to see if the two chains are irrevocably disjoint and to serve somewhat as a "check-pointing" mechanism. The lastest justified is not set in stone. What use case do you see here? Also note, that a newly syncing peer would probably just have a latest finalized epoch/root.

Shall we rename best to head to follow the terminology of the spec and be more neutral?

agreed

Maybe we should have an explicit "Disconnect" message. This would enable peers to do some clean ups and also stop responding to any potential outstanding requests.

Seems reasonable. Could also signal a "why" if you had a particular reason

I think we should try to keep this as simple as possible for now and don't specify any fast or light sync etc.

A method to sync state is necessary in the weakly subjective chain. Clients are expected to show up with some recent finalized root and peers are not expected to serve blocks since genesis in perpetuity. The term is more aptly described here as "State Sync" rather than "fast" as it's primary use is to provide a recent state to a client. Also note that the state size is bound ~600MB even in the worst case and so should actually be fast.

gotta run. I'll get to your other few things in a bit, and will make some edits accordingly.

[FrankSzendzielarz]

After a review (I will study/contemplate this later in more depth) of this lovely piece of work, I just want to raise a couple of questions, which most likely arise out of my own lack of understanding:

1. If nodes are discovered using ENRs, the information on the node is already available. This means that clients will be identified by ENR, and if libp2p multiaddr is used or not is an implementation detail, not a protocol requirement, (right?)
2. Similarly, compression type could be omitted from the message as the default should be the 'highest' supported by the sending node supported by the recipient.
3. Encoding (SSZ) could similarly be optional/variable.
4. Protocol path could be compressed out as the info is in the ENR. A byte/nibble should be sufficient. Why compress the message body and make the protocol path human readable?
5. Eth-current uses Network ID and Chain ID.

On wire encoding is it fair to say that for participation in the existing network, RLP is necessary as a software component for the medium term future regardless?