Upgrade libp2p to v0.47.0 by divagant-martian · Pull Request #3491 · sigp/lighthouse

divagant-martian · 2022-08-22T16:45:14Z

Issue Addressed

Upgrades libp2p to v0.47.0. Addresses the deprecation of some traits and parts of the derive macro that force handling behaviour events above the swarm level. See libp2p swarm release notes

Contains #3495 and #3497 so better to give those a pass to make reading this easier.

This upgrade also requires an explicit/manual installation of protoc. See 0.47.0 release notes

Proposed Changes

last version of libp2p requires an explicit/manual installation of the protobuf compiler (protoc) so this PR:
- Modifies the installation instructions.
- Modifies the build from source instructions.
- Updates Dockerfile to reflect this.
- Updates CI using an action. I've seen Rate limits here so we might want to change this.
Create a minimal Behaviour struct that derives NetworkBehaviour. This generates the event BehaviourEvent.
The above generated BehaviourEvent requires debug, so it's now derived for the sub-behaviour events that did not implement it.
Renames the old Behaviour to Network.
Moves the swarm inside the (new) Network.
Since handling behaviour events now is done by polling the swarm, all the NetworkBehaviourEventProcess implementation are moved to functions in the inject_<sub-behaviour>_event fashion.
Merges the Libp2pEvent into BehaviourEvent (simply by moving the two extra events NewListenAddr and ZeroListeners). This is renamed to NetworkEvent
Merges polling the swarm with polling the old behaviour.
After all this changes the old service.rs file had only utility functions for building the swarm, so this file is moved and renamed to service/utils.rs. Noting this change here explicitly because git does not recognize the file as moved but as deleted.
Updates tests accordingly.
The libp2p upgrade carries a Prometheus client upgrade which solves RUSTSEC-2022-0040

Additional Info

na

michaelsproul · 2022-08-24T00:33:39Z

I couldn't help updating the title - we can change it back to upg-rage if that was an accurate description of writing this PR 😁

divagant-martian · 2022-08-24T01:14:04Z

lmao what an accurate title description the previous one was @michaelsproul. Thanks for fixing it

…inside

This reverts commit c1d4221.

AgeManning

Yeah I like this. The majority of code changes seem to be just re-organising structure.

It all looks fine to me.

AgeManning · 2022-09-06T05:22:04Z

beacon_node/lighthouse_network/src/service/mod.rs

+        Ok((network, network_globals))
+    }
+
+    // TODO: docs


Might as well, while we are here

done in 1e5694f

AgeManning · 2022-09-06T05:27:57Z

beacon_node/lighthouse_network/src/service/mod.rs

+    pub fn poll_network(&mut self, cx: &mut Context) -> Poll<NetworkEvent<AppReqId, TSpec>> {
+        while let Poll::Ready(Some(swarm_event)) = self.swarm.poll_next_unpin(cx) {
+            let maybe_event = match swarm_event {
+                SwarmEvent::Behaviour(behaviour_event) => match behaviour_event {


The new libp2p now makes us do this manually? Thats kinda annoying

But your soln here does look cleaner.

yeah this is essentially the core of the required change

AgeManning

Ok, lets dooo it

## Issue Addressed Upgrades libp2p to v.0.47.0. This is the compilation of - [x] #3495 - [x] #3497 - [x] #3491 - [x] #3546 - [x] #3553 Co-authored-by: Age Manning <Age@AgeManning.com>

## Issue Addressed I missed this from #3491. peers were being banned at the behaviour level only. The identify errors are explained by this as well ## Proposed Changes Add banning and unbanning ## Additional Info Befor,e having tests that catch this was hard because the swarm was outside the behaviour. We could now have tests that prevent something like this in the future

## Issue Addressed I missed this from sigp#3491. peers were being banned at the behaviour level only. The identify errors are explained by this as well ## Proposed Changes Add banning and unbanning ## Additional Info Befor,e having tests that catch this was hard because the swarm was outside the behaviour. We could now have tests that prevent something like this in the future

## Issue Addressed Upgrades libp2p to v.0.47.0. This is the compilation of - [x] sigp#3495 - [x] sigp#3497 - [x] sigp#3491 - [x] sigp#3546 - [x] sigp#3553 Co-authored-by: Age Manning <Age@AgeManning.com>

## Issue Addressed I missed this from sigp#3491. peers were being banned at the behaviour level only. The identify errors are explained by this as well ## Proposed Changes Add banning and unbanning ## Additional Info Befor,e having tests that catch this was hard because the swarm was outside the behaviour. We could now have tests that prevent something like this in the future

divagant-martian added the work-in-progress PR is a work-in-progress label Aug 22, 2022

divagant-martian changed the base branch from stable to unstable August 22, 2022 16:45

divagant-martian force-pushed the networking-version-upgrades branch 2 times, most recently from 775c866 to 03e016a Compare August 23, 2022 20:00

michaelsproul changed the title ~~upgrage libp2p~~ upgrade libp2p Aug 24, 2022

divagant-martian changed the base branch from unstable to libp2p-v0.47.0-upgrade August 26, 2022 20:06

divagant-martian marked this pull request as ready for review August 26, 2022 20:41

divagant-martian changed the title ~~upgrade libp2p~~ Upgrade libp2p to v0.47.0 Aug 26, 2022

divagant-martian requested review from AgeManning and pawanjay176 August 26, 2022 20:47

divagant-martian added ready-for-review The code is ready for review and removed work-in-progress PR is a work-in-progress labels Aug 26, 2022

divagant-martian added 16 commits September 5, 2022 12:11

version upgrades

64023d9

type renames to aid in refactor

01b6b62

remove network behaviour implementation of old behaviour, move swarm …

289a9d0

…inside

fix most compilation issues

7de09e3

create network

5452541

add start function

68132a0

poll without event queueing

ed4d820

file renaming

37e6f5e

fix most compilation issues in the network crate

8e7fcd3

use better name for Old BehaviourEvent

53d9159

get right pollling the network

bb16b07

fix pm tests

4ad7131

revert allowing pm tests in debug

1a76514

further cleanup

43b2eda

fmt and clippy for the day

d427348

more fmt

0f972aa

divagant-martian added 14 commits September 5, 2022 12:12

update installation guide

bcbd9d1

install protoc for windows in CI using installation guide commands

812f34e

only windows tests for quick CI feedback

421fe09

remove more workflows

ca5bf55

enable fmt

3843eaf

enable debug tests with protoc

df17e61

Revert "remove more workflows"

9b999a5

This reverts commit c1d4221.

add protoc step to Dockerfile and CI

fd3b435

update instalation steps in ubuntu

1993263

poll swarm in loop

9538870

fixes

825ff6f

code reorg

1f6db5d

remove advisory ignore

2d04c87

update diff

01cf480

divagant-martian force-pushed the networking-version-upgrades branch from 377af9a to 01cf480 Compare September 5, 2022 17:13

AgeManning approved these changes Sep 6, 2022

View reviewed changes

add docs to the Network's start function

1e5694f

divagant-martian mentioned this pull request Sep 6, 2022

[Merged by Bors] - Libp2p v0.48.0 upgrade #3547

Closed

5 tasks

AgeManning approved these changes Sep 7, 2022

View reviewed changes

AgeManning merged commit d1a9635 into sigp:libp2p-v0.47.0-upgrade Sep 7, 2022

divagant-martian added the networking label Sep 9, 2022

divagant-martian added Networking and removed networking labels Oct 20, 2022

divagant-martian mentioned this pull request Oct 24, 2022

[Merged by Bors] - Ban and unban peers at the swarm level #3653

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade libp2p to v0.47.0#3491

Upgrade libp2p to v0.47.0#3491
AgeManning merged 31 commits intosigp:libp2p-v0.47.0-upgradefrom
divagant-martian:networking-version-upgrades

divagant-martian commented Aug 22, 2022 •

edited

Loading

Uh oh!

michaelsproul commented Aug 24, 2022

Uh oh!

divagant-martian commented Aug 24, 2022

Uh oh!

AgeManning left a comment

Uh oh!

AgeManning Sep 6, 2022

Uh oh!

divagant-martian Sep 6, 2022

Uh oh!

AgeManning Sep 6, 2022

Uh oh!

divagant-martian Sep 6, 2022

Uh oh!

AgeManning left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

divagant-martian commented Aug 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue Addressed

Proposed Changes

Additional Info

Uh oh!

michaelsproul commented Aug 24, 2022

Uh oh!

divagant-martian commented Aug 24, 2022

Uh oh!

AgeManning left a comment

Choose a reason for hiding this comment

Uh oh!

AgeManning Sep 6, 2022

Choose a reason for hiding this comment

Uh oh!

divagant-martian Sep 6, 2022

Choose a reason for hiding this comment

Uh oh!

AgeManning Sep 6, 2022

Choose a reason for hiding this comment

Uh oh!

divagant-martian Sep 6, 2022

Choose a reason for hiding this comment

Uh oh!

AgeManning left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

divagant-martian commented Aug 22, 2022 •

edited

Loading