litep2p: Provide partial results to speedup GetRecord queries#7099
litep2p: Provide partial results to speedup GetRecord queries#7099
Conversation
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Cargo.toml
Outdated
| linregress = { version = "0.5.1" } | ||
| lite-json = { version = "0.2.0", default-features = false } | ||
| litep2p = { version = "0.8.4", features = ["websocket"] } | ||
| litep2p = { git = "https://github.com/paritytech/litep2p.git", branch = "lexnv/kad-found-records", features = ["websocket"] } |
There was a problem hiding this comment.
Yep, will do one soon, wanted to double-check first on our Kusama validator that this indeed solves the problem 🙏
CPULitep2p without this PR was running before ~7:15 (spike—left side). On the right side is Litep2p with this PR. The CPU consumption looks overall stable, with some fluctuations at ~10-minute intervals. Considering the network's randomness and the ~18x speedup improvement in authority-discovery records, this small tradeoff (if any) appears justified. Triage ReportThe
|
Signed-off-by: Alexandru Vasile <[email protected]>
|
Created backport PR for
Please cherry-pick the changes locally and resolve any conflicts. git fetch origin backport-7099-to-stable2407
git worktree add --checkout .worktree/backport-7099-to-stable2407 backport-7099-to-stable2407
cd .worktree/backport-7099-to-stable2407
git reset --hard HEAD^
git cherry-pick -x 77c78e1561bbe5ee0ecf414312bae82396ae6d11
git push --force-with-lease |
|
Created backport PR for
Please cherry-pick the changes locally and resolve any conflicts. git fetch origin backport-7099-to-stable2409
git worktree add --checkout .worktree/backport-7099-to-stable2409 backport-7099-to-stable2409
cd .worktree/backport-7099-to-stable2409
git reset --hard HEAD^
git cherry-pick -x 77c78e1561bbe5ee0ecf414312bae82396ae6d11
git push --force-with-lease |
|
Created backport PR for
Please cherry-pick the changes locally and resolve any conflicts. git fetch origin backport-7099-to-stable2412
git worktree add --checkout .worktree/backport-7099-to-stable2412 backport-7099-to-stable2412
cd .worktree/backport-7099-to-stable2412
git reset --hard HEAD^
git cherry-pick -x 77c78e1561bbe5ee0ecf414312bae82396ae6d11
git push --force-with-lease |
* master: (33 commits) Implement `pallet-asset-rewards` (#3926) [pallet-revive] Add host function `to_account_id` (#7091) [pallet-revive] Remove revive events (#7164) [pallet-revive] Remove debug buffer (#7163) litep2p: Provide partial results to speedup GetRecord queries (#7099) [pallet-revive] Bump asset-hub westend spec version (#7176) Remove 0 as a special case in gas/storage meters (#6890) [pallet-revive] Fix `caller_is_root` return value (#7086) req-resp/litep2p: Reject inbound requests from banned peers (#7158) Add "run to block" tools (#7109) Fix reversed error message in DispatchInfo (#7170) approval-voting: Make importing of duplicate assignment idempotent (#6971) Parachains: Use relay chain slot for velocity measurement (#6825) PRDOC: Document `validate: false` (#7117) xcm: convert properly assets in xcmpayment apis (#7134) CI: Only format umbrella crate during umbrella check (#7139) approval-voting: Fix sending of assignments after restart (#6973) Retry approval on availability failure if the check is still needed (#6807) [pallet-revive-eth-rpc] persist eth transaction hash (#6836) litep2p: Sufix litep2p to the identify agent version for visibility (#7133) ...
…ecord queries (#7192) Backport #7099 into `stable2412` from lexnv. See the [documentation](https://github.com/paritytech/polkadot-sdk/blob/master/docs/BACKPORT.md) on how to use this bot. <!-- # To be used by other automation, do not modify: original-pr-number: #${pull_number} --> --------- Signed-off-by: Alexandru Vasile <[email protected]> Co-authored-by: Alexandru Vasile <[email protected]> Co-authored-by: Alexandru Vasile <[email protected]> Co-authored-by: Egor_P <[email protected]>
…tech#7099) This PR provides the partial results of the `GetRecord` kademlia query. This significantly improves the authority discovery records, from ~37 minutes to ~2/3 minutes. In contrast, libp2p discovers authority records in around ~10 minutes. The authority discovery was slow because litep2p provided the records only after the Kademlia query was completed. A normal Kademlia query completes in around 40 seconds to a few minutes. In this PR, partial records are provided as soon as they are discovered from the network. ### Testing Done Started a node in Kusama with `--validator` and litep2p backend. The node discovered 996/1000 authority records in ~ 1 minute 45 seconds.  ### Before & After In this image, on the left side is libp2p, in the middle litep2p without this PR, on the right litep2p with this PR  Closes: paritytech#7077 cc @paritytech/networking --------- Signed-off-by: Alexandru Vasile <[email protected]>

This PR provides the partial results of the
GetRecordkademlia query.This significantly improves the authority discovery records, from ~37 minutes to ~2/3 minutes.
In contrast, libp2p discovers authority records in around ~10 minutes.
The authority discovery was slow because litep2p provided the records only after the Kademlia query was completed. A normal Kademlia query completes in around 40 seconds to a few minutes.
In this PR, partial records are provided as soon as they are discovered from the network.
Testing Done
Started a node in Kusama with
--validatorand litep2p backend.The node discovered 996/1000 authority records in ~ 1 minute 45 seconds.
Before & After
In this image, on the left side is libp2p, in the middle litep2p without this PR, on the right litep2p with this PR
Closes: #7077
cc @paritytech/networking