kad: Refactor FindNode query, keep K best results and add tests#114
kad: Refactor FindNode query, keep K best results and add tests#114
Conversation
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
candidates Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
dmitry-markin
left a comment
There was a problem hiding this comment.
Nice PR! And good to see we have FIND_NODE tested now.
| self.queried.insert(peer.peer); | ||
|
|
||
| // TODO: could this be written in another way? | ||
| // TODO: only insert nodes from whom a response was received |
There was a problem hiding this comment.
Is this TODO still relevant? It looks like nothing has changed in its respect.
There was a problem hiding this comment.
I dont think this is addressed indeed, will briefly look over libp2p or create an issue if this may be too involved / not covered by our testing atm, thanks!
There was a problem hiding this comment.
While looking at this I realized a few things:
- we were registering the first 20 (repl factor) responses
- whenever we got a new response (better) we tried to update just the peerID_ of the last entry
This has a few implications:
- the first 19 entries are never updated with a better (closer) peer
- the distance of the last entry was never updated. And because of this, we might end up in a situation where the record is updated with a furthest distance
// Presume:
last_entry: distance = 10, peer = A
// Registering a new response
new_response: distance = 2, peer = B
-> last entry: distance = 10, peer = B
// Registering a new response
new_response: distance = 7, peer = C
-> last entry: distance = 10, peer = C
// we should've kept B as bestIn this case, we should have updated other records and only keep the window of the best 20 peers.
To mitigate this, if we receive a response better than the last entry:
- insert the response with distance and peer to our btreemap, if and only if we didn't track that distance before
- pop the last entry to keep only K entries
Have also added a test for this keep_k_best_results
Signed-off-by: Alexandru Vasile <[email protected]>
last entry Signed-off-by: Alexandru Vasile <[email protected]>
|
|
||
| // The response received from the peer is closer than the furthest response. | ||
| if distance < furthest_distance { | ||
| // Update the entries only if the distance is not already present. |
There was a problem hiding this comment.
What is the drawback of overwriting the entry with this distance? I.e., there should be exactly one peer with such distance as it uniquely identifies the hash, so it should be safe to overwrite without checking.
There was a problem hiding this comment.
I think I followed the libp2p here. However, I believe we can overwrite the entries without checking here, and we'll have a "fresher" report in the end. Will do that, thanks!
Signed-off-by: Alexandru Vasile <[email protected]>
## [0.5.0] - 2023-05-24 This is a small patch release that makes the `FindNode` command a bit more robst: - The `FindNode` command now retains the K (replication factor) best results. - The `FindNode` command has been updated to handle errors and unexpected states without panicking. ### Changed - kad: Refactor FindNode query, keep K best results and add tests ([#114](#114)) ## [0.4.0] - 2023-05-23 This release introduces breaking changes to the litep2p crate, primarily affecting the `kad` module. Key updates include: - The `GetRecord` command now exposes all peer records, not just the latest one. - A new `RecordType` has been introduced to clearly distinguish between locally stored records and those discovered from the network. Significant refactoring has been done to enhance the efficiency and accuracy of the `kad` module. The updates are as follows: - The `GetRecord` command now exposes all peer records. - The `GetRecord` command has been updated to handle errors and unexpected states without panicking. Additionally, we've improved code coverage in the `kad` module by adding more tests. ### Added - Add release checklist ([#115](#115)) - Re-export `multihash` & `multiaddr` types ([#79](#79)) - kad: Expose all peer records of `GET_VALUE` query ([#96](#96)) ### Changed - multistream_select: Remove unneeded changelog.md ([#116](#116)) - kad: Refactor `GetRecord` query and add tests ([#97](#97)) - kad/store: Set memory-store on an incoming record for PutRecordTo ([#88](#88)) - multistream: Dialer deny multiple /multistream/1.0.0 headers ([#61](#61)) - kad: Limit MemoryStore entries ([#78](#78)) - Refactor WebRTC code ([#51](#51)) - Revert "Bring `rustfmt.toml` in sync with polkadot-sdk (#71)" ([#74](#74)) - cargo: Update str0m from 0.4.1 to 0.5.1 ([#95](#95)) ### Fixed - Fix clippy ([#83](#83)) - crypto: Don't panic on unsupported key types ([#84](#84)) --------- Signed-off-by: Alexandru Vasile <[email protected]>
## [0.5.0] - 2023-05-24 This is a small patch release that makes the `FindNode` command a bit more robst: - The `FindNode` command now retains the K (replication factor) best results. - The `FindNode` command has been updated to handle errors and unexpected states without panicking. ### Changed - kad: Refactor FindNode query, keep K best results and add tests ([#114](paritytech/litep2p#114)) --------- Signed-off-by: Alexandru Vasile <[email protected]>
## [0.5.0] - 2023-05-24 This is a small patch release that makes the `FindNode` command a bit more robst: - The `FindNode` command now retains the K (replication factor) best results. - The `FindNode` command has been updated to handle errors and unexpected states without panicking. ### Changed - kad: Refactor FindNode query, keep K best results and add tests ([#114](paritytech/litep2p#114)) --------- Signed-off-by: Alexandru Vasile <[email protected]>
This is a small patch release that makes the `FindNode` command a bit more robst: - The `FindNode` command now retains the K (replication factor) best results. - The `FindNode` command has been updated to handle errors and unexpected states without panicking. - kad: Refactor FindNode query, keep K best results and add tests ([#114](paritytech/litep2p#114)) --------- Signed-off-by: Alexandru Vasile <[email protected]>
This is a small patch release that makes the `FindNode` command a bit more robst: - The `FindNode` command now retains the K (replication factor) best results. - The `FindNode` command has been updated to handle errors and unexpected states without panicking. - kad: Refactor FindNode query, keep K best results and add tests ([#114](paritytech/litep2p#114)) --------- Signed-off-by: Alexandru Vasile <[email protected]>
## [0.5.0] - 2023-05-24 This is a small patch release that makes the `FindNode` command a bit more robst: - The `FindNode` command now retains the K (replication factor) best results. - The `FindNode` command has been updated to handle errors and unexpected states without panicking. ### Changed - kad: Refactor FindNode query, keep K best results and add tests ([paritytech#114](paritytech/litep2p#114)) --------- Signed-off-by: Alexandru Vasile <[email protected]>
## [0.5.0] - 2023-05-24 This is a small patch release that makes the `FindNode` command a bit more robst: - The `FindNode` command now retains the K (replication factor) best results. - The `FindNode` command has been updated to handle errors and unexpected states without panicking. ### Changed - kad: Refactor FindNode query, keep K best results and add tests ([paritytech#114](paritytech/litep2p#114)) --------- Signed-off-by: Alexandru Vasile <[email protected]>
## [0.5.0] - 2023-05-24 This is a small patch release that makes the `FindNode` command a bit more robst: - The `FindNode` command now retains the K (replication factor) best results. - The `FindNode` command has been updated to handle errors and unexpected states without panicking. ### Changed - kad: Refactor FindNode query, keep K best results and add tests ([#114](paritytech/litep2p#114)) ## [0.4.0] - 2023-05-23 This release introduces breaking changes to the litep2p crate, primarily affecting the `kad` module. Key updates include: - The `GetRecord` command now exposes all peer records, not just the latest one. - A new `RecordType` has been introduced to clearly distinguish between locally stored records and those discovered from the network. Significant refactoring has been done to enhance the efficiency and accuracy of the `kad` module. The updates are as follows: - The `GetRecord` command now exposes all peer records. - The `GetRecord` command has been updated to handle errors and unexpected states without panicking. Additionally, we've improved code coverage in the `kad` module by adding more tests. ### Added - Add release checklist ([#115](paritytech/litep2p#115)) - Re-export `multihash` & `multiaddr` types ([#79](paritytech/litep2p#79)) - kad: Expose all peer records of `GET_VALUE` query ([#96](paritytech/litep2p#96)) ### Changed - multistream_select: Remove unneeded changelog.md ([#116](paritytech/litep2p#116)) - kad: Refactor `GetRecord` query and add tests ([#97](paritytech/litep2p#97)) - kad/store: Set memory-store on an incoming record for PutRecordTo ([#88](paritytech/litep2p#88)) - multistream: Dialer deny multiple /multistream/1.0.0 headers ([#61](paritytech/litep2p#61)) - kad: Limit MemoryStore entries ([#78](paritytech/litep2p#78)) - Refactor WebRTC code ([#51](paritytech/litep2p#51)) - Revert "Bring `rustfmt.toml` in sync with polkadot-sdk (#71)" ([#74](paritytech/litep2p#74)) - cargo: Update str0m from 0.4.1 to 0.5.1 ([#95](paritytech/litep2p#95)) ### Fixed - Fix clippy ([#83](paritytech/litep2p#83)) - crypto: Don't panic on unsupported key types ([#84](paritytech/litep2p#84)) --------- Signed-off-by: Alexandru Vasile <[email protected]>
This PR refactors the FindNode query to make it more robust.
It also fixes a bug where only the last peer ID of our results was updated.
This had a few implications:
Testing Done
Closes #100
cc @paritytech/networking