100 lines
3.3 KiB
Rust
Raw Normal View History

Implement Kademlia peer discovery (#120) * Impl Clone for SwarmController and remove 'static * Implement Kademlia * Implement ConnectionReuse correctly * Implement ConnectionReuse correctly * Add some tests and fixes * Remove useless boolean in active_connections * Correctly run tests * Optimize the processing * Rustfmt on libp2p-kad * Improve Kademlia example * Next incoming is now in two steps * Some work * Remove log * Fix dialing a node even if we already have a connection * Add a proper PeerId to Peerstore * Turn identify into a transport layer * Expose the dialed multiaddress * Add identified nodes to the peerstore * Allow configuring the TTL of the addresses * Split identify in two modules * Some comments and tweaks * Run rustfmt * More work * Add test and bugfix * Fix everything * Start transition to new identify system * More work * Minor style * Start implementation of Kademlia server upgrade * Continue implementing the Kademlia server * Start reimplementing high-level kademlia code * Continue reimplementing high-level code * More work * More work * More work * Fix wrong address reported when dialing * Make it work * Remove cluster_level field everywhere * Fix bug in varint-rs when encoding * More work * More work * More work * More work * More work * Bugfix * More work * Implement ping * Style in kademlia_handler * More work * Better error handling in query.rs * More work * More work * More work * Debug impls * Some cleanup in swarm * More work * Clean up changes in swarm * Unpublish the kbucket module * Fix examples and some warnings * Fix websocket browser code * Rustfmt on libp2p-kad * Kad initialization process * Add logging to the example * Fix concerns * Fix style
2018-03-15 15:18:21 +01:00
// Copyright 2018 Parity Technologies (UK) Ltd.
//
// Permission is hereby granted, free of charge, to any person obtaining a
// copy of this software and associated documentation files (the "Software"),
// to deal in the Software without restriction, including without limitation
// the rights to use, copy, modify, merge, publish, distribute, sublicense,
// and/or sell copies of the Software, and to permit persons to whom the
// Software is furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall be included in
// all copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
// OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
// DEALINGS IN THE SOFTWARE.
Kademlia: Address some TODOs - Refactoring - API updates. (#1174) * Address some TODOs, refactor queries and public API. The following left-over issues are addressed: * The key for FIND_NODE requests is generalised to any Multihash, instead of just peer IDs. * All queries get a (configurable) timeout. * Finishing queries as soon as enough results have been received is simplified to avoid code duplication. * No more panics in provider-API-related code paths. The provider API is however still untested and (I think) still incomplete (e.g. expiration of provider records). * Numerous smaller TODOs encountered in the code. The following public API changes / additions are made: * Introduce a `KademliaConfig` with new configuration options for the replication factor and query timeouts. * Rename `find_node` to `get_closest_peers`. * Rename `get_value` to `get_record` and `put_value` to `put_record`, introducing a `Quorum` parameter for both functions, replacing the existing `num_results` parameter with clearer semantics. * Rename `add_providing` to `start_providing` and `remove_providing` to `stop_providing`. * Add a `bootstrap` function that implements a (almost) standard Kademlia bootstrapping procedure. * Rename `KademliaOut` to `KademliaEvent` with an updated list of constructors (some renaming). All events that report query results now report a `Result` to uniformly permit reporting of errors. The following refactorings are made: * Introduce some constants. * Consolidate `query.rs` and `write.rs` behind a common query interface to reduce duplication and facilitate better code reuse, introducing the notion of a query peer iterator. `query/peers/closest.rs` contains the code that was formerly in `query.rs`. `query/peers/fixed.rs` contains a modified variant of `write.rs` (which is removed). The new `query.rs` provides an interface for working with a collection of queries, taking over some code from `behaviour.rs`. * Reduce code duplication in tests and use the current_thread runtime for polling swarms to avoid spurious errors in the test output due to aborted connections when a test finishes prematurely (e.g. because a quorum of results has been collected). * Some additions / improvements to the existing tests. * Fix test. * Fix rebase. * Tweak kad-ipfs example. * Incorporate some feedback. * Provide easy access and conversion to keys in error results.
2019-07-03 16:16:25 +02:00
//! Implementation of the libp2p-specific Kademlia protocol.
Implement Kademlia peer discovery (#120) * Impl Clone for SwarmController and remove 'static * Implement Kademlia * Implement ConnectionReuse correctly * Implement ConnectionReuse correctly * Add some tests and fixes * Remove useless boolean in active_connections * Correctly run tests * Optimize the processing * Rustfmt on libp2p-kad * Improve Kademlia example * Next incoming is now in two steps * Some work * Remove log * Fix dialing a node even if we already have a connection * Add a proper PeerId to Peerstore * Turn identify into a transport layer * Expose the dialed multiaddress * Add identified nodes to the peerstore * Allow configuring the TTL of the addresses * Split identify in two modules * Some comments and tweaks * Run rustfmt * More work * Add test and bugfix * Fix everything * Start transition to new identify system * More work * Minor style * Start implementation of Kademlia server upgrade * Continue implementing the Kademlia server * Start reimplementing high-level kademlia code * Continue reimplementing high-level code * More work * More work * More work * Fix wrong address reported when dialing * Make it work * Remove cluster_level field everywhere * Fix bug in varint-rs when encoding * More work * More work * More work * More work * More work * Bugfix * More work * Implement ping * Style in kademlia_handler * More work * Better error handling in query.rs * More work * More work * More work * Debug impls * Some cleanup in swarm * More work * Clean up changes in swarm * Unpublish the kbucket module * Fix examples and some warnings * Fix websocket browser code * Rustfmt on libp2p-kad * Kad initialization process * Add logging to the example * Fix concerns * Fix style
2018-03-15 15:18:21 +01:00
// TODO: we allow dead_code for now because this library contains a lot of unused code that will
// be useful later for record store
#![allow(dead_code)]
pub mod handler;
pub mod kbucket;
pub mod protocol;
Kademlia Records (#1144) * initial implementation of the records * move to multihash keys * correctly process query results * comments and formatting * correctly return closer_peers in query * checking wrong peer id in test * Apply suggestions from code review Co-Authored-By: Roman Borschel <romanb@users.noreply.github.com> * Fix changes from suggestions * Send responses to PUT_VALUE requests * Shortcut in get_value * Update protocols/kad/src/behaviour.rs Co-Authored-By: Roman Borschel <romanb@users.noreply.github.com> * Revert "Update protocols/kad/src/behaviour.rs" This reverts commit 579ce742a7f4c94587f1e1f0866d2a3a37418efb. * Remove duplicate insertion * Adds a record to a PUT_VALUE response * Fix a racy put_value test * Store value ourselves only if we are in K closest * Abstract over storage * Revert "Abstract over storage": bad take This reverts commit eaebf5b6d915712eaf3b05929577fdf697f204d8. * Abstract over records storage using hashmap as default * Constructor for custom records * New Record type and its traits * Fix outdated storage name * Fixes returning an event * Change FindNodeReq key type to Multihash * WriteState for a second stage of a PUT_VALUE request * GET_VALUE should not have a record * Refactor a match arm * Add successes and failures counters to PutValueRes * If value is found no need to return closer peers * Remove a custo storage from tests * Rename a test to get_value_not_found * Adds a TODO to change FindNode request key to Multihash Co-Authored-By: Roman Borschel <romanb@users.noreply.github.com> * Move MemoryRecordStorage to record.rs * Return a Cow-ed Record from get * Fix incorrect GET_VALUE parsing * Various fixes with review * Fixes get_value_not_found * Fix peerids names in test * another fix * PutValue correctly distributes values * Simplify the test * Check that results are actually the closest * Reverts changes to tests * Fix the test topology and checking the results * Run put_value test ten times * Adds a get_value test * Apply suggestions from code review Co-Authored-By: Roman Borschel <romanb@users.noreply.github.com> * Make Record fields public * Moves WriteState to write.rs * A couple of minor fixes * Another few fixes of review * Simplify the put_value test * Dont synchronously return an error from put_value * Formatting fixes and comments * Collect a bunch of results * Take exactly as much elements as neede * Check if the peer is still connected * Adds a multiple GetValueResults results number test * Unnecessary mut iterators in put_value * Ask for num_results in get_value * Dont allocate twice in get_value * Dont count same errored peer multiple times * Apply suggestions from code review Co-Authored-By: Roman Borschel <romanb@users.noreply.github.com> * Fix another review * Apply suggestions from code review Co-Authored-By: Pierre Krieger <pierre.krieger1708@gmail.com> * Bring back FromIterator and improve a panic message * Update protocols/kad/src/behaviour.rs Co-Authored-By: Pierre Krieger <pierre.krieger1708@gmail.com>
2019-06-04 14:44:24 +03:00
pub mod record;
Implement Kademlia peer discovery (#120) * Impl Clone for SwarmController and remove 'static * Implement Kademlia * Implement ConnectionReuse correctly * Implement ConnectionReuse correctly * Add some tests and fixes * Remove useless boolean in active_connections * Correctly run tests * Optimize the processing * Rustfmt on libp2p-kad * Improve Kademlia example * Next incoming is now in two steps * Some work * Remove log * Fix dialing a node even if we already have a connection * Add a proper PeerId to Peerstore * Turn identify into a transport layer * Expose the dialed multiaddress * Add identified nodes to the peerstore * Allow configuring the TTL of the addresses * Split identify in two modules * Some comments and tweaks * Run rustfmt * More work * Add test and bugfix * Fix everything * Start transition to new identify system * More work * Minor style * Start implementation of Kademlia server upgrade * Continue implementing the Kademlia server * Start reimplementing high-level kademlia code * Continue reimplementing high-level code * More work * More work * More work * Fix wrong address reported when dialing * Make it work * Remove cluster_level field everywhere * Fix bug in varint-rs when encoding * More work * More work * More work * More work * More work * Bugfix * More work * Implement ping * Style in kademlia_handler * More work * Better error handling in query.rs * More work * More work * More work * Debug impls * Some cleanup in swarm * More work * Clean up changes in swarm * Unpublish the kbucket module * Fix examples and some warnings * Fix websocket browser code * Rustfmt on libp2p-kad * Kad initialization process * Add logging to the example * Fix concerns * Fix style
2018-03-15 15:18:21 +01:00
mod addresses;
mod behaviour;
Kademlia: Somewhat complete the records implementation. (#1189) * Somewhat complete the implementation of Kademlia records. This commit relates to [libp2p-146] and [libp2p-1089]. * All records expire (by default, configurable). * Provider records are also stored in the RecordStore, and the RecordStore API extended. * Background jobs for periodic (re-)replication and (re-)publication of records. Regular (value-)records are subject to re-replication and re-publication as per standard Kademlia. Provider records are only subject to re-publication. * For standard Kademlia value lookups (quorum = 1), the record is cached at the closest peer to the key that did not return the value, as per standard Kademlia. * Expiration times of regular (value-)records is computed exponentially inversely proportional to the number of nodes between the local node and the closest node known to the key (beyond the k closest), as per standard Kademlia. The protobuf messages are extended with two fields: `ttl` and `publisher` in order to implement the different semantics of re-replication (by any of the k closest peers to the key, not affecting expiry) and re-publication (by the original publisher, resetting the expiry). This is not done yet in other libp2p Kademlia implementations, see e.g. [libp2p-go-323]. The new protobuf fields have been given somewhat unique identifiers to prevent future collision. Similarly, periodic re-publication of provider records does not seem to be done yet in other implementations, see e.g. [libp2p-js-98]. [libp2p-146]: https://github.com/libp2p/rust-libp2p/issues/146 [libp2p-1089]: https://github.com/libp2p/rust-libp2p/issues/1089 [libp2p-go-323]: https://github.com/libp2p/go-libp2p-kad-dht/issues/323 [libp2p-js-98]: https://github.com/libp2p/js-libp2p-kad-dht/issues/98 * Tweak kad-ipfs example. * Add missing files. * Ensure new delays are polled immediately. To ensure task notification, since `NotReady` is returned right after. * Fix ipfs-kad example and use wasm_timer. * Small cleanup. * Incorporate some feedback. * Adjustments after rebase. * Distinguish events further. In order for a user to easily distinguish the result of e.g. a `put_record` operation from the result of a later republication, different event constructors are used. Furthermore, for now, re-replication and "caching" of records (at the closest peer to the key that did not return a value during a successful lookup) do not yield events for now as they are less interesting. * Speed up tests for CI. * Small refinements and more documentation. * Guard a node against overriding records for which it considers itself to be the publisher. * Document the jobs module more extensively. * More inline docs around removal of "unreachable" addresses. * Remove wildcard re-exports. * Use NonZeroUsize for the constants. * Re-add method lost on merge. * Add missing 'pub'. * Further increase the timeout in the ipfs-kad example. * Readd log dependency to libp2p-kad. * Simplify RecordStore API slightly. * Some more commentary. * Change Addresses::remove to return Result<(),()>. Change the semantics of `Addresses::remove` so that the error case is unambiguous, instead of the success case. Use the `Result` for clearer semantics to that effect. * Add some documentation to .
2019-07-17 14:40:48 +02:00
mod jobs;
mod query;
Kademlia: Address some TODOs - Refactoring - API updates. (#1174) * Address some TODOs, refactor queries and public API. The following left-over issues are addressed: * The key for FIND_NODE requests is generalised to any Multihash, instead of just peer IDs. * All queries get a (configurable) timeout. * Finishing queries as soon as enough results have been received is simplified to avoid code duplication. * No more panics in provider-API-related code paths. The provider API is however still untested and (I think) still incomplete (e.g. expiration of provider records). * Numerous smaller TODOs encountered in the code. The following public API changes / additions are made: * Introduce a `KademliaConfig` with new configuration options for the replication factor and query timeouts. * Rename `find_node` to `get_closest_peers`. * Rename `get_value` to `get_record` and `put_value` to `put_record`, introducing a `Quorum` parameter for both functions, replacing the existing `num_results` parameter with clearer semantics. * Rename `add_providing` to `start_providing` and `remove_providing` to `stop_providing`. * Add a `bootstrap` function that implements a (almost) standard Kademlia bootstrapping procedure. * Rename `KademliaOut` to `KademliaEvent` with an updated list of constructors (some renaming). All events that report query results now report a `Result` to uniformly permit reporting of errors. The following refactorings are made: * Introduce some constants. * Consolidate `query.rs` and `write.rs` behind a common query interface to reduce duplication and facilitate better code reuse, introducing the notion of a query peer iterator. `query/peers/closest.rs` contains the code that was formerly in `query.rs`. `query/peers/fixed.rs` contains a modified variant of `write.rs` (which is removed). The new `query.rs` provides an interface for working with a collection of queries, taking over some code from `behaviour.rs`. * Reduce code duplication in tests and use the current_thread runtime for polling swarms to avoid spurious errors in the test output due to aborted connections when a test finishes prematurely (e.g. because a quorum of results has been collected). * Some additions / improvements to the existing tests. * Fix test. * Fix rebase. * Tweak kad-ipfs example. * Incorporate some feedback. * Provide easy access and conversion to keys in error results.
2019-07-03 16:16:25 +02:00
mod dht_proto {
include!(concat!(env!("OUT_DIR"), "/dht.pb.rs"));
}
Kademlia: Somewhat complete the records implementation. (#1189) * Somewhat complete the implementation of Kademlia records. This commit relates to [libp2p-146] and [libp2p-1089]. * All records expire (by default, configurable). * Provider records are also stored in the RecordStore, and the RecordStore API extended. * Background jobs for periodic (re-)replication and (re-)publication of records. Regular (value-)records are subject to re-replication and re-publication as per standard Kademlia. Provider records are only subject to re-publication. * For standard Kademlia value lookups (quorum = 1), the record is cached at the closest peer to the key that did not return the value, as per standard Kademlia. * Expiration times of regular (value-)records is computed exponentially inversely proportional to the number of nodes between the local node and the closest node known to the key (beyond the k closest), as per standard Kademlia. The protobuf messages are extended with two fields: `ttl` and `publisher` in order to implement the different semantics of re-replication (by any of the k closest peers to the key, not affecting expiry) and re-publication (by the original publisher, resetting the expiry). This is not done yet in other libp2p Kademlia implementations, see e.g. [libp2p-go-323]. The new protobuf fields have been given somewhat unique identifiers to prevent future collision. Similarly, periodic re-publication of provider records does not seem to be done yet in other implementations, see e.g. [libp2p-js-98]. [libp2p-146]: https://github.com/libp2p/rust-libp2p/issues/146 [libp2p-1089]: https://github.com/libp2p/rust-libp2p/issues/1089 [libp2p-go-323]: https://github.com/libp2p/go-libp2p-kad-dht/issues/323 [libp2p-js-98]: https://github.com/libp2p/js-libp2p-kad-dht/issues/98 * Tweak kad-ipfs example. * Add missing files. * Ensure new delays are polled immediately. To ensure task notification, since `NotReady` is returned right after. * Fix ipfs-kad example and use wasm_timer. * Small cleanup. * Incorporate some feedback. * Adjustments after rebase. * Distinguish events further. In order for a user to easily distinguish the result of e.g. a `put_record` operation from the result of a later republication, different event constructors are used. Furthermore, for now, re-replication and "caching" of records (at the closest peer to the key that did not return a value during a successful lookup) do not yield events for now as they are less interesting. * Speed up tests for CI. * Small refinements and more documentation. * Guard a node against overriding records for which it considers itself to be the publisher. * Document the jobs module more extensively. * More inline docs around removal of "unreachable" addresses. * Remove wildcard re-exports. * Use NonZeroUsize for the constants. * Re-add method lost on merge. * Add missing 'pub'. * Further increase the timeout in the ipfs-kad example. * Readd log dependency to libp2p-kad. * Simplify RecordStore API slightly. * Some more commentary. * Change Addresses::remove to return Result<(),()>. Change the semantics of `Addresses::remove` so that the error case is unambiguous, instead of the success case. Use the `Result` for clearer semantics to that effect. * Add some documentation to .
2019-07-17 14:40:48 +02:00
pub use addresses::Addresses;
Kademlia: Address some TODOs - Refactoring - API updates. (#1174) * Address some TODOs, refactor queries and public API. The following left-over issues are addressed: * The key for FIND_NODE requests is generalised to any Multihash, instead of just peer IDs. * All queries get a (configurable) timeout. * Finishing queries as soon as enough results have been received is simplified to avoid code duplication. * No more panics in provider-API-related code paths. The provider API is however still untested and (I think) still incomplete (e.g. expiration of provider records). * Numerous smaller TODOs encountered in the code. The following public API changes / additions are made: * Introduce a `KademliaConfig` with new configuration options for the replication factor and query timeouts. * Rename `find_node` to `get_closest_peers`. * Rename `get_value` to `get_record` and `put_value` to `put_record`, introducing a `Quorum` parameter for both functions, replacing the existing `num_results` parameter with clearer semantics. * Rename `add_providing` to `start_providing` and `remove_providing` to `stop_providing`. * Add a `bootstrap` function that implements a (almost) standard Kademlia bootstrapping procedure. * Rename `KademliaOut` to `KademliaEvent` with an updated list of constructors (some renaming). All events that report query results now report a `Result` to uniformly permit reporting of errors. The following refactorings are made: * Introduce some constants. * Consolidate `query.rs` and `write.rs` behind a common query interface to reduce duplication and facilitate better code reuse, introducing the notion of a query peer iterator. `query/peers/closest.rs` contains the code that was formerly in `query.rs`. `query/peers/fixed.rs` contains a modified variant of `write.rs` (which is removed). The new `query.rs` provides an interface for working with a collection of queries, taking over some code from `behaviour.rs`. * Reduce code duplication in tests and use the current_thread runtime for polling swarms to avoid spurious errors in the test output due to aborted connections when a test finishes prematurely (e.g. because a quorum of results has been collected). * Some additions / improvements to the existing tests. * Fix test. * Fix rebase. * Tweak kad-ipfs example. * Incorporate some feedback. * Provide easy access and conversion to keys in error results.
2019-07-03 16:16:25 +02:00
pub use behaviour::{Kademlia, KademliaConfig, KademliaEvent, Quorum};
pub use behaviour::{
BootstrapResult,
BootstrapOk,
BootstrapError,
GetRecordResult,
GetRecordOk,
GetRecordError,
PutRecordResult,
PutRecordOk,
PutRecordError,
GetClosestPeersResult,
GetClosestPeersOk,
GetClosestPeersError,
AddProviderResult,
AddProviderOk,
AddProviderError,
GetProvidersResult,
GetProvidersOk,
GetProvidersError,
};
pub use query::QueryId;
Kademlia: Address some TODOs - Refactoring - API updates. (#1174) * Address some TODOs, refactor queries and public API. The following left-over issues are addressed: * The key for FIND_NODE requests is generalised to any Multihash, instead of just peer IDs. * All queries get a (configurable) timeout. * Finishing queries as soon as enough results have been received is simplified to avoid code duplication. * No more panics in provider-API-related code paths. The provider API is however still untested and (I think) still incomplete (e.g. expiration of provider records). * Numerous smaller TODOs encountered in the code. The following public API changes / additions are made: * Introduce a `KademliaConfig` with new configuration options for the replication factor and query timeouts. * Rename `find_node` to `get_closest_peers`. * Rename `get_value` to `get_record` and `put_value` to `put_record`, introducing a `Quorum` parameter for both functions, replacing the existing `num_results` parameter with clearer semantics. * Rename `add_providing` to `start_providing` and `remove_providing` to `stop_providing`. * Add a `bootstrap` function that implements a (almost) standard Kademlia bootstrapping procedure. * Rename `KademliaOut` to `KademliaEvent` with an updated list of constructors (some renaming). All events that report query results now report a `Result` to uniformly permit reporting of errors. The following refactorings are made: * Introduce some constants. * Consolidate `query.rs` and `write.rs` behind a common query interface to reduce duplication and facilitate better code reuse, introducing the notion of a query peer iterator. `query/peers/closest.rs` contains the code that was formerly in `query.rs`. `query/peers/fixed.rs` contains a modified variant of `write.rs` (which is removed). The new `query.rs` provides an interface for working with a collection of queries, taking over some code from `behaviour.rs`. * Reduce code duplication in tests and use the current_thread runtime for polling swarms to avoid spurious errors in the test output due to aborted connections when a test finishes prematurely (e.g. because a quorum of results has been collected). * Some additions / improvements to the existing tests. * Fix test. * Fix rebase. * Tweak kad-ipfs example. * Incorporate some feedback. * Provide easy access and conversion to keys in error results.
2019-07-03 16:16:25 +02:00
pub use protocol::KadConnectionType;
Kademlia: Somewhat complete the records implementation. (#1189) * Somewhat complete the implementation of Kademlia records. This commit relates to [libp2p-146] and [libp2p-1089]. * All records expire (by default, configurable). * Provider records are also stored in the RecordStore, and the RecordStore API extended. * Background jobs for periodic (re-)replication and (re-)publication of records. Regular (value-)records are subject to re-replication and re-publication as per standard Kademlia. Provider records are only subject to re-publication. * For standard Kademlia value lookups (quorum = 1), the record is cached at the closest peer to the key that did not return the value, as per standard Kademlia. * Expiration times of regular (value-)records is computed exponentially inversely proportional to the number of nodes between the local node and the closest node known to the key (beyond the k closest), as per standard Kademlia. The protobuf messages are extended with two fields: `ttl` and `publisher` in order to implement the different semantics of re-replication (by any of the k closest peers to the key, not affecting expiry) and re-publication (by the original publisher, resetting the expiry). This is not done yet in other libp2p Kademlia implementations, see e.g. [libp2p-go-323]. The new protobuf fields have been given somewhat unique identifiers to prevent future collision. Similarly, periodic re-publication of provider records does not seem to be done yet in other implementations, see e.g. [libp2p-js-98]. [libp2p-146]: https://github.com/libp2p/rust-libp2p/issues/146 [libp2p-1089]: https://github.com/libp2p/rust-libp2p/issues/1089 [libp2p-go-323]: https://github.com/libp2p/go-libp2p-kad-dht/issues/323 [libp2p-js-98]: https://github.com/libp2p/js-libp2p-kad-dht/issues/98 * Tweak kad-ipfs example. * Add missing files. * Ensure new delays are polled immediately. To ensure task notification, since `NotReady` is returned right after. * Fix ipfs-kad example and use wasm_timer. * Small cleanup. * Incorporate some feedback. * Adjustments after rebase. * Distinguish events further. In order for a user to easily distinguish the result of e.g. a `put_record` operation from the result of a later republication, different event constructors are used. Furthermore, for now, re-replication and "caching" of records (at the closest peer to the key that did not return a value during a successful lookup) do not yield events for now as they are less interesting. * Speed up tests for CI. * Small refinements and more documentation. * Guard a node against overriding records for which it considers itself to be the publisher. * Document the jobs module more extensively. * More inline docs around removal of "unreachable" addresses. * Remove wildcard re-exports. * Use NonZeroUsize for the constants. * Re-add method lost on merge. * Add missing 'pub'. * Further increase the timeout in the ipfs-kad example. * Readd log dependency to libp2p-kad. * Simplify RecordStore API slightly. * Some more commentary. * Change Addresses::remove to return Result<(),()>. Change the semantics of `Addresses::remove` so that the error case is unambiguous, instead of the success case. Use the `Result` for clearer semantics to that effect. * Add some documentation to .
2019-07-17 14:40:48 +02:00
pub use record::{store, Record, ProviderRecord};
Kademlia: Address some TODOs - Refactoring - API updates. (#1174) * Address some TODOs, refactor queries and public API. The following left-over issues are addressed: * The key for FIND_NODE requests is generalised to any Multihash, instead of just peer IDs. * All queries get a (configurable) timeout. * Finishing queries as soon as enough results have been received is simplified to avoid code duplication. * No more panics in provider-API-related code paths. The provider API is however still untested and (I think) still incomplete (e.g. expiration of provider records). * Numerous smaller TODOs encountered in the code. The following public API changes / additions are made: * Introduce a `KademliaConfig` with new configuration options for the replication factor and query timeouts. * Rename `find_node` to `get_closest_peers`. * Rename `get_value` to `get_record` and `put_value` to `put_record`, introducing a `Quorum` parameter for both functions, replacing the existing `num_results` parameter with clearer semantics. * Rename `add_providing` to `start_providing` and `remove_providing` to `stop_providing`. * Add a `bootstrap` function that implements a (almost) standard Kademlia bootstrapping procedure. * Rename `KademliaOut` to `KademliaEvent` with an updated list of constructors (some renaming). All events that report query results now report a `Result` to uniformly permit reporting of errors. The following refactorings are made: * Introduce some constants. * Consolidate `query.rs` and `write.rs` behind a common query interface to reduce duplication and facilitate better code reuse, introducing the notion of a query peer iterator. `query/peers/closest.rs` contains the code that was formerly in `query.rs`. `query/peers/fixed.rs` contains a modified variant of `write.rs` (which is removed). The new `query.rs` provides an interface for working with a collection of queries, taking over some code from `behaviour.rs`. * Reduce code duplication in tests and use the current_thread runtime for polling swarms to avoid spurious errors in the test output due to aborted connections when a test finishes prematurely (e.g. because a quorum of results has been collected). * Some additions / improvements to the existing tests. * Fix test. * Fix rebase. * Tweak kad-ipfs example. * Incorporate some feedback. * Provide easy access and conversion to keys in error results.
2019-07-03 16:16:25 +02:00
Kademlia: Somewhat complete the records implementation. (#1189) * Somewhat complete the implementation of Kademlia records. This commit relates to [libp2p-146] and [libp2p-1089]. * All records expire (by default, configurable). * Provider records are also stored in the RecordStore, and the RecordStore API extended. * Background jobs for periodic (re-)replication and (re-)publication of records. Regular (value-)records are subject to re-replication and re-publication as per standard Kademlia. Provider records are only subject to re-publication. * For standard Kademlia value lookups (quorum = 1), the record is cached at the closest peer to the key that did not return the value, as per standard Kademlia. * Expiration times of regular (value-)records is computed exponentially inversely proportional to the number of nodes between the local node and the closest node known to the key (beyond the k closest), as per standard Kademlia. The protobuf messages are extended with two fields: `ttl` and `publisher` in order to implement the different semantics of re-replication (by any of the k closest peers to the key, not affecting expiry) and re-publication (by the original publisher, resetting the expiry). This is not done yet in other libp2p Kademlia implementations, see e.g. [libp2p-go-323]. The new protobuf fields have been given somewhat unique identifiers to prevent future collision. Similarly, periodic re-publication of provider records does not seem to be done yet in other implementations, see e.g. [libp2p-js-98]. [libp2p-146]: https://github.com/libp2p/rust-libp2p/issues/146 [libp2p-1089]: https://github.com/libp2p/rust-libp2p/issues/1089 [libp2p-go-323]: https://github.com/libp2p/go-libp2p-kad-dht/issues/323 [libp2p-js-98]: https://github.com/libp2p/js-libp2p-kad-dht/issues/98 * Tweak kad-ipfs example. * Add missing files. * Ensure new delays are polled immediately. To ensure task notification, since `NotReady` is returned right after. * Fix ipfs-kad example and use wasm_timer. * Small cleanup. * Incorporate some feedback. * Adjustments after rebase. * Distinguish events further. In order for a user to easily distinguish the result of e.g. a `put_record` operation from the result of a later republication, different event constructors are used. Furthermore, for now, re-replication and "caching" of records (at the closest peer to the key that did not return a value during a successful lookup) do not yield events for now as they are less interesting. * Speed up tests for CI. * Small refinements and more documentation. * Guard a node against overriding records for which it considers itself to be the publisher. * Document the jobs module more extensively. * More inline docs around removal of "unreachable" addresses. * Remove wildcard re-exports. * Use NonZeroUsize for the constants. * Re-add method lost on merge. * Add missing 'pub'. * Further increase the timeout in the ipfs-kad example. * Readd log dependency to libp2p-kad. * Simplify RecordStore API slightly. * Some more commentary. * Change Addresses::remove to return Result<(),()>. Change the semantics of `Addresses::remove` so that the error case is unambiguous, instead of the success case. Use the `Result` for clearer semantics to that effect. * Add some documentation to .
2019-07-17 14:40:48 +02:00
use std::num::NonZeroUsize;
Kademlia: Address some TODOs - Refactoring - API updates. (#1174) * Address some TODOs, refactor queries and public API. The following left-over issues are addressed: * The key for FIND_NODE requests is generalised to any Multihash, instead of just peer IDs. * All queries get a (configurable) timeout. * Finishing queries as soon as enough results have been received is simplified to avoid code duplication. * No more panics in provider-API-related code paths. The provider API is however still untested and (I think) still incomplete (e.g. expiration of provider records). * Numerous smaller TODOs encountered in the code. The following public API changes / additions are made: * Introduce a `KademliaConfig` with new configuration options for the replication factor and query timeouts. * Rename `find_node` to `get_closest_peers`. * Rename `get_value` to `get_record` and `put_value` to `put_record`, introducing a `Quorum` parameter for both functions, replacing the existing `num_results` parameter with clearer semantics. * Rename `add_providing` to `start_providing` and `remove_providing` to `stop_providing`. * Add a `bootstrap` function that implements a (almost) standard Kademlia bootstrapping procedure. * Rename `KademliaOut` to `KademliaEvent` with an updated list of constructors (some renaming). All events that report query results now report a `Result` to uniformly permit reporting of errors. The following refactorings are made: * Introduce some constants. * Consolidate `query.rs` and `write.rs` behind a common query interface to reduce duplication and facilitate better code reuse, introducing the notion of a query peer iterator. `query/peers/closest.rs` contains the code that was formerly in `query.rs`. `query/peers/fixed.rs` contains a modified variant of `write.rs` (which is removed). The new `query.rs` provides an interface for working with a collection of queries, taking over some code from `behaviour.rs`. * Reduce code duplication in tests and use the current_thread runtime for polling swarms to avoid spurious errors in the test output due to aborted connections when a test finishes prematurely (e.g. because a quorum of results has been collected). * Some additions / improvements to the existing tests. * Fix test. * Fix rebase. * Tweak kad-ipfs example. * Incorporate some feedback. * Provide easy access and conversion to keys in error results.
2019-07-03 16:16:25 +02:00
/// The `k` parameter of the Kademlia specification.
///
/// This parameter determines:
///
/// 1) The (fixed) maximum number of nodes in a bucket.
/// 2) The (default) replication factor, which in turn determines:
/// a) The number of closer peers returned in response to a request.
/// b) The number of closest peers to a key to search for in an iterative query.
///
/// The choice of (1) is fixed to this constant. The replication factor is configurable
/// but should generally be no greater than `K_VALUE`. All nodes in a Kademlia
/// DHT should agree on the choices made for (1) and (2).
///
/// The current value is `20`.
Kademlia: Somewhat complete the records implementation. (#1189) * Somewhat complete the implementation of Kademlia records. This commit relates to [libp2p-146] and [libp2p-1089]. * All records expire (by default, configurable). * Provider records are also stored in the RecordStore, and the RecordStore API extended. * Background jobs for periodic (re-)replication and (re-)publication of records. Regular (value-)records are subject to re-replication and re-publication as per standard Kademlia. Provider records are only subject to re-publication. * For standard Kademlia value lookups (quorum = 1), the record is cached at the closest peer to the key that did not return the value, as per standard Kademlia. * Expiration times of regular (value-)records is computed exponentially inversely proportional to the number of nodes between the local node and the closest node known to the key (beyond the k closest), as per standard Kademlia. The protobuf messages are extended with two fields: `ttl` and `publisher` in order to implement the different semantics of re-replication (by any of the k closest peers to the key, not affecting expiry) and re-publication (by the original publisher, resetting the expiry). This is not done yet in other libp2p Kademlia implementations, see e.g. [libp2p-go-323]. The new protobuf fields have been given somewhat unique identifiers to prevent future collision. Similarly, periodic re-publication of provider records does not seem to be done yet in other implementations, see e.g. [libp2p-js-98]. [libp2p-146]: https://github.com/libp2p/rust-libp2p/issues/146 [libp2p-1089]: https://github.com/libp2p/rust-libp2p/issues/1089 [libp2p-go-323]: https://github.com/libp2p/go-libp2p-kad-dht/issues/323 [libp2p-js-98]: https://github.com/libp2p/js-libp2p-kad-dht/issues/98 * Tweak kad-ipfs example. * Add missing files. * Ensure new delays are polled immediately. To ensure task notification, since `NotReady` is returned right after. * Fix ipfs-kad example and use wasm_timer. * Small cleanup. * Incorporate some feedback. * Adjustments after rebase. * Distinguish events further. In order for a user to easily distinguish the result of e.g. a `put_record` operation from the result of a later republication, different event constructors are used. Furthermore, for now, re-replication and "caching" of records (at the closest peer to the key that did not return a value during a successful lookup) do not yield events for now as they are less interesting. * Speed up tests for CI. * Small refinements and more documentation. * Guard a node against overriding records for which it considers itself to be the publisher. * Document the jobs module more extensively. * More inline docs around removal of "unreachable" addresses. * Remove wildcard re-exports. * Use NonZeroUsize for the constants. * Re-add method lost on merge. * Add missing 'pub'. * Further increase the timeout in the ipfs-kad example. * Readd log dependency to libp2p-kad. * Simplify RecordStore API slightly. * Some more commentary. * Change Addresses::remove to return Result<(),()>. Change the semantics of `Addresses::remove` so that the error case is unambiguous, instead of the success case. Use the `Result` for clearer semantics to that effect. * Add some documentation to .
2019-07-17 14:40:48 +02:00
pub const K_VALUE: NonZeroUsize = unsafe { NonZeroUsize::new_unchecked(20) };
Kademlia: Address some TODOs - Refactoring - API updates. (#1174) * Address some TODOs, refactor queries and public API. The following left-over issues are addressed: * The key for FIND_NODE requests is generalised to any Multihash, instead of just peer IDs. * All queries get a (configurable) timeout. * Finishing queries as soon as enough results have been received is simplified to avoid code duplication. * No more panics in provider-API-related code paths. The provider API is however still untested and (I think) still incomplete (e.g. expiration of provider records). * Numerous smaller TODOs encountered in the code. The following public API changes / additions are made: * Introduce a `KademliaConfig` with new configuration options for the replication factor and query timeouts. * Rename `find_node` to `get_closest_peers`. * Rename `get_value` to `get_record` and `put_value` to `put_record`, introducing a `Quorum` parameter for both functions, replacing the existing `num_results` parameter with clearer semantics. * Rename `add_providing` to `start_providing` and `remove_providing` to `stop_providing`. * Add a `bootstrap` function that implements a (almost) standard Kademlia bootstrapping procedure. * Rename `KademliaOut` to `KademliaEvent` with an updated list of constructors (some renaming). All events that report query results now report a `Result` to uniformly permit reporting of errors. The following refactorings are made: * Introduce some constants. * Consolidate `query.rs` and `write.rs` behind a common query interface to reduce duplication and facilitate better code reuse, introducing the notion of a query peer iterator. `query/peers/closest.rs` contains the code that was formerly in `query.rs`. `query/peers/fixed.rs` contains a modified variant of `write.rs` (which is removed). The new `query.rs` provides an interface for working with a collection of queries, taking over some code from `behaviour.rs`. * Reduce code duplication in tests and use the current_thread runtime for polling swarms to avoid spurious errors in the test output due to aborted connections when a test finishes prematurely (e.g. because a quorum of results has been collected). * Some additions / improvements to the existing tests. * Fix test. * Fix rebase. * Tweak kad-ipfs example. * Incorporate some feedback. * Provide easy access and conversion to keys in error results.
2019-07-03 16:16:25 +02:00
/// The `α` parameter of the Kademlia specification.
///
/// This parameter determines the default parallelism for iterative queries,
/// i.e. the allowed number of in-flight requests that an iterative query is
/// waiting for at a particular time while it continues to make progress towards
/// locating the closest peers to a key.
///
/// The current value is `3`.
Kademlia: Somewhat complete the records implementation. (#1189) * Somewhat complete the implementation of Kademlia records. This commit relates to [libp2p-146] and [libp2p-1089]. * All records expire (by default, configurable). * Provider records are also stored in the RecordStore, and the RecordStore API extended. * Background jobs for periodic (re-)replication and (re-)publication of records. Regular (value-)records are subject to re-replication and re-publication as per standard Kademlia. Provider records are only subject to re-publication. * For standard Kademlia value lookups (quorum = 1), the record is cached at the closest peer to the key that did not return the value, as per standard Kademlia. * Expiration times of regular (value-)records is computed exponentially inversely proportional to the number of nodes between the local node and the closest node known to the key (beyond the k closest), as per standard Kademlia. The protobuf messages are extended with two fields: `ttl` and `publisher` in order to implement the different semantics of re-replication (by any of the k closest peers to the key, not affecting expiry) and re-publication (by the original publisher, resetting the expiry). This is not done yet in other libp2p Kademlia implementations, see e.g. [libp2p-go-323]. The new protobuf fields have been given somewhat unique identifiers to prevent future collision. Similarly, periodic re-publication of provider records does not seem to be done yet in other implementations, see e.g. [libp2p-js-98]. [libp2p-146]: https://github.com/libp2p/rust-libp2p/issues/146 [libp2p-1089]: https://github.com/libp2p/rust-libp2p/issues/1089 [libp2p-go-323]: https://github.com/libp2p/go-libp2p-kad-dht/issues/323 [libp2p-js-98]: https://github.com/libp2p/js-libp2p-kad-dht/issues/98 * Tweak kad-ipfs example. * Add missing files. * Ensure new delays are polled immediately. To ensure task notification, since `NotReady` is returned right after. * Fix ipfs-kad example and use wasm_timer. * Small cleanup. * Incorporate some feedback. * Adjustments after rebase. * Distinguish events further. In order for a user to easily distinguish the result of e.g. a `put_record` operation from the result of a later republication, different event constructors are used. Furthermore, for now, re-replication and "caching" of records (at the closest peer to the key that did not return a value during a successful lookup) do not yield events for now as they are less interesting. * Speed up tests for CI. * Small refinements and more documentation. * Guard a node against overriding records for which it considers itself to be the publisher. * Document the jobs module more extensively. * More inline docs around removal of "unreachable" addresses. * Remove wildcard re-exports. * Use NonZeroUsize for the constants. * Re-add method lost on merge. * Add missing 'pub'. * Further increase the timeout in the ipfs-kad example. * Readd log dependency to libp2p-kad. * Simplify RecordStore API slightly. * Some more commentary. * Change Addresses::remove to return Result<(),()>. Change the semantics of `Addresses::remove` so that the error case is unambiguous, instead of the success case. Use the `Result` for clearer semantics to that effect. * Add some documentation to .
2019-07-17 14:40:48 +02:00
pub const ALPHA_VALUE: NonZeroUsize = unsafe { NonZeroUsize::new_unchecked(3) };
Kademlia: Address some TODOs - Refactoring - API updates. (#1174) * Address some TODOs, refactor queries and public API. The following left-over issues are addressed: * The key for FIND_NODE requests is generalised to any Multihash, instead of just peer IDs. * All queries get a (configurable) timeout. * Finishing queries as soon as enough results have been received is simplified to avoid code duplication. * No more panics in provider-API-related code paths. The provider API is however still untested and (I think) still incomplete (e.g. expiration of provider records). * Numerous smaller TODOs encountered in the code. The following public API changes / additions are made: * Introduce a `KademliaConfig` with new configuration options for the replication factor and query timeouts. * Rename `find_node` to `get_closest_peers`. * Rename `get_value` to `get_record` and `put_value` to `put_record`, introducing a `Quorum` parameter for both functions, replacing the existing `num_results` parameter with clearer semantics. * Rename `add_providing` to `start_providing` and `remove_providing` to `stop_providing`. * Add a `bootstrap` function that implements a (almost) standard Kademlia bootstrapping procedure. * Rename `KademliaOut` to `KademliaEvent` with an updated list of constructors (some renaming). All events that report query results now report a `Result` to uniformly permit reporting of errors. The following refactorings are made: * Introduce some constants. * Consolidate `query.rs` and `write.rs` behind a common query interface to reduce duplication and facilitate better code reuse, introducing the notion of a query peer iterator. `query/peers/closest.rs` contains the code that was formerly in `query.rs`. `query/peers/fixed.rs` contains a modified variant of `write.rs` (which is removed). The new `query.rs` provides an interface for working with a collection of queries, taking over some code from `behaviour.rs`. * Reduce code duplication in tests and use the current_thread runtime for polling swarms to avoid spurious errors in the test output due to aborted connections when a test finishes prematurely (e.g. because a quorum of results has been collected). * Some additions / improvements to the existing tests. * Fix test. * Fix rebase. * Tweak kad-ipfs example. * Incorporate some feedback. * Provide easy access and conversion to keys in error results.
2019-07-03 16:16:25 +02:00