* Somewhat complete the implementation of Kademlia records.
This commit relates to [libp2p-146] and [libp2p-1089].
* All records expire (by default, configurable).
* Provider records are also stored in the RecordStore, and the RecordStore
API extended.
* Background jobs for periodic (re-)replication and (re-)publication
of records. Regular (value-)records are subject to re-replication and
re-publication as per standard Kademlia. Provider records are only
subject to re-publication.
* For standard Kademlia value lookups (quorum = 1), the record is cached
at the closest peer to the key that did not return the value, as per
standard Kademlia.
* Expiration times of regular (value-)records is computed exponentially
inversely proportional to the number of nodes between the local node
and the closest node known to the key (beyond the k closest), as per
standard Kademlia.
The protobuf messages are extended with two fields: `ttl` and `publisher`
in order to implement the different semantics of re-replication (by any
of the k closest peers to the key, not affecting expiry) and re-publication
(by the original publisher, resetting the expiry). This is not done yet in
other libp2p Kademlia implementations, see e.g. [libp2p-go-323]. The new protobuf fields
have been given somewhat unique identifiers to prevent future collision.
Similarly, periodic re-publication of provider records does not seem to
be done yet in other implementations, see e.g. [libp2p-js-98].
[libp2p-146]: https://github.com/libp2p/rust-libp2p/issues/146
[libp2p-1089]: https://github.com/libp2p/rust-libp2p/issues/1089
[libp2p-go-323]: https://github.com/libp2p/go-libp2p-kad-dht/issues/323
[libp2p-js-98]: https://github.com/libp2p/js-libp2p-kad-dht/issues/98
* Tweak kad-ipfs example.
* Add missing files.
* Ensure new delays are polled immediately.
To ensure task notification, since `NotReady` is returned right after.
* Fix ipfs-kad example and use wasm_timer.
* Small cleanup.
* Incorporate some feedback.
* Adjustments after rebase.
* Distinguish events further.
In order for a user to easily distinguish the result of e.g.
a `put_record` operation from the result of a later republication,
different event constructors are used. Furthermore, for now,
re-replication and "caching" of records (at the closest peer to
the key that did not return a value during a successful lookup)
do not yield events for now as they are less interesting.
* Speed up tests for CI.
* Small refinements and more documentation.
* Guard a node against overriding records for which it considers
itself to be the publisher.
* Document the jobs module more extensively.
* More inline docs around removal of "unreachable" addresses.
* Remove wildcard re-exports.
* Use NonZeroUsize for the constants.
* Re-add method lost on merge.
* Add missing 'pub'.
* Further increase the timeout in the ipfs-kad example.
* Readd log dependency to libp2p-kad.
* Simplify RecordStore API slightly.
* Some more commentary.
* Change Addresses::remove to return Result<(),()>.
Change the semantics of `Addresses::remove` so that the error case
is unambiguous, instead of the success case. Use the `Result` for
clearer semantics to that effect.
* Add some documentation to .
The example typically runs into a lot of connection timeouts,
which may cause the query to time out. A query timeout currently
results in the example to be considered failed, but the example
should only be considered failed if no closest peers are found,
whether the query timed out or not.
* Address some TODOs, refactor queries and public API.
The following left-over issues are addressed:
* The key for FIND_NODE requests is generalised to any Multihash,
instead of just peer IDs.
* All queries get a (configurable) timeout.
* Finishing queries as soon as enough results have been received is simplified
to avoid code duplication.
* No more panics in provider-API-related code paths. The provider API is
however still untested and (I think) still incomplete (e.g. expiration
of provider records).
* Numerous smaller TODOs encountered in the code.
The following public API changes / additions are made:
* Introduce a `KademliaConfig` with new configuration options for
the replication factor and query timeouts.
* Rename `find_node` to `get_closest_peers`.
* Rename `get_value` to `get_record` and `put_value` to `put_record`,
introducing a `Quorum` parameter for both functions, replacing the
existing `num_results` parameter with clearer semantics.
* Rename `add_providing` to `start_providing` and `remove_providing`
to `stop_providing`.
* Add a `bootstrap` function that implements a (almost) standard
Kademlia bootstrapping procedure.
* Rename `KademliaOut` to `KademliaEvent` with an updated list of
constructors (some renaming). All events that report query results
now report a `Result` to uniformly permit reporting of errors.
The following refactorings are made:
* Introduce some constants.
* Consolidate `query.rs` and `write.rs` behind a common query interface
to reduce duplication and facilitate better code reuse, introducing
the notion of a query peer iterator. `query/peers/closest.rs`
contains the code that was formerly in `query.rs`. `query/peers/fixed.rs` contains
a modified variant of `write.rs` (which is removed). The new `query.rs`
provides an interface for working with a collection of queries, taking
over some code from `behaviour.rs`.
* Reduce code duplication in tests and use the current_thread runtime for
polling swarms to avoid spurious errors in the test output due to aborted
connections when a test finishes prematurely (e.g. because a quorum of
results has been collected).
* Some additions / improvements to the existing tests.
* Fix test.
* Fix rebase.
* Tweak kad-ipfs example.
* Incorporate some feedback.
* Provide easy access and conversion to keys in error results.
* initial implementation of the records
* move to multihash keys
* correctly process query results
* comments and formatting
* correctly return closer_peers in query
* checking wrong peer id in test
* Apply suggestions from code review
Co-Authored-By: Roman Borschel <romanb@users.noreply.github.com>
* Fix changes from suggestions
* Send responses to PUT_VALUE requests
* Shortcut in get_value
* Update protocols/kad/src/behaviour.rs
Co-Authored-By: Roman Borschel <romanb@users.noreply.github.com>
* Revert "Update protocols/kad/src/behaviour.rs"
This reverts commit 579ce742a7f4c94587f1e1f0866d2a3a37418efb.
* Remove duplicate insertion
* Adds a record to a PUT_VALUE response
* Fix a racy put_value test
* Store value ourselves only if we are in K closest
* Abstract over storage
* Revert "Abstract over storage": bad take
This reverts commit eaebf5b6d915712eaf3b05929577fdf697f204d8.
* Abstract over records storage using hashmap as default
* Constructor for custom records
* New Record type and its traits
* Fix outdated storage name
* Fixes returning an event
* Change FindNodeReq key type to Multihash
* WriteState for a second stage of a PUT_VALUE request
* GET_VALUE should not have a record
* Refactor a match arm
* Add successes and failures counters to PutValueRes
* If value is found no need to return closer peers
* Remove a custo storage from tests
* Rename a test to get_value_not_found
* Adds a TODO to change FindNode request key to Multihash
Co-Authored-By: Roman Borschel <romanb@users.noreply.github.com>
* Move MemoryRecordStorage to record.rs
* Return a Cow-ed Record from get
* Fix incorrect GET_VALUE parsing
* Various fixes with review
* Fixes get_value_not_found
* Fix peerids names in test
* another fix
* PutValue correctly distributes values
* Simplify the test
* Check that results are actually the closest
* Reverts changes to tests
* Fix the test topology and checking the results
* Run put_value test ten times
* Adds a get_value test
* Apply suggestions from code review
Co-Authored-By: Roman Borschel <romanb@users.noreply.github.com>
* Make Record fields public
* Moves WriteState to write.rs
* A couple of minor fixes
* Another few fixes of review
* Simplify the put_value test
* Dont synchronously return an error from put_value
* Formatting fixes and comments
* Collect a bunch of results
* Take exactly as much elements as neede
* Check if the peer is still connected
* Adds a multiple GetValueResults results number test
* Unnecessary mut iterators in put_value
* Ask for num_results in get_value
* Dont allocate twice in get_value
* Dont count same errored peer multiple times
* Apply suggestions from code review
Co-Authored-By: Roman Borschel <romanb@users.noreply.github.com>
* Fix another review
* Apply suggestions from code review
Co-Authored-By: Pierre Krieger <pierre.krieger1708@gmail.com>
* Bring back FromIterator and improve a panic message
* Update protocols/kad/src/behaviour.rs
Co-Authored-By: Pierre Krieger <pierre.krieger1708@gmail.com>
* Kademlia: Optimise iteration over closest entries.
The current implementation for finding the entries whose keys are closest
to some target key in the Kademlia routing table involves copying the
keys of all buckets into a new `Vec` which is then sorted based on the
distances to the target and turned into an iterator from which only a
small number of elements (by default 20) are drawn.
This commit introduces an iterator over buckets for finding the closest
keys to a target that visits the buckets in the optimal order, based on
the information contained in the distance bit-string representing the
distance between the local key and the target.
Correctness is tested against full-table scans.
Also included:
* Updated documentation.
* The `Entry` API was moved to the `kbucket::entry` sub-module for
ease of maintenance.
* The pending node handling has been slightly refactored in order to
bring code and documentation in agreement and clarify the semantics
a little.
* Rewrite pending node handling and add tests.
This is now a very simple option serving multiple purposes:
* It allows for stable (integration) tests involving a Swarm, which
are otherwise subject to race conditions due to the connection being
allowed to terminate at any time with `KeepAlive::No`
(which remains the default).
* It makes for a more entertaining ping example which continuously
sends pings.
* Maybe someone wants to use the ping protocol for application-layer
connection keep-alive after all.
* libp2p-ping improvements.
* re #950: Removes use of the `OneShotHandler`, but still sending each
ping over a new substream, as seems to be intentional since #828.
* re #842: Adds an integration test that exercises the ping behaviour through
a Swarm, requiring the RTT to be below a threshold. This requires disabling
Nagle's algorithm as it can interact badly with delayed ACKs (and has been
observed to do so in the context of the new ping example and integration test).
* re #864: Control of the inbound and outbound (sub)stream protocol upgrade
timeouts has been moved from the `NodeHandlerWrapperBuilder` to the
`ProtocolsHandler`. That may also alleviate the need for a custom timeout
on an `OutboundSubstreamRequest` as a `ProtocolsHandler` is now free to
adjust these timeouts over time.
Other changes:
* A new ping example.
* Documentation improvements.
* More documentation improvements.
* Add PingPolicy and ensure no event is dropped.
* Remove inbound_timeout/outbound_timeout.
As per review comment, the inbound timeout is now configured
as part of the `listen_protocol` and the outbound timeout as
part of the `OutboundSubstreamRequest`.
* Simplify and generalise.
Generalise `ListenProtocol` to `SubstreamProtocol`, reusing it in
the context of `ProtocolsHandlerEvent::OutboundSubstreamRequest`.
* Doc comments for SubstreamProtocol.
* Adapt to changes in master.
* Relax upper bound for ping integration test rtt.
For "slow" CI build machines?
* Consolidate keypairs in core.
Introduce the concept of a node's identity keypair in libp2p-core,
instead of only the public key:
* New module: libp2p_core::identity with submodules for the currently
supported key types. An identity::Keypair and identity::PublicKey
support the creation and verification of signatures. The public key
supports encoding/decoding according to the libp2p specs.
* The secio protocol is simplified as a result of moving code to libp2p-core.
* The noise protocol is slightly simplified by consolidating ed25519
keypairs in libp2p-core and using x25519-dalek for DH. Furthermore,
Ed25519 to X25519 keypair conversion is now complete and tested.
Generalise over the DH keys in the noise protocol.
Generalise over the DH keys and thus DH parameter in handshake patterns
of the Noise protocol, such that it is easy to support other DH schemes
in the future, e.g. X448.
* Address new review comments.
* Embed the topology in the NetworkBehaviour
* Put topologies inside of Floodsub and Kad
* Fix core tests
* Fix chat example
* More work
* Some cleanup
* Restore external addresses system
* Move the MdnsService to a service module
* Add a Mdns behaviour
* Add a Mdns network behaviour
* Add minor documentation
* Add minor todo
* Use nat_traversal on the observed address
* Don't add self to topology
* Automatically connect to nodes we discover
* Add Debug implementations
* Fix example
* Rework Kademlia for the new design
* Minor work on protocol.rs
* More work
* Remove QueryTarget::FindValue
* Finish work on query
* Query timeout test
* Work on topology
* More work
* Update protocols/kad/src/topology.rs
Co-Authored-By: tomaka <pierre.krieger1708@gmail.com>
* Fix trailing whitespaces
* Use if let
* Add a floodsub example with the new Swarm
* WIP fixes for Floodsub chat-example code
* cleanup
* Address grumbles
* Update docs
* Update to newest master
Refactor multiaddr crate.
- Remove `AddrComponent`. Instead `Protocol` directly contains its
associated data.
- Various smaller changes around conversions to Multiaddr from other
types, e.g. socket addresses.
- Expand tests to include property tests which test encoding/decoding
identity.
* Introduce NULL cipher and allow more configuration.
* Back to using the hash-code for handshake.
Using `Endpoint` would be incompatible with the existing protocol.
* Add comments.
* The SwarmFuture is now a Stream
* Return the produced future in the message
* Remove IncomingConnection event
* Pass error when failing to dial
* Fix loop break mistake
* Fix concern
* Rename SwarmFuture to SwarmEvents
* Increase type length limit
* Remove todo
