Changes needed to get libp2p to run via `wasm32-unknown-unknown` in the browser
(both main thread and inside web workers).
Replaces wasm-timer with futures-timer and instant.
Co-authored-by: Oliver Wangler <oliver@wngr.de>
* Make clippy "happy".
Address all clippy complaints that are not purely stylistic (or even
have corner cases with false positives). Ignore all "style" and "pedantic" lints.
* Fix tests.
* Undo unnecessary API change.
* feat: upgrade to multihash 0.13
`multihash` changes a lot internally, it is using stack allocation instead
of heap allocation. This leads to a few limitations in regards on how
`Multihash` can be used.
Therefore `PeerId` is now using a `Bytes` internally so that only minimal
changes are needed.
* Update versions and changelogs.
Co-authored-by: Roman Borschel <romanb@users.noreply.github.com>
Co-authored-by: Roman S. Borschel <roman@parity.io>
The extension paper S-Kademlia includes a proposal for lookups over
disjoint paths. Within vanilla Kademlia, queries keep track of the
closest nodes in a single bucket. Any adversary along the path can thus
influence all future paths, in case they can come up with the
next-closest (not overall closest) hops. S-Kademlia tries to solve the
attack above by querying over disjoint paths using multiple buckets.
To adjust the libp2p Kademlia implementation accordingly this change-set
introduces an additional peers iterator: `ClosestDisjointPeersIter`.
This new iterator wraps around a set of `ClosestPeersIter`
`ClosestDisjointPeersIter` enforces that each of the `ClosestPeersIter`
explore disjoint paths by having each peer instantly return that was
queried by a different iterator before.
* [libp2p-kad] Provide more insight and control into Kademlia queries.
More insight: The API allows iterating over the active queries and
inspecting their state and execution statistics.
More control: The API allows aborting queries prematurely
at any time.
To that end, API operations that initiate new queries return the query ID
and multi-phase queries such as `put_record` retain the query ID across all
phases, each phase being executed by a new (internal) query.
* Cleanup
* Cleanup
* Update examples and re-exports.
* Incorporate review feedback.
* Update CHANGELOG
* Update CHANGELOG
Co-authored-by: Max Inden <mail@max-inden.de>
`FixedPeersIter` requires the initial set of peers to be passed as
`PeerId`s and not as `Key<PeerId>`s. This commit removes the unnecessary
conversion.
Instead of creating unconstrained random number generators in quickcheck
tests to generate test data, have quickcheck provide a `Seed` to seed
those random number generators and thus make the test execution
deterministic / reproducible.
Make sure to decrease `num_waiting` when being notified of a peer
failure to allow an additional peer to be queried.
Given that `FixedPeersIter` is initialized with `replication_factor` by
`QueryPool` this bug will not surface today.
* protocols/kad/query/peers/closest: Consider K_VALUE nodes at init
By considering `K_VALUE` at `ClosestPeersIter` initialization, the initial peer
set length is independent of `num_results` and thus of the `replication_factor`.
* protocols/kad/src/behaviour/test: Enable building single nodes
Introduces the `build_node` function to build a single not connected
node. Along the way replace the notion of a `port_base` with returning
the actual `Multiaddr` of the node.
* protocols/kad/behaviour/test: Fix bootstrap test initialization
When looking for the closest node to a key, Kademlia considers
ALPHA_VALUE nodes to query at initialization. If `num_groups` is larger
than ALPHA_VALUE the remaining locally known nodes will not be
considered. Given that no other node is aware of them other than node 1,
they would be lost entirely. To prevent the above restrict `num_groups`
to be equal or smaller than ALPHA_VALUE.
* protocols/kad/behaviour/test: Fix put_record and get_provider
In the past, when trying to find the closest nodes to a key, Kademlia
would consider `num_result` amount of nodes to query out of all the
nodes it is aware of.
Both the `put_record` and the `get_provider` tests initialized their
swarms in the same way. The tests took the replication factor to use as
an input. The number of results to get was equal to the replication
factor. The amount of swarms to start was twice the replication factor.
Nodes would be grouped in two groups a replication factor nodes. The
first node would be aware of all of the nodes in the first group. The
last node of the first group would be aware of all the nodes in the
second group.
By coincidence (I assume) these numbers played together very well. At
initialization the first node would consider `num_results` amount of
peers (see first paragraph). It would then contact each of them. As the
first node is aware of the last node of the first group which in turn is
aware of all nodes in the second group, the first node would eventually
discover all nodes.
Recently the amount of nodes Kademlia considers at initialization when
looking for the nodes closest to a key was changed to only consider
ALPHA nodes.
With this in mind playing through the test setup above again would
result in (1) `replication_factor - ALPHA` nodes being entirely lost as
the first node would never consider them and (2) the first node probably
never contacting the last node out of the first group and thus not
discovering any nodes of the second group.
To keep the multi hop discovery in place while not basing ones test
setup on the lucky assumption of Kademlia considering replication factor
amount of nodes at initialization, this patch alters the two tests:
Build a fully connected set of nodes and one addition node (the first
node). Connect the first node to a single node of the fully connected
set (simulating a boot node). Continue as done previously.
Co-authored-by: Roman Borschel <romanb@users.noreply.github.com>
When not making progress for `parallelism` time a `ClosestPeersIter`
becomes `State::Stalled`. When stalled an iterator is allowed to make
more parallel requests up to `num_results`. If `num_results` is smaller
than `parallelism` make sure to still allow up to `parallelism` requests
in-flight.
Co-Authored-By: Roman Borschel <romanb@users.noreply.github.com>
* Fix broken links in rustdoc
This fixes all of the rustdoc warnings on nightly.
* Check documentation intra-link
* Fix config
* Fix bad indent
* Make nightly explicit
* More links fixes
* Fix link broken after master merge
Co-authored-by: Demi Obenour <48690212+DemiMarie-parity@users.noreply.github.com>
* Somewhat complete the implementation of Kademlia records.
This commit relates to [libp2p-146] and [libp2p-1089].
* All records expire (by default, configurable).
* Provider records are also stored in the RecordStore, and the RecordStore
API extended.
* Background jobs for periodic (re-)replication and (re-)publication
of records. Regular (value-)records are subject to re-replication and
re-publication as per standard Kademlia. Provider records are only
subject to re-publication.
* For standard Kademlia value lookups (quorum = 1), the record is cached
at the closest peer to the key that did not return the value, as per
standard Kademlia.
* Expiration times of regular (value-)records is computed exponentially
inversely proportional to the number of nodes between the local node
and the closest node known to the key (beyond the k closest), as per
standard Kademlia.
The protobuf messages are extended with two fields: `ttl` and `publisher`
in order to implement the different semantics of re-replication (by any
of the k closest peers to the key, not affecting expiry) and re-publication
(by the original publisher, resetting the expiry). This is not done yet in
other libp2p Kademlia implementations, see e.g. [libp2p-go-323]. The new protobuf fields
have been given somewhat unique identifiers to prevent future collision.
Similarly, periodic re-publication of provider records does not seem to
be done yet in other implementations, see e.g. [libp2p-js-98].
[libp2p-146]: https://github.com/libp2p/rust-libp2p/issues/146
[libp2p-1089]: https://github.com/libp2p/rust-libp2p/issues/1089
[libp2p-go-323]: https://github.com/libp2p/go-libp2p-kad-dht/issues/323
[libp2p-js-98]: https://github.com/libp2p/js-libp2p-kad-dht/issues/98
* Tweak kad-ipfs example.
* Add missing files.
* Ensure new delays are polled immediately.
To ensure task notification, since `NotReady` is returned right after.
* Fix ipfs-kad example and use wasm_timer.
* Small cleanup.
* Incorporate some feedback.
* Adjustments after rebase.
* Distinguish events further.
In order for a user to easily distinguish the result of e.g.
a `put_record` operation from the result of a later republication,
different event constructors are used. Furthermore, for now,
re-replication and "caching" of records (at the closest peer to
the key that did not return a value during a successful lookup)
do not yield events for now as they are less interesting.
* Speed up tests for CI.
* Small refinements and more documentation.
* Guard a node against overriding records for which it considers
itself to be the publisher.
* Document the jobs module more extensively.
* More inline docs around removal of "unreachable" addresses.
* Remove wildcard re-exports.
* Use NonZeroUsize for the constants.
* Re-add method lost on merge.
* Add missing 'pub'.
* Further increase the timeout in the ipfs-kad example.
* Readd log dependency to libp2p-kad.
* Simplify RecordStore API slightly.
* Some more commentary.
* Change Addresses::remove to return Result<(),()>.
Change the semantics of `Addresses::remove` so that the error case
is unambiguous, instead of the success case. Use the `Result` for
clearer semantics to that effect.
* Add some documentation to .
* Address some TODOs, refactor queries and public API.
The following left-over issues are addressed:
* The key for FIND_NODE requests is generalised to any Multihash,
instead of just peer IDs.
* All queries get a (configurable) timeout.
* Finishing queries as soon as enough results have been received is simplified
to avoid code duplication.
* No more panics in provider-API-related code paths. The provider API is
however still untested and (I think) still incomplete (e.g. expiration
of provider records).
* Numerous smaller TODOs encountered in the code.
The following public API changes / additions are made:
* Introduce a `KademliaConfig` with new configuration options for
the replication factor and query timeouts.
* Rename `find_node` to `get_closest_peers`.
* Rename `get_value` to `get_record` and `put_value` to `put_record`,
introducing a `Quorum` parameter for both functions, replacing the
existing `num_results` parameter with clearer semantics.
* Rename `add_providing` to `start_providing` and `remove_providing`
to `stop_providing`.
* Add a `bootstrap` function that implements a (almost) standard
Kademlia bootstrapping procedure.
* Rename `KademliaOut` to `KademliaEvent` with an updated list of
constructors (some renaming). All events that report query results
now report a `Result` to uniformly permit reporting of errors.
The following refactorings are made:
* Introduce some constants.
* Consolidate `query.rs` and `write.rs` behind a common query interface
to reduce duplication and facilitate better code reuse, introducing
the notion of a query peer iterator. `query/peers/closest.rs`
contains the code that was formerly in `query.rs`. `query/peers/fixed.rs` contains
a modified variant of `write.rs` (which is removed). The new `query.rs`
provides an interface for working with a collection of queries, taking
over some code from `behaviour.rs`.
* Reduce code duplication in tests and use the current_thread runtime for
polling swarms to avoid spurious errors in the test output due to aborted
connections when a test finishes prematurely (e.g. because a quorum of
results has been collected).
* Some additions / improvements to the existing tests.
* Fix test.
* Fix rebase.
* Tweak kad-ipfs example.
* Incorporate some feedback.
* Provide easy access and conversion to keys in error results.
