- Change `PublicKey::into_protobuf_encoding` to
`PublicKey::to_protobuf_encoding`.
- Change `PublicKey::into_peer_id` to `PublicKey::to_peer_id`.
- Change `PeerId::from_public_key(PublicKey)` to
`PeerId::from_public_key(&PublicKey)`.
- Add `From<&PublicKey> for PeerId`.
Co-authored-by: Max Inden <mail@max-inden.de>
Add `ExpandedSwarm::disconnect_peer_id` and
`NetworkBehaviourAction::CloseConnection` to close connections to a specific
peer via an `ExpandedSwarm` or `NetworkBehaviour`.
Co-authored-by: Max Inden <mail@max-inden.de>
Remove `Deref` and `DerefMut` implementations previously dereferencing
to the `NetworkBehaviour` on `Swarm`. Instead one can access the
`NetworkBehaviour` via `Swarm::behaviour` and `Swarm::behaviour_mut`.
Methods on `Swarm` can now be accessed directly, e.g. via
`my_swarm.local_peer_id()`.
Reasoning: Accessing the `NetworkBehaviour` of a `Swarm` through `Deref`
and `DerefMut` instead of a method call is an unnecessary complication,
especially for newcomers. In addition, `Swarm` is not a smart-pointer
and should thus not make use of `Deref` and `DerefMut`, see documentation
from the standard library below.
> Deref should only be implemented for smart pointers to avoid
confusion.
https://doc.rust-lang.org/std/ops/trait.Deref.html
* Make clippy "happy".
Address all clippy complaints that are not purely stylistic (or even
have corner cases with false positives). Ignore all "style" and "pedantic" lints.
* Fix tests.
* Undo unnecessary API change.
Currently, Gossipsub messages may not be published on a topic if there are no
subscribed peers on that topic, however they are being added to a duplicate
cache. If the messages are content-addressed, this prevents the user from
attempting to republish the message in the future.
This commit modifies the logic such that messages are only added to the caches
(duplicate, memcache and published_messages) if the message could be published
to at least one peer.
Previously, a failed publish would be added the memcache. This meant that the
router would gossip about a failed publish, such that if a peer joined the topic
after the failed publish (but within the gossip history time window) the message
could still be propagated to the newly connected peers. With this PR, a failed
publish now definitively indicates the message will not be published on the
network and the user is required to re-submit at a later time, once sufficient
peers exist on the topic.
`futures-codec` has not been updated in the recent months. It still
depends on `bytes` `v0.5` preventing all downstream dependencies to
upgrade to `bytes` `v1.0`.
This commit replaces `futures_codec` in favor of `asynchronous-codec`
The latter is a fully upgraded fork of the former.
In addition this commit upgrades:
- bytes to v1
- unsigned-varint to v0.6.0
- prost to v0.7
This commit upgrades the current gossipsub implementation to support the [v1.1
spec](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md).
It adds a number of features, bug fixes and performance improvements.
Besides support for all new 1.1 features, other improvements that are of particular note:
- Improved duplicate LRU-time cache (this was previously a severe bottleneck for
large message throughput topics)
- Extended message validation configuration options
- Arbitrary topics (users can now implement their own hashing schemes)
- Improved message validation handling - Invalid messages are no longer dropped
but sent to the behaviour for application-level processing (including scoring)
- Support for floodsub, gossipsub v1 and gossipsub v2
- Protobuf encoding has been shifted into the behaviour. This has permitted two
improvements:
1. Message size verification during publishing (report to the user if the
message is too large before attempting to send).
2. Message fragmentation. If an RPC is too large it is fragmented into its
sub components and sent in smaller chunks.
Additional Notes
The peer eXchange protocol defined in the v1.1 spec is inactive in its current
form. The current implementation permits sending `PeerId` in `PRUNE` messages,
however a `PeerId` is not sufficient to form a new connection to a peer. A
`Signed Address Record` is required to safely transmit peer identity
information. Once these are confirmed (https://github.com/libp2p/specs/pull/217)
a future PR will implement these and make PX usable.
Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: Rüdiger Klaehn <rklaehn@protonmail.com>
Co-authored-by: blacktemplar <blacktemplar@a1.net>
Co-authored-by: Rüdiger Klaehn <rklaehn@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Roman S. Borschel <roman@parity.io>
Co-authored-by: Roman Borschel <romanb@users.noreply.github.com>
Co-authored-by: David Craven <david@craven.ch>
* Add "infinite" scores for external addresses.
Extend address scores with an infinite cardinal, permitting
addresses to be retained "forever" or until explicitly removed.
Expose (external) address scores on the API.
* Update swarm/src/registry.rs
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
* Fix compilation.
* Update CHANGELOG
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
This adds optional message signing and verification to the gossipsub protocol as
per the libp2p specifications.
In addition this commit:
- Removes the LruCache received cache and simply uses the memcache in it's
place.
- Send subscriptions to all peers
- Prevent invalid messages from being gossiped
- Send grafts when subscriptions are added to the mesh
Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: Rüdiger Klaehn <rklaehn@protonmail.com>
Co-authored-by: Rüdiger Klaehn <rklaehn@gmail.com>
* Add debug instances for MessageCache and GossipSub behaviour
* Manual impl of Debug for GossipsubMessage
* Add pretty printing of protocol_id in debug
* Use hex_fmt instead of hex
* Inline StringOrBytes helper struct
Since it is used only once here
* Limit data of gossipsub msg to 20 bytes
Otherwise they might become very large and useless for debugging
- Update `libp2p-kad` CHANGELOG and increment version to 0.20.1.
- Update `libp2p` version to 0.20.1.
- Fix some linter warning in `libp2p-gossipsub` and increment
version to 0.19.3 and update CHANGELOG.
If a user sends a message that is over the maximum transmission size gossipsub
will disconnect from the peer being sent the message.
This PR updates the logic to simply emit an error, not send the over-sized
message but maintain the long-lived streams for future messages.
Co-authored-by: Age Manning <Age@AgeManning.com>
* [libp2p-swarm] Make the multiple connections per peer first-class.
This commit makes the notion of multiple connections per peer
first-class in the API of libp2p-swarm, introducing the new
callbacks `inject_connection_established` and
`inject_connection_closed`. The `endpoint` parameter from
`inject_connected` and `inject_disconnected` is removed,
since the first connection to open may not be the last
connection to close, i.e. it cannot be guaranteed,
as was previously the case, that the endpoints passed
to these callbacks match up.
* Have identify track all addresses.
So that identify requests can be answered with the correct
observed address of the connection on which the request
arrives.
* Cleanup
* Cleanup
* Improve the `Peer` state API.
* Remove connection ID from `SwarmEvent::Dialing`.
* Mark `DialPeerCondition` non-exhaustive.
* Re-encapsulate `NetworkConfig`.
To retain the possibility of not re-exposing all
network configuration choices, thereby providing
a more convenient API on the \`SwarmBuilder\`.
* Rework Swarm::dial API.
* Update CHANGELOG.
* Doc formatting tweaks.
* Allow multiple connections per peer in libp2p-core.
Instead of trying to enforce a single connection per peer,
which involves quite a bit of additional complexity e.g.
to prioritise simultaneously opened connections and can
have other undesirable consequences [1], we now
make multiple connections per peer a feature.
The gist of these changes is as follows:
The concept of a "node" with an implicit 1-1 correspondence
to a connection has been replaced with the "first-class"
concept of a "connection". The code from `src/nodes` has moved
(with varying degrees of modification) to `src/connection`.
A `HandledNode` has become a `Connection`, a `NodeHandler` a
`ConnectionHandler`, the `CollectionStream` was the basis for
the new `connection::Pool`, and so forth.
Conceptually, a `Network` contains a `connection::Pool` which
in turn internally employs the `connection::Manager` for
handling the background `connection::manager::Task`s, one
per connection, as before. These are all considered implementation
details. On the public API, `Peer`s are managed as before through
the `Network`, except now the API has changed with the shift of focus
to (potentially multiple) connections per peer. The `NetworkEvent`s have
accordingly also undergone changes.
The Swarm APIs remain largely unchanged, except for the fact that
`inject_replaced` is no longer called. It may now practically happen
that multiple `ProtocolsHandler`s are associated with a single
`NetworkBehaviour`, one per connection. If implementations of
`NetworkBehaviour` rely somehow on communicating with exactly
one `ProtocolsHandler`, this may cause issues, but it is unlikely.
[1]: https://github.com/paritytech/substrate/issues/4272
* Fix intra-rustdoc links.
* Update core/src/connection/pool.rs
Co-Authored-By: Max Inden <mail@max-inden.de>
* Address some review feedback and fix doc links.
* Allow responses to be sent on the same connection.
* Remove unnecessary remainders of inject_replaced.
* Update swarm/src/behaviour.rs
Co-Authored-By: Pierre Krieger <pierre.krieger1708@gmail.com>
* Update swarm/src/lib.rs
Co-Authored-By: Pierre Krieger <pierre.krieger1708@gmail.com>
* Update core/src/connection/manager.rs
Co-Authored-By: Pierre Krieger <pierre.krieger1708@gmail.com>
* Update core/src/connection/manager.rs
Co-Authored-By: Pierre Krieger <pierre.krieger1708@gmail.com>
* Update core/src/connection/pool.rs
Co-Authored-By: Pierre Krieger <pierre.krieger1708@gmail.com>
* Incorporate more review feedback.
* Move module declaration below imports.
* Update core/src/connection/manager.rs
Co-Authored-By: Toralf Wittner <tw@dtex.org>
* Update core/src/connection/manager.rs
Co-Authored-By: Toralf Wittner <tw@dtex.org>
* Simplify as per review.
* Fix rustoc link.
* Add try_notify_handler and simplify.
* Relocate DialingConnection and DialingAttempt.
For better visibility constraints.
* Small cleanup.
* Small cleanup. More robust EstablishedConnectionIter.
* Clarify semantics of `DialingPeer::connect`.
* Don't call inject_disconnected on InvalidPeerId.
To preserve the previous behavior and ensure calls to
`inject_disconnected` are always paired with calls to
`inject_connected`.
* Provide public ConnectionId constructor.
Mainly needed for testing purposes, e.g. in substrate.
* Move the established connection limit check to the right place.
* Clean up connection error handling.
Separate connection errors into those occuring during
connection setup or upon rejecting a newly established
connection (the `PendingConnectionError`) and those
errors occurring on previously established connections,
i.e. for which a `ConnectionEstablished` event has
been emitted by the connection pool earlier.
* Revert change in log level and clarify an invariant.
* Remove inject_replaced entirely.
* Allow notifying all connection handlers.
Thereby simplify by introducing a new enum `NotifyHandler`,
used with a single constructor `NetworkBehaviourAction::NotifyHandler`.
* Finishing touches.
Small API simplifications and code deduplication.
Some more useful debug logging.
Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
Co-authored-by: Toralf Wittner <tw@dtex.org>
* Fix broken links in rustdoc
This fixes all of the rustdoc warnings on nightly.
* Check documentation intra-link
* Fix config
* Fix bad indent
* Make nightly explicit
* More links fixes
* Fix link broken after master merge
Co-authored-by: Demi Obenour <48690212+DemiMarie-parity@users.noreply.github.com>
* Simplify trait bounds requirements
* More work
* Moar
* Finish
* Fix final tests
* More simplification
* Use separate traits for Inbound/Outbound
* Update gossipsub and remove warnings
* Add documentation to swarm
* Remove BoxSubstream
* Fix tests not compiling
* Fix stack overflow
* Address concerns
* For some reason my IDE ignored libp2p-kad
Instead of closing the connection on an error on the inbound substream
(both on `poll_next` as well as `poll_close`), one can set it to None
and depend on the remote to open an new one in case they have more data
to send.
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
* Create gossipsub crate - Basic template, borrowed from floodsub
* Add a GossipsubConfig struct and set up basic structures in the Gossipsub struct
* Begin implementation of join. Adds get_random_peers helper function and adds tests
* Implements gossipsub leave()
* Update publishMany to incorporate gossipsub mesh and fanout logic
* Use the gossipsub mesh for determining peer subscription
* Remove subscribed_topics field from the Gossipsub struct
* Rename gossipsubconfig to ProtocolConfig
* Implement the gossipsub control messages into the Codec's Encode/Decode and modifies GossipsubRpc
* Modify GossipsubActions to enums for succinctness.
* Modify the memcache to store Gossipsub messages
* Implement control message handling.
* Update control message handling to handle multiple messages.
* Handle received gossipsub messages using pre-built handlers.
* Remove excess connected peer hashmap
* Add extra peer mapping and consistent topic naming.
* Implement heartbeat, emit_gossip and send_graft_prune.
* Group logic in forwarding messages. Add messages to memcache.
* Add heartbeat timer and move location of helper function.
* Add gossipsub the libp2p workspace, makes layer structs public
* Add logging to gossipsub
- Adds the log crate and implements logging macros
- Specifies versions for external crates
* Add example chat for debugging purposes
* Implement #868 for gossipsub.
* Add rust documentation to gossipsub crate.
- Adds basic documentation, overview and examples to the gossipsub
crate.
* Re-introduce the initial heartbeat time config.
This commit also adds the inject_connected test.
* Add subscribe tests.
- Modifies `handle_received_subscriptions` to take a reference of
subscriptions
- Adds `test_subscribe`
- Adds `test_handle_received_subscriptions`
- Adds tests for the filter in `get_random_peers`
* Add Bug fixes and further testing for gossipsub.
- Corrects the tuple use of topic_hashes
- Corrects JOIN logic around fanout and adding peers to the mesh
- Adds test_unsubscribe
- Adds test_join
* Rename GossipsubMessage::msg_id -> id
* Add bug fix for handling disconnected peers.
* Implements (partially) #889 for Gossipsub.
* handle_iwant event count tests
* handle_ihave event count tests
* Move layer.rs tests into separate file.
* Implement clippy suggestions for gossipsub.
* Modify control message tests for specific types.
* Implement builder pattern for GossipsubConfig.
As suggested by @twittner - The builder pattern for building
GossipsubConfig struct is implemented.
* Package version updates as suggested by @twittner.
* Correct line lengths in gossipsub.
* Correct braces in found by @twittner.
* Implement @twittner's suggestions.
- Uses `HashSet` where applicable
- Update `FnvHashMap` to standard `HashMap`
- Uses `min` function in code simplification.
* Add NodeList struct to clarify topic_peers.
* Cleaner handling of messagelist
Co-Authored-By: AgeManning <Age@AgeManning.com>
* Cleaner handling of added peers.
Co-Authored-By: AgeManning <Age@AgeManning.com>
* handle_prune peer removed test
* basic grafting tests
* multiple topic grafting test
* Convert &vec to slice.
Co-Authored-By: AgeManning <Age@AgeManning.com>
* Convert to lazy insert.
Co-Authored-By: AgeManning <Age@AgeManning.com>
* Cleaner topic handling.
Co-Authored-By: AgeManning <Age@AgeManning.com>
* control pool piggybacking
using HashMap.drain() in control_pool_flush
going to squash this
* Add Debug derives to gossipsub and correct tests.
* changes from PR
squash this
all tests passing, but still some that need to be reconsidered
test reform
* Implements Arc for GossipsubRpc events
* Remove support for floodsub nodes
* Reconnected to disconnected peers, to mitigate timeout
* Use ReadOne WriteOne with configurable max gossip sizes
* Remove length delimination from RPC encoding
* Prevent peer duplication in mesh
* Allow oneshot handler's inactivity_timeout to be configurable
* Correct peer duplication in mesh bug
* Remove auto-reconnect to allow for user-level disconnects
* Single long-lived inbound/outbound streams to match go implementation
* Allow gossipsub topics to be optionally hashable
* Improves gossipsub stream handling
- Corrects the handler's keep alive.
- Correct the chat example.
- Instantly add peers to the mesh on subscription if the mesh is low.
* Allows message validation in gossipsub
* Replaces Cuckoofilter with LRUCache
The false positive rate was unacceptable for rejecting messages.
* Renames configuration parameter and corrects logic
* Removes peer from fanout on disconnection
* Add publish and fanout tests
* Apply @mxinden suggestions
* Resend message if outbound stream negotiated
- Downgrades log warnings
* Implement further reviewer suggestions
- Created associated functions to avoid unnecessary cloning
- Messages are rejected if their sequence numbers are not u64
- `GossipsbuConfigBuilder` has the same defaults as `GossipsubConfig`
- Miscellaneous typos
* Add MessageId type and remove unnecessary comments
* Add a return value to propagate_message function
* Adds user-customised gossipsub message ids
* Adds the message id to GossipsubEvent
* Implement Debug for GossipsubConfig
* protocols/gossipsub: Add basic smoke test
Implement a basic smoke test that:
1. Builds a fully connected graph of size N.
2. Subscribes each node to the same topic.
3. Publishes a single message.
4. Waits for all nodes to receive the above message.
N and the structure of the graph are reproducibly randomized via
Quickcheck.
* Corrections pointed out by @mxinden
* Add option to remove source id publishing
* protocols/gossipsub/tests/smoke: Remove unused variable
* Merge latest master
* protocols/gossipsub: Move to stable futures
* examples/gossipsub-chat.rs: Move to stable futures
* protocols/gossipsub/src/behaviour/tests: Update to stable futures
* protocols/gossipsub/tests: Update to stable futures
* protocols/gossipsub: Log substream errors
* protocols/gossipsub: Log outbound substream errors
* Remove rust-fmt formatting
* Shift to prost for protobuf compiling
* Use wasm_timer for wasm compatibility
Co-authored-by: Grant Wuerker <gwuerker@gmail.com>
Co-authored-by: Toralf Wittner <tw@dtex.org>
Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
