rust-libp2p/src/lib.rs
Age Manning 37c7d73b11 Gossipsub Protocol (#898)
* Create gossipsub crate - Basic template, borrowed from floodsub

* Add a GossipsubConfig struct and set up basic structures in the Gossipsub struct

* Begin implementation of join. Adds get_random_peers helper function and adds tests

* Implements gossipsub leave()

* Update publishMany to incorporate gossipsub mesh and fanout logic

* Use the gossipsub mesh for determining peer subscription

* Remove subscribed_topics field from the Gossipsub struct

* Rename gossipsubconfig to ProtocolConfig

* Implement the gossipsub control messages into the Codec's Encode/Decode and modifies GossipsubRpc

* Modify GossipsubActions to enums for succinctness.

* Modify the memcache to store Gossipsub messages

* Implement control message handling.

* Update control message handling to handle multiple messages.

* Handle received gossipsub messages using pre-built handlers.

* Remove excess connected peer hashmap

* Add extra peer mapping and consistent topic naming.

* Implement heartbeat, emit_gossip and send_graft_prune.

* Group logic in forwarding messages. Add messages to memcache.

* Add heartbeat timer and move location of helper function.

* Add gossipsub the libp2p workspace, makes layer structs public

* Add logging to gossipsub

- Adds the log crate and implements logging macros
- Specifies versions for external crates

* Add example chat for debugging purposes

* Implement #868 for gossipsub.

* Add rust documentation to gossipsub crate.

- Adds basic documentation, overview and examples to the gossipsub
crate.

* Re-introduce the initial heartbeat time config.

This commit also adds the inject_connected test.

* Add subscribe tests.

- Modifies `handle_received_subscriptions` to take a reference of
subscriptions
- Adds `test_subscribe`
- Adds `test_handle_received_subscriptions`
- Adds tests for the filter in `get_random_peers`

* Add Bug fixes and further testing for gossipsub.

- Corrects the tuple use of topic_hashes
- Corrects JOIN logic around fanout and adding peers to the mesh
- Adds test_unsubscribe
- Adds test_join

* Rename GossipsubMessage::msg_id -> id

* Add bug fix for handling disconnected peers.

* Implements (partially) #889 for Gossipsub.

* handle_iwant event count tests

* handle_ihave event count tests

* Move layer.rs tests into separate file.

* Implement clippy suggestions for gossipsub.

* Modify control message tests for specific types.

* Implement builder pattern for GossipsubConfig.

As suggested by @twittner - The builder pattern for building
GossipsubConfig struct is implemented.

* Package version updates as suggested by @twittner.

* Correct line lengths in gossipsub.

* Correct braces in  found by @twittner.

* Implement @twittner's suggestions.

- Uses `HashSet` where applicable
- Update `FnvHashMap` to standard `HashMap`
- Uses `min` function in code simplification.

* Add NodeList struct to clarify topic_peers.

* Cleaner handling of messagelist

Co-Authored-By: AgeManning <Age@AgeManning.com>

* Cleaner handling of added peers.

Co-Authored-By: AgeManning <Age@AgeManning.com>

* handle_prune peer removed test

* basic grafting tests

* multiple topic grafting test

* Convert &vec to slice.

Co-Authored-By: AgeManning <Age@AgeManning.com>

* Convert to lazy insert.

Co-Authored-By: AgeManning <Age@AgeManning.com>

* Cleaner topic handling.

Co-Authored-By: AgeManning <Age@AgeManning.com>

* control pool piggybacking

using HashMap.drain() in control_pool_flush

going to squash this

* Add Debug derives to gossipsub and correct tests.

* changes from PR

squash this

all tests passing, but still some that need to be reconsidered

test reform

* Implements Arc for GossipsubRpc events

* Remove support for floodsub nodes

* Reconnected to disconnected peers, to mitigate timeout

* Use ReadOne WriteOne with configurable max gossip sizes

* Remove length delimination from RPC encoding

* Prevent peer duplication in mesh

* Allow oneshot handler's inactivity_timeout to be configurable

* Correct peer duplication in mesh bug

* Remove auto-reconnect to allow for user-level disconnects

* Single long-lived inbound/outbound streams to match go implementation

* Allow gossipsub topics to be optionally hashable

* Improves gossipsub stream handling

- Corrects the handler's keep alive.
- Correct the chat example.
- Instantly add peers to the mesh on subscription if the mesh is low.

* Allows message validation in gossipsub

* Replaces Cuckoofilter with LRUCache

The false positive rate was unacceptable for rejecting messages.

* Renames configuration parameter and corrects logic

* Removes peer from fanout on disconnection

* Add publish and fanout tests

* Apply @mxinden suggestions

* Resend message if outbound stream negotiated

- Downgrades log warnings

* Implement further reviewer suggestions

- Created associated functions to avoid unnecessary cloning
- Messages are rejected if their sequence numbers are not u64
- `GossipsbuConfigBuilder` has the same defaults as `GossipsubConfig`
- Miscellaneous typos

* Add MessageId type and remove unnecessary comments

* Add a return value to propagate_message function

* Adds user-customised gossipsub message ids

* Adds the message id to GossipsubEvent

* Implement Debug for GossipsubConfig

* protocols/gossipsub: Add basic smoke test

Implement a basic smoke test that:

1. Builds a fully connected graph of size N.

2. Subscribes each node to the same topic.

3. Publishes a single message.

4. Waits for all nodes to receive the above message.

N and the structure of the graph are reproducibly randomized via
Quickcheck.

* Corrections pointed out by @mxinden

* Add option to remove source id publishing

* protocols/gossipsub/tests/smoke: Remove unused variable

* Merge latest master

* protocols/gossipsub: Move to stable futures

* examples/gossipsub-chat.rs: Move to stable futures

* protocols/gossipsub/src/behaviour/tests: Update to stable futures

* protocols/gossipsub/tests: Update to stable futures

* protocols/gossipsub: Log substream errors

* protocols/gossipsub: Log outbound substream errors

* Remove rust-fmt formatting

* Shift to prost for protobuf compiling

* Use wasm_timer for wasm compatibility

Co-authored-by: Grant Wuerker <gwuerker@gmail.com>
Co-authored-by: Toralf Wittner <tw@dtex.org>
Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
2020-01-24 16:16:02 +01:00

318 lines
13 KiB
Rust

// Copyright 2018 Parity Technologies (UK) Ltd.
//
// Permission is hereby granted, free of charge, to any person obtaining a
// copy of this software and associated documentation files (the "Software"),
// to deal in the Software without restriction, including without limitation
// the rights to use, copy, modify, merge, publish, distribute, sublicense,
// and/or sell copies of the Software, and to permit persons to whom the
// Software is furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall be included in
// all copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
// OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
// DEALINGS IN THE SOFTWARE.
//! Libp2p is a peer-to-peer framework.
//!
//! # Major libp2p concepts
//!
//! Here is a list of all the major concepts of libp2p.
//!
//! ## Multiaddr
//!
//! A [`Multiaddr`] is a self-describing network address and protocol stack
//! that is used to establish connections to peers. Some examples:
//!
//! * `/ip4/80.123.90.4/tcp/5432`
//! * `/ip6/[::1]/udp/10560/quic`
//! * `/unix//path/to/socket`
//!
//! ## Transport
//!
//! [`Transport`] is a trait for types that provide connection-oriented communication channels
//! based on dialing to or listening on a [`Multiaddr`]. To that end a transport
//! produces as output a type of data stream that varies depending on the concrete type of
//! transport.
//!
//! An implementation of transport typically supports only certain multi-addresses.
//! For example, the [`TcpConfig`] only supports multi-addresses of the format
//! `/ip4/.../tcp/...`.
//!
//! Example (Dialing a TCP/IP multi-address):
//!
//! ```rust
//! use libp2p::{Multiaddr, Transport, tcp::TcpConfig};
//! let tcp = TcpConfig::new();
//! let addr: Multiaddr = "/ip4/98.97.96.95/tcp/20500".parse().expect("invalid multiaddr");
//! let _conn = tcp.dial(addr);
//! ```
//! In the above example, `_conn` is a [`Future`] that needs to be polled in order for
//! the dialing to take place and eventually resolve to a connection. Polling
//! futures is typically done through a [tokio] runtime.
//!
//! The easiest way to create a transport is to use [`build_development_transport`].
//! This function provides support for the most common protocols but it is also
//! subject to change over time and should thus not be used in production
//! configurations.
//!
//! Example (Creating a development transport):
//!
//! ```rust
//! let keypair = libp2p::identity::Keypair::generate_ed25519();
//! let _transport = libp2p::build_development_transport(keypair);
//! // _transport.dial(...);
//! ```
//!
//! The keypair that is passed as an argument in the above example is used
//! to set up transport-layer encryption using a newly generated long-term
//! identity keypair. The public key of this keypair uniquely identifies
//! the node in the network in the form of a [`PeerId`].
//!
//! See the documentation of the [`Transport`] trait for more details.
//!
//! ### Connection Upgrades
//!
//! Once a connection has been established with a remote through a [`Transport`], it can be
//! *upgraded*. Upgrading a transport is the process of negotiating an additional protocol
//! with the remote, mediated through a negotiation protocol called [`multistream-select`].
//!
//! Example ([`secio`] + [`yamux`] Protocol Upgrade):
//!
//! ```rust
//! # #[cfg(all(not(any(target_os = "emscripten", target_os = "unknown")), feature = "libp2p-secio"))] {
//! use libp2p::{Transport, tcp::TcpConfig, secio::SecioConfig, identity::Keypair, yamux};
//! let tcp = TcpConfig::new();
//! let secio = SecioConfig::new(Keypair::generate_ed25519());
//! let yamux = yamux::Config::default();
//! let transport = tcp.upgrade().authenticate(secio).multiplex(yamux);
//! # }
//! ```
//! In this example, `tcp_secio` is a new [`Transport`] that negotiates the secio protocol
//! on all connections.
//!
//! ## Network Behaviour
//!
//! The [`NetworkBehaviour`] trait is implemented on types that provide some capability to the
//! network. Examples of network behaviours include:
//!
//! * Periodically pinging other nodes on established connections.
//! * Periodically asking for information from other nodes.
//! * Querying information from a DHT and propagating it to other nodes.
//!
//! ## Swarm
//!
//! A [`Swarm`] manages a pool of connections established through a [`Transport`]
//! and drives a [`NetworkBehaviour`] through emitting events triggered by activity
//! on the managed connections. Creating a [`Swarm`] thus involves combining a
//! [`Transport`] with a [`NetworkBehaviour`].
//!
//! See the documentation of the [`core`] module for more details about swarms.
//!
//! # Using libp2p
//!
//! The easiest way to get started with libp2p involves the following steps:
//!
//! 1. Creating an identity [`Keypair`] for the local node, obtaining the local
//! [`PeerId`] from the [`PublicKey`].
//! 2. Creating an instance of a base [`Transport`], e.g. [`TcpConfig`], upgrading it with
//! all the desired protocols, such as for transport security and multiplexing.
//! In order to be usable with a [`Swarm`] later, the [`Output`](Transport::Output)
//! of the final transport must be a tuple of a [`PeerId`] and a value whose type
//! implements [`StreamMuxer`] (e.g. [`Yamux`]). The peer ID must be the
//! identity of the remote peer of the established connection, which is
//! usually obtained through a transport encryption protocol such as
//! [`secio`] that authenticates the peer. See the implementation of
//! [`build_development_transport`] for an example.
//! 3. Creating a struct that implements the [`NetworkBehaviour`] trait and combines all the
//! desired network behaviours, implementing the event handlers as per the
//! desired application's networking logic.
//! 4. Instantiating a [`Swarm`] with the transport, the network behaviour and the
//! local peer ID from the previous steps.
//!
//! The swarm instance can then be polled with the [tokio] library, in order to
//! continuously drive the network activity of the program.
//!
//! [`Keypair`]: identity::Keypair
//! [`PublicKey`]: identity::PublicKey
//! [`Future`]: futures::Future
//! [`TcpConfig`]: tcp::TcpConfig
//! [`NetworkBehaviour`]: swarm::NetworkBehaviour
//! [`StreamMuxer`]: core::muxing::StreamMuxer
//! [`Yamux`]: yamux::Yamux
//!
//! [tokio]: https://tokio.rs
//! [`multistream-select`]: https://github.com/multiformats/multistream-select
#![doc(html_logo_url = "https://libp2p.io/img/logo_small.png")]
#![doc(html_favicon_url = "https://libp2p.io/img/favicon.png")]
pub use bytes;
pub use futures;
#[doc(inline)]
pub use multiaddr;
#[doc(inline)]
pub use multihash;
#[doc(inline)]
pub use libp2p_core as core;
#[cfg(not(any(target_os = "emscripten", target_os = "unknown")))]
#[doc(inline)]
pub use libp2p_deflate as deflate;
#[cfg(not(any(target_os = "emscripten", target_os = "unknown")))]
#[doc(inline)]
pub use libp2p_dns as dns;
#[doc(inline)]
pub use libp2p_identify as identify;
#[doc(inline)]
pub use libp2p_kad as kad;
#[doc(inline)]
pub use libp2p_floodsub as floodsub;
#[doc(inline)]
pub use libp2p_gossipsub as gossipsub;
#[doc(inline)]
pub use libp2p_mplex as mplex;
#[cfg(not(any(target_os = "emscripten", target_os = "unknown")))]
#[doc(inline)]
pub use libp2p_mdns as mdns;
#[cfg(not(any(target_os = "emscripten", target_os = "unknown")))]
#[doc(inline)]
pub use libp2p_noise as noise;
#[doc(inline)]
pub use libp2p_ping as ping;
#[doc(inline)]
pub use libp2p_plaintext as plaintext;
#[doc(inline)]
pub use libp2p_secio as secio;
#[doc(inline)]
pub use libp2p_swarm as swarm;
#[cfg(not(any(target_os = "emscripten", target_os = "unknown")))]
#[doc(inline)]
pub use libp2p_tcp as tcp;
#[doc(inline)]
pub use libp2p_uds as uds;
#[doc(inline)]
pub use libp2p_wasm_ext as wasm_ext;
#[cfg(all(feature = "libp2p-websocket", not(any(target_os = "emscripten", target_os = "unknown"))))]
#[doc(inline)]
pub use libp2p_websocket as websocket;
#[doc(inline)]
pub use libp2p_yamux as yamux;
mod transport_ext;
pub mod bandwidth;
pub mod simple;
pub use self::core::{
identity,
PeerId,
Transport,
transport::TransportError,
upgrade::{InboundUpgrade, InboundUpgradeExt, OutboundUpgrade, OutboundUpgradeExt}
};
pub use libp2p_core_derive::NetworkBehaviour;
pub use self::multiaddr::{Multiaddr, multiaddr as build_multiaddr};
pub use self::simple::SimpleProtocol;
pub use self::swarm::Swarm;
pub use self::transport_ext::TransportExt;
use std::{error, io, time::Duration};
/// Builds a `Transport` that supports the most commonly-used protocols that libp2p supports.
///
/// > **Note**: This `Transport` is not suitable for production usage, as its implementation
/// > reserves the right to support additional protocols or remove deprecated protocols.
pub fn build_development_transport(keypair: identity::Keypair)
-> io::Result<impl Transport<Output = (PeerId, impl core::muxing::StreamMuxer<OutboundSubstream = impl Send, Substream = impl Send, Error = impl Into<io::Error>> + Send + Sync), Error = impl error::Error + Send, Listener = impl Send, Dial = impl Send, ListenerUpgrade = impl Send> + Clone>
{
build_tcp_ws_secio_mplex_yamux(keypair)
}
/// Builds an implementation of `Transport` that is suitable for usage with the `Swarm`.
///
/// The implementation supports TCP/IP, WebSockets over TCP/IP, secio as the encryption layer,
/// and mplex or yamux as the multiplexing layer.
///
/// > **Note**: If you ever need to express the type of this `Transport`.
pub fn build_tcp_ws_secio_mplex_yamux(keypair: identity::Keypair)
-> io::Result<impl Transport<Output = (PeerId, impl core::muxing::StreamMuxer<OutboundSubstream = impl Send, Substream = impl Send, Error = impl Into<io::Error>> + Send + Sync), Error = impl error::Error + Send, Listener = impl Send, Dial = impl Send, ListenerUpgrade = impl Send> + Clone>
{
Ok(CommonTransport::new()?
.upgrade(core::upgrade::Version::V1)
.authenticate(secio::SecioConfig::new(keypair))
.multiplex(core::upgrade::SelectUpgrade::new(yamux::Config::default(), mplex::MplexConfig::new()))
.map(|(peer, muxer), _| (peer, core::muxing::StreamMuxerBox::new(muxer)))
.timeout(Duration::from_secs(20)))
}
/// Implementation of `Transport` that supports the most common protocols.
///
/// The list currently is TCP/IP, DNS, and WebSockets. However this list could change in the
/// future to get new transports.
#[derive(Debug, Clone)]
struct CommonTransport {
// The actual implementation of everything.
inner: CommonTransportInner
}
#[cfg(all(not(any(target_os = "emscripten", target_os = "unknown")), feature = "libp2p-websocket"))]
type InnerImplementation = core::transport::OrTransport<dns::DnsConfig<tcp::TcpConfig>, websocket::WsConfig<dns::DnsConfig<tcp::TcpConfig>>>;
#[cfg(all(not(any(target_os = "emscripten", target_os = "unknown")), not(feature = "libp2p-websocket")))]
type InnerImplementation = dns::DnsConfig<tcp::TcpConfig>;
#[cfg(any(target_os = "emscripten", target_os = "unknown"))]
type InnerImplementation = core::transport::dummy::DummyTransport;
#[derive(Debug, Clone)]
struct CommonTransportInner {
inner: InnerImplementation,
}
impl CommonTransport {
/// Initializes the `CommonTransport`.
#[cfg(not(any(target_os = "emscripten", target_os = "unknown")))]
pub fn new() -> io::Result<CommonTransport> {
let tcp = tcp::TcpConfig::new().nodelay(true);
let transport = dns::DnsConfig::new(tcp)?;
#[cfg(feature = "libp2p-websocket")]
let transport = {
let trans_clone = transport.clone();
transport.or_transport(websocket::WsConfig::new(trans_clone))
};
Ok(CommonTransport {
inner: CommonTransportInner { inner: transport }
})
}
/// Initializes the `CommonTransport`.
#[cfg(any(target_os = "emscripten", target_os = "unknown"))]
pub fn new() -> io::Result<CommonTransport> {
let inner = core::transport::dummy::DummyTransport::new();
Ok(CommonTransport {
inner: CommonTransportInner { inner }
})
}
}
impl Transport for CommonTransport {
type Output = <InnerImplementation as Transport>::Output;
type Error = <InnerImplementation as Transport>::Error;
type Listener = <InnerImplementation as Transport>::Listener;
type ListenerUpgrade = <InnerImplementation as Transport>::ListenerUpgrade;
type Dial = <InnerImplementation as Transport>::Dial;
fn listen_on(self, addr: Multiaddr) -> Result<Self::Listener, TransportError<Self::Error>> {
self.inner.inner.listen_on(addr)
}
fn dial(self, addr: Multiaddr) -> Result<Self::Dial, TransportError<Self::Error>> {
self.inner.inner.dial(addr)
}
}