1030 Commits

Author SHA1 Message Date
Pierre Krieger
0a45f7310f
Support tokio in libp2p-tcp and libp2p-uds (#1402) 2020-01-24 16:40:48 +01:00
Age Manning
37c7d73b11 Gossipsub Protocol (#898)
* Create gossipsub crate - Basic template, borrowed from floodsub

* Add a GossipsubConfig struct and set up basic structures in the Gossipsub struct

* Begin implementation of join. Adds get_random_peers helper function and adds tests

* Implements gossipsub leave()

* Update publishMany to incorporate gossipsub mesh and fanout logic

* Use the gossipsub mesh for determining peer subscription

* Remove subscribed_topics field from the Gossipsub struct

* Rename gossipsubconfig to ProtocolConfig

* Implement the gossipsub control messages into the Codec's Encode/Decode and modifies GossipsubRpc

* Modify GossipsubActions to enums for succinctness.

* Modify the memcache to store Gossipsub messages

* Implement control message handling.

* Update control message handling to handle multiple messages.

* Handle received gossipsub messages using pre-built handlers.

* Remove excess connected peer hashmap

* Add extra peer mapping and consistent topic naming.

* Implement heartbeat, emit_gossip and send_graft_prune.

* Group logic in forwarding messages. Add messages to memcache.

* Add heartbeat timer and move location of helper function.

* Add gossipsub the libp2p workspace, makes layer structs public

* Add logging to gossipsub

- Adds the log crate and implements logging macros
- Specifies versions for external crates

* Add example chat for debugging purposes

* Implement #868 for gossipsub.

* Add rust documentation to gossipsub crate.

- Adds basic documentation, overview and examples to the gossipsub
crate.

* Re-introduce the initial heartbeat time config.

This commit also adds the inject_connected test.

* Add subscribe tests.

- Modifies `handle_received_subscriptions` to take a reference of
subscriptions
- Adds `test_subscribe`
- Adds `test_handle_received_subscriptions`
- Adds tests for the filter in `get_random_peers`

* Add Bug fixes and further testing for gossipsub.

- Corrects the tuple use of topic_hashes
- Corrects JOIN logic around fanout and adding peers to the mesh
- Adds test_unsubscribe
- Adds test_join

* Rename GossipsubMessage::msg_id -> id

* Add bug fix for handling disconnected peers.

* Implements (partially) #889 for Gossipsub.

* handle_iwant event count tests

* handle_ihave event count tests

* Move layer.rs tests into separate file.

* Implement clippy suggestions for gossipsub.

* Modify control message tests for specific types.

* Implement builder pattern for GossipsubConfig.

As suggested by @twittner - The builder pattern for building
GossipsubConfig struct is implemented.

* Package version updates as suggested by @twittner.

* Correct line lengths in gossipsub.

* Correct braces in  found by @twittner.

* Implement @twittner's suggestions.

- Uses `HashSet` where applicable
- Update `FnvHashMap` to standard `HashMap`
- Uses `min` function in code simplification.

* Add NodeList struct to clarify topic_peers.

* Cleaner handling of messagelist

Co-Authored-By: AgeManning <Age@AgeManning.com>

* Cleaner handling of added peers.

Co-Authored-By: AgeManning <Age@AgeManning.com>

* handle_prune peer removed test

* basic grafting tests

* multiple topic grafting test

* Convert &vec to slice.

Co-Authored-By: AgeManning <Age@AgeManning.com>

* Convert to lazy insert.

Co-Authored-By: AgeManning <Age@AgeManning.com>

* Cleaner topic handling.

Co-Authored-By: AgeManning <Age@AgeManning.com>

* control pool piggybacking

using HashMap.drain() in control_pool_flush

going to squash this

* Add Debug derives to gossipsub and correct tests.

* changes from PR

squash this

all tests passing, but still some that need to be reconsidered

test reform

* Implements Arc for GossipsubRpc events

* Remove support for floodsub nodes

* Reconnected to disconnected peers, to mitigate timeout

* Use ReadOne WriteOne with configurable max gossip sizes

* Remove length delimination from RPC encoding

* Prevent peer duplication in mesh

* Allow oneshot handler's inactivity_timeout to be configurable

* Correct peer duplication in mesh bug

* Remove auto-reconnect to allow for user-level disconnects

* Single long-lived inbound/outbound streams to match go implementation

* Allow gossipsub topics to be optionally hashable

* Improves gossipsub stream handling

- Corrects the handler's keep alive.
- Correct the chat example.
- Instantly add peers to the mesh on subscription if the mesh is low.

* Allows message validation in gossipsub

* Replaces Cuckoofilter with LRUCache

The false positive rate was unacceptable for rejecting messages.

* Renames configuration parameter and corrects logic

* Removes peer from fanout on disconnection

* Add publish and fanout tests

* Apply @mxinden suggestions

* Resend message if outbound stream negotiated

- Downgrades log warnings

* Implement further reviewer suggestions

- Created associated functions to avoid unnecessary cloning
- Messages are rejected if their sequence numbers are not u64
- `GossipsbuConfigBuilder` has the same defaults as `GossipsubConfig`
- Miscellaneous typos

* Add MessageId type and remove unnecessary comments

* Add a return value to propagate_message function

* Adds user-customised gossipsub message ids

* Adds the message id to GossipsubEvent

* Implement Debug for GossipsubConfig

* protocols/gossipsub: Add basic smoke test

Implement a basic smoke test that:

1. Builds a fully connected graph of size N.

2. Subscribes each node to the same topic.

3. Publishes a single message.

4. Waits for all nodes to receive the above message.

N and the structure of the graph are reproducibly randomized via
Quickcheck.

* Corrections pointed out by @mxinden

* Add option to remove source id publishing

* protocols/gossipsub/tests/smoke: Remove unused variable

* Merge latest master

* protocols/gossipsub: Move to stable futures

* examples/gossipsub-chat.rs: Move to stable futures

* protocols/gossipsub/src/behaviour/tests: Update to stable futures

* protocols/gossipsub/tests: Update to stable futures

* protocols/gossipsub: Log substream errors

* protocols/gossipsub: Log outbound substream errors

* Remove rust-fmt formatting

* Shift to prost for protobuf compiling

* Use wasm_timer for wasm compatibility

Co-authored-by: Grant Wuerker <gwuerker@gmail.com>
Co-authored-by: Toralf Wittner <tw@dtex.org>
Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
2020-01-24 16:16:02 +01:00
Pierre Krieger
0cb3cd4262
Fix WouldBlock being returned in wasm-ext (#1407)
* Fix WouldBlock being returned

* Fix other WouldBlocks
2020-01-24 15:18:45 +01:00
Pierre Krieger
d8bafc1f0a
Force prost 0.6.1 (#1404) 2020-01-23 11:06:11 +01:00
Pierre Krieger
d6bf8a5aeb
Remove some Unpin requirements (#1403) 2020-01-23 10:44:52 +01:00
Volker Mische
3405fc8b32 Fix link to upgrade::Builder (#1401)
The auto-linking didn't work properly as there is a method called `upgrade`
within the transport module.
2020-01-22 15:06:54 +01:00
Pierre Krieger
7d0c80fce1
Executor + Send (#1399) 2020-01-20 15:22:47 +01:00
Pierre Krieger
f89683419a
Allow configuring the tasks executor (#1391)
* Allow configuring the tasks executor

* Minor tweaks

* Add executor_fn

* Create ThreadsPool at the end if necessary

* Allow configuring the tasks executor

* Minor tweaks

* Add executor_fn

* Create ThreadsPool at the end if necessary

* WIP

* Don't depend on async-std and tokio in core

* Replace FutureObj with PinBoxFuture

* Some docs on Executor

* Fix tests

Co-authored-by: Toralf Wittner <tw@dtex.org>
2020-01-20 14:18:35 +01:00
Pierre Krieger
0ed684ee30
Don't set KeepAlive to Until all the time (#1398)
* Don't set KeepAlive to Until all the time

* Address concerns
2020-01-20 13:55:25 +01:00
Roman Borschel
c241014ddb
Log connection close in swarm. Simplify bounds. (#1393) 2020-01-16 13:17:39 +01:00
Pierre Krieger
89acb0d9f1
Allow building an upgrade from an async function (#1383)
* Allow building an upgrade from an async function

* Some styling
2020-01-15 14:57:02 +01:00
Max Inden
991f5af993 misc/mdns: Make MdnsService::new sync by using std::net::UdpSocket::bind (#1382)
* misc/mdns: Make MdnsService::new sync by using std::net::UdpSocket::bind

MdnsService uses an IP address to create a UDP socket. The address does
not need to be resolved. Therefore one can use std's UdpSocket::bind
instead of the async counterpart from async-std. As a result
MdnsService::new and MdnsService::silent don't need to be async.

* examples/mdns-passive-discovery: Don't await sync MdnsService::new

* misc/mdns/src/behaviour: Make Mdns::new sync

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
2020-01-15 13:43:09 +01:00
Toralf Wittner
680c467f7e Replace protobuf crate with prost! (#1390)
* Replace protobuf crate with prost!

* Add copyright headers to build.rs files.

* kad: Fix error when mapping connection types.

Co-Authored-By: Pierre Krieger <pierre.krieger1708@gmail.com>

* Fix more mapping mistakes.

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
2020-01-15 12:02:02 +01:00
Max Inden
9d2df148cd core/src/transport/memory: Add basic dialer listener test (#1283)
* core/src/transport/memory: Add basic dialer listener test

Adds a basic test using two memory transport instances, one dialing the
other, sending a single message over the /wire/.

* core/src/nodes/network: Fix typo

* core/src/transport: Randomize listening port in unit test

* core/src/transport/memory: Update test to stable future

* core/src/transport/memory.rs: Use saturating_add for test port

Co-Authored-By: Pierre Krieger <pierre.krieger1708@gmail.com>

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
Co-authored-by: Roman Borschel <romanb@users.noreply.github.com>
2020-01-14 18:19:10 +01:00
Joonas Koivunen
7df3e74d79 Enhance NetworkBehaviour docs (#1362)
* add docs about deriving NetworkBehaviour

* add commentary to the chat example

also minor stuff like reordering to match the struct elements, note why the listening address is
reported in poll.

* fix remove the warning on the example ignored field

* Apply suggestions from code review

Thanks for suggestions!

Co-Authored-By: Max Inden <mail@max-inden.de>

* chore cleanup added docs after suggestions

* fix print all listening addresses in examples/chat

not that there should be more than one but it makes sense to repeat the full example here.

* chore remove confusing writing on printing the listening addrs

* fix use intra-docs, spelling

Co-Authored-By: Roman S. Borschel <roman@parity.io>

Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: Roman Borschel <romanb@users.noreply.github.com>
2020-01-14 12:48:16 +01:00
Pierre Krieger
3f968cbf92
Remove some Unpin requirements on Futures (#1384)
* Remove lots of Unpin requirements

* Make Transport::and_then accept pinned futures

* Finish the PR

* Work on secio

* Fix BandwidthTransport

* Adjust ListenersStrema

* Fix nodes/tasks

* Fix nodes

* Various more fixes

* Fix yamux

* Fix Swarm

* Fix WebSockets

* Fix rw-stream-sink
2020-01-14 12:03:10 +01:00
Pierre Krieger
42a45e2630
Remove Negotiated from upgrade traits (#1388)
* Remove Negotiated from upgrade traits

* Remove import
2020-01-13 14:34:43 +01:00
Toralf Wittner
ff780b5bff
Make InboundUpgrade for IK pattern more generic. (#1379)
The current implementation defines the InboundUpgrade only for `R = ()`
so notably a `NoiseConfig` created by `NoiseConfig::ik_dialer` can not
be an `InboundUpgrade`.

Fixes #1378.
2020-01-10 15:09:36 +01:00
Qinxuan Chen
af464e18c5 Update some dependenices; Remove some useless dependencies (#1387)
Signed-off-by: koushiro <koushiro.cqx@gmail.com>

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
2020-01-10 14:03:59 +01:00
Toralf Wittner
655609dfe0 dns: Resolve dns4 to IPv4 and dns6 to IPv6. (#1381) 2020-01-10 13:10:16 +01:00
Max Inden
b939aefdd6 protocols/plaintext: Adapt protobuf generation to consolidated scripts (#1309)
* protocols/plaintext: Adapt protobuf generation to consolidated scripts

* protocols/plaintext: Fix protobuf import

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
2020-01-09 14:12:43 +01:00
Pierre Krieger
324f0dc326
Bump rw-stream-sink to 0.2 (#1377) v0.14.0-alpha.1 2020-01-07 15:54:39 +01:00
Pierre Krieger
84487cf904
Publish 0.14.0-alpha.1 (#1376)
* Publish 0.14.0-alpha.1

* multiaddr -> 0.7.0
2020-01-07 15:30:39 +01:00
Toralf Wittner
ab2eb7a51d multiaddr: Replace Bytes with Arc<Vec<u8>>. (#1370)
* multiaddr: Replace `Bytes` with `Arc<Vec<u8>>`.

* Annotate type of `io::Cursor`.

Co-Authored-By: Pierre Krieger <pierre.krieger1708@gmail.com>

* Annotate type of `io::Cursor`.

Co-Authored-By: Pierre Krieger <pierre.krieger1708@gmail.com>

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
2020-01-07 13:06:16 +01:00
Akihito Nakano
d83619109b Add protocol string to Error::UnknownProtocolString (#1355)
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
2020-01-07 12:36:37 +01:00
Pierre Krieger
84b6a7d04d
Add Swarm::next_extended (#1374)
* Add Swarm::next_extended

* Fix ipfs-kad example

* Fix tests

* Renames
2020-01-07 11:57:00 +01:00
Pierre Krieger
56ca671376
Merge pull request #1328 from libp2p/stable-futures
Refactor the crate to use stable futures
2020-01-06 19:20:30 +01:00
Pierre Krieger
ced66a319c
Merge pull request #1373 from tomaka/loggin-stable-fut
Address review on stable-futures
2020-01-06 18:38:28 +01:00
Pierre Krieger
fd7c06ed23
Merge pull request #1375 from tomaka/another-review-fix2
Another review fix for stable-futures branch
2020-01-06 17:30:34 +01:00
Pierre Krieger
55d6925006
Terminology 2020-01-06 17:22:09 +01:00
Pierre Krieger
5e0f219555
Use a different fix 2020-01-06 17:20:44 +01:00
Pierre Krieger
1333d5b8d9
Another review fix for stable-futures branch 2020-01-06 17:13:39 +01:00
Pierre Krieger
be0f18d162
Update core/src/nodes/tasks/manager.rs
Co-Authored-By: Max Inden <mail@max-inden.de>
2020-01-06 14:50:33 +01:00
Pierre Krieger
65a7de49b2
Address review on stable-futures 2020-01-06 14:08:02 +01:00
Pierre Krieger
84ebd20d01
Merge pull request #1372 from tomaka/another-review-fix
Another small review fix
2020-01-06 13:32:11 +01:00
Pierre Krieger
e21657107d
Another small review fix 2020-01-06 11:57:12 +01:00
Max Inden
74790cd514
Apply suggestions from code review
Co-Authored-By: Demi Obenour <48690212+DemiMarie-parity@users.noreply.github.com>
2020-01-06 11:13:23 +01:00
Pierre Krieger
90e0044dd6
Apply suggestions from code review
Co-Authored-By: Max Inden <mail@max-inden.de>
2020-01-03 11:58:08 +01:00
Pierre Krieger
e29f36e32b
Merge pull request #1369 from tomaka/merge-master
Merge master in stable-futures
2020-01-03 11:22:29 +01:00
Pierre Krieger
17e90f0f2c
Merge master in stable-futures 2020-01-02 17:47:41 +01:00
Pierre Krieger
b7146f4918
Merge pull request #1368 from tomaka/0.13.2
Pin protobuf versin and publish 0.13.2
v0.13.2 0.13.2
2020-01-02 16:21:14 +01:00
Pierre Krieger
c6a303cc45
Comment 2020-01-02 16:04:00 +01:00
Pierre Krieger
455a5a92f4
Pin protobuf versin and publish 0.13.2 2020-01-02 15:29:43 +01:00
Pierre Krieger
d0944ed50c
Merge pull request #1366 from tomaka/review-stable-fut
Address some review comments on #1328
2020-01-02 13:24:40 +01:00
Pierre Krieger
23fc6ee524
Address some review comments on #1328 2020-01-02 12:59:10 +01:00
Pierre Krieger
ff0d2d55bd
Merge pull request #1353 from twittner/fixes
Several changes.
2020-01-02 11:42:07 +01:00
Toralf Wittner
d870c734ee Pin protobuf to version 2.8.1 2020-01-02 10:48:51 +01:00
Toralf Wittner
72f1018acd Update to unsigned-varint v0.3 2020-01-02 10:45:43 +01:00
Toralf Wittner
83e4c60923 Update yamux dependency. 2019-12-30 12:08:17 +01:00
Toralf Wittner
34b36b464b Cover more cases in simultaneous connect test. 2019-12-21 17:14:59 +01:00