rust-libp2p/examples/ipfs-kad.rs

121 lines
4.8 KiB
Rust
Raw Normal View History

// Copyright 2018 Parity Technologies (UK) Ltd.
//
// Permission is hereby granted, free of charge, to any person obtaining a
// copy of this software and associated documentation files (the "Software"),
// to deal in the Software without restriction, including without limitation
// the rights to use, copy, modify, merge, publish, distribute, sublicense,
// and/or sell copies of the Software, and to permit persons to whom the
// Software is furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall be included in
// all copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
// OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
// DEALINGS IN THE SOFTWARE.
//! Demonstrates how to perform Kademlia queries on the IPFS network.
//!
//! You can pass as parameter a base58 peer ID to search for. If you don't pass any parameter, a
//! peer ID will be generated randomly.
use async_std::task;
use futures::StreamExt;
use libp2p::kad::record::store::MemoryStore;
use libp2p::kad::{GetClosestPeersError, Kademlia, KademliaConfig, KademliaEvent, QueryResult};
use libp2p::{
development_transport, identity,
swarm::{Swarm, SwarmEvent},
Multiaddr, PeerId,
};
use std::{env, error::Error, str::FromStr, time::Duration};
const BOOTNODES: [&'static str; 4] = [
"QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN",
"QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa",
"QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb",
"QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt",
];
#[async_std::main]
async fn main() -> Result<(), Box<dyn Error>> {
Kademlia: Address some TODOs - Refactoring - API updates. (#1174) * Address some TODOs, refactor queries and public API. The following left-over issues are addressed: * The key for FIND_NODE requests is generalised to any Multihash, instead of just peer IDs. * All queries get a (configurable) timeout. * Finishing queries as soon as enough results have been received is simplified to avoid code duplication. * No more panics in provider-API-related code paths. The provider API is however still untested and (I think) still incomplete (e.g. expiration of provider records). * Numerous smaller TODOs encountered in the code. The following public API changes / additions are made: * Introduce a `KademliaConfig` with new configuration options for the replication factor and query timeouts. * Rename `find_node` to `get_closest_peers`. * Rename `get_value` to `get_record` and `put_value` to `put_record`, introducing a `Quorum` parameter for both functions, replacing the existing `num_results` parameter with clearer semantics. * Rename `add_providing` to `start_providing` and `remove_providing` to `stop_providing`. * Add a `bootstrap` function that implements a (almost) standard Kademlia bootstrapping procedure. * Rename `KademliaOut` to `KademliaEvent` with an updated list of constructors (some renaming). All events that report query results now report a `Result` to uniformly permit reporting of errors. The following refactorings are made: * Introduce some constants. * Consolidate `query.rs` and `write.rs` behind a common query interface to reduce duplication and facilitate better code reuse, introducing the notion of a query peer iterator. `query/peers/closest.rs` contains the code that was formerly in `query.rs`. `query/peers/fixed.rs` contains a modified variant of `write.rs` (which is removed). The new `query.rs` provides an interface for working with a collection of queries, taking over some code from `behaviour.rs`. * Reduce code duplication in tests and use the current_thread runtime for polling swarms to avoid spurious errors in the test output due to aborted connections when a test finishes prematurely (e.g. because a quorum of results has been collected). * Some additions / improvements to the existing tests. * Fix test. * Fix rebase. * Tweak kad-ipfs example. * Incorporate some feedback. * Provide easy access and conversion to keys in error results.
2019-07-03 16:16:25 +02:00
env_logger::init();
// Create a random key for ourselves.
let local_key = identity::Keypair::generate_ed25519();
let local_peer_id = PeerId::from(local_key.public());
// Set up a an encrypted DNS-enabled TCP Transport over the Mplex protocol
let transport = development_transport(local_key).await?;
// Create a swarm to manage peers and events.
let mut swarm = {
// Create a Kademlia behaviour.
Kademlia: Somewhat complete the records implementation. (#1189) * Somewhat complete the implementation of Kademlia records. This commit relates to [libp2p-146] and [libp2p-1089]. * All records expire (by default, configurable). * Provider records are also stored in the RecordStore, and the RecordStore API extended. * Background jobs for periodic (re-)replication and (re-)publication of records. Regular (value-)records are subject to re-replication and re-publication as per standard Kademlia. Provider records are only subject to re-publication. * For standard Kademlia value lookups (quorum = 1), the record is cached at the closest peer to the key that did not return the value, as per standard Kademlia. * Expiration times of regular (value-)records is computed exponentially inversely proportional to the number of nodes between the local node and the closest node known to the key (beyond the k closest), as per standard Kademlia. The protobuf messages are extended with two fields: `ttl` and `publisher` in order to implement the different semantics of re-replication (by any of the k closest peers to the key, not affecting expiry) and re-publication (by the original publisher, resetting the expiry). This is not done yet in other libp2p Kademlia implementations, see e.g. [libp2p-go-323]. The new protobuf fields have been given somewhat unique identifiers to prevent future collision. Similarly, periodic re-publication of provider records does not seem to be done yet in other implementations, see e.g. [libp2p-js-98]. [libp2p-146]: https://github.com/libp2p/rust-libp2p/issues/146 [libp2p-1089]: https://github.com/libp2p/rust-libp2p/issues/1089 [libp2p-go-323]: https://github.com/libp2p/go-libp2p-kad-dht/issues/323 [libp2p-js-98]: https://github.com/libp2p/js-libp2p-kad-dht/issues/98 * Tweak kad-ipfs example. * Add missing files. * Ensure new delays are polled immediately. To ensure task notification, since `NotReady` is returned right after. * Fix ipfs-kad example and use wasm_timer. * Small cleanup. * Incorporate some feedback. * Adjustments after rebase. * Distinguish events further. In order for a user to easily distinguish the result of e.g. a `put_record` operation from the result of a later republication, different event constructors are used. Furthermore, for now, re-replication and "caching" of records (at the closest peer to the key that did not return a value during a successful lookup) do not yield events for now as they are less interesting. * Speed up tests for CI. * Small refinements and more documentation. * Guard a node against overriding records for which it considers itself to be the publisher. * Document the jobs module more extensively. * More inline docs around removal of "unreachable" addresses. * Remove wildcard re-exports. * Use NonZeroUsize for the constants. * Re-add method lost on merge. * Add missing 'pub'. * Further increase the timeout in the ipfs-kad example. * Readd log dependency to libp2p-kad. * Simplify RecordStore API slightly. * Some more commentary. * Change Addresses::remove to return Result<(),()>. Change the semantics of `Addresses::remove` so that the error case is unambiguous, instead of the success case. Use the `Result` for clearer semantics to that effect. * Add some documentation to .
2019-07-17 14:40:48 +02:00
let mut cfg = KademliaConfig::default();
Kademlia: Address some TODOs - Refactoring - API updates. (#1174) * Address some TODOs, refactor queries and public API. The following left-over issues are addressed: * The key for FIND_NODE requests is generalised to any Multihash, instead of just peer IDs. * All queries get a (configurable) timeout. * Finishing queries as soon as enough results have been received is simplified to avoid code duplication. * No more panics in provider-API-related code paths. The provider API is however still untested and (I think) still incomplete (e.g. expiration of provider records). * Numerous smaller TODOs encountered in the code. The following public API changes / additions are made: * Introduce a `KademliaConfig` with new configuration options for the replication factor and query timeouts. * Rename `find_node` to `get_closest_peers`. * Rename `get_value` to `get_record` and `put_value` to `put_record`, introducing a `Quorum` parameter for both functions, replacing the existing `num_results` parameter with clearer semantics. * Rename `add_providing` to `start_providing` and `remove_providing` to `stop_providing`. * Add a `bootstrap` function that implements a (almost) standard Kademlia bootstrapping procedure. * Rename `KademliaOut` to `KademliaEvent` with an updated list of constructors (some renaming). All events that report query results now report a `Result` to uniformly permit reporting of errors. The following refactorings are made: * Introduce some constants. * Consolidate `query.rs` and `write.rs` behind a common query interface to reduce duplication and facilitate better code reuse, introducing the notion of a query peer iterator. `query/peers/closest.rs` contains the code that was formerly in `query.rs`. `query/peers/fixed.rs` contains a modified variant of `write.rs` (which is removed). The new `query.rs` provides an interface for working with a collection of queries, taking over some code from `behaviour.rs`. * Reduce code duplication in tests and use the current_thread runtime for polling swarms to avoid spurious errors in the test output due to aborted connections when a test finishes prematurely (e.g. because a quorum of results has been collected). * Some additions / improvements to the existing tests. * Fix test. * Fix rebase. * Tweak kad-ipfs example. * Incorporate some feedback. * Provide easy access and conversion to keys in error results.
2019-07-03 16:16:25 +02:00
cfg.set_query_timeout(Duration::from_secs(5 * 60));
let store = MemoryStore::new(local_peer_id);
let mut behaviour = Kademlia::with_config(local_peer_id, store, cfg);
Kademlia: Address some TODOs - Refactoring - API updates. (#1174) * Address some TODOs, refactor queries and public API. The following left-over issues are addressed: * The key for FIND_NODE requests is generalised to any Multihash, instead of just peer IDs. * All queries get a (configurable) timeout. * Finishing queries as soon as enough results have been received is simplified to avoid code duplication. * No more panics in provider-API-related code paths. The provider API is however still untested and (I think) still incomplete (e.g. expiration of provider records). * Numerous smaller TODOs encountered in the code. The following public API changes / additions are made: * Introduce a `KademliaConfig` with new configuration options for the replication factor and query timeouts. * Rename `find_node` to `get_closest_peers`. * Rename `get_value` to `get_record` and `put_value` to `put_record`, introducing a `Quorum` parameter for both functions, replacing the existing `num_results` parameter with clearer semantics. * Rename `add_providing` to `start_providing` and `remove_providing` to `stop_providing`. * Add a `bootstrap` function that implements a (almost) standard Kademlia bootstrapping procedure. * Rename `KademliaOut` to `KademliaEvent` with an updated list of constructors (some renaming). All events that report query results now report a `Result` to uniformly permit reporting of errors. The following refactorings are made: * Introduce some constants. * Consolidate `query.rs` and `write.rs` behind a common query interface to reduce duplication and facilitate better code reuse, introducing the notion of a query peer iterator. `query/peers/closest.rs` contains the code that was formerly in `query.rs`. `query/peers/fixed.rs` contains a modified variant of `write.rs` (which is removed). The new `query.rs` provides an interface for working with a collection of queries, taking over some code from `behaviour.rs`. * Reduce code duplication in tests and use the current_thread runtime for polling swarms to avoid spurious errors in the test output due to aborted connections when a test finishes prematurely (e.g. because a quorum of results has been collected). * Some additions / improvements to the existing tests. * Fix test. * Fix rebase. * Tweak kad-ipfs example. * Incorporate some feedback. * Provide easy access and conversion to keys in error results.
2019-07-03 16:16:25 +02:00
// Add the bootnodes to the local routing table. `libp2p-dns` built
// into the `transport` resolves the `dnsaddr` when Kademlia tries
// to dial these nodes.
let bootaddr = Multiaddr::from_str("/dnsaddr/bootstrap.libp2p.io")?;
for peer in &BOOTNODES {
behaviour.add_address(&PeerId::from_str(peer)?, bootaddr.clone());
}
Kademlia: Address some TODOs - Refactoring - API updates. (#1174) * Address some TODOs, refactor queries and public API. The following left-over issues are addressed: * The key for FIND_NODE requests is generalised to any Multihash, instead of just peer IDs. * All queries get a (configurable) timeout. * Finishing queries as soon as enough results have been received is simplified to avoid code duplication. * No more panics in provider-API-related code paths. The provider API is however still untested and (I think) still incomplete (e.g. expiration of provider records). * Numerous smaller TODOs encountered in the code. The following public API changes / additions are made: * Introduce a `KademliaConfig` with new configuration options for the replication factor and query timeouts. * Rename `find_node` to `get_closest_peers`. * Rename `get_value` to `get_record` and `put_value` to `put_record`, introducing a `Quorum` parameter for both functions, replacing the existing `num_results` parameter with clearer semantics. * Rename `add_providing` to `start_providing` and `remove_providing` to `stop_providing`. * Add a `bootstrap` function that implements a (almost) standard Kademlia bootstrapping procedure. * Rename `KademliaOut` to `KademliaEvent` with an updated list of constructors (some renaming). All events that report query results now report a `Result` to uniformly permit reporting of errors. The following refactorings are made: * Introduce some constants. * Consolidate `query.rs` and `write.rs` behind a common query interface to reduce duplication and facilitate better code reuse, introducing the notion of a query peer iterator. `query/peers/closest.rs` contains the code that was formerly in `query.rs`. `query/peers/fixed.rs` contains a modified variant of `write.rs` (which is removed). The new `query.rs` provides an interface for working with a collection of queries, taking over some code from `behaviour.rs`. * Reduce code duplication in tests and use the current_thread runtime for polling swarms to avoid spurious errors in the test output due to aborted connections when a test finishes prematurely (e.g. because a quorum of results has been collected). * Some additions / improvements to the existing tests. * Fix test. * Fix rebase. * Tweak kad-ipfs example. * Incorporate some feedback. * Provide easy access and conversion to keys in error results.
2019-07-03 16:16:25 +02:00
Swarm::new(transport, behaviour, local_peer_id)
};
// Order Kademlia to search for a peer.
Kademlia: Address some TODOs - Refactoring - API updates. (#1174) * Address some TODOs, refactor queries and public API. The following left-over issues are addressed: * The key for FIND_NODE requests is generalised to any Multihash, instead of just peer IDs. * All queries get a (configurable) timeout. * Finishing queries as soon as enough results have been received is simplified to avoid code duplication. * No more panics in provider-API-related code paths. The provider API is however still untested and (I think) still incomplete (e.g. expiration of provider records). * Numerous smaller TODOs encountered in the code. The following public API changes / additions are made: * Introduce a `KademliaConfig` with new configuration options for the replication factor and query timeouts. * Rename `find_node` to `get_closest_peers`. * Rename `get_value` to `get_record` and `put_value` to `put_record`, introducing a `Quorum` parameter for both functions, replacing the existing `num_results` parameter with clearer semantics. * Rename `add_providing` to `start_providing` and `remove_providing` to `stop_providing`. * Add a `bootstrap` function that implements a (almost) standard Kademlia bootstrapping procedure. * Rename `KademliaOut` to `KademliaEvent` with an updated list of constructors (some renaming). All events that report query results now report a `Result` to uniformly permit reporting of errors. The following refactorings are made: * Introduce some constants. * Consolidate `query.rs` and `write.rs` behind a common query interface to reduce duplication and facilitate better code reuse, introducing the notion of a query peer iterator. `query/peers/closest.rs` contains the code that was formerly in `query.rs`. `query/peers/fixed.rs` contains a modified variant of `write.rs` (which is removed). The new `query.rs` provides an interface for working with a collection of queries, taking over some code from `behaviour.rs`. * Reduce code duplication in tests and use the current_thread runtime for polling swarms to avoid spurious errors in the test output due to aborted connections when a test finishes prematurely (e.g. because a quorum of results has been collected). * Some additions / improvements to the existing tests. * Fix test. * Fix rebase. * Tweak kad-ipfs example. * Incorporate some feedback. * Provide easy access and conversion to keys in error results.
2019-07-03 16:16:25 +02:00
let to_search: PeerId = if let Some(peer_id) = env::args().nth(1) {
peer_id.parse()?
} else {
identity::Keypair::generate_ed25519().public().into()
};
Kademlia: Address some TODOs - Refactoring - API updates. (#1174) * Address some TODOs, refactor queries and public API. The following left-over issues are addressed: * The key for FIND_NODE requests is generalised to any Multihash, instead of just peer IDs. * All queries get a (configurable) timeout. * Finishing queries as soon as enough results have been received is simplified to avoid code duplication. * No more panics in provider-API-related code paths. The provider API is however still untested and (I think) still incomplete (e.g. expiration of provider records). * Numerous smaller TODOs encountered in the code. The following public API changes / additions are made: * Introduce a `KademliaConfig` with new configuration options for the replication factor and query timeouts. * Rename `find_node` to `get_closest_peers`. * Rename `get_value` to `get_record` and `put_value` to `put_record`, introducing a `Quorum` parameter for both functions, replacing the existing `num_results` parameter with clearer semantics. * Rename `add_providing` to `start_providing` and `remove_providing` to `stop_providing`. * Add a `bootstrap` function that implements a (almost) standard Kademlia bootstrapping procedure. * Rename `KademliaOut` to `KademliaEvent` with an updated list of constructors (some renaming). All events that report query results now report a `Result` to uniformly permit reporting of errors. The following refactorings are made: * Introduce some constants. * Consolidate `query.rs` and `write.rs` behind a common query interface to reduce duplication and facilitate better code reuse, introducing the notion of a query peer iterator. `query/peers/closest.rs` contains the code that was formerly in `query.rs`. `query/peers/fixed.rs` contains a modified variant of `write.rs` (which is removed). The new `query.rs` provides an interface for working with a collection of queries, taking over some code from `behaviour.rs`. * Reduce code duplication in tests and use the current_thread runtime for polling swarms to avoid spurious errors in the test output due to aborted connections when a test finishes prematurely (e.g. because a quorum of results has been collected). * Some additions / improvements to the existing tests. * Fix test. * Fix rebase. * Tweak kad-ipfs example. * Incorporate some feedback. * Provide easy access and conversion to keys in error results.
2019-07-03 16:16:25 +02:00
println!("Searching for the closest peers to {:?}", to_search);
swarm.behaviour_mut().get_closest_peers(to_search);
// Kick it off!
task::block_on(async move {
loop {
let event = swarm.select_next_some().await;
if let SwarmEvent::Behaviour(KademliaEvent::OutboundQueryCompleted {
result: QueryResult::GetClosestPeers(result),
..
}) = event
{
match result {
Ok(ok) => {
if !ok.peers.is_empty() {
println!("Query finished with closest peers: {:#?}", ok.peers)
} else {
// The example is considered failed as there
// should always be at least 1 reachable peer.
2019-12-09 16:50:08 +01:00
println!("Query finished with no closest peers.")
Kademlia: Address some TODOs - Refactoring - API updates. (#1174) * Address some TODOs, refactor queries and public API. The following left-over issues are addressed: * The key for FIND_NODE requests is generalised to any Multihash, instead of just peer IDs. * All queries get a (configurable) timeout. * Finishing queries as soon as enough results have been received is simplified to avoid code duplication. * No more panics in provider-API-related code paths. The provider API is however still untested and (I think) still incomplete (e.g. expiration of provider records). * Numerous smaller TODOs encountered in the code. The following public API changes / additions are made: * Introduce a `KademliaConfig` with new configuration options for the replication factor and query timeouts. * Rename `find_node` to `get_closest_peers`. * Rename `get_value` to `get_record` and `put_value` to `put_record`, introducing a `Quorum` parameter for both functions, replacing the existing `num_results` parameter with clearer semantics. * Rename `add_providing` to `start_providing` and `remove_providing` to `stop_providing`. * Add a `bootstrap` function that implements a (almost) standard Kademlia bootstrapping procedure. * Rename `KademliaOut` to `KademliaEvent` with an updated list of constructors (some renaming). All events that report query results now report a `Result` to uniformly permit reporting of errors. The following refactorings are made: * Introduce some constants. * Consolidate `query.rs` and `write.rs` behind a common query interface to reduce duplication and facilitate better code reuse, introducing the notion of a query peer iterator. `query/peers/closest.rs` contains the code that was formerly in `query.rs`. `query/peers/fixed.rs` contains a modified variant of `write.rs` (which is removed). The new `query.rs` provides an interface for working with a collection of queries, taking over some code from `behaviour.rs`. * Reduce code duplication in tests and use the current_thread runtime for polling swarms to avoid spurious errors in the test output due to aborted connections when a test finishes prematurely (e.g. because a quorum of results has been collected). * Some additions / improvements to the existing tests. * Fix test. * Fix rebase. * Tweak kad-ipfs example. * Incorporate some feedback. * Provide easy access and conversion to keys in error results.
2019-07-03 16:16:25 +02:00
}
}
Err(GetClosestPeersError::Timeout { peers, .. }) => {
if !peers.is_empty() {
println!("Query timed out with closest peers: {:#?}", peers)
} else {
// The example is considered failed as there
// should always be at least 1 reachable peer.
2019-12-09 16:50:08 +01:00
println!("Query timed out with no closest peers.");
Kademlia: Address some TODOs - Refactoring - API updates. (#1174) * Address some TODOs, refactor queries and public API. The following left-over issues are addressed: * The key for FIND_NODE requests is generalised to any Multihash, instead of just peer IDs. * All queries get a (configurable) timeout. * Finishing queries as soon as enough results have been received is simplified to avoid code duplication. * No more panics in provider-API-related code paths. The provider API is however still untested and (I think) still incomplete (e.g. expiration of provider records). * Numerous smaller TODOs encountered in the code. The following public API changes / additions are made: * Introduce a `KademliaConfig` with new configuration options for the replication factor and query timeouts. * Rename `find_node` to `get_closest_peers`. * Rename `get_value` to `get_record` and `put_value` to `put_record`, introducing a `Quorum` parameter for both functions, replacing the existing `num_results` parameter with clearer semantics. * Rename `add_providing` to `start_providing` and `remove_providing` to `stop_providing`. * Add a `bootstrap` function that implements a (almost) standard Kademlia bootstrapping procedure. * Rename `KademliaOut` to `KademliaEvent` with an updated list of constructors (some renaming). All events that report query results now report a `Result` to uniformly permit reporting of errors. The following refactorings are made: * Introduce some constants. * Consolidate `query.rs` and `write.rs` behind a common query interface to reduce duplication and facilitate better code reuse, introducing the notion of a query peer iterator. `query/peers/closest.rs` contains the code that was formerly in `query.rs`. `query/peers/fixed.rs` contains a modified variant of `write.rs` (which is removed). The new `query.rs` provides an interface for working with a collection of queries, taking over some code from `behaviour.rs`. * Reduce code duplication in tests and use the current_thread runtime for polling swarms to avoid spurious errors in the test output due to aborted connections when a test finishes prematurely (e.g. because a quorum of results has been collected). * Some additions / improvements to the existing tests. * Fix test. * Fix rebase. * Tweak kad-ipfs example. * Incorporate some feedback. * Provide easy access and conversion to keys in error results.
2019-07-03 16:16:25 +02:00
}
}
2019-12-09 16:50:08 +01:00
};
break;
}
}
2019-12-09 16:50:08 +01:00
Ok(())
})
}