fluencelabs/rust-libp2p
1
0
Fork 0
mirror of https://github.com/fluencelabs/rust-libp2p synced 2025-06-28 09:11:34 +00:00
Code Issues Projects Releases Wiki Activity

Kademlia: Somewhat complete the records implementation. (#1189)

Browse Source 
* Somewhat complete the implementation of Kademlia records.

This commit relates to [libp2p-146] and [libp2p-1089].

  * All records expire (by default, configurable).
  * Provider records are also stored in the RecordStore, and the RecordStore
    API extended.
  * Background jobs for periodic (re-)replication and (re-)publication
    of records. Regular (value-)records are subject to re-replication and
    re-publication as per standard Kademlia. Provider records are only
    subject to re-publication.
  * For standard Kademlia value lookups (quorum = 1), the record is cached
    at the closest peer to the key that did not return the value, as per
    standard Kademlia.
  * Expiration times of regular (value-)records is computed exponentially
    inversely proportional to the number of nodes between the local node
    and the closest node known to the key (beyond the k closest), as per
    standard Kademlia.

The protobuf messages are extended with two fields: `ttl` and `publisher`
in order to implement the different semantics of re-replication (by any
of the k closest peers to the key, not affecting expiry) and re-publication
(by the original publisher, resetting the expiry). This is not done yet in
other libp2p Kademlia implementations, see e.g. [libp2p-go-323]. The new protobuf fields
have been given somewhat unique identifiers to prevent future collision.

Similarly, periodic re-publication of provider records does not seem to
be done yet in other implementations, see e.g. [libp2p-js-98].

[libp2p-146]: https://github.com/libp2p/rust-libp2p/issues/146
[libp2p-1089]: https://github.com/libp2p/rust-libp2p/issues/1089
[libp2p-go-323]: https://github.com/libp2p/go-libp2p-kad-dht/issues/323
[libp2p-js-98]: https://github.com/libp2p/js-libp2p-kad-dht/issues/98

* Tweak kad-ipfs example.

* Add missing files.

* Ensure new delays are polled immediately.

To ensure task notification, since `NotReady` is returned right after.

* Fix ipfs-kad example and use wasm_timer.

* Small cleanup.

* Incorporate some feedback.

* Adjustments after rebase.

* Distinguish events further.

In order for a user to easily distinguish the result of e.g.
a `put_record` operation from the result of a later republication,
different event constructors are used. Furthermore, for now,
re-replication and "caching" of records (at the closest peer to
the key that did not return a value during a successful lookup)
do not yield events for now as they are less interesting.

* Speed up tests for CI.

* Small refinements and more documentation.

  * Guard a node against overriding records for which it considers
    itself to be the publisher.

  * Document the jobs module more extensively.

* More inline docs around removal of "unreachable" addresses.

* Remove wildcard re-exports.

* Use NonZeroUsize for the constants.

* Re-add method lost on merge.

* Add missing 'pub'.

* Further increase the timeout in the ipfs-kad example.

* Readd log dependency to libp2p-kad.

* Simplify RecordStore API slightly.

* Some more commentary.

* Change Addresses::remove to return Result<(),()>.

Change the semantics of `Addresses::remove` so that the error case
is unambiguous, instead of the success case. Use the `Result` for
clearer semantics to that effect.

* Add some documentation to .
This commit is contained in:
Roman Borschel
2019-07-17 14:40:48 +02:00
committed by GitHub
parent 01bce16d09
commit cde93f5432
21 changed files with 2715 additions and 947 deletions
Download Patch File Download Diff File Expand all files Collapse all files

103
protocols/kad/src/protocol.rs
View File

@ -39,11 +39,12 @@ use libp2p_core::{Multiaddr, PeerId};
use libp2p_core::upgrade::{InboundUpgrade, OutboundUpgrade, UpgradeInfo, Negotiated};
use multihash::Multihash;
use protobuf::{self, Message};
use std::{borrow::Cow, convert::TryFrom};
use std::{borrow::Cow, convert::TryFrom, time::Duration};
use std::{io, iter};
use tokio_codec::Framed;
use tokio_io::{AsyncRead, AsyncWrite};
use unsigned_varint::codec;
use wasm_timer::Instant;
/// Status of our connection to a node reported by the Kademlia protocol.
#[derive(Copy, Clone, PartialEq, Eq, Debug, Hash)]
@ -272,7 +273,7 @@ pub enum KadRequestMsg {
/// Key for which we should add providers.
key: Multihash,
/// Known provider for this key.
provider_peer: KadPeer,
provider: KadPeer,
},
/// Request to get a value from the dht records.
@ -283,10 +284,7 @@ pub enum KadRequestMsg {
/// Request to put a value into the dht records.
PutValue {
/// The key of the record.
key: Multihash,
/// The value of the record.
value: Vec<u8>,
record: Record,
}
}
@ -313,7 +311,7 @@ pub enum KadResponseMsg {
/// Response to a `GetValue`.
GetValue {
/// Result that might have been found
result: Option<Record>,
record: Option<Record>,
/// Nodes closest to the key
closer_peers: Vec<KadPeer>,
},
@ -349,12 +347,12 @@ fn req_msg_to_proto(kad_msg: KadRequestMsg) -> proto::Message {
msg.set_clusterLevelRaw(10);
msg
}
KadRequestMsg::AddProvider { key, provider_peer } => {
KadRequestMsg::AddProvider { key, provider } => {
let mut msg = proto::Message::new();
msg.set_field_type(proto::Message_MessageType::ADD_PROVIDER);
msg.set_clusterLevelRaw(10);
msg.set_key(key.into_bytes());
msg.mut_providerPeers().push(provider_peer.into());
msg.mut_providerPeers().push(provider.into());
msg
}
KadRequestMsg::GetValue { key } => {
@ -365,14 +363,10 @@ fn req_msg_to_proto(kad_msg: KadRequestMsg) -> proto::Message {
msg
}
KadRequestMsg::PutValue { key, value} => {
KadRequestMsg::PutValue { record } => {
let mut msg = proto::Message::new();
msg.set_field_type(proto::Message_MessageType::PUT_VALUE);
let mut record = proto::Record::new();
record.set_value(value);
record.set_key(key.into_bytes());
msg.set_record(record);
msg.set_record(record_to_proto(record));
msg
}
}
@ -411,7 +405,7 @@ fn resp_msg_to_proto(kad_msg: KadResponseMsg) -> proto::Message {
msg
}
KadResponseMsg::GetValue {
result,
record,
closer_peers,
} => {
let mut msg = proto::Message::new();
@ -420,12 +414,8 @@ fn resp_msg_to_proto(kad_msg: KadResponseMsg) -> proto::Message {
for peer in closer_peers {
msg.mut_closerPeers().push(peer.into());
}
if let Some(Record{ key, value }) = result {
let mut record = proto::Record::new();
record.set_key(key.into_bytes());
record.set_value(value);
msg.set_record(record);
if let Some(record) = record {
msg.set_record(record_to_proto(record));
}
msg
@ -456,9 +446,8 @@ fn proto_to_req_msg(mut message: proto::Message) -> Result<KadRequestMsg, io::Er
proto::Message_MessageType::PING => Ok(KadRequestMsg::Ping),
proto::Message_MessageType::PUT_VALUE => {
let record = message.mut_record();
let key = Multihash::from_bytes(record.take_key()).map_err(invalid_data)?;
Ok(KadRequestMsg::PutValue { key, value: record.take_value() })
let record = record_from_proto(message.take_record())?;
Ok(KadRequestMsg::PutValue { record })
}
proto::Message_MessageType::GET_VALUE => {
@ -481,14 +470,14 @@ fn proto_to_req_msg(mut message: proto::Message) -> Result<KadRequestMsg, io::Er
// TODO: for now we don't parse the peer properly, so it is possible that we get
// parsing errors for peers even when they are valid; we ignore these
// errors for now, but ultimately we should just error altogether
let provider_peer = message
let provider = message
.mut_providerPeers()
.iter_mut()
.find_map(|peer| KadPeer::try_from(peer).ok());
if let Some(provider_peer) = provider_peer {
if let Some(provider) = provider {
let key = Multihash::from_bytes(message.take_key()).map_err(invalid_data)?;
Ok(KadRequestMsg::AddProvider { key, provider_peer })
Ok(KadRequestMsg::AddProvider { key, provider })
} else {
Err(invalid_data("ADD_PROVIDER message with no valid peer."))
}
@ -504,14 +493,12 @@ fn proto_to_resp_msg(mut message: proto::Message) -> Result<KadResponseMsg, io::
proto::Message_MessageType::PING => Ok(KadResponseMsg::Pong),
proto::Message_MessageType::GET_VALUE => {
let result = match message.has_record() {
true => {
let mut record = message.take_record();
let key = Multihash::from_bytes(record.take_key()).map_err(invalid_data)?;
Some(Record { key, value: record.take_value() })
}
false => None,
};
let record =
if message.has_record() {
Some(record_from_proto(message.take_record())?)
} else {
None
};
let closer_peers = message
.mut_closerPeers()
@ -519,7 +506,7 @@ fn proto_to_resp_msg(mut message: proto::Message) -> Result<KadResponseMsg, io::
.filter_map(|peer| KadPeer::try_from(peer).ok())
.collect::<Vec<_>>();
Ok(KadResponseMsg::GetValue { result, closer_peers })
Ok(KadResponseMsg::GetValue { record, closer_peers })
},
proto::Message_MessageType::FIND_NODE => {
@ -569,6 +556,48 @@ fn proto_to_resp_msg(mut message: proto::Message) -> Result<KadResponseMsg, io::
}
}
fn record_from_proto(mut record: proto::Record) -> Result<Record, io::Error> {
let key = Multihash::from_bytes(record.take_key()).map_err(invalid_data)?;
let value = record.take_value();
let publisher =
if record.publisher.len() > 0 {
PeerId::from_bytes(record.take_publisher())
.map(Some)
.map_err(|_| invalid_data("Invalid publisher peer ID."))?
} else {
None
};
let expires =
if record.ttl > 0 {
Some(Instant::now() + Duration::from_secs(record.ttl as u64))
} else {
None
};
Ok(Record { key, value, publisher, expires })
}
fn record_to_proto(record: Record) -> proto::Record {
let mut pb_record = proto::Record::new();
pb_record.key = record.key.into_bytes();
pb_record.value = record.value;
if let Some(p) = record.publisher {
pb_record.publisher = p.into_bytes();
}
if let Some(t) = record.expires {
let now = Instant::now();
if t > now {
pb_record.ttl = (t - now).as_secs() as u32;
} else {
pb_record.ttl = 1; // because 0 means "does not expire"
}
}
pb_record
}
/// Creates an `io::Error` with `io::ErrorKind::InvalidData`.
fn invalid_data<E>(e: E) -> io::Error
where