Merge branch 'develop' into abci-spec-docs

docs/specification/new-spec/README.md

@@ -9,6 +9,7 @@ It contains the following components:
 - [Encoding and Digests](encoding.md)
 - [Blockchain](blockchain.md)
 - [State](state.md)
+- [P2P](p2p/node.md)
 
 ## Overview
 

docs/specification/new-spec/p2p/config.md

@@ -0,0 +1,39 @@
+# P2P Config
+
+Here we describe configuration options around the Peer Exchange.
+
+## Seed Mode
+
+`--p2p.seed_mode`
+
+The node operates in seed mode. It will kick incoming peers after sharing some peers.
+It will continually crawl the network for peers.
+
+## Seeds
+
+`--p2p.seeds “1.2.3.4:466656,2.3.4.5:4444”`
+
+Dials these seeds when we need more peers. They will return a list of peers and then disconnect.
+If we already have enough peers in the address book, we may never need to dial them.
+
+## Persistent Peers
+
+`--p2p.persistent_peers “1.2.3.4:46656,2.3.4.5:466656”`
+
+Dial these peers and auto-redial them if the connection fails.
+These are intended to be trusted persistent peers that can help
+anchor us in the p2p network.
+
+Note that the auto-redial uses exponential backoff and will give up
+after a day of trying to connect.
+
+NOTE: If `dial_seeds` and `persistent_peers` intersect,
+the user will be WARNED that seeds may auto-close connections
+and the node may not be able to keep the connection persistent.
+
+## Private Persistent Peers
+
+`--p2p.private_persistent_peers “1.2.3.4:46656,2.3.4.5:466656”`
+
+These are persistent peers that we do not add to the address book or
+gossip to other peers. They stay private to us.

docs/specification/new-spec/p2p/node.md

@@ -3,11 +3,6 @@
 A Tendermint P2P network has different kinds of nodes with different requirements for connectivity to others.
 This document describes what kind of nodes Tendermint should enable and how they should work.
 
-## Node startup options
---p2p.seed_mode // If present, this node operates in seed mode.  It will kick incoming peers after sharing some peers.
---p2p.seeds “1.2.3.4:466656,2.3.4.5:4444” // Dials these seeds to get peers and disconnects.
---p2p.persistent_peers “1.2.3.4:46656,2.3.4.5:466656” // These connections will be auto-redialed.  If dial_seeds and persistent intersect, the user will be WARNED that seeds may auto-close connections and the node may not be able to keep the connection persistent
-
 ## Seeds
 
 Seeds are the first point of contact for a new node.
@@ -17,22 +12,36 @@ Seeds should operate full nodes, and with the PEX reactor in a "crawler" mode
 that continuously explores to validate the availability of peers.
 
 Seeds should only respond with some top percentile of the best peers it knows about.
+See [reputation] for details on peer quality.
 
 ## New Full Node
 
-A new node has seeds hardcoded into the software, but they can also be set manually (config file or flags).
+A new node needs a few things to connect to the network:
-The new node must also have access to a recent block height, H, and hash, HASH.
+- a list of seeds, which can be provided to Tendermint via config file or flags,
+or hardcoded into the software by in-process apps
+- a `ChainID`, also called `Network` at the p2p layer
+- a recent block height, H, and hash, HASH for the blockchain.
 
-The node then queries some seeds for peers for its chain,
+The values `H` and `HASH` must be received and corroborated by means external to Tendermint, and specific to the user - ie. via the user's trusted social consensus.
+This requirement to validate `H` and `HASH` out-of-band and via social consensus
+is the essential difference in security models between Proof-of-Work and Proof-of-Stake blockchains.
+
+With the above, the node then queries some seeds for peers for its chain,
 dials those peers, and runs the Tendermint protocols with those it successfully connects to.
 
 When the peer catches up to height H, it ensures the block hash matches HASH.
+If not, Tendermint will exit, and the user must try again - either they are connected
+to bad peers or their social consensus was invalidated.
 
 ## Restarted Full Node
 
 A node checks its address book on startup and attempts to connect to peers from there.
 If it can't connect to any peers after some time, it falls back to the seeds to find more.
 
+Restarted full nodes can run the `blockchain` or `consensus` reactor protocols to sync up
+to the latest state of the blockchain, assuming they aren't too far behind.
+If they are too far behind, they may need to validate a recent `H` and `HASH` out-of-band again.
+
 ## Validator Node
 
 A validator node is a node that interfaces with a validator signing key.

docs/specification/new-spec/p2p/peer.md

@@ -10,19 +10,19 @@ Each peer has an ID defined as `peer.ID == peer.PrivKey.Address()`, where `Addre
 
 Peer ID's must come with some Proof-of-Work; that is,
 they must satisfy `peer.PrivKey.Address() < target` for some difficulty target.
-This ensures they are not too easy to generate.
+This ensures they are not too easy to generate. To begin, let `target == 2^240`.
 
-A single peer ID can have multiple IP addresses associated with - for simplicity, we only keep track
+A single peer ID can have multiple IP addresses associated with it.
-of the latest one.
+For simplicity, we only keep track of the latest one.
 
 When attempting to connect to a peer, we use the PeerURL: `<ID>@<IP>:<PORT>`.
 We will attempt to connect to the peer at IP:PORT, and verify,
 via authenticated encryption, that it is in possession of the private key
 corresponding to `<ID>`. This prevents man-in-the-middle attacks on the peer layer.
 
-Peers can also be connected to without specifying an ID, ie. `<IP>:<PORT>`.
+Peers can also be connected to without specifying an ID, ie. just `<IP>:<PORT>`.
-In this case, the peer cannot be authenticated and other means, such as a VPN,
+In this case, the peer must be authenticated out-of-band of Tendermint,
-must be used.
+for instance via VPN
 
 ## Connections
 
@@ -40,18 +40,27 @@ It goes as follows:
 - send the ephemeral public key to the peer
 - wait to receive the peer's ephemeral public key
 - compute the Diffie-Hellman shared secret using the peers ephemeral public key and our ephemeral private key
-- generate nonces to use for encryption
+- generate two nonces to use for encryption (sending and receiving) as follows:
-    - TODO
+    - sort the ephemeral public keys in ascending order and concatenate them
-- all communications from now on are encrypted using the shared secret
+    - RIPEMD160 the result
-- generate a common challenge to sign
+    - append 4 empty bytes (extending the hash to 24-bytes)
+    - the result is nonce1
+    - flip the last bit of nonce1 to get nonce2
+    - if we had the smaller ephemeral pubkey, use nonce1 for receiving, nonce2 for sending;
+        else the opposite
+- all communications from now on are encrypted using the shared secret and the nonces, where each nonce
+- we now have an encrypted channel, but still need to authenticate
+increments by 2 every time it is used
+- generate a common challenge to sign:
+    - SHA256 of the sorted (lowest first) and concatenated ephemeral pub keys
 - sign the common challenge with our persistent private key
-- send the signed challenge and persistent public key to the peer
+- send the go-wire encoded persistent pubkey and signature to the peer
-- wait to receive the signed challenge and persistent public key from the peer
+- wait to receive the persistent public key and signature from the peer
-- verify the signature in the signed challenge using the peers persistent public key
+- verify the signature on the challenge using the peer's persistent public key
 
 
 If this is an outgoing connection (we dialed the peer) and we used a peer ID,
-then finally verify that the `peer.PubKey` corresponds to the peer ID we dialed,
+then finally verify that the peer's persistent public key corresponds to the peer ID we dialed,
 ie. `peer.PubKey.Address() == <ID>`.
 
 The connection has now been authenticated. All traffic is encrypted.
@@ -60,21 +69,20 @@ Note that only the dialer can authenticate the identity of the peer,
 but this is what we care about since when we join the network we wish to
 ensure we have reached the intended peer (and are not being MITMd).
 
-
 ### Peer Filter
 
-Before continuing, we check if the new peer has the same ID has ourselves or
+Before continuing, we check if the new peer has the same ID as ourselves or
 an existing peer. If so, we disconnect.
 
 We also check the peer's address and public key against
 an optional whitelist which can be managed through the ABCI app -
-if the whitelist is enabled and the peer is not on it, the connection is
+if the whitelist is enabled and the peer does not qualigy, the connection is
 terminated.
 
 
 ### Tendermint Version Handshake
 
-The Tendermint Version Handshake allows the peers to exchange their NodeInfo, which contains:
+The Tendermint Version Handshake allows the peers to exchange their NodeInfo:
 
 ```
 type NodeInfo struct {
@@ -95,11 +103,16 @@ The connection is disconnected if:
 - `peer.NodeInfo.Version` Major is not the same as ours
 - `peer.NodeInfo.Version` Minor is not the same as ours
 - `peer.NodeInfo.Network` is not the same as ours
+- `peer.Channels` does not intersect with our known Channels.
 
 
-At this point, if we have not disconnected, the peer is valid and added to the switch,
+At this point, if we have not disconnected, the peer is valid.
-so it is added to all reactors.
+It is added to the switch and hence all reactors via the `AddPeer` method.
+Note that each reactor may handle multiple channels.
 
+## Connection Activity
 
-### Connection Activity
+Once a peer is added, incoming messages for a given reactor are handled through
+that reactor's `Receive` method, and output messages are sent directly by the Reactors
+on each peer. A typical reactor maintains per-peer go-routine/s that handle this.
 

docs/specification/new-spec/p2p/pex.md

@@ -0,0 +1,94 @@
+# Peer Strategy and Exchange
+
+Here we outline the design of the AddressBook
+and how it used by the Peer Exchange Reactor (PEX) to ensure we are connected
+to good peers and to gossip peers to others.
+
+## Peer Types
+
+Certain peers are special in that they are specified by the user as `persistent`,
+which means we auto-redial them if the connection fails.
+Some such peers can additional be marked as `private`, which means
+we will not gossip them to others.
+
+All others peers are tracked using an address book.
+
+## Discovery
+
+Peer discovery begins with a list of seeds.
+When we have no peers, or have been unable to find enough peers from existing ones,
+we dial a randomly selected seed to get a list of peers to dial.
+
+So long as we have less than `MaxPeers`, we periodically request additional peers
+from each of our own. If sufficient time goes by and we still can't find enough peers,
+we try the seeds again.
+
+## Address Book
+
+Peers are tracked via their ID (their PubKey.Address()).
+For each ID, the address book keeps the most recent IP:PORT.
+Peers are added to the address book from the PEX when they first connect to us or
+when we hear about them from other peers.
+
+The address book is arranged in sets of buckets, and distinguishes between
+vetted and unvetted peers. It keeps different sets of buckets for vetted and
+unvetted peers. Buckets provide randomization over peer selection.
+
+A vetted peer can only be in one bucket. An unvetted peer can be in multiple buckets.
+
+## Vetting
+
+When a peer is first added, it is unvetted.
+Marking a peer as vetted is outside the scope of the `p2p` package.
+For Tendermint, a Peer becomes vetted once it has contributed sufficiently
+at the consensus layer; ie. once it has sent us valid and not-yet-known
+votes and/or block parts for `NumBlocksForVetted` blocks.
+Other users of the p2p package can determine their own conditions for when a peer is marked vetted.
+
+If a peer becomes vetted but there are already too many vetted peers,
+a randomly selected one of the vetted peers becomes unvetted.
+
+If a peer becomes unvetted (either a new peer, or one that was previously vetted),
+a randomly selected one of the unvetted peers is removed from the address book.
+
+More fine-grained tracking of peer behaviour can be done using
+a Trust Metric, but it's best to start with something simple.
+
+## Select Peers to Dial
+
+When we need more peers, we pick them randomly from the addrbook with some
+configurable bias for unvetted peers. The bias should be lower when we have fewer peers,
+and can increase as we obtain more, ensuring that our first peers are more trustworthy,
+but always giving us the chance to discover new good peers.
+
+## Select Peers to Exchange
+
+When we’re asked for peers, we select them as follows:
+- select at most `maxGetSelection` peers
+- try to select at least `minGetSelection` peers - if we have less than that, select them all.
+- select a random, unbiased `getSelectionPercent` of the peers
+
+Send the selected peers. Note we select peers for sending without bias for vetted/unvetted.
+
+## Preventing Spam
+
+There are various cases where we decide a peer has misbehaved and we disconnect from them.
+When this happens, the peer is removed from the address book and black listed for
+some amount of time. We call this "Disconnect and Mark".
+Note that the bad behaviour may be detected outside the PEX reactor itseld
+(for instance, in the mconnection, or another reactor), but it must be communicated to the PEX reactor
+so it can remove and mark the peer.
+
+In the PEX, if a peer sends us unsolicited lists of peers,
+or if the peer sends too many requests for more peers in a given amount of time,
+we Disconnect and Mark.
+
+## Trust Metric
+
+The quality of peers can be tracked in more fine-grained detail using a
+Proportional-Integral-Derrivative (PID) controller that incorporates
+current, past, and rate-of-change data to inform peer quality.
+
+While a PID trust metric has been implemented, it remains for future work
+to use it in the PEX.
+

docs/specification/new-spec/p2p/trustmetric.md

@@ -1,11 +1,4 @@
 
-# Peer Strategy
-
-Peers are managed using an address book and a trust metric.
-The book keeps a record of vetted peers and unvetted peers.
-When we need more peers, we pick them randomly from the addrbook with some
-configurable bias for unvetted peers. When we’re asked for peers, we provide a random selection with no bias.
-
 The trust metric tracks the quality of the peers.
 When a peer exceeds a certain quality for a certain amount of time,
 it is marked as vetted in the addrbook.
@@ -17,7 +10,7 @@ possibly only if its below some absolute minimum ?
 Peer quality is tracked in the connection and across the reactors.
 Behaviours are defined as one of:
     - fatal - something outright malicious. we should disconnect and remember them.
-        - bad - any kind of timeout, msgs that dont unmarshal, or fail other validity checks, or msgs we didn't ask for or arent expecting
+    - bad - any kind of timeout, msgs that dont unmarshal, or fail other validity checks, or msgs we didn't ask for or arent expecting
-            - neutral - normal correct behaviour. unknown channels/msg types (version upgrades).
+    - neutral - normal correct behaviour. unknown channels/msg types (version upgrades).
-                - good - some random majority of peers per reactor sending us useful messages
+    - good - some random majority of peers per reactor sending us useful messages
 

p2p/README.md

@@ -9,5 +9,6 @@ See:
 - [docs/connection] for details on how connections and multiplexing work
 - [docs/peer] for details on peer ID, handshakes, and peer exchange
 - [docs/node] for details about different types of nodes and how they should work
-- [docs/reputation] for details on how peer reputation is managed
+- [docs/pex] for details on peer discovery and exchange
+- [docs/config] for details on some config options