docs consolidation

docs/specification/new-spec/README.md

@@ -1,7 +1,8 @@
 # Tendermint Specification
 
 This is a markdown specification of the Tendermint blockchain.
-It defines the base data structures used in the blockchain and how they are validated.
+It defines the base data structures, how they are validated,
+and how they are communicated over the network.
 
 XXX: this spec is a work in progress and not yet complete - see github
 [isses](https://github.com/tendermint/tendermint/issues) and
@@ -14,12 +15,25 @@ please submit them to our [bug bounty](https://tendermint.com/security)!
 
 ## Contents
 
+### Data Structures
+
 - [Overview](#overview)
 - [Encoding and Digests](encoding.md)
 - [Blockchain](blockchain.md)
 - [State](state.md)
-- [Consensus](consensus.md)
-- [P2P](p2p/node.md)
+
+### P2P and Network Protocols
+
+- [The Base P2P Layer](p2p/README.md): multiplex the protocols ("reactors") on authenticated and encrypted TCP conns
+- [Peer Exchange (PEX)](pex/README.md): gossip known peer addresses so peers can find eachother
+- [Block Sync](block_sync/README.md): gossip blocks so peers can catch up quickly
+- [Consensus](consensus/README.md): gossip votes and block parts so new blocks can be committed
+- [Mempool](mempool/README.md): gossip transactions so they get included in blocks
+- [Evidence](evidence/README.md): TODO
+
+### More
+- [Light Client](light_client/README.md): TODO
+- [Persistence](persistence/README.md): TODO
 
 ## Overview
 
@@ -60,10 +74,3 @@ Also note that information like the transaction results and the validator set ar
 directly included in the block - only their cryptographic digests (Merkle roots) are.
 Hence, verification of a block requires a separate data structure to store this information.
 We call this the `State`. Block verification also requires access to the previous block.
-
-## TODO
-
-- Light Client
-- P2P
-- Reactor protocols (consensus, mempool, blockchain, pex)
-

docs/specification/new-spec/blockchain.md

@@ -10,7 +10,7 @@ The Tendermint blockchains consists of a short list of basic data types:
 ## Block
 
 A block consists of a header, a list of transactions, a list of votes (the commit),
-and a list of evidence if malfeasance (ie. signing conflicting votes).
+and a list of evidence of malfeasance (ie. signing conflicting votes).
 
 ```go
 type Block struct {
@@ -366,10 +366,10 @@ against the given signature and message bytes.
 
 ## Evidence
 
+TODO
 
 ```
-
-
+TODO
 ```
 
 Every piece of evidence contains two conflicting votes from a single validator that
@@ -384,7 +384,35 @@ Once a block is validated, it can be executed against the state.
 The state follows the recursive equation:
 
 ```go
-app = NewABCIApp
 state(1) = InitialState
-state(h+1) <- Execute(state(h), app, block(h))
+state(h+1) <- Execute(state(h), ABCIApp, block(h))
 ```
+
+Where `InitialState` includes the initial consensus parameters and validator set,
+and `ABCIApp` is an ABCI application that can return results and changes to the validator
+set (TODO). Execute is defined as:
+
+```go
+Execute(s State, app ABCIApp, block Block) State {
+    TODO: just spell out ApplyBlock here
+    and remove ABCIResponses struct.
+    abciResponses := app.ApplyBlock(block)
+
+    return State{
+        LastResults: abciResponses.DeliverTxResults,
+        AppHash: abciResponses.AppHash,
+        Validators: UpdateValidators(state.Validators, abciResponses.ValidatorChanges),
+        LastValidators: state.Validators,
+        ConsensusParams: UpdateConsensusParams(state.ConsensusParams, abci.Responses.ConsensusParamChanges),
+    }
+}
+
+type ABCIResponses struct {
+    DeliverTxResults        []Result
+    ValidatorChanges        []Validator
+    ConsensusParamChanges   ConsensusParams
+    AppHash                 []byte
+}
+```
+
+

docs/specification/new-spec/reactors/block_sync/impl.md

@@ -1,47 +1,3 @@
-# Blockchain Reactor
-
-The Blockchain Reactor's high level responsibility is to enable peers who are
-far behind the current state of the consensus to quickly catch up by downloading
-many blocks in parallel, verifying their commits, and executing them against the
-ABCI application.
-
-Tendermint full nodes run the Blockchain Reactor as a service to provide blocks
-to new nodes. New nodes run the Blockchain Reactor in "fast_sync" mode,
-where they actively make requests for more blocks until they sync up.
-Once caught up, they disable "fast_sync" mode, and turn on the Consensus Reactor.
-
-## Message Types
-
-```go
-const (
-    msgTypeBlockRequest    = byte(0x10)
-    msgTypeBlockResponse   = byte(0x11)
-    msgTypeNoBlockResponse = byte(0x12)
-    msgTypeStatusResponse  = byte(0x20)
-    msgTypeStatusRequest   = byte(0x21)
-)
-```
-
-```go
-type bcBlockRequestMessage struct {
-    Height int64
-}
-
-type bcNoBlockResponseMessage struct {
-    Height int64
-}
-
-type bcBlockResponseMessage struct {
-    Block Block
-}
-
-type bcStatusRequestMessage struct {
-    Height int64
-
-type bcStatusResponseMessage struct {
-    Height int64
-}
-```
 
 ## Blockchain Reactor
 

docs/specification/new-spec/reactors/block_sync/reactor.md

@@ -0,0 +1,49 @@
+# Blockchain Reactor
+
+The Blockchain Reactor's high level responsibility is to enable peers who are
+far behind the current state of the consensus to quickly catch up by downloading
+many blocks in parallel, verifying their commits, and executing them against the
+ABCI application.
+
+Tendermint full nodes run the Blockchain Reactor as a service to provide blocks
+to new nodes. New nodes run the Blockchain Reactor in "fast_sync" mode,
+where they actively make requests for more blocks until they sync up.
+Once caught up, "fast_sync" mode is disabled and the node switches to
+using the Consensus Reactor. , and turn on the Consensus Reactor.
+
+## Message Types
+
+```go
+const (
+    msgTypeBlockRequest    = byte(0x10)
+    msgTypeBlockResponse   = byte(0x11)
+    msgTypeNoBlockResponse = byte(0x12)
+    msgTypeStatusResponse  = byte(0x20)
+    msgTypeStatusRequest   = byte(0x21)
+)
+```
+
+```go
+type bcBlockRequestMessage struct {
+    Height int64
+}
+
+type bcNoBlockResponseMessage struct {
+    Height int64
+}
+
+type bcBlockResponseMessage struct {
+    Block Block
+}
+
+type bcStatusRequestMessage struct {
+    Height int64
+
+type bcStatusResponseMessage struct {
+    Height int64
+}
+```
+
+## Protocol
+
+TODO

docs/specification/new-spec/reactors/mempool/README.md

@@ -0,0 +1,11 @@
+# Mempool Specification
+
+This package contains documents specifying the functionality
+of the mempool module.
+
+Components:
+
+* [Config](./config.md) - how to configure it
+* [External Messages](./messages.md) - The messages we accept over p2p and rpc interfaces
+* [Functionality](./functionality.md) - high-level description of the functionality it provides
+* [Concurrency Model](./concurrency.md) - What guarantees we provide, what locks we require.

docs/specification/new-spec/reactors/mempool/concurrency.md

@@ -0,0 +1,8 @@
+# Mempool Concurrency
+
+Look at the concurrency model this uses...
+
+* Receiving CheckTx
+* Broadcasting new tx
+* Interfaces with consensus engine, reap/update while checking
+* Calling the ABCI app (ordering. callbacks. how proxy works alongside the blockchain proxy which actually writes blocks)

docs/specification/new-spec/reactors/mempool/config.md

@@ -0,0 +1,59 @@
+# Mempool Configuration
+
+Here we describe configuration options around mempool.
+For the purposes of this document, they are described
+as command-line flags, but they can also be passed in as
+environmental variables or in the config.toml file. The
+following are all equivalent:
+
+Flag: `--mempool.recheck_empty=false`
+
+Environment: `TM_MEMPOOL_RECHECK_EMPTY=false`
+
+Config:
+```
+[mempool]
+recheck_empty = false
+```
+
+
+## Recheck
+
+`--mempool.recheck=false` (default: true)
+
+`--mempool.recheck_empty=false` (default: true)
+
+Recheck determines if the mempool rechecks all pending
+transactions after a block was committed. Once a block
+is committed, the mempool removes all valid transactions
+that were successfully included in the block.
+
+If `recheck` is true, then it will rerun CheckTx on
+all remaining transactions with the new block state.
+
+If the block contained no transactions, it will skip the
+recheck unless `recheck_empty` is true.
+
+## Broadcast
+
+`--mempool.broadcast=false` (default: true)
+
+Determines whether this node gossips any valid transactions
+that arrive in mempool. Default is to gossip anything that
+passes checktx. If this is disabled, transactions are not
+gossiped, but instead stored locally and added to the next
+block this node is the proposer.
+
+## WalDir
+
+`--mempool.wal_dir=/tmp/gaia/mempool.wal` (default: $TM_HOME/data/mempool.wal)
+
+This defines the directory where mempool writes the write-ahead
+logs. These files can be used to reload unbroadcasted
+transactions if the node crashes.
+
+If the directory passed in is an absolute path, the wal file is
+created there. If the directory is a relative path, the path is
+appended to home directory of the tendermint process to
+generate an absolute path to the wal directory
+(default `$HOME/.tendermint` or set via `TM_HOME` or `--home``)

docs/specification/new-spec/reactors/mempool/functionality.md

@@ -0,0 +1,37 @@
+# Mempool Functionality
+
+The mempool maintains a list of potentially valid transactions,
+both to broadcast to other nodes, as well as to provide to the
+consensus reactor when it is selected as the block proposer.
+
+There are two sides to the mempool state:
+
+* External: get, check, and broadcast new transactions
+* Internal: return valid transaction, update list after block commit
+
+
+## External functionality
+
+External functionality is exposed via network interfaces
+to potentially untrusted actors.
+
+* CheckTx - triggered via RPC or P2P
+* Broadcast - gossip messages after a successful check
+
+## Internal functionality
+
+Internal functionality is exposed via method calls to other
+code compiled into the tendermint binary.
+
+* Reap - get tx to propose in next block
+* Update - remove tx that were included in last block
+* ABCI.CheckTx - call ABCI app to validate the tx
+
+What does it provide the consensus reactor?
+What guarantees does it need from the ABCI app?
+(talk about interleaving processes in concurrency)
+
+## Optimizations
+
+Talk about the LRU cache to make sure we don't process any
+tx that we have seen before

docs/specification/new-spec/reactors/mempool/messages.md

@@ -0,0 +1,60 @@
+# Mempool Messages
+
+## P2P Messages
+
+There is currently only one message that Mempool broadcasts
+and receives over the p2p gossip network (via the reactor):
+`TxMessage`
+
+```go
+// TxMessage is a MempoolMessage containing a transaction.
+type TxMessage struct {
+    Tx types.Tx
+}
+```
+
+TxMessage is go-wire encoded and prepended with `0x1` as a
+"type byte". This is followed by a go-wire encoded byte-slice.
+Prefix of 40=0x28 byte tx is: `0x010128...` followed by
+the actual 40-byte tx. Prefix of 350=0x015e byte tx is:
+`0x0102015e...` followed by the actual 350 byte tx.
+
+(Please see the [go-wire repo](https://github.com/tendermint/go-wire#an-interface-example) for more information)
+
+## RPC Messages
+
+Mempool exposes `CheckTx([]byte)` over the RPC interface.
+
+It can be posted via `broadcast_commit`, `broadcast_sync` or
+`broadcast_async`. They all parse a message with one argument,
+`"tx": "HEX_ENCODED_BINARY"` and differ in only how long they
+wait before returning (sync makes sure CheckTx passes, commit
+makes sure it was included in a signed block).
+
+Request (`POST http://gaia.zone:46657/`):
+```json
+{
+ "id": "",
+ "jsonrpc": "2.0",
+ "method": "broadcast_sync",
+ "params": {
+    "tx": "F012A4BC68..."
+ }
+}
+```
+
+
+Response:
+```json
+{
+ "error": "",
+ "result": {
+     "hash": "E39AAB7A537ABAA237831742DCE1117F187C3C52",
+     "log": "",
+     "data": "",
+     "code": 0
+ },
+ "id": "",
+ "jsonrpc": "2.0"
+}
+```

docs/specification/new-spec/state.md

@@ -3,10 +3,15 @@
 ## State
 
 The state contains information whose cryptographic digest is included in block headers, and thus is
-necessary for validating new blocks. For instance, the Merkle root of the results from executing the
-previous block, or the Merkle root of the current validators. While neither the results of
-transactions now the validators are ever included in the blockchain itself, the Merkle roots are,
-and hence we need a separate data structure to track them.
+necessary for validating new blocks. For instance, the set of validators and the results of
+transactions are never included in blocks, but their Merkle roots are - the state keeps track of them.
+
+Note that the `State` object itself is an implementation detail, since it is never
+included in a block or gossipped over the network, and we never compute
+its hash. However, the types it contains are part of the specification, since
+their Merkle roots are included in blocks.
+
+For details on an implementation of `State` with persistence, see TODO
 
 ```go
 type State struct {
@@ -77,28 +82,3 @@ TODO:
 
 TODO:
 
-## Execution
-
-We define an `Execute` function that takes a state and a block,
-executes the block against the application, and returns an updated state.
-
-```go
-Execute(s State, app ABCIApp, block Block) State {
-    abciResponses := app.ApplyBlock(block)
-
-    return State{
-        LastResults: abciResponses.DeliverTxResults,
-        AppHash: abciResponses.AppHash,
-        Validators: UpdateValidators(state.Validators, abciResponses.ValidatorChanges),
-        LastValidators: state.Validators,
-        ConsensusParams: UpdateConsensusParams(state.ConsensusParams, abci.Responses.ConsensusParamChanges),
-    }
-}
-
-type ABCIResponses struct {
-    DeliverTxResults        []Result
-    ValidatorChanges        []Validator
-    ConsensusParamChanges   ConsensusParams
-    AppHash                 []byte
-}
-```