Merge pull request #793 from tendermint/release-v0.12.0

Release v0.12.0

.codecov.yml

@@ -0,0 +1,21 @@
+
+#
+# This codecov.yml is the default configuration for
+# all repositories on Codecov. You may adjust the settings
+# below in your own codecov.yml in your repository.
+#
+coverage:
+  precision: 2
+  round: down
+  range: 70...100
+
+  status:
+    # Learn more at https://codecov.io/docs#yaml_default_commit_status
+    project: 
+      default:
+        threshold: 1% # allow this much decrease on project
+    changes: false
+
+comment:
+  layout: "header, diff"
+  behavior: default  # update if exists else create new

.editorconfig

@@ -0,0 +1,19 @@
+# top-most EditorConfig file
+root = true
+
+# Unix-style newlines with a newline ending every file
+[*]
+charset = utf-8
+end_of_line = lf
+insert_final_newline = true
+trim_trailing_whitespace = true
+
+[Makefile]
+indent_style = tab
+
+[*.sh]
+indent_style = tab
+
+[*.proto]
+indent_style = space
+indent_size = 2

.github/CODEOWNERS

@@ -0,0 +1,5 @@
+# CODEOWNERS: https://help.github.com/articles/about-codeowners/
+
+# Everything goes through Bucky. For now.
+*       @ebuchman
+

.github/ISSUE_TEMPLATE

@@ -0,0 +1,41 @@
+<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.-->
+
+**Is this a BUG REPORT or FEATURE REQUEST?** (choose one):
+
+<!--
+If this is a BUG REPORT, please:
+  - Fill in as much of the template below as you can.
+
+If this is a FEATURE REQUEST, please:
+  - Describe *in detail* the feature/behavior/change you'd like to see.
+
+In both cases, be ready for followup questions, and please respond in a timely
+manner. We might ask you to provide additional logs and data (tendermint & app)
+in a case of bug.
+-->
+
+**Tendermint version** (use `tendermint version` or `git rev-parse --verify HEAD` if installed from source):
+
+
+**ABCI app** (name for built-in, URL for self-written if it's publicly available):
+
+
+**Merkleeyes version** (use `git rev-parse --verify HEAD`, skip if you don't use it):
+
+
+**Environment**:
+- **OS** (e.g. from /etc/os-release):
+- **Install tools**:
+- **Others**:
+
+
+**What happened**:
+
+
+**What you expected to happen**:
+
+
+**How to reproduce it** (as minimally and precisely as possible):
+
+
+**Anything else do we need to know**:

.gitignore

@@ -10,3 +10,13 @@ rpc/test/.tendermint
 remote_dump
 .revision
 vendor
+.vagrant
+test/p2p/data/
+test/logs
+.glide
+coverage.txt
+docs/_build
+docs/tools
+
+scripts/wal2json/wal2json
+scripts/cutWALUntil/cutWALUntil

CHANGELOG.md

@@ -0,0 +1,491 @@
+# Changelog
+
+## Roadmap
+
+BREAKING CHANGES:
+- Upgrade the header to support better proofs on validtors, results, evidence, and possibly more
+- Better support for injecting randomness
+- Pass evidence/voteInfo through ABCI
+- Upgrade consensus for more real-time use of evidence
+
+FEATURES:
+- Peer reputation management
+- Use the chain as its own CA for nodes and validators
+- Tooling to run multiple blockchains/apps, possibly in a single process
+- State syncing (without transaction replay)
+- Add authentication and rate-limitting to the RPC
+
+IMPROVEMENTS:
+- Improve subtleties around mempool caching and logic
+- Consensus optimizations:
+	- cache block parts for faster agreement after round changes
+	- propagate block parts rarest first
+- Better testing of the consensus state machine (ie. use a DSL)
+- Auto compiled serialization/deserialization code instead of go-wire reflection
+
+BUG FIXES:
+- Graceful handling/recovery for apps that have non-determinism or fail to halt
+- Graceful handling/recovery for violations of safety, or liveness
+
+## 0.12.0 (October 27, 2017)
+
+BREAKING CHANGES:
+ - rpc/client: websocket ResultsCh and ErrorsCh unified in ResponsesCh.
+ - rpc/client: ABCIQuery no longer takes `prove`
+ - state: remove GenesisDoc from state.
+ - consensus: new binary WAL format provides efficiency and uses checksums to detect corruption
+    - use scripts/wal2json to convert to json for debugging
+
+FEATURES:
+ - new `certifiers` pkg contains the tendermint light-client library (name subject to change)!
+ - rpc: `/genesis` includes the `app_options` .
+ - rpc: `/abci_query` takes an additional `height` parameter to support historical queries.
+ - rpc/client: new ABCIQueryWithOptions supports options like `trusted` (set false to get a proof) and `height` to query a historical height.
+
+IMPROVEMENTS:
+ - rpc: `/genesis` result includes `app_options`
+ - rpc/lib/client: add jitter to reconnects.
+ - rpc/lib/types: `RPCError` satisfies the `error` interface.
+
+BUG FIXES:
+ - rpc/client: fix ws deadlock after stopping
+ - blockchain: fix panic on AddBlock when peer is nil
+ - mempool: fix sending on TxsAvailable when a tx has been invalidated
+ - consensus: dont run WAL catchup if we fast synced
+
+## 0.11.1 (October 10, 2017)
+
+IMPROVEMENTS:
+ - blockchain/reactor: respondWithNoResponseMessage for missing height
+
+BUG FIXES:
+ - rpc: fixed client WebSocket timeout
+ - rpc: client now resubscribes on reconnection
+ - rpc: fix panics on missing params
+ - rpc: fix `/dump_consensus_state` to have normal json output (NOTE: technically breaking, but worth a bug fix label)
+ - types: fixed out of range error in VoteSet.addVote
+ - consensus: fix wal autofile via https://github.com/tendermint/tmlibs/blob/master/CHANGELOG.md#032-october-2-2017
+
+## 0.11.0 (September 22, 2017)
+
+BREAKING:
+ - genesis file: validator `amount` is now `power`
+ - abci: Info, BeginBlock, InitChain all take structs
+ - rpc: various changes to match JSONRPC spec (http://www.jsonrpc.org/specification), including breaking ones:
+    - requests that previously returned HTTP code 4XX now return 200 with an error code in the JSONRPC.
+    - `rpctypes.RPCResponse` uses new `RPCError` type instead of `string`.
+
+ - cmd: if there is no genesis, exit immediately instead of waiting around for one to show.
+ - types: `Signer.Sign` returns an error.
+ - state: every validator set change is persisted to disk, which required some changes to the `State` structure.
+ - p2p: new `p2p.Peer` interface used for all reactor methods (instead of `*p2p.Peer` struct).
+
+FEATURES:
+ - rpc: `/validators?height=X` allows querying of validators at previous heights.
+ - rpc: Leaving the `height` param empty for `/block`, `/validators`, and `/commit` will return the value for the latest height.
+
+IMPROVEMENTS:
+ - docs: Moved all docs from the website and tools repo in, converted to `.rst`, and cleaned up for presentation on `tendermint.readthedocs.io`
+
+BUG FIXES:
+ - fix WAL openning issue on Windows
+
+## 0.10.4 (September 5, 2017)
+
+IMPROVEMENTS:
+- docs: Added Slate docs to each rpc function (see rpc/core)
+- docs: Ported all website docs to Read The Docs
+- config: expose some p2p params to tweak performance: RecvRate, SendRate, and MaxMsgPacketPayloadSize
+- rpc: Upgrade the websocket client and server, including improved auto reconnect, and proper ping/pong
+
+BUG FIXES:
+- consensus: fix panic on getVoteBitArray
+- consensus: hang instead of panicking on byzantine consensus failures
+- cmd: dont load config for version command
+
+## 0.10.3 (August 10, 2017)
+
+FEATURES:
+- control over empty block production:
+  - new flag, `--consensus.create_empty_blocks`; when set to false, blocks are only created when there are txs or when the AppHash changes.
+  - new config option, `consensus.create_empty_blocks_interval`; an empty block is created after this many seconds.
+  - in normal operation, `create_empty_blocks = true` and `create_empty_blocks_interval = 0`, so blocks are being created all the time (as in all previous versions of tendermint). The number of empty blocks can be reduced by increasing `create_empty_blocks_interval` or by setting `create_empty_blocks = false`.
+  - new `TxsAvailable()` method added to Mempool that returns a channel which fires when txs are available.
+  - new heartbeat message added to consensus reactor to notify peers that a node is waiting for txs before entering propose step.
+- rpc: Add `syncing` field to response returned by `/status`. Is `true` while in fast-sync mode.
+
+IMPROVEMENTS:
+- various improvements to documentation and code comments
+
+BUG FIXES:
+- mempool: pass height into constructor so it doesn't always start at 0
+
+## 0.10.2 (July 10, 2017)
+
+FEATURES:
+- Enable lower latency block commits by adding consensus reactor sleep durations and p2p flush throttle timeout to the config
+
+IMPROVEMENTS:
+- More detailed logging in the consensus reactor and state machine
+- More in-code documentation for many exposed functions, especially in consensus/reactor.go and p2p/switch.go
+- Improved readability for some function definitions and code blocks with long lines
+
+## 0.10.1 (June 28, 2017)
+
+FEATURES:
+- Use `--trace` to get stack traces for logged errors
+- types: GenesisDoc.ValidatorHash returns the hash of the genesis validator set
+- types: GenesisDocFromFile parses a GenesiDoc from a JSON file
+
+IMPROVEMENTS:
+- Add a Code of Conduct
+- Variety of improvements as suggested by `megacheck` tool
+- rpc: deduplicate tests between rpc/client and rpc/tests
+- rpc: addresses without a protocol prefix default to `tcp://`. `http://` is also accepted as an alias for `tcp://`
+- cmd: commands are more easily reuseable from other tools
+- DOCKER: automate build/push
+
+BUG FIXES:
+- Fix log statements using keys with spaces (logger does not currently support spaces)
+- rpc: set logger on websocket connection
+- rpc: fix ws connection stability by setting write deadline on pings
+
+## 0.10.0 (June 2, 2017)
+
+Includes major updates to configuration, logging, and json serialization.
+Also includes the Grand Repo-Merge of 2017.
+
+BREAKING CHANGES:
+
+- Config and Flags:
+  - The `config` map is replaced with a [`Config` struct](https://github.com/tendermint/tendermint/blob/master/config/config.go#L11),
+containing substructs: `BaseConfig`, `P2PConfig`, `MempoolConfig`, `ConsensusConfig`, `RPCConfig`
+  - This affects the following flags:
+    - `--seeds` is now `--p2p.seeds`
+    - `--node_laddr` is now `--p2p.laddr`
+    - `--pex` is now `--p2p.pex`
+    - `--skip_upnp` is now `--p2p.skip_upnp`
+    - `--rpc_laddr` is now `--rpc.laddr`
+    - `--grpc_laddr` is now `--rpc.grpc_laddr`
+  - Any configuration option now within a substract must come under that heading in the `config.toml`, for instance:
+    ```
+    [p2p]
+    laddr="tcp://1.2.3.4:46656"
+
+    [consensus]
+    timeout_propose=1000
+    ```
+  - Use viper and `DefaultConfig() / TestConfig()` functions to handle defaults, and remove `config/tendermint` and `config/tendermint_test`
+  - Change some function and method signatures to
+  - Change some [function and method signatures](https://gist.github.com/ebuchman/640d5fc6c2605f73497992fe107ebe0b) accomodate new config
+
+- Logger
+  - Replace static `log15` logger with a simple interface, and provide a new implementation using `go-kit`.
+See our new [logging library](https://github.com/tendermint/tmlibs/log) and [blog post](https://tendermint.com/blog/abstracting-the-logger-interface-in-go) for more details
+  - Levels `warn` and `notice` are removed (you may need to change them in your `config.toml`!)
+  - Change some [function and method signatures](https://gist.github.com/ebuchman/640d5fc6c2605f73497992fe107ebe0b) to accept a logger
+
+- JSON serialization:
+  - Replace `[TypeByte, Xxx]` with `{"type": "some-type", "data": Xxx}` in RPC and all `.json` files by using `go-wire/data`. For instance, a public key is now:
+    ```
+    "pub_key": {
+      "type": "ed25519",
+      "data": "83DDF8775937A4A12A2704269E2729FCFCD491B933C4B0A7FFE37FE41D7760D0"
+    }
+    ```
+  - Remove type information about RPC responses, so `[TypeByte, {"jsonrpc": "2.0", ... }]` is now just `{"jsonrpc": "2.0", ... }`
+  - Change `[]byte` to `data.Bytes` in all serialized types (for hex encoding)
+  - Lowercase the JSON tags in `ValidatorSet` fields
+  - Introduce `EventDataInner` for serializing events
+
+- Other:
+  - Send InitChain message in handshake if `appBlockHeight == 0`
+  - Do not include the `Accum` field when computing the validator hash. This makes the ValidatorSetHash unique for a given validator set, rather than changing with every block (as the Accum changes)
+  - Unsafe RPC calls are not enabled by default. This includes `/dial_seeds`, and all calls prefixed with `unsafe`. Use the `--rpc.unsafe` flag to enable.
+
+
+FEATURES:
+
+- Per-module log levels. For instance, the new default is `state:info,*:error`, which means the `state` package logs at `info` level, and everything else logs at `error` level
+- Log if a node is validator or not in every consensus round
+- Use ldflags to set git hash as part of the version
+- Ignore `address` and `pub_key` fields in `priv_validator.json` and overwrite them with the values derrived from the `priv_key`
+
+IMPROVEMENTS:
+
+- Merge `tendermint/go-p2p -> tendermint/tendermint/p2p` and `tendermint/go-rpc -> tendermint/tendermint/rpc/lib`
+- Update paths for grand repo merge:
+  - `go-common -> tmlibs/common`
+  - `go-data -> go-wire/data`
+  - All other `go-` libs, except `go-crypto` and `go-wire`, are merged under `tmlibs`
+- No global loggers (loggers are passed into constructors, or preferably set with a SetLogger method)
+- Return HTTP status codes with errors for RPC responses
+- Limit `/blockchain_info` call to return a maximum of 20 blocks
+- Use `.Wrap()` and `.Unwrap()` instead of eg. `PubKeyS` for `go-crypto` types
+- RPC JSON responses use pretty printing (via `json.MarshalIndent`)
+- Color code different instances of the consensus for tests
+- Isolate viper to `cmd/tendermint/commands` and do not read config from file for tests
+
+
+## 0.9.2 (April 26, 2017)
+
+BUG FIXES:
+
+- Fix bug in `ResetPrivValidator` where we were using the global config and log (causing external consumers, eg. basecoin, to fail).
+
+## 0.9.1 (April 21, 2017)
+
+FEATURES:
+
+- Transaction indexing - txs are indexed by their hash using a simple key-value store; easily extended to more advanced indexers
+- New `/tx?hash=X` endpoint to query for transactions and their DeliverTx result by hash. Optionally returns a proof of the tx's inclusion in the block
+- `tendermint testnet` command initializes files for a testnet
+
+IMPROVEMENTS:
+
+- CLI now uses Cobra framework
+- TMROOT is now TMHOME (TMROOT will stop working in 0.10.0)
+- `/broadcast_tx_XXX` also returns the Hash (can be used to query for the tx)
+- `/broadcast_tx_commit` also returns the height the block was committed in
+- ABCIResponses struct persisted to disk before calling Commit; makes handshake replay much cleaner
+- WAL uses #ENDHEIGHT instead of #HEIGHT (#HEIGHT will stop working in 0.10.0)
+- Peers included via `--seeds`, under `seeds` in the config, or in `/dial_seeds` are now persistent, and will be reconnected to if the connection breaks
+
+BUG FIXES:
+
+- Fix bug in fast-sync where we stop syncing after a peer is removed, even if they're re-added later
+- Fix handshake replay to handle validator set changes and results of DeliverTx when we crash after app.Commit but before state.Save()
+
+## 0.9.0 (March 6, 2017)
+
+BREAKING CHANGES:
+
+- Update ABCI to v0.4.0, where Query is now `Query(RequestQuery) ResponseQuery`, enabling precise proofs at particular heights:
+
+```
+message RequestQuery{
+	bytes data = 1;
+	string path = 2;
+	uint64 height = 3;
+	bool prove = 4;
+}
+
+message ResponseQuery{
+	CodeType          code        = 1;
+	int64             index       = 2;
+	bytes             key         = 3;
+	bytes             value       = 4;
+	bytes             proof       = 5;
+	uint64            height      = 6;
+	string            log         = 7;
+}
+```
+
+
+- `BlockMeta` data type unifies its Hash and PartSetHash under a `BlockID`:
+
+```
+type BlockMeta struct {
+	BlockID BlockID `json:"block_id"` // the block hash and partsethash
+	Header  *Header `json:"header"`   // The block's Header
+}
+```
+
+- `ValidatorSet.Proposer` is exposed as a field and persisted with the `State`. Use `GetProposer()` to initialize or update after validator-set changes.
+
+- `tendermint gen_validator` command output is now pure JSON
+
+FEATURES:
+
+- New RPC endpoint `/commit?height=X` returns header and commit for block at height `X`
+- Client API for each endpoint, including mocks for testing
+
+IMPROVEMENTS:
+
+- `Node` is now a `BaseService`
+- Simplified starting Tendermint in-process from another application
+- Better organized Makefile
+- Scripts for auto-building binaries across platforms
+- Docker image improved, slimmed down (using Alpine), and changed from tendermint/tmbase to tendermint/tendermint
+- New repo files: `CONTRIBUTING.md`, Github `ISSUE_TEMPLATE`, `CHANGELOG.md`
+- Improvements on CircleCI for managing build/test artifacts
+- Handshake replay is doen through the consensus package, possibly using a mockApp
+- Graceful shutdown of RPC listeners
+- Tests for the PEX reactor and DialSeeds
+
+BUG FIXES:
+
+- Check peer.Send for failure before updating PeerState in consensus
+- Fix panic in `/dial_seeds` with invalid addresses
+- Fix proposer selection logic in ValidatorSet by taking the address into account in the `accumComparable`
+- Fix inconcistencies with `ValidatorSet.Proposer` across restarts by persisting it in the `State`
+
+
+## 0.8.0 (January 13, 2017)
+
+BREAKING CHANGES:
+
+- New data type `BlockID` to represent blocks:
+
+```
+type BlockID struct {
+	Hash        []byte        `json:"hash"`
+	PartsHeader PartSetHeader `json:"parts"`
+}
+```
+
+- `Vote` data type now includes validator address and index:
+
+```
+type Vote struct {
+	ValidatorAddress []byte           `json:"validator_address"`
+	ValidatorIndex   int              `json:"validator_index"`
+	Height           int              `json:"height"`
+	Round            int              `json:"round"`
+	Type             byte             `json:"type"`
+	BlockID          BlockID          `json:"block_id"` // zero if vote is nil.
+	Signature        crypto.Signature `json:"signature"`
+}
+```
+
+- Update TMSP to v0.3.0, where it is now called ABCI and AppendTx is DeliverTx
+- Hex strings in the RPC are now "0x" prefixed
+
+
+FEATURES:
+
+- New message type on the ConsensusReactor, `Maj23Msg`, for peers to alert others they've seen a Maj23,
+in order to track and handle conflicting votes intelligently to prevent Byzantine faults from causing halts:
+
+```
+type VoteSetMaj23Message struct {
+	Height  int
+	Round   int
+	Type    byte
+	BlockID types.BlockID
+}
+```
+
+- Configurable block part set size
+- Validator set changes
+- Optionally skip TimeoutCommit if we have all the votes
+- Handshake between Tendermint and App on startup to sync latest state and ensure consistent recovery from crashes
+- GRPC server for BroadcastTx endpoint
+
+IMPROVEMENTS:
+
+- Less verbose logging
+- Better test coverage (37% -> 49%)
+- Canonical SignBytes for signable types
+- Write-Ahead Log for Mempool and Consensus via tmlibs/autofile
+- Better in-process testing for the consensus reactor and byzantine faults
+- Better crash/restart testing for individual nodes at preset failure points, and of networks at arbitrary points
+- Better abstraction over timeout mechanics
+
+BUG FIXES:
+
+- Fix memory leak in mempool peer
+- Fix panic on POLRound=-1
+- Actually set the CommitTime
+- Actually send BeginBlock message
+- Fix a liveness issues caused by Byzantine proposals/votes. Uses the new `Maj23Msg`.
+
+
+## 0.7.4 (December 14, 2016)
+
+FEATURES:
+
+- Enable the Peer Exchange reactor with the `--pex` flag for more resilient gossip network (feature still in development, beware dragons)
+
+IMPROVEMENTS:
+
+- Remove restrictions on RPC endpoint `/dial_seeds` to enable manual network configuration
+
+## 0.7.3 (October 20, 2016)
+
+IMPROVEMENTS:
+
+- Type safe FireEvent
+- More WAL/replay tests
+- Cleanup some docs
+
+BUG FIXES:
+
+- Fix deadlock in mempool for synchronous apps
+- Replay handles non-empty blocks
+- Fix race condition in HeightVoteSet
+
+## 0.7.2 (September 11, 2016)
+
+BUG FIXES:
+
+- Set mustConnect=false so tendermint will retry connecting to the app
+
+## 0.7.1 (September 10, 2016)
+
+FEATURES:
+
+- New TMSP connection for Query/Info
+- New RPC endpoints:
+	- `tmsp_query`
+	- `tmsp_info`
+- Allow application to filter peers through Query (off by default)
+
+IMPROVEMENTS:
+
+- TMSP connection type enforced at compile time
+- All listen/client urls use a "tcp://" or "unix://" prefix
+
+BUG FIXES:
+
+- Save LastSignature/LastSignBytes to `priv_validator.json` for recovery
+- Fix event unsubscribe
+- Fix fastsync/blockchain reactor
+
+## 0.7.0 (August 7, 2016)
+
+BREAKING CHANGES:
+
+- Strict SemVer starting now!
+- Update to ABCI v0.2.0
+- Validation types now called Commit
+- NewBlock event only returns the block header
+
+
+FEATURES:
+
+- TMSP and RPC support TCP and UNIX sockets
+- Addition config options including block size and consensus parameters
+- New WAL mode `cswal_light`; logs only the validator's own votes
+- New RPC endpoints:
+	- for starting/stopping profilers, and for updating config
+	- `/broadcast_tx_commit`, returns when tx is included in a block, else an error
+	- `/unsafe_flush_mempool`, empties the mempool
+
+
+IMPROVEMENTS:
+
+- Various optimizations
+- Remove bad or invalidated transactions from the mempool cache (allows later duplicates)
+- More elaborate testing using CircleCI including benchmarking throughput on 4 digitalocean droplets
+
+BUG FIXES:
+
+- Various fixes to WAL and replay logic
+- Various race conditions
+
+## PreHistory
+
+Strict versioning only began with the release of v0.7.0, in late summer 2016.
+The project itself began in early summer 2014 and was workable decentralized cryptocurrency software by the end of that year.
+Through the course of 2015, in collaboration with Eris Industries (now Monax Indsutries),
+many additional features were integrated, including an implementation from scratch of the Ethereum Virtual Machine.
+That implementation now forms the heart of [Burrow](https://github.com/hyperledger/burrow).
+In the later half of 2015, the consensus algorithm was upgraded with a more asynchronous design and a more deterministic and robust implementation.
+
+By late 2015, frustration with the difficulty of forking a large monolithic stack to create alternative cryptocurrency designs led to the
+invention of the Application Blockchain Interface (ABCI), then called the Tendermint Socket Protocol (TMSP).
+The Ethereum Virtual Machine and various other transaction features were removed, and Tendermint was whittled down to a core consensus engine
+driving an application running in another process.
+The ABCI interface and implementation were iterated on and improved over the course of 2016,
+until versioned history kicked in with v0.7.0.

CODE_OF_CONDUCT.md

@@ -0,0 +1,56 @@
+# The Tendermint Code of Conduct
+This code of conduct applies to all projects run by the Tendermint/COSMOS team and hence to tendermint.
+
+
+----
+
+
+# Conduct
+## Contact: adrian@tendermint.com
+
+* We are committed to providing a friendly, safe and welcoming environment for all, regardless of level of experience, gender, gender identity and expression, sexual orientation, disability, personal appearance, body size, race, ethnicity, age, religion, nationality, or other similar characteristic.
+
+* On Slack, please avoid using overtly sexual nicknames or other nicknames that might detract from a friendly, safe and welcoming environment for all.
+
+* Please be kind and courteous. There’s no need to be mean or rude.
+
+* Respect that people have differences of opinion and that every design or implementation choice carries a trade-off and numerous costs. There is seldom a right answer.
+
+* Please keep unstructured critique to a minimum. If you have solid ideas you want to experiment with, make a fork and see how it works.
+
+* We will exclude you from interaction if you insult, demean or harass anyone. That is not welcome behaviour. We interpret the term “harassment” as including the definition in the [Citizen Code of Conduct](http://citizencodeofconduct.org/); if you have any lack of clarity about what might be included in that concept, please read their definition. In particular, we don’t tolerate behavior that excludes people in socially marginalized groups.
+
+* Private harassment is also unacceptable. No matter who you are, if you feel you have been or are being harassed or made uncomfortable by a community member, please contact one of the channel admins or the person mentioned above immediately. Whether you’re a regular contributor or a newcomer, we care about making this community a safe place for you and we’ve got your back.
+
+* Likewise any spamming, trolling, flaming, baiting or other attention-stealing behaviour is not welcome.
+
+
+----
+
+
+# Moderation
+These are the policies for upholding our community’s standards of conduct. If you feel that a thread needs moderation, please contact the above mentioned person.
+
+1. Remarks that violate the Tendermint/COSMOS standards of conduct, including hateful, hurtful, oppressive, or exclusionary remarks, are not allowed. (Cursing is allowed, but never targeting another user, and never in a hateful manner.)
+
+2. Remarks that moderators find inappropriate, whether listed in the code of conduct or not, are also not allowed.
+
+3. Moderators will first respond to such remarks with a warning.
+
+4. If the warning is unheeded, the user will be “kicked,” i.e., kicked out of the communication channel to cool off.
+
+5. If the user comes back and continues to make trouble, they will be banned, i.e., indefinitely excluded.
+
+6. Moderators may choose at their discretion to un-ban the user if it was a first offense and they offer the offended party a genuine apology.
+
+7. If a moderator bans someone and you think it was unjustified, please take it up with that moderator, or with a different moderator, in private. Complaints about bans in-channel are not allowed.
+
+8. Moderators are held to a higher standard than other community members. If a moderator creates an inappropriate situation, they should expect less leeway than others.
+
+In the Tendermint/COSMOS community we strive to go the extra step to look out for each other. Don’t just aim to be technically unimpeachable, try to be your best self. In particular, avoid flirting with offensive or sensitive issues, particularly if they’re off-topic; this all too often leads to unnecessary fights, hurt feelings, and damaged trust; worse, it can drive people away from the community entirely.
+
+And if someone takes issue with something you said or did, resist the urge to be defensive. Just stop doing what it was they complained about and apologize. Even if you feel you were misinterpreted or unfairly accused, chances are good there was something you could’ve communicated better — remember that it’s your responsibility to make your fellow Cosmonauts comfortable. Everyone wants to get along and we are all here first and foremost because we want to talk about cool technology. You will find that people will be eager to assume good intent and forgive as long as you earn their trust.
+
+The enforcement policies listed above apply to all official Tendermint/COSMOS venues.For other projects adopting the Tendermint/COSMOS Code of Conduct, please contact the maintainers of those projects for enforcement. If you wish to use this code of conduct for your own project, consider explicitly mentioning your moderation policy or making a copy with your own moderation policy so as to avoid confusion.
+
+*Adapted from the [Node.js Policy on Trolling](http://blog.izs.me/post/30036893703/policy-on-trolling), the [Contributor Covenant v1.3.0](http://contributor-covenant.org/version/1/3/0/) and the [Rust Code of Conduct](https://www.rust-lang.org/en-US/conduct.html).

CONTRIBUTING.md

@@ -0,0 +1,77 @@
+# Contributing
+
+Thank you for considering making contributions to Tendermint and related repositories! Start by taking a look at the [coding repo](https://github.com/tendermint/coding) for overall information on repository workflow and standards.
+
+Please follow standard github best practices: fork the repo, branch from the tip of develop, make some commits, and submit a pull request to develop. See the [open issues](https://github.com/tendermint/tendermint/issues) for things we need help with!
+
+Please make sure to use `gofmt` before every commit - the easiest way to do this is have your editor run it for you upon saving a file.
+
+## Forking
+
+Please note that Go requires code to live under absolute paths, which complicates forking. 
+While my fork lives at `https://github.com/ebuchman/tendermint`, 
+the code should never exist at  `$GOPATH/src/github.com/ebuchman/tendermint`. 
+Instead, we use `git remote` to add the fork as a new remote for the original repo,
+`$GOPATH/src/github.com/tendermint/tendermint `, and do all the work there.
+
+For instance, to create a fork and work on a branch of it, I would:
+
+  * Create the fork on github, using the fork button.
+  * Go to the original repo checked out locally (ie. `$GOPATH/src/github.com/tendermint/tendermint`)
+  * `git remote rename origin upstream`
+  * `git remote add origin git@github.com:ebuchman/basecoin.git`
+
+Now `origin` refers to my fork and `upstream` refers to the tendermint version.
+So I can `git push -u origin master` to update my fork, and make pull requests to tendermint from there.
+Of course, replace `ebuchman` with your git handle.
+
+To pull in updates from the origin repo, run
+
+    * `git fetch upstream`
+    * `git rebase upstream/master` (or whatever branch you want)
+
+Please don't make Pull Requests to `master`.
+
+## Dependencies
+
+We use [glide](https://github.com/masterminds/glide) to manage dependencies.
+That said, the master branch of every Tendermint repository should just build with `go get`, which means they should be kept up-to-date with their dependencies so we can get away with telling people they can just `go get` our software.
+Since some dependencies are not under our control, a third party may break our build, in which case we can fall back on `glide install`. Even for dependencies under our control, glide helps us keeps multiple repos in sync as they evolve. Anything with an executable, such as apps, tools, and the core, should use glide.
+
+Run `bash scripts/glide/status.sh` to get a list of vendored dependencies that may not be up-to-date. 
+
+## Testing
+
+All repos should be hooked up to circle. 
+If they have `.go` files in the root directory, they will be automatically tested by circle using `go test -v -race ./...`. If not, they will need a `circle.yml`. Ideally, every repo has a `Makefile` that defines `make test` and includes its continuous integration status using a badge in the `README.md`.
+
+## Branching Model and Release
+
+User-facing repos should adhere to the branching model: http://nvie.com/posts/a-successful-git-branching-model/.
+That is, these repos should be well versioned, and any merge to master requires a version bump and tagged release.
+
+Libraries need not follow the model strictly, but would be wise to,
+especially `go-p2p` and `go-rpc`, as their versions are referenced in tendermint core.
+
+### Development Procedure:
+- the latest state of development is on `develop`
+- `develop` must never fail `make test`
+- no --force onto `develop` (except when reverting a broken commit, which should seldom happen)
+- create a development branch either on github.com/tendermint/tendermint, or your fork (using `git add origin`)
+- before submitting a pull request, begin `git rebase` on top of `develop`
+
+### Pull Merge Procedure:
+- ensure pull branch is rebased on develop
+- run `make test` to ensure that all tests pass
+- merge pull request
+- the `unstable` branch may be used to aggregate pull merges before testing once
+- push master may request that pull requests be rebased on top of `unstable`
+
+### Release Procedure:
+- start on `develop`
+- run integration tests (see `test_integrations` in Makefile)
+- prepare changelog/release issue
+- bump versions
+- push to release-vX.X.X to run the extended integration tests on the CI
+- merge to master
+- merge master back to develop

DOCKER/Dockerfile

@@ -1,54 +1,45 @@
-# Pull base image.
-FROM golang:1.6
+FROM alpine:3.6
 
-ENV USER tmuser
-ENV DATA_ROOT /data/tendermint
+# This is the release of tendermint to pull in.
+ENV TM_VERSION 0.11.0
+ENV TM_SHA256SUM 7e443bac4d42f12e7beaf9cee63b4a565dad8c58895291fdedde8057088b70c5
+
+# Tendermint will be looking for genesis file in /tendermint (unless you change
+# `genesis_file` in config.toml). You can put your config.toml and private
+# validator file into /tendermint.
+#
+# The /tendermint/data dir is used by tendermint to store state.
+ENV DATA_ROOT /tendermint
+ENV TMHOME $DATA_ROOT
 
 # Set user right away for determinism
-RUN groupadd -r $USER \
-  && useradd -r -s /bin/false -g $USER $USER
-
-# Create home directory for USER
-# Needed for nodejs/nom
-RUN mkdir -p /home/$USER \
-  && chown -R $USER:$USER /home/$USER
+RUN addgroup tmuser && \
+    adduser -S -G tmuser tmuser
 
 # Create directory for persistence and give our user ownership
-RUN mkdir -p $DATA_ROOT \
-  && chown -R $USER:$USER $DATA_ROOT
+RUN mkdir -p $DATA_ROOT && \
+    chown -R tmuser:tmuser $DATA_ROOT
 
-# Set the env variables to non-interactive
-ENV DEBIAN_FRONTEND noninteractive
-ENV DEBIAN_PRIORITY critical
-ENV DEBCONF_NOWARNINGS yes
-ENV TERM linux
-RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections
+# jq and curl used for extracting `pub_key` from private validator while
+# deploying tendermint with Kubernetes. It is nice to have bash so the users
+# could execute bash commands.
+RUN apk add --no-cache bash curl jq
 
-# Grab deps (git)
-RUN apt-get update && \
-  apt-get install -y --no-install-recommends \
-    git && \
-  rm -rf /var/lib/apt/lists/*
+RUN apk add --no-cache openssl && \
+    wget https://s3-us-west-2.amazonaws.com/tendermint/binaries/tendermint/v${TM_VERSION}/tendermint_${TM_VERSION}_linux_amd64.zip && \
+    echo "${TM_SHA256SUM}  tendermint_${TM_VERSION}_linux_amd64.zip" | sha256sum -c && \
+    unzip -d /bin tendermint_${TM_VERSION}_linux_amd64.zip && \
+    apk del openssl && \
+    rm -f tendermint_${TM_VERSION}_linux_amd64.zip
 
-# Grab deps (node)
-RUN curl -sL https://deb.nodesource.com/setup_5.x | bash -
-RUN apt-get update && \
-  apt-get install -y --no-install-recommends \
-    nodejs && \
-  rm -rf /var/lib/apt/lists/*
-
-# Copy run.sh
-COPY ./run.sh $DATA_ROOT/run.sh
-RUN chmod +x $DATA_ROOT/run.sh
-
-# Persist data, set user
-WORKDIR $DATA_ROOT
+# Expose the data directory as a volume since there's mutable state in there
 VOLUME $DATA_ROOT
-USER $USER
-ENV TMROOT $DATA_ROOT
 
+# p2p port
 EXPOSE 46656
+# rpc port
 EXPOSE 46657
 
-# Run tendermint
-CMD ["./run.sh"]
+ENTRYPOINT ["tendermint"]
+
+CMD ["node", "--moniker=`hostname`"]

DOCKER/Dockerfile.develop

@@ -0,0 +1,35 @@
+FROM alpine:3.6
+
+ENV DATA_ROOT /tendermint
+ENV TMHOME $DATA_ROOT
+
+RUN addgroup tmuser && \
+    adduser -S -G tmuser tmuser
+
+RUN mkdir -p $DATA_ROOT && \
+    chown -R tmuser:tmuser $DATA_ROOT
+
+RUN apk add --no-cache bash curl jq
+
+ENV GOPATH /go
+ENV PATH "$PATH:/go/bin"
+RUN mkdir -p /go/src/github.com/tendermint/tendermint && \
+    apk add --no-cache go build-base git && \
+    cd /go/src/github.com/tendermint/tendermint && \
+    git clone https://github.com/tendermint/tendermint . && \
+    git checkout develop && \
+    make get_vendor_deps && \
+    make install && \
+    glide cc && \
+    cd - && \
+    rm -rf /go/src/github.com/tendermint/tendermint && \
+    apk del go build-base git
+
+VOLUME $DATA_ROOT
+
+EXPOSE 46656
+EXPOSE 46657
+
+ENTRYPOINT ["tendermint"]
+
+CMD ["node", "--moniker=`hostname`", "--proxy_app=dummy"]

DOCKER/Makefile

@@ -0,0 +1,13 @@
+build:
+	@sh -c "'$(CURDIR)/build.sh'"
+
+push:
+	@sh -c "'$(CURDIR)/push.sh'"
+
+build_develop:
+	docker build -t "tendermint/tendermint:develop" -f Dockerfile.develop .
+
+push_develop:
+	docker push "tendermint/tendermint:develop"
+
+.PHONY: build build_develop push push_develop

DOCKER/README.md

@@ -1,13 +1,56 @@
-# Build and run a docker image and container
+# Supported tags and respective `Dockerfile` links
+
+- `0.11.0`, `latest` [(Dockerfile)](https://github.com/tendermint/tendermint/blob/9177cc1f64ca88a4a0243c5d1773d10fba67e201/DOCKER/Dockerfile)
+- `0.10.0` [(Dockerfile)](https://github.com/tendermint/tendermint/blob/e5342f4054ab784b2cd6150e14f01053d7c8deb2/DOCKER/Dockerfile)
+- `0.9.1`, `0.9`, [(Dockerfile)](https://github.com/tendermint/tendermint/blob/809e0e8c5933604ba8b2d096803ada7c5ec4dfd3/DOCKER/Dockerfile)
+- `0.9.0` [(Dockerfile)](https://github.com/tendermint/tendermint/blob/d474baeeea6c22b289e7402449572f7c89ee21da/DOCKER/Dockerfile)
+- `0.8.0`, `0.8` [(Dockerfile)](https://github.com/tendermint/tendermint/blob/bf64dd21fdb193e54d8addaaaa2ecf7ac371de8c/DOCKER/Dockerfile)
+- `develop` [(Dockerfile)](https://github.com/tendermint/tendermint/blob/master/DOCKER/Dockerfile.develop)
+
+`develop` tag points to the [develop](https://github.com/tendermint/tendermint/tree/develop) branch.
+
+# Quick reference
+
+* **Where to get help:**
+  [Chat on Rocket](https://cosmos.rocket.chat/)
+
+* **Where to file issues:**
+  https://github.com/tendermint/tendermint/issues
+
+* **Supported Docker versions:**
+  [the latest release](https://github.com/moby/moby/releases) (down to 1.6 on a best-effort basis)
+
+# Tendermint
+
+Tendermint Core is Byzantine Fault Tolerant (BFT) middleware that takes a state transition machine, written in any programming language, and securely replicates it on many machines.
+
+For more background, see the [introduction](https://tendermint.readthedocs.io/en/master/introduction.html).
+
+To get started developing applications, see the [application developers guide](https://tendermint.readthedocs.io/en/master/getting-started.html).
+
+# How to use this image
+
+## Start one instance of the Tendermint core with the `dummy` app
+
+A very simple example of a built-in app and Tendermint core in one container.
 
 ```
-# Build base Docker image
-# Make sure ./run.sh exists.
-docker build -t tendermint/tmbase -f Dockerfile .
-
-# Log into dockerhub
-docker login
-
-# Push latest build to dockerhub
-docker push tendermint/tmbase
+docker run -it --rm -v "/tmp:/tendermint" tendermint/tendermint init
+docker run -it --rm -v "/tmp:/tendermint" tendermint/tendermint node --proxy_app=dummy
 ```
+
+## mintnet-kubernetes
+
+If you want to see many containers talking to each other, consider using [mintnet-kubernetes](https://github.com/tendermint/tools/tree/master/mintnet-kubernetes), which is a tool for running Tendermint-based applications on a Kubernetes cluster.
+
+# License
+
+View [license information](https://raw.githubusercontent.com/tendermint/tendermint/master/LICENSE) for the software contained in this image.
+
+# User Feedback
+
+## Contributing
+
+You are invited to contribute new features, fixes, or updates, large or small; we are always thrilled to receive pull requests, and do our best to process them as fast as we can.
+
+Before you start to code, we recommend discussing your plans through a [GitHub](https://github.com/tendermint/tendermint/issues) issue, especially for more ambitious contributions. This gives other contributors a chance to point you in the right direction, give you feedback on your design, and help you find out if someone else is working on the same thing.

DOCKER/build.sh

@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+set -e
+
+# Get the tag from the version, or try to figure it out.
+if [ -z "$TAG" ]; then
+	TAG=$(awk -F\" '/Version =/ { print $2; exit }' < ../version/version.go)
+fi
+if [ -z "$TAG" ]; then
+		echo "Please specify a tag."
+		exit 1
+fi
+
+TAG_NO_PATCH=${TAG%.*}
+
+read -p "==> Build 3 docker images with the following tags (latest, $TAG, $TAG_NO_PATCH)? y/n" -n 1 -r
+echo
+if [[ $REPLY =~ ^[Yy]$ ]]
+then
+		docker build -t "tendermint/tendermint" -t "tendermint/tendermint:$TAG" -t "tendermint/tendermint:$TAG_NO_PATCH" .
+fi

DOCKER/push.sh

@@ -0,0 +1,22 @@
+#!/usr/bin/env bash
+set -e
+
+# Get the tag from the version, or try to figure it out.
+if [ -z "$TAG" ]; then
+	TAG=$(awk -F\" '/Version =/ { print $2; exit }' < ../version/version.go)
+fi
+if [ -z "$TAG" ]; then
+		echo "Please specify a tag."
+		exit 1
+fi
+
+TAG_NO_PATCH=${TAG%.*}
+
+read -p "==> Push 3 docker images with the following tags (latest, $TAG, $TAG_NO_PATCH)? y/n" -n 1 -r
+echo
+if [[ $REPLY =~ ^[Yy]$ ]]
+then
+	docker push "tendermint/tendermint:latest"
+	docker push "tendermint/tendermint:$TAG"
+	docker push "tendermint/tendermint:$TAG_NO_PATCH"
+fi

DOCKER/run.sh

@@ -1,10 +0,0 @@
-#! /bin/bash
-
-mkdir -p $GOPATH/src/$TMREPO
-cd $GOPATH/src/$TMREPO
-git clone https://$TMREPO.git .
-git fetch
-git reset --hard $TMHEAD
-go get -d $TMREPO/cmd/tendermint
-make
-tendermint node --seeds="$TMSEEDS" --moniker="$TMNAME"

INSTALL.md

@@ -0,0 +1 @@
+The installation guide has moved to the [docs directory](docs/guides/install-from-source.md) in order to easily be rendered by the website. Please update your links accordingly.

INSTALL/FORK_TESTNET.md

@@ -1,21 +0,0 @@
-1. Fork github.com/tendermint/tendermint.
-2. Run "make", it should install the daemon, which we named "tendermint".
-3. Run "tendermint gen_account".  Save the address, pub_key bytes, and priv_key bytes.
-   This is your developer key for controlling the cloud nodes.
-4. Also run "tendermint gen_validator" 5 times, once for each cloud node.  Save the output.
-5. Create a directory ~/.debora/ and copy cmd/debora/default.cfg into ~/.debora/default.cfg
-   Copy the priv_key bytes from step 4 into ~/.debora/default.cfg where it says so.
-   Change the list of hosts in ~/.debora/default.cfg with your own set of 5 cloud nodes.
-6. Replace cmd/barak/seed's pubkey with the pub_key bytes from step 3.
-7. Update config/tendermint/config.go's genesis with validator pubkeys from step 4.
-   Give each of your nodes the same amount of voting power.
-   Set up the accounts however you want.
-8. On each cloud node, follow the instructions here: https://github.com/tendermint/tendermint/tree/master/INSTALL
-   Create tmuser, install go, and also install 'barak'.
-   Then, run `barak -config="cmd/barak/seed"`.
-   You don't need to start the node at this time.
-9. Now you can run "debora list" on your development machine and post commands to each cloud node.
-10. Run scripts/unsafe_upgrade_barak.sh to test that barak is running.
-    The old barak you started on step 8 should now have quit.
-    A new instance of barak should be running.  Check with `ps -ef | grep "barak"`
-11. Run scripts/unsafe_restart_net.sh start your new testnet.

INSTALL/README.md

@@ -1,30 +0,0 @@
-NOTE: Only Ubuntu 14.04 64bit is supported at this time.
-
-### Server setup / create `tmuser`
-
-Secure the server, install dependencies, and create a new user `tmuser`
-
-    curl -L https://raw.githubusercontent.com/tendermint/tendermint/master/INSTALL/install_env.sh > install_env.sh
-    source install_env.sh
-    cd /home/tmuser
-
-### Install Go as `tmuser`
-
-Don't use `apt-get install golang`, it's still on an old version.
-
-    curl -L https://raw.githubusercontent.com/tendermint/tendermint/master/INSTALL/install_golang.sh > install_golang.sh
-    source install_golang.sh
-
-### Run Barak
-
-WARNING: THIS STEP WILL GIVE CONTROL OF THE CURRENT USER TO THE DEV TEAM.
-
-    go get -u github.com/tendermint/tendermint/cmd/barak
-    nohup barak -config="$GOPATH/src/github.com/tendermint/tendermint/cmd/barak/seed" &
-
-### Install/Update MintDB
-
-    go get -u github.com/tendermint/tendermint/cmd/tendermint
-    mkdir -p ~/.tendermint
-    cp $GOPATH/src/github.com/tendermint/tendermint/config/tendermint/genesis.json ~/.tendermint/
-    tendermint node --seeds="goldenalchemist.chaintest.net:46656"

INSTALL/install_env.sh

@@ -1,63 +0,0 @@
-#!/bin/bash
-# Run this as root user
-# This part is for hardening the server and setting up a user account
-
-if [ `whoami` != "root" ];
-then
-  echo "You must run this script as root"
-  exit 1
-fi
-
-USER="tmuser"
-OPEN_PORTS=(46656 46657 46658 46659 46660 46661 46662 46663 46664 46665 46666 46667 46668 46669 46670 46671)
-SSH_PORT=22
-WHITELIST=()
-
-# update and upgrade
-apt-get update -y
-apt-get upgrade -y
-
-# fail2ban for monitoring logins
-apt-get install -y fail2ban
-
-# set up the network time daemon
-apt-get install -y ntp
-
-# install dependencies
-apt-get install -y make screen gcc git mercurial libc6-dev pkg-config libgmp-dev
-
-# set up firewall
-echo "ENABLE FIREWALL ..."
-set -x
-# white list ssh access 
-for ip in "${WHITELIST[@]}"; do
-	ufw allow from $ip to any port $SSH_PORT
-done
-if [ ${#WHITELIST[@]} -eq 0 ]; then
-	ufw allow $SSH_PORT
-fi
-# open ports
-for port in "${OPEN_PORTS[@]}"; do
-	ufw allow $port
-done
-# apply
-ufw --force enable
-set +x
-# set up firewall END
-
-# watch the logs and have them emailed to me
-# apt-get install -y logwatch
-# echo "/usr/sbin/logwatch --output mail --mailto $ADMIN_EMAIL --detail high" >> /etc/cron.daily/00logwatch
-
-# set up user account
-echo "CREATE USER $USER ..."
-useradd $USER -d /home/$USER
-# This user should not have root access.
-# usermod -aG sudo $USER
-mkdir /home/$USER
-cp /etc/skel/.bashrc .
-cp /etc/skel/.profile .
-chown -R $USER:$USER /home/$USER
-
-echo "Done setting env.  Switching to $USER..."
-su $USER

INSTALL/install_golang.sh

@@ -1,29 +0,0 @@
-#!/bin/bash
-# Run this as tmuser user
-# This part is for installing go
-
-if [ `whoami` == "root" ];
-then
-  echo "You should not run this script as root"
-  exit 1
-fi
-
-USER=`whoami`
-PWD=`pwd`
-
-# get dependencies
-# sudo apt-get install -y make screen gcc git mercurial libc6-dev pkg-config libgmp-dev
-
-# install golang
-cd /home/$USER
-mkdir gocode
-wget https://storage.googleapis.com/golang/go1.4.2.src.tar.gz
-tar -xzvf go*.tar.gz
-cd go/src
-./make.bash
-mkdir -p /home/$USER/go/src
-echo 'export GOROOT=/home/$USER/go' >> /home/$USER/.bashrc
-echo 'export GOPATH=/home/$USER/gocode' >> /home/$USER/.bashrc
-echo 'export PATH=$PATH:$GOROOT/bin:$GOPATH/bin' >> /home/$USER/.bashrc
-source /home/$USER/.bashrc
-cd $PWD

Makefile

@@ -1,58 +1,86 @@
-.PHONY: get_deps build all list_deps install
+GOTOOLS = \
+					github.com/mitchellh/gox \
+					github.com/tcnksm/ghr \
+					github.com/Masterminds/glide \
 
-all: get_deps install test
+PACKAGES=$(shell go list ./... | grep -v '/vendor/')
+BUILD_TAGS?=tendermint
+TMHOME = $${TMHOME:-$$HOME/.tendermint}
 
-TMROOT = $${TMROOT:-$$HOME/.tendermint}
-define NEWLINE
+all: install test
 
+install: get_vendor_deps
+	@go install --ldflags '-extldflags "-static"' \
+		--ldflags "-X github.com/tendermint/tendermint/version.GitCommit=`git rev-parse HEAD`" ./cmd/tendermint
 
-endef
-NOVENDOR = go list github.com/tendermint/tendermint/... | grep -v /vendor/
+build:
+	go build \
+		--ldflags "-X github.com/tendermint/tendermint/version.GitCommit=`git rev-parse HEAD`"  -o build/tendermint ./cmd/tendermint/
 
-install: get_deps
-	go install github.com/tendermint/tendermint/cmd/tendermint
+build_race:
+	go build -race -o build/tendermint ./cmd/tendermint
 
-build: 
-	go build -o build/tendermint github.com/tendermint/tendermint/cmd/tendermint
+# dist builds binaries for all platforms and packages them for distribution
+dist:
+	@BUILD_TAGS='$(BUILD_TAGS)' sh -c "'$(CURDIR)/scripts/dist.sh'"
 
-build_race: 
-	go build -race -o build/tendermint github.com/tendermint/tendermint/cmd/tendermint
+test:
+	@echo "--> Running go test"
+	@go test $(PACKAGES)
 
-test: build
-	go test `${NOVENDOR}`
-	
-test_race: build
-	go test -race `${NOVENDOR}`
+test_race:
+	@echo "--> Running go test --race"
+	@go test -v -race $(PACKAGES)
 
-test_integrations: 
-	bash ./test/test.sh
+test_integrations:
+	@bash ./test/test.sh
 
-test100: build
-	for i in {1..100}; do make test; done
+release:
+	@go test -tags release $(PACKAGES)
+
+test100:
+	@for i in {1..100}; do make test; done
 
 draw_deps:
-	# requires brew install graphviz
-	go get github.com/hirokidaichi/goviz
-	goviz -i github.com/tendermint/tendermint/cmd/tendermint | dot -Tpng -o huge.png
+	# requires brew install graphviz or apt-get install graphviz
+	go get github.com/RobotsAndPencils/goviz
+	@goviz -i github.com/tendermint/tendermint/cmd/tendermint -d 3 | dot -Tpng -o dependency-graph.png
 
 list_deps:
-	go list -f '{{join .Deps "\n"}}' github.com/tendermint/tendermint/... | \
+	@go list -f '{{join .Deps "\n"}}' ./... | \
 		grep -v /vendor/ | sort | uniq | \
-	  xargs go list -f '{{if not .Standard}}{{.ImportPath}}{{end}}'
+		xargs go list -f '{{if not .Standard}}{{.ImportPath}}{{end}}'
 
 get_deps:
-	go get -d `${NOVENDOR}`
-	go list -f '{{join .TestImports "\n"}}' github.com/tendermint/tendermint/... | \
+	@echo "--> Running go get"
+	@go get -v -d $(PACKAGES)
+	@go list -f '{{join .TestImports "\n"}}' ./... | \
 		grep -v /vendor/ | sort | uniq | \
-		xargs go get
+		xargs go get -v -d
 
-get_vendor_deps:
-	go get github.com/Masterminds/glide
-	glide install
+get_vendor_deps: ensure_tools
+	@rm -rf vendor/
+	@echo "--> Running glide install"
+	@glide install
 
-update_deps:
-	go get -d -u github.com/tendermint/tendermint/...
+update_deps: tools
+	@echo "--> Updating dependencies"
+	@go get -d -u ./...
 
 revision:
-	-echo `git rev-parse --verify HEAD` > $(TMROOT)/revision
-	-echo `git rev-parse --verify HEAD` >> $(TMROOT)/revision_history
+	-echo `git rev-parse --verify HEAD` > $(TMHOME)/revision
+	-echo `git rev-parse --verify HEAD` >> $(TMHOME)/revision_history
+
+tools:
+	go get -u -v $(GOTOOLS)
+
+ensure_tools:
+	go get $(GOTOOLS)
+
+### Formatting, linting, and vetting
+
+megacheck:
+	@for pkg in ${PACKAGES}; do megacheck "$$pkg"; done
+
+
+.PHONY: install build build_race dist test test_race test_integrations test100 draw_deps list_deps get_deps get_vendor_deps update_deps revision tools

README.md

@@ -1,43 +1,105 @@
 # Tendermint
-Simple, Secure, Scalable Blockchain Platform
 
-[![CircleCI](https://circleci.com/gh/tendermint/tendermint.svg?style=svg)](https://circleci.com/gh/tendermint/tendermint)
-[![codecov](https://codecov.io/gh/tendermint/tendermint/branch/develop/graph/badge.svg)](https://codecov.io/gh/tendermint/tendermint)
+[Byzantine-Fault Tolerant](https://en.wikipedia.org/wiki/Byzantine_fault_tolerance)
+[State Machine Replication](https://en.wikipedia.org/wiki/State_machine_replication).
+Or [Blockchain](https://en.wikipedia.org/wiki/Blockchain_(database)) for short.
 
-_NOTE: This is yet pre-alpha non-production-quality software._
+[![version](https://img.shields.io/github/tag/tendermint/tendermint.svg)](https://github.com/tendermint/tendermint/releases/latest)
+[![API Reference](
+https://camo.githubusercontent.com/915b7be44ada53c290eb157634330494ebe3e30a/68747470733a2f2f676f646f632e6f72672f6769746875622e636f6d2f676f6c616e672f6764646f3f7374617475732e737667
+)](https://godoc.org/github.com/tendermint/tendermint)
+[![Rocket.Chat](https://demo.rocket.chat/images/join-chat.svg)](https://cosmos.rocket.chat/)
+[![license](https://img.shields.io/github/license/tendermint/tendermint.svg)](https://github.com/tendermint/tendermint/blob/master/LICENSE)
+[![](https://tokei.rs/b1/github/tendermint/tendermint?category=lines)](https://github.com/tendermint/tendermint)
 
-Tendermint Core is Byzantine Fault Tolerant (BFT) middleware that takes a state transition machine, written in any programming language,
-and replicates it on many machines.
-See the [application developers guide](https://github.com/tendermint/tendermint/wiki/Application-Developers) to get started.
 
-## Contributing
+Branch    | Tests | Coverage
+----------|-------|----------
+master    | [![CircleCI](https://circleci.com/gh/tendermint/tendermint/tree/master.svg?style=shield)](https://circleci.com/gh/tendermint/tendermint/tree/master) | [![codecov](https://codecov.io/gh/tendermint/tendermint/branch/master/graph/badge.svg)](https://codecov.io/gh/tendermint/tendermint)
+develop   | [![CircleCI](https://circleci.com/gh/tendermint/tendermint/tree/develop.svg?style=shield)](https://circleci.com/gh/tendermint/tendermint/tree/develop) | [![codecov](https://codecov.io/gh/tendermint/tendermint/branch/develop/graph/badge.svg)](https://codecov.io/gh/tendermint/tendermint)
 
-Yay open source! Please see our [contributing guidelines](https://github.com/tendermint/tendermint/wiki/Contributing).
+_NOTE: This is alpha software. Please contact us if you intend to run it in production._
+
+Tendermint Core is Byzantine Fault Tolerant (BFT) middleware that takes a state transition machine - written in any programming language -
+and securely replicates it on many machines.
+
+For more information, from introduction to install to application development, [Read The Docs](http://tendermint.readthedocs.io/projects/tools/en/master).
+
+## Install
+
+To download pre-built binaries, see our [downloads page](https://tendermint.com/downloads).
+
+To install from source, you should be able to:
+
+`go get -u github.com/tendermint/tendermint/cmd/tendermint`
+
+For more details (or if it fails), [read the docs](http://tendermint.readthedocs.io/projects/tools/en/master/install.html).
 
 ## Resources
 
 ### Tendermint Core
 
-- [Introduction](https://github.com/tendermint/tendermint/wiki/Introduction)
-- [Validators](https://github.com/tendermint/tendermint/wiki/Validators)
-- [Byzantine Consensus Algorithm](https://github.com/tendermint/tendermint/wiki/Byzantine-Consensus-Algorithm)
-- [Block Structure](https://github.com/tendermint/tendermint/wiki/Block-Structure)
-- [RPC](https://github.com/tendermint/tendermint/wiki/RPC)
-- [Genesis](https://github.com/tendermint/tendermint/wiki/Genesis)
-- [Configuration](https://github.com/tendermint/tendermint/wiki/Configuration)
-- [Light Client Protocol](https://github.com/tendermint/tendermint/wiki/Light-Client-Protocol)
-- [Roadmap for V2](https://github.com/tendermint/tendermint/wiki/Roadmap-for-V2)
+All resources involving the use of, building application on, or developing for, tendermint, can be found at [Read The Docs](http://tendermint.readthedocs.io/projects/tools/en/master). Additional information about some - and eventually all - of the sub-projects below, can be found at Read The Docs.
 
 ### Sub-projects
 
-* [TMSP](http://github.com/tendermint/tmsp)
-* [Mintnet](http://github.com/tendermint/mintnet)
-* [Go-Wire](http://github.com/tendermint/go-wire)
-* [Go-P2P](http://github.com/tendermint/go-p2p)
-* [Go-Merkle](http://github.com/tendermint/go-merkle)
+* [ABCI](http://github.com/tendermint/abci), the Application Blockchain Interface
+* [Go-Wire](http://github.com/tendermint/go-wire), a deterministic serialization library
+* [Go-Crypto](http://github.com/tendermint/go-crypto), an elliptic curve cryptography library
+* [TmLibs](http://github.com/tendermint/tmlibs), an assortment of Go libraries used internally
+* [IAVL](http://github.com/tendermint/iavl), Merkleized IAVL+ Tree implementation
 
-## Install
+### Tools
+* [Deployment, Benchmarking, and Monitoring](http://tendermint.readthedocs.io/projects/tools/en/develop/index.html#tendermint-tools)
 
-`go get -u github.com/tendermint/tendermint/cmd/tendermint`
+### Applications
 
-For more details, see the [install guide](https://github.com/tendermint/tendermint/wiki/Installation).
+* [Ethermint](http://github.com/tendermint/ethermint); Ethereum on Tendermint
+* [Cosmos SDK](http://github.com/cosmos/cosmos-sdk); a cryptocurrency application framework
+* [Many more](https://tendermint.readthedocs.io/en/master/ecosystem.html#abci-applications)
+
+### More
+
+* [Master's Thesis on Tendermint](https://atrium.lib.uoguelph.ca/xmlui/handle/10214/9769)
+* [Original Whitepaper](https://tendermint.com/static/docs/tendermint.pdf)
+* [Tendermint Blog](https://blog.cosmos.network/tendermint/home)
+* [Cosmos Blog](https://blog.cosmos.network)
+
+## Contributing
+
+Yay open source! Please see our [contributing guidelines](CONTRIBUTING.md).
+
+## Versioning
+
+### SemVer
+
+Tendermint uses [SemVer](http://semver.org/) to determine when and how the version changes.
+According to SemVer, anything in the public API can change at any time before version 1.0.0
+
+To provide some stability to Tendermint users in these 0.X.X days, the MINOR version is used
+to signal breaking changes across a subset of the total public API. This subset includes all
+interfaces exposed to other processes (cli, rpc, p2p, etc.), as well as parts of the following packages:
+
+- types
+- rpc/client
+- config
+- node
+
+Exported objects in these packages that are not covered by the versioning scheme
+are explicitly marked by `// UNSTABLE` in their go doc comment and may change at any time.
+Functions, types, and values in any other package may also change at any time.
+
+### Upgrades
+
+In an effort to avoid accumulating technical debt prior to 1.0.0,
+we do not guarantee that breaking changes (ie. bumps in the MINOR version)
+will work with existing tendermint blockchains. In these cases you will
+have to start a new blockchain, or write something custom to get the old
+data into the new chain.
+
+However, any bump in the PATCH version should be compatible with existing histories
+(if not please open an [issue](https://github.com/tendermint/tendermint/issues)).
+
+## Code of Conduct
+
+Please read, understand and adhere to our [code of conduct](CODE_OF_CONDUCT.md).

Vagrantfile

@@ -1,25 +1,38 @@
 # -*- mode: ruby -*-
 # vi: set ft=ruby :
 
-# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
-VAGRANTFILE_API_VERSION = "2"
+Vagrant.configure("2") do |config|
+  config.vm.box = "ubuntu/trusty64"
 
-Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
-  config.vm.box = "phusion-open-ubuntu-14.04-amd64"
-  config.vm.box_url = "https://oss-binaries.phusionpassenger.com/vagrant/boxes/latest/ubuntu-14.04-amd64-vbox.box"
-  # Or, for Ubuntu 12.04:
-
-  config.vm.provider :vmware_fusion do |f, override|
-    override.vm.box_url = "https://oss-binaries.phusionpassenger.com/vagrant/boxes/latest/ubuntu-14.04-amd64-vmwarefusion.box"
+  config.vm.provider "virtualbox" do |v|
+    v.memory = 4096
+    v.cpus = 2
   end
 
-  if Dir.glob("#{File.dirname(__FILE__)}/.vagrant/machines/default/*/id").empty?
-    # Install Docker
-    pkg_cmd = "wget -q -O - https://get.docker.io/gpg | apt-key add -;" \
-      "echo deb http://get.docker.io/ubuntu docker main > /etc/apt/sources.list.d/docker.list;" \
-      "apt-get update -qq; apt-get install -q -y --force-yes lxc-docker; "
-    # Add vagrant user to the docker group
-    pkg_cmd << "usermod -a -G docker vagrant; "
-    config.vm.provision :shell, :inline => pkg_cmd
-  end
+  config.vm.provision "shell", inline: <<-SHELL
+    apt-get update
+    apt-get install -y --no-install-recommends wget curl jq shellcheck bsdmainutils psmisc
+
+    wget -qO- https://get.docker.com/ | sh
+    usermod -a -G docker vagrant
+    apt-get autoremove -y
+
+    curl -O https://storage.googleapis.com/golang/go1.9.linux-amd64.tar.gz
+    tar -xvf go1.9.linux-amd64.tar.gz
+    rm -rf /usr/local/go
+    mv go /usr/local
+    rm -f go1.9.linux-amd64.tar.gz
+    mkdir -p /home/vagrant/go/bin
+    echo 'export PATH=$PATH:/usr/local/go/bin:/home/vagrant/go/bin' >> /home/vagrant/.bash_profile
+    echo 'export GOPATH=/home/vagrant/go' >> /home/vagrant/.bash_profile
+
+    echo 'export LC_ALL=en_US.UTF-8' >> /home/vagrant/.bash_profile
+
+    mkdir -p /home/vagrant/go/src/github.com/tendermint
+    ln -s /vagrant /home/vagrant/go/src/github.com/tendermint/tendermint
+
+    chown -R vagrant:vagrant /home/vagrant/go
+
+    su - vagrant -c 'cd /home/vagrant/go/src/github.com/tendermint/tendermint && make get_vendor_deps'
+  SHELL
 end

benchmarks/blockchain/.gitignore

@@ -0,0 +1,2 @@
+data
+

benchmarks/blockchain/localsync.sh

@@ -0,0 +1,80 @@
+#!/bin/bash
+
+DATA=$GOPATH/src/github.com/tendermint/tendermint/benchmarks/blockchain/data
+if [ ! -d $DATA ]; then
+  echo "no data found, generating a chain... (this only has to happen once)"
+
+  tendermint init --home $DATA
+  cp $DATA/config.toml $DATA/config2.toml
+  echo "
+  [consensus]
+  timeout_commit = 0
+  " >> $DATA/config.toml
+
+  echo "starting node"
+  tendermint node \
+    --home $DATA \
+    --proxy_app dummy \
+    --p2p.laddr tcp://127.0.0.1:56656 \
+    --rpc.laddr tcp://127.0.0.1:56657 \
+    --log_level error &
+
+  echo "making blocks for 60s"
+  sleep 60
+
+  mv $DATA/config2.toml $DATA/config.toml
+
+  kill %1
+
+  echo "done generating chain."
+fi
+
+# validator node
+HOME1=$TMPDIR$RANDOM$RANDOM
+cp -R $DATA $HOME1
+echo "starting validator node"
+tendermint node \
+  --home $HOME1 \
+  --proxy_app dummy \
+  --p2p.laddr tcp://127.0.0.1:56656 \
+  --rpc.laddr tcp://127.0.0.1:56657 \
+  --log_level error &
+sleep 1
+
+# downloader node
+HOME2=$TMPDIR$RANDOM$RANDOM
+tendermint init --home $HOME2
+cp $HOME1/genesis.json $HOME2
+printf "starting downloader node"
+tendermint node \
+  --home $HOME2 \
+  --proxy_app dummy \
+  --p2p.laddr tcp://127.0.0.1:56666 \
+  --rpc.laddr tcp://127.0.0.1:56667 \
+  --p2p.seeds 127.0.0.1:56656 \
+  --log_level error &
+
+# wait for node to start up so we only count time where we are actually syncing
+sleep 0.5
+while curl localhost:56667/status 2> /dev/null | grep "\"latest_block_height\": 0," > /dev/null
+do
+  printf '.'
+  sleep 0.2
+done
+echo
+
+echo "syncing blockchain for 10s"
+for i in {1..10}
+do
+  sleep 1
+  HEIGHT="$(curl localhost:56667/status 2> /dev/null \
+    | grep 'latest_block_height' \
+    | grep -o ' [0-9]*' \
+    | xargs)"
+  let 'RATE = HEIGHT / i'
+  echo "height: $HEIGHT, blocks/sec: $RATE"
+done
+
+kill %1
+kill %2
+rm -rf $HOME1 $HOME2

benchmarks/codec_test.go

@@ -4,7 +4,7 @@ import (
 	"testing"
 
 	"github.com/tendermint/go-crypto"
-	"github.com/tendermint/go-p2p"
+	"github.com/tendermint/tendermint/p2p"
 	"github.com/tendermint/go-wire"
 	proto "github.com/tendermint/tendermint/benchmarks/proto"
 	ctypes "github.com/tendermint/tendermint/rpc/core/types"
@@ -12,10 +12,10 @@ import (
 
 func BenchmarkEncodeStatusWire(b *testing.B) {
 	b.StopTimer()
-	pubKey := crypto.GenPrivKeyEd25519().PubKey().(crypto.PubKeyEd25519)
+	pubKey := crypto.GenPrivKeyEd25519().PubKey()
 	status := &ctypes.ResultStatus{
 		NodeInfo: &p2p.NodeInfo{
-			PubKey:     pubKey,
+			PubKey:     pubKey.Unwrap().(crypto.PubKeyEd25519),
 			Moniker:    "SOMENAME",
 			Network:    "SOMENAME",
 			RemoteAddr: "SOMEADDR",
@@ -40,7 +40,7 @@ func BenchmarkEncodeStatusWire(b *testing.B) {
 
 func BenchmarkEncodeNodeInfoWire(b *testing.B) {
 	b.StopTimer()
-	pubKey := crypto.GenPrivKeyEd25519().PubKey().(crypto.PubKeyEd25519)
+	pubKey := crypto.GenPrivKeyEd25519().PubKey().Unwrap().(crypto.PubKeyEd25519)
 	nodeInfo := &p2p.NodeInfo{
 		PubKey:     pubKey,
 		Moniker:    "SOMENAME",
@@ -61,7 +61,7 @@ func BenchmarkEncodeNodeInfoWire(b *testing.B) {
 
 func BenchmarkEncodeNodeInfoBinary(b *testing.B) {
 	b.StopTimer()
-	pubKey := crypto.GenPrivKeyEd25519().PubKey().(crypto.PubKeyEd25519)
+	pubKey := crypto.GenPrivKeyEd25519().PubKey().Unwrap().(crypto.PubKeyEd25519)
 	nodeInfo := &p2p.NodeInfo{
 		PubKey:     pubKey,
 		Moniker:    "SOMENAME",
@@ -83,7 +83,7 @@ func BenchmarkEncodeNodeInfoBinary(b *testing.B) {
 
 func BenchmarkEncodeNodeInfoProto(b *testing.B) {
 	b.StopTimer()
-	pubKey := crypto.GenPrivKeyEd25519().PubKey().(crypto.PubKeyEd25519)
+	pubKey := crypto.GenPrivKeyEd25519().PubKey().Unwrap().(crypto.PubKeyEd25519)
 	pubKey2 := &proto.PubKey{Ed25519: &proto.PubKeyEd25519{Bytes: pubKey[:]}}
 	nodeInfo := &proto.NodeInfo{
 		PubKey:     pubKey2,

benchmarks/map_test.go

@@ -1,8 +1,9 @@
 package benchmarks
 
 import (
-	. "github.com/tendermint/go-common"
 	"testing"
+
+	cmn "github.com/tendermint/tmlibs/common"
 )
 
 func BenchmarkSomething(b *testing.B) {
@@ -11,11 +12,11 @@ func BenchmarkSomething(b *testing.B) {
 	numChecks := 100000
 	keys := make([]string, numItems)
 	for i := 0; i < numItems; i++ {
-		keys[i] = RandStr(100)
+		keys[i] = cmn.RandStr(100)
 	}
 	txs := make([]string, numChecks)
 	for i := 0; i < numChecks; i++ {
-		txs[i] = RandStr(100)
+		txs[i] = cmn.RandStr(100)
 	}
 	b.StartTimer()
 

benchmarks/os_test.go

@@ -4,7 +4,7 @@ import (
 	"os"
 	"testing"
 
-	. "github.com/tendermint/go-common"
+	cmn "github.com/tendermint/tmlibs/common"
 )
 
 func BenchmarkFileWrite(b *testing.B) {
@@ -14,7 +14,7 @@ func BenchmarkFileWrite(b *testing.B) {
 	if err != nil {
 		b.Error(err)
 	}
-	testString := RandStr(200) + "\n"
+	testString := cmn.RandStr(200) + "\n"
 	b.StartTimer()
 
 	for i := 0; i < b.N; i++ {

benchmarks/simu/counter.go

@@ -1,30 +1,27 @@
 package main
 
 import (
+	"context"
 	"encoding/binary"
-	"time"
-	//"encoding/hex"
 	"fmt"
+	"time"
 
-	"github.com/gorilla/websocket"
-	. "github.com/tendermint/go-common"
-	"github.com/tendermint/go-rpc/client"
-	"github.com/tendermint/go-rpc/types"
-	"github.com/tendermint/go-wire"
-	_ "github.com/tendermint/tendermint/rpc/core/types" // Register RPCResponse > Result types
+	rpcclient "github.com/tendermint/tendermint/rpc/lib/client"
+	cmn "github.com/tendermint/tmlibs/common"
 )
 
 func main() {
-	ws := rpcclient.NewWSClient("127.0.0.1:46657", "/websocket")
-	_, err := ws.Start()
+	wsc := rpcclient.NewWSClient("127.0.0.1:46657", "/websocket")
+	_, err := wsc.Start()
 	if err != nil {
-		Exit(err.Error())
+		cmn.Exit(err.Error())
 	}
+	defer wsc.Stop()
 
 	// Read a bunch of responses
 	go func() {
 		for {
-			_, ok := <-ws.ResultsCh
+			_, ok := <-wsc.ResponsesCh
 			if !ok {
 				break
 			}
@@ -37,19 +34,14 @@ func main() {
 	for i := 0; ; i++ {
 		binary.BigEndian.PutUint64(buf, uint64(i))
 		//txBytes := hex.EncodeToString(buf[:n])
-		request := rpctypes.NewRPCRequest("fakeid", "broadcast_tx", Arr(buf[:8]))
-		reqBytes := wire.JSONBytes(request)
-		//fmt.Println("!!", string(reqBytes))
 		fmt.Print(".")
-		err := ws.WriteMessage(websocket.TextMessage, reqBytes)
+		err = wsc.Call(context.TODO(), "broadcast_tx", map[string]interface{}{"tx": buf[:8]})
 		if err != nil {
-			Exit(err.Error())
+			cmn.Exit(err.Error())
 		}
 		if i%1000 == 0 {
 			fmt.Println(i)
 		}
 		time.Sleep(time.Microsecond * 1000)
 	}
-
-	ws.Stop()
 }

blockchain/log.go

@@ -1,7 +0,0 @@
-package blockchain
-
-import (
-	"github.com/tendermint/go-logger"
-)
-
-var log = logger.New("module", "blockchain")

blockchain/pool.go

@@ -5,16 +5,31 @@ import (
 	"sync"
 	"time"
 
-	flow "github.com/tendermint/flowcontrol"
-	. "github.com/tendermint/go-common"
 	"github.com/tendermint/tendermint/types"
+	cmn "github.com/tendermint/tmlibs/common"
+	flow "github.com/tendermint/tmlibs/flowrate"
+	"github.com/tendermint/tmlibs/log"
 )
 
+/*
+
+eg, L = latency = 0.1s
+	P = num peers = 10
+	FN = num full nodes
+	BS = 1kB block size
+	CB = 1 Mbit/s = 128 kB/s
+	CB/P = 12.8 kB
+	B/S = CB/P/BS = 12.8 blocks/s
+
+	12.8 * 0.1 = 1.28 blocks on conn
+
+*/
+
 const (
-	requestIntervalMS         = 250
-	maxTotalRequesters        = 300
+	requestIntervalMS         = 100
+	maxTotalRequesters        = 1000
 	maxPendingRequests        = maxTotalRequesters
-	maxPendingRequestsPerPeer = 75
+	maxPendingRequestsPerPeer = 50
 	minRecvRate               = 10240 // 10Kb/s
 )
 
@@ -26,13 +41,13 @@ var peerTimeoutSeconds = time.Duration(15) // not const so we can override with
 	in sequence from peers that reported higher heights than ours.
 	Every so often we ask peers what height they're on so we can keep going.
 
-	Requests are continuously made for blocks of heigher heights until
-	the limits. If most of the requests have no available peers, and we
+	Requests are continuously made for blocks of higher heights until
+	the limit is reached. If most of the requests have no available peers, and we
 	are not at peer limits, we can probably switch to consensus reactor
 */
 
 type BlockPool struct {
-	QuitService
+	cmn.BaseService
 	startTime time.Time
 
 	mtx sync.Mutex
@@ -41,7 +56,8 @@ type BlockPool struct {
 	height     int   // the lowest key in requesters.
 	numPending int32 // number of requests pending assignment or block response
 	// peers
-	peers map[string]*bpPeer
+	peers         map[string]*bpPeer
+	maxPeerHeight int
 
 	requestsCh chan<- BlockRequest
 	timeoutsCh chan<- string
@@ -58,27 +74,26 @@ func NewBlockPool(start int, requestsCh chan<- BlockRequest, timeoutsCh chan<- s
 		requestsCh: requestsCh,
 		timeoutsCh: timeoutsCh,
 	}
-	bp.QuitService = *NewQuitService(log, "BlockPool", bp)
+	bp.BaseService = *cmn.NewBaseService(nil, "BlockPool", bp)
 	return bp
 }
 
 func (pool *BlockPool) OnStart() error {
-	pool.QuitService.OnStart()
 	go pool.makeRequestersRoutine()
 	pool.startTime = time.Now()
 	return nil
 }
 
-func (pool *BlockPool) OnStop() {
-	pool.QuitService.OnStop()
-}
+func (pool *BlockPool) OnStop() {}
 
 // Run spawns requesters as needed.
 func (pool *BlockPool) makeRequestersRoutine() {
+
 	for {
 		if !pool.IsRunning() {
 			break
 		}
+
 		_, numPending, lenRequesters := pool.GetStatus()
 		if numPending >= maxPendingRequests {
 			// sleep for a bit.
@@ -107,7 +122,7 @@ func (pool *BlockPool) removeTimedoutPeers() {
 			// XXX remove curRate != 0
 			if curRate != 0 && curRate < minRecvRate {
 				pool.sendTimeout(peer.id)
-				log.Warn("SendTimeout", "peer", peer.id, "reason", "curRate too low")
+				pool.Logger.Error("SendTimeout", "peer", peer.id, "reason", "curRate too low")
 				peer.didTimeout = true
 			}
 		}
@@ -129,21 +144,16 @@ func (pool *BlockPool) IsCaughtUp() bool {
 	pool.mtx.Lock()
 	defer pool.mtx.Unlock()
 
-	height := pool.height
-
 	// Need at least 1 peer to be considered caught up.
 	if len(pool.peers) == 0 {
-		log.Debug("Blockpool has no peers")
+		pool.Logger.Debug("Blockpool has no peers")
 		return false
 	}
 
-	maxPeerHeight := 0
-	for _, peer := range pool.peers {
-		maxPeerHeight = MaxInt(maxPeerHeight, peer.height)
-	}
-
-	isCaughtUp := (height > 0 || time.Now().Sub(pool.startTime) > 5*time.Second) && (maxPeerHeight == 0 || height >= maxPeerHeight)
-	log.Notice(Fmt("IsCaughtUp: %v", isCaughtUp), "height", height, "maxPeerHeight", maxPeerHeight)
+	// some conditions to determine if we're caught up
+	receivedBlockOrTimedOut := (pool.height > 0 || time.Since(pool.startTime) > 5*time.Second)
+	ourChainIsLongestAmongPeers := pool.maxPeerHeight == 0 || pool.height >= pool.maxPeerHeight
+	isCaughtUp := receivedBlockOrTimedOut && ourChainIsLongestAmongPeers
 	return isCaughtUp
 }
 
@@ -179,7 +189,7 @@ func (pool *BlockPool) PopRequest() {
 		delete(pool.requesters, pool.height)
 		pool.height++
 	} else {
-		PanicSanity(Fmt("Expected requester to pop, got nothing at height %v", pool.height))
+		cmn.PanicSanity(cmn.Fmt("Expected requester to pop, got nothing at height %v", pool.height))
 	}
 }
 
@@ -187,15 +197,16 @@ func (pool *BlockPool) PopRequest() {
 // Remove the peer and redo request from others.
 func (pool *BlockPool) RedoRequest(height int) {
 	pool.mtx.Lock()
+	defer pool.mtx.Unlock()
+
 	request := pool.requesters[height]
-	pool.mtx.Unlock()
 
 	if request.block == nil {
-		PanicSanity("Expected block to be non-nil")
+		cmn.PanicSanity("Expected block to be non-nil")
 	}
 	// RemovePeer will redo all requesters associated with this peer.
 	// TODO: record this malfeasance
-	pool.RemovePeer(request.peerID)
+	pool.removePeer(request.peerID)
 }
 
 // TODO: ensure that blocks come in order for each peer.
@@ -205,18 +216,29 @@ func (pool *BlockPool) AddBlock(peerID string, block *types.Block, blockSize int
 
 	requester := pool.requesters[block.Height]
 	if requester == nil {
+		// a block we didn't expect.
+		// TODO:if height is too far ahead, punish peer
 		return
 	}
 
 	if requester.setBlock(block, peerID) {
 		pool.numPending--
 		peer := pool.peers[peerID]
-		peer.decrPending(blockSize)
+		if peer != nil {
+			peer.decrPending(blockSize)
+		}
 	} else {
 		// Bad peer?
 	}
 }
 
+// MaxPeerHeight returns the heighest height reported by a peer
+func (pool *BlockPool) MaxPeerHeight() int {
+	pool.mtx.Lock()
+	defer pool.mtx.Unlock()
+	return pool.maxPeerHeight
+}
+
 // Sets the peer's alleged blockchain height.
 func (pool *BlockPool) SetPeerHeight(peerID string, height int) {
 	pool.mtx.Lock()
@@ -227,8 +249,13 @@ func (pool *BlockPool) SetPeerHeight(peerID string, height int) {
 		peer.height = height
 	} else {
 		peer = newBPPeer(pool, peerID, height)
+		peer.setLogger(pool.Logger.With("peer", peerID))
 		pool.peers[peerID] = peer
 	}
+
+	if height > pool.maxPeerHeight {
+		pool.maxPeerHeight = height
+	}
 }
 
 func (pool *BlockPool) RemovePeer(peerID string) {
@@ -241,7 +268,9 @@ func (pool *BlockPool) RemovePeer(peerID string) {
 func (pool *BlockPool) removePeer(peerID string) {
 	for _, requester := range pool.requesters {
 		if requester.getPeerID() == peerID {
-			pool.numPending++
+			if requester.getBlock() != nil {
+				pool.numPending++
+			}
 			go requester.redo() // pick another peer and ...
 		}
 	}
@@ -258,7 +287,6 @@ func (pool *BlockPool) pickIncrAvailablePeer(minHeight int) *bpPeer {
 		if peer.didTimeout {
 			pool.removePeer(peer.id)
 			continue
-		} else {
 		}
 		if peer.numPending >= maxPendingRequestsPerPeer {
 			continue
@@ -278,6 +306,7 @@ func (pool *BlockPool) makeNextRequester() {
 
 	nextHeight := pool.height + len(pool.requesters)
 	request := newBPRequester(pool, nextHeight)
+	// request.SetLogger(pool.Logger.With("height", nextHeight))
 
 	pool.requesters[nextHeight] = request
 	pool.numPending++
@@ -299,6 +328,7 @@ func (pool *BlockPool) sendTimeout(peerID string) {
 	pool.timeoutsCh <- peerID
 }
 
+// unused by tendermint; left for debugging purposes
 func (pool *BlockPool) debug() string {
 	pool.mtx.Lock() // Lock
 	defer pool.mtx.Unlock()
@@ -306,10 +336,10 @@ func (pool *BlockPool) debug() string {
 	str := ""
 	for h := pool.height; h < pool.height+len(pool.requesters); h++ {
 		if pool.requesters[h] == nil {
-			str += Fmt("H(%v):X ", h)
+			str += cmn.Fmt("H(%v):X ", h)
 		} else {
-			str += Fmt("H(%v):", h)
-			str += Fmt("B?(%v) ", pool.requesters[h].block != nil)
+			str += cmn.Fmt("H(%v):", h)
+			str += cmn.Fmt("B?(%v) ", pool.requesters[h].block != nil)
 		}
 	}
 	return str
@@ -322,11 +352,12 @@ type bpPeer struct {
 	id          string
 	recvMonitor *flow.Monitor
 
-	mtx        sync.Mutex
 	height     int
 	numPending int32
 	timeout    *time.Timer
 	didTimeout bool
+
+	logger log.Logger
 }
 
 func newBPPeer(pool *BlockPool, peerID string, height int) *bpPeer {
@@ -335,13 +366,18 @@ func newBPPeer(pool *BlockPool, peerID string, height int) *bpPeer {
 		id:         peerID,
 		height:     height,
 		numPending: 0,
+		logger:     log.NewNopLogger(),
 	}
 	return peer
 }
 
+func (peer *bpPeer) setLogger(l log.Logger) {
+	peer.logger = l
+}
+
 func (peer *bpPeer) resetMonitor() {
 	peer.recvMonitor = flow.New(time.Second, time.Second*40)
-	var initialValue = float64(minRecvRate) * math.E
+	initialValue := float64(minRecvRate) * math.E
 	peer.recvMonitor.SetREMA(initialValue)
 }
 
@@ -376,14 +412,14 @@ func (peer *bpPeer) onTimeout() {
 	defer peer.pool.mtx.Unlock()
 
 	peer.pool.sendTimeout(peer.id)
-	log.Warn("SendTimeout", "peer", peer.id, "reason", "onTimeout")
+	peer.logger.Error("SendTimeout", "reason", "onTimeout")
 	peer.didTimeout = true
 }
 
 //-------------------------------------
 
 type bpRequester struct {
-	QuitService
+	cmn.BaseService
 	pool       *BlockPool
 	height     int
 	gotBlockCh chan struct{}
@@ -404,12 +440,11 @@ func newBPRequester(pool *BlockPool, height int) *bpRequester {
 		peerID: "",
 		block:  nil,
 	}
-	bpr.QuitService = *NewQuitService(nil, "bpRequester", bpr)
+	bpr.BaseService = *cmn.NewBaseService(nil, "bpRequester", bpr)
 	return bpr
 }
 
 func (bpr *bpRequester) OnStart() error {
-	bpr.QuitService.OnStart()
 	go bpr.requestRoutine()
 	return nil
 }

blockchain/pool_test.go

@@ -5,8 +5,9 @@ import (
 	"testing"
 	"time"
 
-	. "github.com/tendermint/go-common"
 	"github.com/tendermint/tendermint/types"
+	cmn "github.com/tendermint/tmlibs/common"
+	"github.com/tendermint/tmlibs/log"
 )
 
 func init() {
@@ -21,7 +22,7 @@ type testPeer struct {
 func makePeers(numPeers int, minHeight, maxHeight int) map[string]testPeer {
 	peers := make(map[string]testPeer, numPeers)
 	for i := 0; i < numPeers; i++ {
-		peerID := RandStr(12)
+		peerID := cmn.RandStr(12)
 		height := minHeight + rand.Intn(maxHeight-minHeight)
 		peers[peerID] = testPeer{peerID, height}
 	}
@@ -34,7 +35,9 @@ func TestBasic(t *testing.T) {
 	timeoutsCh := make(chan string, 100)
 	requestsCh := make(chan BlockRequest, 100)
 	pool := NewBlockPool(start, requestsCh, timeoutsCh)
+	pool.SetLogger(log.TestingLogger())
 	pool.Start()
+	defer pool.Stop()
 
 	// Introduce each peer.
 	go func() {
@@ -64,7 +67,7 @@ func TestBasic(t *testing.T) {
 		case peerID := <-timeoutsCh:
 			t.Errorf("timeout: %v", peerID)
 		case request := <-requestsCh:
-			log.Info("TEST: Pulled new BlockRequest", "request", request)
+			t.Logf("Pulled new BlockRequest %v", request)
 			if request.Height == 300 {
 				return // Done!
 			}
@@ -72,12 +75,10 @@ func TestBasic(t *testing.T) {
 			go func() {
 				block := &types.Block{Header: &types.Header{Height: request.Height}}
 				pool.AddBlock(request.PeerID, block, 123)
-				log.Info("TEST: Added block", "block", request.Height, "peer", request.PeerID)
+				t.Logf("Added block from peer %v (height: %v)", request.PeerID, request.Height)
 			}()
 		}
 	}
-
-	pool.Stop()
 }
 
 func TestTimeout(t *testing.T) {
@@ -86,10 +87,12 @@ func TestTimeout(t *testing.T) {
 	timeoutsCh := make(chan string, 100)
 	requestsCh := make(chan BlockRequest, 100)
 	pool := NewBlockPool(start, requestsCh, timeoutsCh)
+	pool.SetLogger(log.TestingLogger())
 	pool.Start()
+	defer pool.Stop()
 
 	for _, peer := range peers {
-		log.Info("Peer", "id", peer.id)
+		t.Logf("Peer %v", peer.id)
 	}
 
 	// Introduce each peer.
@@ -120,7 +123,7 @@ func TestTimeout(t *testing.T) {
 	for {
 		select {
 		case peerID := <-timeoutsCh:
-			log.Info("Timeout", "peerID", peerID)
+			t.Logf("Peer %v timeouted", peerID)
 			if _, ok := timedOut[peerID]; !ok {
 				counter++
 				if counter == len(peers) {
@@ -128,9 +131,7 @@ func TestTimeout(t *testing.T) {
 				}
 			}
 		case request := <-requestsCh:
-			log.Info("TEST: Pulled new BlockRequest", "request", request)
+			t.Logf("Pulled new BlockRequest %+v", request)
 		}
 	}
-
-	pool.Stop()
 }

blockchain/reactor.go

@@ -3,24 +3,24 @@ package blockchain
 import (
 	"bytes"
 	"errors"
-	"fmt"
 	"reflect"
 	"time"
 
-	. "github.com/tendermint/go-common"
-	"github.com/tendermint/go-events"
-	"github.com/tendermint/go-p2p"
-	"github.com/tendermint/go-wire"
+	wire "github.com/tendermint/go-wire"
+	"github.com/tendermint/tendermint/p2p"
 	"github.com/tendermint/tendermint/proxy"
 	sm "github.com/tendermint/tendermint/state"
 	"github.com/tendermint/tendermint/types"
+	cmn "github.com/tendermint/tmlibs/common"
+	"github.com/tendermint/tmlibs/log"
 )
 
 const (
-	BlockchainChannel      = byte(0x40)
-	defaultChannelCapacity = 100
-	defaultSleepIntervalMS = 500
-	trySyncIntervalMS      = 100
+	// BlockchainChannel is a channel for blocks and status updates (`BlockStore` height)
+	BlockchainChannel = byte(0x40)
+
+	defaultChannelCapacity = 1000
+	trySyncIntervalMS      = 50
 	// stop syncing when last block's time is
 	// within this much of the system time.
 	// stopSyncingDurationMinutes = 10
@@ -29,20 +29,18 @@ const (
 	statusUpdateIntervalSeconds = 10
 	// check if we should switch to consensus reactor
 	switchToConsensusIntervalSeconds = 1
-	maxBlockchainResponseSize        = types.MaxBlockSize + 2
 )
 
 type consensusReactor interface {
 	// for when we switch from blockchain reactor and fast sync to
 	// the consensus machine
-	SwitchToConsensus(*sm.State)
+	SwitchToConsensus(*sm.State, int)
 }
 
 // BlockchainReactor handles long-term catchup syncing.
 type BlockchainReactor struct {
 	p2p.BaseReactor
 
-	sw           *p2p.Switch
 	state        *sm.State
 	proxyAppConn proxy.AppConnConsensus // same as consensus.proxyAppConn
 	store        *BlockStore
@@ -50,17 +48,17 @@ type BlockchainReactor struct {
 	fastSync     bool
 	requestsCh   chan BlockRequest
 	timeoutsCh   chan string
-	lastBlock    *types.Block
 
-	evsw *events.EventSwitch
+	evsw types.EventSwitch
 }
 
+// NewBlockchainReactor returns new reactor instance.
 func NewBlockchainReactor(state *sm.State, proxyAppConn proxy.AppConnConsensus, store *BlockStore, fastSync bool) *BlockchainReactor {
 	if state.LastBlockHeight == store.Height()-1 {
-		store.height -= 1 // XXX HACK, make this better
+		store.height-- // XXX HACK, make this better
 	}
 	if state.LastBlockHeight != store.Height() {
-		PanicSanity(Fmt("state (%v) and store (%v) height mismatch", state.LastBlockHeight, store.Height()))
+		cmn.PanicSanity(cmn.Fmt("state (%v) and store (%v) height mismatch", state.LastBlockHeight, store.Height()))
 	}
 	requestsCh := make(chan BlockRequest, defaultChannelCapacity)
 	timeoutsCh := make(chan string, defaultChannelCapacity)
@@ -78,10 +76,17 @@ func NewBlockchainReactor(state *sm.State, proxyAppConn proxy.AppConnConsensus,
 		requestsCh:   requestsCh,
 		timeoutsCh:   timeoutsCh,
 	}
-	bcR.BaseReactor = *p2p.NewBaseReactor(log, "BlockchainReactor", bcR)
+	bcR.BaseReactor = *p2p.NewBaseReactor("BlockchainReactor", bcR)
 	return bcR
 }
 
+// SetLogger implements cmn.Service by setting the logger on reactor and pool.
+func (bcR *BlockchainReactor) SetLogger(l log.Logger) {
+	bcR.BaseService.Logger = l
+	bcR.pool.Logger = l
+}
+
+// OnStart implements cmn.Service.
 func (bcR *BlockchainReactor) OnStart() error {
 	bcR.BaseReactor.OnStart()
 	if bcR.fastSync {
@@ -94,60 +99,72 @@ func (bcR *BlockchainReactor) OnStart() error {
 	return nil
 }
 
+// OnStop implements cmn.Service.
 func (bcR *BlockchainReactor) OnStop() {
 	bcR.BaseReactor.OnStop()
 	bcR.pool.Stop()
 }
 
-// Implements Reactor
+// GetChannels implements Reactor
 func (bcR *BlockchainReactor) GetChannels() []*p2p.ChannelDescriptor {
 	return []*p2p.ChannelDescriptor{
 		&p2p.ChannelDescriptor{
 			ID:                BlockchainChannel,
-			Priority:          5,
-			SendQueueCapacity: 100,
+			Priority:          10,
+			SendQueueCapacity: 1000,
 		},
 	}
 }
 
-// Implements Reactor
-func (bcR *BlockchainReactor) AddPeer(peer *p2p.Peer) {
-	// Send peer our state.
-	peer.Send(BlockchainChannel, struct{ BlockchainMessage }{&bcStatusResponseMessage{bcR.store.Height()}})
+// AddPeer implements Reactor by sending our state to peer.
+func (bcR *BlockchainReactor) AddPeer(peer p2p.Peer) {
+	if !peer.Send(BlockchainChannel, struct{ BlockchainMessage }{&bcStatusResponseMessage{bcR.store.Height()}}) {
+		// doing nothing, will try later in `poolRoutine`
+	}
 }
 
-// Implements Reactor
-func (bcR *BlockchainReactor) RemovePeer(peer *p2p.Peer, reason interface{}) {
-	// Remove peer from the pool.
-	bcR.pool.RemovePeer(peer.Key)
+// RemovePeer implements Reactor by removing peer from the pool.
+func (bcR *BlockchainReactor) RemovePeer(peer p2p.Peer, reason interface{}) {
+	bcR.pool.RemovePeer(peer.Key())
 }
 
-// Implements Reactor
-func (bcR *BlockchainReactor) Receive(chID byte, src *p2p.Peer, msgBytes []byte) {
-	_, msg, err := DecodeMessage(msgBytes)
+// respondToPeer loads a block and sends it to the requesting peer,
+// if we have it. Otherwise, we'll respond saying we don't have it.
+// According to the Tendermint spec, if all nodes are honest,
+// no node should be requesting for a block that's non-existent.
+func (bcR *BlockchainReactor) respondToPeer(msg *bcBlockRequestMessage, src p2p.Peer) (queued bool) {
+	block := bcR.store.LoadBlock(msg.Height)
+	if block != nil {
+		msg := &bcBlockResponseMessage{Block: block}
+		return src.TrySend(BlockchainChannel, struct{ BlockchainMessage }{msg})
+	}
+
+	bcR.Logger.Info("Peer asking for a block we don't have", "src", src, "height", msg.Height)
+
+	return src.TrySend(BlockchainChannel, struct{ BlockchainMessage }{
+		&bcNoBlockResponseMessage{Height: msg.Height},
+	})
+}
+
+// Receive implements Reactor by handling 4 types of messages (look below).
+func (bcR *BlockchainReactor) Receive(chID byte, src p2p.Peer, msgBytes []byte) {
+	_, msg, err := DecodeMessage(msgBytes, bcR.maxMsgSize())
 	if err != nil {
-		log.Warn("Error decoding message", "error", err)
+		bcR.Logger.Error("Error decoding message", "err", err)
 		return
 	}
 
-	log.Notice("Receive", "src", src, "chID", chID, "msg", msg)
+	bcR.Logger.Debug("Receive", "src", src, "chID", chID, "msg", msg)
 
+	// TODO: improve logic to satisfy megacheck
 	switch msg := msg.(type) {
 	case *bcBlockRequestMessage:
-		// Got a request for a block. Respond with block if we have it.
-		block := bcR.store.LoadBlock(msg.Height)
-		if block != nil {
-			msg := &bcBlockResponseMessage{Block: block}
-			queued := src.TrySend(BlockchainChannel, struct{ BlockchainMessage }{msg})
-			if !queued {
-				// queue is full, just ignore.
-			}
-		} else {
-			// TODO peer is asking for things we don't have.
+		if queued := bcR.respondToPeer(msg, src); !queued {
+			// Unfortunately not queued since the queue is full.
 		}
 	case *bcBlockResponseMessage:
 		// Got a block.
-		bcR.pool.AddBlock(src.Key, msg.Block, len(msgBytes))
+		bcR.pool.AddBlock(src.Key(), msg.Block, len(msgBytes))
 	case *bcStatusRequestMessage:
 		// Send peer our state.
 		queued := src.TrySend(BlockchainChannel, struct{ BlockchainMessage }{&bcStatusResponseMessage{bcR.store.Height()}})
@@ -156,12 +173,18 @@ func (bcR *BlockchainReactor) Receive(chID byte, src *p2p.Peer, msgBytes []byte)
 		}
 	case *bcStatusResponseMessage:
 		// Got a peer status. Unverified.
-		bcR.pool.SetPeerHeight(src.Key, msg.Height)
+		bcR.pool.SetPeerHeight(src.Key(), msg.Height)
 	default:
-		log.Warn(Fmt("Unknown message type %v", reflect.TypeOf(msg)))
+		bcR.Logger.Error(cmn.Fmt("Unknown message type %v", reflect.TypeOf(msg)))
 	}
 }
 
+// maxMsgSize returns the maximum allowable size of a
+// message on the blockchain reactor.
+func (bcR *BlockchainReactor) maxMsgSize() int {
+	return bcR.state.Params.BlockSizeParams.MaxBytes + 2
+}
+
 // Handle messages from the poolReactor telling the reactor what to do.
 // NOTE: Don't sleep in the FOR_LOOP or otherwise slow it down!
 // (Except for the SYNC_LOOP, which is the primary purpose and must be synchronous.)
@@ -171,6 +194,13 @@ func (bcR *BlockchainReactor) poolRoutine() {
 	statusUpdateTicker := time.NewTicker(statusUpdateIntervalSeconds * time.Second)
 	switchToConsensusTicker := time.NewTicker(switchToConsensusIntervalSeconds * time.Second)
 
+	blocksSynced := 0
+
+	chainID := bcR.state.ChainID
+
+	lastHundred := time.Now()
+	lastRate := 0.0
+
 FOR_LOOP:
 	for {
 		select {
@@ -192,62 +222,68 @@ FOR_LOOP:
 			if peer != nil {
 				bcR.Switch.StopPeerForError(peer, errors.New("BlockchainReactor Timeout"))
 			}
-		case _ = <-statusUpdateTicker.C:
+		case <-statusUpdateTicker.C:
 			// ask for status updates
 			go bcR.BroadcastStatusRequest()
-		case _ = <-switchToConsensusTicker.C:
-			height, numPending, _ := bcR.pool.GetStatus()
+		case <-switchToConsensusTicker.C:
+			height, numPending, lenRequesters := bcR.pool.GetStatus()
 			outbound, inbound, _ := bcR.Switch.NumPeers()
-			log.Info("Consensus ticker", "numPending", numPending, "total", len(bcR.pool.requesters),
+			bcR.Logger.Debug("Consensus ticker", "numPending", numPending, "total", lenRequesters,
 				"outbound", outbound, "inbound", inbound)
 			if bcR.pool.IsCaughtUp() {
-				log.Notice("Time to switch to consensus reactor!", "height", height)
+				bcR.Logger.Info("Time to switch to consensus reactor!", "height", height)
 				bcR.pool.Stop()
 
 				conR := bcR.Switch.Reactor("CONSENSUS").(consensusReactor)
-				conR.SwitchToConsensus(bcR.state)
+				conR.SwitchToConsensus(bcR.state, blocksSynced)
 
 				break FOR_LOOP
 			}
-		case _ = <-trySyncTicker.C: // chan time
+		case <-trySyncTicker.C: // chan time
 			// This loop can be slow as long as it's doing syncing work.
 		SYNC_LOOP:
 			for i := 0; i < 10; i++ {
 				// See if there are any blocks to sync.
 				first, second := bcR.pool.PeekTwoBlocks()
-				//log.Info("TrySync peeked", "first", first, "second", second)
+				//bcR.Logger.Info("TrySync peeked", "first", first, "second", second)
 				if first == nil || second == nil {
 					// We need both to sync the first block.
 					break SYNC_LOOP
 				}
-				firstParts := first.MakePartSet()
+				firstParts := first.MakePartSet(bcR.state.Params.BlockPartSizeBytes)
 				firstPartsHeader := firstParts.Header()
 				// Finally, verify the first block using the second's commit
+				// NOTE: we can probably make this more efficient, but note that calling
+				// first.Hash() doesn't verify the tx contents, so MakePartSet() is
+				// currently necessary.
 				err := bcR.state.Validators.VerifyCommit(
-					bcR.state.ChainID, first.Hash(), firstPartsHeader, first.Height, second.LastCommit)
+					chainID, types.BlockID{first.Hash(), firstPartsHeader}, first.Height, second.LastCommit)
 				if err != nil {
-					log.Info("error in validation", "error", err)
+					bcR.Logger.Error("Error in validation", "err", err)
 					bcR.pool.RedoRequest(first.Height)
 					break SYNC_LOOP
 				} else {
 					bcR.pool.PopRequest()
-					// TODO: use ApplyBlock instead of Exec/Commit/SetAppHash/Save
-					err := bcR.state.ExecBlock(bcR.evsw, bcR.proxyAppConn, first, firstPartsHeader)
-					if err != nil {
-						// TODO This is bad, are we zombie?
-						PanicQ(Fmt("Failed to process committed block (%d:%X): %v", first.Height, first.Hash(), err))
-					}
+
+					bcR.store.SaveBlock(first, firstParts, second.LastCommit)
+
+					// TODO: should we be firing events? need to fire NewBlock events manually ...
 					// NOTE: we could improve performance if we
 					// didn't make the app commit to disk every block
 					// ... but we would need a way to get the hash without it persisting
-					res := bcR.proxyAppConn.CommitSync()
-					if res.IsErr() {
-						// TODO Handle gracefully.
-						PanicQ(Fmt("Failed to commit block at application: %v", res))
+					err := bcR.state.ApplyBlock(bcR.evsw, bcR.proxyAppConn, first, firstPartsHeader, types.MockMempool{})
+					if err != nil {
+						// TODO This is bad, are we zombie?
+						cmn.PanicQ(cmn.Fmt("Failed to process committed block (%d:%X): %v", first.Height, first.Hash(), err))
+					}
+					blocksSynced += 1
+
+					if blocksSynced%100 == 0 {
+						lastRate = 0.9*lastRate + 0.1*(100/time.Since(lastHundred).Seconds())
+						bcR.Logger.Info("Fast Sync Rate", "height", bcR.pool.height,
+							"max_peer_height", bcR.pool.MaxPeerHeight(), "blocks/s", lastRate)
+						lastHundred = time.Now()
 					}
-					bcR.store.SaveBlock(first, firstParts, second.LastCommit)
-					bcR.state.AppHash = res.Data
-					bcR.state.Save()
 				}
 			}
 			continue FOR_LOOP
@@ -257,18 +293,14 @@ FOR_LOOP:
 	}
 }
 
-func (bcR *BlockchainReactor) BroadcastStatusResponse() error {
-	bcR.Switch.Broadcast(BlockchainChannel, struct{ BlockchainMessage }{&bcStatusResponseMessage{bcR.store.Height()}})
-	return nil
-}
-
+// BroadcastStatusRequest broadcasts `BlockStore` height.
 func (bcR *BlockchainReactor) BroadcastStatusRequest() error {
 	bcR.Switch.Broadcast(BlockchainChannel, struct{ BlockchainMessage }{&bcStatusRequestMessage{bcR.store.Height()}})
 	return nil
 }
 
-// implements events.Eventable
-func (bcR *BlockchainReactor) SetEventSwitch(evsw *events.EventSwitch) {
+// SetEventSwitch implements events.Eventable
+func (bcR *BlockchainReactor) SetEventSwitch(evsw types.EventSwitch) {
 	bcR.evsw = evsw
 }
 
@@ -276,30 +308,34 @@ func (bcR *BlockchainReactor) SetEventSwitch(evsw *events.EventSwitch) {
 // Messages
 
 const (
-	msgTypeBlockRequest   = byte(0x10)
-	msgTypeBlockResponse  = byte(0x11)
-	msgTypeStatusResponse = byte(0x20)
-	msgTypeStatusRequest  = byte(0x21)
+	msgTypeBlockRequest    = byte(0x10)
+	msgTypeBlockResponse   = byte(0x11)
+	msgTypeNoBlockResponse = byte(0x12)
+	msgTypeStatusResponse  = byte(0x20)
+	msgTypeStatusRequest   = byte(0x21)
 )
 
+// BlockchainMessage is a generic message for this reactor.
 type BlockchainMessage interface{}
 
 var _ = wire.RegisterInterface(
 	struct{ BlockchainMessage }{},
 	wire.ConcreteType{&bcBlockRequestMessage{}, msgTypeBlockRequest},
 	wire.ConcreteType{&bcBlockResponseMessage{}, msgTypeBlockResponse},
+	wire.ConcreteType{&bcNoBlockResponseMessage{}, msgTypeNoBlockResponse},
 	wire.ConcreteType{&bcStatusResponseMessage{}, msgTypeStatusResponse},
 	wire.ConcreteType{&bcStatusRequestMessage{}, msgTypeStatusRequest},
 )
 
+// DecodeMessage decodes BlockchainMessage.
 // TODO: ensure that bz is completely read.
-func DecodeMessage(bz []byte) (msgType byte, msg BlockchainMessage, err error) {
+func DecodeMessage(bz []byte, maxSize int) (msgType byte, msg BlockchainMessage, err error) {
 	msgType = bz[0]
 	n := int(0)
 	r := bytes.NewReader(bz)
-	msg = wire.ReadBinary(struct{ BlockchainMessage }{}, r, maxBlockchainResponseSize, &n, &err).(struct{ BlockchainMessage }).BlockchainMessage
+	msg = wire.ReadBinary(struct{ BlockchainMessage }{}, r, maxSize, &n, &err).(struct{ BlockchainMessage }).BlockchainMessage
 	if err != nil && n != len(bz) {
-		err = errors.New("DecodeMessage() had bytes left over.")
+		err = errors.New("DecodeMessage() had bytes left over")
 	}
 	return
 }
@@ -311,7 +347,15 @@ type bcBlockRequestMessage struct {
 }
 
 func (m *bcBlockRequestMessage) String() string {
-	return fmt.Sprintf("[bcBlockRequestMessage %v]", m.Height)
+	return cmn.Fmt("[bcBlockRequestMessage %v]", m.Height)
+}
+
+type bcNoBlockResponseMessage struct {
+	Height int
+}
+
+func (brm *bcNoBlockResponseMessage) String() string {
+	return cmn.Fmt("[bcNoBlockResponseMessage %d]", brm.Height)
 }
 
 //-------------------------------------
@@ -322,7 +366,7 @@ type bcBlockResponseMessage struct {
 }
 
 func (m *bcBlockResponseMessage) String() string {
-	return fmt.Sprintf("[bcBlockResponseMessage %v]", m.Block.Height)
+	return cmn.Fmt("[bcBlockResponseMessage %v]", m.Block.Height)
 }
 
 //-------------------------------------
@@ -332,7 +376,7 @@ type bcStatusRequestMessage struct {
 }
 
 func (m *bcStatusRequestMessage) String() string {
-	return fmt.Sprintf("[bcStatusRequestMessage %v]", m.Height)
+	return cmn.Fmt("[bcStatusRequestMessage %v]", m.Height)
 }
 
 //-------------------------------------
@@ -342,5 +386,5 @@ type bcStatusResponseMessage struct {
 }
 
 func (m *bcStatusResponseMessage) String() string {
-	return fmt.Sprintf("[bcStatusResponseMessage %v]", m.Height)
+	return cmn.Fmt("[bcStatusResponseMessage %v]", m.Height)
 }

blockchain/reactor_test.go

@@ -0,0 +1,151 @@
+package blockchain
+
+import (
+	"testing"
+
+	wire "github.com/tendermint/go-wire"
+	cmn "github.com/tendermint/tmlibs/common"
+	dbm "github.com/tendermint/tmlibs/db"
+	"github.com/tendermint/tmlibs/log"
+
+	cfg "github.com/tendermint/tendermint/config"
+	"github.com/tendermint/tendermint/p2p"
+	sm "github.com/tendermint/tendermint/state"
+	"github.com/tendermint/tendermint/types"
+)
+
+func newBlockchainReactor(maxBlockHeight int) *BlockchainReactor {
+	logger := log.TestingLogger()
+	config := cfg.ResetTestRoot("blockchain_reactor_test")
+
+	blockStore := NewBlockStore(dbm.NewMemDB())
+
+	// Get State
+	state, _ := sm.GetState(dbm.NewMemDB(), config.GenesisFile())
+	state.SetLogger(logger.With("module", "state"))
+	state.Save()
+
+	// Make the blockchainReactor itself
+	fastSync := true
+	bcReactor := NewBlockchainReactor(state.Copy(), nil, blockStore, fastSync)
+	bcReactor.SetLogger(logger.With("module", "blockchain"))
+
+	// Next: we need to set a switch in order for peers to be added in
+	bcReactor.Switch = p2p.NewSwitch(cfg.DefaultP2PConfig())
+
+	// Lastly: let's add some blocks in
+	for blockHeight := 1; blockHeight <= maxBlockHeight; blockHeight++ {
+		firstBlock := makeBlock(blockHeight, state)
+		secondBlock := makeBlock(blockHeight+1, state)
+		firstParts := firstBlock.MakePartSet(state.Params.BlockGossipParams.BlockPartSizeBytes)
+		blockStore.SaveBlock(firstBlock, firstParts, secondBlock.LastCommit)
+	}
+
+	return bcReactor
+}
+
+func TestNoBlockMessageResponse(t *testing.T) {
+	maxBlockHeight := 20
+
+	bcr := newBlockchainReactor(maxBlockHeight)
+	bcr.Start()
+	defer bcr.Stop()
+
+	// Add some peers in
+	peer := newbcrTestPeer(cmn.RandStr(12))
+	bcr.AddPeer(peer)
+
+	chID := byte(0x01)
+
+	tests := []struct {
+		height   int
+		existent bool
+	}{
+		{maxBlockHeight + 2, false},
+		{10, true},
+		{1, true},
+		{100, false},
+	}
+
+	for _, tt := range tests {
+		reqBlockMsg := &bcBlockRequestMessage{tt.height}
+		reqBlockBytes := wire.BinaryBytes(struct{ BlockchainMessage }{reqBlockMsg})
+		bcr.Receive(chID, peer, reqBlockBytes)
+		value := peer.lastValue()
+		msg := value.(struct{ BlockchainMessage }).BlockchainMessage
+
+		if tt.existent {
+			if blockMsg, ok := msg.(*bcBlockResponseMessage); !ok {
+				t.Fatalf("Expected to receive a block response for height %d", tt.height)
+			} else if blockMsg.Block.Height != tt.height {
+				t.Fatalf("Expected response to be for height %d, got %d", tt.height, blockMsg.Block.Height)
+			}
+		} else {
+			if noBlockMsg, ok := msg.(*bcNoBlockResponseMessage); !ok {
+				t.Fatalf("Expected to receive a no block response for height %d", tt.height)
+			} else if noBlockMsg.Height != tt.height {
+				t.Fatalf("Expected response to be for height %d, got %d", tt.height, noBlockMsg.Height)
+			}
+		}
+	}
+}
+
+//----------------------------------------------
+// utility funcs
+
+func makeTxs(blockNumber int) (txs []types.Tx) {
+	for i := 0; i < 10; i++ {
+		txs = append(txs, types.Tx([]byte{byte(blockNumber), byte(i)}))
+	}
+	return txs
+}
+
+func makeBlock(blockNumber int, state *sm.State) *types.Block {
+	prevHash := state.LastBlockID.Hash
+	prevParts := types.PartSetHeader{}
+	valHash := state.Validators.Hash()
+	prevBlockID := types.BlockID{prevHash, prevParts}
+	block, _ := types.MakeBlock(blockNumber, "test_chain", makeTxs(blockNumber),
+		new(types.Commit), prevBlockID, valHash, state.AppHash, state.Params.BlockGossipParams.BlockPartSizeBytes)
+	return block
+}
+
+// The Test peer
+type bcrTestPeer struct {
+	cmn.Service
+	key string
+	ch  chan interface{}
+}
+
+var _ p2p.Peer = (*bcrTestPeer)(nil)
+
+func newbcrTestPeer(key string) *bcrTestPeer {
+	return &bcrTestPeer{
+		Service: cmn.NewBaseService(nil, "bcrTestPeer", nil),
+		key:     key,
+		ch:      make(chan interface{}, 2),
+	}
+}
+
+func (tp *bcrTestPeer) lastValue() interface{} { return <-tp.ch }
+
+func (tp *bcrTestPeer) TrySend(chID byte, value interface{}) bool {
+	if _, ok := value.(struct{ BlockchainMessage }).BlockchainMessage.(*bcStatusResponseMessage); ok {
+		// Discard status response messages since they skew our results
+		// We only want to deal with:
+		// + bcBlockResponseMessage
+		// + bcNoBlockResponseMessage
+	} else {
+		tp.ch <- value
+	}
+	return true
+}
+
+func (tp *bcrTestPeer) Send(chID byte, data interface{}) bool { return tp.TrySend(chID, data) }
+func (tp *bcrTestPeer) NodeInfo() *p2p.NodeInfo               { return nil }
+func (tp *bcrTestPeer) Status() p2p.ConnectionStatus          { return p2p.ConnectionStatus{} }
+func (tp *bcrTestPeer) Key() string                           { return tp.key }
+func (tp *bcrTestPeer) IsOutbound() bool                      { return false }
+func (tp *bcrTestPeer) IsPersistent() bool                    { return true }
+func (tp *bcrTestPeer) Get(s string) interface{}              { return s }
+func (tp *bcrTestPeer) Set(string, interface{})               {}

blockchain/store.go

@@ -5,11 +5,12 @@ import (
 	"encoding/json"
 	"fmt"
 	"io"
+	"sync"
 
-	. "github.com/tendermint/go-common"
-	dbm "github.com/tendermint/go-db"
-	"github.com/tendermint/go-wire"
+	wire "github.com/tendermint/go-wire"
 	"github.com/tendermint/tendermint/types"
+	. "github.com/tendermint/tmlibs/common"
+	dbm "github.com/tendermint/tmlibs/db"
 )
 
 /*
@@ -24,11 +25,14 @@ Currently the precommit signatures are duplicated in the Block parts as
 well as the Commit.  In the future this may change, perhaps by moving
 the Commit data outside the Block.
 
-Panics indicate probable corruption in the data
+// NOTE: BlockStore methods will panic if they encounter errors
+// deserializing loaded data, indicating probable corruption on disk.
 */
 type BlockStore struct {
+	db dbm.DB
+
+	mtx    sync.RWMutex
 	height int
-	db     dbm.DB
 }
 
 func NewBlockStore(db dbm.DB) *BlockStore {
@@ -41,6 +45,8 @@ func NewBlockStore(db dbm.DB) *BlockStore {
 
 // Height() returns the last known contiguous block height.
 func (bs *BlockStore) Height() int {
+	bs.mtx.RLock()
+	defer bs.mtx.RUnlock()
 	return bs.height
 }
 
@@ -59,12 +65,12 @@ func (bs *BlockStore) LoadBlock(height int) *types.Block {
 	if r == nil {
 		return nil
 	}
-	meta := wire.ReadBinary(&types.BlockMeta{}, r, 0, &n, &err).(*types.BlockMeta)
+	blockMeta := wire.ReadBinary(&types.BlockMeta{}, r, 0, &n, &err).(*types.BlockMeta)
 	if err != nil {
 		PanicCrisis(Fmt("Error reading block meta: %v", err))
 	}
 	bytez := []byte{}
-	for i := 0; i < meta.PartsHeader.Total; i++ {
+	for i := 0; i < blockMeta.BlockID.PartsHeader.Total; i++ {
 		part := bs.LoadBlockPart(height, i)
 		bytez = append(bytez, part.Bytes...)
 	}
@@ -96,11 +102,11 @@ func (bs *BlockStore) LoadBlockMeta(height int) *types.BlockMeta {
 	if r == nil {
 		return nil
 	}
-	meta := wire.ReadBinary(&types.BlockMeta{}, r, 0, &n, &err).(*types.BlockMeta)
+	blockMeta := wire.ReadBinary(&types.BlockMeta{}, r, 0, &n, &err).(*types.BlockMeta)
 	if err != nil {
 		PanicCrisis(Fmt("Error reading block meta: %v", err))
 	}
-	return meta
+	return blockMeta
 }
 
 // The +2/3 and other Precommit-votes for block at `height`.
@@ -141,16 +147,16 @@ func (bs *BlockStore) LoadSeenCommit(height int) *types.Commit {
 //             most recent height.  Otherwise they'd stall at H-1.
 func (bs *BlockStore) SaveBlock(block *types.Block, blockParts *types.PartSet, seenCommit *types.Commit) {
 	height := block.Height
-	if height != bs.height+1 {
-		PanicSanity(Fmt("BlockStore can only save contiguous blocks. Wanted %v, got %v", bs.height+1, height))
+	if height != bs.Height()+1 {
+		PanicSanity(Fmt("BlockStore can only save contiguous blocks. Wanted %v, got %v", bs.Height()+1, height))
 	}
 	if !blockParts.IsComplete() {
 		PanicSanity(Fmt("BlockStore can only save complete block part sets"))
 	}
 
 	// Save block meta
-	meta := types.NewBlockMeta(block, blockParts)
-	metaBytes := wire.BinaryBytes(meta)
+	blockMeta := types.NewBlockMeta(block, blockParts)
+	metaBytes := wire.BinaryBytes(blockMeta)
 	bs.db.Set(calcBlockMetaKey(height), metaBytes)
 
 	// Save block parts
@@ -163,6 +169,7 @@ func (bs *BlockStore) SaveBlock(block *types.Block, blockParts *types.PartSet, s
 	bs.db.Set(calcBlockCommitKey(height-1), blockCommitBytes)
 
 	// Save seen commit (seen +2/3 precommits for block)
+	// NOTE: we can delete this at a later height
 	seenCommitBytes := wire.BinaryBytes(seenCommit)
 	bs.db.Set(calcSeenCommitKey(height), seenCommitBytes)
 
@@ -170,12 +177,17 @@ func (bs *BlockStore) SaveBlock(block *types.Block, blockParts *types.PartSet, s
 	BlockStoreStateJSON{Height: height}.Save(bs.db)
 
 	// Done!
+	bs.mtx.Lock()
 	bs.height = height
+	bs.mtx.Unlock()
+
+	// Flush
+	bs.db.SetSync(nil, nil)
 }
 
 func (bs *BlockStore) saveBlockPart(height int, index int, part *types.Part) {
-	if height != bs.height+1 {
-		PanicSanity(Fmt("BlockStore can only save contiguous blocks. Wanted %v, got %v", bs.height+1, height))
+	if height != bs.Height()+1 {
+		PanicSanity(Fmt("BlockStore can only save contiguous blocks. Wanted %v, got %v", bs.Height()+1, height))
 	}
 	partBytes := wire.BinaryBytes(part)
 	bs.db.Set(calcBlockPartKey(height, index), partBytes)
@@ -212,7 +224,7 @@ func (bsj BlockStoreStateJSON) Save(db dbm.DB) {
 	if err != nil {
 		PanicSanity(Fmt("Could not marshal state bytes: %v", err))
 	}
-	db.Set(blockStoreKey, bytes)
+	db.SetSync(blockStoreKey, bytes)
 }
 
 func LoadBlockStoreStateJSON(db dbm.DB) BlockStoreStateJSON {

certifiers/client/main_test.go

@@ -0,0 +1,25 @@
+package client_test
+
+import (
+	"os"
+	"testing"
+
+	"github.com/tendermint/abci/example/dummy"
+
+	nm "github.com/tendermint/tendermint/node"
+	rpctest "github.com/tendermint/tendermint/rpc/test"
+)
+
+var node *nm.Node
+
+func TestMain(m *testing.M) {
+	// start a tendermint node (and merkleeyes) in the background to test against
+	app := dummy.NewDummyApplication()
+	node = rpctest.StartTendermint(app)
+	code := m.Run()
+
+	// and shut down proper at the end
+	node.Stop()
+	node.Wait()
+	os.Exit(code)
+}

certifiers/client/provider.go

@@ -0,0 +1,133 @@
+/*
+Package client defines a provider that uses a rpcclient
+to get information, which is used to get new headers
+and validators directly from a node.
+*/
+package client
+
+import (
+	"bytes"
+
+	rpcclient "github.com/tendermint/tendermint/rpc/client"
+	ctypes "github.com/tendermint/tendermint/rpc/core/types"
+	"github.com/tendermint/tendermint/types"
+
+	"github.com/tendermint/tendermint/certifiers"
+	certerr "github.com/tendermint/tendermint/certifiers/errors"
+)
+
+type SignStatusClient interface {
+	rpcclient.SignClient
+	rpcclient.StatusClient
+}
+
+type provider struct {
+	node       SignStatusClient
+	lastHeight int
+}
+
+// NewProvider can wrap any rpcclient to expose it as
+// a read-only provider.
+func NewProvider(node SignStatusClient) certifiers.Provider {
+	return &provider{node: node}
+}
+
+// NewProvider can connects to a tendermint json-rpc endpoint
+// at the given url, and uses that as a read-only provider.
+func NewHTTPProvider(remote string) certifiers.Provider {
+	return &provider{
+		node: rpcclient.NewHTTP(remote, "/websocket"),
+	}
+}
+
+// StoreCommit is a noop, as clients can only read from the chain...
+func (p *provider) StoreCommit(_ certifiers.FullCommit) error { return nil }
+
+// GetHash gets the most recent validator and sees if it matches
+//
+// TODO: improve when the rpc interface supports more functionality
+func (p *provider) GetByHash(hash []byte) (certifiers.FullCommit, error) {
+	var fc certifiers.FullCommit
+	vals, err := p.node.Validators(nil)
+	// if we get no validators, or a different height, return an error
+	if err != nil {
+		return fc, err
+	}
+	p.updateHeight(vals.BlockHeight)
+	vhash := types.NewValidatorSet(vals.Validators).Hash()
+	if !bytes.Equal(hash, vhash) {
+		return fc, certerr.ErrCommitNotFound()
+	}
+	return p.seedFromVals(vals)
+}
+
+// GetByHeight gets the validator set by height
+func (p *provider) GetByHeight(h int) (fc certifiers.FullCommit, err error) {
+	commit, err := p.node.Commit(&h)
+	if err != nil {
+		return fc, err
+	}
+	return p.seedFromCommit(commit)
+}
+
+func (p *provider) LatestCommit() (fc certifiers.FullCommit, err error) {
+	commit, err := p.GetLatestCommit()
+	if err != nil {
+		return fc, err
+	}
+	return p.seedFromCommit(commit)
+}
+
+// GetLatestCommit should return the most recent commit there is,
+// which handles queries for future heights as per the semantics
+// of GetByHeight.
+func (p *provider) GetLatestCommit() (*ctypes.ResultCommit, error) {
+	status, err := p.node.Status()
+	if err != nil {
+		return nil, err
+	}
+	return p.node.Commit(&status.LatestBlockHeight)
+}
+
+func CommitFromResult(result *ctypes.ResultCommit) certifiers.Commit {
+	return (certifiers.Commit)(result.SignedHeader)
+}
+
+func (p *provider) seedFromVals(vals *ctypes.ResultValidators) (certifiers.FullCommit, error) {
+	// now get the commits and build a full commit
+	commit, err := p.node.Commit(&vals.BlockHeight)
+	if err != nil {
+		return certifiers.FullCommit{}, err
+	}
+	fc := certifiers.NewFullCommit(
+		CommitFromResult(commit),
+		types.NewValidatorSet(vals.Validators),
+	)
+	return fc, nil
+}
+
+func (p *provider) seedFromCommit(commit *ctypes.ResultCommit) (fc certifiers.FullCommit, err error) {
+	fc.Commit = CommitFromResult(commit)
+
+	// now get the proper validators
+	vals, err := p.node.Validators(&commit.Header.Height)
+	if err != nil {
+		return fc, err
+	}
+
+	// make sure they match the commit (as we cannot enforce height)
+	vset := types.NewValidatorSet(vals.Validators)
+	if !bytes.Equal(vset.Hash(), commit.Header.ValidatorsHash) {
+		return fc, certerr.ErrValidatorsChanged()
+	}
+
+	p.updateHeight(commit.Header.Height)
+	fc.Validators = vset
+	return fc, nil
+}
+
+func (p *provider) updateHeight(h int) {
+	if h > p.lastHeight {
+		p.lastHeight = h
+	}
+}

certifiers/client/provider_test.go

@@ -0,0 +1,62 @@
+package client_test
+
+import (
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+
+	rpctest "github.com/tendermint/tendermint/rpc/test"
+
+	"github.com/tendermint/tendermint/certifiers"
+	"github.com/tendermint/tendermint/certifiers/client"
+	certerr "github.com/tendermint/tendermint/certifiers/errors"
+)
+
+func TestProvider(t *testing.T) {
+	assert, require := assert.New(t), require.New(t)
+
+	cfg := rpctest.GetConfig()
+	rpcAddr := cfg.RPC.ListenAddress
+	chainID := cfg.ChainID
+	p := client.NewHTTPProvider(rpcAddr)
+	require.NotNil(t, p)
+
+	// let it produce some blocks
+	time.Sleep(500 * time.Millisecond)
+
+	// let's get the highest block
+	seed, err := p.LatestCommit()
+
+	require.Nil(err, "%+v", err)
+	sh := seed.Height()
+	vhash := seed.Header.ValidatorsHash
+	assert.True(sh < 5000)
+
+	// let's check this is valid somehow
+	assert.Nil(seed.ValidateBasic(chainID))
+	cert := certifiers.NewStatic(chainID, seed.Validators)
+
+	// historical queries now work :)
+	lower := sh - 5
+	seed, err = p.GetByHeight(lower)
+	assert.Nil(err, "%+v", err)
+	assert.Equal(lower, seed.Height())
+
+	// also get by hash (given the match)
+	seed, err = p.GetByHash(vhash)
+	require.Nil(err, "%+v", err)
+	require.Equal(vhash, seed.Header.ValidatorsHash)
+	err = cert.Certify(seed.Commit)
+	assert.Nil(err, "%+v", err)
+
+	// get by hash fails without match
+	seed, err = p.GetByHash([]byte("foobar"))
+	assert.NotNil(err)
+	assert.True(certerr.IsCommitNotFoundErr(err))
+
+	// storing the seed silently ignored
+	err = p.StoreCommit(seed)
+	assert.Nil(err, "%+v", err)
+}

certifiers/commit.go

@@ -0,0 +1,96 @@
+package certifiers
+
+import (
+	"bytes"
+
+	"github.com/pkg/errors"
+
+	"github.com/tendermint/tendermint/types"
+
+	certerr "github.com/tendermint/tendermint/certifiers/errors"
+)
+
+// Certifier checks the votes to make sure the block really is signed properly.
+// Certifier must know the current set of validitors by some other means.
+type Certifier interface {
+	Certify(check Commit) error
+	ChainID() string
+}
+
+// Commit is basically the rpc /commit response, but extended
+//
+// This is the basepoint for proving anything on the blockchain. It contains
+// a signed header.  If the signatures are valid and > 2/3 of the known set,
+// we can store this checkpoint and use it to prove any number of aspects of
+// the system: such as txs, abci state, validator sets, etc...
+type Commit types.SignedHeader
+
+// FullCommit is a commit and the actual validator set,
+// the base info you need to update to a given point,
+// assuming knowledge of some previous validator set
+type FullCommit struct {
+	Commit     `json:"commit"`
+	Validators *types.ValidatorSet `json:"validator_set"`
+}
+
+func NewFullCommit(commit Commit, vals *types.ValidatorSet) FullCommit {
+	return FullCommit{
+		Commit:     commit,
+		Validators: vals,
+	}
+}
+
+func (c Commit) Height() int {
+	if c.Header == nil {
+		return 0
+	}
+	return c.Header.Height
+}
+
+func (c Commit) ValidatorsHash() []byte {
+	if c.Header == nil {
+		return nil
+	}
+	return c.Header.ValidatorsHash
+}
+
+// ValidateBasic does basic consistency checks and makes sure the headers
+// and commits are all consistent and refer to our chain.
+//
+// Make sure to use a Verifier to validate the signatures actually provide
+// a significantly strong proof for this header's validity.
+func (c Commit) ValidateBasic(chainID string) error {
+	// make sure the header is reasonable
+	if c.Header == nil {
+		return errors.New("Commit missing header")
+	}
+	if c.Header.ChainID != chainID {
+		return errors.Errorf("Header belongs to another chain '%s' not '%s'",
+			c.Header.ChainID, chainID)
+	}
+
+	if c.Commit == nil {
+		return errors.New("Commit missing signatures")
+	}
+
+	// make sure the header and commit match (height and hash)
+	if c.Commit.Height() != c.Header.Height {
+		return certerr.ErrHeightMismatch(c.Commit.Height(), c.Header.Height)
+	}
+	hhash := c.Header.Hash()
+	chash := c.Commit.BlockID.Hash
+	if !bytes.Equal(hhash, chash) {
+		return errors.Errorf("Commits sign block %X header is block %X",
+			chash, hhash)
+	}
+
+	// make sure the commit is reasonable
+	err := c.Commit.ValidateBasic()
+	if err != nil {
+		return errors.WithStack(err)
+	}
+
+	// looks good, we just need to make sure the signatures are really from
+	// empowered validators
+	return nil
+}

certifiers/doc.go

@@ -0,0 +1,133 @@
+/*
+Package certifiers allows you to securely validate headers
+without a full node.
+
+This library pulls together all the crypto and algorithms,
+so given a relatively recent (< unbonding period) known
+validator set, one can get indisputable proof that data is in
+the chain (current state) or detect if the node is lying to
+the client.
+
+Tendermint RPC exposes a lot of info, but a malicious node
+could return any data it wants to queries, or even to block
+headers, even making up fake signatures from non-existent
+validators to justify it. This is a lot of logic to get
+right, to be contained in a small, easy to use library,
+that does this for you, so you can just build nice UI.
+
+We design for clients who have no strong trust relationship
+with any tendermint node, just the validator set as a whole.
+Beyond building nice mobile or desktop applications, the
+cosmos hub is another important example of a client,
+that needs undeniable proof without syncing the full chain,
+in order to efficiently implement IBC.
+
+Commits
+
+There are two main data structures that we pass around - Commit
+and FullCommit. Both of them mirror what information is
+exposed in tendermint rpc.
+
+Commit is a block header along with enough validator signatures
+to prove its validity (> 2/3 of the voting power). A FullCommit
+is a Commit along with the full validator set. When the
+validator set doesn't change, the Commit is enough, but since
+the block header only has a hash, we need the FullCommit to
+follow any changes to the validator set.
+
+Certifiers
+
+A Certifier validates a new Commit given the currently known
+state. There are three different types of Certifiers exposed,
+each one building on the last one, with additional complexity.
+
+Static - given the validator set upon initialization. Verifies
+all signatures against that set and if the validator set
+changes, it will reject all headers.
+
+Dynamic - This wraps Static and has the same Certify
+method. However, it adds an Update method, which can be called
+with a FullCommit when the validator set changes. If it can
+prove this is a valid transition, it will update the validator
+set.
+
+Inquiring - this wraps Dynamic and implements an auto-update
+strategy on top of the Dynamic update. If a call to
+Certify fails as the validator set has changed, then it
+attempts to find a FullCommit and Update to that header.
+To get these FullCommits, it makes use of a Provider.
+
+Providers
+
+A Provider allows us to store and retrieve the FullCommits,
+to provide memory to the Inquiring Certifier.
+
+NewMemStoreProvider - in-memory cache.
+
+files.NewProvider - disk backed storage.
+
+client.NewHTTPProvider - query tendermint rpc.
+
+NewCacheProvider - combine multiple providers.
+
+The suggested use for local light clients is
+client.NewHTTPProvider for getting new data (Source),
+and NewCacheProvider(NewMemStoreProvider(),
+files.NewProvider()) to store confirmed headers (Trusted)
+
+How We Track Validators
+
+Unless you want to blindly trust the node you talk with, you
+need to trace every response back to a hash in a block header
+and validate the commit signatures of that block header match
+the proper validator set.  If there is a contant validator
+set, you store it locally upon initialization of the client,
+and check against that every time.
+
+Once there is a dynamic validator set, the issue of
+verifying a block becomes a bit more tricky. There is
+background information in a
+github issue (https://github.com/tendermint/tendermint/issues/377).
+
+In short, if there is a block at height H with a known
+(trusted) validator set V, and another block at height H'
+(H' > H) with validator set V' != V, then we want a way to
+safely update it.
+
+First, get the new (unconfirmed) validator set V' and
+verify H' is internally consistent and properly signed by
+this V'. Assuming it is a valid block, we check that at
+least 2/3 of the validators in V also signed it, meaning
+it would also be valid under our old assumptions.
+That should be enough, but we can also check that the
+V counts for at least 2/3 of the total votes in H'
+for extra safety (we can have a discussion if this is
+strictly required). If we can verify all this,
+then we can accept H' and V' as valid and use that to
+validate all blocks X > H'.
+
+If we cannot update directly from H -> H' because there was
+too much change to the validator set, then we can look for
+some Hm (H < Hm < H') with a validator set Vm.  Then we try
+to update H -> Hm and Hm -> H' in two separate steps.
+If one of these steps doesn't work, then we continue
+bisecting, until we eventually have to externally
+validate the valdiator set changes at every block.
+
+Since we never trust any server in this protocol, only the
+signatures themselves, it doesn't matter if the seed comes
+from a (possibly malicious) node or a (possibly malicious) user.
+We can accept it or reject it based only on our trusted
+validator set and cryptographic proofs. This makes it
+extremely important to verify that you have the proper
+validator set when initializing the client, as that is the
+root of all trust.
+
+Or course, this assumes that the known block is within the
+unbonding period to avoid the "nothing at stake" problem.
+If you haven't seen the state in a few months, you will need
+to manually verify the new validator set hash using off-chain
+means (the same as getting the initial hash).
+
+*/
+package certifiers

certifiers/dynamic.go

@@ -0,0 +1,89 @@
+package certifiers
+
+import (
+	"github.com/tendermint/tendermint/types"
+
+	certerr "github.com/tendermint/tendermint/certifiers/errors"
+)
+
+var _ Certifier = &Dynamic{}
+
+// Dynamic uses a Static for Certify, but adds an
+// Update method to allow for a change of validators.
+//
+// You can pass in a FullCommit with another validator set,
+// and if this is a provably secure transition (< 1/3 change,
+// sufficient signatures), then it will update the
+// validator set for the next Certify call.
+// For security, it will only follow validator set changes
+// going forward.
+type Dynamic struct {
+	cert       *Static
+	lastHeight int
+}
+
+func NewDynamic(chainID string, vals *types.ValidatorSet, height int) *Dynamic {
+	return &Dynamic{
+		cert:       NewStatic(chainID, vals),
+		lastHeight: height,
+	}
+}
+
+func (c *Dynamic) ChainID() string {
+	return c.cert.ChainID()
+}
+
+func (c *Dynamic) Validators() *types.ValidatorSet {
+	return c.cert.vSet
+}
+
+func (c *Dynamic) Hash() []byte {
+	return c.cert.Hash()
+}
+
+func (c *Dynamic) LastHeight() int {
+	return c.lastHeight
+}
+
+// Certify handles this with
+func (c *Dynamic) Certify(check Commit) error {
+	err := c.cert.Certify(check)
+	if err == nil {
+		// update last seen height if input is valid
+		c.lastHeight = check.Height()
+	}
+	return err
+}
+
+// Update will verify if this is a valid change and update
+// the certifying validator set if safe to do so.
+//
+// Returns an error if update is impossible (invalid proof or IsTooMuchChangeErr)
+func (c *Dynamic) Update(fc FullCommit) error {
+	// ignore all checkpoints in the past -> only to the future
+	h := fc.Height()
+	if h <= c.lastHeight {
+		return certerr.ErrPastTime()
+	}
+
+	// first, verify if the input is self-consistent....
+	err := fc.ValidateBasic(c.ChainID())
+	if err != nil {
+		return err
+	}
+
+	// now, make sure not too much change... meaning this commit
+	// would be approved by the currently known validator set
+	// as well as the new set
+	commit := fc.Commit.Commit
+	err = c.Validators().VerifyCommitAny(fc.Validators, c.ChainID(),
+		commit.BlockID, h, commit)
+	if err != nil {
+		return certerr.ErrTooMuchChange()
+	}
+
+	// looks good, we can update
+	c.cert = NewStatic(c.ChainID(), fc.Validators)
+	c.lastHeight = h
+	return nil
+}

certifiers/dynamic_test.go

@@ -0,0 +1,130 @@
+package certifiers_test
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+
+	"github.com/tendermint/tendermint/types"
+
+	"github.com/tendermint/tendermint/certifiers"
+	"github.com/tendermint/tendermint/certifiers/errors"
+)
+
+// TestDynamicCert just makes sure it still works like StaticCert
+func TestDynamicCert(t *testing.T) {
+	// assert, require := assert.New(t), require.New(t)
+	assert := assert.New(t)
+	// require := require.New(t)
+
+	keys := certifiers.GenValKeys(4)
+	// 20, 30, 40, 50 - the first 3 don't have 2/3, the last 3 do!
+	vals := keys.ToValidators(20, 10)
+	// and a certifier based on our known set
+	chainID := "test-dyno"
+	cert := certifiers.NewDynamic(chainID, vals, 0)
+
+	cases := []struct {
+		keys        certifiers.ValKeys
+		vals        *types.ValidatorSet
+		height      int
+		first, last int  // who actually signs
+		proper      bool // true -> expect no error
+		changed     bool // true -> expect validator change error
+	}{
+		// perfect, signed by everyone
+		{keys, vals, 1, 0, len(keys), true, false},
+		// skip little guy is okay
+		{keys, vals, 2, 1, len(keys), true, false},
+		// but not the big guy
+		{keys, vals, 3, 0, len(keys) - 1, false, false},
+		// even changing the power a little bit breaks the static validator
+		// the sigs are enough, but the validator hash is unknown
+		{keys, keys.ToValidators(20, 11), 4, 0, len(keys), false, true},
+	}
+
+	for _, tc := range cases {
+		check := tc.keys.GenCommit(chainID, tc.height, nil, tc.vals,
+			[]byte("bar"), tc.first, tc.last)
+		err := cert.Certify(check)
+		if tc.proper {
+			assert.Nil(err, "%+v", err)
+			assert.Equal(cert.LastHeight(), tc.height)
+		} else {
+			assert.NotNil(err)
+			if tc.changed {
+				assert.True(errors.IsValidatorsChangedErr(err), "%+v", err)
+			}
+		}
+	}
+}
+
+// TestDynamicUpdate makes sure we update safely and sanely
+func TestDynamicUpdate(t *testing.T) {
+	assert, require := assert.New(t), require.New(t)
+
+	chainID := "test-dyno-up"
+	keys := certifiers.GenValKeys(5)
+	vals := keys.ToValidators(20, 0)
+	cert := certifiers.NewDynamic(chainID, vals, 40)
+
+	// one valid block to give us a sense of time
+	h := 100
+	good := keys.GenCommit(chainID, h, nil, vals, []byte("foo"), 0, len(keys))
+	err := cert.Certify(good)
+	require.Nil(err, "%+v", err)
+
+	// some new sets to try later
+	keys2 := keys.Extend(2)
+	keys3 := keys2.Extend(4)
+
+	// we try to update with some blocks
+	cases := []struct {
+		keys        certifiers.ValKeys
+		vals        *types.ValidatorSet
+		height      int
+		first, last int  // who actually signs
+		proper      bool // true -> expect no error
+		changed     bool // true -> expect too much change error
+	}{
+		// same validator set, well signed, of course it is okay
+		{keys, vals, h + 10, 0, len(keys), true, false},
+		// same validator set, poorly signed, fails
+		{keys, vals, h + 20, 2, len(keys), false, false},
+
+		// shift the power a little, works if properly signed
+		{keys, keys.ToValidators(10, 0), h + 30, 1, len(keys), true, false},
+		// but not on a poor signature
+		{keys, keys.ToValidators(10, 0), h + 40, 2, len(keys), false, false},
+		// and not if it was in the past
+		{keys, keys.ToValidators(10, 0), h + 25, 0, len(keys), false, false},
+
+		// let's try to adjust to a whole new validator set (we have 5/7 of the votes)
+		{keys2, keys2.ToValidators(10, 0), h + 33, 0, len(keys2), true, false},
+
+		// properly signed but too much change, not allowed (only 7/11 validators known)
+		{keys3, keys3.ToValidators(10, 0), h + 50, 0, len(keys3), false, true},
+	}
+
+	for _, tc := range cases {
+		fc := tc.keys.GenFullCommit(chainID, tc.height, nil, tc.vals,
+			[]byte("bar"), tc.first, tc.last)
+		err := cert.Update(fc)
+		if tc.proper {
+			assert.Nil(err, "%d: %+v", tc.height, err)
+			// we update last seen height
+			assert.Equal(cert.LastHeight(), tc.height)
+			// and we update the proper validators
+			assert.EqualValues(fc.Header.ValidatorsHash, cert.Hash())
+		} else {
+			assert.NotNil(err, "%d", tc.height)
+			// we don't update the height
+			assert.NotEqual(cert.LastHeight(), tc.height)
+			if tc.changed {
+				assert.True(errors.IsTooMuchChangeErr(err),
+					"%d: %+v", tc.height, err)
+			}
+		}
+	}
+}

certifiers/errors/errors.go

@@ -0,0 +1,86 @@
+package errors
+
+import (
+	"fmt"
+
+	"github.com/pkg/errors"
+)
+
+var (
+	errValidatorsChanged = fmt.Errorf("Validators differ between header and certifier")
+	errCommitNotFound    = fmt.Errorf("Commit not found by provider")
+	errTooMuchChange     = fmt.Errorf("Validators change too much to safely update")
+	errPastTime          = fmt.Errorf("Update older than certifier height")
+	errNoPathFound       = fmt.Errorf("Cannot find a path of validators")
+)
+
+// IsCommitNotFoundErr checks whether an error is due to missing data
+func IsCommitNotFoundErr(err error) bool {
+	return err != nil && (errors.Cause(err) == errCommitNotFound)
+}
+
+func ErrCommitNotFound() error {
+	return errors.WithStack(errCommitNotFound)
+}
+
+// IsValidatorsChangedErr checks whether an error is due
+// to a differing validator set
+func IsValidatorsChangedErr(err error) bool {
+	return err != nil && (errors.Cause(err) == errValidatorsChanged)
+}
+
+func ErrValidatorsChanged() error {
+	return errors.WithStack(errValidatorsChanged)
+}
+
+// IsTooMuchChangeErr checks whether an error is due to too much change
+// between these validators sets
+func IsTooMuchChangeErr(err error) bool {
+	return err != nil && (errors.Cause(err) == errTooMuchChange)
+}
+
+func ErrTooMuchChange() error {
+	return errors.WithStack(errTooMuchChange)
+}
+
+func IsPastTimeErr(err error) bool {
+	return err != nil && (errors.Cause(err) == errPastTime)
+}
+
+func ErrPastTime() error {
+	return errors.WithStack(errPastTime)
+}
+
+// IsNoPathFoundErr checks whether an error is due to no path of
+// validators in provider from where we are to where we want to be
+func IsNoPathFoundErr(err error) bool {
+	return err != nil && (errors.Cause(err) == errNoPathFound)
+}
+
+func ErrNoPathFound() error {
+	return errors.WithStack(errNoPathFound)
+}
+
+//--------------------------------------------
+
+type errHeightMismatch struct {
+	h1, h2 int
+}
+
+func (e errHeightMismatch) Error() string {
+	return fmt.Sprintf("Blocks don't match - %d vs %d", e.h1, e.h2)
+}
+
+// IsHeightMismatchErr checks whether an error is due to data from different blocks
+func IsHeightMismatchErr(err error) bool {
+	if err == nil {
+		return false
+	}
+	_, ok := errors.Cause(err).(errHeightMismatch)
+	return ok
+}
+
+// ErrHeightMismatch returns an mismatch error with stack-trace
+func ErrHeightMismatch(h1, h2 int) error {
+	return errors.WithStack(errHeightMismatch{h1, h2})
+}

certifiers/errors/errors_test.go

@@ -0,0 +1,18 @@
+package errors
+
+import (
+	"errors"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+)
+
+func TestErrorHeight(t *testing.T) {
+	e1 := ErrHeightMismatch(2, 3)
+	e1.Error()
+	assert.True(t, IsHeightMismatchErr(e1))
+
+	e2 := errors.New("foobar")
+	assert.False(t, IsHeightMismatchErr(e2))
+	assert.False(t, IsHeightMismatchErr(nil))
+}

certifiers/files/commit.go

@@ -0,0 +1,77 @@
+package files
+
+import (
+	"encoding/json"
+	"os"
+
+	"github.com/pkg/errors"
+
+	wire "github.com/tendermint/go-wire"
+
+	"github.com/tendermint/tendermint/certifiers"
+	certerr "github.com/tendermint/tendermint/certifiers/errors"
+)
+
+const (
+	// MaxFullCommitSize is the maximum number of bytes we will
+	// read in for a full commit to avoid excessive allocations
+	// in the deserializer
+	MaxFullCommitSize = 1024 * 1024
+)
+
+// SaveFullCommit exports the seed in binary / go-wire style
+func SaveFullCommit(fc certifiers.FullCommit, path string) error {
+	f, err := os.Create(path)
+	if err != nil {
+		return errors.WithStack(err)
+	}
+	defer f.Close()
+
+	var n int
+	wire.WriteBinary(fc, f, &n, &err)
+	return errors.WithStack(err)
+}
+
+// SaveFullCommitJSON exports the seed in a json format
+func SaveFullCommitJSON(fc certifiers.FullCommit, path string) error {
+	f, err := os.Create(path)
+	if err != nil {
+		return errors.WithStack(err)
+	}
+	defer f.Close()
+	stream := json.NewEncoder(f)
+	err = stream.Encode(fc)
+	return errors.WithStack(err)
+}
+
+func LoadFullCommit(path string) (certifiers.FullCommit, error) {
+	var fc certifiers.FullCommit
+	f, err := os.Open(path)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return fc, certerr.ErrCommitNotFound()
+		}
+		return fc, errors.WithStack(err)
+	}
+	defer f.Close()
+
+	var n int
+	wire.ReadBinaryPtr(&fc, f, MaxFullCommitSize, &n, &err)
+	return fc, errors.WithStack(err)
+}
+
+func LoadFullCommitJSON(path string) (certifiers.FullCommit, error) {
+	var fc certifiers.FullCommit
+	f, err := os.Open(path)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return fc, certerr.ErrCommitNotFound()
+		}
+		return fc, errors.WithStack(err)
+	}
+	defer f.Close()
+
+	stream := json.NewDecoder(f)
+	err = stream.Decode(&fc)
+	return fc, errors.WithStack(err)
+}

certifiers/files/commit_test.go

@@ -0,0 +1,66 @@
+package files
+
+import (
+	"os"
+	"path/filepath"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+
+	cmn "github.com/tendermint/tmlibs/common"
+
+	"github.com/tendermint/tendermint/certifiers"
+)
+
+func tmpFile() string {
+	suffix := cmn.RandStr(16)
+	return filepath.Join(os.TempDir(), "fc-test-"+suffix)
+}
+
+func TestSerializeFullCommits(t *testing.T) {
+	assert, require := assert.New(t), require.New(t)
+
+	// some constants
+	appHash := []byte("some crazy thing")
+	chainID := "ser-ial"
+	h := 25
+
+	// build a fc
+	keys := certifiers.GenValKeys(5)
+	vals := keys.ToValidators(10, 0)
+	fc := keys.GenFullCommit(chainID, h, nil, vals, appHash, 0, 5)
+
+	require.Equal(h, fc.Height())
+	require.Equal(vals.Hash(), fc.ValidatorsHash())
+
+	// try read/write with json
+	jfile := tmpFile()
+	defer os.Remove(jfile)
+	jseed, err := LoadFullCommitJSON(jfile)
+	assert.NotNil(err)
+	err = SaveFullCommitJSON(fc, jfile)
+	require.Nil(err)
+	jseed, err = LoadFullCommitJSON(jfile)
+	assert.Nil(err, "%+v", err)
+	assert.Equal(h, jseed.Height())
+	assert.Equal(vals.Hash(), jseed.ValidatorsHash())
+
+	// try read/write with binary
+	bfile := tmpFile()
+	defer os.Remove(bfile)
+	bseed, err := LoadFullCommit(bfile)
+	assert.NotNil(err)
+	err = SaveFullCommit(fc, bfile)
+	require.Nil(err)
+	bseed, err = LoadFullCommit(bfile)
+	assert.Nil(err, "%+v", err)
+	assert.Equal(h, bseed.Height())
+	assert.Equal(vals.Hash(), bseed.ValidatorsHash())
+
+	// make sure they don't read the other format (different)
+	_, err = LoadFullCommit(jfile)
+	assert.NotNil(err)
+	_, err = LoadFullCommitJSON(bfile)
+	assert.NotNil(err)
+}

certifiers/files/provider.go

@@ -0,0 +1,134 @@
+/*
+Package files defines a Provider that stores all data in the filesystem
+
+We assume the same validator hash may be reused by many different
+headers/Commits, and thus store it separately. This leaves us
+with three issues:
+
+  1. Given a validator hash, retrieve the validator set if previously stored
+  2. Given a block height, find the Commit with the highest height <= h
+  3. Given a FullCommit, store it quickly to satisfy 1 and 2
+
+Note that we do not worry about caching, as that can be achieved by
+pairing this with a MemStoreProvider and CacheProvider from certifiers
+*/
+package files
+
+import (
+	"encoding/hex"
+	"fmt"
+	"math"
+	"os"
+	"path/filepath"
+	"sort"
+
+	"github.com/pkg/errors"
+
+	"github.com/tendermint/tendermint/certifiers"
+	certerr "github.com/tendermint/tendermint/certifiers/errors"
+)
+
+const (
+	Ext      = ".tsd"
+	ValDir   = "validators"
+	CheckDir = "checkpoints"
+	dirPerm  = os.FileMode(0755)
+	filePerm = os.FileMode(0644)
+)
+
+type provider struct {
+	valDir   string
+	checkDir string
+}
+
+// NewProvider creates the parent dir and subdirs
+// for validators and checkpoints as needed
+func NewProvider(dir string) certifiers.Provider {
+	valDir := filepath.Join(dir, ValDir)
+	checkDir := filepath.Join(dir, CheckDir)
+	for _, d := range []string{valDir, checkDir} {
+		err := os.MkdirAll(d, dirPerm)
+		if err != nil {
+			panic(err)
+		}
+	}
+	return &provider{valDir: valDir, checkDir: checkDir}
+}
+
+func (p *provider) encodeHash(hash []byte) string {
+	return hex.EncodeToString(hash) + Ext
+}
+
+func (p *provider) encodeHeight(h int) string {
+	// pad up to 10^12 for height...
+	return fmt.Sprintf("%012d%s", h, Ext)
+}
+
+func (p *provider) StoreCommit(fc certifiers.FullCommit) error {
+	// make sure the fc is self-consistent before saving
+	err := fc.ValidateBasic(fc.Commit.Header.ChainID)
+	if err != nil {
+		return err
+	}
+
+	paths := []string{
+		filepath.Join(p.checkDir, p.encodeHeight(fc.Height())),
+		filepath.Join(p.valDir, p.encodeHash(fc.Header.ValidatorsHash)),
+	}
+	for _, path := range paths {
+		err := SaveFullCommit(fc, path)
+		// unknown error in creating or writing immediately breaks
+		if err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+func (p *provider) GetByHeight(h int) (certifiers.FullCommit, error) {
+	// first we look for exact match, then search...
+	path := filepath.Join(p.checkDir, p.encodeHeight(h))
+	fc, err := LoadFullCommit(path)
+	if certerr.IsCommitNotFoundErr(err) {
+		path, err = p.searchForHeight(h)
+		if err == nil {
+			fc, err = LoadFullCommit(path)
+		}
+	}
+	return fc, err
+}
+
+func (p *provider) LatestCommit() (fc certifiers.FullCommit, err error) {
+	// Note to future: please update by 2077 to avoid rollover
+	return p.GetByHeight(math.MaxInt32 - 1)
+}
+
+// search for height, looks for a file with highest height < h
+// return certifiers.ErrCommitNotFound() if not there...
+func (p *provider) searchForHeight(h int) (string, error) {
+	d, err := os.Open(p.checkDir)
+	if err != nil {
+		return "", errors.WithStack(err)
+	}
+	files, err := d.Readdirnames(0)
+
+	d.Close()
+	if err != nil {
+		return "", errors.WithStack(err)
+	}
+
+	desired := p.encodeHeight(h)
+	sort.Strings(files)
+	i := sort.SearchStrings(files, desired)
+	if i == 0 {
+		return "", certerr.ErrCommitNotFound()
+	}
+	found := files[i-1]
+	path := filepath.Join(p.checkDir, found)
+	return path, errors.WithStack(err)
+}
+
+func (p *provider) GetByHash(hash []byte) (certifiers.FullCommit, error) {
+	path := filepath.Join(p.valDir, p.encodeHash(hash))
+	return LoadFullCommit(path)
+}

certifiers/files/provider_test.go

@@ -0,0 +1,96 @@
+package files_test
+
+import (
+	"bytes"
+	"errors"
+	"io/ioutil"
+	"os"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+
+	"github.com/tendermint/tendermint/certifiers"
+	certerr "github.com/tendermint/tendermint/certifiers/errors"
+	"github.com/tendermint/tendermint/certifiers/files"
+)
+
+func checkEqual(stored, loaded certifiers.FullCommit, chainID string) error {
+	err := loaded.ValidateBasic(chainID)
+	if err != nil {
+		return err
+	}
+	if !bytes.Equal(stored.ValidatorsHash(), loaded.ValidatorsHash()) {
+		return errors.New("Different block hashes")
+	}
+	return nil
+}
+
+func TestFileProvider(t *testing.T) {
+	assert, require := assert.New(t), require.New(t)
+
+	dir, err := ioutil.TempDir("", "fileprovider-test")
+	assert.Nil(err)
+	defer os.RemoveAll(dir)
+	p := files.NewProvider(dir)
+
+	chainID := "test-files"
+	appHash := []byte("some-data")
+	keys := certifiers.GenValKeys(5)
+	count := 10
+
+	// make a bunch of seeds...
+	seeds := make([]certifiers.FullCommit, count)
+	for i := 0; i < count; i++ {
+		// two seeds for each validator, to check how we handle dups
+		// (10, 0), (10, 1), (10, 1), (10, 2), (10, 2), ...
+		vals := keys.ToValidators(10, int64(count/2))
+		h := 20 + 10*i
+		check := keys.GenCommit(chainID, h, nil, vals, appHash, 0, 5)
+		seeds[i] = certifiers.NewFullCommit(check, vals)
+	}
+
+	// check provider is empty
+	seed, err := p.GetByHeight(20)
+	require.NotNil(err)
+	assert.True(certerr.IsCommitNotFoundErr(err))
+
+	seed, err = p.GetByHash(seeds[3].ValidatorsHash())
+	require.NotNil(err)
+	assert.True(certerr.IsCommitNotFoundErr(err))
+
+	// now add them all to the provider
+	for _, s := range seeds {
+		err = p.StoreCommit(s)
+		require.Nil(err)
+		// and make sure we can get it back
+		s2, err := p.GetByHash(s.ValidatorsHash())
+		assert.Nil(err)
+		err = checkEqual(s, s2, chainID)
+		assert.Nil(err)
+		// by height as well
+		s2, err = p.GetByHeight(s.Height())
+		err = checkEqual(s, s2, chainID)
+		assert.Nil(err)
+	}
+
+	// make sure we get the last hash if we overstep
+	seed, err = p.GetByHeight(5000)
+	if assert.Nil(err, "%+v", err) {
+		assert.Equal(seeds[count-1].Height(), seed.Height())
+		err = checkEqual(seeds[count-1], seed, chainID)
+		assert.Nil(err)
+	}
+
+	// and middle ones as well
+	seed, err = p.GetByHeight(47)
+	if assert.Nil(err, "%+v", err) {
+		// we only step by 10, so 40 must be the one below this
+		assert.Equal(40, seed.Height())
+	}
+
+	// and proper error for too low
+	_, err = p.GetByHeight(5)
+	assert.NotNil(err)
+	assert.True(certerr.IsCommitNotFoundErr(err))
+}

certifiers/helper.go

@@ -0,0 +1,147 @@
+package certifiers
+
+import (
+	"time"
+
+	crypto "github.com/tendermint/go-crypto"
+
+	"github.com/tendermint/tendermint/types"
+)
+
+// ValKeys is a helper for testing.
+//
+// It lets us simulate signing with many keys, either ed25519 or secp256k1.
+// The main use case is to create a set, and call GenCommit
+// to get propely signed header for testing.
+//
+// You can set different weights of validators each time you call
+// ToValidators, and can optionally extend the validator set later
+// with Extend or ExtendSecp
+type ValKeys []crypto.PrivKey
+
+// GenValKeys produces an array of private keys to generate commits
+func GenValKeys(n int) ValKeys {
+	res := make(ValKeys, n)
+	for i := range res {
+		res[i] = crypto.GenPrivKeyEd25519().Wrap()
+	}
+	return res
+}
+
+// Change replaces the key at index i
+func (v ValKeys) Change(i int) ValKeys {
+	res := make(ValKeys, len(v))
+	copy(res, v)
+	res[i] = crypto.GenPrivKeyEd25519().Wrap()
+	return res
+}
+
+// Extend adds n more keys (to remove, just take a slice)
+func (v ValKeys) Extend(n int) ValKeys {
+	extra := GenValKeys(n)
+	return append(v, extra...)
+}
+
+// GenSecpValKeys produces an array of secp256k1 private keys to generate commits
+func GenSecpValKeys(n int) ValKeys {
+	res := make(ValKeys, n)
+	for i := range res {
+		res[i] = crypto.GenPrivKeySecp256k1().Wrap()
+	}
+	return res
+}
+
+// ExtendSecp adds n more secp256k1 keys (to remove, just take a slice)
+func (v ValKeys) ExtendSecp(n int) ValKeys {
+	extra := GenSecpValKeys(n)
+	return append(v, extra...)
+}
+
+// ToValidators produces a list of validators from the set of keys
+// The first key has weight `init` and it increases by `inc` every step
+// so we can have all the same weight, or a simple linear distribution
+// (should be enough for testing)
+func (v ValKeys) ToValidators(init, inc int64) *types.ValidatorSet {
+	res := make([]*types.Validator, len(v))
+	for i, k := range v {
+		res[i] = types.NewValidator(k.PubKey(), init+int64(i)*inc)
+	}
+	return types.NewValidatorSet(res)
+}
+
+// signHeader properly signs the header with all keys from first to last exclusive
+func (v ValKeys) signHeader(header *types.Header, first, last int) *types.Commit {
+	votes := make([]*types.Vote, len(v))
+
+	// we need this list to keep the ordering...
+	vset := v.ToValidators(1, 0)
+
+	// fill in the votes we want
+	for i := first; i < last; i++ {
+		vote := makeVote(header, vset, v[i])
+		votes[vote.ValidatorIndex] = vote
+	}
+
+	res := &types.Commit{
+		BlockID:    types.BlockID{Hash: header.Hash()},
+		Precommits: votes,
+	}
+	return res
+}
+
+func makeVote(header *types.Header, vals *types.ValidatorSet, key crypto.PrivKey) *types.Vote {
+	addr := key.PubKey().Address()
+	idx, _ := vals.GetByAddress(addr)
+	vote := &types.Vote{
+		ValidatorAddress: addr,
+		ValidatorIndex:   idx,
+		Height:           header.Height,
+		Round:            1,
+		Type:             types.VoteTypePrecommit,
+		BlockID:          types.BlockID{Hash: header.Hash()},
+	}
+	// Sign it
+	signBytes := types.SignBytes(header.ChainID, vote)
+	vote.Signature = key.Sign(signBytes)
+	return vote
+}
+
+func genHeader(chainID string, height int, txs types.Txs,
+	vals *types.ValidatorSet, appHash []byte) *types.Header {
+
+	return &types.Header{
+		ChainID: chainID,
+		Height:  height,
+		Time:    time.Now(),
+		NumTxs:  len(txs),
+		// LastBlockID
+		// LastCommitHash
+		ValidatorsHash: vals.Hash(),
+		DataHash:       txs.Hash(),
+		AppHash:        appHash,
+	}
+}
+
+// GenCommit calls genHeader and signHeader and combines them into a Commit
+func (v ValKeys) GenCommit(chainID string, height int, txs types.Txs,
+	vals *types.ValidatorSet, appHash []byte, first, last int) Commit {
+
+	header := genHeader(chainID, height, txs, vals, appHash)
+	check := Commit{
+		Header: header,
+		Commit: v.signHeader(header, first, last),
+	}
+	return check
+}
+
+// GenFullCommit calls genHeader and signHeader and combines them into a Commit
+func (v ValKeys) GenFullCommit(chainID string, height int, txs types.Txs,
+	vals *types.ValidatorSet, appHash []byte, first, last int) FullCommit {
+
+	header := genHeader(chainID, height, txs, vals, appHash)
+	commit := Commit{
+		Header: header,
+		Commit: v.signHeader(header, first, last),
+	}
+	return NewFullCommit(commit, vals)
+}

certifiers/inquirer.go

@@ -0,0 +1,142 @@
+package certifiers
+
+import (
+	"github.com/tendermint/tendermint/types"
+
+	certerr "github.com/tendermint/tendermint/certifiers/errors"
+)
+
+type Inquiring struct {
+	cert *Dynamic
+	// These are only properly validated data, from local system
+	trusted Provider
+	// This is a source of new info, like a node rpc, or other import method
+	Source Provider
+}
+
+func NewInquiring(chainID string, fc FullCommit, trusted Provider, source Provider) *Inquiring {
+	// store the data in trusted
+	trusted.StoreCommit(fc)
+
+	return &Inquiring{
+		cert:    NewDynamic(chainID, fc.Validators, fc.Height()),
+		trusted: trusted,
+		Source:  source,
+	}
+}
+
+func (c *Inquiring) ChainID() string {
+	return c.cert.ChainID()
+}
+
+func (c *Inquiring) Validators() *types.ValidatorSet {
+	return c.cert.cert.vSet
+}
+
+func (c *Inquiring) LastHeight() int {
+	return c.cert.lastHeight
+}
+
+// Certify makes sure this is checkpoint is valid.
+//
+// If the validators have changed since the last know time, it looks
+// for a path to prove the new validators.
+//
+// On success, it will store the checkpoint in the store for later viewing
+func (c *Inquiring) Certify(commit Commit) error {
+	err := c.useClosestTrust(commit.Height())
+	if err != nil {
+		return err
+	}
+
+	err = c.cert.Certify(commit)
+	if !certerr.IsValidatorsChangedErr(err) {
+		return err
+	}
+	err = c.updateToHash(commit.Header.ValidatorsHash)
+	if err != nil {
+		return err
+	}
+
+	err = c.cert.Certify(commit)
+	if err != nil {
+		return err
+	}
+
+	// store the new checkpoint
+	c.trusted.StoreCommit(
+		NewFullCommit(commit, c.Validators()))
+	return nil
+}
+
+func (c *Inquiring) Update(fc FullCommit) error {
+	err := c.useClosestTrust(fc.Height())
+	if err != nil {
+		return err
+	}
+
+	err = c.cert.Update(fc)
+	if err == nil {
+		c.trusted.StoreCommit(fc)
+	}
+	return err
+}
+
+func (c *Inquiring) useClosestTrust(h int) error {
+	closest, err := c.trusted.GetByHeight(h)
+	if err != nil {
+		return err
+	}
+
+	// if the best seed is not the one we currently use,
+	// let's just reset the dynamic validator
+	if closest.Height() != c.LastHeight() {
+		c.cert = NewDynamic(c.ChainID(), closest.Validators, closest.Height())
+	}
+	return nil
+}
+
+// updateToHash gets the validator hash we want to update to
+// if IsTooMuchChangeErr, we try to find a path by binary search over height
+func (c *Inquiring) updateToHash(vhash []byte) error {
+	// try to get the match, and update
+	fc, err := c.Source.GetByHash(vhash)
+	if err != nil {
+		return err
+	}
+	err = c.cert.Update(fc)
+	// handle IsTooMuchChangeErr by using divide and conquer
+	if certerr.IsTooMuchChangeErr(err) {
+		err = c.updateToHeight(fc.Height())
+	}
+	return err
+}
+
+// updateToHeight will use divide-and-conquer to find a path to h
+func (c *Inquiring) updateToHeight(h int) error {
+	// try to update to this height (with checks)
+	fc, err := c.Source.GetByHeight(h)
+	if err != nil {
+		return err
+	}
+	start, end := c.LastHeight(), fc.Height()
+	if end <= start {
+		return certerr.ErrNoPathFound()
+	}
+	err = c.Update(fc)
+
+	// we can handle IsTooMuchChangeErr specially
+	if !certerr.IsTooMuchChangeErr(err) {
+		return err
+	}
+
+	// try to update to mid
+	mid := (start + end) / 2
+	err = c.updateToHeight(mid)
+	if err != nil {
+		return err
+	}
+
+	// if we made it to mid, we recurse
+	return c.updateToHeight(h)
+}

certifiers/inquirer_test.go

@@ -0,0 +1,165 @@
+package certifiers_test
+
+import (
+	"fmt"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+
+	"github.com/tendermint/tendermint/certifiers"
+)
+
+func TestInquirerValidPath(t *testing.T) {
+	assert, require := assert.New(t), require.New(t)
+	trust := certifiers.NewMemStoreProvider()
+	source := certifiers.NewMemStoreProvider()
+
+	// set up the validators to generate test blocks
+	var vote int64 = 10
+	keys := certifiers.GenValKeys(5)
+	vals := keys.ToValidators(vote, 0)
+
+	// construct a bunch of commits, each with one more height than the last
+	chainID := "inquiry-test"
+	count := 50
+	commits := make([]certifiers.FullCommit, count)
+	for i := 0; i < count; i++ {
+		// extend the keys by 1 each time
+		keys = keys.Extend(1)
+		vals = keys.ToValidators(vote, 0)
+		h := 20 + 10*i
+		appHash := []byte(fmt.Sprintf("h=%d", h))
+		commits[i] = keys.GenFullCommit(chainID, h, nil, vals, appHash, 0, len(keys))
+	}
+
+	// initialize a certifier with the initial state
+	cert := certifiers.NewInquiring(chainID, commits[0], trust, source)
+
+	// this should fail validation....
+	commit := commits[count-1].Commit
+	err := cert.Certify(commit)
+	require.NotNil(err)
+
+	// add a few seed in the middle should be insufficient
+	for i := 10; i < 13; i++ {
+		err := source.StoreCommit(commits[i])
+		require.Nil(err)
+	}
+	err = cert.Certify(commit)
+	assert.NotNil(err)
+
+	// with more info, we succeed
+	for i := 0; i < count; i++ {
+		err := source.StoreCommit(commits[i])
+		require.Nil(err)
+	}
+	err = cert.Certify(commit)
+	assert.Nil(err, "%+v", err)
+}
+
+func TestInquirerMinimalPath(t *testing.T) {
+	assert, require := assert.New(t), require.New(t)
+	trust := certifiers.NewMemStoreProvider()
+	source := certifiers.NewMemStoreProvider()
+
+	// set up the validators to generate test blocks
+	var vote int64 = 10
+	keys := certifiers.GenValKeys(5)
+	vals := keys.ToValidators(vote, 0)
+
+	// construct a bunch of commits, each with one more height than the last
+	chainID := "minimal-path"
+	count := 12
+	commits := make([]certifiers.FullCommit, count)
+	for i := 0; i < count; i++ {
+		// extend the validators, so we are just below 2/3
+		keys = keys.Extend(len(keys)/2 - 1)
+		vals = keys.ToValidators(vote, 0)
+		h := 5 + 10*i
+		appHash := []byte(fmt.Sprintf("h=%d", h))
+		commits[i] = keys.GenFullCommit(chainID, h, nil, vals, appHash, 0, len(keys))
+	}
+
+	// initialize a certifier with the initial state
+	cert := certifiers.NewInquiring(chainID, commits[0], trust, source)
+
+	// this should fail validation....
+	commit := commits[count-1].Commit
+	err := cert.Certify(commit)
+	require.NotNil(err)
+
+	// add a few seed in the middle should be insufficient
+	for i := 5; i < 8; i++ {
+		err := source.StoreCommit(commits[i])
+		require.Nil(err)
+	}
+	err = cert.Certify(commit)
+	assert.NotNil(err)
+
+	// with more info, we succeed
+	for i := 0; i < count; i++ {
+		err := source.StoreCommit(commits[i])
+		require.Nil(err)
+	}
+	err = cert.Certify(commit)
+	assert.Nil(err, "%+v", err)
+}
+
+func TestInquirerVerifyHistorical(t *testing.T) {
+	assert, require := assert.New(t), require.New(t)
+	trust := certifiers.NewMemStoreProvider()
+	source := certifiers.NewMemStoreProvider()
+
+	// set up the validators to generate test blocks
+	var vote int64 = 10
+	keys := certifiers.GenValKeys(5)
+	vals := keys.ToValidators(vote, 0)
+
+	// construct a bunch of commits, each with one more height than the last
+	chainID := "inquiry-test"
+	count := 10
+	commits := make([]certifiers.FullCommit, count)
+	for i := 0; i < count; i++ {
+		// extend the keys by 1 each time
+		keys = keys.Extend(1)
+		vals = keys.ToValidators(vote, 0)
+		h := 20 + 10*i
+		appHash := []byte(fmt.Sprintf("h=%d", h))
+		commits[i] = keys.GenFullCommit(chainID, h, nil, vals, appHash, 0, len(keys))
+	}
+
+	// initialize a certifier with the initial state
+	cert := certifiers.NewInquiring(chainID, commits[0], trust, source)
+
+	// store a few commits as trust
+	for _, i := range []int{2, 5} {
+		trust.StoreCommit(commits[i])
+	}
+
+	// let's see if we can jump forward using trusted commits
+	err := source.StoreCommit(commits[7])
+	require.Nil(err, "%+v", err)
+	check := commits[7].Commit
+	err = cert.Certify(check)
+	require.Nil(err, "%+v", err)
+	assert.Equal(check.Height(), cert.LastHeight())
+
+	// add access to all commits via untrusted source
+	for i := 0; i < count; i++ {
+		err := source.StoreCommit(commits[i])
+		require.Nil(err)
+	}
+
+	// try to check an unknown seed in the past
+	mid := commits[3].Commit
+	err = cert.Certify(mid)
+	require.Nil(err, "%+v", err)
+	assert.Equal(mid.Height(), cert.LastHeight())
+
+	// and jump all the way forward again
+	end := commits[count-1].Commit
+	err = cert.Certify(end)
+	require.Nil(err, "%+v", err)
+	assert.Equal(end.Height(), cert.LastHeight())
+}

certifiers/memprovider.go

@@ -0,0 +1,78 @@
+package certifiers
+
+import (
+	"encoding/hex"
+	"sort"
+
+	certerr "github.com/tendermint/tendermint/certifiers/errors"
+)
+
+type memStoreProvider struct {
+	// byHeight is always sorted by Height... need to support range search (nil, h]
+	// btree would be more efficient for larger sets
+	byHeight fullCommits
+	byHash   map[string]FullCommit
+}
+
+// fullCommits just exists to allow easy sorting
+type fullCommits []FullCommit
+
+func (s fullCommits) Len() int      { return len(s) }
+func (s fullCommits) Swap(i, j int) { s[i], s[j] = s[j], s[i] }
+func (s fullCommits) Less(i, j int) bool {
+	return s[i].Height() < s[j].Height()
+}
+
+func NewMemStoreProvider() Provider {
+	return &memStoreProvider{
+		byHeight: fullCommits{},
+		byHash:   map[string]FullCommit{},
+	}
+}
+
+func (m *memStoreProvider) encodeHash(hash []byte) string {
+	return hex.EncodeToString(hash)
+}
+
+func (m *memStoreProvider) StoreCommit(fc FullCommit) error {
+	// make sure the fc is self-consistent before saving
+	err := fc.ValidateBasic(fc.Commit.Header.ChainID)
+	if err != nil {
+		return err
+	}
+
+	// store the valid fc
+	key := m.encodeHash(fc.ValidatorsHash())
+	m.byHash[key] = fc
+	m.byHeight = append(m.byHeight, fc)
+	sort.Sort(m.byHeight)
+	return nil
+}
+
+func (m *memStoreProvider) GetByHeight(h int) (FullCommit, error) {
+	// search from highest to lowest
+	for i := len(m.byHeight) - 1; i >= 0; i-- {
+		fc := m.byHeight[i]
+		if fc.Height() <= h {
+			return fc, nil
+		}
+	}
+	return FullCommit{}, certerr.ErrCommitNotFound()
+}
+
+func (m *memStoreProvider) GetByHash(hash []byte) (FullCommit, error) {
+	var err error
+	fc, ok := m.byHash[m.encodeHash(hash)]
+	if !ok {
+		err = certerr.ErrCommitNotFound()
+	}
+	return fc, err
+}
+
+func (m *memStoreProvider) LatestCommit() (FullCommit, error) {
+	l := len(m.byHeight)
+	if l == 0 {
+		return FullCommit{}, certerr.ErrCommitNotFound()
+	}
+	return m.byHeight[l-1], nil
+}

certifiers/performance_test.go

@@ -0,0 +1,116 @@
+package certifiers_test
+
+import (
+	"fmt"
+	"testing"
+
+	"github.com/tendermint/tendermint/certifiers"
+)
+
+func BenchmarkGenCommit20(b *testing.B) {
+	keys := certifiers.GenValKeys(20)
+	benchmarkGenCommit(b, keys)
+}
+
+func BenchmarkGenCommit100(b *testing.B) {
+	keys := certifiers.GenValKeys(100)
+	benchmarkGenCommit(b, keys)
+}
+
+func BenchmarkGenCommitSec20(b *testing.B) {
+	keys := certifiers.GenSecpValKeys(20)
+	benchmarkGenCommit(b, keys)
+}
+
+func BenchmarkGenCommitSec100(b *testing.B) {
+	keys := certifiers.GenSecpValKeys(100)
+	benchmarkGenCommit(b, keys)
+}
+
+func benchmarkGenCommit(b *testing.B, keys certifiers.ValKeys) {
+	chainID := fmt.Sprintf("bench-%d", len(keys))
+	vals := keys.ToValidators(20, 10)
+	for i := 0; i < b.N; i++ {
+		h := 1 + i
+		appHash := []byte(fmt.Sprintf("h=%d", h))
+		keys.GenCommit(chainID, h, nil, vals, appHash, 0, len(keys))
+	}
+}
+
+// this benchmarks generating one key
+func BenchmarkGenValKeys(b *testing.B) {
+	keys := certifiers.GenValKeys(20)
+	for i := 0; i < b.N; i++ {
+		keys = keys.Extend(1)
+	}
+}
+
+// this benchmarks generating one key
+func BenchmarkGenSecpValKeys(b *testing.B) {
+	keys := certifiers.GenSecpValKeys(20)
+	for i := 0; i < b.N; i++ {
+		keys = keys.Extend(1)
+	}
+}
+
+func BenchmarkToValidators20(b *testing.B) {
+	benchmarkToValidators(b, 20)
+}
+
+func BenchmarkToValidators100(b *testing.B) {
+	benchmarkToValidators(b, 100)
+}
+
+// this benchmarks constructing the validator set (.PubKey() * nodes)
+func benchmarkToValidators(b *testing.B, nodes int) {
+	keys := certifiers.GenValKeys(nodes)
+	for i := 1; i <= b.N; i++ {
+		keys.ToValidators(int64(2*i), int64(i))
+	}
+}
+
+func BenchmarkToValidatorsSec100(b *testing.B) {
+	benchmarkToValidatorsSec(b, 100)
+}
+
+// this benchmarks constructing the validator set (.PubKey() * nodes)
+func benchmarkToValidatorsSec(b *testing.B, nodes int) {
+	keys := certifiers.GenSecpValKeys(nodes)
+	for i := 1; i <= b.N; i++ {
+		keys.ToValidators(int64(2*i), int64(i))
+	}
+}
+
+func BenchmarkCertifyCommit20(b *testing.B) {
+	keys := certifiers.GenValKeys(20)
+	benchmarkCertifyCommit(b, keys)
+}
+
+func BenchmarkCertifyCommit100(b *testing.B) {
+	keys := certifiers.GenValKeys(100)
+	benchmarkCertifyCommit(b, keys)
+}
+
+func BenchmarkCertifyCommitSec20(b *testing.B) {
+	keys := certifiers.GenSecpValKeys(20)
+	benchmarkCertifyCommit(b, keys)
+}
+
+func BenchmarkCertifyCommitSec100(b *testing.B) {
+	keys := certifiers.GenSecpValKeys(100)
+	benchmarkCertifyCommit(b, keys)
+}
+
+func benchmarkCertifyCommit(b *testing.B, keys certifiers.ValKeys) {
+	chainID := "bench-certify"
+	vals := keys.ToValidators(20, 10)
+	cert := certifiers.NewStatic(chainID, vals)
+	check := keys.GenCommit(chainID, 123, nil, vals, []byte("foo"), 0, len(keys))
+	for i := 0; i < b.N; i++ {
+		err := cert.Certify(check)
+		if err != nil {
+			panic(err)
+		}
+	}
+
+}

certifiers/provider.go

@@ -0,0 +1,125 @@
+package certifiers
+
+import (
+	certerr "github.com/tendermint/tendermint/certifiers/errors"
+)
+
+// Provider is used to get more validators by other means
+//
+// Examples: MemProvider, files.Provider, client.Provider....
+type Provider interface {
+	// StoreCommit saves a FullCommit after we have verified it,
+	// so we can query for it later. Important for updating our
+	// store of trusted commits
+	StoreCommit(fc FullCommit) error
+	// GetByHeight returns the closest commit with height <= h
+	GetByHeight(h int) (FullCommit, error)
+	// GetByHash returns a commit exactly matching this validator hash
+	GetByHash(hash []byte) (FullCommit, error)
+	// LatestCommit returns the newest commit stored
+	LatestCommit() (FullCommit, error)
+}
+
+// cacheProvider allows you to place one or more caches in front of a source
+// Provider.  It runs through them in order until a match is found.
+// So you can keep a local cache, and check with the network if
+// no data is there.
+type cacheProvider struct {
+	Providers []Provider
+}
+
+func NewCacheProvider(providers ...Provider) Provider {
+	return cacheProvider{
+		Providers: providers,
+	}
+}
+
+// StoreCommit tries to add the seed to all providers.
+//
+// Aborts on first error it encounters (closest provider)
+func (c cacheProvider) StoreCommit(fc FullCommit) (err error) {
+	for _, p := range c.Providers {
+		err = p.StoreCommit(fc)
+		if err != nil {
+			break
+		}
+	}
+	return err
+}
+
+/*
+GetByHeight should return the closest possible match from all providers.
+
+The Cache is usually organized in order from cheapest call (memory)
+to most expensive calls (disk/network). However, since GetByHeight returns
+a FullCommit at h' <= h, if the memory has a seed at h-10, but the network would
+give us the exact match, a naive "stop at first non-error" would hide
+the actual desired results.
+
+Thus, we query each provider in order until we find an exact match
+or we finished querying them all.  If at least one returned a non-error,
+then this returns the best match (minimum h-h').
+*/
+func (c cacheProvider) GetByHeight(h int) (fc FullCommit, err error) {
+	for _, p := range c.Providers {
+		var tfc FullCommit
+		tfc, err = p.GetByHeight(h)
+		if err == nil {
+			if tfc.Height() > fc.Height() {
+				fc = tfc
+			}
+			if tfc.Height() == h {
+				break
+			}
+		}
+	}
+	// even if the last one had an error, if any was a match, this is good
+	if fc.Height() > 0 {
+		err = nil
+	}
+	return fc, err
+}
+
+func (c cacheProvider) GetByHash(hash []byte) (fc FullCommit, err error) {
+	for _, p := range c.Providers {
+		fc, err = p.GetByHash(hash)
+		if err == nil {
+			break
+		}
+	}
+	return fc, err
+}
+
+func (c cacheProvider) LatestCommit() (fc FullCommit, err error) {
+	for _, p := range c.Providers {
+		var tfc FullCommit
+		tfc, err = p.LatestCommit()
+		if err == nil && tfc.Height() > fc.Height() {
+			fc = tfc
+		}
+	}
+	// even if the last one had an error, if any was a match, this is good
+	if fc.Height() > 0 {
+		err = nil
+	}
+	return fc, err
+}
+
+// missingProvider doens't store anything, always a miss
+// Designed as a mock for testing
+type missingProvider struct{}
+
+func NewMissingProvider() Provider {
+	return missingProvider{}
+}
+
+func (missingProvider) StoreCommit(_ FullCommit) error { return nil }
+func (missingProvider) GetByHeight(_ int) (FullCommit, error) {
+	return FullCommit{}, certerr.ErrCommitNotFound()
+}
+func (missingProvider) GetByHash(_ []byte) (FullCommit, error) {
+	return FullCommit{}, certerr.ErrCommitNotFound()
+}
+func (missingProvider) LatestCommit() (FullCommit, error) {
+	return FullCommit{}, certerr.ErrCommitNotFound()
+}

certifiers/provider_test.go

@@ -0,0 +1,128 @@
+package certifiers_test
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+
+	"github.com/tendermint/tendermint/certifiers"
+	"github.com/tendermint/tendermint/certifiers/errors"
+)
+
+func TestMemProvider(t *testing.T) {
+	p := certifiers.NewMemStoreProvider()
+	checkProvider(t, p, "test-mem", "empty")
+}
+
+func TestCacheProvider(t *testing.T) {
+	p := certifiers.NewCacheProvider(
+		certifiers.NewMissingProvider(),
+		certifiers.NewMemStoreProvider(),
+		certifiers.NewMissingProvider(),
+	)
+	checkProvider(t, p, "test-cache", "kjfhekfhkewhgit")
+}
+
+func checkProvider(t *testing.T, p certifiers.Provider, chainID, app string) {
+	assert, require := assert.New(t), require.New(t)
+	appHash := []byte(app)
+	keys := certifiers.GenValKeys(5)
+	count := 10
+
+	// make a bunch of commits...
+	commits := make([]certifiers.FullCommit, count)
+	for i := 0; i < count; i++ {
+		// two commits for each validator, to check how we handle dups
+		// (10, 0), (10, 1), (10, 1), (10, 2), (10, 2), ...
+		vals := keys.ToValidators(10, int64(count/2))
+		h := 20 + 10*i
+		commits[i] = keys.GenFullCommit(chainID, h, nil, vals, appHash, 0, 5)
+	}
+
+	// check provider is empty
+	fc, err := p.GetByHeight(20)
+	require.NotNil(err)
+	assert.True(errors.IsCommitNotFoundErr(err))
+
+	fc, err = p.GetByHash(commits[3].ValidatorsHash())
+	require.NotNil(err)
+	assert.True(errors.IsCommitNotFoundErr(err))
+
+	// now add them all to the provider
+	for _, s := range commits {
+		err = p.StoreCommit(s)
+		require.Nil(err)
+		// and make sure we can get it back
+		s2, err := p.GetByHash(s.ValidatorsHash())
+		assert.Nil(err)
+		assert.Equal(s, s2)
+		// by height as well
+		s2, err = p.GetByHeight(s.Height())
+		assert.Nil(err)
+		assert.Equal(s, s2)
+	}
+
+	// make sure we get the last hash if we overstep
+	fc, err = p.GetByHeight(5000)
+	if assert.Nil(err) {
+		assert.Equal(commits[count-1].Height(), fc.Height())
+		assert.Equal(commits[count-1], fc)
+	}
+
+	// and middle ones as well
+	fc, err = p.GetByHeight(47)
+	if assert.Nil(err) {
+		// we only step by 10, so 40 must be the one below this
+		assert.Equal(40, fc.Height())
+	}
+
+}
+
+// this will make a get height, and if it is good, set the data as well
+func checkGetHeight(t *testing.T, p certifiers.Provider, ask, expect int) {
+	fc, err := p.GetByHeight(ask)
+	require.Nil(t, err, "%+v", err)
+	if assert.Equal(t, expect, fc.Height()) {
+		err = p.StoreCommit(fc)
+		require.Nil(t, err, "%+v", err)
+	}
+}
+
+func TestCacheGetsBestHeight(t *testing.T) {
+	// assert, require := assert.New(t), require.New(t)
+	require := require.New(t)
+
+	// we will write data to the second level of the cache (p2),
+	// and see what gets cached, stored in
+	p := certifiers.NewMemStoreProvider()
+	p2 := certifiers.NewMemStoreProvider()
+	cp := certifiers.NewCacheProvider(p, p2)
+
+	chainID := "cache-best-height"
+	appHash := []byte("01234567")
+	keys := certifiers.GenValKeys(5)
+	count := 10
+
+	// set a bunch of commits
+	for i := 0; i < count; i++ {
+		vals := keys.ToValidators(10, int64(count/2))
+		h := 10 * (i + 1)
+		fc := keys.GenFullCommit(chainID, h, nil, vals, appHash, 0, 5)
+		err := p2.StoreCommit(fc)
+		require.NoError(err)
+	}
+
+	// let's get a few heights from the cache and set them proper
+	checkGetHeight(t, cp, 57, 50)
+	checkGetHeight(t, cp, 33, 30)
+
+	// make sure they are set in p as well (but nothing else)
+	checkGetHeight(t, p, 44, 30)
+	checkGetHeight(t, p, 50, 50)
+	checkGetHeight(t, p, 99, 50)
+
+	// now, query the cache for a higher value
+	checkGetHeight(t, p2, 99, 90)
+	checkGetHeight(t, cp, 99, 90)
+}

certifiers/static.go

@@ -0,0 +1,66 @@
+package certifiers
+
+import (
+	"bytes"
+
+	"github.com/pkg/errors"
+
+	"github.com/tendermint/tendermint/types"
+
+	certerr "github.com/tendermint/tendermint/certifiers/errors"
+)
+
+var _ Certifier = &Static{}
+
+// Static assumes a static set of validators, set on
+// initilization and checks against them.
+// The signatures on every header is checked for > 2/3 votes
+// against the known validator set upon Certify
+//
+// Good for testing or really simple chains.  Building block
+// to support real-world functionality.
+type Static struct {
+	chainID string
+	vSet    *types.ValidatorSet
+	vhash   []byte
+}
+
+func NewStatic(chainID string, vals *types.ValidatorSet) *Static {
+	return &Static{
+		chainID: chainID,
+		vSet:    vals,
+	}
+}
+
+func (c *Static) ChainID() string {
+	return c.chainID
+}
+
+func (c *Static) Validators() *types.ValidatorSet {
+	return c.vSet
+}
+
+func (c *Static) Hash() []byte {
+	if len(c.vhash) == 0 {
+		c.vhash = c.vSet.Hash()
+	}
+	return c.vhash
+}
+
+func (c *Static) Certify(commit Commit) error {
+	// do basic sanity checks
+	err := commit.ValidateBasic(c.chainID)
+	if err != nil {
+		return err
+	}
+
+	// make sure it has the same validator set we have (static means static)
+	if !bytes.Equal(c.Hash(), commit.Header.ValidatorsHash) {
+		return certerr.ErrValidatorsChanged()
+	}
+
+	// then make sure we have the proper signatures for this
+	err = c.vSet.VerifyCommit(c.chainID, commit.Commit.BlockID,
+		commit.Header.Height, commit.Commit)
+	return errors.WithStack(err)
+}

certifiers/static_test.go

@@ -0,0 +1,59 @@
+package certifiers_test
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+
+	"github.com/tendermint/tendermint/types"
+
+	"github.com/tendermint/tendermint/certifiers"
+	errors "github.com/tendermint/tendermint/certifiers/errors"
+)
+
+func TestStaticCert(t *testing.T) {
+	// assert, require := assert.New(t), require.New(t)
+	assert := assert.New(t)
+	// require := require.New(t)
+
+	keys := certifiers.GenValKeys(4)
+	// 20, 30, 40, 50 - the first 3 don't have 2/3, the last 3 do!
+	vals := keys.ToValidators(20, 10)
+	// and a certifier based on our known set
+	chainID := "test-static"
+	cert := certifiers.NewStatic(chainID, vals)
+
+	cases := []struct {
+		keys        certifiers.ValKeys
+		vals        *types.ValidatorSet
+		height      int
+		first, last int  // who actually signs
+		proper      bool // true -> expect no error
+		changed     bool // true -> expect validator change error
+	}{
+		// perfect, signed by everyone
+		{keys, vals, 1, 0, len(keys), true, false},
+		// skip little guy is okay
+		{keys, vals, 2, 1, len(keys), true, false},
+		// but not the big guy
+		{keys, vals, 3, 0, len(keys) - 1, false, false},
+		// even changing the power a little bit breaks the static validator
+		// the sigs are enough, but the validator hash is unknown
+		{keys, keys.ToValidators(20, 11), 4, 0, len(keys), false, true},
+	}
+
+	for _, tc := range cases {
+		check := tc.keys.GenCommit(chainID, tc.height, nil, tc.vals,
+			[]byte("foo"), tc.first, tc.last)
+		err := cert.Certify(check)
+		if tc.proper {
+			assert.Nil(err, "%+v", err)
+		} else {
+			assert.NotNil(err)
+			if tc.changed {
+				assert.True(errors.IsValidatorsChangedErr(err), "%+v", err)
+			}
+		}
+	}
+
+}

circle.yml

@@ -1,34 +1,33 @@
+---
 machine:
   environment:
     MACH_PREFIX: tendermint-test-mach
-    GOPATH: /home/ubuntu/.go_workspace
-    REPO: $GOPATH/src/github.com/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME
     DOCKER_VERSION: 1.10.0
-    DOCKER_MACHINE_VERSION: 0.6.0
+    DOCKER_MACHINE_VERSION: 0.9.0
+    GOPATH: "$HOME/.go_project"
+    PROJECT_PARENT_PATH: "$GOPATH/src/github.com/$CIRCLE_PROJECT_USERNAME"
+    PROJECT_PATH: "$PROJECT_PARENT_PATH/$CIRCLE_PROJECT_REPONAME"
   hosts:
-    circlehost: 127.0.0.1
     localhost: 127.0.0.1
 
-checkout:
-  post:
-    - rm -rf $REPO
-    - mkdir -p $HOME/.go_workspace/src/github.com/$CIRCLE_PROJECT_USERNAME
-    - mv $HOME/$CIRCLE_PROJECT_REPONAME $REPO
-    # - git submodule sync
-    # - git submodule update --init # use submodules
-
 dependencies:
   override:
-    - echo $MACH_PREFIX $GOPATH $REPO $DOCKER_VERSION $DOCKER_MACHINE_VERSION
     - curl -sSL https://s3.amazonaws.com/circle-downloads/install-circleci-docker.sh | sudo bash -s -- $DOCKER_VERSION
-    - sudo curl -sSL -o /usr/bin/docker-machine https://github.com/docker/machine/releases/download/v$DOCKER_MACHINE_VERSION/docker-machine-linux-x86_64; sudo chmod 0755 /usr/bin/docker-machine
     - sudo start docker
+    - sudo curl -sSL -o /usr/bin/docker-machine "https://github.com/docker/machine/releases/download/v$DOCKER_MACHINE_VERSION/docker-machine-`uname -s`-`uname -m`"; sudo chmod 0755 /usr/bin/docker-machine
+    - mkdir -p "$PROJECT_PARENT_PATH"
+    - ln -sf "$HOME/$CIRCLE_PROJECT_REPONAME/" "$PROJECT_PATH"
+  post:
     - go version
     - docker version
     - docker-machine version
 
 test:
   override:
-    - "cd $REPO && make test_integrations"
+    - cd "$PROJECT_PATH" && set -o pipefail && make test_integrations 2>&1 | tee test_integrations.log:
+        timeout: 1800
   post:
-    - bash <(curl -s https://codecov.io/bash)
+    - cd "$PROJECT_PATH" && mv test_integrations.log "${CIRCLE_ARTIFACTS}"
+    - cd "$PROJECT_PATH" && bash <(curl -s https://codecov.io/bash) -f coverage.txt
+    - cd "$PROJECT_PATH" && mv coverage.txt "${CIRCLE_ARTIFACTS}"
+    - cd "$PROJECT_PATH" && cp test/logs/messages "${CIRCLE_ARTIFACTS}/docker_logs.txt"

cmd/tendermint/commands/gen_validator.go

@@ -0,0 +1,25 @@
+package commands
+
+import (
+	"encoding/json"
+	"fmt"
+
+	"github.com/spf13/cobra"
+
+	"github.com/tendermint/tendermint/types"
+)
+
+// GenValidatorCmd allows the generation of a keypair for a
+// validator.
+var GenValidatorCmd = &cobra.Command{
+	Use:   "gen_validator",
+	Short: "Generate new validator keypair",
+	Run:   genValidator,
+}
+
+func genValidator(cmd *cobra.Command, args []string) {
+	privValidator := types.GenPrivValidatorFS("")
+	privValidatorJSONBytes, _ := json.MarshalIndent(privValidator, "", "\t")
+	fmt.Printf(`%v
+`, string(privValidatorJSONBytes))
+}

cmd/tendermint/commands/init.go

@@ -0,0 +1,43 @@
+package commands
+
+import (
+	"os"
+
+	"github.com/spf13/cobra"
+
+	"github.com/tendermint/tendermint/types"
+	cmn "github.com/tendermint/tmlibs/common"
+)
+
+// InitFilesCmd initialises a fresh Tendermint Core instance.
+var InitFilesCmd = &cobra.Command{
+	Use:   "init",
+	Short: "Initialize Tendermint",
+	Run:   initFiles,
+}
+
+func initFiles(cmd *cobra.Command, args []string) {
+	privValFile := config.PrivValidatorFile()
+	if _, err := os.Stat(privValFile); os.IsNotExist(err) {
+		privValidator := types.GenPrivValidatorFS(privValFile)
+		privValidator.Save()
+
+		genFile := config.GenesisFile()
+
+		if _, err := os.Stat(genFile); os.IsNotExist(err) {
+			genDoc := types.GenesisDoc{
+				ChainID: cmn.Fmt("test-chain-%v", cmn.RandStr(6)),
+			}
+			genDoc.Validators = []types.GenesisValidator{types.GenesisValidator{
+				PubKey: privValidator.GetPubKey(),
+				Power:  10,
+			}}
+
+			genDoc.SaveAs(genFile)
+		}
+
+		logger.Info("Initialized tendermint", "genesis", config.GenesisFile(), "priv_validator", config.PrivValidatorFile())
+	} else {
+		logger.Info("Already initialized", "priv_validator", config.PrivValidatorFile())
+	}
+}

cmd/tendermint/commands/probe_upnp.go

@@ -0,0 +1,32 @@
+package commands
+
+import (
+	"encoding/json"
+	"fmt"
+
+	"github.com/spf13/cobra"
+
+	"github.com/tendermint/tendermint/p2p/upnp"
+)
+
+// ProbeUpnpCmd adds capabilities to test the UPnP functionality.
+var ProbeUpnpCmd = &cobra.Command{
+	Use:   "probe_upnp",
+	Short: "Test UPnP functionality",
+	RunE:  probeUpnp,
+}
+
+func probeUpnp(cmd *cobra.Command, args []string) error {
+	capabilities, err := upnp.Probe(logger)
+	if err != nil {
+		fmt.Println("Probe failed: %v", err)
+	} else {
+		fmt.Println("Probe success!")
+		jsonBytes, err := json.Marshal(capabilities)
+		if err != nil {
+			return err
+		}
+		fmt.Println(string(jsonBytes))
+	}
+	return nil
+}

cmd/tendermint/commands/replay.go

@@ -0,0 +1,26 @@
+package commands
+
+import (
+	"github.com/spf13/cobra"
+
+	"github.com/tendermint/tendermint/consensus"
+)
+
+// ReplayCmd allows replaying of messages from the WAL.
+var ReplayCmd = &cobra.Command{
+	Use:   "replay",
+	Short: "Replay messages from WAL",
+	Run: func(cmd *cobra.Command, args []string) {
+		consensus.RunReplayFile(config.BaseConfig, config.Consensus, false)
+	},
+}
+
+// ReplayConsoleCmd allows replaying of messages from the WAL in a
+// console.
+var ReplayConsoleCmd = &cobra.Command{
+	Use:   "replay_console",
+	Short: "Replay messages from WAL in a console",
+	Run: func(cmd *cobra.Command, args []string) {
+		consensus.RunReplayFile(config.BaseConfig, config.Consensus, true)
+	},
+}

cmd/tendermint/commands/reset_priv_validator.go

@@ -0,0 +1,58 @@
+package commands
+
+import (
+	"os"
+
+	"github.com/spf13/cobra"
+
+	"github.com/tendermint/tendermint/types"
+	"github.com/tendermint/tmlibs/log"
+)
+
+// ResetAllCmd removes the database of this Tendermint core
+// instance.
+var ResetAllCmd = &cobra.Command{
+	Use:   "unsafe_reset_all",
+	Short: "(unsafe) Remove all the data and WAL, reset this node's validator",
+	Run:   resetAll,
+}
+
+// ResetPrivValidatorCmd resets the private validator files.
+var ResetPrivValidatorCmd = &cobra.Command{
+	Use:   "unsafe_reset_priv_validator",
+	Short: "(unsafe) Reset this node's validator",
+	Run:   resetPrivValidator,
+}
+
+// ResetAll removes the privValidator files.
+// Exported so other CLI tools can use  it
+func ResetAll(dbDir, privValFile string, logger log.Logger) {
+	resetPrivValidatorFS(privValFile, logger)
+	os.RemoveAll(dbDir)
+	logger.Info("Removed all data", "dir", dbDir)
+}
+
+// XXX: this is totally unsafe.
+// it's only suitable for testnets.
+func resetAll(cmd *cobra.Command, args []string) {
+	ResetAll(config.DBDir(), config.PrivValidatorFile(), logger)
+}
+
+// XXX: this is totally unsafe.
+// it's only suitable for testnets.
+func resetPrivValidator(cmd *cobra.Command, args []string) {
+	resetPrivValidatorFS(config.PrivValidatorFile(), logger)
+}
+
+func resetPrivValidatorFS(privValFile string, logger log.Logger) {
+	// Get PrivValidator
+	if _, err := os.Stat(privValFile); err == nil {
+		privValidator := types.LoadPrivValidatorFS(privValFile)
+		privValidator.Reset()
+		logger.Info("Reset PrivValidator", "file", privValFile)
+	} else {
+		privValidator := types.GenPrivValidatorFS(privValFile)
+		privValidator.Save()
+		logger.Info("Generated PrivValidator", "file", privValFile)
+	}
+}

cmd/tendermint/commands/root.go

@@ -0,0 +1,58 @@
+package commands
+
+import (
+	"os"
+
+	"github.com/spf13/cobra"
+	"github.com/spf13/viper"
+
+	cfg "github.com/tendermint/tendermint/config"
+	"github.com/tendermint/tmlibs/cli"
+	tmflags "github.com/tendermint/tmlibs/cli/flags"
+	"github.com/tendermint/tmlibs/log"
+)
+
+var (
+	config = cfg.DefaultConfig()
+	logger = log.NewTMLogger(log.NewSyncWriter(os.Stdout)).With("module", "main")
+)
+
+func init() {
+	RootCmd.PersistentFlags().String("log_level", config.LogLevel, "Log level")
+}
+
+// ParseConfig retrieves the default environment configuration,
+// sets up the Tendermint root and ensures that the root exists
+func ParseConfig() (*cfg.Config, error) {
+	conf := cfg.DefaultConfig()
+	err := viper.Unmarshal(conf)
+	if err != nil {
+		return nil, err
+	}
+	conf.SetRoot(conf.RootDir)
+	cfg.EnsureRoot(conf.RootDir)
+	return conf, err
+}
+
+// RootCmd is the root command for Tendermint core.
+var RootCmd = &cobra.Command{
+	Use:   "tendermint",
+	Short: "Tendermint Core (BFT Consensus) in Go",
+	PersistentPreRunE: func(cmd *cobra.Command, args []string) (err error) {
+		if cmd.Name() == VersionCmd.Name() {
+			return nil
+		}
+		config, err = ParseConfig()
+		if err != nil {
+			return err
+		}
+		logger, err = tmflags.ParseLogLevel(config.LogLevel, logger, cfg.DefaultLogLevel())
+		if err != nil {
+			return err
+		}
+		if viper.GetBool(cli.TraceFlag) {
+			logger = log.NewTracingLogger(logger)
+		}
+		return nil
+	},
+}

cmd/tendermint/commands/root_test.go

@@ -0,0 +1,105 @@
+package commands
+
+import (
+	"os"
+	"strconv"
+	"testing"
+
+	"github.com/spf13/cobra"
+	"github.com/spf13/viper"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+
+	cfg "github.com/tendermint/tendermint/config"
+	"github.com/tendermint/tmlibs/cli"
+)
+
+var (
+	defaultRoot = os.ExpandEnv("$HOME/.some/test/dir")
+)
+
+const (
+	rootName = "root"
+)
+
+// isolate provides a clean setup and returns a copy of RootCmd you can
+// modify in the test cases.
+// NOTE: it unsets all TM* env variables.
+func isolate(cmds ...*cobra.Command) cli.Executable {
+	os.Unsetenv("TMHOME")
+	os.Unsetenv("TM_HOME")
+	os.Unsetenv("TMROOT")
+	os.Unsetenv("TM_ROOT")
+
+	viper.Reset()
+	config = cfg.DefaultConfig()
+	r := &cobra.Command{
+		Use:               rootName,
+		PersistentPreRunE: RootCmd.PersistentPreRunE,
+	}
+	r.AddCommand(cmds...)
+	wr := cli.PrepareBaseCmd(r, "TM", defaultRoot)
+	return wr
+}
+
+func TestRootConfig(t *testing.T) {
+	assert, require := assert.New(t), require.New(t)
+
+	// we pre-create a config file we can refer to in the rest of
+	// the test cases.
+	cvals := map[string]string{
+		"moniker":   "monkey",
+		"fast_sync": "false",
+	}
+	// proper types of the above settings
+	cfast := false
+	conf, err := cli.WriteDemoConfig(cvals)
+	require.Nil(err)
+
+	defaults := cfg.DefaultConfig()
+	dmax := defaults.P2P.MaxNumPeers
+
+	cases := []struct {
+		args     []string
+		env      map[string]string
+		root     string
+		moniker  string
+		fastSync bool
+		maxPeer  int
+	}{
+		{nil, nil, defaultRoot, defaults.Moniker, defaults.FastSync, dmax},
+		// try multiple ways of setting root (two flags, cli vs. env)
+		{[]string{"--home", conf}, nil, conf, cvals["moniker"], cfast, dmax},
+		{nil, map[string]string{"TMROOT": conf}, conf, cvals["moniker"], cfast, dmax},
+		// check setting p2p subflags two different ways
+		{[]string{"--p2p.max_num_peers", "420"}, nil, defaultRoot, defaults.Moniker, defaults.FastSync, 420},
+		{nil, map[string]string{"TM_P2P_MAX_NUM_PEERS": "17"}, defaultRoot, defaults.Moniker, defaults.FastSync, 17},
+		// try to set env that have no flags attached...
+		{[]string{"--home", conf}, map[string]string{"TM_MONIKER": "funny"}, conf, "funny", cfast, dmax},
+	}
+
+	for idx, tc := range cases {
+		i := strconv.Itoa(idx)
+		// test command that does nothing, except trigger unmarshalling in root
+		noop := &cobra.Command{
+			Use: "noop",
+			RunE: func(cmd *cobra.Command, args []string) error {
+				return nil
+			},
+		}
+		noop.Flags().Int("p2p.max_num_peers", defaults.P2P.MaxNumPeers, "")
+		cmd := isolate(noop)
+
+		args := append([]string{rootName, noop.Use}, tc.args...)
+		err := cli.RunWithArgs(cmd, args, tc.env)
+		require.Nil(err, i)
+		assert.Equal(tc.root, config.RootDir, i)
+		assert.Equal(tc.root, config.P2P.RootDir, i)
+		assert.Equal(tc.root, config.Consensus.RootDir, i)
+		assert.Equal(tc.root, config.Mempool.RootDir, i)
+		assert.Equal(tc.moniker, config.Moniker, i)
+		assert.Equal(tc.fastSync, config.FastSync, i)
+		assert.Equal(tc.maxPeer, config.P2P.MaxNumPeers, i)
+	}
+
+}

cmd/tendermint/commands/run_node.go

@@ -0,0 +1,67 @@
+package commands
+
+import (
+	"fmt"
+
+	"github.com/spf13/cobra"
+
+	nm "github.com/tendermint/tendermint/node"
+)
+
+// AddNodeFlags exposes some common configuration options on the command-line
+// These are exposed for convenience of commands embedding a tendermint node
+func AddNodeFlags(cmd *cobra.Command) {
+	// bind flags
+	cmd.Flags().String("moniker", config.Moniker, "Node Name")
+
+	// node flags
+	cmd.Flags().Bool("fast_sync", config.FastSync, "Fast blockchain syncing")
+
+	// abci flags
+	cmd.Flags().String("proxy_app", config.ProxyApp, "Proxy app address, or 'nilapp' or 'dummy' for local testing.")
+	cmd.Flags().String("abci", config.ABCI, "Specify abci transport (socket | grpc)")
+
+	// rpc flags
+	cmd.Flags().String("rpc.laddr", config.RPC.ListenAddress, "RPC listen address. Port required")
+	cmd.Flags().String("rpc.grpc_laddr", config.RPC.GRPCListenAddress, "GRPC listen address (BroadcastTx only). Port required")
+	cmd.Flags().Bool("rpc.unsafe", config.RPC.Unsafe, "Enabled unsafe rpc methods")
+
+	// p2p flags
+	cmd.Flags().String("p2p.laddr", config.P2P.ListenAddress, "Node listen address. (0.0.0.0:0 means any interface, any port)")
+	cmd.Flags().String("p2p.seeds", config.P2P.Seeds, "Comma delimited host:port seed nodes")
+	cmd.Flags().Bool("p2p.skip_upnp", config.P2P.SkipUPNP, "Skip UPNP configuration")
+	cmd.Flags().Bool("p2p.pex", config.P2P.PexReactor, "Enable Peer-Exchange (dev feature)")
+
+	// consensus flags
+	cmd.Flags().Bool("consensus.create_empty_blocks", config.Consensus.CreateEmptyBlocks, "Set this to false to only produce blocks when there are txs or when the AppHash changes")
+}
+
+// NewRunNodeCmd returns the command that allows the CLI to start a
+// node. It can be used with a custom PrivValidator and in-process ABCI application.
+func NewRunNodeCmd(nodeProvider nm.NodeProvider) *cobra.Command {
+	cmd := &cobra.Command{
+		Use:   "node",
+		Short: "Run the tendermint node",
+		RunE: func(cmd *cobra.Command, args []string) error {
+			// Create & start node
+			n, err := nodeProvider(config, logger)
+			if err != nil {
+				return fmt.Errorf("Failed to create node: %v", err)
+			}
+
+			if _, err := n.Start(); err != nil {
+				return fmt.Errorf("Failed to start node: %v", err)
+			} else {
+				logger.Info("Started node", "nodeInfo", n.Switch().NodeInfo())
+			}
+
+			// Trap signal, run forever.
+			n.RunForever()
+
+			return nil
+		},
+	}
+
+	AddNodeFlags(cmd)
+	return cmd
+}

cmd/tendermint/commands/show_validator.go

@@ -0,0 +1,23 @@
+package commands
+
+import (
+	"fmt"
+
+	"github.com/spf13/cobra"
+
+	"github.com/tendermint/go-wire/data"
+	"github.com/tendermint/tendermint/types"
+)
+
+// ShowValidatorCmd adds capabilities for showing the validator info.
+var ShowValidatorCmd = &cobra.Command{
+	Use:   "show_validator",
+	Short: "Show this node's validator info",
+	Run:   showValidator,
+}
+
+func showValidator(cmd *cobra.Command, args []string) {
+	privValidator := types.LoadOrGenPrivValidatorFS(config.PrivValidatorFile())
+	pubKeyJSONBytes, _ := data.ToJSON(privValidator.PubKey)
+	fmt.Println(string(pubKeyJSONBytes))
+}

cmd/tendermint/commands/testnet.go

@@ -0,0 +1,92 @@
+package commands
+
+import (
+	"fmt"
+	"path"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"github.com/tendermint/tendermint/types"
+	cmn "github.com/tendermint/tmlibs/common"
+)
+
+//flags
+var (
+	nValidators int
+	dataDir     string
+)
+
+func init() {
+	TestnetFilesCmd.Flags().IntVar(&nValidators, "n", 4,
+		"Number of validators to initialize the testnet with")
+	TestnetFilesCmd.Flags().StringVar(&dataDir, "dir", "mytestnet",
+		"Directory to store initialization data for the testnet")
+}
+
+// TestnetFilesCmd allows initialisation of files for a
+// Tendermint testnet.
+var TestnetFilesCmd = &cobra.Command{
+	Use:   "testnet",
+	Short: "Initialize files for a Tendermint testnet",
+	Run:   testnetFiles,
+}
+
+func testnetFiles(cmd *cobra.Command, args []string) {
+
+	genVals := make([]types.GenesisValidator, nValidators)
+
+	// Initialize core dir and priv_validator.json's
+	for i := 0; i < nValidators; i++ {
+		mach := cmn.Fmt("mach%d", i)
+		err := initMachCoreDirectory(dataDir, mach)
+		if err != nil {
+			cmn.Exit(err.Error())
+		}
+		// Read priv_validator.json to populate vals
+		privValFile := path.Join(dataDir, mach, "priv_validator.json")
+		privVal := types.LoadPrivValidatorFS(privValFile)
+		genVals[i] = types.GenesisValidator{
+			PubKey: privVal.GetPubKey(),
+			Power:  1,
+			Name:   mach,
+		}
+	}
+
+	// Generate genesis doc from generated validators
+	genDoc := &types.GenesisDoc{
+		GenesisTime: time.Now(),
+		ChainID:     "chain-" + cmn.RandStr(6),
+		Validators:  genVals,
+	}
+
+	// Write genesis file.
+	for i := 0; i < nValidators; i++ {
+		mach := cmn.Fmt("mach%d", i)
+		genDoc.SaveAs(path.Join(dataDir, mach, "genesis.json"))
+	}
+
+	fmt.Println(cmn.Fmt("Successfully initialized %v node directories", nValidators))
+}
+
+// Initialize per-machine core directory
+func initMachCoreDirectory(base, mach string) error {
+	dir := path.Join(base, mach)
+	err := cmn.EnsureDir(dir, 0777)
+	if err != nil {
+		return err
+	}
+
+	// Create priv_validator.json file if not present
+	ensurePrivValidator(path.Join(dir, "priv_validator.json"))
+	return nil
+
+}
+
+func ensurePrivValidator(file string) {
+	if cmn.FileExists(file) {
+		return
+	}
+	privValidator := types.GenPrivValidatorFS(file)
+	privValidator.Save()
+}

cmd/tendermint/commands/version.go

@@ -0,0 +1,18 @@
+package commands
+
+import (
+	"fmt"
+
+	"github.com/spf13/cobra"
+
+	"github.com/tendermint/tendermint/version"
+)
+
+// VersionCmd ...
+var VersionCmd = &cobra.Command{
+	Use:   "version",
+	Short: "Show version info",
+	Run: func(cmd *cobra.Command, args []string) {
+		fmt.Println(version.Version)
+	},
+}

cmd/tendermint/flags.go

@@ -1,53 +0,0 @@
-package main
-
-import (
-	flag "github.com/spf13/pflag"
-	"os"
-
-	cfg "github.com/tendermint/go-config"
-)
-
-func parseFlags(config cfg.Config, args []string) {
-	var (
-		printHelp     bool
-		moniker       string
-		nodeLaddr     string
-		seeds         string
-		fastSync      bool
-		skipUPNP      bool
-		rpcLaddr      string
-		logLevel      string
-		proxyApp      string
-		tmspTransport string
-	)
-
-	// Declare flags
-	var flags = flag.NewFlagSet("main", flag.ExitOnError)
-	flags.BoolVar(&printHelp, "help", false, "Print this help message.")
-	flags.StringVar(&moniker, "moniker", config.GetString("moniker"), "Node Name")
-	flags.StringVar(&nodeLaddr, "node_laddr", config.GetString("node_laddr"), "Node listen address. (0.0.0.0:0 means any interface, any port)")
-	flags.StringVar(&seeds, "seeds", config.GetString("seeds"), "Comma delimited host:port seed nodes")
-	flags.BoolVar(&fastSync, "fast_sync", config.GetBool("fast_sync"), "Fast blockchain syncing")
-	flags.BoolVar(&skipUPNP, "skip_upnp", config.GetBool("skip_upnp"), "Skip UPNP configuration")
-	flags.StringVar(&rpcLaddr, "rpc_laddr", config.GetString("rpc_laddr"), "RPC listen address. Port required")
-	flags.StringVar(&logLevel, "log_level", config.GetString("log_level"), "Log level")
-	flags.StringVar(&proxyApp, "proxy_app", config.GetString("proxy_app"),
-		"Proxy app address, or 'nilapp' or 'dummy' for local testing.")
-	flags.StringVar(&tmspTransport, "tmsp", config.GetString("tmsp"), "Specify tmsp transport (socket | grpc)")
-	flags.Parse(args)
-	if printHelp {
-		flags.PrintDefaults()
-		os.Exit(0)
-	}
-
-	// Merge parsed flag values onto app.
-	config.Set("moniker", moniker)
-	config.Set("node_laddr", nodeLaddr)
-	config.Set("seeds", seeds)
-	config.Set("fast_sync", fastSync)
-	config.Set("skip_upnp", skipUPNP)
-	config.Set("rpc_laddr", rpcLaddr)
-	config.Set("log_level", logLevel)
-	config.Set("proxy_app", proxyApp)
-	config.Set("tmsp", tmspTransport)
-}

cmd/tendermint/gen_validator.go

@@ -1,23 +0,0 @@
-package main
-
-import (
-	"fmt"
-
-	"github.com/tendermint/go-wire"
-	"github.com/tendermint/tendermint/types"
-)
-
-func gen_validator() {
-
-	privValidator := types.GenPrivValidator()
-	privValidatorJSONBytes := wire.JSONBytesPretty(privValidator)
-	fmt.Printf(`Generated a new validator!
-Paste the following JSON into your %v file
-
-%v
-
-`,
-		config.GetString("priv_validator_file"),
-		string(privValidatorJSONBytes),
-	)
-}

cmd/tendermint/init.go

@@ -1,23 +0,0 @@
-package main
-
-import (
-	. "github.com/tendermint/go-common"
-	"github.com/tendermint/tendermint/types"
-)
-
-func init_files() {
-	privValidator := types.GenPrivValidator()
-	privValidator.SetFile(config.GetString("priv_validator_file"))
-	privValidator.Save()
-
-	genDoc := types.GenesisDoc{
-		ChainID: Fmt("test-chain-%v", RandStr(6)),
-	}
-	genDoc.Validators = []types.GenesisValidator{types.GenesisValidator{
-		PubKey: privValidator.PubKey,
-		Amount: 10,
-	}}
-
-	genDoc.SaveAs(config.GetString("genesis_file"))
-
-}

cmd/tendermint/log.go

@@ -1,7 +0,0 @@
-package main
-
-import (
-	"github.com/tendermint/go-logger"
-)
-
-var log = logger.New("module", "main")

cmd/tendermint/main.go

@@ -1,66 +1,41 @@
 package main
 
 import (
-	"fmt"
 	"os"
 
-	. "github.com/tendermint/go-common"
-	cfg "github.com/tendermint/go-config"
-	"github.com/tendermint/go-logger"
-	tmcfg "github.com/tendermint/tendermint/config/tendermint"
-	"github.com/tendermint/tendermint/node"
-	"github.com/tendermint/tendermint/version"
+	"github.com/tendermint/tmlibs/cli"
+
+	cmd "github.com/tendermint/tendermint/cmd/tendermint/commands"
+	nm "github.com/tendermint/tendermint/node"
 )
 
-var config cfg.Config
-
 func main() {
+	rootCmd := cmd.RootCmd
+	rootCmd.AddCommand(
+		cmd.GenValidatorCmd,
+		cmd.InitFilesCmd,
+		cmd.ProbeUpnpCmd,
+		cmd.ReplayCmd,
+		cmd.ReplayConsoleCmd,
+		cmd.ResetAllCmd,
+		cmd.ResetPrivValidatorCmd,
+		cmd.ShowValidatorCmd,
+		cmd.TestnetFilesCmd,
+		cmd.VersionCmd)
 
-	args := os.Args[1:]
-	if len(args) == 0 {
-		fmt.Println(`Tendermint
+	// NOTE:
+	// Users wishing to:
+	//	* Use an external signer for their validators
+	//	* Supply an in-proc abci app
+	//	* Supply a genesis doc file from another source
+	//	* Provide their own DB implementation
+	// can copy this file and use something other than the
+	// DefaultNewNode function
+	nodeFunc := nm.DefaultNewNode
 
-Commands:
-    node            Run the tendermint node
-    show_validator  Show this node's validator info
-    gen_validator   Generate new validator keypair
-    probe_upnp      Test UPnP functionality
-    version         Show version info
-`)
-		return
-	}
+	// Create & start node
+	rootCmd.AddCommand(cmd.NewRunNodeCmd(nodeFunc))
 
-	// Get configuration
-	config = tmcfg.GetConfig("")
-	parseFlags(config, args[1:]) // Command line overrides
-
-	// set the log level
-	logger.SetLogLevel(config.GetString("log_level"))
-
-	switch args[0] {
-	case "node":
-		node.RunNode(config)
-	case "replay":
-		if len(args) > 1 && args[1] == "console" {
-			node.RunReplayConsole(config)
-		} else {
-			node.RunReplay(config)
-		}
-	case "init":
-		init_files()
-	case "show_validator":
-		show_validator()
-	case "gen_validator":
-		gen_validator()
-	case "probe_upnp":
-		probe_upnp()
-	case "unsafe_reset_all":
-		reset_all()
-	case "unsafe_reset_priv_validator":
-		reset_priv_validator()
-	case "version":
-		fmt.Println(version.Version)
-	default:
-		Exit(Fmt("Unknown command %v\n", args[0]))
-	}
+	cmd := cli.PrepareBaseCmd(rootCmd, "TM", os.ExpandEnv("$HOME/.tendermint"))
+	cmd.Execute()
 }

cmd/tendermint/probe_upnp.go

@@ -1,24 +0,0 @@
-package main
-
-import (
-	"encoding/json"
-	"fmt"
-
-	"github.com/tendermint/go-p2p/upnp"
-)
-
-func probe_upnp() {
-
-	capabilities, err := upnp.Probe()
-	if err != nil {
-		fmt.Println("Probe failed: %v", err)
-	} else {
-		fmt.Println("Probe success!")
-		jsonBytes, err := json.Marshal(capabilities)
-		if err != nil {
-			panic(err)
-		}
-		fmt.Println(string(jsonBytes))
-	}
-
-}

cmd/tendermint/reset_priv_validator.go

@@ -1,33 +0,0 @@
-package main
-
-import (
-	"os"
-
-	"github.com/tendermint/tendermint/types"
-)
-
-// NOTE: this is totally unsafe.
-// it's only suitable for testnets.
-func reset_all() {
-	reset_priv_validator()
-	os.RemoveAll(config.GetString("db_dir"))
-	os.Remove(config.GetString("cswal"))
-}
-
-// NOTE: this is totally unsafe.
-// it's only suitable for testnets.
-func reset_priv_validator() {
-	// Get PrivValidator
-	var privValidator *types.PrivValidator
-	privValidatorFile := config.GetString("priv_validator_file")
-	if _, err := os.Stat(privValidatorFile); err == nil {
-		privValidator = types.LoadPrivValidator(privValidatorFile)
-		privValidator.Reset()
-		log.Notice("Reset PrivValidator", "file", privValidatorFile)
-	} else {
-		privValidator = types.GenPrivValidator()
-		privValidator.SetFile(privValidatorFile)
-		privValidator.Save()
-		log.Notice("Generated PrivValidator", "file", privValidatorFile)
-	}
-}

cmd/tendermint/show_validator.go

@@ -1,14 +0,0 @@
-package main
-
-import (
-	"fmt"
-
-	"github.com/tendermint/go-wire"
-	"github.com/tendermint/tendermint/types"
-)
-
-func show_validator() {
-	privValidatorFile := config.GetString("priv_validator_file")
-	privValidator := types.LoadOrGenPrivValidator(privValidatorFile)
-	fmt.Println(string(wire.JSONBytesPretty(privValidator.PubKey)))
-}

config/config.go

@@ -0,0 +1,424 @@
+package config
+
+import (
+	"fmt"
+	"path/filepath"
+	"time"
+)
+
+// Config defines the top level configuration for a Tendermint node
+type Config struct {
+	// Top level options use an anonymous struct
+	BaseConfig `mapstructure:",squash"`
+
+	// Options for services
+	RPC       *RPCConfig       `mapstructure:"rpc"`
+	P2P       *P2PConfig       `mapstructure:"p2p"`
+	Mempool   *MempoolConfig   `mapstructure:"mempool"`
+	Consensus *ConsensusConfig `mapstructure:"consensus"`
+}
+
+// DefaultConfig returns a default configuration for a Tendermint node
+func DefaultConfig() *Config {
+	return &Config{
+		BaseConfig: DefaultBaseConfig(),
+		RPC:        DefaultRPCConfig(),
+		P2P:        DefaultP2PConfig(),
+		Mempool:    DefaultMempoolConfig(),
+		Consensus:  DefaultConsensusConfig(),
+	}
+}
+
+// TestConfig returns a configuration that can be used for testing
+func TestConfig() *Config {
+	return &Config{
+		BaseConfig: TestBaseConfig(),
+		RPC:        TestRPCConfig(),
+		P2P:        TestP2PConfig(),
+		Mempool:    DefaultMempoolConfig(),
+		Consensus:  TestConsensusConfig(),
+	}
+}
+
+// SetRoot sets the RootDir for all Config structs
+func (cfg *Config) SetRoot(root string) *Config {
+	cfg.BaseConfig.RootDir = root
+	cfg.RPC.RootDir = root
+	cfg.P2P.RootDir = root
+	cfg.Mempool.RootDir = root
+	cfg.Consensus.RootDir = root
+	return cfg
+}
+
+//-----------------------------------------------------------------------------
+// BaseConfig
+
+// BaseConfig defines the base configuration for a Tendermint node
+type BaseConfig struct {
+	// The root directory for all data.
+	// This should be set in viper so it can unmarshal into this struct
+	RootDir string `mapstructure:"home"`
+
+	// The ID of the chain to join (should be signed with every transaction and vote)
+	ChainID string `mapstructure:"chain_id"`
+
+	// A JSON file containing the initial validator set and other meta data
+	Genesis string `mapstructure:"genesis_file"`
+
+	// A JSON file containing the private key to use as a validator in the consensus protocol
+	PrivValidator string `mapstructure:"priv_validator_file"`
+
+	// A custom human readable name for this node
+	Moniker string `mapstructure:"moniker"`
+
+	// TCP or UNIX socket address of the ABCI application,
+	// or the name of an ABCI application compiled in with the Tendermint binary
+	ProxyApp string `mapstructure:"proxy_app"`
+
+	// Mechanism to connect to the ABCI application: socket | grpc
+	ABCI string `mapstructure:"abci"`
+
+	// Output level for logging
+	LogLevel string `mapstructure:"log_level"`
+
+	// TCP or UNIX socket address for the profiling server to listen on
+	ProfListenAddress string `mapstructure:"prof_laddr"`
+
+	// If this node is many blocks behind the tip of the chain, FastSync
+	// allows them to catchup quickly by downloading blocks in parallel
+	// and verifying their commits
+	FastSync bool `mapstructure:"fast_sync"`
+
+	// If true, query the ABCI app on connecting to a new peer
+	// so the app can decide if we should keep the connection or not
+	FilterPeers bool `mapstructure:"filter_peers"` // false
+
+	// What indexer to use for transactions
+	TxIndex string `mapstructure:"tx_index"`
+
+	// Database backend: leveldb | memdb
+	DBBackend string `mapstructure:"db_backend"`
+
+	// Database directory
+	DBPath string `mapstructure:"db_dir"`
+}
+
+// DefaultBaseConfig returns a default base configuration for a Tendermint node
+func DefaultBaseConfig() BaseConfig {
+	return BaseConfig{
+		Genesis:           "genesis.json",
+		PrivValidator:     "priv_validator.json",
+		Moniker:           "anonymous",
+		ProxyApp:          "tcp://127.0.0.1:46658",
+		ABCI:              "socket",
+		LogLevel:          DefaultPackageLogLevels(),
+		ProfListenAddress: "",
+		FastSync:          true,
+		FilterPeers:       false,
+		TxIndex:           "kv",
+		DBBackend:         "leveldb",
+		DBPath:            "data",
+	}
+}
+
+// TestBaseConfig returns a base configuration for testing a Tendermint node
+func TestBaseConfig() BaseConfig {
+	conf := DefaultBaseConfig()
+	conf.ChainID = "tendermint_test"
+	conf.ProxyApp = "dummy"
+	conf.FastSync = false
+	conf.DBBackend = "memdb"
+	return conf
+}
+
+// GenesisFile returns the full path to the genesis.json file
+func (b BaseConfig) GenesisFile() string {
+	return rootify(b.Genesis, b.RootDir)
+}
+
+// PrivValidatorFile returns the full path to the priv_validator.json file
+func (b BaseConfig) PrivValidatorFile() string {
+	return rootify(b.PrivValidator, b.RootDir)
+}
+
+// DBDir returns the full path to the database directory
+func (b BaseConfig) DBDir() string {
+	return rootify(b.DBPath, b.RootDir)
+}
+
+// DefaultLogLevel returns a default log level of "error"
+func DefaultLogLevel() string {
+	return "error"
+}
+
+// DefaultPackageLogLevels returns a default log level setting so all packages log at "error", while the `state` package logs at "info"
+func DefaultPackageLogLevels() string {
+	return fmt.Sprintf("state:info,*:%s", DefaultLogLevel())
+}
+
+//-----------------------------------------------------------------------------
+// RPCConfig
+
+// RPCConfig defines the configuration options for the Tendermint RPC server
+type RPCConfig struct {
+	RootDir string `mapstructure:"home"`
+
+	// TCP or UNIX socket address for the RPC server to listen on
+	ListenAddress string `mapstructure:"laddr"`
+
+	// TCP or UNIX socket address for the gRPC server to listen on
+	// NOTE: This server only supports /broadcast_tx_commit
+	GRPCListenAddress string `mapstructure:"grpc_laddr"`
+
+	// Activate unsafe RPC commands like /dial_seeds and /unsafe_flush_mempool
+	Unsafe bool `mapstructure:"unsafe"`
+}
+
+// DefaultRPCConfig returns a default configuration for the RPC server
+func DefaultRPCConfig() *RPCConfig {
+	return &RPCConfig{
+		ListenAddress:     "tcp://0.0.0.0:46657",
+		GRPCListenAddress: "",
+		Unsafe:            false,
+	}
+}
+
+// TestRPCConfig returns a configuration for testing the RPC server
+func TestRPCConfig() *RPCConfig {
+	conf := DefaultRPCConfig()
+	conf.ListenAddress = "tcp://0.0.0.0:36657"
+	conf.GRPCListenAddress = "tcp://0.0.0.0:36658"
+	conf.Unsafe = true
+	return conf
+}
+
+//-----------------------------------------------------------------------------
+// P2PConfig
+
+// P2PConfig defines the configuration options for the Tendermint peer-to-peer networking layer
+type P2PConfig struct {
+	RootDir string `mapstructure:"home"`
+
+	// Address to listen for incoming connections
+	ListenAddress string `mapstructure:"laddr"`
+
+	// Comma separated list of seed nodes to connect to
+	Seeds string `mapstructure:"seeds"`
+
+	// Skip UPNP port forwarding
+	SkipUPNP bool `mapstructure:"skip_upnp"`
+
+	// Path to address book
+	AddrBook string `mapstructure:"addr_book_file"`
+
+	// Set true for strict address routability rules
+	AddrBookStrict bool `mapstructure:"addr_book_strict"`
+
+	// Set true to enable the peer-exchange reactor
+	PexReactor bool `mapstructure:"pex"`
+
+	// Maximum number of peers to connect to
+	MaxNumPeers int `mapstructure:"max_num_peers"`
+
+	// Time to wait before flushing messages out on the connection, in ms
+	FlushThrottleTimeout int `mapstructure:"flush_throttle_timeout"`
+
+	// Maximum size of a message packet payload, in bytes
+	MaxMsgPacketPayloadSize int `mapstructure:"max_msg_packet_payload_size"`
+
+	// Rate at which packets can be sent, in bytes/second
+	SendRate int64 `mapstructure:"send_rate"`
+
+	// Rate at which packets can be received, in bytes/second
+	RecvRate int64 `mapstructure:"recv_rate"`
+}
+
+// DefaultP2PConfig returns a default configuration for the peer-to-peer layer
+func DefaultP2PConfig() *P2PConfig {
+	return &P2PConfig{
+		ListenAddress:           "tcp://0.0.0.0:46656",
+		AddrBook:                "addrbook.json",
+		AddrBookStrict:          true,
+		MaxNumPeers:             50,
+		FlushThrottleTimeout:    100,
+		MaxMsgPacketPayloadSize: 1024,   // 1 kB
+		SendRate:                512000, // 500 kB/s
+		RecvRate:                512000, // 500 kB/s
+	}
+}
+
+// TestP2PConfig returns a configuration for testing the peer-to-peer layer
+func TestP2PConfig() *P2PConfig {
+	conf := DefaultP2PConfig()
+	conf.ListenAddress = "tcp://0.0.0.0:36656"
+	conf.SkipUPNP = true
+	return conf
+}
+
+// AddrBookFile returns the full path to the address bool
+func (p *P2PConfig) AddrBookFile() string {
+	return rootify(p.AddrBook, p.RootDir)
+}
+
+//-----------------------------------------------------------------------------
+// MempoolConfig
+
+// MempoolConfig defines the configuration options for the Tendermint mempool
+type MempoolConfig struct {
+	RootDir      string `mapstructure:"home"`
+	Recheck      bool   `mapstructure:"recheck"`
+	RecheckEmpty bool   `mapstructure:"recheck_empty"`
+	Broadcast    bool   `mapstructure:"broadcast"`
+	WalPath      string `mapstructure:"wal_dir"`
+}
+
+// DefaultMempoolConfig returns a default configuration for the Tendermint mempool
+func DefaultMempoolConfig() *MempoolConfig {
+	return &MempoolConfig{
+		Recheck:      true,
+		RecheckEmpty: true,
+		Broadcast:    true,
+		WalPath:      "data/mempool.wal",
+	}
+}
+
+// WalDir returns the full path to the mempool's write-ahead log
+func (m *MempoolConfig) WalDir() string {
+	return rootify(m.WalPath, m.RootDir)
+}
+
+//-----------------------------------------------------------------------------
+// ConsensusConfig
+
+// ConsensusConfig defines the confuguration for the Tendermint consensus service,
+// including timeouts and details about the WAL and the block structure.
+type ConsensusConfig struct {
+	RootDir  string `mapstructure:"home"`
+	WalPath  string `mapstructure:"wal_file"`
+	WalLight bool   `mapstructure:"wal_light"`
+	walFile  string // overrides WalPath if set
+
+	// All timeouts are in ms
+	TimeoutPropose        int `mapstructure:"timeout_propose"`
+	TimeoutProposeDelta   int `mapstructure:"timeout_propose_delta"`
+	TimeoutPrevote        int `mapstructure:"timeout_prevote"`
+	TimeoutPrevoteDelta   int `mapstructure:"timeout_prevote_delta"`
+	TimeoutPrecommit      int `mapstructure:"timeout_precommit"`
+	TimeoutPrecommitDelta int `mapstructure:"timeout_precommit_delta"`
+	TimeoutCommit         int `mapstructure:"timeout_commit"`
+
+	// Make progress as soon as we have all the precommits (as if TimeoutCommit = 0)
+	SkipTimeoutCommit bool `mapstructure:"skip_timeout_commit"`
+
+	// BlockSize
+	MaxBlockSizeTxs   int `mapstructure:"max_block_size_txs"`
+	MaxBlockSizeBytes int `mapstructure:"max_block_size_bytes"`
+
+	// EmptyBlocks mode and possible interval between empty blocks in seconds
+	CreateEmptyBlocks         bool `mapstructure:"create_empty_blocks"`
+	CreateEmptyBlocksInterval int  `mapstructure:"create_empty_blocks_interval"`
+
+	// Reactor sleep duration parameters are in ms
+	PeerGossipSleepDuration     int `mapstructure:"peer_gossip_sleep_duration"`
+	PeerQueryMaj23SleepDuration int `mapstructure:"peer_query_maj23_sleep_duration"`
+}
+
+// WaitForTxs returns true if the consensus should wait for transactions before entering the propose step
+func (cfg *ConsensusConfig) WaitForTxs() bool {
+	return !cfg.CreateEmptyBlocks || cfg.CreateEmptyBlocksInterval > 0
+}
+
+// EmptyBlocks returns the amount of time to wait before proposing an empty block or starting the propose timer if there are no txs available
+func (cfg *ConsensusConfig) EmptyBlocksInterval() time.Duration {
+	return time.Duration(cfg.CreateEmptyBlocksInterval) * time.Second
+}
+
+// Propose returns the amount of time to wait for a proposal
+func (cfg *ConsensusConfig) Propose(round int) time.Duration {
+	return time.Duration(cfg.TimeoutPropose+cfg.TimeoutProposeDelta*round) * time.Millisecond
+}
+
+// Prevote returns the amount of time to wait for straggler votes after receiving any +2/3 prevotes
+func (cfg *ConsensusConfig) Prevote(round int) time.Duration {
+	return time.Duration(cfg.TimeoutPrevote+cfg.TimeoutPrevoteDelta*round) * time.Millisecond
+}
+
+// Precommit returns the amount of time to wait for straggler votes after receiving any +2/3 precommits
+func (cfg *ConsensusConfig) Precommit(round int) time.Duration {
+	return time.Duration(cfg.TimeoutPrecommit+cfg.TimeoutPrecommitDelta*round) * time.Millisecond
+}
+
+// Commit returns the amount of time to wait for straggler votes after receiving +2/3 precommits for a single block (ie. a commit).
+func (cfg *ConsensusConfig) Commit(t time.Time) time.Time {
+	return t.Add(time.Duration(cfg.TimeoutCommit) * time.Millisecond)
+}
+
+// PeerGossipSleep returns the amount of time to sleep if there is nothing to send from the ConsensusReactor
+func (cfg *ConsensusConfig) PeerGossipSleep() time.Duration {
+	return time.Duration(cfg.PeerGossipSleepDuration) * time.Millisecond
+}
+
+// PeerQueryMaj23Sleep returns the amount of time to sleep after each VoteSetMaj23Message is sent in the ConsensusReactor
+func (cfg *ConsensusConfig) PeerQueryMaj23Sleep() time.Duration {
+	return time.Duration(cfg.PeerQueryMaj23SleepDuration) * time.Millisecond
+}
+
+// DefaultConsensusConfig returns a default configuration for the consensus service
+func DefaultConsensusConfig() *ConsensusConfig {
+	return &ConsensusConfig{
+		WalPath:                     "data/cs.wal/wal",
+		WalLight:                    false,
+		TimeoutPropose:              3000,
+		TimeoutProposeDelta:         500,
+		TimeoutPrevote:              1000,
+		TimeoutPrevoteDelta:         500,
+		TimeoutPrecommit:            1000,
+		TimeoutPrecommitDelta:       500,
+		TimeoutCommit:               1000,
+		SkipTimeoutCommit:           false,
+		MaxBlockSizeTxs:             10000,
+		MaxBlockSizeBytes:           1, // TODO
+		CreateEmptyBlocks:           true,
+		CreateEmptyBlocksInterval:   0,
+		PeerGossipSleepDuration:     100,
+		PeerQueryMaj23SleepDuration: 2000,
+	}
+}
+
+// TestConsensusConfig returns a configuration for testing the consensus service
+func TestConsensusConfig() *ConsensusConfig {
+	config := DefaultConsensusConfig()
+	config.TimeoutPropose = 2000
+	config.TimeoutProposeDelta = 1
+	config.TimeoutPrevote = 10
+	config.TimeoutPrevoteDelta = 1
+	config.TimeoutPrecommit = 10
+	config.TimeoutPrecommitDelta = 1
+	config.TimeoutCommit = 10
+	config.SkipTimeoutCommit = true
+	return config
+}
+
+// WalFile returns the full path to the write-ahead log file
+func (c *ConsensusConfig) WalFile() string {
+	if c.walFile != "" {
+		return c.walFile
+	}
+	return rootify(c.WalPath, c.RootDir)
+}
+
+// SetWalFile sets the path to the write-ahead log file
+func (c *ConsensusConfig) SetWalFile(walFile string) {
+	c.walFile = walFile
+}
+
+//-----------------------------------------------------------------------------
+// Utils
+
+// helper function to make config creation independent of root dir
+func rootify(path, root string) string {
+	if filepath.IsAbs(path) {
+		return path
+	}
+	return filepath.Join(root, path)
+}

config/config_test.go

@@ -0,0 +1,28 @@
+package config
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+)
+
+func TestDefaultConfig(t *testing.T) {
+	assert := assert.New(t)
+
+	// set up some defaults
+	cfg := DefaultConfig()
+	assert.NotNil(cfg.P2P)
+	assert.NotNil(cfg.Mempool)
+	assert.NotNil(cfg.Consensus)
+
+	// check the root dir stuff...
+	cfg.SetRoot("/foo")
+	cfg.Genesis = "bar"
+	cfg.DBPath = "/opt/data"
+	cfg.Mempool.WalPath = "wal/mem/"
+
+	assert.Equal("/foo/bar", cfg.GenesisFile())
+	assert.Equal("/opt/data", cfg.DBDir())
+	assert.Equal("/foo/wal/mem", cfg.Mempool.WalDir())
+
+}

config/tendermint/config.go

@@ -1,107 +0,0 @@
-package tendermint
-
-import (
-	"os"
-	"path"
-	"strings"
-
-	. "github.com/tendermint/go-common"
-	cfg "github.com/tendermint/go-config"
-)
-
-func getTMRoot(rootDir string) string {
-	if rootDir == "" {
-		rootDir = os.Getenv("TMROOT")
-	}
-	if rootDir == "" {
-		rootDir = os.Getenv("HOME") + "/.tendermint"
-	}
-	return rootDir
-}
-
-func initTMRoot(rootDir string) {
-	rootDir = getTMRoot(rootDir)
-	EnsureDir(rootDir, 0700)
-
-	configFilePath := path.Join(rootDir, "config.toml")
-
-	// Write default config file if missing.
-	if !FileExists(configFilePath) {
-		// Ask user for moniker
-		// moniker := cfg.Prompt("Type hostname: ", "anonymous")
-		MustWriteFile(configFilePath, []byte(defaultConfig("anonymous")), 0644)
-	}
-}
-
-func GetConfig(rootDir string) cfg.Config {
-	rootDir = getTMRoot(rootDir)
-	initTMRoot(rootDir)
-
-	configFilePath := path.Join(rootDir, "config.toml")
-	mapConfig, err := cfg.ReadMapConfigFromFile(configFilePath)
-	if err != nil {
-		Exit(Fmt("Could not read config: %v", err))
-	}
-
-	// Set defaults or panic
-	if mapConfig.IsSet("chain_id") {
-		Exit("Cannot set 'chain_id' via config.toml")
-	}
-	if mapConfig.IsSet("revision_file") {
-		Exit("Cannot set 'revision_file' via config.toml. It must match what's in the Makefile")
-	}
-	mapConfig.SetRequired("chain_id") // blows up if you try to use it before setting.
-	mapConfig.SetDefault("genesis_file", rootDir+"/genesis.json")
-	mapConfig.SetDefault("proxy_app", "tcp://127.0.0.1:46658")
-	mapConfig.SetDefault("tmsp", "socket")
-	mapConfig.SetDefault("moniker", "anonymous")
-	mapConfig.SetDefault("node_laddr", "tcp://0.0.0.0:46656")
-	mapConfig.SetDefault("seeds", "")
-	// mapConfig.SetDefault("seeds", "goldenalchemist.chaintest.net:46656")
-	mapConfig.SetDefault("fast_sync", true)
-	mapConfig.SetDefault("skip_upnp", false)
-	mapConfig.SetDefault("addrbook_file", rootDir+"/addrbook.json")
-	mapConfig.SetDefault("priv_validator_file", rootDir+"/priv_validator.json")
-	mapConfig.SetDefault("db_backend", "leveldb")
-	mapConfig.SetDefault("db_dir", rootDir+"/data")
-	mapConfig.SetDefault("log_level", "info")
-	mapConfig.SetDefault("rpc_laddr", "tcp://0.0.0.0:46657")
-	mapConfig.SetDefault("prof_laddr", "")
-	mapConfig.SetDefault("revision_file", rootDir+"/revision")
-	mapConfig.SetDefault("cswal", rootDir+"/data/cswal")
-	mapConfig.SetDefault("cswal_light", false)
-	mapConfig.SetDefault("filter_peers", false)
-
-	mapConfig.SetDefault("block_size", 10000)
-	mapConfig.SetDefault("disable_data_hash", false)
-	mapConfig.SetDefault("timeout_propose", 3000)
-	mapConfig.SetDefault("timeout_propose_delta", 500)
-	mapConfig.SetDefault("timeout_prevote", 1000)
-	mapConfig.SetDefault("timeout_prevote_delta", 500)
-	mapConfig.SetDefault("timeout_precommit", 1000)
-	mapConfig.SetDefault("timeout_precommit_delta", 500)
-	mapConfig.SetDefault("timeout_commit", 1000)
-	mapConfig.SetDefault("mempool_recheck", true)
-	mapConfig.SetDefault("mempool_recheck_empty", true)
-	mapConfig.SetDefault("mempool_broadcast", true)
-
-	return mapConfig
-}
-
-var defaultConfigTmpl = `# This is a TOML config file.
-# For more information, see https://github.com/toml-lang/toml
-
-proxy_app = "tcp://127.0.0.1:46658"
-moniker = "__MONIKER__"
-node_laddr = "tcp://0.0.0.0:46656"
-seeds = ""
-fast_sync = true
-db_backend = "leveldb"
-log_level = "notice"
-rpc_laddr = "tcp://0.0.0.0:46657"
-`
-
-func defaultConfig(moniker string) (defaultConfig string) {
-	defaultConfig = strings.Replace(defaultConfigTmpl, "__MONIKER__", moniker, -1)
-	return
-}

config/tendermint/logrotate.config

@@ -1,22 +0,0 @@
-// If you wanted to use logrotate, I suppose this might be the config you want.
-// Instead, I'll just write our own, that way we don't need sudo to install.
-
-$HOME/.tendermint/logs/tendermint.log {
-  missingok
-  notifempty
-  rotate 12
-  daily
-  size 10M
-  compress
-  delaycompress
-}
-
-$HOME/.barak/logs/barak.log {
-  missingok
-  notifempty
-  rotate 12
-  weekly
-  size 10M
-  compress
-  delaycompress
-}

config/tendermint_test/config.go

@@ -1,151 +0,0 @@
-// Import this in all *_test.go files to initialize ~/.tendermint_test.
-
-package tendermint_test
-
-import (
-	"os"
-	"path"
-	"strings"
-
-	. "github.com/tendermint/go-common"
-	cfg "github.com/tendermint/go-config"
-)
-
-func init() {
-	// Creates ~/.tendermint_test
-	EnsureDir(os.Getenv("HOME")+"/.tendermint_test", 0700)
-}
-
-func initTMRoot(rootDir string) {
-	// Remove ~/.tendermint_test_bak
-	if FileExists(rootDir + "_bak") {
-		err := os.RemoveAll(rootDir + "_bak")
-		if err != nil {
-			PanicSanity(err.Error())
-		}
-	}
-	// Move ~/.tendermint_test to ~/.tendermint_test_bak
-	if FileExists(rootDir) {
-		err := os.Rename(rootDir, rootDir+"_bak")
-		if err != nil {
-			PanicSanity(err.Error())
-		}
-	}
-	// Create new dir
-	EnsureDir(rootDir, 0700)
-
-	configFilePath := path.Join(rootDir, "config.toml")
-	genesisFilePath := path.Join(rootDir, "genesis.json")
-	privFilePath := path.Join(rootDir, "priv_validator.json")
-
-	// Write default config file if missing.
-	if !FileExists(configFilePath) {
-		// Ask user for moniker
-		// moniker := cfg.Prompt("Type hostname: ", "anonymous")
-		MustWriteFile(configFilePath, []byte(defaultConfig("anonymous")), 0644)
-	}
-	if !FileExists(genesisFilePath) {
-		MustWriteFile(genesisFilePath, []byte(defaultGenesis), 0644)
-	}
-	// we always overwrite the priv val
-	MustWriteFile(privFilePath, []byte(defaultPrivValidator), 0644)
-}
-
-func ResetConfig(localPath string) cfg.Config {
-	rootDir := os.Getenv("HOME") + "/.tendermint_test/" + localPath
-	initTMRoot(rootDir)
-
-	configFilePath := path.Join(rootDir, "config.toml")
-	mapConfig, err := cfg.ReadMapConfigFromFile(configFilePath)
-	if err != nil {
-		Exit(Fmt("Could not read config: %v", err))
-	}
-
-	// Set defaults or panic
-	if mapConfig.IsSet("chain_id") {
-		Exit("Cannot set 'chain_id' via config.toml")
-	}
-	mapConfig.SetDefault("chain_id", "tendermint_test")
-	mapConfig.SetDefault("genesis_file", rootDir+"/genesis.json")
-	mapConfig.SetDefault("proxy_app", "dummy")
-	mapConfig.SetDefault("tmsp", "socket")
-	mapConfig.SetDefault("moniker", "anonymous")
-	mapConfig.SetDefault("node_laddr", "tcp://0.0.0.0:36656")
-	mapConfig.SetDefault("fast_sync", false)
-	mapConfig.SetDefault("skip_upnp", true)
-	mapConfig.SetDefault("addrbook_file", rootDir+"/addrbook.json")
-	mapConfig.SetDefault("priv_validator_file", rootDir+"/priv_validator.json")
-	mapConfig.SetDefault("db_backend", "memdb")
-	mapConfig.SetDefault("db_dir", rootDir+"/data")
-	mapConfig.SetDefault("log_level", "debug")
-	mapConfig.SetDefault("rpc_laddr", "tcp://0.0.0.0:36657")
-	mapConfig.SetDefault("prof_laddr", "")
-	mapConfig.SetDefault("revision_file", rootDir+"/revision")
-	mapConfig.SetDefault("cswal", rootDir+"/data/cswal")
-	mapConfig.SetDefault("cswal_light", false)
-	mapConfig.SetDefault("filter_peers", false)
-
-	mapConfig.SetDefault("block_size", 10000)
-	mapConfig.SetDefault("disable_data_hash", false)
-	mapConfig.SetDefault("timeout_propose", 2000)
-	mapConfig.SetDefault("timeout_propose_delta", 500)
-	mapConfig.SetDefault("timeout_prevote", 1000)
-	mapConfig.SetDefault("timeout_prevote_delta", 500)
-	mapConfig.SetDefault("timeout_precommit", 1000)
-	mapConfig.SetDefault("timeout_precommit_delta", 500)
-	mapConfig.SetDefault("timeout_commit", 100)
-	mapConfig.SetDefault("mempool_recheck", true)
-	mapConfig.SetDefault("mempool_recheck_empty", true)
-	mapConfig.SetDefault("mempool_broadcast", true)
-
-	return mapConfig
-}
-
-var defaultConfigTmpl = `# This is a TOML config file.
-# For more information, see https://github.com/toml-lang/toml
-
-proxy_app = "dummy"
-moniker = "__MONIKER__"
-node_laddr = "tcp://0.0.0.0:36656"
-seeds = ""
-fast_sync = false
-db_backend = "memdb"
-log_level = "debug"
-rpc_laddr = "tcp://0.0.0.0:36657"
-`
-
-func defaultConfig(moniker string) (defaultConfig string) {
-	defaultConfig = strings.Replace(defaultConfigTmpl, "__MONIKER__", moniker, -1)
-	return
-}
-
-var defaultGenesis = `{
-  "genesis_time": "0001-01-01T00:00:00.000Z",
-  "chain_id": "tendermint_test",
-  "validators": [
-    {
-      "pub_key": [
-        1,
-        "3B3069C422E19688B45CBFAE7BB009FC0FA1B1EA86593519318B7214853803C8"
-      ],
-      "amount": 10,
-      "name": ""
-    }
-  ],
-  "app_hash": ""
-}`
-
-var defaultPrivValidator = `{
-  "address": "D028C9981F7A87F3093672BF0D5B0E2A1B3ED456",
-  "pub_key": [
-    1,
-    "3B3069C422E19688B45CBFAE7BB009FC0FA1B1EA86593519318B7214853803C8"
-  ],
-  "priv_key": [
-    1,
-    "27F82582AEFAE7AB151CFB01C48BB6C1A0DA78F9BDDA979A9F70A84D074EB07D3B3069C422E19688B45CBFAE7BB009FC0FA1B1EA86593519318B7214853803C8"
-  ],
-  "last_height": 0,
-  "last_round": 0,
-  "last_step": 0
-}`

config/toml.go

@@ -0,0 +1,142 @@
+package config
+
+import (
+	"os"
+	"path"
+	"path/filepath"
+	"strings"
+
+	cmn "github.com/tendermint/tmlibs/common"
+)
+
+/****** these are for production settings ***********/
+
+func EnsureRoot(rootDir string) {
+	cmn.EnsureDir(rootDir, 0700)
+	cmn.EnsureDir(rootDir+"/data", 0700)
+
+	configFilePath := path.Join(rootDir, "config.toml")
+
+	// Write default config file if missing.
+	if !cmn.FileExists(configFilePath) {
+		// Ask user for moniker
+		// moniker := cfg.Prompt("Type hostname: ", "anonymous")
+		cmn.MustWriteFile(configFilePath, []byte(defaultConfig("anonymous")), 0644)
+	}
+}
+
+var defaultConfigTmpl = `# This is a TOML config file.
+# For more information, see https://github.com/toml-lang/toml
+
+proxy_app = "tcp://127.0.0.1:46658"
+moniker = "__MONIKER__"
+fast_sync = true
+db_backend = "leveldb"
+log_level = "state:info,*:error"
+
+[rpc]
+laddr = "tcp://0.0.0.0:46657"
+
+[p2p]
+laddr = "tcp://0.0.0.0:46656"
+seeds = ""
+`
+
+func defaultConfig(moniker string) string {
+	return strings.Replace(defaultConfigTmpl, "__MONIKER__", moniker, -1)
+}
+
+/****** these are for test settings ***********/
+
+func ResetTestRoot(testName string) *Config {
+	rootDir := os.ExpandEnv("$HOME/.tendermint_test")
+	rootDir = filepath.Join(rootDir, testName)
+	// Remove ~/.tendermint_test_bak
+	if cmn.FileExists(rootDir + "_bak") {
+		err := os.RemoveAll(rootDir + "_bak")
+		if err != nil {
+			cmn.PanicSanity(err.Error())
+		}
+	}
+	// Move ~/.tendermint_test to ~/.tendermint_test_bak
+	if cmn.FileExists(rootDir) {
+		err := os.Rename(rootDir, rootDir+"_bak")
+		if err != nil {
+			cmn.PanicSanity(err.Error())
+		}
+	}
+	// Create new dir
+	cmn.EnsureDir(rootDir, 0700)
+	cmn.EnsureDir(rootDir+"/data", 0700)
+
+	configFilePath := path.Join(rootDir, "config.toml")
+	genesisFilePath := path.Join(rootDir, "genesis.json")
+	privFilePath := path.Join(rootDir, "priv_validator.json")
+
+	// Write default config file if missing.
+	if !cmn.FileExists(configFilePath) {
+		// Ask user for moniker
+		cmn.MustWriteFile(configFilePath, []byte(testConfig("anonymous")), 0644)
+	}
+	if !cmn.FileExists(genesisFilePath) {
+		cmn.MustWriteFile(genesisFilePath, []byte(testGenesis), 0644)
+	}
+	// we always overwrite the priv val
+	cmn.MustWriteFile(privFilePath, []byte(testPrivValidator), 0644)
+
+	config := TestConfig().SetRoot(rootDir)
+	return config
+}
+
+var testConfigTmpl = `# This is a TOML config file.
+# For more information, see https://github.com/toml-lang/toml
+
+proxy_app = "dummy"
+moniker = "__MONIKER__"
+fast_sync = false
+db_backend = "memdb"
+log_level = "info"
+
+[rpc]
+laddr = "tcp://0.0.0.0:36657"
+
+[p2p]
+laddr = "tcp://0.0.0.0:36656"
+seeds = ""
+`
+
+func testConfig(moniker string) (testConfig string) {
+	testConfig = strings.Replace(testConfigTmpl, "__MONIKER__", moniker, -1)
+	return
+}
+
+var testGenesis = `{
+  "genesis_time": "0001-01-01T00:00:00.000Z",
+  "chain_id": "tendermint_test",
+  "validators": [
+    {
+      "pub_key": {
+        "type": "ed25519",
+        "data":"3B3069C422E19688B45CBFAE7BB009FC0FA1B1EA86593519318B7214853803C8"
+      },
+      "power": 10,
+      "name": ""
+    }
+  ],
+  "app_hash": ""
+}`
+
+var testPrivValidator = `{
+  "address": "D028C9981F7A87F3093672BF0D5B0E2A1B3ED456",
+  "pub_key": {
+    "type": "ed25519",
+    "data": "3B3069C422E19688B45CBFAE7BB009FC0FA1B1EA86593519318B7214853803C8"
+  },
+  "priv_key": {
+    "type": "ed25519",
+    "data": "27F82582AEFAE7AB151CFB01C48BB6C1A0DA78F9BDDA979A9F70A84D074EB07D3B3069C422E19688B45CBFAE7BB009FC0FA1B1EA86593519318B7214853803C8"
+  },
+  "last_height": 0,
+  "last_round": 0,
+  "last_step": 0
+}`

config/toml_test.go

@@ -0,0 +1,57 @@
+package config
+
+import (
+	"io/ioutil"
+	"os"
+	"path/filepath"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func ensureFiles(t *testing.T, rootDir string, files ...string) {
+	for _, f := range files {
+		p := rootify(rootDir, f)
+		_, err := os.Stat(p)
+		assert.Nil(t, err, p)
+	}
+}
+
+func TestEnsureRoot(t *testing.T) {
+	assert, require := assert.New(t), require.New(t)
+
+	// setup temp dir for test
+	tmpDir, err := ioutil.TempDir("", "config-test")
+	require.Nil(err)
+	defer os.RemoveAll(tmpDir)
+
+	// create root dir
+	EnsureRoot(tmpDir)
+
+	// make sure config is set properly
+	data, err := ioutil.ReadFile(filepath.Join(tmpDir, "config.toml"))
+	require.Nil(err)
+	assert.Equal([]byte(defaultConfig("anonymous")), data)
+
+	ensureFiles(t, tmpDir, "data")
+}
+
+func TestEnsureTestRoot(t *testing.T) {
+	assert, require := assert.New(t), require.New(t)
+
+	testName := "ensureTestRoot"
+
+	// create root dir
+	cfg := ResetTestRoot(testName)
+	rootDir := cfg.RootDir
+
+	// make sure config is set properly
+	data, err := ioutil.ReadFile(filepath.Join(rootDir, "config.toml"))
+	require.Nil(err)
+	assert.Equal([]byte(testConfig("anonymous")), data)
+
+	// TODO: make sure the cfg returned and testconfig are the same!
+
+	ensureFiles(t, rootDir, "data", "genesis.json", "priv_validator.json")
+}

consensus/byzantine_test.go

@@ -0,0 +1,300 @@
+package consensus
+
+import (
+	"sync"
+	"testing"
+	"time"
+
+	crypto "github.com/tendermint/go-crypto"
+	data "github.com/tendermint/go-wire/data"
+	"github.com/tendermint/tendermint/p2p"
+	"github.com/tendermint/tendermint/types"
+	cmn "github.com/tendermint/tmlibs/common"
+	"github.com/tendermint/tmlibs/events"
+)
+
+func init() {
+	config = ResetConfig("consensus_byzantine_test")
+}
+
+//----------------------------------------------
+// byzantine failures
+
+// 4 validators. 1 is byzantine. The other three are partitioned into A (1 val) and B (2 vals).
+// byzantine validator sends conflicting proposals into A and B,
+// and prevotes/precommits on both of them.
+// B sees a commit, A doesn't.
+// Byzantine validator refuses to prevote.
+// Heal partition and ensure A sees the commit
+func TestByzantine(t *testing.T) {
+	N := 4
+	logger := consensusLogger()
+	css := randConsensusNet(N, "consensus_byzantine_test", newMockTickerFunc(false), newCounter)
+
+	// give the byzantine validator a normal ticker
+	css[0].SetTimeoutTicker(NewTimeoutTicker())
+
+	switches := make([]*p2p.Switch, N)
+	p2pLogger := logger.With("module", "p2p")
+	for i := 0; i < N; i++ {
+		switches[i] = p2p.NewSwitch(config.P2P)
+		switches[i].SetLogger(p2pLogger.With("validator", i))
+	}
+
+	reactors := make([]p2p.Reactor, N)
+	defer func() {
+		for _, r := range reactors {
+			if rr, ok := r.(*ByzantineReactor); ok {
+				rr.reactor.Switch.Stop()
+			} else {
+				r.(*ConsensusReactor).Switch.Stop()
+			}
+		}
+	}()
+	eventChans := make([]chan interface{}, N)
+	eventLogger := logger.With("module", "events")
+	for i := 0; i < N; i++ {
+		if i == 0 {
+			css[i].privValidator = NewByzantinePrivValidator(css[i].privValidator)
+			// make byzantine
+			css[i].decideProposal = func(j int) func(int, int) {
+				return func(height, round int) {
+					byzantineDecideProposalFunc(t, height, round, css[j], switches[j])
+				}
+			}(i)
+			css[i].doPrevote = func(height, round int) {}
+		}
+
+		eventSwitch := events.NewEventSwitch()
+		eventSwitch.SetLogger(eventLogger.With("validator", i))
+		_, err := eventSwitch.Start()
+		if err != nil {
+			t.Fatalf("Failed to start switch: %v", err)
+		}
+		eventChans[i] = subscribeToEvent(eventSwitch, "tester", types.EventStringNewBlock(), 1)
+
+		conR := NewConsensusReactor(css[i], true) // so we dont start the consensus states
+		conR.SetLogger(logger.With("validator", i))
+		conR.SetEventSwitch(eventSwitch)
+
+		var conRI p2p.Reactor
+		conRI = conR
+
+		if i == 0 {
+			conRI = NewByzantineReactor(conR)
+		}
+		reactors[i] = conRI
+	}
+
+	p2p.MakeConnectedSwitches(config.P2P, N, func(i int, s *p2p.Switch) *p2p.Switch {
+		// ignore new switch s, we already made ours
+		switches[i].AddReactor("CONSENSUS", reactors[i])
+		return switches[i]
+	}, func(sws []*p2p.Switch, i, j int) {
+		// the network starts partitioned with globally active adversary
+		if i != 0 {
+			return
+		}
+		p2p.Connect2Switches(sws, i, j)
+	})
+
+	// start the state machines
+	byzR := reactors[0].(*ByzantineReactor)
+	s := byzR.reactor.conS.GetState()
+	byzR.reactor.SwitchToConsensus(s, 0)
+	for i := 1; i < N; i++ {
+		cr := reactors[i].(*ConsensusReactor)
+		cr.SwitchToConsensus(cr.conS.GetState(), 0)
+	}
+
+	// byz proposer sends one block to peers[0]
+	// and the other block to peers[1] and peers[2].
+	// note peers and switches order don't match.
+	peers := switches[0].Peers().List()
+	ind0 := getSwitchIndex(switches, peers[0])
+	ind1 := getSwitchIndex(switches, peers[1])
+	ind2 := getSwitchIndex(switches, peers[2])
+
+	// connect the 2 peers in the larger partition
+	p2p.Connect2Switches(switches, ind1, ind2)
+
+	// wait for someone in the big partition to make a block
+	<-eventChans[ind2]
+
+	t.Log("A block has been committed. Healing partition")
+
+	// connect the partitions
+	p2p.Connect2Switches(switches, ind0, ind1)
+	p2p.Connect2Switches(switches, ind0, ind2)
+
+	// wait till everyone makes the first new block
+	// (one of them already has)
+	wg := new(sync.WaitGroup)
+	wg.Add(2)
+	for i := 1; i < N-1; i++ {
+		go func(j int) {
+			<-eventChans[j]
+			wg.Done()
+		}(i)
+	}
+
+	done := make(chan struct{})
+	go func() {
+		wg.Wait()
+		close(done)
+	}()
+
+	tick := time.NewTicker(time.Second * 10)
+	select {
+	case <-done:
+	case <-tick.C:
+		for i, reactor := range reactors {
+			t.Log(cmn.Fmt("Consensus Reactor %v", i))
+			t.Log(cmn.Fmt("%v", reactor))
+		}
+		t.Fatalf("Timed out waiting for all validators to commit first block")
+	}
+}
+
+//-------------------------------
+// byzantine consensus functions
+
+func byzantineDecideProposalFunc(t *testing.T, height, round int, cs *ConsensusState, sw *p2p.Switch) {
+	// byzantine user should create two proposals and try to split the vote.
+	// Avoid sending on internalMsgQueue and running consensus state.
+
+	// Create a new proposal block from state/txs from the mempool.
+	block1, blockParts1 := cs.createProposalBlock()
+	polRound, polBlockID := cs.Votes.POLInfo()
+	proposal1 := types.NewProposal(height, round, blockParts1.Header(), polRound, polBlockID)
+	cs.privValidator.SignProposal(cs.state.ChainID, proposal1) // byzantine doesnt err
+
+	// Create a new proposal block from state/txs from the mempool.
+	block2, blockParts2 := cs.createProposalBlock()
+	polRound, polBlockID = cs.Votes.POLInfo()
+	proposal2 := types.NewProposal(height, round, blockParts2.Header(), polRound, polBlockID)
+	cs.privValidator.SignProposal(cs.state.ChainID, proposal2) // byzantine doesnt err
+
+	block1Hash := block1.Hash()
+	block2Hash := block2.Hash()
+
+	// broadcast conflicting proposals/block parts to peers
+	peers := sw.Peers().List()
+	t.Logf("Byzantine: broadcasting conflicting proposals to %d peers", len(peers))
+	for i, peer := range peers {
+		if i < len(peers)/2 {
+			go sendProposalAndParts(height, round, cs, peer, proposal1, block1Hash, blockParts1)
+		} else {
+			go sendProposalAndParts(height, round, cs, peer, proposal2, block2Hash, blockParts2)
+		}
+	}
+}
+
+func sendProposalAndParts(height, round int, cs *ConsensusState, peer p2p.Peer, proposal *types.Proposal, blockHash []byte, parts *types.PartSet) {
+	// proposal
+	msg := &ProposalMessage{Proposal: proposal}
+	peer.Send(DataChannel, struct{ ConsensusMessage }{msg})
+
+	// parts
+	for i := 0; i < parts.Total(); i++ {
+		part := parts.GetPart(i)
+		msg := &BlockPartMessage{
+			Height: height, // This tells peer that this part applies to us.
+			Round:  round,  // This tells peer that this part applies to us.
+			Part:   part,
+		}
+		peer.Send(DataChannel, struct{ ConsensusMessage }{msg})
+	}
+
+	// votes
+	cs.mtx.Lock()
+	prevote, _ := cs.signVote(types.VoteTypePrevote, blockHash, parts.Header())
+	precommit, _ := cs.signVote(types.VoteTypePrecommit, blockHash, parts.Header())
+	cs.mtx.Unlock()
+
+	peer.Send(VoteChannel, struct{ ConsensusMessage }{&VoteMessage{prevote}})
+	peer.Send(VoteChannel, struct{ ConsensusMessage }{&VoteMessage{precommit}})
+}
+
+//----------------------------------------
+// byzantine consensus reactor
+
+type ByzantineReactor struct {
+	cmn.Service
+	reactor *ConsensusReactor
+}
+
+func NewByzantineReactor(conR *ConsensusReactor) *ByzantineReactor {
+	return &ByzantineReactor{
+		Service: conR,
+		reactor: conR,
+	}
+}
+
+func (br *ByzantineReactor) SetSwitch(s *p2p.Switch)               { br.reactor.SetSwitch(s) }
+func (br *ByzantineReactor) GetChannels() []*p2p.ChannelDescriptor { return br.reactor.GetChannels() }
+func (br *ByzantineReactor) AddPeer(peer p2p.Peer) {
+	if !br.reactor.IsRunning() {
+		return
+	}
+
+	// Create peerState for peer
+	peerState := NewPeerState(peer).SetLogger(br.reactor.Logger)
+	peer.Set(types.PeerStateKey, peerState)
+
+	// Send our state to peer.
+	// If we're fast_syncing, broadcast a RoundStepMessage later upon SwitchToConsensus().
+	if !br.reactor.fastSync {
+		br.reactor.sendNewRoundStepMessages(peer)
+	}
+}
+func (br *ByzantineReactor) RemovePeer(peer p2p.Peer, reason interface{}) {
+	br.reactor.RemovePeer(peer, reason)
+}
+func (br *ByzantineReactor) Receive(chID byte, peer p2p.Peer, msgBytes []byte) {
+	br.reactor.Receive(chID, peer, msgBytes)
+}
+
+//----------------------------------------
+// byzantine privValidator
+
+type ByzantinePrivValidator struct {
+	types.Signer
+
+	pv types.PrivValidator
+}
+
+// Return a priv validator that will sign anything
+func NewByzantinePrivValidator(pv types.PrivValidator) *ByzantinePrivValidator {
+	return &ByzantinePrivValidator{
+		Signer: pv.(*types.PrivValidatorFS).Signer,
+		pv:     pv,
+	}
+}
+
+func (privVal *ByzantinePrivValidator) GetAddress() data.Bytes {
+	return privVal.pv.GetAddress()
+}
+
+func (privVal *ByzantinePrivValidator) GetPubKey() crypto.PubKey {
+	return privVal.pv.GetPubKey()
+}
+
+func (privVal *ByzantinePrivValidator) SignVote(chainID string, vote *types.Vote) (err error) {
+	vote.Signature, err = privVal.Sign(types.SignBytes(chainID, vote))
+	return err
+}
+
+func (privVal *ByzantinePrivValidator) SignProposal(chainID string, proposal *types.Proposal) (err error) {
+	proposal.Signature, err = privVal.Sign(types.SignBytes(chainID, proposal))
+	return nil
+}
+
+func (privVal *ByzantinePrivValidator) SignHeartbeat(chainID string, heartbeat *types.Heartbeat) (err error) {
+	heartbeat.Signature, err = privVal.Sign(types.SignBytes(chainID, heartbeat))
+	return nil
+}
+
+func (privVal *ByzantinePrivValidator) String() string {
+	return cmn.Fmt("PrivValidator{%X}", privVal.GetAddress())
+}

consensus/common.go

@@ -1,15 +1,35 @@
 package consensus
 
 import (
-	"github.com/tendermint/go-events"
+	"github.com/tendermint/tendermint/types"
 )
 
-// NOTE: this is blocking
-func subscribeToEvent(evsw *events.EventSwitch, receiver, eventID string, chanCap int) chan interface{} {
-	// listen for new round
+// XXX: WARNING: these functions can halt the consensus as firing events is synchronous.
+// Make sure to read off the channels, and in the case of subscribeToEventRespond, to write back on it
+
+// NOTE: if chanCap=0, this blocks on the event being consumed
+func subscribeToEvent(evsw types.EventSwitch, receiver, eventID string, chanCap int) chan interface{} {
+	// listen for event
 	ch := make(chan interface{}, chanCap)
-	evsw.AddListenerForEvent(receiver, eventID, func(data events.EventData) {
+	types.AddListenerForEvent(evsw, receiver, eventID, func(data types.TMEventData) {
 		ch <- data
 	})
 	return ch
 }
+
+// NOTE: this blocks on receiving a response after the event is consumed
+func subscribeToEventRespond(evsw types.EventSwitch, receiver, eventID string) chan interface{} {
+	// listen for event
+	ch := make(chan interface{})
+	types.AddListenerForEvent(evsw, receiver, eventID, func(data types.TMEventData) {
+		ch <- data
+		<-ch
+	})
+	return ch
+}
+
+func discardFromChan(ch chan interface{}, n int) {
+	for i := 0; i < n; i++ {
+		<-ch
+	}
+}

consensus/common_test.go

@@ -3,52 +3,80 @@ package consensus
 import (
 	"bytes"
 	"fmt"
+	"io/ioutil"
+	"os"
+	"path"
 	"sort"
 	"sync"
 	"testing"
 	"time"
 
-	cfg "github.com/tendermint/go-config"
-	dbm "github.com/tendermint/go-db"
-	"github.com/tendermint/go-events"
+	abcicli "github.com/tendermint/abci/client"
+	abci "github.com/tendermint/abci/types"
 	bc "github.com/tendermint/tendermint/blockchain"
+	cfg "github.com/tendermint/tendermint/config"
+	cstypes "github.com/tendermint/tendermint/consensus/types"
 	mempl "github.com/tendermint/tendermint/mempool"
+	"github.com/tendermint/tendermint/p2p"
 	sm "github.com/tendermint/tendermint/state"
 	"github.com/tendermint/tendermint/types"
-	tmspcli "github.com/tendermint/tmsp/client"
-	tmsp "github.com/tendermint/tmsp/types"
+	cmn "github.com/tendermint/tmlibs/common"
+	dbm "github.com/tendermint/tmlibs/db"
+	"github.com/tendermint/tmlibs/log"
 
-	"github.com/tendermint/tmsp/example/counter"
+	"github.com/tendermint/abci/example/counter"
+	"github.com/tendermint/abci/example/dummy"
+
+	"github.com/go-kit/kit/log/term"
 )
 
-var config cfg.Config // NOTE: must be reset for each _test.go file
-var ensureTimeout = time.Duration(2)
+// genesis, chain_id, priv_val
+var config *cfg.Config // NOTE: must be reset for each _test.go file
+var ensureTimeout = time.Second * 2
 
-type validatorStub struct {
-	Height int
-	Round  int
-	*types.PrivValidator
+func ensureDir(dir string, mode os.FileMode) {
+	if err := cmn.EnsureDir(dir, mode); err != nil {
+		panic(err)
+	}
 }
 
-func NewValidatorStub(privValidator *types.PrivValidator) *validatorStub {
+func ResetConfig(name string) *cfg.Config {
+	return cfg.ResetTestRoot(name)
+}
+
+//-------------------------------------------------------------------------------
+// validator stub (a dummy consensus peer we control)
+
+type validatorStub struct {
+	Index  int // Validator index. NOTE: we don't assume validator set changes.
+	Height int
+	Round  int
+	types.PrivValidator
+}
+
+var testMinPower = 10
+
+func NewValidatorStub(privValidator types.PrivValidator, valIndex int) *validatorStub {
 	return &validatorStub{
+		Index:         valIndex,
 		PrivValidator: privValidator,
 	}
 }
 
 func (vs *validatorStub) signVote(voteType byte, hash []byte, header types.PartSetHeader) (*types.Vote, error) {
 	vote := &types.Vote{
+		ValidatorIndex:   vs.Index,
+		ValidatorAddress: vs.PrivValidator.GetAddress(),
 		Height:           vs.Height,
 		Round:            vs.Round,
 		Type:             voteType,
-		BlockHash:        hash,
-		BlockPartsHeader: header,
+		BlockID:          types.BlockID{hash, header},
 	}
-	err := vs.PrivValidator.SignVote(config.GetString("chain_id"), vote)
+	err := vs.PrivValidator.SignVote(config.ChainID, vote)
 	return vote, err
 }
 
-// convenienve function for testing
+// Sign vote for type/hash/header
 func signVote(vs *validatorStub, voteType byte, hash []byte, header types.PartSetHeader) *types.Vote {
 	v, err := vs.signVote(voteType, hash, header)
 	if err != nil {
@@ -57,102 +85,7 @@ func signVote(vs *validatorStub, voteType byte, hash []byte, header types.PartSe
 	return v
 }
 
-// create proposal block from cs1 but sign it with vs
-func decideProposal(cs1 *ConsensusState, cs2 *validatorStub, height, round int) (proposal *types.Proposal, block *types.Block) {
-	block, blockParts := cs1.createProposalBlock()
-	if block == nil { // on error
-		panic("error creating proposal block")
-	}
-
-	// Make proposal
-	proposal = types.NewProposal(height, round, blockParts.Header(), cs1.Votes.POLRound())
-	if err := cs2.SignProposal(config.GetString("chain_id"), proposal); err != nil {
-		panic(err)
-	}
-	return
-}
-
-//-------------------------------------------------------------------------------
-// utils
-
-/*
-func nilRound(t *testing.T, cs1 *ConsensusState, vss ...*validatorStub) {
-	cs1.mtx.Lock()
-	height, round := cs1.Height, cs1.Round
-	cs1.mtx.Unlock()
-
-	waitFor(t, cs1, height, round, RoundStepPrevote)
-
-	signAddVoteToFromMany(types.VoteTypePrevote, cs1, nil, cs1.ProposalBlockParts.Header(), vss...)
-
-	waitFor(t, cs1, height, round, RoundStepPrecommit)
-
-	signAddVoteToFromMany(types.VoteTypePrecommit, cs1, nil, cs1.ProposalBlockParts.Header(), vss...)
-
-	waitFor(t, cs1, height, round+1, RoundStepNewRound)
-}
-*/
-
-// NOTE: this switches the propser as far as `perspectiveOf` is concerned,
-// but for simplicity we return a block it generated.
-func changeProposer(t *testing.T, perspectiveOf *ConsensusState, newProposer *validatorStub) *types.Block {
-	_, v1 := perspectiveOf.Validators.GetByAddress(perspectiveOf.privValidator.Address)
-	v1.Accum, v1.VotingPower = 0, 0
-	if updated := perspectiveOf.Validators.Update(v1); !updated {
-		panic("failed to update validator")
-	}
-	_, v2 := perspectiveOf.Validators.GetByAddress(newProposer.Address)
-	v2.Accum, v2.VotingPower = 100, 100
-	if updated := perspectiveOf.Validators.Update(v2); !updated {
-		panic("failed to update validator")
-	}
-
-	// make the proposal
-	propBlock, _ := perspectiveOf.createProposalBlock()
-	if propBlock == nil {
-		panic("Failed to create proposal block with cs2")
-	}
-	return propBlock
-}
-
-func fixVotingPower(t *testing.T, cs1 *ConsensusState, addr2 []byte) {
-	_, v1 := cs1.Validators.GetByAddress(cs1.privValidator.Address)
-	_, v2 := cs1.Validators.GetByAddress(addr2)
-	v1.Accum, v1.VotingPower = v2.Accum, v2.VotingPower
-	if updated := cs1.Validators.Update(v1); !updated {
-		panic("failed to update validator")
-	}
-}
-
-func addVoteToFromMany(to *ConsensusState, votes []*types.Vote, froms ...*validatorStub) {
-	if len(votes) != len(froms) {
-		panic("len(votes) and len(froms) must match")
-	}
-
-	for i, from := range froms {
-		addVoteToFrom(to, from, votes[i])
-	}
-}
-
-func addVoteToFrom(to *ConsensusState, from *validatorStub, vote *types.Vote) {
-	to.mtx.Lock() // NOTE: wont need this when the vote comes with the index!
-	valIndex, _ := to.Validators.GetByAddress(from.PrivValidator.Address)
-	to.mtx.Unlock()
-
-	to.peerMsgQueue <- msgInfo{Msg: &VoteMessage{valIndex, vote}}
-	// added, err := to.TryAddVote(valIndex, vote, "")
-	/*
-		if _, ok := err.(*types.ErrVoteConflictingSignature); ok {
-			// let it fly
-		} else if !added {
-			fmt.Println("to, from, vote:", to.Height, from.Height, vote.Height)
-			panic(fmt.Sprintln("Failed to add vote. Err:", err))
-		} else if err != nil {
-			panic(fmt.Sprintln("Failed to add vote:", err))
-		}*/
-}
-
-func signVoteMany(voteType byte, hash []byte, header types.PartSetHeader, vss ...*validatorStub) []*types.Vote {
+func signVotes(voteType byte, hash []byte, header types.PartSetHeader, vss ...*validatorStub) []*types.Vote {
 	votes := make([]*types.Vote, len(vss))
 	for i, vs := range vss {
 		votes[i] = signVote(vs, voteType, hash, header)
@@ -160,79 +93,6 @@ func signVoteMany(voteType byte, hash []byte, header types.PartSetHeader, vss ..
 	return votes
 }
 
-// add vote to one cs from another
-// if voteCh is not nil, read all votes
-func signAddVoteToFromMany(voteType byte, to *ConsensusState, hash []byte, header types.PartSetHeader, voteCh chan interface{}, froms ...*validatorStub) {
-	var wg chan struct{} // when done reading all votes
-	if voteCh != nil {
-		wg = readVotes(voteCh, len(froms))
-	}
-	for _, from := range froms {
-		vote := signVote(from, voteType, hash, header)
-		addVoteToFrom(to, from, vote)
-	}
-
-	if voteCh != nil {
-		<-wg
-	}
-}
-
-func signAddVoteToFrom(voteType byte, to *ConsensusState, from *validatorStub, hash []byte, header types.PartSetHeader, voteCh chan interface{}) *types.Vote {
-	var wg chan struct{} // when done reading all votes
-	if voteCh != nil {
-		wg = readVotes(voteCh, 1)
-	}
-	vote := signVote(from, voteType, hash, header)
-	addVoteToFrom(to, from, vote)
-	if voteCh != nil {
-		<-wg
-	}
-	return vote
-}
-
-func ensureNoNewStep(stepCh chan interface{}) {
-	timeout := time.NewTicker(ensureTimeout * time.Second)
-	select {
-	case <-timeout.C:
-		break
-	case <-stepCh:
-		panic("We should be stuck waiting for more votes, not moving to the next step")
-	}
-}
-
-/*
-func ensureNoNewStep(t *testing.T, cs *ConsensusState) {
-	timeout := time.NewTicker(ensureTimeout * time.Second)
-	select {
-	case <-timeout.C:
-		break
-	case <-cs.NewStepCh():
-		panic("We should be stuck waiting for more votes, not moving to the next step")
-	}
-}
-
-func ensureNewStep(t *testing.T, cs *ConsensusState) *RoundState {
-	timeout := time.NewTicker(ensureTimeout * time.Second)
-	select {
-	case <-timeout.C:
-		panic("We should have gone to the next step, not be stuck waiting")
-	case rs := <-cs.NewStepCh():
-		return rs
-	}
-}
-
-func waitFor(t *testing.T, cs *ConsensusState, height int, round int, step RoundStepType) {
-	for {
-		rs := ensureNewStep(t, cs)
-		if CompareHRS(rs.Height, rs.Round, rs.Step, height, round, step) < 0 {
-			continue
-		} else {
-			break
-		}
-	}
-}
-*/
-
 func incrementHeight(vss ...*validatorStub) {
 	for _, vs := range vss {
 		vs.Height += 1
@@ -245,19 +105,54 @@ func incrementRound(vss ...*validatorStub) {
 	}
 }
 
+//-------------------------------------------------------------------------------
+// Functions for transitioning the consensus state
+
+func startTestRound(cs *ConsensusState, height, round int) {
+	cs.enterNewRound(height, round)
+	cs.startRoutines(0)
+}
+
+// Create proposal block from cs1 but sign it with vs
+func decideProposal(cs1 *ConsensusState, vs *validatorStub, height, round int) (proposal *types.Proposal, block *types.Block) {
+	block, blockParts := cs1.createProposalBlock()
+	if block == nil { // on error
+		panic("error creating proposal block")
+	}
+
+	// Make proposal
+	polRound, polBlockID := cs1.Votes.POLInfo()
+	proposal = types.NewProposal(height, round, blockParts.Header(), polRound, polBlockID)
+	if err := vs.SignProposal(config.ChainID, proposal); err != nil {
+		panic(err)
+	}
+	return
+}
+
+func addVotes(to *ConsensusState, votes ...*types.Vote) {
+	for _, vote := range votes {
+		to.peerMsgQueue <- msgInfo{Msg: &VoteMessage{vote}}
+	}
+}
+
+func signAddVotes(to *ConsensusState, voteType byte, hash []byte, header types.PartSetHeader, vss ...*validatorStub) {
+	votes := signVotes(voteType, hash, header, vss...)
+	addVotes(to, votes...)
+}
+
 func validatePrevote(t *testing.T, cs *ConsensusState, round int, privVal *validatorStub, blockHash []byte) {
 	prevotes := cs.Votes.Prevotes(round)
 	var vote *types.Vote
-	if vote = prevotes.GetByAddress(privVal.Address); vote == nil {
+	if vote = prevotes.GetByAddress(privVal.GetAddress()); vote == nil {
 		panic("Failed to find prevote from validator")
 	}
 	if blockHash == nil {
-		if vote.BlockHash != nil {
-			panic(fmt.Sprintf("Expected prevote to be for nil, got %X", vote.BlockHash))
+		if vote.BlockID.Hash != nil {
+			panic(fmt.Sprintf("Expected prevote to be for nil, got %X", vote.BlockID.Hash))
 		}
 	} else {
-		if !bytes.Equal(vote.BlockHash, blockHash) {
-			panic(fmt.Sprintf("Expected prevote to be for %X, got %X", blockHash, vote.BlockHash))
+		if !bytes.Equal(vote.BlockID.Hash, blockHash) {
+			panic(fmt.Sprintf("Expected prevote to be for %X, got %X", blockHash, vote.BlockID.Hash))
 		}
 	}
 }
@@ -265,27 +160,27 @@ func validatePrevote(t *testing.T, cs *ConsensusState, round int, privVal *valid
 func validateLastPrecommit(t *testing.T, cs *ConsensusState, privVal *validatorStub, blockHash []byte) {
 	votes := cs.LastCommit
 	var vote *types.Vote
-	if vote = votes.GetByAddress(privVal.Address); vote == nil {
+	if vote = votes.GetByAddress(privVal.GetAddress()); vote == nil {
 		panic("Failed to find precommit from validator")
 	}
-	if !bytes.Equal(vote.BlockHash, blockHash) {
-		panic(fmt.Sprintf("Expected precommit to be for %X, got %X", blockHash, vote.BlockHash))
+	if !bytes.Equal(vote.BlockID.Hash, blockHash) {
+		panic(fmt.Sprintf("Expected precommit to be for %X, got %X", blockHash, vote.BlockID.Hash))
 	}
 }
 
 func validatePrecommit(t *testing.T, cs *ConsensusState, thisRound, lockRound int, privVal *validatorStub, votedBlockHash, lockedBlockHash []byte) {
 	precommits := cs.Votes.Precommits(thisRound)
 	var vote *types.Vote
-	if vote = precommits.GetByAddress(privVal.Address); vote == nil {
+	if vote = precommits.GetByAddress(privVal.GetAddress()); vote == nil {
 		panic("Failed to find precommit from validator")
 	}
 
 	if votedBlockHash == nil {
-		if vote.BlockHash != nil {
+		if vote.BlockID.Hash != nil {
 			panic("Expected precommit to be for nil")
 		}
 	} else {
-		if !bytes.Equal(vote.BlockHash, votedBlockHash) {
+		if !bytes.Equal(vote.BlockID.Hash, votedBlockHash) {
 			panic("Expected precommit to be for proposal block")
 		}
 	}
@@ -311,39 +206,77 @@ func validatePrevoteAndPrecommit(t *testing.T, cs *ConsensusState, thisRound, lo
 	cs.mtx.Unlock()
 }
 
-func fixedConsensusState() *ConsensusState {
-	stateDB := dbm.NewMemDB()
-	state := sm.MakeGenesisStateFromFile(stateDB, config.GetString("genesis_file"))
-	privValidatorFile := config.GetString("priv_validator_file")
-	privValidator := types.LoadOrGenPrivValidator(privValidatorFile)
-	privValidator.Reset()
-	cs := newConsensusState(state, privValidator, counter.NewCounterApplication(true))
-	return cs
+// genesis
+func subscribeToVoter(cs *ConsensusState, addr []byte) chan interface{} {
+	voteCh0 := subscribeToEvent(cs.evsw, "tester", types.EventStringVote(), 1)
+	voteCh := make(chan interface{})
+	go func() {
+		for {
+			v := <-voteCh0
+			vote := v.(types.TMEventData).Unwrap().(types.EventDataVote)
+			// we only fire for our own votes
+			if bytes.Equal(addr, vote.Vote.ValidatorAddress) {
+				voteCh <- v
+			}
+		}
+	}()
+	return voteCh
 }
 
-func newConsensusState(state *sm.State, pv *types.PrivValidator, app tmsp.Application) *ConsensusState {
+//-------------------------------------------------------------------------------
+// consensus states
+
+func newConsensusState(state *sm.State, pv types.PrivValidator, app abci.Application) *ConsensusState {
+	return newConsensusStateWithConfig(config, state, pv, app)
+}
+
+func newConsensusStateWithConfig(thisConfig *cfg.Config, state *sm.State, pv types.PrivValidator, app abci.Application) *ConsensusState {
 	// Get BlockStore
 	blockDB := dbm.NewMemDB()
 	blockStore := bc.NewBlockStore(blockDB)
 
 	// one for mempool, one for consensus
 	mtx := new(sync.Mutex)
-	proxyAppConnMem := tmspcli.NewLocalClient(mtx, app)
-	proxyAppConnCon := tmspcli.NewLocalClient(mtx, app)
+	proxyAppConnMem := abcicli.NewLocalClient(mtx, app)
+	proxyAppConnCon := abcicli.NewLocalClient(mtx, app)
 
 	// Make Mempool
-	mempool := mempl.NewMempool(config, proxyAppConnMem)
+	mempool := mempl.NewMempool(thisConfig.Mempool, proxyAppConnMem, 0)
+	mempool.SetLogger(log.TestingLogger().With("module", "mempool"))
+	if thisConfig.Consensus.WaitForTxs() {
+		mempool.EnableTxsAvailable()
+	}
 
 	// Make ConsensusReactor
-	cs := NewConsensusState(config, state, proxyAppConnCon, blockStore, mempool)
+	cs := NewConsensusState(thisConfig.Consensus, state, proxyAppConnCon, blockStore, mempool)
+	cs.SetLogger(log.TestingLogger())
 	cs.SetPrivValidator(pv)
 
-	evsw := events.NewEventSwitch()
+	evsw := types.NewEventSwitch()
+	evsw.SetLogger(log.TestingLogger().With("module", "events"))
 	cs.SetEventSwitch(evsw)
 	evsw.Start()
 	return cs
 }
 
+func loadPrivValidator(config *cfg.Config) *types.PrivValidatorFS {
+	privValidatorFile := config.PrivValidatorFile()
+	ensureDir(path.Dir(privValidatorFile), 0700)
+	privValidator := types.LoadOrGenPrivValidatorFS(privValidatorFile)
+	privValidator.Reset()
+	return privValidator
+}
+
+func fixedConsensusStateDummy() *ConsensusState {
+	stateDB := dbm.NewMemDB()
+	state, _ := sm.MakeGenesisStateFromFile(stateDB, config.GenesisFile())
+	state.SetLogger(log.TestingLogger().With("module", "state"))
+	privValidator := loadPrivValidator(config)
+	cs := newConsensusState(state, privValidator, dummy.NewDummyApplication())
+	cs.SetLogger(log.TestingLogger())
+	return cs
+}
+
 func randConsensusState(nValidators int) (*ConsensusState, []*validatorStub) {
 	// Get State
 	state, privVals := randGenesisState(nValidators, false, 10)
@@ -351,9 +284,10 @@ func randConsensusState(nValidators int) (*ConsensusState, []*validatorStub) {
 	vss := make([]*validatorStub, nValidators)
 
 	cs := newConsensusState(state, privVals[0], counter.NewCounterApplication(true))
+	cs.SetLogger(log.TestingLogger())
 
 	for i := 0; i < nValidators; i++ {
-		vss[i] = NewValidatorStub(privVals[i])
+		vss[i] = NewValidatorStub(privVals[i], i)
 	}
 	// since cs1 starts at 1
 	incrementHeight(vss[1:]...)
@@ -361,62 +295,196 @@ func randConsensusState(nValidators int) (*ConsensusState, []*validatorStub) {
 	return cs, vss
 }
 
-func subscribeToVoter(cs *ConsensusState, addr []byte) chan interface{} {
-	voteCh0 := subscribeToEvent(cs.evsw, "tester", types.EventStringVote(), 1)
-	voteCh := make(chan interface{})
-	go func() {
-		for {
-			v := <-voteCh0
-			vote := v.(types.EventDataVote)
-			// we only fire for our own votes
-			if bytes.Equal(addr, vote.Address) {
-				voteCh <- v
+//-------------------------------------------------------------------------------
+
+func ensureNoNewStep(stepCh chan interface{}) {
+	timer := time.NewTimer(ensureTimeout)
+	select {
+	case <-timer.C:
+		break
+	case <-stepCh:
+		panic("We should be stuck waiting, not moving to the next step")
+	}
+}
+
+func ensureNewStep(stepCh chan interface{}) {
+	timer := time.NewTimer(ensureTimeout)
+	select {
+	case <-timer.C:
+		panic("We shouldnt be stuck waiting")
+	case <-stepCh:
+		break
+	}
+}
+
+//-------------------------------------------------------------------------------
+// consensus nets
+
+// consensusLogger is a TestingLogger which uses a different
+// color for each validator ("validator" key must exist).
+func consensusLogger() log.Logger {
+	return log.TestingLoggerWithColorFn(func(keyvals ...interface{}) term.FgBgColor {
+		for i := 0; i < len(keyvals)-1; i += 2 {
+			if keyvals[i] == "validator" {
+				return term.FgBgColor{Fg: term.Color(uint8(keyvals[i+1].(int) + 1))}
 			}
 		}
-	}()
-	return voteCh
+		return term.FgBgColor{}
+	})
 }
 
-func readVotes(ch chan interface{}, reads int) chan struct{} {
-	wg := make(chan struct{})
-	go func() {
-		for i := 0; i < reads; i++ {
-			<-ch // read the precommit event
+func randConsensusNet(nValidators int, testName string, tickerFunc func() TimeoutTicker, appFunc func() abci.Application, configOpts ...func(*cfg.Config)) []*ConsensusState {
+	genDoc, privVals := randGenesisDoc(nValidators, false, 10)
+	css := make([]*ConsensusState, nValidators)
+	logger := consensusLogger()
+	for i := 0; i < nValidators; i++ {
+		db := dbm.NewMemDB() // each state needs its own db
+		state, _ := sm.MakeGenesisState(db, genDoc)
+		state.SetLogger(logger.With("module", "state", "validator", i))
+		state.Save()
+		thisConfig := ResetConfig(cmn.Fmt("%s_%d", testName, i))
+		for _, opt := range configOpts {
+			opt(thisConfig)
 		}
-		close(wg)
-	}()
-	return wg
+		ensureDir(path.Dir(thisConfig.Consensus.WalFile()), 0700) // dir for wal
+		app := appFunc()
+		vals := types.TM2PB.Validators(state.Validators)
+		app.InitChain(abci.RequestInitChain{Validators: vals})
+
+		css[i] = newConsensusStateWithConfig(thisConfig, state, privVals[i], app)
+		css[i].SetLogger(logger.With("validator", i))
+		css[i].SetTimeoutTicker(tickerFunc())
+	}
+	return css
 }
 
-func randGenesisState(numValidators int, randPower bool, minPower int64) (*sm.State, []*types.PrivValidator) {
-	db := dbm.NewMemDB()
-	genDoc, privValidators := randGenesisDoc(numValidators, randPower, minPower)
-	s0 := sm.MakeGenesisState(db, genDoc)
-	s0.Save()
-	return s0, privValidators
+// nPeers = nValidators + nNotValidator
+func randConsensusNetWithPeers(nValidators, nPeers int, testName string, tickerFunc func() TimeoutTicker, appFunc func() abci.Application) []*ConsensusState {
+	genDoc, privVals := randGenesisDoc(nValidators, false, int64(testMinPower))
+	css := make([]*ConsensusState, nPeers)
+	for i := 0; i < nPeers; i++ {
+		db := dbm.NewMemDB() // each state needs its own db
+		state, _ := sm.MakeGenesisState(db, genDoc)
+		state.SetLogger(log.TestingLogger().With("module", "state"))
+		state.Save()
+		thisConfig := ResetConfig(cmn.Fmt("%s_%d", testName, i))
+		ensureDir(path.Dir(thisConfig.Consensus.WalFile()), 0700) // dir for wal
+		var privVal types.PrivValidator
+		if i < nValidators {
+			privVal = privVals[i]
+		} else {
+			_, tempFilePath := cmn.Tempfile("priv_validator_")
+			privVal = types.GenPrivValidatorFS(tempFilePath)
+		}
+
+		app := appFunc()
+		vals := types.TM2PB.Validators(state.Validators)
+		app.InitChain(abci.RequestInitChain{Validators: vals})
+
+		css[i] = newConsensusStateWithConfig(thisConfig, state, privVal, app)
+		css[i].SetLogger(log.TestingLogger())
+		css[i].SetTimeoutTicker(tickerFunc())
+	}
+	return css
 }
 
-func randGenesisDoc(numValidators int, randPower bool, minPower int64) (*types.GenesisDoc, []*types.PrivValidator) {
+func getSwitchIndex(switches []*p2p.Switch, peer p2p.Peer) int {
+	for i, s := range switches {
+		if bytes.Equal(peer.NodeInfo().PubKey.Address(), s.NodeInfo().PubKey.Address()) {
+			return i
+		}
+	}
+	panic("didnt find peer in switches")
+	return -1
+}
+
+//-------------------------------------------------------------------------------
+// genesis
+
+func randGenesisDoc(numValidators int, randPower bool, minPower int64) (*types.GenesisDoc, []*types.PrivValidatorFS) {
 	validators := make([]types.GenesisValidator, numValidators)
-	privValidators := make([]*types.PrivValidator, numValidators)
+	privValidators := make([]*types.PrivValidatorFS, numValidators)
 	for i := 0; i < numValidators; i++ {
 		val, privVal := types.RandValidator(randPower, minPower)
 		validators[i] = types.GenesisValidator{
 			PubKey: val.PubKey,
-			Amount: val.VotingPower,
+			Power:  val.VotingPower,
 		}
 		privValidators[i] = privVal
 	}
 	sort.Sort(types.PrivValidatorsByAddress(privValidators))
 	return &types.GenesisDoc{
 		GenesisTime: time.Now(),
-		ChainID:     config.GetString("chain_id"),
+		ChainID:     config.ChainID,
 		Validators:  validators,
 	}, privValidators
-
 }
 
-func startTestRound(cs *ConsensusState, height, round int) {
-	cs.enterNewRound(height, round)
-	cs.startRoutines(0)
+func randGenesisState(numValidators int, randPower bool, minPower int64) (*sm.State, []*types.PrivValidatorFS) {
+	genDoc, privValidators := randGenesisDoc(numValidators, randPower, minPower)
+	db := dbm.NewMemDB()
+	s0, _ := sm.MakeGenesisState(db, genDoc)
+	s0.SetLogger(log.TestingLogger().With("module", "state"))
+	s0.Save()
+	return s0, privValidators
+}
+
+//------------------------------------
+// mock ticker
+
+func newMockTickerFunc(onlyOnce bool) func() TimeoutTicker {
+	return func() TimeoutTicker {
+		return &mockTicker{
+			c:        make(chan timeoutInfo, 10),
+			onlyOnce: onlyOnce,
+		}
+	}
+}
+
+// mock ticker only fires on RoundStepNewHeight
+// and only once if onlyOnce=true
+type mockTicker struct {
+	c chan timeoutInfo
+
+	mtx      sync.Mutex
+	onlyOnce bool
+	fired    bool
+}
+
+func (m *mockTicker) Start() (bool, error) {
+	return true, nil
+}
+
+func (m *mockTicker) Stop() bool {
+	return true
+}
+
+func (m *mockTicker) ScheduleTimeout(ti timeoutInfo) {
+	m.mtx.Lock()
+	defer m.mtx.Unlock()
+	if m.onlyOnce && m.fired {
+		return
+	}
+	if ti.Step == cstypes.RoundStepNewHeight {
+		m.c <- ti
+		m.fired = true
+	}
+}
+
+func (m *mockTicker) Chan() <-chan timeoutInfo {
+	return m.c
+}
+
+func (mockTicker) SetLogger(log.Logger) {
+}
+
+//------------------------------------
+
+func newCounter() abci.Application {
+	return counter.NewCounterApplication(true)
+}
+
+func newPersistentDummy() abci.Application {
+	dir, _ := ioutil.TempDir("/tmp", "persistent-dummy")
+	return dummy.NewPersistentDummyApplication(dir)
 }

consensus/height_vote_set_test.go

@@ -1,53 +0,0 @@
-package consensus
-
-import (
-	"testing"
-
-	. "github.com/tendermint/go-common"
-	"github.com/tendermint/tendermint/config/tendermint_test"
-	"github.com/tendermint/tendermint/types"
-)
-
-func init() {
-	config = tendermint_test.ResetConfig("consensus_height_vote_set_test")
-}
-
-func TestPeerCatchupRounds(t *testing.T) {
-	valSet, privVals := types.RandValidatorSet(10, 1)
-
-	hvs := NewHeightVoteSet(config.GetString("chain_id"), 1, valSet)
-
-	vote999_0 := makeVoteHR(t, 1, 999, privVals[0])
-	added, _, err := hvs.AddByIndex(0, vote999_0, "peer1")
-	if !added || err != nil {
-		t.Error("Expected to successfully add vote from peer", added, err)
-	}
-
-	vote1000_0 := makeVoteHR(t, 1, 1000, privVals[0])
-	added, _, err = hvs.AddByIndex(0, vote1000_0, "peer1")
-	if added {
-		t.Error("Expected to *not* add vote from peer, too many catchup rounds.")
-	}
-
-	added, _, err = hvs.AddByIndex(0, vote1000_0, "peer2")
-	if !added || err != nil {
-		t.Error("Expected to successfully add vote from another peer")
-	}
-
-}
-
-func makeVoteHR(t *testing.T, height, round int, privVal *types.PrivValidator) *types.Vote {
-	vote := &types.Vote{
-		Height:    height,
-		Round:     round,
-		Type:      types.VoteTypePrecommit,
-		BlockHash: []byte("fakehash"),
-	}
-	chainID := config.GetString("chain_id")
-	err := privVal.SignVote(chainID, vote)
-	if err != nil {
-		panic(Fmt("Error signing vote: %v", err))
-		return nil
-	}
-	return vote
-}

consensus/log.go

@@ -1,18 +0,0 @@
-package consensus
-
-import (
-	"github.com/tendermint/go-logger"
-)
-
-var log = logger.New("module", "consensus")
-
-/*
-func init() {
-	log.SetHandler(
-		logger.LvlFilterHandler(
-			logger.LvlDebug,
-			logger.BypassHandler(),
-		),
-	)
-}
-*/

consensus/mempool_test.go

@@ -2,19 +2,93 @@ package consensus
 
 import (
 	"encoding/binary"
-	//	"math/rand"
 	"testing"
 	"time"
 
-	"github.com/tendermint/tendermint/config/tendermint_test"
+	abci "github.com/tendermint/abci/types"
 	"github.com/tendermint/tendermint/types"
-	tmsp "github.com/tendermint/tmsp/types"
 
-	. "github.com/tendermint/go-common"
+	cmn "github.com/tendermint/tmlibs/common"
 )
 
 func init() {
-	config = tendermint_test.ResetConfig("consensus_mempool_test")
+	config = ResetConfig("consensus_mempool_test")
+}
+
+func TestNoProgressUntilTxsAvailable(t *testing.T) {
+	config := ResetConfig("consensus_mempool_txs_available_test")
+	config.Consensus.CreateEmptyBlocks = false
+	state, privVals := randGenesisState(1, false, 10)
+	cs := newConsensusStateWithConfig(config, state, privVals[0], NewCounterApplication())
+	cs.mempool.EnableTxsAvailable()
+	height, round := cs.Height, cs.Round
+	newBlockCh := subscribeToEvent(cs.evsw, "tester", types.EventStringNewBlock(), 1)
+	startTestRound(cs, height, round)
+
+	ensureNewStep(newBlockCh) // first block gets committed
+	ensureNoNewStep(newBlockCh)
+	deliverTxsRange(cs, 0, 2)
+	ensureNewStep(newBlockCh) // commit txs
+	ensureNewStep(newBlockCh) // commit updated app hash
+	ensureNoNewStep(newBlockCh)
+
+}
+
+func TestProgressAfterCreateEmptyBlocksInterval(t *testing.T) {
+	config := ResetConfig("consensus_mempool_txs_available_test")
+	config.Consensus.CreateEmptyBlocksInterval = int(ensureTimeout.Seconds())
+	state, privVals := randGenesisState(1, false, 10)
+	cs := newConsensusStateWithConfig(config, state, privVals[0], NewCounterApplication())
+	cs.mempool.EnableTxsAvailable()
+	height, round := cs.Height, cs.Round
+	newBlockCh := subscribeToEvent(cs.evsw, "tester", types.EventStringNewBlock(), 1)
+	startTestRound(cs, height, round)
+
+	ensureNewStep(newBlockCh)   // first block gets committed
+	ensureNoNewStep(newBlockCh) // then we dont make a block ...
+	ensureNewStep(newBlockCh)   // until the CreateEmptyBlocksInterval has passed
+}
+
+func TestProgressInHigherRound(t *testing.T) {
+	config := ResetConfig("consensus_mempool_txs_available_test")
+	config.Consensus.CreateEmptyBlocks = false
+	state, privVals := randGenesisState(1, false, 10)
+	cs := newConsensusStateWithConfig(config, state, privVals[0], NewCounterApplication())
+	cs.mempool.EnableTxsAvailable()
+	height, round := cs.Height, cs.Round
+	newBlockCh := subscribeToEvent(cs.evsw, "tester", types.EventStringNewBlock(), 1)
+	newRoundCh := subscribeToEvent(cs.evsw, "tester", types.EventStringNewRound(), 1)
+	timeoutCh := subscribeToEvent(cs.evsw, "tester", types.EventStringTimeoutPropose(), 1)
+	cs.setProposal = func(proposal *types.Proposal) error {
+		if cs.Height == 2 && cs.Round == 0 {
+			// dont set the proposal in round 0 so we timeout and
+			// go to next round
+			cs.Logger.Info("Ignoring set proposal at height 2, round 0")
+			return nil
+		}
+		return cs.defaultSetProposal(proposal)
+	}
+	startTestRound(cs, height, round)
+
+	ensureNewStep(newRoundCh) // first round at first height
+	ensureNewStep(newBlockCh) // first block gets committed
+	ensureNewStep(newRoundCh) // first round at next height
+	deliverTxsRange(cs, 0, 2) // we deliver txs, but dont set a proposal so we get the next round
+	<-timeoutCh
+	ensureNewStep(newRoundCh) // wait for the next round
+	ensureNewStep(newBlockCh) // now we can commit the block
+}
+
+func deliverTxsRange(cs *ConsensusState, start, end int) {
+	// Deliver some txs.
+	for i := start; i < end; i++ {
+		txBytes := make([]byte, 8)
+		binary.BigEndian.PutUint64(txBytes, uint64(i))
+		err := cs.mempool.CheckTx(txBytes, nil)
+		if err != nil {
+			panic(cmn.Fmt("Error after CheckTx: %v", err))
+		}
+	}
 }
 
 func TestTxConcurrentWithCommit(t *testing.T) {
@@ -24,36 +98,83 @@ func TestTxConcurrentWithCommit(t *testing.T) {
 	height, round := cs.Height, cs.Round
 	newBlockCh := subscribeToEvent(cs.evsw, "tester", types.EventStringNewBlock(), 1)
 
-	appendTxsRange := func(start, end int) {
-		// Append some txs.
-		for i := start; i < end; i++ {
-			txBytes := make([]byte, 8)
-			binary.BigEndian.PutUint64(txBytes, uint64(i))
-			err := cs.mempool.CheckTx(txBytes, nil)
-			if err != nil {
-				panic(Fmt("Error after CheckTx: %v", err))
-			}
-			//	time.Sleep(time.Microsecond * time.Duration(rand.Int63n(3000)))
-		}
-	}
-
 	NTxs := 10000
-	go appendTxsRange(0, NTxs)
+	go deliverTxsRange(cs, 0, NTxs)
 
 	startTestRound(cs, height, round)
 	ticker := time.NewTicker(time.Second * 20)
 	for nTxs := 0; nTxs < NTxs; {
 		select {
 		case b := <-newBlockCh:
-			nTxs += b.(types.EventDataNewBlock).Block.Header.NumTxs
+			nTxs += b.(types.TMEventData).Unwrap().(types.EventDataNewBlock).Block.Header.NumTxs
 		case <-ticker.C:
 			panic("Timed out waiting to commit blocks with transactions")
 		}
 	}
 }
 
+func TestRmBadTx(t *testing.T) {
+	state, privVals := randGenesisState(1, false, 10)
+	app := NewCounterApplication()
+	cs := newConsensusState(state, privVals[0], app)
+
+	// increment the counter by 1
+	txBytes := make([]byte, 8)
+	binary.BigEndian.PutUint64(txBytes, uint64(0))
+	app.DeliverTx(txBytes)
+	app.Commit()
+
+	ch := make(chan struct{})
+	cbCh := make(chan struct{})
+	go func() {
+		// Try to send the tx through the mempool.
+		// CheckTx should not err, but the app should return a bad abci code
+		// and the tx should get removed from the pool
+		err := cs.mempool.CheckTx(txBytes, func(r *abci.Response) {
+			if r.GetCheckTx().Code != abci.CodeType_BadNonce {
+				t.Fatalf("expected checktx to return bad nonce, got %v", r)
+			}
+			cbCh <- struct{}{}
+		})
+		if err != nil {
+			t.Fatal("Error after CheckTx: %v", err)
+		}
+
+		// check for the tx
+		for {
+			time.Sleep(time.Second)
+			txs := cs.mempool.Reap(1)
+			if len(txs) == 0 {
+				ch <- struct{}{}
+				return
+			}
+
+		}
+	}()
+
+	// Wait until the tx returns
+	ticker := time.After(time.Second * 5)
+	select {
+	case <-cbCh:
+		// success
+	case <-ticker:
+		t.Fatalf("Timed out waiting for tx to return")
+	}
+
+	// Wait until the tx is removed
+	ticker = time.After(time.Second * 5)
+	select {
+	case <-ch:
+		// success
+	case <-ticker:
+		t.Fatalf("Timed out waiting for tx to be removed")
+	}
+}
+
 // CounterApplication that maintains a mempool state and resets it upon commit
 type CounterApplication struct {
+	abci.BaseApplication
+
 	txCount        int
 	mempoolTxCount int
 }
@@ -62,49 +183,37 @@ func NewCounterApplication() *CounterApplication {
 	return &CounterApplication{}
 }
 
-func (app *CounterApplication) Info() string {
-	return Fmt("txs:%v", app.txCount)
+func (app *CounterApplication) Info(req abci.RequestInfo) abci.ResponseInfo {
+	return abci.ResponseInfo{Data: cmn.Fmt("txs:%v", app.txCount)}
 }
 
-func (app *CounterApplication) SetOption(key string, value string) (log string) {
-	return ""
-}
-
-func (app *CounterApplication) AppendTx(tx []byte) tmsp.Result {
+func (app *CounterApplication) DeliverTx(tx []byte) abci.Result {
 	return runTx(tx, &app.txCount)
 }
 
-func (app *CounterApplication) CheckTx(tx []byte) tmsp.Result {
+func (app *CounterApplication) CheckTx(tx []byte) abci.Result {
 	return runTx(tx, &app.mempoolTxCount)
 }
 
-func runTx(tx []byte, countPtr *int) tmsp.Result {
+func runTx(tx []byte, countPtr *int) abci.Result {
 	count := *countPtr
 	tx8 := make([]byte, 8)
 	copy(tx8[len(tx8)-len(tx):], tx)
 	txValue := binary.BigEndian.Uint64(tx8)
 	if txValue != uint64(count) {
-		return tmsp.Result{
-			Code: tmsp.CodeType_BadNonce,
-			Data: nil,
-			Log:  Fmt("Invalid nonce. Expected %v, got %v", count, txValue),
-		}
+		return abci.ErrBadNonce.AppendLog(cmn.Fmt("Invalid nonce. Expected %v, got %v", count, txValue))
 	}
 	*countPtr += 1
-	return tmsp.OK
+	return abci.OK
 }
 
-func (app *CounterApplication) Commit() tmsp.Result {
+func (app *CounterApplication) Commit() abci.Result {
 	app.mempoolTxCount = app.txCount
 	if app.txCount == 0 {
-		return tmsp.OK
+		return abci.OK
 	} else {
 		hash := make([]byte, 8)
 		binary.BigEndian.PutUint64(hash, uint64(app.txCount))
-		return tmsp.NewResultOK(hash, "")
+		return abci.NewResultOK(hash, "")
 	}
 }
-
-func (app *CounterApplication) Query(query []byte) tmsp.Result {
-	return tmsp.NewResultOK(nil, Fmt("Query is not supported"))
-}

consensus/reactor_test.go

@@ -0,0 +1,340 @@
+package consensus
+
+import (
+	"fmt"
+	"sync"
+	"testing"
+	"time"
+
+	"github.com/tendermint/abci/example/dummy"
+	"github.com/tendermint/tmlibs/events"
+
+	cfg "github.com/tendermint/tendermint/config"
+	"github.com/tendermint/tendermint/p2p"
+	"github.com/tendermint/tendermint/types"
+)
+
+func init() {
+	config = ResetConfig("consensus_reactor_test")
+}
+
+//----------------------------------------------
+// in-process testnets
+
+func startConsensusNet(t *testing.T, css []*ConsensusState, N int, subscribeEventRespond bool) ([]*ConsensusReactor, []chan interface{}) {
+	reactors := make([]*ConsensusReactor, N)
+	eventChans := make([]chan interface{}, N)
+	logger := consensusLogger()
+	for i := 0; i < N; i++ {
+		reactors[i] = NewConsensusReactor(css[i], true) // so we dont start the consensus states
+		reactors[i].SetLogger(logger.With("validator", i))
+
+		eventSwitch := events.NewEventSwitch()
+		eventSwitch.SetLogger(logger.With("module", "events", "validator", i))
+		_, err := eventSwitch.Start()
+		if err != nil {
+			t.Fatalf("Failed to start switch: %v", err)
+		}
+
+		reactors[i].SetEventSwitch(eventSwitch)
+		if subscribeEventRespond {
+			eventChans[i] = subscribeToEventRespond(eventSwitch, "tester", types.EventStringNewBlock())
+		} else {
+			eventChans[i] = subscribeToEvent(eventSwitch, "tester", types.EventStringNewBlock(), 1)
+		}
+	}
+	// make connected switches and start all reactors
+	p2p.MakeConnectedSwitches(config.P2P, N, func(i int, s *p2p.Switch) *p2p.Switch {
+		s.AddReactor("CONSENSUS", reactors[i])
+		return s
+	}, p2p.Connect2Switches)
+
+	// now that everyone is connected,  start the state machines
+	// If we started the state machines before everyone was connected,
+	// we'd block when the cs fires NewBlockEvent and the peers are trying to start their reactors
+	for i := 0; i < N; i++ {
+		s := reactors[i].conS.GetState()
+		reactors[i].SwitchToConsensus(s, 0)
+	}
+	return reactors, eventChans
+}
+
+func stopConsensusNet(reactors []*ConsensusReactor) {
+	for _, r := range reactors {
+		r.Switch.Stop()
+	}
+}
+
+// Ensure a testnet makes blocks
+func TestReactor(t *testing.T) {
+	N := 4
+	css := randConsensusNet(N, "consensus_reactor_test", newMockTickerFunc(true), newCounter)
+	reactors, eventChans := startConsensusNet(t, css, N, false)
+	defer stopConsensusNet(reactors)
+	// wait till everyone makes the first new block
+	timeoutWaitGroup(t, N, func(wg *sync.WaitGroup, j int) {
+		<-eventChans[j]
+		wg.Done()
+	}, css)
+}
+
+// Ensure a testnet sends proposal heartbeats and makes blocks when there are txs
+func TestReactorProposalHeartbeats(t *testing.T) {
+	N := 4
+	css := randConsensusNet(N, "consensus_reactor_test", newMockTickerFunc(true), newCounter,
+		func(c *cfg.Config) {
+			c.Consensus.CreateEmptyBlocks = false
+		})
+	reactors, eventChans := startConsensusNet(t, css, N, false)
+	defer stopConsensusNet(reactors)
+	heartbeatChans := make([]chan interface{}, N)
+	for i := 0; i < N; i++ {
+		heartbeatChans[i] = subscribeToEvent(css[i].evsw, "tester", types.EventStringProposalHeartbeat(), 1)
+	}
+	// wait till everyone sends a proposal heartbeat
+	timeoutWaitGroup(t, N, func(wg *sync.WaitGroup, j int) {
+		<-heartbeatChans[j]
+		wg.Done()
+	}, css)
+
+	// send a tx
+	css[3].mempool.CheckTx([]byte{1, 2, 3}, nil)
+
+	// wait till everyone makes the first new block
+	timeoutWaitGroup(t, N, func(wg *sync.WaitGroup, j int) {
+		<-eventChans[j]
+		wg.Done()
+	}, css)
+}
+
+//-------------------------------------------------------------
+// ensure we can make blocks despite cycling a validator set
+
+func TestVotingPowerChange(t *testing.T) {
+	nVals := 4
+	css := randConsensusNet(nVals, "consensus_voting_power_changes_test", newMockTickerFunc(true), newPersistentDummy)
+	reactors, eventChans := startConsensusNet(t, css, nVals, true)
+	defer stopConsensusNet(reactors)
+
+	// map of active validators
+	activeVals := make(map[string]struct{})
+	for i := 0; i < nVals; i++ {
+		activeVals[string(css[i].privValidator.GetAddress())] = struct{}{}
+	}
+
+	// wait till everyone makes block 1
+	timeoutWaitGroup(t, nVals, func(wg *sync.WaitGroup, j int) {
+		<-eventChans[j]
+		eventChans[j] <- struct{}{}
+		wg.Done()
+	}, css)
+
+	//---------------------------------------------------------------------------
+	t.Log("---------------------------- Testing changing the voting power of one validator a few times")
+
+	val1PubKey := css[0].privValidator.GetPubKey()
+	updateValidatorTx := dummy.MakeValSetChangeTx(val1PubKey.Bytes(), 25)
+	previousTotalVotingPower := css[0].GetRoundState().LastValidators.TotalVotingPower()
+
+	waitForAndValidateBlock(t, nVals, activeVals, eventChans, css, updateValidatorTx)
+	waitForAndValidateBlock(t, nVals, activeVals, eventChans, css)
+	waitForAndValidateBlock(t, nVals, activeVals, eventChans, css)
+	waitForAndValidateBlock(t, nVals, activeVals, eventChans, css)
+
+	if css[0].GetRoundState().LastValidators.TotalVotingPower() == previousTotalVotingPower {
+		t.Fatalf("expected voting power to change (before: %d, after: %d)", previousTotalVotingPower, css[0].GetRoundState().LastValidators.TotalVotingPower())
+	}
+
+	updateValidatorTx = dummy.MakeValSetChangeTx(val1PubKey.Bytes(), 2)
+	previousTotalVotingPower = css[0].GetRoundState().LastValidators.TotalVotingPower()
+
+	waitForAndValidateBlock(t, nVals, activeVals, eventChans, css, updateValidatorTx)
+	waitForAndValidateBlock(t, nVals, activeVals, eventChans, css)
+	waitForAndValidateBlock(t, nVals, activeVals, eventChans, css)
+	waitForAndValidateBlock(t, nVals, activeVals, eventChans, css)
+
+	if css[0].GetRoundState().LastValidators.TotalVotingPower() == previousTotalVotingPower {
+		t.Fatalf("expected voting power to change (before: %d, after: %d)", previousTotalVotingPower, css[0].GetRoundState().LastValidators.TotalVotingPower())
+	}
+
+	updateValidatorTx = dummy.MakeValSetChangeTx(val1PubKey.Bytes(), 100)
+	previousTotalVotingPower = css[0].GetRoundState().LastValidators.TotalVotingPower()
+
+	waitForAndValidateBlock(t, nVals, activeVals, eventChans, css, updateValidatorTx)
+	waitForAndValidateBlock(t, nVals, activeVals, eventChans, css)
+	waitForAndValidateBlock(t, nVals, activeVals, eventChans, css)
+	waitForAndValidateBlock(t, nVals, activeVals, eventChans, css)
+
+	if css[0].GetRoundState().LastValidators.TotalVotingPower() == previousTotalVotingPower {
+		t.Fatalf("expected voting power to change (before: %d, after: %d)", previousTotalVotingPower, css[0].GetRoundState().LastValidators.TotalVotingPower())
+	}
+}
+
+func TestValidatorSetChanges(t *testing.T) {
+	nPeers := 7
+	nVals := 4
+	css := randConsensusNetWithPeers(nVals, nPeers, "consensus_val_set_changes_test", newMockTickerFunc(true), newPersistentDummy)
+	reactors, eventChans := startConsensusNet(t, css, nPeers, true)
+	defer stopConsensusNet(reactors)
+
+	// map of active validators
+	activeVals := make(map[string]struct{})
+	for i := 0; i < nVals; i++ {
+		activeVals[string(css[i].privValidator.GetAddress())] = struct{}{}
+	}
+
+	// wait till everyone makes block 1
+	timeoutWaitGroup(t, nPeers, func(wg *sync.WaitGroup, j int) {
+		<-eventChans[j]
+		eventChans[j] <- struct{}{}
+		wg.Done()
+	}, css)
+
+	//---------------------------------------------------------------------------
+	t.Log("---------------------------- Testing adding one validator")
+
+	newValidatorPubKey1 := css[nVals].privValidator.GetPubKey()
+	newValidatorTx1 := dummy.MakeValSetChangeTx(newValidatorPubKey1.Bytes(), uint64(testMinPower))
+
+	// wait till everyone makes block 2
+	// ensure the commit includes all validators
+	// send newValTx to change vals in block 3
+	waitForAndValidateBlock(t, nPeers, activeVals, eventChans, css, newValidatorTx1)
+
+	// wait till everyone makes block 3.
+	// it includes the commit for block 2, which is by the original validator set
+	waitForAndValidateBlock(t, nPeers, activeVals, eventChans, css)
+
+	// wait till everyone makes block 4.
+	// it includes the commit for block 3, which is by the original validator set
+	waitForAndValidateBlock(t, nPeers, activeVals, eventChans, css)
+
+	// the commits for block 4 should be with the updated validator set
+	activeVals[string(newValidatorPubKey1.Address())] = struct{}{}
+
+	// wait till everyone makes block 5
+	// it includes the commit for block 4, which should have the updated validator set
+	waitForAndValidateBlock(t, nPeers, activeVals, eventChans, css)
+
+	//---------------------------------------------------------------------------
+	t.Log("---------------------------- Testing changing the voting power of one validator")
+
+	updateValidatorPubKey1 := css[nVals].privValidator.GetPubKey()
+	updateValidatorTx1 := dummy.MakeValSetChangeTx(updateValidatorPubKey1.Bytes(), 25)
+	previousTotalVotingPower := css[nVals].GetRoundState().LastValidators.TotalVotingPower()
+
+	waitForAndValidateBlock(t, nPeers, activeVals, eventChans, css, updateValidatorTx1)
+	waitForAndValidateBlock(t, nPeers, activeVals, eventChans, css)
+	waitForAndValidateBlock(t, nPeers, activeVals, eventChans, css)
+	waitForAndValidateBlock(t, nPeers, activeVals, eventChans, css)
+
+	if css[nVals].GetRoundState().LastValidators.TotalVotingPower() == previousTotalVotingPower {
+		t.Errorf("expected voting power to change (before: %d, after: %d)", previousTotalVotingPower, css[nVals].GetRoundState().LastValidators.TotalVotingPower())
+	}
+
+	//---------------------------------------------------------------------------
+	t.Log("---------------------------- Testing adding two validators at once")
+
+	newValidatorPubKey2 := css[nVals+1].privValidator.GetPubKey()
+	newValidatorTx2 := dummy.MakeValSetChangeTx(newValidatorPubKey2.Bytes(), uint64(testMinPower))
+
+	newValidatorPubKey3 := css[nVals+2].privValidator.GetPubKey()
+	newValidatorTx3 := dummy.MakeValSetChangeTx(newValidatorPubKey3.Bytes(), uint64(testMinPower))
+
+	waitForAndValidateBlock(t, nPeers, activeVals, eventChans, css, newValidatorTx2, newValidatorTx3)
+	waitForAndValidateBlock(t, nPeers, activeVals, eventChans, css)
+	waitForAndValidateBlock(t, nPeers, activeVals, eventChans, css)
+	activeVals[string(newValidatorPubKey2.Address())] = struct{}{}
+	activeVals[string(newValidatorPubKey3.Address())] = struct{}{}
+	waitForAndValidateBlock(t, nPeers, activeVals, eventChans, css)
+
+	//---------------------------------------------------------------------------
+	t.Log("---------------------------- Testing removing two validators at once")
+
+	removeValidatorTx2 := dummy.MakeValSetChangeTx(newValidatorPubKey2.Bytes(), 0)
+	removeValidatorTx3 := dummy.MakeValSetChangeTx(newValidatorPubKey3.Bytes(), 0)
+
+	waitForAndValidateBlock(t, nPeers, activeVals, eventChans, css, removeValidatorTx2, removeValidatorTx3)
+	waitForAndValidateBlock(t, nPeers, activeVals, eventChans, css)
+	waitForAndValidateBlock(t, nPeers, activeVals, eventChans, css)
+	delete(activeVals, string(newValidatorPubKey2.Address()))
+	delete(activeVals, string(newValidatorPubKey3.Address()))
+	waitForAndValidateBlock(t, nPeers, activeVals, eventChans, css)
+}
+
+// Check we can make blocks with skip_timeout_commit=false
+func TestReactorWithTimeoutCommit(t *testing.T) {
+	N := 4
+	css := randConsensusNet(N, "consensus_reactor_with_timeout_commit_test", newMockTickerFunc(false), newCounter)
+	// override default SkipTimeoutCommit == true for tests
+	for i := 0; i < N; i++ {
+		css[i].config.SkipTimeoutCommit = false
+	}
+
+	reactors, eventChans := startConsensusNet(t, css, N-1, false)
+	defer stopConsensusNet(reactors)
+
+	// wait till everyone makes the first new block
+	timeoutWaitGroup(t, N-1, func(wg *sync.WaitGroup, j int) {
+		<-eventChans[j]
+		wg.Done()
+	}, css)
+}
+
+func waitForAndValidateBlock(t *testing.T, n int, activeVals map[string]struct{}, eventChans []chan interface{}, css []*ConsensusState, txs ...[]byte) {
+	timeoutWaitGroup(t, n, func(wg *sync.WaitGroup, j int) {
+		newBlockI := <-eventChans[j]
+		newBlock := newBlockI.(types.TMEventData).Unwrap().(types.EventDataNewBlock).Block
+		t.Logf("[WARN] Got block height=%v validator=%v", newBlock.Height, j)
+		err := validateBlock(newBlock, activeVals)
+		if err != nil {
+			t.Fatal(err)
+		}
+		for _, tx := range txs {
+			css[j].mempool.CheckTx(tx, nil)
+		}
+
+		eventChans[j] <- struct{}{}
+		wg.Done()
+	}, css)
+}
+
+// expects high synchrony!
+func validateBlock(block *types.Block, activeVals map[string]struct{}) error {
+	if block.LastCommit.Size() != len(activeVals) {
+		return fmt.Errorf("Commit size doesn't match number of active validators. Got %d, expected %d", block.LastCommit.Size(), len(activeVals))
+	}
+
+	for _, vote := range block.LastCommit.Precommits {
+		if _, ok := activeVals[string(vote.ValidatorAddress)]; !ok {
+			return fmt.Errorf("Found vote for unactive validator %X", vote.ValidatorAddress)
+		}
+	}
+	return nil
+}
+
+func timeoutWaitGroup(t *testing.T, n int, f func(*sync.WaitGroup, int), css []*ConsensusState) {
+	wg := new(sync.WaitGroup)
+	wg.Add(n)
+	for i := 0; i < n; i++ {
+		go f(wg, i)
+	}
+
+	done := make(chan struct{})
+	go func() {
+		wg.Wait()
+		close(done)
+	}()
+
+	select {
+	case <-done:
+	case <-time.After(time.Second * 10):
+		for i, cs := range css {
+			fmt.Println("#################")
+			fmt.Println("Validator", i)
+			fmt.Println(cs.GetRoundState())
+			fmt.Println("")
+		}
+		panic("Timed out waiting for all validators to commit a block")
+	}
+}

consensus/replay.go

@@ -1,39 +1,52 @@
 package consensus
 
 import (
-	"bufio"
+	"bytes"
 	"errors"
 	"fmt"
+	"hash/crc32"
 	"io"
-	"os"
 	"reflect"
 	"strconv"
 	"strings"
 	"time"
 
-	. "github.com/tendermint/go-common"
-	"github.com/tendermint/go-wire"
+	abci "github.com/tendermint/abci/types"
+	auto "github.com/tendermint/tmlibs/autofile"
+	cmn "github.com/tendermint/tmlibs/common"
+	"github.com/tendermint/tmlibs/log"
 
+	"github.com/tendermint/tendermint/proxy"
 	sm "github.com/tendermint/tendermint/state"
 	"github.com/tendermint/tendermint/types"
+	"github.com/tendermint/tendermint/version"
 )
 
-// unmarshal and apply a single message to the consensus state
+var crc32c = crc32.MakeTable(crc32.Castagnoli)
+
+// Functionality to replay blocks and messages on recovery from a crash.
+// There are two general failure scenarios: failure during consensus, and failure while applying the block.
+// The former is handled by the WAL, the latter by the proxyApp Handshake on restart,
+// which ultimately hands off the work to the WAL.
+
+//-----------------------------------------
+// recover from failure during consensus
+// by replaying messages from the WAL
+
+// Unmarshal and apply a single message to the consensus state
 // as if it were received in receiveRoutine
+// Lines that start with "#" are ignored.
 // NOTE: receiveRoutine should not be running
-func (cs *ConsensusState) readReplayMessage(msgBytes []byte, newStepCh chan interface{}) error {
-	var err error
-	var msg ConsensusLogMessage
-	wire.ReadJSON(&msg, msgBytes, &err)
-	if err != nil {
-		fmt.Println("MsgBytes:", msgBytes, string(msgBytes))
-		return fmt.Errorf("Error reading json data: %v", err)
+func (cs *ConsensusState) readReplayMessage(msg *TimedWALMessage, newStepCh chan interface{}) error {
+	// skip meta messages
+	if _, ok := msg.Msg.(EndHeightMessage); ok {
+		return nil
 	}
 
 	// for logging
 	switch m := msg.Msg.(type) {
 	case types.EventDataRoundState:
-		log.Notice("Replay: New Step", "height", m.Height, "round", m.Round, "step", m.Step)
+		cs.Logger.Info("Replay: New Step", "height", m.Height, "round", m.Round, "step", m.Step)
 		// these are playback checks
 		ticker := time.After(time.Second * 2)
 		if newStepCh != nil {
@@ -55,306 +68,330 @@ func (cs *ConsensusState) readReplayMessage(msgBytes []byte, newStepCh chan inte
 		switch msg := m.Msg.(type) {
 		case *ProposalMessage:
 			p := msg.Proposal
-			log.Notice("Replay: Proposal", "height", p.Height, "round", p.Round, "header",
+			cs.Logger.Info("Replay: Proposal", "height", p.Height, "round", p.Round, "header",
 				p.BlockPartsHeader, "pol", p.POLRound, "peer", peerKey)
 		case *BlockPartMessage:
-			log.Notice("Replay: BlockPart", "height", msg.Height, "round", msg.Round, "peer", peerKey)
+			cs.Logger.Info("Replay: BlockPart", "height", msg.Height, "round", msg.Round, "peer", peerKey)
 		case *VoteMessage:
 			v := msg.Vote
-			log.Notice("Replay: Vote", "height", v.Height, "round", v.Round, "type", v.Type,
-				"hash", v.BlockHash, "header", v.BlockPartsHeader, "peer", peerKey)
+			cs.Logger.Info("Replay: Vote", "height", v.Height, "round", v.Round, "type", v.Type,
+				"blockID", v.BlockID, "peer", peerKey)
 		}
 
-		cs.handleMsg(m, cs.RoundState)
+		cs.handleMsg(m)
 	case timeoutInfo:
-		log.Notice("Replay: Timeout", "height", m.Height, "round", m.Round, "step", m.Step, "dur", m.Duration)
+		cs.Logger.Info("Replay: Timeout", "height", m.Height, "round", m.Round, "step", m.Step, "dur", m.Duration)
 		cs.handleTimeout(m, cs.RoundState)
 	default:
-		return fmt.Errorf("Replay: Unknown ConsensusLogMessage type: %v", reflect.TypeOf(msg.Msg))
+		return fmt.Errorf("Replay: Unknown TimedWALMessage type: %v", reflect.TypeOf(msg.Msg))
 	}
 	return nil
 }
 
 // replay only those messages since the last block.
 // timeoutRoutine should run concurrently to read off tickChan
-func (cs *ConsensusState) catchupReplay(height int) error {
-	if !cs.wal.Exists() {
-		return nil
-	}
+func (cs *ConsensusState) catchupReplay(csHeight int) error {
 
 	// set replayMode
 	cs.replayMode = true
 	defer func() { cs.replayMode = false }()
 
-	// starting from end of file,
-	// read messages until a new height is found
-	nLines, err := cs.wal.SeekFromEnd(func(lineBytes []byte) bool {
-		var err error
-		var msg ConsensusLogMessage
-		wire.ReadJSON(&msg, lineBytes, &err)
-		if err != nil {
-			panic(Fmt("Failed to read cs_msg_log json: %v", err))
-		}
-		m, ok := msg.Msg.(types.EventDataRoundState)
-		if ok && m.Step == RoundStepNewHeight.String() {
-			// TODO: ensure the height matches
-			return true
-		}
-		return false
-	})
+	// Ensure that ENDHEIGHT for this height doesn't exist
+	// NOTE: This is just a sanity check. As far as we know things work fine without it,
+	// and Handshake could reuse ConsensusState if it weren't for this check (since we can crash after writing ENDHEIGHT).
+	gr, found, err := cs.wal.SearchForEndHeight(uint64(csHeight))
+	if gr != nil {
+		gr.Close()
+	}
+	if found {
+		return errors.New(cmn.Fmt("WAL should not contain #ENDHEIGHT %d.", csHeight))
+	}
 
-	if err != nil {
+	// Search for last height marker
+	gr, found, err = cs.wal.SearchForEndHeight(uint64(csHeight - 1))
+	if err == io.EOF {
+		cs.Logger.Error("Replay: wal.group.Search returned EOF", "#ENDHEIGHT", csHeight-1)
+	} else if err != nil {
 		return err
 	}
-
-	var beginning bool // if we had to go back to the beginning
-	if c, _ := cs.wal.fp.Seek(0, 1); c == 0 {
-		beginning = true
+	if !found {
+		return errors.New(cmn.Fmt("Cannot replay height %d. WAL does not contain #ENDHEIGHT for %d.", csHeight, csHeight-1))
 	}
+	defer gr.Close()
 
-	log.Notice("Catchup by replaying consensus messages", "n", nLines)
+	cs.Logger.Info("Catchup by replaying consensus messages", "height", csHeight)
 
-	// now we can replay the latest nLines on consensus state
-	// note we can't use scan because we've already been reading from the file
-	reader := bufio.NewReader(cs.wal.fp)
-	for i := 0; i < nLines; i++ {
-		msgBytes, err := reader.ReadBytes('\n')
+	var msg *TimedWALMessage
+	dec := WALDecoder{gr}
+
+	for {
+		msg, err = dec.Decode()
 		if err == io.EOF {
 			break
 		} else if err != nil {
 			return err
-		} else if len(msgBytes) == 0 {
-			continue
-		} else if len(msgBytes) == 1 && msgBytes[0] == '\n' {
-			continue
 		}
-		// the first msg is the NewHeight event (if we're not at the beginning), so we can ignore it
-		if !beginning && i == 1 {
-			continue
-		}
-
 		// NOTE: since the priv key is set when the msgs are received
 		// it will attempt to eg double sign but we can just ignore it
 		// since the votes will be replayed and we'll get to the next step
-		if err := cs.readReplayMessage(msgBytes, nil); err != nil {
+		if err := cs.readReplayMessage(msg, nil); err != nil {
 			return err
 		}
 	}
-	log.Notice("Done catchup replay")
+	cs.Logger.Info("Replay: Done")
 	return nil
 }
 
-//--------------------------------------------------------
-// replay messages interactively or all at once
+//--------------------------------------------------------------------------------
 
-// Interactive playback
-func (cs ConsensusState) ReplayConsole(file string) error {
-	return cs.replay(file, true)
+// Parses marker lines of the form:
+// #ENDHEIGHT: 12345
+func makeHeightSearchFunc(height int) auto.SearchFunc {
+	return func(line string) (int, error) {
+		line = strings.TrimRight(line, "\n")
+		parts := strings.Split(line, " ")
+		if len(parts) != 2 {
+			return -1, errors.New("Line did not have 2 parts")
+		}
+		i, err := strconv.Atoi(parts[1])
+		if err != nil {
+			return -1, errors.New("Failed to parse INFO: " + err.Error())
+		}
+		if height < i {
+			return 1, nil
+		} else if height == i {
+			return 0, nil
+		} else {
+			return -1, nil
+		}
+	}
 }
 
-// Full playback, with tests
-func (cs ConsensusState) ReplayMessages(file string) error {
-	return cs.replay(file, false)
+//----------------------------------------------
+// Recover from failure during block processing
+// by handshaking with the app to figure out where
+// we were last and using the WAL to recover there
+
+type Handshaker struct {
+	state  *sm.State
+	store  types.BlockStore
+	logger log.Logger
+
+	nBlocks int // number of blocks applied to the state
 }
 
-// replay all msgs or start the console
-func (cs *ConsensusState) replay(file string, console bool) error {
-	if cs.IsRunning() {
-		return errors.New("cs is already running, cannot replay")
-	}
-	if cs.wal != nil {
-		return errors.New("cs wal is open, cannot replay")
-	}
+func NewHandshaker(state *sm.State, store types.BlockStore) *Handshaker {
+	return &Handshaker{state, store, log.NewNopLogger(), 0}
+}
 
-	cs.startForReplay()
+func (h *Handshaker) SetLogger(l log.Logger) {
+	h.logger = l
+}
 
-	// ensure all new step events are regenerated as expected
-	newStepCh := subscribeToEvent(cs.evsw, "replay-test", types.EventStringNewRoundStep(), 1)
+func (h *Handshaker) NBlocks() int {
+	return h.nBlocks
+}
 
-	// just open the file for reading, no need to use wal
-	fp, err := os.OpenFile(file, os.O_RDONLY, 0666)
+// TODO: retry the handshake/replay if it fails ?
+func (h *Handshaker) Handshake(proxyApp proxy.AppConns) error {
+	// handshake is done via info request on the query conn
+	res, err := proxyApp.Query().InfoSync(abci.RequestInfo{version.Version})
 	if err != nil {
-		return err
+		return errors.New(cmn.Fmt("Error calling Info: %v", err))
 	}
 
-	pb := newPlayback(file, fp, cs, cs.state.Copy())
-	defer pb.fp.Close()
+	blockHeight := int(res.LastBlockHeight) // XXX: beware overflow
+	appHash := res.LastBlockAppHash
 
-	var nextN int // apply N msgs in a row
-	for pb.scanner.Scan() {
-		if nextN == 0 && console {
-			nextN = pb.replayConsoleLoop()
-		}
+	h.logger.Info("ABCI Handshake", "appHeight", blockHeight, "appHash", fmt.Sprintf("%X", appHash))
 
-		if err := pb.cs.readReplayMessage(pb.scanner.Bytes(), newStepCh); err != nil {
-			return err
-		}
+	// TODO: check version
 
-		if nextN > 0 {
-			nextN -= 1
-		}
-		pb.count += 1
-	}
-	return nil
-}
-
-//------------------------------------------------
-// playback manager
-
-type playback struct {
-	cs *ConsensusState
-
-	fp      *os.File
-	scanner *bufio.Scanner
-	count   int // how many lines/msgs into the file are we
-
-	// replays can be reset to beginning
-	fileName     string    // so we can close/reopen the file
-	genesisState *sm.State // so the replay session knows where to restart from
-}
-
-func newPlayback(fileName string, fp *os.File, cs *ConsensusState, genState *sm.State) *playback {
-	return &playback{
-		cs:           cs,
-		fp:           fp,
-		fileName:     fileName,
-		genesisState: genState,
-		scanner:      bufio.NewScanner(fp),
-	}
-}
-
-// go back count steps by resetting the state and running (pb.count - count) steps
-func (pb *playback) replayReset(count int, newStepCh chan interface{}) error {
-
-	pb.cs.Stop()
-
-	newCS := NewConsensusState(pb.cs.config, pb.genesisState.Copy(), pb.cs.proxyAppConn, pb.cs.blockStore, pb.cs.mempool)
-	newCS.SetEventSwitch(pb.cs.evsw)
-	newCS.startForReplay()
-
-	pb.fp.Close()
-	fp, err := os.OpenFile(pb.fileName, os.O_RDONLY, 0666)
+	// replay blocks up to the latest in the blockstore
+	_, err = h.ReplayBlocks(appHash, blockHeight, proxyApp)
 	if err != nil {
-		return err
+		return errors.New(cmn.Fmt("Error on replay: %v", err))
 	}
-	pb.fp = fp
-	pb.scanner = bufio.NewScanner(fp)
-	count = pb.count - count
-	log.Notice(Fmt("Reseting from %d to %d", pb.count, count))
-	pb.count = 0
-	pb.cs = newCS
-	for i := 0; pb.scanner.Scan() && i < count; i++ {
-		if err := pb.cs.readReplayMessage(pb.scanner.Bytes(), newStepCh); err != nil {
-			return err
+
+	h.logger.Info("Completed ABCI Handshake - Tendermint and App are synced", "appHeight", blockHeight, "appHash", fmt.Sprintf("%X", appHash))
+
+	// TODO: (on restart) replay mempool
+
+	return nil
+}
+
+// Replay all blocks since appBlockHeight and ensure the result matches the current state.
+// Returns the final AppHash or an error
+func (h *Handshaker) ReplayBlocks(appHash []byte, appBlockHeight int, proxyApp proxy.AppConns) ([]byte, error) {
+
+	storeBlockHeight := h.store.Height()
+	stateBlockHeight := h.state.LastBlockHeight
+	h.logger.Info("ABCI Replay Blocks", "appHeight", appBlockHeight, "storeHeight", storeBlockHeight, "stateHeight", stateBlockHeight)
+
+	// If appBlockHeight == 0 it means that we are at genesis and hence should send InitChain
+	if appBlockHeight == 0 {
+		validators := types.TM2PB.Validators(h.state.Validators)
+		proxyApp.Consensus().InitChainSync(abci.RequestInitChain{validators})
+	}
+
+	// First handle edge cases and constraints on the storeBlockHeight
+	if storeBlockHeight == 0 {
+		return appHash, h.checkAppHash(appHash)
+
+	} else if storeBlockHeight < appBlockHeight {
+		// the app should never be ahead of the store (but this is under app's control)
+		return appHash, sm.ErrAppBlockHeightTooHigh{storeBlockHeight, appBlockHeight}
+
+	} else if storeBlockHeight < stateBlockHeight {
+		// the state should never be ahead of the store (this is under tendermint's control)
+		cmn.PanicSanity(cmn.Fmt("StateBlockHeight (%d) > StoreBlockHeight (%d)", stateBlockHeight, storeBlockHeight))
+
+	} else if storeBlockHeight > stateBlockHeight+1 {
+		// store should be at most one ahead of the state (this is under tendermint's control)
+		cmn.PanicSanity(cmn.Fmt("StoreBlockHeight (%d) > StateBlockHeight + 1 (%d)", storeBlockHeight, stateBlockHeight+1))
+	}
+
+	// Now either store is equal to state, or one ahead.
+	// For each, consider all cases of where the app could be, given app <= store
+	if storeBlockHeight == stateBlockHeight {
+		// Tendermint ran Commit and saved the state.
+		// Either the app is asking for replay, or we're all synced up.
+		if appBlockHeight < storeBlockHeight {
+			// the app is behind, so replay blocks, but no need to go through WAL (state is already synced to store)
+			return h.replayBlocks(proxyApp, appBlockHeight, storeBlockHeight, false)
+
+		} else if appBlockHeight == storeBlockHeight {
+			// We're good!
+			return appHash, h.checkAppHash(appHash)
 		}
-		pb.count += 1
+
+	} else if storeBlockHeight == stateBlockHeight+1 {
+		// We saved the block in the store but haven't updated the state,
+		// so we'll need to replay a block using the WAL.
+		if appBlockHeight < stateBlockHeight {
+			// the app is further behind than it should be, so replay blocks
+			// but leave the last block to go through the WAL
+			return h.replayBlocks(proxyApp, appBlockHeight, storeBlockHeight, true)
+
+		} else if appBlockHeight == stateBlockHeight {
+			// We haven't run Commit (both the state and app are one block behind),
+			// so replayBlock with the real app.
+			// NOTE: We could instead use the cs.WAL on cs.Start,
+			// but we'd have to allow the WAL to replay a block that wrote it's ENDHEIGHT
+			h.logger.Info("Replay last block using real app")
+			return h.replayBlock(storeBlockHeight, proxyApp.Consensus())
+
+		} else if appBlockHeight == storeBlockHeight {
+			// We ran Commit, but didn't save the state, so replayBlock with mock app
+			abciResponses := h.state.LoadABCIResponses()
+			mockApp := newMockProxyApp(appHash, abciResponses)
+			h.logger.Info("Replay last block using mock app")
+			return h.replayBlock(storeBlockHeight, mockApp)
+		}
+
+	}
+
+	cmn.PanicSanity("Should never happen")
+	return nil, nil
+}
+
+func (h *Handshaker) replayBlocks(proxyApp proxy.AppConns, appBlockHeight, storeBlockHeight int, mutateState bool) ([]byte, error) {
+	// App is further behind than it should be, so we need to replay blocks.
+	// We replay all blocks from appBlockHeight+1.
+	//
+	// Note that we don't have an old version of the state,
+	// so we by-pass state validation/mutation using sm.ExecCommitBlock.
+	// This also means we won't be saving validator sets if they change during this period.
+	//
+	// If mutateState == true, the final block is replayed with h.replayBlock()
+
+	var appHash []byte
+	var err error
+	finalBlock := storeBlockHeight
+	if mutateState {
+		finalBlock -= 1
+	}
+	for i := appBlockHeight + 1; i <= finalBlock; i++ {
+		h.logger.Info("Applying block", "height", i)
+		block := h.store.LoadBlock(i)
+		appHash, err = sm.ExecCommitBlock(proxyApp.Consensus(), block, h.logger)
+		if err != nil {
+			return nil, err
+		}
+
+		h.nBlocks += 1
+	}
+
+	if mutateState {
+		// sync the final block
+		return h.replayBlock(storeBlockHeight, proxyApp.Consensus())
+	}
+
+	return appHash, h.checkAppHash(appHash)
+}
+
+// ApplyBlock on the proxyApp with the last block.
+func (h *Handshaker) replayBlock(height int, proxyApp proxy.AppConnConsensus) ([]byte, error) {
+	mempool := types.MockMempool{}
+
+	var eventCache types.Fireable // nil
+	block := h.store.LoadBlock(height)
+	meta := h.store.LoadBlockMeta(height)
+
+	if err := h.state.ApplyBlock(eventCache, proxyApp, block, meta.BlockID.PartsHeader, mempool); err != nil {
+		return nil, err
+	}
+
+	h.nBlocks += 1
+
+	return h.state.AppHash, nil
+}
+
+func (h *Handshaker) checkAppHash(appHash []byte) error {
+	if !bytes.Equal(h.state.AppHash, appHash) {
+		panic(errors.New(cmn.Fmt("Tendermint state.AppHash does not match AppHash after replay. Got %X, expected %X", appHash, h.state.AppHash)).Error())
+		return nil
 	}
 	return nil
 }
 
-func (cs *ConsensusState) startForReplay() {
-	// don't want to start full cs
-	cs.BaseService.OnStart()
+//--------------------------------------------------------------------------------
+// mockProxyApp uses ABCIResponses to give the right results
+// Useful because we don't want to call Commit() twice for the same block on the real app.
 
-	// since we replay tocks we just ignore ticks
-	go func() {
-		for {
-			select {
-			case <-cs.tickChan:
-			case <-cs.Quit:
-				return
-			}
-		}
-	}()
+func newMockProxyApp(appHash []byte, abciResponses *sm.ABCIResponses) proxy.AppConnConsensus {
+	clientCreator := proxy.NewLocalClientCreator(&mockProxyApp{
+		appHash:       appHash,
+		abciResponses: abciResponses,
+	})
+	cli, _ := clientCreator.NewABCIClient()
+	cli.Start()
+	return proxy.NewAppConnConsensus(cli)
 }
 
-// console function for parsing input and running commands
-func (pb *playback) replayConsoleLoop() int {
-	for {
-		fmt.Printf("> ")
-		bufReader := bufio.NewReader(os.Stdin)
-		line, more, err := bufReader.ReadLine()
-		if more {
-			Exit("input is too long")
-		} else if err != nil {
-			Exit(err.Error())
-		}
+type mockProxyApp struct {
+	abci.BaseApplication
 
-		tokens := strings.Split(string(line), " ")
-		if len(tokens) == 0 {
-			continue
-		}
+	appHash       []byte
+	txCount       int
+	abciResponses *sm.ABCIResponses
+}
 
-		switch tokens[0] {
-		case "next":
-			// "next" -> replay next message
-			// "next N" -> replay next N messages
-
-			if len(tokens) == 1 {
-				return 0
-			} else {
-				i, err := strconv.Atoi(tokens[1])
-				if err != nil {
-					fmt.Println("next takes an integer argument")
-				} else {
-					return i
-				}
-			}
-
-		case "back":
-			// "back" -> go back one message
-			// "back N" -> go back N messages
-
-			// NOTE: "back" is not supported in the state machine design,
-			// so we restart and replay up to
-
-			// ensure all new step events are regenerated as expected
-			newStepCh := subscribeToEvent(pb.cs.evsw, "replay-test", types.EventStringNewRoundStep(), 1)
-			if len(tokens) == 1 {
-				pb.replayReset(1, newStepCh)
-			} else {
-				i, err := strconv.Atoi(tokens[1])
-				if err != nil {
-					fmt.Println("back takes an integer argument")
-				} else if i > pb.count {
-					fmt.Printf("argument to back must not be larger than the current count (%d)\n", pb.count)
-				} else {
-					pb.replayReset(i, newStepCh)
-				}
-			}
-
-		case "rs":
-			// "rs" -> print entire round state
-			// "rs short" -> print height/round/step
-			// "rs <field>" -> print another field of the round state
-
-			rs := pb.cs.RoundState
-			if len(tokens) == 1 {
-				fmt.Println(rs)
-			} else {
-				switch tokens[1] {
-				case "short":
-					fmt.Printf("%v/%v/%v\n", rs.Height, rs.Round, rs.Step)
-				case "validators":
-					fmt.Println(rs.Validators)
-				case "proposal":
-					fmt.Println(rs.Proposal)
-				case "proposal_block":
-					fmt.Printf("%v %v\n", rs.ProposalBlockParts.StringShort(), rs.ProposalBlock.StringShort())
-				case "locked_round":
-					fmt.Println(rs.LockedRound)
-				case "locked_block":
-					fmt.Printf("%v %v\n", rs.LockedBlockParts.StringShort(), rs.LockedBlock.StringShort())
-				case "votes":
-					fmt.Println(rs.Votes.StringIndented("    "))
-
-				default:
-					fmt.Println("Unknown option", tokens[1])
-				}
-			}
-		case "n":
-			fmt.Println(pb.count)
-		}
+func (mock *mockProxyApp) DeliverTx(tx []byte) abci.Result {
+	r := mock.abciResponses.DeliverTx[mock.txCount]
+	mock.txCount += 1
+	return abci.Result{
+		r.Code,
+		r.Data,
+		r.Log,
 	}
-	return 0
+}
+
+func (mock *mockProxyApp) EndBlock(height uint64) abci.ResponseEndBlock {
+	mock.txCount = 0
+	return mock.abciResponses.EndBlock
+}
+
+func (mock *mockProxyApp) Commit() abci.Result {
+	return abci.NewResultOK(mock.appHash, "")
 }

consensus/replay_file.go

@@ -0,0 +1,283 @@
+package consensus
+
+import (
+	"bufio"
+	"errors"
+	"fmt"
+	"io"
+	"os"
+	"strconv"
+	"strings"
+
+	bc "github.com/tendermint/tendermint/blockchain"
+	cfg "github.com/tendermint/tendermint/config"
+	"github.com/tendermint/tendermint/proxy"
+	sm "github.com/tendermint/tendermint/state"
+	"github.com/tendermint/tendermint/types"
+	cmn "github.com/tendermint/tmlibs/common"
+	dbm "github.com/tendermint/tmlibs/db"
+)
+
+//--------------------------------------------------------
+// replay messages interactively or all at once
+
+func RunReplayFile(config cfg.BaseConfig, csConfig *cfg.ConsensusConfig, console bool) {
+	consensusState := newConsensusStateForReplay(config, csConfig)
+
+	if err := consensusState.ReplayFile(csConfig.WalFile(), console); err != nil {
+		cmn.Exit(cmn.Fmt("Error during consensus replay: %v", err))
+	}
+}
+
+// Replay msgs in file or start the console
+func (cs *ConsensusState) ReplayFile(file string, console bool) error {
+
+	if cs.IsRunning() {
+		return errors.New("cs is already running, cannot replay")
+	}
+	if cs.wal != nil {
+		return errors.New("cs wal is open, cannot replay")
+	}
+
+	cs.startForReplay()
+
+	// ensure all new step events are regenerated as expected
+	newStepCh := subscribeToEvent(cs.evsw, "replay-test", types.EventStringNewRoundStep(), 1)
+
+	// just open the file for reading, no need to use wal
+	fp, err := os.OpenFile(file, os.O_RDONLY, 0666)
+	if err != nil {
+		return err
+	}
+
+	pb := newPlayback(file, fp, cs, cs.state.Copy())
+	defer pb.fp.Close()
+
+	var nextN int // apply N msgs in a row
+	var msg *TimedWALMessage
+	for {
+		if nextN == 0 && console {
+			nextN = pb.replayConsoleLoop()
+		}
+
+		msg, err = pb.dec.Decode()
+		if err == io.EOF {
+			return nil
+		} else if err != nil {
+			return err
+		}
+
+		if err := pb.cs.readReplayMessage(msg, newStepCh); err != nil {
+			return err
+		}
+
+		if nextN > 0 {
+			nextN -= 1
+		}
+		pb.count += 1
+	}
+	return nil
+}
+
+//------------------------------------------------
+// playback manager
+
+type playback struct {
+	cs *ConsensusState
+
+	fp    *os.File
+	dec   *WALDecoder
+	count int // how many lines/msgs into the file are we
+
+	// replays can be reset to beginning
+	fileName     string    // so we can close/reopen the file
+	genesisState *sm.State // so the replay session knows where to restart from
+}
+
+func newPlayback(fileName string, fp *os.File, cs *ConsensusState, genState *sm.State) *playback {
+	return &playback{
+		cs:           cs,
+		fp:           fp,
+		fileName:     fileName,
+		genesisState: genState,
+		dec:          NewWALDecoder(fp),
+	}
+}
+
+// go back count steps by resetting the state and running (pb.count - count) steps
+func (pb *playback) replayReset(count int, newStepCh chan interface{}) error {
+
+	pb.cs.Stop()
+	pb.cs.Wait()
+
+	newCS := NewConsensusState(pb.cs.config, pb.genesisState.Copy(), pb.cs.proxyAppConn, pb.cs.blockStore, pb.cs.mempool)
+	newCS.SetEventSwitch(pb.cs.evsw)
+	newCS.startForReplay()
+
+	pb.fp.Close()
+	fp, err := os.OpenFile(pb.fileName, os.O_RDONLY, 0666)
+	if err != nil {
+		return err
+	}
+	pb.fp = fp
+	pb.dec = NewWALDecoder(fp)
+	count = pb.count - count
+	fmt.Printf("Reseting from %d to %d\n", pb.count, count)
+	pb.count = 0
+	pb.cs = newCS
+	var msg *TimedWALMessage
+	for i := 0; i < count; i++ {
+		msg, err = pb.dec.Decode()
+		if err == io.EOF {
+			return nil
+		} else if err != nil {
+			return err
+		}
+		if err := pb.cs.readReplayMessage(msg, newStepCh); err != nil {
+			return err
+		}
+		pb.count += 1
+	}
+	return nil
+}
+
+func (cs *ConsensusState) startForReplay() {
+	cs.Logger.Error("Replay commands are disabled until someone updates them and writes tests")
+	/* TODO:!
+	// since we replay tocks we just ignore ticks
+		go func() {
+			for {
+				select {
+				case <-cs.tickChan:
+				case <-cs.Quit:
+					return
+				}
+			}
+		}()*/
+}
+
+// console function for parsing input and running commands
+func (pb *playback) replayConsoleLoop() int {
+	for {
+		fmt.Printf("> ")
+		bufReader := bufio.NewReader(os.Stdin)
+		line, more, err := bufReader.ReadLine()
+		if more {
+			cmn.Exit("input is too long")
+		} else if err != nil {
+			cmn.Exit(err.Error())
+		}
+
+		tokens := strings.Split(string(line), " ")
+		if len(tokens) == 0 {
+			continue
+		}
+
+		switch tokens[0] {
+		case "next":
+			// "next" -> replay next message
+			// "next N" -> replay next N messages
+
+			if len(tokens) == 1 {
+				return 0
+			} else {
+				i, err := strconv.Atoi(tokens[1])
+				if err != nil {
+					fmt.Println("next takes an integer argument")
+				} else {
+					return i
+				}
+			}
+
+		case "back":
+			// "back" -> go back one message
+			// "back N" -> go back N messages
+
+			// NOTE: "back" is not supported in the state machine design,
+			// so we restart and replay up to
+
+			// ensure all new step events are regenerated as expected
+			newStepCh := subscribeToEvent(pb.cs.evsw, "replay-test", types.EventStringNewRoundStep(), 1)
+			if len(tokens) == 1 {
+				pb.replayReset(1, newStepCh)
+			} else {
+				i, err := strconv.Atoi(tokens[1])
+				if err != nil {
+					fmt.Println("back takes an integer argument")
+				} else if i > pb.count {
+					fmt.Printf("argument to back must not be larger than the current count (%d)\n", pb.count)
+				} else {
+					pb.replayReset(i, newStepCh)
+				}
+			}
+
+		case "rs":
+			// "rs" -> print entire round state
+			// "rs short" -> print height/round/step
+			// "rs <field>" -> print another field of the round state
+
+			rs := pb.cs.RoundState
+			if len(tokens) == 1 {
+				fmt.Println(rs)
+			} else {
+				switch tokens[1] {
+				case "short":
+					fmt.Printf("%v/%v/%v\n", rs.Height, rs.Round, rs.Step)
+				case "validators":
+					fmt.Println(rs.Validators)
+				case "proposal":
+					fmt.Println(rs.Proposal)
+				case "proposal_block":
+					fmt.Printf("%v %v\n", rs.ProposalBlockParts.StringShort(), rs.ProposalBlock.StringShort())
+				case "locked_round":
+					fmt.Println(rs.LockedRound)
+				case "locked_block":
+					fmt.Printf("%v %v\n", rs.LockedBlockParts.StringShort(), rs.LockedBlock.StringShort())
+				case "votes":
+					fmt.Println(rs.Votes.StringIndented("    "))
+
+				default:
+					fmt.Println("Unknown option", tokens[1])
+				}
+			}
+		case "n":
+			fmt.Println(pb.count)
+		}
+	}
+	return 0
+}
+
+//--------------------------------------------------------------------------------
+
+// convenience for replay mode
+func newConsensusStateForReplay(config cfg.BaseConfig, csConfig *cfg.ConsensusConfig) *ConsensusState {
+	// Get BlockStore
+	blockStoreDB := dbm.NewDB("blockstore", config.DBBackend, config.DBDir())
+	blockStore := bc.NewBlockStore(blockStoreDB)
+
+	// Get State
+	stateDB := dbm.NewDB("state", config.DBBackend, config.DBDir())
+	state, err := sm.MakeGenesisStateFromFile(stateDB, config.GenesisFile())
+	if err != nil {
+		cmn.Exit(err.Error())
+	}
+
+	// Create proxyAppConn connection (consensus, mempool, query)
+	clientCreator := proxy.DefaultClientCreator(config.ProxyApp, config.ABCI, config.DBDir())
+	proxyApp := proxy.NewAppConns(clientCreator, NewHandshaker(state, blockStore))
+	_, err = proxyApp.Start()
+	if err != nil {
+		cmn.Exit(cmn.Fmt("Error starting proxy app conns: %v", err))
+	}
+
+	// Make event switch
+	eventSwitch := types.NewEventSwitch()
+	if _, err := eventSwitch.Start(); err != nil {
+		cmn.Exit(cmn.Fmt("Failed to start event switch: %v", err))
+	}
+
+	consensusState := NewConsensusState(csConfig, state.Copy(), proxyApp.Consensus(), blockStore, types.MockMempool{})
+
+	consensusState.SetEventSwitch(eventSwitch)
+	return consensusState
+}

consensus/replay_test.go

@@ -1,151 +1,230 @@
 package consensus
 
 import (
+	"bytes"
+	"errors"
 	"fmt"
+	"io"
 	"io/ioutil"
 	"os"
-	"strings"
+	"path"
 	"testing"
 	"time"
 
-	. "github.com/tendermint/go-common"
-	"github.com/tendermint/go-events"
-	"github.com/tendermint/go-wire"
+	"github.com/tendermint/abci/example/dummy"
+	abci "github.com/tendermint/abci/types"
+	crypto "github.com/tendermint/go-crypto"
+	wire "github.com/tendermint/go-wire"
+	cmn "github.com/tendermint/tmlibs/common"
+	dbm "github.com/tendermint/tmlibs/db"
+
+	cfg "github.com/tendermint/tendermint/config"
+	"github.com/tendermint/tendermint/proxy"
+	sm "github.com/tendermint/tendermint/state"
 	"github.com/tendermint/tendermint/types"
+	"github.com/tendermint/tmlibs/log"
 )
 
-/*
-	The easiest way to generate this data is to copy ~/.tendermint_test/somedir/* to ~/.tendermint
-	and to run a local node.
-	Be sure to set the db to "leveldb" to create a cswal file in ~/.tendermint/data/cswal.
-
-	If you need to change the signatures, you can use a script as follows:
-	The privBytes comes from config/tendermint_test/...
-
-	```
-	package main
-
-	import (
-		"encoding/hex"
-		"fmt"
-
-		"github.com/tendermint/go-crypto"
-	)
-
-	func main() {
-		signBytes, err := hex.DecodeString("7B22636861696E5F6964223A2274656E6465726D696E745F74657374222C22766F7465223A7B22626C6F636B5F68617368223A2242453544373939433846353044354645383533364334333932464443384537423342313830373638222C22626C6F636B5F70617274735F686561646572223A506172745365747B543A31204236323237323535464632307D2C22686569676874223A312C22726F756E64223A302C2274797065223A327D7D")
-		if err != nil {
-			panic(err)
-		}
-		privBytes, err := hex.DecodeString("27F82582AEFAE7AB151CFB01C48BB6C1A0DA78F9BDDA979A9F70A84D074EB07D3B3069C422E19688B45CBFAE7BB009FC0FA1B1EA86593519318B7214853803C8")
-		if err != nil {
-			panic(err)
-		}
-		privKey := crypto.PrivKeyEd25519{}
-		copy(privKey[:], privBytes)
-		signature := privKey.Sign(signBytes)
-		signatureEd25519 := signature.(crypto.SignatureEd25519)
-		fmt.Printf("Signature Bytes: %X\n", signatureEd25519[:])
-	}
-	```
-*/
-
-var testLog = `{"time":"2016-04-03T11:23:54.387Z","msg":[3,{"duration":972835254,"height":1,"round":0,"step":1}]}
-{"time":"2016-04-03T11:23:54.388Z","msg":[1,{"height":1,"round":0,"step":"RoundStepPropose"}]}
-{"time":"2016-04-03T11:23:54.388Z","msg":[2,{"msg":[17,{"Proposal":{"height":1,"round":0,"block_parts_header":{"total":1,"hash":"3BA1E90CB868DA6B4FD7F3589826EC461E9EB4EF"},"pol_round":-1,"signature":"3A2ECD5023B21EC144EC16CFF1B992A4321317B83EEDD8969FDFEA6EB7BF4389F38DDA3E7BB109D63A07491C16277A197B241CF1F05F5E485C59882ECACD9E07"}}],"peer_key":""}]}
-{"time":"2016-04-03T11:23:54.389Z","msg":[2,{"msg":[19,{"Height":1,"Round":0,"Part":{"index":0,"bytes":"0101010F74656E6465726D696E745F7465737401011441D59F4B718AC00000000000000114C4B01D3810579550997AC5641E759E20D99B51C10001000100","proof":{"aunts":[]}}}],"peer_key":""}]}
-{"time":"2016-04-03T11:23:54.390Z","msg":[1,{"height":1,"round":0,"step":"RoundStepPrevote"}]}
-{"time":"2016-04-03T11:23:54.390Z","msg":[2,{"msg":[20,{"ValidatorIndex":0,"Vote":{"height":1,"round":0,"type":1,"block_hash":"4291966B8A9DFBA00AEC7C700F2718E61DF4331D","block_parts_header":{"total":1,"hash":"3BA1E90CB868DA6B4FD7F3589826EC461E9EB4EF"},"signature":"47D2A75A4E2F15DB1F0D1B656AC0637AF9AADDFEB6A156874F6553C73895E5D5DC948DBAEF15E61276C5342D0E638DFCB77C971CD282096EA8735A564A90F008"}}],"peer_key":""}]}
-{"time":"2016-04-03T11:23:54.392Z","msg":[1,{"height":1,"round":0,"step":"RoundStepPrecommit"}]}
-{"time":"2016-04-03T11:23:54.392Z","msg":[2,{"msg":[20,{"ValidatorIndex":0,"Vote":{"height":1,"round":0,"type":2,"block_hash":"4291966B8A9DFBA00AEC7C700F2718E61DF4331D","block_parts_header":{"total":1,"hash":"3BA1E90CB868DA6B4FD7F3589826EC461E9EB4EF"},"signature":"39147DA595F08B73CF8C899967C8403B5872FD9042FFA4E239159E0B6C5D9665C9CA81D766EACA2AE658872F94C2FCD1E34BF51859CD5B274DA8512BACE4B50D"}}],"peer_key":""}]}
-`
-
-// map lines in the above wal to privVal step
-var mapPrivValStep = map[int]int8{
-	0: 0,
-	1: 0,
-	2: 1,
-	3: 1,
-	4: 1,
-	5: 2,
-	6: 2,
-	7: 3,
+func init() {
+	config = ResetConfig("consensus_replay_test")
 }
 
-func writeWAL(log string) string {
-	fmt.Println("writing", log)
-	// write the needed wal to file
-	f, err := ioutil.TempFile(os.TempDir(), "replay_test_")
+// These tests ensure we can always recover from failure at any part of the consensus process.
+// There are two general failure scenarios: failure during consensus, and failure while applying the block.
+// Only the latter interacts with the app and store,
+// but the former has to deal with restrictions on re-use of priv_validator keys.
+// The `WAL Tests` are for failures during the consensus;
+// the `Handshake Tests` are for failures in applying the block.
+// With the help of the WAL, we can recover from it all!
+
+// NOTE: Files in this dir are generated by running the `build.sh` therein.
+// It's a simple way to generate wals for a single block, or multiple blocks, with random transactions,
+// and different part sizes. The output is not deterministic, and the stepChanges may need to be adjusted
+// after running it (eg. sometimes small_block2 will have 5 block parts, sometimes 6).
+// It should only have to be re-run if there is some breaking change to the consensus data structures (eg. blocks, votes)
+// or to the behaviour of the app (eg. computes app hash differently)
+var data_dir = path.Join(cmn.GoPath(), "src/github.com/tendermint/tendermint/consensus", "test_data")
+
+//------------------------------------------------------------------------------------------
+// WAL Tests
+
+// TODO: It would be better to verify explicitly which states we can recover from without the wal
+// and which ones we need the wal for - then we'd also be able to only flush the
+// wal writer when we need to, instead of with every message.
+
+// the priv validator changes step at these lines for a block with 1 val and 1 part
+var baseStepChanges = []int{3, 6, 8}
+
+// test recovery from each line in each testCase
+var testCases = []*testCase{
+	newTestCase("empty_block", baseStepChanges),   // empty block (has 1 block part)
+	newTestCase("small_block1", baseStepChanges),  // small block with txs in 1 block part
+	newTestCase("small_block2", []int{3, 12, 14}), // small block with txs across 6 smaller block parts
+}
+
+type testCase struct {
+	name    string
+	log     []byte       //full cs wal
+	stepMap map[int]int8 // map lines of log to privval step
+
+	proposeLine   int
+	prevoteLine   int
+	precommitLine int
+}
+
+func newTestCase(name string, stepChanges []int) *testCase {
+	if len(stepChanges) != 3 {
+		panic(cmn.Fmt("a full wal has 3 step changes! Got array %v", stepChanges))
+	}
+	return &testCase{
+		name:    name,
+		log:     readWAL(path.Join(data_dir, name+".cswal")),
+		stepMap: newMapFromChanges(stepChanges),
+
+		proposeLine:   stepChanges[0],
+		prevoteLine:   stepChanges[1],
+		precommitLine: stepChanges[2],
+	}
+}
+
+func newMapFromChanges(changes []int) map[int]int8 {
+	changes = append(changes, changes[2]+1) // so we add the last step change to the map
+	m := make(map[int]int8)
+	var count int
+	for changeNum, nextChange := range changes {
+		for ; count < nextChange; count++ {
+			m[count] = int8(changeNum)
+		}
+	}
+	return m
+}
+
+func readWAL(p string) []byte {
+	b, err := ioutil.ReadFile(p)
 	if err != nil {
 		panic(err)
 	}
-
-	_, err = f.WriteString(log)
-	if err != nil {
-		panic(err)
-	}
-	name := f.Name()
-	f.Close()
-	return name
+	return b
 }
 
-func waitForBlock(newBlockCh chan interface{}) {
+func writeWAL(walMsgs []byte) string {
+	walFile, err := ioutil.TempFile("", "wal")
+	if err != nil {
+		panic(fmt.Errorf("failed to create temp WAL file: %v", err))
+	}
+	_, err = walFile.Write(walMsgs)
+	if err != nil {
+		panic(fmt.Errorf("failed to write to temp WAL file: %v", err))
+	}
+	if err := walFile.Close(); err != nil {
+		panic(fmt.Errorf("failed to close temp WAL file: %v", err))
+	}
+	return walFile.Name()
+}
+
+func waitForBlock(newBlockCh chan interface{}, thisCase *testCase, i int) {
 	after := time.After(time.Second * 10)
 	select {
 	case <-newBlockCh:
 	case <-after:
-		panic("Timed out waiting for new block")
+		panic(cmn.Fmt("Timed out waiting for new block for case '%s' line %d", thisCase.name, i))
 	}
 }
 
-func runReplayTest(t *testing.T, cs *ConsensusState, fileName string, newBlockCh chan interface{}) {
-	cs.config.Set("cswal", fileName)
-	cs.Start()
+func runReplayTest(t *testing.T, cs *ConsensusState, walFile string, newBlockCh chan interface{},
+	thisCase *testCase, i int) {
+
+	cs.config.SetWalFile(walFile)
+	started, err := cs.Start()
+	if err != nil {
+		t.Fatalf("Cannot start consensus: %v", err)
+	}
+	if !started {
+		t.Error("Consensus did not start")
+	}
 	// Wait to make a new block.
 	// This is just a signal that we haven't halted; its not something contained in the WAL itself.
 	// Assuming the consensus state is running, replay of any WAL, including the empty one,
 	// should eventually be followed by a new block, or else something is wrong
-	waitForBlock(newBlockCh)
+	waitForBlock(newBlockCh, thisCase, i)
+	cs.evsw.Stop()
 	cs.Stop()
+LOOP:
+	for {
+		select {
+		case <-newBlockCh:
+		default:
+			break LOOP
+		}
+	}
+	cs.Wait()
 }
 
-func setupReplayTest(nLines int, crashAfter bool) (*ConsensusState, chan interface{}, string, string) {
-	fmt.Println("-------------------------------------")
-	log.Notice(Fmt("Starting replay test of %d lines of WAL (crash before write)", nLines))
+func toPV(pv types.PrivValidator) *types.PrivValidatorFS {
+	return pv.(*types.PrivValidatorFS)
+}
+
+func setupReplayTest(t *testing.T, thisCase *testCase, nLines int, crashAfter bool) (*ConsensusState, chan interface{}, []byte, string) {
+	t.Log("-------------------------------------")
+	t.Logf("Starting replay test %v (of %d lines of WAL). Crash after = %v", thisCase.name, nLines, crashAfter)
 
 	lineStep := nLines
 	if crashAfter {
 		lineStep -= 1
 	}
 
-	split := strings.Split(testLog, "\n")
+	split := bytes.Split(thisCase.log, walSeparator)
 	lastMsg := split[nLines]
 
 	// we write those lines up to (not including) one with the signature
-	fileName := writeWAL(strings.Join(split[:nLines], "\n") + "\n")
+	b := bytes.Join(split[:nLines], walSeparator)
+	b = append(b, walSeparator...)
+	walFile := writeWAL(b)
 
-	cs := fixedConsensusState()
+	cs := fixedConsensusStateDummy()
 
 	// set the last step according to when we crashed vs the wal
-	cs.privValidator.LastHeight = 1 // first block
-	cs.privValidator.LastStep = mapPrivValStep[lineStep]
+	toPV(cs.privValidator).LastHeight = 1 // first block
+	toPV(cs.privValidator).LastStep = thisCase.stepMap[lineStep]
 
-	fmt.Println("LAST STEP", cs.privValidator.LastStep)
+	t.Logf("[WARN] setupReplayTest LastStep=%v", toPV(cs.privValidator).LastStep)
 
 	newBlockCh := subscribeToEvent(cs.evsw, "tester", types.EventStringNewBlock(), 1)
 
-	return cs, newBlockCh, lastMsg, fileName
+	return cs, newBlockCh, lastMsg, walFile
+}
+
+func readTimedWALMessage(t *testing.T, rawMsg []byte) TimedWALMessage {
+	b := bytes.NewBuffer(rawMsg)
+	// because rawMsg does not contain a separator and WALDecoder#Decode expects it
+	_, err := b.Write(walSeparator)
+	if err != nil {
+		t.Fatal(err)
+	}
+	dec := NewWALDecoder(b)
+	msg, err := dec.Decode()
+	if err != nil {
+		t.Fatalf("Error reading json data: %v", err)
+	}
+	return *msg
 }
 
 //-----------------------------------------------
 // Test the log at every iteration, and set the privVal last step
 // as if the log was written after signing, before the crash
 
-func TestReplayCrashAfterWrite(t *testing.T) {
-	split := strings.Split(testLog, "\n")
-	for i := 0; i < len(split)-1; i++ {
-		cs, newBlockCh, _, f := setupReplayTest(i+1, true)
-		runReplayTest(t, cs, f, newBlockCh)
+func TestWALCrashAfterWrite(t *testing.T) {
+	for _, thisCase := range testCases {
+		splitSize := bytes.Count(thisCase.log, walSeparator)
+		for i := 0; i < splitSize-1; i++ {
+			t.Run(fmt.Sprintf("%s:%d", thisCase.name, i), func(t *testing.T) {
+				cs, newBlockCh, _, walFile := setupReplayTest(t, thisCase, i+1, true)
+				cs.config.TimeoutPropose = 100
+				runReplayTest(t, cs, walFile, newBlockCh, thisCase, i+1)
+				// cleanup
+				os.Remove(walFile)
+			})
+		}
 	}
 }
 
@@ -153,51 +232,375 @@ func TestReplayCrashAfterWrite(t *testing.T) {
 // Test the log as if we crashed after signing but before writing.
 // This relies on privValidator.LastSignature being set
 
-func TestReplayCrashBeforeWritePropose(t *testing.T) {
-	cs, newBlockCh, proposalMsg, f := setupReplayTest(2, false) // propose
-	// Set LastSig
-	var err error
-	var msg ConsensusLogMessage
-	wire.ReadJSON(&msg, []byte(proposalMsg), &err)
-	proposal := msg.Msg.(msgInfo).Msg.(*ProposalMessage)
-	if err != nil {
-		t.Fatalf("Error reading json data: %v", err)
+func TestWALCrashBeforeWritePropose(t *testing.T) {
+	for _, thisCase := range testCases {
+		lineNum := thisCase.proposeLine
+		t.Run(fmt.Sprintf("%s:%d", thisCase.name, lineNum), func(t *testing.T) {
+			// setup replay test where last message is a proposal
+			cs, newBlockCh, proposalMsg, walFile := setupReplayTest(t, thisCase, lineNum, false)
+			cs.config.TimeoutPropose = 100
+			msg := readTimedWALMessage(t, proposalMsg)
+			proposal := msg.Msg.(msgInfo).Msg.(*ProposalMessage)
+			// Set LastSig
+			toPV(cs.privValidator).LastSignBytes = types.SignBytes(cs.state.ChainID, proposal.Proposal)
+			toPV(cs.privValidator).LastSignature = proposal.Proposal.Signature
+			runReplayTest(t, cs, walFile, newBlockCh, thisCase, lineNum)
+			// cleanup
+			os.Remove(walFile)
+		})
 	}
-	cs.privValidator.LastSignBytes = types.SignBytes(cs.state.ChainID, proposal.Proposal)
-	cs.privValidator.LastSignature = proposal.Proposal.Signature
-	runReplayTest(t, cs, f, newBlockCh)
 }
 
-func TestReplayCrashBeforeWritePrevote(t *testing.T) {
-	cs, newBlockCh, voteMsg, f := setupReplayTest(5, false) // prevote
-	cs.evsw.AddListenerForEvent("tester", types.EventStringCompleteProposal(), func(data events.EventData) {
-		// Set LastSig
-		var err error
-		var msg ConsensusLogMessage
-		wire.ReadJSON(&msg, []byte(voteMsg), &err)
-		vote := msg.Msg.(msgInfo).Msg.(*VoteMessage)
-		if err != nil {
-			t.Fatalf("Error reading json data: %v", err)
-		}
-		cs.privValidator.LastSignBytes = types.SignBytes(cs.state.ChainID, vote.Vote)
-		cs.privValidator.LastSignature = vote.Vote.Signature
-	})
-	runReplayTest(t, cs, f, newBlockCh)
+func TestWALCrashBeforeWritePrevote(t *testing.T) {
+	for _, thisCase := range testCases {
+		testReplayCrashBeforeWriteVote(t, thisCase, thisCase.prevoteLine, types.EventStringCompleteProposal())
+	}
 }
 
-func TestReplayCrashBeforeWritePrecommit(t *testing.T) {
-	cs, newBlockCh, voteMsg, f := setupReplayTest(7, false) // precommit
-	cs.evsw.AddListenerForEvent("tester", types.EventStringPolka(), func(data events.EventData) {
-		// Set LastSig
-		var err error
-		var msg ConsensusLogMessage
-		wire.ReadJSON(&msg, []byte(voteMsg), &err)
-		vote := msg.Msg.(msgInfo).Msg.(*VoteMessage)
-		if err != nil {
-			t.Fatalf("Error reading json data: %v", err)
-		}
-		cs.privValidator.LastSignBytes = types.SignBytes(cs.state.ChainID, vote.Vote)
-		cs.privValidator.LastSignature = vote.Vote.Signature
-	})
-	runReplayTest(t, cs, f, newBlockCh)
+func TestWALCrashBeforeWritePrecommit(t *testing.T) {
+	for _, thisCase := range testCases {
+		testReplayCrashBeforeWriteVote(t, thisCase, thisCase.precommitLine, types.EventStringPolka())
+	}
+}
+
+func testReplayCrashBeforeWriteVote(t *testing.T, thisCase *testCase, lineNum int, eventString string) {
+	// setup replay test where last message is a vote
+	cs, newBlockCh, voteMsg, walFile := setupReplayTest(t, thisCase, lineNum, false)
+	types.AddListenerForEvent(cs.evsw, "tester", eventString, func(data types.TMEventData) {
+		msg := readTimedWALMessage(t, voteMsg)
+		vote := msg.Msg.(msgInfo).Msg.(*VoteMessage)
+		// Set LastSig
+		toPV(cs.privValidator).LastSignBytes = types.SignBytes(cs.state.ChainID, vote.Vote)
+		toPV(cs.privValidator).LastSignature = vote.Vote.Signature
+	})
+	runReplayTest(t, cs, walFile, newBlockCh, thisCase, lineNum)
+}
+
+//------------------------------------------------------------------------------------------
+// Handshake Tests
+
+var (
+	NUM_BLOCKS = 6 // number of blocks in the test_data/many_blocks.cswal
+	mempool    = types.MockMempool{}
+)
+
+//---------------------------------------
+// Test handshake/replay
+
+// 0 - all synced up
+// 1 - saved block but app and state are behind
+// 2 - save block and committed but state is behind
+var modes = []uint{0, 1, 2}
+
+// Sync from scratch
+func TestHandshakeReplayAll(t *testing.T) {
+	for _, m := range modes {
+		testHandshakeReplay(t, 0, m)
+	}
+}
+
+// Sync many, not from scratch
+func TestHandshakeReplaySome(t *testing.T) {
+	for _, m := range modes {
+		testHandshakeReplay(t, 1, m)
+	}
+}
+
+// Sync from lagging by one
+func TestHandshakeReplayOne(t *testing.T) {
+	for _, m := range modes {
+		testHandshakeReplay(t, NUM_BLOCKS-1, m)
+	}
+}
+
+// Sync from caught up
+func TestHandshakeReplayNone(t *testing.T) {
+	for _, m := range modes {
+		testHandshakeReplay(t, NUM_BLOCKS, m)
+	}
+}
+
+// Make some blocks. Start a fresh app and apply nBlocks blocks. Then restart the app and sync it up with the remaining blocks
+func testHandshakeReplay(t *testing.T, nBlocks int, mode uint) {
+	config := ResetConfig("proxy_test_")
+
+	// copy the many_blocks file
+	walBody, err := cmn.ReadFile(path.Join(data_dir, "many_blocks.cswal"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	walFile := writeWAL(walBody)
+	config.Consensus.SetWalFile(walFile)
+
+	privVal := types.LoadPrivValidatorFS(config.PrivValidatorFile())
+
+	wal, err := NewWAL(walFile, false)
+	if err != nil {
+		t.Fatal(err)
+	}
+	wal.SetLogger(log.TestingLogger())
+	if _, err := wal.Start(); err != nil {
+		t.Fatal(err)
+	}
+	chain, commits, err := makeBlockchainFromWAL(wal)
+	if err != nil {
+		t.Fatalf(err.Error())
+	}
+
+	state, store := stateAndStore(config, privVal.GetPubKey())
+	store.chain = chain
+	store.commits = commits
+
+	// run the chain through state.ApplyBlock to build up the tendermint state
+	latestAppHash := buildTMStateFromChain(config, state, chain, mode)
+
+	// make a new client creator
+	dummyApp := dummy.NewPersistentDummyApplication(path.Join(config.DBDir(), "2"))
+	clientCreator2 := proxy.NewLocalClientCreator(dummyApp)
+	if nBlocks > 0 {
+		// run nBlocks against a new client to build up the app state.
+		// use a throwaway tendermint state
+		proxyApp := proxy.NewAppConns(clientCreator2, nil)
+		state, _ := stateAndStore(config, privVal.GetPubKey())
+		buildAppStateFromChain(proxyApp, state, chain, nBlocks, mode)
+	}
+
+	// now start the app using the handshake - it should sync
+	handshaker := NewHandshaker(state, store)
+	proxyApp := proxy.NewAppConns(clientCreator2, handshaker)
+	if _, err := proxyApp.Start(); err != nil {
+		t.Fatalf("Error starting proxy app connections: %v", err)
+	}
+
+	// get the latest app hash from the app
+	res, err := proxyApp.Query().InfoSync(abci.RequestInfo{""})
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	// the app hash should be synced up
+	if !bytes.Equal(latestAppHash, res.LastBlockAppHash) {
+		t.Fatalf("Expected app hashes to match after handshake/replay. got %X, expected %X", res.LastBlockAppHash, latestAppHash)
+	}
+
+	expectedBlocksToSync := NUM_BLOCKS - nBlocks
+	if nBlocks == NUM_BLOCKS && mode > 0 {
+		expectedBlocksToSync += 1
+	} else if nBlocks > 0 && mode == 1 {
+		expectedBlocksToSync += 1
+	}
+
+	if handshaker.NBlocks() != expectedBlocksToSync {
+		t.Fatalf("Expected handshake to sync %d blocks, got %d", expectedBlocksToSync, handshaker.NBlocks())
+	}
+}
+
+func applyBlock(st *sm.State, blk *types.Block, proxyApp proxy.AppConns) {
+	testPartSize := st.Params.BlockPartSizeBytes
+	err := st.ApplyBlock(nil, proxyApp.Consensus(), blk, blk.MakePartSet(testPartSize).Header(), mempool)
+	if err != nil {
+		panic(err)
+	}
+}
+
+func buildAppStateFromChain(proxyApp proxy.AppConns,
+	state *sm.State, chain []*types.Block, nBlocks int, mode uint) {
+	// start a new app without handshake, play nBlocks blocks
+	if _, err := proxyApp.Start(); err != nil {
+		panic(err)
+	}
+
+	validators := types.TM2PB.Validators(state.Validators)
+	proxyApp.Consensus().InitChainSync(abci.RequestInitChain{validators})
+
+	defer proxyApp.Stop()
+	switch mode {
+	case 0:
+		for i := 0; i < nBlocks; i++ {
+			block := chain[i]
+			applyBlock(state, block, proxyApp)
+		}
+	case 1, 2:
+		for i := 0; i < nBlocks-1; i++ {
+			block := chain[i]
+			applyBlock(state, block, proxyApp)
+		}
+
+		if mode == 2 {
+			// update the dummy height and apphash
+			// as if we ran commit but not
+			applyBlock(state, chain[nBlocks-1], proxyApp)
+		}
+	}
+
+}
+
+func buildTMStateFromChain(config *cfg.Config, state *sm.State, chain []*types.Block, mode uint) []byte {
+	// run the whole chain against this client to build up the tendermint state
+	clientCreator := proxy.NewLocalClientCreator(dummy.NewPersistentDummyApplication(path.Join(config.DBDir(), "1")))
+	proxyApp := proxy.NewAppConns(clientCreator, nil) // sm.NewHandshaker(config, state, store, ReplayLastBlock))
+	if _, err := proxyApp.Start(); err != nil {
+		panic(err)
+	}
+	defer proxyApp.Stop()
+
+	validators := types.TM2PB.Validators(state.Validators)
+	proxyApp.Consensus().InitChainSync(abci.RequestInitChain{validators})
+
+	var latestAppHash []byte
+
+	switch mode {
+	case 0:
+		// sync right up
+		for _, block := range chain {
+			applyBlock(state, block, proxyApp)
+		}
+
+		latestAppHash = state.AppHash
+	case 1, 2:
+		// sync up to the penultimate as if we stored the block.
+		// whether we commit or not depends on the appHash
+		for _, block := range chain[:len(chain)-1] {
+			applyBlock(state, block, proxyApp)
+		}
+
+		// apply the final block to a state copy so we can
+		// get the right next appHash but keep the state back
+		stateCopy := state.Copy()
+		applyBlock(stateCopy, chain[len(chain)-1], proxyApp)
+		latestAppHash = stateCopy.AppHash
+	}
+
+	return latestAppHash
+}
+
+//--------------------------
+// utils for making blocks
+
+func makeBlockchainFromWAL(wal *WAL) ([]*types.Block, []*types.Commit, error) {
+	// Search for height marker
+	gr, found, err := wal.SearchForEndHeight(0)
+	if err != nil {
+		return nil, nil, err
+	}
+	if !found {
+		return nil, nil, errors.New(cmn.Fmt("WAL does not contain height %d.", 1))
+	}
+	defer gr.Close()
+
+	// log.Notice("Build a blockchain by reading from the WAL")
+
+	var blockParts *types.PartSet
+	var blocks []*types.Block
+	var commits []*types.Commit
+
+	dec := NewWALDecoder(gr)
+	for {
+		msg, err := dec.Decode()
+		if err == io.EOF {
+			break
+		} else if err != nil {
+			return nil, nil, err
+		}
+
+		piece := readPieceFromWAL(msg)
+		if piece == nil {
+			continue
+		}
+
+		switch p := piece.(type) {
+		case *types.PartSetHeader:
+			// if its not the first one, we have a full block
+			if blockParts != nil {
+				var n int
+				block := wire.ReadBinary(&types.Block{}, blockParts.GetReader(), 0, &n, &err).(*types.Block)
+				blocks = append(blocks, block)
+			}
+			blockParts = types.NewPartSetFromHeader(*p)
+		case *types.Part:
+			_, err := blockParts.AddPart(p, false)
+			if err != nil {
+				return nil, nil, err
+			}
+		case *types.Vote:
+			if p.Type == types.VoteTypePrecommit {
+				commit := &types.Commit{
+					BlockID:    p.BlockID,
+					Precommits: []*types.Vote{p},
+				}
+				commits = append(commits, commit)
+			}
+		}
+	}
+	// grab the last block too
+	var n int
+	block := wire.ReadBinary(&types.Block{}, blockParts.GetReader(), 0, &n, &err).(*types.Block)
+	blocks = append(blocks, block)
+	return blocks, commits, nil
+}
+
+func readPieceFromWAL(msg *TimedWALMessage) interface{} {
+	// skip meta messages
+	if _, ok := msg.Msg.(EndHeightMessage); ok {
+		return nil
+	}
+
+	// for logging
+	switch m := msg.Msg.(type) {
+	case msgInfo:
+		switch msg := m.Msg.(type) {
+		case *ProposalMessage:
+			return &msg.Proposal.BlockPartsHeader
+		case *BlockPartMessage:
+			return msg.Part
+		case *VoteMessage:
+			return msg.Vote
+		}
+	}
+
+	return nil
+}
+
+// fresh state and mock store
+func stateAndStore(config *cfg.Config, pubKey crypto.PubKey) (*sm.State, *mockBlockStore) {
+	stateDB := dbm.NewMemDB()
+	state, _ := sm.MakeGenesisStateFromFile(stateDB, config.GenesisFile())
+	state.SetLogger(log.TestingLogger().With("module", "state"))
+
+	store := NewMockBlockStore(config, state.Params)
+	return state, store
+}
+
+//----------------------------------
+// mock block store
+
+type mockBlockStore struct {
+	config  *cfg.Config
+	params  types.ConsensusParams
+	chain   []*types.Block
+	commits []*types.Commit
+}
+
+// TODO: NewBlockStore(db.NewMemDB) ...
+func NewMockBlockStore(config *cfg.Config, params types.ConsensusParams) *mockBlockStore {
+	return &mockBlockStore{config, params, nil, nil}
+}
+
+func (bs *mockBlockStore) Height() int                       { return len(bs.chain) }
+func (bs *mockBlockStore) LoadBlock(height int) *types.Block { return bs.chain[height-1] }
+func (bs *mockBlockStore) LoadBlockMeta(height int) *types.BlockMeta {
+	block := bs.chain[height-1]
+	return &types.BlockMeta{
+		BlockID: types.BlockID{block.Hash(), block.MakePartSet(bs.params.BlockPartSizeBytes).Header()},
+		Header:  block.Header,
+	}
+}
+func (bs *mockBlockStore) LoadBlockPart(height int, index int) *types.Part { return nil }
+func (bs *mockBlockStore) SaveBlock(block *types.Block, blockParts *types.PartSet, seenCommit *types.Commit) {
+}
+func (bs *mockBlockStore) LoadBlockCommit(height int) *types.Commit {
+	return bs.commits[height-1]
+}
+func (bs *mockBlockStore) LoadSeenCommit(height int) *types.Commit {
+	return bs.commits[height-1]
 }

consensus/state_test.go

@@ -6,18 +6,17 @@ import (
 	"testing"
 	"time"
 
-	"github.com/tendermint/tendermint/config/tendermint_test"
-	//"github.com/tendermint/go-events"
-	. "github.com/tendermint/go-common"
+	cstypes "github.com/tendermint/tendermint/consensus/types"
 	"github.com/tendermint/tendermint/types"
+	cmn "github.com/tendermint/tmlibs/common"
 )
 
 func init() {
-	config = tendermint_test.ResetConfig("consensus_state_test")
+	config = ResetConfig("consensus_state_test")
 }
 
-func (tp *TimeoutParams) ensureProposeTimeout() time.Duration {
-	return time.Duration(tp.Propose0*2) * time.Millisecond
+func ensureProposeTimeout(timeoutPropose int) time.Duration {
+	return time.Duration(timeoutPropose*2) * time.Millisecond
 }
 
 /*
@@ -31,7 +30,7 @@ x * TestBadProposal - 2 vals, bad proposal (bad block state hash), should prevot
 FullRoundSuite
 x * TestFullRound1 - 1 val, full successful round
 x * TestFullRoundNil - 1 val, full round of nil
-x * TestFullRound2 - 2 vals, both required for fuill round
+x * TestFullRound2 - 2 vals, both required for full round
 LockSuite
 x * TestLockNoPOL - 2 vals, 4 rounds. one val locked, precommits nil every round except first.
 x * TestLockPOLRelock - 4 vals, one precommits, other 3 polka at next round, so we unlock and precomit the polka
@@ -66,23 +65,23 @@ func TestProposerSelection0(t *testing.T) {
 	<-newRoundCh
 
 	// lets commit a block and ensure proposer for the next height is correct
-	prop := cs1.GetRoundState().Validators.Proposer()
-	if !bytes.Equal(prop.Address, cs1.privValidator.Address) {
-		panic(Fmt("expected proposer to be validator %d. Got %X", 0, prop.Address))
+	prop := cs1.GetRoundState().Validators.GetProposer()
+	if !bytes.Equal(prop.Address, cs1.privValidator.GetAddress()) {
+		t.Fatalf("expected proposer to be validator %d. Got %X", 0, prop.Address)
 	}
 
 	// wait for complete proposal
 	<-proposalCh
 
 	rs := cs1.GetRoundState()
-	signAddVoteToFromMany(types.VoteTypePrecommit, cs1, rs.ProposalBlock.Hash(), rs.ProposalBlockParts.Header(), nil, vss[1:]...)
+	signAddVotes(cs1, types.VoteTypePrecommit, rs.ProposalBlock.Hash(), rs.ProposalBlockParts.Header(), vss[1:]...)
 
 	// wait for new round so next validator is set
 	<-newRoundCh
 
-	prop = cs1.GetRoundState().Validators.Proposer()
-	if !bytes.Equal(prop.Address, vss[1].Address) {
-		panic(Fmt("expected proposer to be validator %d. Got %X", 1, prop.Address))
+	prop = cs1.GetRoundState().Validators.GetProposer()
+	if !bytes.Equal(prop.Address, vss[1].GetAddress()) {
+		panic(cmn.Fmt("expected proposer to be validator %d. Got %X", 1, prop.Address))
 	}
 }
 
@@ -101,13 +100,13 @@ func TestProposerSelection2(t *testing.T) {
 
 	// everyone just votes nil. we get a new proposer each round
 	for i := 0; i < len(vss); i++ {
-		prop := cs1.GetRoundState().Validators.Proposer()
-		if !bytes.Equal(prop.Address, vss[(i+2)%len(vss)].Address) {
-			panic(Fmt("expected proposer to be validator %d. Got %X", (i+2)%len(vss), prop.Address))
+		prop := cs1.GetRoundState().Validators.GetProposer()
+		if !bytes.Equal(prop.Address, vss[(i+2)%len(vss)].GetAddress()) {
+			panic(cmn.Fmt("expected proposer to be validator %d. Got %X", (i+2)%len(vss), prop.Address))
 		}
 
 		rs := cs1.GetRoundState()
-		signAddVoteToFromMany(types.VoteTypePrecommit, cs1, nil, rs.ProposalBlockParts.Header(), nil, vss[1:]...)
+		signAddVotes(cs1, types.VoteTypePrecommit, nil, rs.ProposalBlockParts.Header(), vss[1:]...)
 		<-newRoundCh // wait for the new round event each round
 
 		incrementRound(vss[1:]...)
@@ -127,7 +126,7 @@ func TestEnterProposeNoPrivValidator(t *testing.T) {
 	startTestRound(cs, height, round)
 
 	// if we're not a validator, EnterPropose should timeout
-	ticker := time.NewTicker(cs.timeoutParams.ensureProposeTimeout())
+	ticker := time.NewTicker(ensureProposeTimeout(cs.config.TimeoutPropose))
 	select {
 	case <-timeoutCh:
 	case <-ticker.C:
@@ -168,7 +167,7 @@ func TestEnterProposeYesPrivValidator(t *testing.T) {
 	}
 
 	// if we're a validator, enterPropose should not timeout
-	ticker := time.NewTicker(cs.timeoutParams.ensureProposeTimeout())
+	ticker := time.NewTicker(ensureProposeTimeout(cs.config.TimeoutPropose))
 	select {
 	case <-timeoutCh:
 		panic("Expected EnterPropose not to timeout")
@@ -180,12 +179,14 @@ func TestEnterProposeYesPrivValidator(t *testing.T) {
 func TestBadProposal(t *testing.T) {
 	cs1, vss := randConsensusState(2)
 	height, round := cs1.Height, cs1.Round
-	cs2 := vss[1]
+	vs2 := vss[1]
+
+	partSize := cs1.state.Params.BlockPartSizeBytes
 
 	proposalCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringCompleteProposal(), 1)
 	voteCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringVote(), 1)
 
-	propBlock, _ := cs1.createProposalBlock() //changeProposer(t, cs1, cs2)
+	propBlock, _ := cs1.createProposalBlock() //changeProposer(t, cs1, vs2)
 
 	// make the second validator the proposer by incrementing round
 	round = round + 1
@@ -198,10 +199,10 @@ func TestBadProposal(t *testing.T) {
 	}
 	stateHash[0] = byte((stateHash[0] + 1) % 255)
 	propBlock.AppHash = stateHash
-	propBlockParts := propBlock.MakePartSet()
-	proposal := types.NewProposal(cs2.Height, round, propBlockParts.Header(), -1)
-	if err := cs2.SignProposal(config.GetString("chain_id"), proposal); err != nil {
-		panic("failed to sign bad proposal: " + err.Error())
+	propBlockParts := propBlock.MakePartSet(partSize)
+	proposal := types.NewProposal(vs2.Height, round, propBlockParts.Header(), -1, types.BlockID{})
+	if err := vs2.SignProposal(config.ChainID, proposal); err != nil {
+		t.Fatal("failed to sign bad proposal", err)
 	}
 
 	// set the proposal block
@@ -218,14 +219,15 @@ func TestBadProposal(t *testing.T) {
 
 	validatePrevote(t, cs1, round, vss[0], nil)
 
-	// add bad prevote from cs2 and wait for it
-	signAddVoteToFrom(types.VoteTypePrevote, cs1, cs2, propBlock.Hash(), propBlock.MakePartSet().Header(), voteCh)
+	// add bad prevote from vs2 and wait for it
+	signAddVotes(cs1, types.VoteTypePrevote, propBlock.Hash(), propBlock.MakePartSet(partSize).Header(), vs2)
+	<-voteCh
 
 	// wait for precommit
 	<-voteCh
 
 	validatePrecommit(t, cs1, round, 0, vss[0], nil, nil)
-	signAddVoteToFrom(types.VoteTypePrecommit, cs1, cs2, propBlock.Hash(), propBlock.MakePartSet().Header(), voteCh)
+	signAddVotes(cs1, types.VoteTypePrecommit, propBlock.Hash(), propBlock.MakePartSet(partSize).Header(), vs2)
 }
 
 //----------------------------------------------------------------------------------------------------
@@ -246,7 +248,7 @@ func TestFullRound1(t *testing.T) {
 
 	// grab proposal
 	re := <-propCh
-	propBlockHash := re.(types.EventDataRoundState).RoundState.(*RoundState).ProposalBlock.Hash()
+	propBlockHash := re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*cstypes.RoundState).ProposalBlock.Hash()
 
 	<-voteCh // wait for prevote
 	// NOTE: voteChan cap of 0 ensures we can complete this
@@ -282,7 +284,7 @@ func TestFullRoundNil(t *testing.T) {
 // where the first validator has to wait for votes from the second
 func TestFullRound2(t *testing.T) {
 	cs1, vss := randConsensusState(2)
-	cs2 := vss[1]
+	vs2 := vss[1]
 	height, round := cs1.Height, cs1.Round
 
 	voteCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringVote(), 1)
@@ -297,8 +299,9 @@ func TestFullRound2(t *testing.T) {
 	rs := cs1.GetRoundState()
 	propBlockHash, propPartsHeader := rs.ProposalBlock.Hash(), rs.ProposalBlockParts.Header()
 
-	// prevote arrives from cs2:
-	signAddVoteToFrom(types.VoteTypePrevote, cs1, cs2, propBlockHash, propPartsHeader, voteCh)
+	// prevote arrives from vs2:
+	signAddVotes(cs1, types.VoteTypePrevote, propBlockHash, propPartsHeader, vs2)
+	<-voteCh
 
 	<-voteCh //precommit
 
@@ -307,8 +310,9 @@ func TestFullRound2(t *testing.T) {
 
 	// we should be stuck in limbo waiting for more precommits
 
-	// precommit arrives from cs2:
-	signAddVoteToFrom(types.VoteTypePrecommit, cs1, cs2, propBlockHash, propPartsHeader, voteCh)
+	// precommit arrives from vs2:
+	signAddVotes(cs1, types.VoteTypePrecommit, propBlockHash, propPartsHeader, vs2)
+	<-voteCh
 
 	// wait to finish commit, propose in next height
 	<-newBlockCh
@@ -321,9 +325,11 @@ func TestFullRound2(t *testing.T) {
 // two vals take turns proposing. val1 locks on first one, precommits nil on everything else
 func TestLockNoPOL(t *testing.T) {
 	cs1, vss := randConsensusState(2)
-	cs2 := vss[1]
+	vs2 := vss[1]
 	height := cs1.Height
 
+	partSize := cs1.state.Params.BlockPartSizeBytes
+
 	timeoutProposeCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringTimeoutPropose(), 1)
 	timeoutWaitCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringTimeoutWait(), 1)
 	voteCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringVote(), 1)
@@ -339,14 +345,15 @@ func TestLockNoPOL(t *testing.T) {
 	cs1.startRoutines(0)
 
 	re := <-proposalCh
-	rs := re.(types.EventDataRoundState).RoundState.(*RoundState)
+	rs := re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*cstypes.RoundState)
 	theBlockHash := rs.ProposalBlock.Hash()
 
 	<-voteCh // prevote
 
 	// we should now be stuck in limbo forever, waiting for more prevotes
-	// prevote arrives from cs2:
-	signAddVoteToFrom(types.VoteTypePrevote, cs1, cs2, rs.ProposalBlock.Hash(), rs.ProposalBlockParts.Header(), voteCh)
+	// prevote arrives from vs2:
+	signAddVotes(cs1, types.VoteTypePrevote, cs1.ProposalBlock.Hash(), cs1.ProposalBlockParts.Header(), vs2)
+	<-voteCh // prevote
 
 	<-voteCh // precommit
 
@@ -359,7 +366,8 @@ func TestLockNoPOL(t *testing.T) {
 	hash := make([]byte, len(theBlockHash))
 	copy(hash, theBlockHash)
 	hash[0] = byte((hash[0] + 1) % 255)
-	signAddVoteToFrom(types.VoteTypePrecommit, cs1, cs2, hash, rs.ProposalBlock.MakePartSet().Header(), voteCh)
+	signAddVotes(cs1, types.VoteTypePrecommit, hash, rs.ProposalBlock.MakePartSet(partSize).Header(), vs2)
+	<-voteCh // precommit
 
 	// (note we're entering precommit for a second time this round)
 	// but with invalid args. then we enterPrecommitWait, and the timeout to new round
@@ -368,16 +376,16 @@ func TestLockNoPOL(t *testing.T) {
 	///
 
 	<-newRoundCh
-	log.Notice("#### ONTO ROUND 1")
+	t.Log("#### ONTO ROUND 1")
 	/*
 		Round2 (cs1, B) // B B2
 	*/
 
-	incrementRound(cs2)
+	incrementRound(vs2)
 
 	// now we're on a new round and not the proposer, so wait for timeout
 	re = <-timeoutProposeCh
-	rs = re.(types.EventDataRoundState).RoundState.(*RoundState)
+	rs = re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*cstypes.RoundState)
 
 	if rs.ProposalBlock != nil {
 		panic("Expected proposal block to be nil")
@@ -390,7 +398,8 @@ func TestLockNoPOL(t *testing.T) {
 	validatePrevote(t, cs1, 1, vss[0], rs.LockedBlock.Hash())
 
 	// add a conflicting prevote from the other validator
-	signAddVoteToFrom(types.VoteTypePrevote, cs1, cs2, hash, rs.ProposalBlock.MakePartSet().Header(), voteCh)
+	signAddVotes(cs1, types.VoteTypePrevote, hash, rs.ProposalBlock.MakePartSet(partSize).Header(), vs2)
+	<-voteCh
 
 	// now we're going to enter prevote again, but with invalid args
 	// and then prevote wait, which should timeout. then wait for precommit
@@ -402,61 +411,65 @@ func TestLockNoPOL(t *testing.T) {
 	// we should precommit nil and be locked on the proposal
 	validatePrecommit(t, cs1, 1, 0, vss[0], nil, theBlockHash)
 
-	// add conflicting precommit from cs2
+	// add conflicting precommit from vs2
 	// NOTE: in practice we should never get to a point where there are precommits for different blocks at the same round
-	signAddVoteToFrom(types.VoteTypePrecommit, cs1, cs2, hash, rs.ProposalBlock.MakePartSet().Header(), voteCh)
+	signAddVotes(cs1, types.VoteTypePrecommit, hash, rs.ProposalBlock.MakePartSet(partSize).Header(), vs2)
+	<-voteCh
 
 	// (note we're entering precommit for a second time this round, but with invalid args
 	// then we enterPrecommitWait and timeout into NewRound
 	<-timeoutWaitCh
 
 	<-newRoundCh
-	log.Notice("#### ONTO ROUND 2")
+	t.Log("#### ONTO ROUND 2")
 	/*
-		Round3 (cs2, _) // B, B2
+		Round3 (vs2, _) // B, B2
 	*/
 
-	incrementRound(cs2)
+	incrementRound(vs2)
 
 	re = <-proposalCh
-	rs = re.(types.EventDataRoundState).RoundState.(*RoundState)
+	rs = re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*cstypes.RoundState)
 
 	// now we're on a new round and are the proposer
 	if !bytes.Equal(rs.ProposalBlock.Hash(), rs.LockedBlock.Hash()) {
-		panic(Fmt("Expected proposal block to be locked block. Got %v, Expected %v", rs.ProposalBlock, rs.LockedBlock))
+		panic(cmn.Fmt("Expected proposal block to be locked block. Got %v, Expected %v", rs.ProposalBlock, rs.LockedBlock))
 	}
 
 	<-voteCh // prevote
 
 	validatePrevote(t, cs1, 2, vss[0], rs.LockedBlock.Hash())
 
-	signAddVoteToFrom(types.VoteTypePrevote, cs1, cs2, hash, rs.ProposalBlock.MakePartSet().Header(), voteCh)
+	signAddVotes(cs1, types.VoteTypePrevote, hash, rs.ProposalBlock.MakePartSet(partSize).Header(), vs2)
+	<-voteCh
 
 	<-timeoutWaitCh // prevote wait
 	<-voteCh        // precommit
 
-	validatePrecommit(t, cs1, 2, 0, vss[0], nil, theBlockHash)                                                  // precommit nil but be locked on proposal
-	signAddVoteToFrom(types.VoteTypePrecommit, cs1, cs2, hash, rs.ProposalBlock.MakePartSet().Header(), voteCh) // NOTE: conflicting precommits at same height
+	validatePrecommit(t, cs1, 2, 0, vss[0], nil, theBlockHash) // precommit nil but be locked on proposal
+
+	signAddVotes(cs1, types.VoteTypePrecommit, hash, rs.ProposalBlock.MakePartSet(partSize).Header(), vs2) // NOTE: conflicting precommits at same height
+	<-voteCh
 
 	<-timeoutWaitCh
 
 	// before we time out into new round, set next proposal block
-	prop, propBlock := decideProposal(cs1, cs2, cs2.Height, cs2.Round+1)
+	prop, propBlock := decideProposal(cs1, vs2, vs2.Height, vs2.Round+1)
 	if prop == nil || propBlock == nil {
-		panic("Failed to create proposal block with cs2")
+		t.Fatal("Failed to create proposal block with vs2")
 	}
 
-	incrementRound(cs2)
+	incrementRound(vs2)
 
 	<-newRoundCh
-	log.Notice("#### ONTO ROUND 3")
+	t.Log("#### ONTO ROUND 3")
 	/*
-		Round4 (cs2, C) // B C // B C
+		Round4 (vs2, C) // B C // B C
 	*/
 
 	// now we're on a new round and not the proposer
 	// so set the proposal block
-	cs1.SetProposalAndBlock(prop, propBlock, propBlock.MakePartSet(), "")
+	cs1.SetProposalAndBlock(prop, propBlock, propBlock.MakePartSet(partSize), "")
 
 	<-proposalCh
 	<-voteCh // prevote
@@ -464,19 +477,24 @@ func TestLockNoPOL(t *testing.T) {
 	// prevote for locked block (not proposal)
 	validatePrevote(t, cs1, 0, vss[0], cs1.LockedBlock.Hash())
 
-	signAddVoteToFrom(types.VoteTypePrevote, cs1, cs2, propBlock.Hash(), propBlock.MakePartSet().Header(), voteCh)
+	signAddVotes(cs1, types.VoteTypePrevote, propBlock.Hash(), propBlock.MakePartSet(partSize).Header(), vs2)
+	<-voteCh
 
 	<-timeoutWaitCh
 	<-voteCh
 
-	validatePrecommit(t, cs1, 2, 0, vss[0], nil, theBlockHash)                                                       // precommit nil but locked on proposal
-	signAddVoteToFrom(types.VoteTypePrecommit, cs1, cs2, propBlock.Hash(), propBlock.MakePartSet().Header(), voteCh) // NOTE: conflicting precommits at same height
+	validatePrecommit(t, cs1, 2, 0, vss[0], nil, theBlockHash) // precommit nil but locked on proposal
+
+	signAddVotes(cs1, types.VoteTypePrecommit, propBlock.Hash(), propBlock.MakePartSet(partSize).Header(), vs2) // NOTE: conflicting precommits at same height
+	<-voteCh
 }
 
 // 4 vals, one precommits, other 3 polka at next round, so we unlock and precomit the polka
 func TestLockPOLRelock(t *testing.T) {
 	cs1, vss := randConsensusState(4)
-	cs2, cs3, cs4 := vss[1], vss[2], vss[3]
+	vs2, vs3, vs4 := vss[1], vss[2], vss[3]
+
+	partSize := cs1.state.Params.BlockPartSizeBytes
 
 	timeoutProposeCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringTimeoutPropose(), 1)
 	timeoutWaitCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringTimeoutWait(), 1)
@@ -485,14 +503,12 @@ func TestLockPOLRelock(t *testing.T) {
 	newRoundCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringNewRound(), 1)
 	newBlockCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringNewBlockHeader(), 1)
 
-	log.Debug("cs2 last round", "lr", cs2.PrivValidator.LastRound)
-
 	// everything done from perspective of cs1
 
 	/*
 		Round1 (cs1, B) // B B B B// B nil B nil
 
-		eg. cs2 and cs4 didn't see the 2/3 prevotes
+		eg. vs2 and vs4 didn't see the 2/3 prevotes
 	*/
 
 	// start round and wait for propose and prevote
@@ -500,40 +516,43 @@ func TestLockPOLRelock(t *testing.T) {
 
 	<-newRoundCh
 	re := <-proposalCh
-	rs := re.(types.EventDataRoundState).RoundState.(*RoundState)
+	rs := re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*cstypes.RoundState)
 	theBlockHash := rs.ProposalBlock.Hash()
-	theBlockPartsHeader := rs.ProposalBlockParts.Header()
 
 	<-voteCh // prevote
 
-	signAddVoteToFromMany(types.VoteTypePrevote, cs1, theBlockHash, theBlockPartsHeader, voteCh, cs2, cs3, cs4)
+	signAddVotes(cs1, types.VoteTypePrevote, cs1.ProposalBlock.Hash(), cs1.ProposalBlockParts.Header(), vs2, vs3, vs4)
+	// prevotes
+	discardFromChan(voteCh, 3)
 
 	<-voteCh // our precommit
 	// the proposed block should now be locked and our precommit added
 	validatePrecommit(t, cs1, 0, 0, vss[0], theBlockHash, theBlockHash)
 
 	// add precommits from the rest
-	signAddVoteToFromMany(types.VoteTypePrecommit, cs1, nil, types.PartSetHeader{}, voteCh, cs2, cs4)
-	signAddVoteToFrom(types.VoteTypePrecommit, cs1, cs3, theBlockHash, theBlockPartsHeader, voteCh)
+	signAddVotes(cs1, types.VoteTypePrecommit, nil, types.PartSetHeader{}, vs2, vs4)
+	signAddVotes(cs1, types.VoteTypePrecommit, cs1.ProposalBlock.Hash(), cs1.ProposalBlockParts.Header(), vs3)
+	// precommites
+	discardFromChan(voteCh, 3)
 
 	// before we timeout to the new round set the new proposal
-	prop, propBlock := decideProposal(cs1, cs2, cs2.Height, cs2.Round+1)
-	propBlockParts := propBlock.MakePartSet()
+	prop, propBlock := decideProposal(cs1, vs2, vs2.Height, vs2.Round+1)
+	propBlockParts := propBlock.MakePartSet(partSize)
 	propBlockHash := propBlock.Hash()
 
-	incrementRound(cs2, cs3, cs4)
+	incrementRound(vs2, vs3, vs4)
 
 	// timeout to new round
 	<-timeoutWaitCh
 
-	//XXX: this isnt gauranteed to get there before the timeoutPropose ...
+	//XXX: this isnt guaranteed to get there before the timeoutPropose ...
 	cs1.SetProposalAndBlock(prop, propBlock, propBlockParts, "some peer")
 
 	<-newRoundCh
-	log.Notice("### ONTO ROUND 1")
+	t.Log("### ONTO ROUND 1")
 
 	/*
-		Round2 (cs2, C) // B C C C // C C C _)
+		Round2 (vs2, C) // B C C C // C C C _)
 
 		cs1 changes lock!
 	*/
@@ -551,7 +570,9 @@ func TestLockPOLRelock(t *testing.T) {
 	validatePrevote(t, cs1, 0, vss[0], theBlockHash)
 
 	// now lets add prevotes from everyone else for the new block
-	signAddVoteToFromMany(types.VoteTypePrevote, cs1, propBlockHash, propBlockParts.Header(), voteCh, cs2, cs3, cs4)
+	signAddVotes(cs1, types.VoteTypePrevote, propBlockHash, propBlockParts.Header(), vs2, vs3, vs4)
+	// prevotes
+	discardFromChan(voteCh, 3)
 
 	// now either we go to PrevoteWait or Precommit
 	select {
@@ -565,12 +586,13 @@ func TestLockPOLRelock(t *testing.T) {
 	// we should have unlocked and locked on the new block
 	validatePrecommit(t, cs1, 1, 1, vss[0], propBlockHash, propBlockHash)
 
-	signAddVoteToFromMany(types.VoteTypePrecommit, cs1, propBlockHash, propBlockParts.Header(), voteCh, cs2, cs3)
+	signAddVotes(cs1, types.VoteTypePrecommit, propBlockHash, propBlockParts.Header(), vs2, vs3)
+	discardFromChan(voteCh, 2)
 
 	be := <-newBlockCh
-	b := be.(types.EventDataNewBlockHeader)
+	b := be.(types.TMEventData).Unwrap().(types.EventDataNewBlockHeader)
 	re = <-newRoundCh
-	rs = re.(types.EventDataRoundState).RoundState.(*RoundState)
+	rs = re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*cstypes.RoundState)
 	if rs.Height != 2 {
 		panic("Expected height to increment")
 	}
@@ -583,14 +605,16 @@ func TestLockPOLRelock(t *testing.T) {
 // 4 vals, one precommits, other 3 polka at next round, so we unlock and precomit the polka
 func TestLockPOLUnlock(t *testing.T) {
 	cs1, vss := randConsensusState(4)
-	cs2, cs3, cs4 := vss[1], vss[2], vss[3]
+	vs2, vs3, vs4 := vss[1], vss[2], vss[3]
+
+	partSize := cs1.state.Params.BlockPartSizeBytes
 
 	proposalCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringCompleteProposal(), 1)
 	timeoutProposeCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringTimeoutPropose(), 1)
 	timeoutWaitCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringTimeoutWait(), 1)
 	newRoundCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringNewRound(), 1)
 	unlockCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringUnlock(), 1)
-	voteCh := subscribeToVoter(cs1, cs1.privValidator.Address)
+	voteCh := subscribeToVoter(cs1, cs1.privValidator.GetAddress())
 
 	// everything done from perspective of cs1
 
@@ -604,12 +628,12 @@ func TestLockPOLUnlock(t *testing.T) {
 	startTestRound(cs1, cs1.Height, 0)
 	<-newRoundCh
 	re := <-proposalCh
-	rs := re.(types.EventDataRoundState).RoundState.(*RoundState)
+	rs := re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*cstypes.RoundState)
 	theBlockHash := rs.ProposalBlock.Hash()
 
 	<-voteCh // prevote
 
-	signAddVoteToFromMany(types.VoteTypePrevote, cs1, rs.ProposalBlock.Hash(), rs.ProposalBlockParts.Header(), nil, cs2, cs3, cs4)
+	signAddVotes(cs1, types.VoteTypePrevote, cs1.ProposalBlock.Hash(), cs1.ProposalBlockParts.Header(), vs2, vs3, vs4)
 
 	<-voteCh //precommit
 
@@ -619,27 +643,27 @@ func TestLockPOLUnlock(t *testing.T) {
 	rs = cs1.GetRoundState()
 
 	// add precommits from the rest
-	signAddVoteToFromMany(types.VoteTypePrecommit, cs1, nil, types.PartSetHeader{}, nil, cs2, cs4)
-	signAddVoteToFrom(types.VoteTypePrecommit, cs1, cs3, rs.ProposalBlock.Hash(), rs.ProposalBlockParts.Header(), nil)
+	signAddVotes(cs1, types.VoteTypePrecommit, nil, types.PartSetHeader{}, vs2, vs4)
+	signAddVotes(cs1, types.VoteTypePrecommit, cs1.ProposalBlock.Hash(), cs1.ProposalBlockParts.Header(), vs3)
 
 	// before we time out into new round, set next proposal block
-	prop, propBlock := decideProposal(cs1, cs2, cs2.Height, cs2.Round+1)
-	propBlockParts := propBlock.MakePartSet()
+	prop, propBlock := decideProposal(cs1, vs2, vs2.Height, vs2.Round+1)
+	propBlockParts := propBlock.MakePartSet(partSize)
 
-	incrementRound(cs2, cs3, cs4)
+	incrementRound(vs2, vs3, vs4)
 
 	// timeout to new round
 	re = <-timeoutWaitCh
-	rs = re.(types.EventDataRoundState).RoundState.(*RoundState)
+	rs = re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*cstypes.RoundState)
 	lockedBlockHash := rs.LockedBlock.Hash()
 
-	//XXX: this isnt gauranteed to get there before the timeoutPropose ...
+	//XXX: this isnt guaranteed to get there before the timeoutPropose ...
 	cs1.SetProposalAndBlock(prop, propBlock, propBlockParts, "some peer")
 
 	<-newRoundCh
-	log.Notice("#### ONTO ROUND 1")
+	t.Log("#### ONTO ROUND 1")
 	/*
-		Round2 (cs2, C) // B nil nil nil // nil nil nil _
+		Round2 (vs2, C) // B nil nil nil // nil nil nil _
 
 		cs1 unlocks!
 	*/
@@ -656,7 +680,7 @@ func TestLockPOLUnlock(t *testing.T) {
 	<-voteCh
 	validatePrevote(t, cs1, 0, vss[0], lockedBlockHash)
 	// now lets add prevotes from everyone else for nil (a polka!)
-	signAddVoteToFromMany(types.VoteTypePrevote, cs1, nil, types.PartSetHeader{}, nil, cs2, cs3, cs4)
+	signAddVotes(cs1, types.VoteTypePrevote, nil, types.PartSetHeader{}, vs2, vs3, vs4)
 
 	// the polka makes us unlock and precommit nil
 	<-unlockCh
@@ -666,7 +690,7 @@ func TestLockPOLUnlock(t *testing.T) {
 	// NOTE: since we don't relock on nil, the lock round is 0
 	validatePrecommit(t, cs1, 1, 0, vss[0], nil, nil)
 
-	signAddVoteToFromMany(types.VoteTypePrecommit, cs1, nil, types.PartSetHeader{}, nil, cs2, cs3)
+	signAddVotes(cs1, types.VoteTypePrecommit, nil, types.PartSetHeader{}, vs2, vs3)
 	<-newRoundCh
 }
 
@@ -676,19 +700,21 @@ func TestLockPOLUnlock(t *testing.T) {
 // then we see the polka from round 1 but shouldn't unlock
 func TestLockPOLSafety1(t *testing.T) {
 	cs1, vss := randConsensusState(4)
-	cs2, cs3, cs4 := vss[1], vss[2], vss[3]
+	vs2, vs3, vs4 := vss[1], vss[2], vss[3]
+
+	partSize := cs1.state.Params.BlockPartSizeBytes
 
 	proposalCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringCompleteProposal(), 1)
 	timeoutProposeCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringTimeoutPropose(), 1)
 	timeoutWaitCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringTimeoutWait(), 1)
 	newRoundCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringNewRound(), 1)
-	voteCh := subscribeToVoter(cs1, cs1.privValidator.Address)
+	voteCh := subscribeToVoter(cs1, cs1.privValidator.GetAddress())
 
 	// start round and wait for propose and prevote
 	startTestRound(cs1, cs1.Height, 0)
 	<-newRoundCh
 	re := <-proposalCh
-	rs := re.(types.EventDataRoundState).RoundState.(*RoundState)
+	rs := re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*cstypes.RoundState)
 	propBlock := rs.ProposalBlock
 
 	<-voteCh // prevote
@@ -696,7 +722,7 @@ func TestLockPOLSafety1(t *testing.T) {
 	validatePrevote(t, cs1, 0, vss[0], propBlock.Hash())
 
 	// the others sign a polka but we don't see it
-	prevotes := signVoteMany(types.VoteTypePrevote, propBlock.Hash(), propBlock.MakePartSet().Header(), cs2, cs3, cs4)
+	prevotes := signVotes(types.VoteTypePrevote, propBlock.Hash(), propBlock.MakePartSet(partSize).Header(), vs2, vs3, vs4)
 
 	// before we time out into new round, set next proposer
 	// and next proposal block
@@ -707,22 +733,22 @@ func TestLockPOLSafety1(t *testing.T) {
 			panic("failed to update validator")
 		}*/
 
-	log.Warn("old prop", "hash", fmt.Sprintf("%X", propBlock.Hash()))
+	t.Logf("old prop hash %v", fmt.Sprintf("%X", propBlock.Hash()))
 
 	// we do see them precommit nil
-	signAddVoteToFromMany(types.VoteTypePrecommit, cs1, nil, types.PartSetHeader{}, nil, cs2, cs3, cs4)
+	signAddVotes(cs1, types.VoteTypePrecommit, nil, types.PartSetHeader{}, vs2, vs3, vs4)
 
-	prop, propBlock := decideProposal(cs1, cs2, cs2.Height, cs2.Round+1)
+	prop, propBlock := decideProposal(cs1, vs2, vs2.Height, vs2.Round+1)
 	propBlockHash := propBlock.Hash()
-	propBlockParts := propBlock.MakePartSet()
+	propBlockParts := propBlock.MakePartSet(partSize)
 
-	incrementRound(cs2, cs3, cs4)
+	incrementRound(vs2, vs3, vs4)
 
-	//XXX: this isnt gauranteed to get there before the timeoutPropose ...
+	//XXX: this isnt guaranteed to get there before the timeoutPropose ...
 	cs1.SetProposalAndBlock(prop, propBlock, propBlockParts, "some peer")
 
 	<-newRoundCh
-	log.Notice("### ONTO ROUND 1")
+	t.Log("### ONTO ROUND 1")
 	/*Round2
 	// we timeout and prevote our lock
 	// a polka happened but we didn't see it!
@@ -736,33 +762,33 @@ func TestLockPOLSafety1(t *testing.T) {
 		re = <-proposalCh
 	}
 
-	rs = re.(types.EventDataRoundState).RoundState.(*RoundState)
+	rs = re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*cstypes.RoundState)
 
 	if rs.LockedBlock != nil {
 		panic("we should not be locked!")
 	}
-	log.Warn("new prop", "hash", fmt.Sprintf("%X", propBlockHash))
+	t.Logf("new prop hash %v", fmt.Sprintf("%X", propBlockHash))
 	// go to prevote, prevote for proposal block
 	<-voteCh
 	validatePrevote(t, cs1, 1, vss[0], propBlockHash)
 
 	// now we see the others prevote for it, so we should lock on it
-	signAddVoteToFromMany(types.VoteTypePrevote, cs1, propBlockHash, propBlockParts.Header(), nil, cs2, cs3, cs4)
+	signAddVotes(cs1, types.VoteTypePrevote, propBlockHash, propBlockParts.Header(), vs2, vs3, vs4)
 
 	<-voteCh // precommit
 
 	// we should have precommitted
 	validatePrecommit(t, cs1, 1, 1, vss[0], propBlockHash, propBlockHash)
 
-	signAddVoteToFromMany(types.VoteTypePrecommit, cs1, nil, types.PartSetHeader{}, nil, cs2, cs3)
+	signAddVotes(cs1, types.VoteTypePrecommit, nil, types.PartSetHeader{}, vs2, vs3)
 
 	<-timeoutWaitCh
 
-	incrementRound(cs2, cs3, cs4)
+	incrementRound(vs2, vs3, vs4)
 
 	<-newRoundCh
 
-	log.Notice("### ONTO ROUND 2")
+	t.Log("### ONTO ROUND 2")
 	/*Round3
 	we see the polka from round 1 but we shouldn't unlock!
 	*/
@@ -779,9 +805,9 @@ func TestLockPOLSafety1(t *testing.T) {
 	newStepCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringNewRoundStep(), 1)
 
 	// add prevotes from the earlier round
-	addVoteToFromMany(cs1, prevotes, cs2, cs3, cs4)
+	addVotes(cs1, prevotes...)
 
-	log.Warn("Done adding prevotes!")
+	t.Log("Done adding prevotes!")
 
 	ensureNoNewStep(newStepCh)
 }
@@ -795,34 +821,37 @@ func TestLockPOLSafety1(t *testing.T) {
 // dont see P0, lock on P1 at R1, dont unlock using P0 at R2
 func TestLockPOLSafety2(t *testing.T) {
 	cs1, vss := randConsensusState(4)
-	cs2, cs3, cs4 := vss[1], vss[2], vss[3]
+	vs2, vs3, vs4 := vss[1], vss[2], vss[3]
+
+	partSize := cs1.state.Params.BlockPartSizeBytes
 
 	proposalCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringCompleteProposal(), 1)
 	timeoutProposeCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringTimeoutPropose(), 1)
 	timeoutWaitCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringTimeoutWait(), 1)
 	newRoundCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringNewRound(), 1)
 	unlockCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringUnlock(), 1)
-	voteCh := subscribeToVoter(cs1, cs1.privValidator.Address)
+	voteCh := subscribeToVoter(cs1, cs1.privValidator.GetAddress())
 
 	// the block for R0: gets polkad but we miss it
 	// (even though we signed it, shhh)
 	_, propBlock0 := decideProposal(cs1, vss[0], cs1.Height, cs1.Round)
 	propBlockHash0 := propBlock0.Hash()
-	propBlockParts0 := propBlock0.MakePartSet()
+	propBlockParts0 := propBlock0.MakePartSet(partSize)
 
 	// the others sign a polka but we don't see it
-	prevotes := signVoteMany(types.VoteTypePrevote, propBlockHash0, propBlockParts0.Header(), cs2, cs3, cs4)
+	prevotes := signVotes(types.VoteTypePrevote, propBlockHash0, propBlockParts0.Header(), vs2, vs3, vs4)
 
 	// the block for round 1
-	prop1, propBlock1 := decideProposal(cs1, cs2, cs2.Height, cs2.Round+1)
+	prop1, propBlock1 := decideProposal(cs1, vs2, vs2.Height, vs2.Round+1)
 	propBlockHash1 := propBlock1.Hash()
-	propBlockParts1 := propBlock1.MakePartSet()
+	propBlockParts1 := propBlock1.MakePartSet(partSize)
+	propBlockID1 := types.BlockID{propBlockHash1, propBlockParts1.Header()}
 
-	incrementRound(cs2, cs3, cs4)
+	incrementRound(vs2, vs3, vs4)
 
-	cs1.updateRoundStep(0, RoundStepPrecommitWait)
+	cs1.updateRoundStep(0, cstypes.RoundStepPrecommitWait)
 
-	log.Notice("### ONTO Round 1")
+	t.Log("### ONTO Round 1")
 	// jump in at round 1
 	height := cs1.Height
 	startTestRound(cs1, height, 1)
@@ -833,31 +862,33 @@ func TestLockPOLSafety2(t *testing.T) {
 
 	<-voteCh // prevote
 
-	signAddVoteToFromMany(types.VoteTypePrevote, cs1, propBlockHash1, propBlockParts1.Header(), nil, cs2, cs3, cs4)
+	signAddVotes(cs1, types.VoteTypePrevote, propBlockHash1, propBlockParts1.Header(), vs2, vs3, vs4)
 
 	<-voteCh // precommit
 	// the proposed block should now be locked and our precommit added
 	validatePrecommit(t, cs1, 1, 1, vss[0], propBlockHash1, propBlockHash1)
 
 	// add precommits from the rest
-	signAddVoteToFromMany(types.VoteTypePrecommit, cs1, nil, types.PartSetHeader{}, nil, cs2, cs4)
-	signAddVoteToFrom(types.VoteTypePrecommit, cs1, cs3, propBlockHash1, propBlockParts1.Header(), nil)
+	signAddVotes(cs1, types.VoteTypePrecommit, nil, types.PartSetHeader{}, vs2, vs4)
+	signAddVotes(cs1, types.VoteTypePrecommit, propBlockHash1, propBlockParts1.Header(), vs3)
 
-	incrementRound(cs2, cs3, cs4)
+	incrementRound(vs2, vs3, vs4)
 
 	// timeout of precommit wait to new round
 	<-timeoutWaitCh
 
 	// in round 2 we see the polkad block from round 0
-	newProp := types.NewProposal(height, 2, propBlockParts0.Header(), 0)
-	if err := cs3.SignProposal(config.GetString("chain_id"), newProp); err != nil {
-		panic(err)
+	newProp := types.NewProposal(height, 2, propBlockParts0.Header(), 0, propBlockID1)
+	if err := vs3.SignProposal(config.ChainID, newProp); err != nil {
+		t.Fatal(err)
 	}
 	cs1.SetProposalAndBlock(newProp, propBlock0, propBlockParts0, "some peer")
-	addVoteToFromMany(cs1, prevotes, cs2, cs3, cs4) // add the pol votes
+
+	// Add the pol votes
+	addVotes(cs1, prevotes...)
 
 	<-newRoundCh
-	log.Notice("### ONTO Round 2")
+	t.Log("### ONTO Round 2")
 	/*Round2
 	// now we see the polka from round 1, but we shouldnt unlock
 	*/
@@ -885,13 +916,13 @@ func TestLockPOLSafety2(t *testing.T) {
 /*
 func TestSlashingPrevotes(t *testing.T) {
 	cs1, vss := randConsensusState(2)
-	cs2 := vss[1]
+	vs2 := vss[1]
 
 
 	proposalCh := subscribeToEvent(cs1.evsw,"tester",types.EventStringCompleteProposal() , 1)
 	timeoutWaitCh := subscribeToEvent(cs1.evsw,"tester",types.EventStringTimeoutWait() , 1)
 	newRoundCh := subscribeToEvent(cs1.evsw,"tester",types.EventStringNewRound() , 1)
-	voteCh := subscribeToVoter(cs1, cs1.privValidator.Address)
+	voteCh := subscribeToVoter(cs1, cs1.privValidator.GetAddress())
 
 	// start round and wait for propose and prevote
 	startTestRound(cs1, cs1.Height, 0)
@@ -899,13 +930,13 @@ func TestSlashingPrevotes(t *testing.T) {
 	re := <-proposalCh
 	<-voteCh // prevote
 
-	rs := re.(types.EventDataRoundState).RoundState.(*RoundState)
+	rs := re.(types.EventDataRoundState).RoundState.(*cstypes.RoundState)
 
 	// we should now be stuck in limbo forever, waiting for more prevotes
 	// add one for a different block should cause us to go into prevote wait
 	hash := rs.ProposalBlock.Hash()
 	hash[0] = byte(hash[0]+1) % 255
-	signAddVoteToFrom(types.VoteTypePrevote, cs1, cs2, hash, rs.ProposalBlockParts.Header(), nil)
+	signAddVotes(cs1, types.VoteTypePrevote, hash, rs.ProposalBlockParts.Header(), vs2)
 
 	<-timeoutWaitCh
 
@@ -913,20 +944,20 @@ func TestSlashingPrevotes(t *testing.T) {
 	// away and ignore more prevotes (and thus fail to slash!)
 
 	// add the conflicting vote
-	signAddVoteToFrom(types.VoteTypePrevote, cs1, cs2, rs.ProposalBlock.Hash(), rs.ProposalBlockParts.Header(),nil)
+	signAddVotes(cs1, types.VoteTypePrevote, rs.ProposalBlock.Hash(), rs.ProposalBlockParts.Header(), vs2)
 
 	// XXX: Check for existence of Dupeout info
 }
 
 func TestSlashingPrecommits(t *testing.T) {
 	cs1, vss := randConsensusState(2)
-	cs2 := vss[1]
+	vs2 := vss[1]
 
 
 	proposalCh := subscribeToEvent(cs1.evsw,"tester",types.EventStringCompleteProposal() , 1)
 	timeoutWaitCh := subscribeToEvent(cs1.evsw,"tester",types.EventStringTimeoutWait() , 1)
 	newRoundCh := subscribeToEvent(cs1.evsw,"tester",types.EventStringNewRound() , 1)
-	voteCh := subscribeToVoter(cs1, cs1.privValidator.Address)
+	voteCh := subscribeToVoter(cs1, cs1.privValidator.GetAddress())
 
 	// start round and wait for propose and prevote
 	startTestRound(cs1, cs1.Height, 0)
@@ -934,8 +965,8 @@ func TestSlashingPrecommits(t *testing.T) {
 	re := <-proposalCh
 	<-voteCh // prevote
 
-	// add prevote from cs2
-	signAddVoteToFrom(types.VoteTypePrevote, cs1, cs2, rs.ProposalBlock.Hash(), rs.ProposalBlockParts.Header(), nil)
+	// add prevote from vs2
+	signAddVotes(cs1, types.VoteTypePrevote, rs.ProposalBlock.Hash(), rs.ProposalBlockParts.Header(), vs2)
 
 	<-voteCh // precommit
 
@@ -943,13 +974,13 @@ func TestSlashingPrecommits(t *testing.T) {
 	// add one for a different block should cause us to go into prevote wait
 	hash := rs.ProposalBlock.Hash()
 	hash[0] = byte(hash[0]+1) % 255
-	signAddVoteToFrom(types.VoteTypePrecommit, cs1, cs2, hash, rs.ProposalBlockParts.Header(),nil)
+	signAddVotes(cs1, types.VoteTypePrecommit, hash, rs.ProposalBlockParts.Header(), vs2)
 
 	// NOTE: we have to send the vote for different block first so we don't just go into precommit round right
 	// away and ignore more prevotes (and thus fail to slash!)
 
-	// add precommit from cs2
-	signAddVoteToFrom(types.VoteTypePrecommit, cs1, cs2, rs.ProposalBlock.Hash(), rs.ProposalBlockParts.Header(),nil)
+	// add precommit from vs2
+	signAddVotes(cs1, types.VoteTypePrecommit, rs.ProposalBlock.Hash(), rs.ProposalBlockParts.Header(), vs2)
 
 	// XXX: Check for existence of Dupeout info
 }
@@ -965,44 +996,46 @@ func TestSlashingPrecommits(t *testing.T) {
 // we receive a final precommit after going into next round, but others might have gone to commit already!
 func TestHalt1(t *testing.T) {
 	cs1, vss := randConsensusState(4)
-	cs2, cs3, cs4 := vss[1], vss[2], vss[3]
+	vs2, vs3, vs4 := vss[1], vss[2], vss[3]
+
+	partSize := cs1.state.Params.BlockPartSizeBytes
 
 	proposalCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringCompleteProposal(), 1)
 	timeoutWaitCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringTimeoutWait(), 1)
 	newRoundCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringNewRound(), 1)
 	newBlockCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringNewBlock(), 1)
-	voteCh := subscribeToVoter(cs1, cs1.privValidator.Address)
+	voteCh := subscribeToVoter(cs1, cs1.privValidator.GetAddress())
 
 	// start round and wait for propose and prevote
 	startTestRound(cs1, cs1.Height, 0)
 	<-newRoundCh
 	re := <-proposalCh
-	rs := re.(types.EventDataRoundState).RoundState.(*RoundState)
+	rs := re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*cstypes.RoundState)
 	propBlock := rs.ProposalBlock
-	propBlockParts := propBlock.MakePartSet()
+	propBlockParts := propBlock.MakePartSet(partSize)
 
 	<-voteCh // prevote
 
-	signAddVoteToFromMany(types.VoteTypePrevote, cs1, propBlock.Hash(), propBlockParts.Header(), nil, cs3, cs4)
+	signAddVotes(cs1, types.VoteTypePrevote, propBlock.Hash(), propBlockParts.Header(), vs3, vs4)
 	<-voteCh // precommit
 
 	// the proposed block should now be locked and our precommit added
 	validatePrecommit(t, cs1, 0, 0, vss[0], propBlock.Hash(), propBlock.Hash())
 
 	// add precommits from the rest
-	signAddVoteToFrom(types.VoteTypePrecommit, cs1, cs2, nil, types.PartSetHeader{}, nil) // didnt receive proposal
-	signAddVoteToFrom(types.VoteTypePrecommit, cs1, cs3, propBlock.Hash(), propBlockParts.Header(), nil)
-	// we receive this later, but cs3 might receive it earlier and with ours will go to commit!
-	precommit4 := signVote(cs4, types.VoteTypePrecommit, propBlock.Hash(), propBlockParts.Header())
+	signAddVotes(cs1, types.VoteTypePrecommit, nil, types.PartSetHeader{}, vs2) // didnt receive proposal
+	signAddVotes(cs1, types.VoteTypePrecommit, propBlock.Hash(), propBlockParts.Header(), vs3)
+	// we receive this later, but vs3 might receive it earlier and with ours will go to commit!
+	precommit4 := signVote(vs4, types.VoteTypePrecommit, propBlock.Hash(), propBlockParts.Header())
 
-	incrementRound(cs2, cs3, cs4)
+	incrementRound(vs2, vs3, vs4)
 
 	// timeout to new round
 	<-timeoutWaitCh
 	re = <-newRoundCh
-	rs = re.(types.EventDataRoundState).RoundState.(*RoundState)
+	rs = re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*cstypes.RoundState)
 
-	log.Notice("### ONTO ROUND 1")
+	t.Log("### ONTO ROUND 1")
 	/*Round2
 	// we timeout and prevote our lock
 	// a polka happened but we didn't see it!
@@ -1013,12 +1046,12 @@ func TestHalt1(t *testing.T) {
 	validatePrevote(t, cs1, 0, vss[0], rs.LockedBlock.Hash())
 
 	// now we receive the precommit from the previous round
-	addVoteToFrom(cs1, cs4, precommit4)
+	addVotes(cs1, precommit4)
 
 	// receiving that precommit should take us straight to commit
 	<-newBlockCh
 	re = <-newRoundCh
-	rs = re.(types.EventDataRoundState).RoundState.(*RoundState)
+	rs = re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*cstypes.RoundState)
 
 	if rs.Height != 2 {
 		panic("expected height to increment")

consensus/test_data/README.md

@@ -0,0 +1,36 @@
+# Generating test data
+
+To generate the data, run `build.sh`. See that script for more details.
+
+Make sure to adjust the stepChanges in the testCases if the number of messages changes.
+This sometimes happens for the `small_block2.cswal`, where the number of block parts changes between 4 and 5.
+
+If you need to change the signatures, you can use a script as follows:
+The privBytes comes from `config/tendermint_test/...`:
+
+```
+package main
+
+import (
+	"encoding/hex"
+	"fmt"
+
+	"github.com/tendermint/go-crypto"
+)
+
+func main() {
+	signBytes, err := hex.DecodeString("7B22636861696E5F6964223A2274656E6465726D696E745F74657374222C22766F7465223A7B22626C6F636B5F68617368223A2242453544373939433846353044354645383533364334333932464443384537423342313830373638222C22626C6F636B5F70617274735F686561646572223A506172745365747B543A31204236323237323535464632307D2C22686569676874223A312C22726F756E64223A302C2274797065223A327D7D")
+	if err != nil {
+		panic(err)
+	}
+	privBytes, err := hex.DecodeString("27F82582AEFAE7AB151CFB01C48BB6C1A0DA78F9BDDA979A9F70A84D074EB07D3B3069C422E19688B45CBFAE7BB009FC0FA1B1EA86593519318B7214853803C8")
+	if err != nil {
+		panic(err)
+	}
+	privKey := crypto.PrivKeyEd25519{}
+	copy(privKey[:], privBytes)
+	signature := privKey.Sign(signBytes)
+	fmt.Printf("Signature Bytes: %X\n", signature.Bytes())
+}
+```
+

consensus/test_data/build.sh

@@ -0,0 +1,148 @@
+#!/usr/bin/env bash
+
+# Requires: killall command and jq JSON processor.
+
+# Get the parent directory of where this script is.
+SOURCE="${BASH_SOURCE[0]}"
+while [ -h "$SOURCE" ] ; do SOURCE="$(readlink "$SOURCE")"; done
+DIR="$( cd -P "$( dirname "$SOURCE" )/../.." && pwd )"
+
+# Change into that dir because we expect that.
+cd "$DIR" || exit 1
+
+# Make sure we have a tendermint command.
+if ! hash tendermint 2>/dev/null; then
+	make install
+fi
+
+# Make sure we have a cutWALUntil binary.
+cutWALUntil=./scripts/cutWALUntil/cutWALUntil
+cutWALUntilDir=$(dirname $cutWALUntil)
+if ! hash $cutWALUntil 2>/dev/null; then
+	cd "$cutWALUntilDir" && go build && cd - || exit 1
+fi
+
+TMHOME=$(mktemp -d)
+export TMHOME="$TMHOME"
+
+if [[ ! -d "$TMHOME" ]]; then
+	echo "Could not create temp directory"
+  exit 1
+else
+	echo "TMHOME: ${TMHOME}"
+fi
+
+# TODO: eventually we should replace with `tendermint init --test`
+DIR_TO_COPY=$HOME/.tendermint_test/consensus_state_test
+if [ ! -d "$DIR_TO_COPY" ]; then
+    echo "$DIR_TO_COPY does not exist. Please run: go test ./consensus"
+    exit 1
+fi
+echo "==> Copying ${DIR_TO_COPY} to ${TMHOME} directory..."
+cp -r "$DIR_TO_COPY"/* "$TMHOME"
+
+# preserve original genesis file because later it will be modified (see small_block2)
+cp "$TMHOME/genesis.json" "$TMHOME/genesis.json.bak"
+
+function reset(){
+	echo "==> Resetting tendermint..."
+	tendermint unsafe_reset_all
+	cp "$TMHOME/genesis.json.bak" "$TMHOME/genesis.json"
+}
+
+reset
+
+function empty_block(){
+	echo "==> Starting tendermint..."
+	tendermint node --proxy_app=persistent_dummy &> /dev/null &
+	sleep 5
+	echo "==> Killing tendermint..."
+	killall tendermint
+
+	echo "==> Copying WAL log..."
+	$cutWALUntil "$TMHOME/data/cs.wal/wal" 1 consensus/test_data/new_empty_block.cswal
+	mv consensus/test_data/new_empty_block.cswal consensus/test_data/empty_block.cswal
+
+	reset
+}
+
+function many_blocks(){
+	bash scripts/txs/random.sh 1000 36657 &> /dev/null &
+	PID=$!
+	echo "==> Starting tendermint..."
+	tendermint node --proxy_app=persistent_dummy &> /dev/null &
+	sleep 10
+	echo "==> Killing tendermint..."
+	kill -9 $PID
+	killall tendermint
+
+	echo "==> Copying WAL log..."
+	$cutWALUntil "$TMHOME/data/cs.wal/wal" 6 consensus/test_data/new_many_blocks.cswal
+	mv consensus/test_data/new_many_blocks.cswal consensus/test_data/many_blocks.cswal
+
+	reset
+}
+
+
+function small_block1(){
+	bash scripts/txs/random.sh 1000 36657 &> /dev/null &
+	PID=$!
+	echo "==> Starting tendermint..."
+	tendermint node --proxy_app=persistent_dummy &> /dev/null &
+	sleep 10
+	echo "==> Killing tendermint..."
+	kill -9 $PID
+	killall tendermint
+
+	echo "==> Copying WAL log..."
+	$cutWALUntil "$TMHOME/data/cs.wal/wal" 1 consensus/test_data/new_small_block1.cswal
+	mv consensus/test_data/new_small_block1.cswal consensus/test_data/small_block1.cswal
+
+	reset
+}
+
+
+# block part size = 512
+function small_block2(){
+	cat "$TMHOME/genesis.json" | jq '. + {consensus_params: {block_size_params: {max_bytes: 22020096}, block_gossip_params: {block_part_size_bytes: 512}}}' > "$TMHOME/new_genesis.json"
+	mv "$TMHOME/new_genesis.json" "$TMHOME/genesis.json"
+	bash scripts/txs/random.sh 1000 36657 &> /dev/null &
+	PID=$!
+	echo "==> Starting tendermint..."
+	tendermint node --proxy_app=persistent_dummy &> /dev/null &
+	sleep 5
+	echo "==> Killing tendermint..."
+	kill -9 $PID
+	killall tendermint
+
+	echo "==> Copying WAL log..."
+	$cutWALUntil "$TMHOME/data/cs.wal/wal" 1 consensus/test_data/new_small_block2.cswal
+	mv consensus/test_data/new_small_block2.cswal consensus/test_data/small_block2.cswal
+
+	reset
+}
+
+
+
+case "$1" in
+	"small_block1")
+		small_block1
+		;;
+	"small_block2")
+		small_block2
+		;;
+	"empty_block")
+		empty_block
+		;;
+	"many_blocks")
+		many_blocks
+		;;
+	*)
+		small_block1
+		small_block2
+		empty_block
+		many_blocks
+esac
+
+echo "==> Cleaning up..."
+rm -rf "$TMHOME"