Merge pull request #903 from tendermint/hotfix-v0.12.1

Upgrade tmlibs dependency to build on Windows

.editorconfig

@@ -13,3 +13,7 @@ indent_style = tab
 
 [*.sh]
 indent_style = tab
+
+[*.proto]
+indent_style = space
+indent_size = 2

.github/CODEOWNERS

@@ -0,0 +1,5 @@
+# CODEOWNERS: https://help.github.com/articles/about-codeowners/
+
+# Everything goes through Bucky. For now.
+*       @ebuchman
+

.gitignore

@@ -15,3 +15,8 @@ test/p2p/data/
 test/logs
 .glide
 coverage.txt
+docs/_build
+docs/tools
+
+scripts/wal2json/wal2json
+scripts/cutWALUntil/cutWALUntil

CHANGELOG.md

@@ -1,5 +1,160 @@
 # Changelog
 
+## Roadmap
+
+BREAKING CHANGES:
+- Upgrade the header to support better proofs on validtors, results, evidence, and possibly more
+- Better support for injecting randomness
+- Pass evidence/voteInfo through ABCI
+- Upgrade consensus for more real-time use of evidence
+
+FEATURES:
+- Peer reputation management
+- Use the chain as its own CA for nodes and validators
+- Tooling to run multiple blockchains/apps, possibly in a single process
+- State syncing (without transaction replay)
+- Add authentication and rate-limitting to the RPC
+
+IMPROVEMENTS:
+- Improve subtleties around mempool caching and logic
+- Consensus optimizations:
+	- cache block parts for faster agreement after round changes
+	- propagate block parts rarest first
+- Better testing of the consensus state machine (ie. use a DSL)
+- Auto compiled serialization/deserialization code instead of go-wire reflection
+
+BUG FIXES:
+- Graceful handling/recovery for apps that have non-determinism or fail to halt
+- Graceful handling/recovery for violations of safety, or liveness
+
+## 0.12.1 (November 27, 2017)
+
+BUG FIXES:
+- upgrade tmlibs dependency to enable Windows builds for Tendermint
+
+## 0.12.0 (October 27, 2017)
+
+BREAKING CHANGES:
+ - rpc/client: websocket ResultsCh and ErrorsCh unified in ResponsesCh.
+ - rpc/client: ABCIQuery no longer takes `prove`
+ - state: remove GenesisDoc from state.
+ - consensus: new binary WAL format provides efficiency and uses checksums to detect corruption
+    - use scripts/wal2json to convert to json for debugging
+
+FEATURES:
+ - new `certifiers` pkg contains the tendermint light-client library (name subject to change)!
+ - rpc: `/genesis` includes the `app_options` .
+ - rpc: `/abci_query` takes an additional `height` parameter to support historical queries.
+ - rpc/client: new ABCIQueryWithOptions supports options like `trusted` (set false to get a proof) and `height` to query a historical height.
+
+IMPROVEMENTS:
+ - rpc: `/genesis` result includes `app_options`
+ - rpc/lib/client: add jitter to reconnects.
+ - rpc/lib/types: `RPCError` satisfies the `error` interface.
+
+BUG FIXES:
+ - rpc/client: fix ws deadlock after stopping
+ - blockchain: fix panic on AddBlock when peer is nil
+ - mempool: fix sending on TxsAvailable when a tx has been invalidated
+ - consensus: dont run WAL catchup if we fast synced
+
+## 0.11.1 (October 10, 2017)
+
+IMPROVEMENTS:
+ - blockchain/reactor: respondWithNoResponseMessage for missing height
+
+BUG FIXES:
+ - rpc: fixed client WebSocket timeout
+ - rpc: client now resubscribes on reconnection
+ - rpc: fix panics on missing params
+ - rpc: fix `/dump_consensus_state` to have normal json output (NOTE: technically breaking, but worth a bug fix label)
+ - types: fixed out of range error in VoteSet.addVote
+ - consensus: fix wal autofile via https://github.com/tendermint/tmlibs/blob/master/CHANGELOG.md#032-october-2-2017
+
+## 0.11.0 (September 22, 2017)
+
+BREAKING:
+ - genesis file: validator `amount` is now `power`
+ - abci: Info, BeginBlock, InitChain all take structs
+ - rpc: various changes to match JSONRPC spec (http://www.jsonrpc.org/specification), including breaking ones:
+    - requests that previously returned HTTP code 4XX now return 200 with an error code in the JSONRPC.
+    - `rpctypes.RPCResponse` uses new `RPCError` type instead of `string`.
+
+ - cmd: if there is no genesis, exit immediately instead of waiting around for one to show.
+ - types: `Signer.Sign` returns an error.
+ - state: every validator set change is persisted to disk, which required some changes to the `State` structure.
+ - p2p: new `p2p.Peer` interface used for all reactor methods (instead of `*p2p.Peer` struct).
+
+FEATURES:
+ - rpc: `/validators?height=X` allows querying of validators at previous heights.
+ - rpc: Leaving the `height` param empty for `/block`, `/validators`, and `/commit` will return the value for the latest height.
+
+IMPROVEMENTS:
+ - docs: Moved all docs from the website and tools repo in, converted to `.rst`, and cleaned up for presentation on `tendermint.readthedocs.io`
+
+BUG FIXES:
+ - fix WAL openning issue on Windows
+
+## 0.10.4 (September 5, 2017)
+
+IMPROVEMENTS:
+- docs: Added Slate docs to each rpc function (see rpc/core)
+- docs: Ported all website docs to Read The Docs
+- config: expose some p2p params to tweak performance: RecvRate, SendRate, and MaxMsgPacketPayloadSize
+- rpc: Upgrade the websocket client and server, including improved auto reconnect, and proper ping/pong
+
+BUG FIXES:
+- consensus: fix panic on getVoteBitArray
+- consensus: hang instead of panicking on byzantine consensus failures
+- cmd: dont load config for version command
+
+## 0.10.3 (August 10, 2017)
+
+FEATURES:
+- control over empty block production:
+  - new flag, `--consensus.create_empty_blocks`; when set to false, blocks are only created when there are txs or when the AppHash changes.
+  - new config option, `consensus.create_empty_blocks_interval`; an empty block is created after this many seconds.
+  - in normal operation, `create_empty_blocks = true` and `create_empty_blocks_interval = 0`, so blocks are being created all the time (as in all previous versions of tendermint). The number of empty blocks can be reduced by increasing `create_empty_blocks_interval` or by setting `create_empty_blocks = false`.
+  - new `TxsAvailable()` method added to Mempool that returns a channel which fires when txs are available.
+  - new heartbeat message added to consensus reactor to notify peers that a node is waiting for txs before entering propose step.
+- rpc: Add `syncing` field to response returned by `/status`. Is `true` while in fast-sync mode.
+
+IMPROVEMENTS:
+- various improvements to documentation and code comments
+
+BUG FIXES:
+- mempool: pass height into constructor so it doesn't always start at 0
+
+## 0.10.2 (July 10, 2017)
+
+FEATURES:
+- Enable lower latency block commits by adding consensus reactor sleep durations and p2p flush throttle timeout to the config
+
+IMPROVEMENTS:
+- More detailed logging in the consensus reactor and state machine
+- More in-code documentation for many exposed functions, especially in consensus/reactor.go and p2p/switch.go
+- Improved readability for some function definitions and code blocks with long lines
+
+## 0.10.1 (June 28, 2017)
+
+FEATURES:
+- Use `--trace` to get stack traces for logged errors
+- types: GenesisDoc.ValidatorHash returns the hash of the genesis validator set
+- types: GenesisDocFromFile parses a GenesiDoc from a JSON file
+
+IMPROVEMENTS:
+- Add a Code of Conduct
+- Variety of improvements as suggested by `megacheck` tool
+- rpc: deduplicate tests between rpc/client and rpc/tests
+- rpc: addresses without a protocol prefix default to `tcp://`. `http://` is also accepted as an alias for `tcp://`
+- cmd: commands are more easily reuseable from other tools
+- DOCKER: automate build/push
+
+BUG FIXES:
+- Fix log statements using keys with spaces (logger does not currently support spaces)
+- rpc: set logger on websocket connection
+- rpc: fix ws connection stability by setting write deadline on pings
+
 ## 0.10.0 (June 2, 2017)
 
 Includes major updates to configuration, logging, and json serialization.
@@ -8,7 +163,7 @@ Also includes the Grand Repo-Merge of 2017.
 BREAKING CHANGES:
 
 - Config and Flags:
-  - The `config` map is replaced with a [`Config` struct](https://github.com/tendermint/tendermint/blob/master/config/config.go#L11), 
+  - The `config` map is replaced with a [`Config` struct](https://github.com/tendermint/tendermint/blob/master/config/config.go#L11),
 containing substructs: `BaseConfig`, `P2PConfig`, `MempoolConfig`, `ConsensusConfig`, `RPCConfig`
   - This affects the following flags:
     - `--seeds` is now `--p2p.seeds`
@@ -21,16 +176,16 @@ containing substructs: `BaseConfig`, `P2PConfig`, `MempoolConfig`, `ConsensusCon
     ```
     [p2p]
     laddr="tcp://1.2.3.4:46656"
-    
+
     [consensus]
     timeout_propose=1000
     ```
   - Use viper and `DefaultConfig() / TestConfig()` functions to handle defaults, and remove `config/tendermint` and `config/tendermint_test`
-  - Change some function and method signatures to 
+  - Change some function and method signatures to
   - Change some [function and method signatures](https://gist.github.com/ebuchman/640d5fc6c2605f73497992fe107ebe0b) accomodate new config
 
 - Logger
-  - Replace static `log15` logger with a simple interface, and provide a new implementation using `go-kit`. 
+  - Replace static `log15` logger with a simple interface, and provide a new implementation using `go-kit`.
 See our new [logging library](https://github.com/tendermint/tmlibs/log) and [blog post](https://tendermint.com/blog/abstracting-the-logger-interface-in-go) for more details
   - Levels `warn` and `notice` are removed (you may need to change them in your `config.toml`!)
   - Change some [function and method signatures](https://gist.github.com/ebuchman/640d5fc6c2605f73497992fe107ebe0b) to accept a logger
@@ -73,7 +228,7 @@ IMPROVEMENTS:
 - Limit `/blockchain_info` call to return a maximum of 20 blocks
 - Use `.Wrap()` and `.Unwrap()` instead of eg. `PubKeyS` for `go-crypto` types
 - RPC JSON responses use pretty printing (via `json.MarshalIndent`)
-- Color code different instances of the consensus for tests 
+- Color code different instances of the consensus for tests
 - Isolate viper to `cmd/tendermint/commands` and do not read config from file for tests
 
 
@@ -101,7 +256,7 @@ IMPROVEMENTS:
 - WAL uses #ENDHEIGHT instead of #HEIGHT (#HEIGHT will stop working in 0.10.0)
 - Peers included via `--seeds`, under `seeds` in the config, or in `/dial_seeds` are now persistent, and will be reconnected to if the connection breaks
 
-BUG FIXES: 
+BUG FIXES:
 
 - Fix bug in fast-sync where we stop syncing after a peer is removed, even if they're re-added later
 - Fix handshake replay to handle validator set changes and results of DeliverTx when we crash after app.Commit but before state.Save()
@@ -117,7 +272,7 @@ message RequestQuery{
 	bytes data = 1;
 	string path = 2;
 	uint64 height = 3;
-	bool prove = 4; 
+	bool prove = 4;
 }
 
 message ResponseQuery{
@@ -141,7 +296,7 @@ type BlockMeta struct {
 }
 ```
 
-- `ValidatorSet.Proposer` is exposed as a field and persisted with the `State`. Use `GetProposer()` to initialize or update after validator-set changes. 
+- `ValidatorSet.Proposer` is exposed as a field and persisted with the `State`. Use `GetProposer()` to initialize or update after validator-set changes.
 
 - `tendermint gen_validator` command output is now pure JSON
 
@@ -184,7 +339,7 @@ type BlockID struct {
 }
 ```
 
-- `Vote` data type now includes validator address and index: 
+- `Vote` data type now includes validator address and index:
 
 ```
 type Vote struct {
@@ -204,7 +359,7 @@ type Vote struct {
 
 FEATURES:
 
-- New message type on the ConsensusReactor, `Maj23Msg`, for peers to alert others they've seen a Maj23, 
+- New message type on the ConsensusReactor, `Maj23Msg`, for peers to alert others they've seen a Maj23,
 in order to track and handle conflicting votes intelligently to prevent Byzantine faults from causing halts:
 
 ```
@@ -227,7 +382,7 @@ IMPROVEMENTS:
 - Less verbose logging
 - Better test coverage (37% -> 49%)
 - Canonical SignBytes for signable types
-- Write-Ahead Log for Mempool and Consensus via tmlibs/autofile 
+- Write-Ahead Log for Mempool and Consensus via tmlibs/autofile
 - Better in-process testing for the consensus reactor and byzantine faults
 - Better crash/restart testing for individual nodes at preset failure points, and of networks at arbitrary points
 - Better abstraction over timeout mechanics
@@ -307,7 +462,7 @@ FEATURES:
 - TMSP and RPC support TCP and UNIX sockets
 - Addition config options including block size and consensus parameters
 - New WAL mode `cswal_light`; logs only the validator's own votes
-- New RPC endpoints: 
+- New RPC endpoints:
 	- for starting/stopping profilers, and for updating config
 	- `/broadcast_tx_commit`, returns when tx is included in a block, else an error
 	- `/unsafe_flush_mempool`, empties the mempool
@@ -328,14 +483,14 @@ BUG FIXES:
 
 Strict versioning only began with the release of v0.7.0, in late summer 2016.
 The project itself began in early summer 2014 and was workable decentralized cryptocurrency software by the end of that year.
-Through the course of 2015, in collaboration with Eris Industries (now Monax Indsutries), 
+Through the course of 2015, in collaboration with Eris Industries (now Monax Indsutries),
 many additional features were integrated, including an implementation from scratch of the Ethereum Virtual Machine.
-That implementation now forms the heart of [ErisDB](https://github.com/eris-ltd/eris-db).
+That implementation now forms the heart of [Burrow](https://github.com/hyperledger/burrow).
 In the later half of 2015, the consensus algorithm was upgraded with a more asynchronous design and a more deterministic and robust implementation.
 
-By late 2015, frustration with the difficulty of forking a large monolithic stack to create alternative cryptocurrency designs led to the 
+By late 2015, frustration with the difficulty of forking a large monolithic stack to create alternative cryptocurrency designs led to the
 invention of the Application Blockchain Interface (ABCI), then called the Tendermint Socket Protocol (TMSP).
 The Ethereum Virtual Machine and various other transaction features were removed, and Tendermint was whittled down to a core consensus engine
-driving an application running in another process. 
+driving an application running in another process.
 The ABCI interface and implementation were iterated on and improved over the course of 2016,
 until versioned history kicked in with v0.7.0.

CODE_OF_CONDUCT.md

@@ -0,0 +1,56 @@
+# The Tendermint Code of Conduct
+This code of conduct applies to all projects run by the Tendermint/COSMOS team and hence to tendermint.
+
+
+----
+
+
+# Conduct
+## Contact: adrian@tendermint.com
+
+* We are committed to providing a friendly, safe and welcoming environment for all, regardless of level of experience, gender, gender identity and expression, sexual orientation, disability, personal appearance, body size, race, ethnicity, age, religion, nationality, or other similar characteristic.
+
+* On Slack, please avoid using overtly sexual nicknames or other nicknames that might detract from a friendly, safe and welcoming environment for all.
+
+* Please be kind and courteous. There’s no need to be mean or rude.
+
+* Respect that people have differences of opinion and that every design or implementation choice carries a trade-off and numerous costs. There is seldom a right answer.
+
+* Please keep unstructured critique to a minimum. If you have solid ideas you want to experiment with, make a fork and see how it works.
+
+* We will exclude you from interaction if you insult, demean or harass anyone. That is not welcome behaviour. We interpret the term “harassment” as including the definition in the [Citizen Code of Conduct](http://citizencodeofconduct.org/); if you have any lack of clarity about what might be included in that concept, please read their definition. In particular, we don’t tolerate behavior that excludes people in socially marginalized groups.
+
+* Private harassment is also unacceptable. No matter who you are, if you feel you have been or are being harassed or made uncomfortable by a community member, please contact one of the channel admins or the person mentioned above immediately. Whether you’re a regular contributor or a newcomer, we care about making this community a safe place for you and we’ve got your back.
+
+* Likewise any spamming, trolling, flaming, baiting or other attention-stealing behaviour is not welcome.
+
+
+----
+
+
+# Moderation
+These are the policies for upholding our community’s standards of conduct. If you feel that a thread needs moderation, please contact the above mentioned person.
+
+1. Remarks that violate the Tendermint/COSMOS standards of conduct, including hateful, hurtful, oppressive, or exclusionary remarks, are not allowed. (Cursing is allowed, but never targeting another user, and never in a hateful manner.)
+
+2. Remarks that moderators find inappropriate, whether listed in the code of conduct or not, are also not allowed.
+
+3. Moderators will first respond to such remarks with a warning.
+
+4. If the warning is unheeded, the user will be “kicked,” i.e., kicked out of the communication channel to cool off.
+
+5. If the user comes back and continues to make trouble, they will be banned, i.e., indefinitely excluded.
+
+6. Moderators may choose at their discretion to un-ban the user if it was a first offense and they offer the offended party a genuine apology.
+
+7. If a moderator bans someone and you think it was unjustified, please take it up with that moderator, or with a different moderator, in private. Complaints about bans in-channel are not allowed.
+
+8. Moderators are held to a higher standard than other community members. If a moderator creates an inappropriate situation, they should expect less leeway than others.
+
+In the Tendermint/COSMOS community we strive to go the extra step to look out for each other. Don’t just aim to be technically unimpeachable, try to be your best self. In particular, avoid flirting with offensive or sensitive issues, particularly if they’re off-topic; this all too often leads to unnecessary fights, hurt feelings, and damaged trust; worse, it can drive people away from the community entirely.
+
+And if someone takes issue with something you said or did, resist the urge to be defensive. Just stop doing what it was they complained about and apologize. Even if you feel you were misinterpreted or unfairly accused, chances are good there was something you could’ve communicated better — remember that it’s your responsibility to make your fellow Cosmonauts comfortable. Everyone wants to get along and we are all here first and foremost because we want to talk about cool technology. You will find that people will be eager to assume good intent and forgive as long as you earn their trust.
+
+The enforcement policies listed above apply to all official Tendermint/COSMOS venues.For other projects adopting the Tendermint/COSMOS Code of Conduct, please contact the maintainers of those projects for enforcement. If you wish to use this code of conduct for your own project, consider explicitly mentioning your moderation policy or making a copy with your own moderation policy so as to avoid confusion.
+
+*Adapted from the [Node.js Policy on Trolling](http://blog.izs.me/post/30036893703/policy-on-trolling), the [Contributor Covenant v1.3.0](http://contributor-covenant.org/version/1/3/0/) and the [Rust Code of Conduct](https://www.rust-lang.org/en-US/conduct.html).

CONTRIBUTING.md

@@ -1,16 +1,89 @@
-# Contributing guidelines
+# Contributing
 
-**Thanks for considering making contributions to Tendermint!**
+Thank you for considering making contributions to Tendermint and related repositories! Start by taking a look at the [coding repo](https://github.com/tendermint/coding) for overall information on repository workflow and standards.
 
-Please follow standard github best practices: fork the repo, **branch from the
-tip of develop**, make some commits, test your code changes with `make test`,
-and submit a pull request to develop.
+Please follow standard github best practices: fork the repo, branch from the tip of develop, make some commits, and submit a pull request to develop. See the [open issues](https://github.com/tendermint/tendermint/issues) for things we need help with!
 
-See the [open issues](https://github.com/tendermint/tendermint/issues) for
-things we need help with!
+Please make sure to use `gofmt` before every commit - the easiest way to do this is have your editor run it for you upon saving a file.
 
-Please make sure to use `gofmt` before every commit - the easiest way to do
-this is have your editor run it for you upon saving a file.
+## Forking
 
-You can read the full guide [on our
-site](https://tendermint.com/docs/guides/contributing).
+Please note that Go requires code to live under absolute paths, which complicates forking. 
+While my fork lives at `https://github.com/ebuchman/tendermint`, 
+the code should never exist at  `$GOPATH/src/github.com/ebuchman/tendermint`. 
+Instead, we use `git remote` to add the fork as a new remote for the original repo,
+`$GOPATH/src/github.com/tendermint/tendermint `, and do all the work there.
+
+For instance, to create a fork and work on a branch of it, I would:
+
+  * Create the fork on github, using the fork button.
+  * Go to the original repo checked out locally (ie. `$GOPATH/src/github.com/tendermint/tendermint`)
+  * `git remote rename origin upstream`
+  * `git remote add origin git@github.com:ebuchman/basecoin.git`
+
+Now `origin` refers to my fork and `upstream` refers to the tendermint version.
+So I can `git push -u origin master` to update my fork, and make pull requests to tendermint from there.
+Of course, replace `ebuchman` with your git handle.
+
+To pull in updates from the origin repo, run
+
+    * `git fetch upstream`
+    * `git rebase upstream/master` (or whatever branch you want)
+
+Please don't make Pull Requests to `master`.
+
+## Dependencies
+
+We use [glide](https://github.com/masterminds/glide) to manage dependencies.
+That said, the master branch of every Tendermint repository should just build with `go get`, which means they should be kept up-to-date with their dependencies so we can get away with telling people they can just `go get` our software.
+Since some dependencies are not under our control, a third party may break our build, in which case we can fall back on `glide install`. Even for dependencies under our control, glide helps us keeps multiple repos in sync as they evolve. Anything with an executable, such as apps, tools, and the core, should use glide.
+
+Run `bash scripts/glide/status.sh` to get a list of vendored dependencies that may not be up-to-date. 
+
+## Testing
+
+All repos should be hooked up to circle. 
+If they have `.go` files in the root directory, they will be automatically tested by circle using `go test -v -race ./...`. If not, they will need a `circle.yml`. Ideally, every repo has a `Makefile` that defines `make test` and includes its continuous integration status using a badge in the `README.md`.
+
+## Branching Model and Release
+
+User-facing repos should adhere to the branching model: http://nvie.com/posts/a-successful-git-branching-model/.
+That is, these repos should be well versioned, and any merge to master requires a version bump and tagged release.
+
+Libraries need not follow the model strictly, but would be wise to,
+especially `go-p2p` and `go-rpc`, as their versions are referenced in tendermint core.
+
+### Development Procedure:
+- the latest state of development is on `develop`
+- `develop` must never fail `make test`
+- no --force onto `develop` (except when reverting a broken commit, which should seldom happen)
+- create a development branch either on github.com/tendermint/tendermint, or your fork (using `git add origin`)
+- before submitting a pull request, begin `git rebase` on top of `develop`
+
+### Pull Merge Procedure:
+- ensure pull branch is rebased on develop
+- run `make test` to ensure that all tests pass
+- merge pull request
+- the `unstable` branch may be used to aggregate pull merges before testing once
+- push master may request that pull requests be rebased on top of `unstable`
+
+### Release Procedure:
+- start on `develop`
+- run integration tests (see `test_integrations` in Makefile)
+- prepare changelog/release issue
+- bump versions
+- push to release-vX.X.X to run the extended integration tests on the CI
+- merge to master
+- merge master back to develop
+
+### Hotfix Procedure:
+- start on `master`
+- checkout a new branch named hotfix-vX.X.X
+- make the required changes
+  - these changes should be small and an absolute necessity
+  - add a note to CHANGELOG.md
+- bumb versions
+- push to hotfix-vX.X.X to run the extended integration tests on the CI
+- merge hotfix-vX.X.X to master
+- merge hotfix-vX.X.X to develop
+- delete the hotfix-vX.X.X branch

DOCKER/Dockerfile

@@ -1,8 +1,8 @@
-FROM alpine:3.5
+FROM alpine:3.6
 
 # This is the release of tendermint to pull in.
-ENV TM_VERSION 0.9.1
-ENV TM_SHA256SUM da34234755937140dcd953afcc965555fad7e05afd546711bc5bdc2df3d54226
+ENV TM_VERSION 0.11.0
+ENV TM_SHA256SUM 7e443bac4d42f12e7beaf9cee63b4a565dad8c58895291fdedde8057088b70c5
 
 # Tendermint will be looking for genesis file in /tendermint (unless you change
 # `genesis_file` in config.toml). You can put your config.toml and private
@@ -26,7 +26,7 @@ RUN mkdir -p $DATA_ROOT && \
 RUN apk add --no-cache bash curl jq
 
 RUN apk add --no-cache openssl && \
-    wget https://s3-us-west-2.amazonaws.com/tendermint/${TM_VERSION}/tendermint_${TM_VERSION}_linux_amd64.zip && \
+    wget https://s3-us-west-2.amazonaws.com/tendermint/binaries/tendermint/v${TM_VERSION}/tendermint_${TM_VERSION}_linux_amd64.zip && \
     echo "${TM_SHA256SUM}  tendermint_${TM_VERSION}_linux_amd64.zip" | sha256sum -c && \
     unzip -d /bin tendermint_${TM_VERSION}_linux_amd64.zip && \
     apk del openssl && \
@@ -42,5 +42,4 @@ EXPOSE 46657
 
 ENTRYPOINT ["tendermint"]
 
-# By default you'll get the dummy app
-CMD ["node", "--moniker=`hostname`", "--proxy_app=dummy"]
+CMD ["node", "--moniker=`hostname`"]

DOCKER/Dockerfile.develop

@@ -1,4 +1,4 @@
-FROM alpine:3.5
+FROM alpine:3.6
 
 ENV DATA_ROOT /tendermint
 ENV TMHOME $DATA_ROOT

DOCKER/Makefile

@@ -1,12 +1,8 @@
 build:
-	# TAG=0.8.0 TAG_NO_PATCH=0.8
-	docker build -t "tendermint/tendermint" -t "tendermint/tendermint:$(TAG)" -t "tendermint/tendermint:$(TAG_NO_PATCH)" .
+	@sh -c "'$(CURDIR)/build.sh'"
 
 push:
-	# TAG=0.8.0 TAG_NO_PATCH=0.8
-	docker push "tendermint/tendermint:latest"
-	docker push "tendermint/tendermint:$(TAG)"
-	docker push "tendermint/tendermint:$(TAG_NO_PATCH)"
+	@sh -c "'$(CURDIR)/push.sh'"
 
 build_develop:
 	docker build -t "tendermint/tendermint:develop" -f Dockerfile.develop .

DOCKER/README.md

@@ -1,19 +1,32 @@
 # Supported tags and respective `Dockerfile` links
 
-- `0.9.1`, `0.9`, `latest` [(Dockerfile)](https://github.com/tendermint/tendermint/blob/809e0e8c5933604ba8b2d096803ada7c5ec4dfd3/DOCKER/Dockerfile)
+- `0.11.0`, `latest` [(Dockerfile)](https://github.com/tendermint/tendermint/blob/9177cc1f64ca88a4a0243c5d1773d10fba67e201/DOCKER/Dockerfile)
+- `0.10.0` [(Dockerfile)](https://github.com/tendermint/tendermint/blob/e5342f4054ab784b2cd6150e14f01053d7c8deb2/DOCKER/Dockerfile)
+- `0.9.1`, `0.9`, [(Dockerfile)](https://github.com/tendermint/tendermint/blob/809e0e8c5933604ba8b2d096803ada7c5ec4dfd3/DOCKER/Dockerfile)
 - `0.9.0` [(Dockerfile)](https://github.com/tendermint/tendermint/blob/d474baeeea6c22b289e7402449572f7c89ee21da/DOCKER/Dockerfile)
 - `0.8.0`, `0.8` [(Dockerfile)](https://github.com/tendermint/tendermint/blob/bf64dd21fdb193e54d8addaaaa2ecf7ac371de8c/DOCKER/Dockerfile)
 - `develop` [(Dockerfile)](https://github.com/tendermint/tendermint/blob/master/DOCKER/Dockerfile.develop)
 
 `develop` tag points to the [develop](https://github.com/tendermint/tendermint/tree/develop) branch.
 
+# Quick reference
+
+* **Where to get help:**
+  [Chat on Rocket](https://cosmos.rocket.chat/)
+
+* **Where to file issues:**
+  https://github.com/tendermint/tendermint/issues
+
+* **Supported Docker versions:**
+  [the latest release](https://github.com/moby/moby/releases) (down to 1.6 on a best-effort basis)
+
 # Tendermint
 
 Tendermint Core is Byzantine Fault Tolerant (BFT) middleware that takes a state transition machine, written in any programming language, and securely replicates it on many machines.
 
-For more background, see the [introduction](https://tendermint.com/intro).
+For more background, see the [introduction](https://tendermint.readthedocs.io/en/master/introduction.html).
 
-To get started developing applications, see the [application developers guide](https://tendermint.com/docs/guides/app-development).
+To get started developing applications, see the [application developers guide](https://tendermint.readthedocs.io/en/master/getting-started.html).
 
 # How to use this image
 
@@ -23,20 +36,12 @@ A very simple example of a built-in app and Tendermint core in one container.
 
 ```
 docker run -it --rm -v "/tmp:/tendermint" tendermint/tendermint init
-docker run -it --rm -v "/tmp:/tendermint" tendermint/tendermint
+docker run -it --rm -v "/tmp:/tendermint" tendermint/tendermint node --proxy_app=dummy
 ```
 
 ## mintnet-kubernetes
 
-If you want to see many containers talking to each other, consider using [mintnet-kubernetes](https://github.com/tendermint/mintnet-kubernetes), which is a tool for running Tendermint-based applications on a Kubernetes cluster.
-
-# Supported Docker versions
-
-This image is officially supported on Docker version 1.13.1.
-
-Support for older versions (down to 1.6) is provided on a best-effort basis.
-
-Please see [the Docker installation documentation](https://docs.docker.com/installation/) for details on how to upgrade your Docker daemon.
+If you want to see many containers talking to each other, consider using [mintnet-kubernetes](https://github.com/tendermint/tools/tree/master/mintnet-kubernetes), which is a tool for running Tendermint-based applications on a Kubernetes cluster.
 
 # License
 
@@ -44,12 +49,6 @@ View [license information](https://raw.githubusercontent.com/tendermint/tendermi
 
 # User Feedback
 
-## Issues
-
-If you have any problems with or questions about this image, please contact us through a [GitHub](https://github.com/tendermint/tendermint/issues) issue. If the issue is related to a CVE, please check for [a `cve-tracker` issue on the `official-images` repository](https://github.com/docker-library/official-images/issues?q=label%3Acve-tracker) first.
-
-You can also reach the image maintainers via [Slack](http://forum.tendermint.com:3000/).
-
 ## Contributing
 
 You are invited to contribute new features, fixes, or updates, large or small; we are always thrilled to receive pull requests, and do our best to process them as fast as we can.

DOCKER/build.sh

@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+set -e
+
+# Get the tag from the version, or try to figure it out.
+if [ -z "$TAG" ]; then
+	TAG=$(awk -F\" '/Version =/ { print $2; exit }' < ../version/version.go)
+fi
+if [ -z "$TAG" ]; then
+		echo "Please specify a tag."
+		exit 1
+fi
+
+TAG_NO_PATCH=${TAG%.*}
+
+read -p "==> Build 3 docker images with the following tags (latest, $TAG, $TAG_NO_PATCH)? y/n" -n 1 -r
+echo
+if [[ $REPLY =~ ^[Yy]$ ]]
+then
+		docker build -t "tendermint/tendermint" -t "tendermint/tendermint:$TAG" -t "tendermint/tendermint:$TAG_NO_PATCH" .
+fi

DOCKER/push.sh

@@ -0,0 +1,22 @@
+#!/usr/bin/env bash
+set -e
+
+# Get the tag from the version, or try to figure it out.
+if [ -z "$TAG" ]; then
+	TAG=$(awk -F\" '/Version =/ { print $2; exit }' < ../version/version.go)
+fi
+if [ -z "$TAG" ]; then
+		echo "Please specify a tag."
+		exit 1
+fi
+
+TAG_NO_PATCH=${TAG%.*}
+
+read -p "==> Push 3 docker images with the following tags (latest, $TAG, $TAG_NO_PATCH)? y/n" -n 1 -r
+echo
+if [[ $REPLY =~ ^[Yy]$ ]]
+then
+	docker push "tendermint/tendermint:latest"
+	docker push "tendermint/tendermint:$TAG"
+	docker push "tendermint/tendermint:$TAG_NO_PATCH"
+fi

INSTALL.md

@@ -1,57 +1 @@
-# Install Go
-
-[Install Go, set the `GOPATH`, and put `GOPATH/bin` on your `PATH`](https://github.com/tendermint/tendermint/wiki/Setting-GOPATH).
-
-# Install Tendermint
-
-You should be able to install the latest with a simple `go get -u github.com/tendermint/tendermint/cmd/tendermint`.
-The `-u` makes sure all dependencies are updated as well. 
-
-Run `tendermint version` and `tendermint --help`.
-
-If the install falied, see [vendored dependencies below](#vendored-dependencies).
-
-To start a one-node blockchain with a simple in-process application: 
-
-```
-tendermint init
-tendermint node --proxy_app=dummy
-```
-
-See the [application developers guide](https://github.com/tendermint/tendermint/wiki/Application-Developers) for more details on building and running applications.
-
-
-## Vendored dependencies
-
-If the `go get` failed, updated dependencies may have broken the build.
-Install the correct version of each dependency using `glide`.
-
-Fist, install `glide`:
-
-```
-go get github.com/Masterminds/glide
-```
-
-Now, fetch the dependencies and install them with `glide` and `go`:
-
-```
-cd $GOPATH/src/github.com/tendermint/tendermint
-glide install
-go install ./cmd/tendermint
-```
-
-Sometimes `glide install` is painfully slow. Hang in there champ.
-
-The latest Tendermint Core version is now installed. Check by running `tendermint version`.
-
-## Troubleshooting
-
-If `go get` failing bothers you, fetch the code using `git`:
-
-```
-mkdir -p $GOPATH/src/github.com/tendermint
-git clone https://github.com/tendermint/tendermint $GOPATH/src/github.com/tendermint/tendermint
-cd $GOPATH/src/github.com/tendermint/tendermint
-glide install
-go install ./cmd/tendermint
-```
+The installation guide has moved to the [docs directory](docs/guides/install-from-source.md) in order to easily be rendered by the website. Please update your links accordingly.

Makefile

@@ -1,6 +1,8 @@
 GOTOOLS = \
 					github.com/mitchellh/gox \
-					github.com/Masterminds/glide
+					github.com/tcnksm/ghr \
+					github.com/Masterminds/glide \
+
 PACKAGES=$(shell go list ./... | grep -v '/vendor/')
 BUILD_TAGS?=tendermint
 TMHOME = $${TMHOME:-$$HOME/.tendermint}
@@ -33,6 +35,9 @@ test_race:
 test_integrations:
 	@bash ./test/test.sh
 
+release:
+	@go test -tags release $(PACKAGES)
+
 test100:
 	@for i in {1..100}; do make test; done
 
@@ -72,5 +77,10 @@ tools:
 ensure_tools:
 	go get $(GOTOOLS)
 
+### Formatting, linting, and vetting
+
+megacheck:
+	@for pkg in ${PACKAGES}; do megacheck "$$pkg"; done
+
 
 .PHONY: install build build_race dist test test_race test_integrations test100 draw_deps list_deps get_deps get_vendor_deps update_deps revision tools

README.md

@@ -1,74 +1,105 @@
 # Tendermint
 
 [Byzantine-Fault Tolerant](https://en.wikipedia.org/wiki/Byzantine_fault_tolerance)
-[State Machine Replication](https://en.wikipedia.org/wiki/State_machine_replication). 
+[State Machine Replication](https://en.wikipedia.org/wiki/State_machine_replication).
 Or [Blockchain](https://en.wikipedia.org/wiki/Blockchain_(database)) for short.
 
 [![version](https://img.shields.io/github/tag/tendermint/tendermint.svg)](https://github.com/tendermint/tendermint/releases/latest)
 [![API Reference](
 https://camo.githubusercontent.com/915b7be44ada53c290eb157634330494ebe3e30a/68747470733a2f2f676f646f632e6f72672f6769746875622e636f6d2f676f6c616e672f6764646f3f7374617475732e737667
 )](https://godoc.org/github.com/tendermint/tendermint)
-[![chat](https://img.shields.io/badge/slack-join%20chat-pink.svg)](http://forum.tendermint.com:3000/)
+[![Rocket.Chat](https://demo.rocket.chat/images/join-chat.svg)](https://cosmos.rocket.chat/)
 [![license](https://img.shields.io/github/license/tendermint/tendermint.svg)](https://github.com/tendermint/tendermint/blob/master/LICENSE)
 [![](https://tokei.rs/b1/github/tendermint/tendermint?category=lines)](https://github.com/tendermint/tendermint)
 
 
-Branch    | Tests | Coverage | Report Card
-----------|-------|----------|-------------
-develop   | [![CircleCI](https://circleci.com/gh/tendermint/tendermint/tree/develop.svg?style=shield)](https://circleci.com/gh/tendermint/tendermint/tree/develop) | [![codecov](https://codecov.io/gh/tendermint/tendermint/branch/develop/graph/badge.svg)](https://codecov.io/gh/tendermint/tendermint) | [![Go Report Card](https://goreportcard.com/badge/github.com/tendermint/tendermint/tree/develop)](https://goreportcard.com/report/github.com/tendermint/tendermint/tree/develop)
-master    | [![CircleCI](https://circleci.com/gh/tendermint/tendermint/tree/master.svg?style=shield)](https://circleci.com/gh/tendermint/tendermint/tree/master) | [![codecov](https://codecov.io/gh/tendermint/tendermint/branch/master/graph/badge.svg)](https://codecov.io/gh/tendermint/tendermint) | [![Go Report Card](https://goreportcard.com/badge/github.com/tendermint/tendermint/tree/master)](https://goreportcard.com/report/github.com/tendermint/tendermint/tree/master)
+Branch    | Tests | Coverage
+----------|-------|----------
+master    | [![CircleCI](https://circleci.com/gh/tendermint/tendermint/tree/master.svg?style=shield)](https://circleci.com/gh/tendermint/tendermint/tree/master) | [![codecov](https://codecov.io/gh/tendermint/tendermint/branch/master/graph/badge.svg)](https://codecov.io/gh/tendermint/tendermint)
+develop   | [![CircleCI](https://circleci.com/gh/tendermint/tendermint/tree/develop.svg?style=shield)](https://circleci.com/gh/tendermint/tendermint/tree/develop) | [![codecov](https://codecov.io/gh/tendermint/tendermint/branch/develop/graph/badge.svg)](https://codecov.io/gh/tendermint/tendermint)
 
 _NOTE: This is alpha software. Please contact us if you intend to run it in production._
 
-Tendermint Core is Byzantine Fault Tolerant (BFT) middleware that takes a state transition machine, written in any programming language,
+Tendermint Core is Byzantine Fault Tolerant (BFT) middleware that takes a state transition machine - written in any programming language -
 and securely replicates it on many machines.
 
-For more background, see the [introduction](https://tendermint.com/intro).
-
-To get started developing applications, see the [application developers guide](https://tendermint.com/docs/guides/app-development).
+For more information, from introduction to install to application development, [Read The Docs](http://tendermint.readthedocs.io/projects/tools/en/master).
 
 ## Install
 
-To download pre-built binaries, see our [downloads page](https://tendermint.com/intro/getting-started/download).
+To download pre-built binaries, see our [downloads page](https://tendermint.com/downloads).
 
 To install from source, you should be able to:
 
 `go get -u github.com/tendermint/tendermint/cmd/tendermint`
 
-For more details (or if it fails), see the [install guide](https://tendermint.com/docs/guides/install).
-
-## Contributing
-
-Yay open source! Please see our [contributing guidelines](https://tendermint.com/docs/guides/contributing).
+For more details (or if it fails), [read the docs](http://tendermint.readthedocs.io/projects/tools/en/master/install.html).
 
 ## Resources
 
 ### Tendermint Core
 
-- [Introduction](https://tendermint.com/intro)
-- [Docs](https://tendermint.com/docs)
-- [Software using Tendermint](https://tendermint.com/ecosystem)
+All resources involving the use of, building application on, or developing for, tendermint, can be found at [Read The Docs](http://tendermint.readthedocs.io/projects/tools/en/master). Additional information about some - and eventually all - of the sub-projects below, can be found at Read The Docs.
 
 ### Sub-projects
 
 * [ABCI](http://github.com/tendermint/abci), the Application Blockchain Interface
 * [Go-Wire](http://github.com/tendermint/go-wire), a deterministic serialization library
-* [Go-Crypto](http://github.com/tendermint/go-crypto), an elliptic curve cryptography library 
-* [TmLibs](http://github.com/tendermint/tmlibs), an assortment of Go libraries
-* [Merkleeyes](http://github.com/tendermint/merkleeyes), a balanced, binary Merkle tree for ABCI apps
+* [Go-Crypto](http://github.com/tendermint/go-crypto), an elliptic curve cryptography library
+* [TmLibs](http://github.com/tendermint/tmlibs), an assortment of Go libraries used internally
+* [IAVL](http://github.com/tendermint/iavl), Merkleized IAVL+ Tree implementation
 
 ### Tools
-* [Deployment, Benchmarking, and Monitoring](https://github.com/tendermint/tools)
+* [Deployment, Benchmarking, and Monitoring](http://tendermint.readthedocs.io/projects/tools/en/develop/index.html#tendermint-tools)
 
 ### Applications
 
-* [Ethermint](http://github.com/tendermint/ethermint): Ethereum on Tendermint
-* [Basecoin](http://github.com/tendermint/basecoin), a cryptocurrency application framework
+* [Ethermint](http://github.com/tendermint/ethermint); Ethereum on Tendermint
+* [Cosmos SDK](http://github.com/cosmos/cosmos-sdk); a cryptocurrency application framework
+* [Many more](https://tendermint.readthedocs.io/en/master/ecosystem.html#abci-applications)
 
-### More 
+### More
 
-* [Tendermint Blog](https://tendermint.com/blog)
-* [Cosmos Blog](https://cosmos.network/blog)
-* [Original Whitepaper (out-of-date)](http://www.the-blockchain.com/docs/Tendermint%20Consensus%20without%20Mining.pdf)
 * [Master's Thesis on Tendermint](https://atrium.lib.uoguelph.ca/xmlui/handle/10214/9769)
+* [Original Whitepaper](https://tendermint.com/static/docs/tendermint.pdf)
+* [Tendermint Blog](https://blog.cosmos.network/tendermint/home)
+* [Cosmos Blog](https://blog.cosmos.network)
 
+## Contributing
+
+Yay open source! Please see our [contributing guidelines](CONTRIBUTING.md).
+
+## Versioning
+
+### SemVer
+
+Tendermint uses [SemVer](http://semver.org/) to determine when and how the version changes.
+According to SemVer, anything in the public API can change at any time before version 1.0.0
+
+To provide some stability to Tendermint users in these 0.X.X days, the MINOR version is used
+to signal breaking changes across a subset of the total public API. This subset includes all
+interfaces exposed to other processes (cli, rpc, p2p, etc.), as well as parts of the following packages:
+
+- types
+- rpc/client
+- config
+- node
+
+Exported objects in these packages that are not covered by the versioning scheme
+are explicitly marked by `// UNSTABLE` in their go doc comment and may change at any time.
+Functions, types, and values in any other package may also change at any time.
+
+### Upgrades
+
+In an effort to avoid accumulating technical debt prior to 1.0.0,
+we do not guarantee that breaking changes (ie. bumps in the MINOR version)
+will work with existing tendermint blockchains. In these cases you will
+have to start a new blockchain, or write something custom to get the old
+data into the new chain.
+
+However, any bump in the PATCH version should be compatible with existing histories
+(if not please open an [issue](https://github.com/tendermint/tendermint/issues)).
+
+## Code of Conduct
+
+Please read, understand and adhere to our [code of conduct](CODE_OF_CONDUCT.md).

Vagrantfile

@@ -17,11 +17,11 @@ Vagrant.configure("2") do |config|
     usermod -a -G docker vagrant
     apt-get autoremove -y
 
-    curl -O https://storage.googleapis.com/golang/go1.8.linux-amd64.tar.gz
-    tar -xvf go1.8.linux-amd64.tar.gz
+    curl -O https://storage.googleapis.com/golang/go1.9.linux-amd64.tar.gz
+    tar -xvf go1.9.linux-amd64.tar.gz
     rm -rf /usr/local/go
     mv go /usr/local
-    rm -f go1.8.linux-amd64.tar.gz
+    rm -f go1.9.linux-amd64.tar.gz
     mkdir -p /home/vagrant/go/bin
     echo 'export PATH=$PATH:/usr/local/go/bin:/home/vagrant/go/bin' >> /home/vagrant/.bash_profile
     echo 'export GOPATH=/home/vagrant/go' >> /home/vagrant/.bash_profile

benchmarks/blockchain/.gitignore

@@ -0,0 +1,2 @@
+data
+

benchmarks/blockchain/localsync.sh

@@ -0,0 +1,80 @@
+#!/bin/bash
+
+DATA=$GOPATH/src/github.com/tendermint/tendermint/benchmarks/blockchain/data
+if [ ! -d $DATA ]; then
+  echo "no data found, generating a chain... (this only has to happen once)"
+
+  tendermint init --home $DATA
+  cp $DATA/config.toml $DATA/config2.toml
+  echo "
+  [consensus]
+  timeout_commit = 0
+  " >> $DATA/config.toml
+
+  echo "starting node"
+  tendermint node \
+    --home $DATA \
+    --proxy_app dummy \
+    --p2p.laddr tcp://127.0.0.1:56656 \
+    --rpc.laddr tcp://127.0.0.1:56657 \
+    --log_level error &
+
+  echo "making blocks for 60s"
+  sleep 60
+
+  mv $DATA/config2.toml $DATA/config.toml
+
+  kill %1
+
+  echo "done generating chain."
+fi
+
+# validator node
+HOME1=$TMPDIR$RANDOM$RANDOM
+cp -R $DATA $HOME1
+echo "starting validator node"
+tendermint node \
+  --home $HOME1 \
+  --proxy_app dummy \
+  --p2p.laddr tcp://127.0.0.1:56656 \
+  --rpc.laddr tcp://127.0.0.1:56657 \
+  --log_level error &
+sleep 1
+
+# downloader node
+HOME2=$TMPDIR$RANDOM$RANDOM
+tendermint init --home $HOME2
+cp $HOME1/genesis.json $HOME2
+printf "starting downloader node"
+tendermint node \
+  --home $HOME2 \
+  --proxy_app dummy \
+  --p2p.laddr tcp://127.0.0.1:56666 \
+  --rpc.laddr tcp://127.0.0.1:56667 \
+  --p2p.seeds 127.0.0.1:56656 \
+  --log_level error &
+
+# wait for node to start up so we only count time where we are actually syncing
+sleep 0.5
+while curl localhost:56667/status 2> /dev/null | grep "\"latest_block_height\": 0," > /dev/null
+do
+  printf '.'
+  sleep 0.2
+done
+echo
+
+echo "syncing blockchain for 10s"
+for i in {1..10}
+do
+  sleep 1
+  HEIGHT="$(curl localhost:56667/status 2> /dev/null \
+    | grep 'latest_block_height' \
+    | grep -o ' [0-9]*' \
+    | xargs)"
+  let 'RATE = HEIGHT / i'
+  echo "height: $HEIGHT, blocks/sec: $RATE"
+done
+
+kill %1
+kill %2
+rm -rf $HOME1 $HOME2

benchmarks/map_test.go

@@ -1,8 +1,9 @@
 package benchmarks
 
 import (
-	. "github.com/tendermint/tmlibs/common"
 	"testing"
+
+	cmn "github.com/tendermint/tmlibs/common"
 )
 
 func BenchmarkSomething(b *testing.B) {
@@ -11,11 +12,11 @@ func BenchmarkSomething(b *testing.B) {
 	numChecks := 100000
 	keys := make([]string, numItems)
 	for i := 0; i < numItems; i++ {
-		keys[i] = RandStr(100)
+		keys[i] = cmn.RandStr(100)
 	}
 	txs := make([]string, numChecks)
 	for i := 0; i < numChecks; i++ {
-		txs[i] = RandStr(100)
+		txs[i] = cmn.RandStr(100)
 	}
 	b.StartTimer()
 

benchmarks/os_test.go

@@ -4,7 +4,7 @@ import (
 	"os"
 	"testing"
 
-	. "github.com/tendermint/tmlibs/common"
+	cmn "github.com/tendermint/tmlibs/common"
 )
 
 func BenchmarkFileWrite(b *testing.B) {
@@ -14,7 +14,7 @@ func BenchmarkFileWrite(b *testing.B) {
 	if err != nil {
 		b.Error(err)
 	}
-	testString := RandStr(200) + "\n"
+	testString := cmn.RandStr(200) + "\n"
 	b.StartTimer()
 
 	for i := 0; i < b.N; i++ {

benchmarks/simu/counter.go

@@ -1,30 +1,27 @@
 package main
 
 import (
+	"context"
 	"encoding/binary"
-	"time"
-	//"encoding/hex"
 	"fmt"
+	"time"
 
-	"github.com/gorilla/websocket"
-	"github.com/tendermint/go-wire"
-	_ "github.com/tendermint/tendermint/rpc/core/types" // Register RPCResponse > Result types
-	"github.com/tendermint/tendermint/rpc/lib/client"
-	"github.com/tendermint/tendermint/rpc/lib/types"
-	. "github.com/tendermint/tmlibs/common"
+	rpcclient "github.com/tendermint/tendermint/rpc/lib/client"
+	cmn "github.com/tendermint/tmlibs/common"
 )
 
 func main() {
-	ws := rpcclient.NewWSClient("127.0.0.1:46657", "/websocket")
-	_, err := ws.Start()
+	wsc := rpcclient.NewWSClient("127.0.0.1:46657", "/websocket")
+	_, err := wsc.Start()
 	if err != nil {
-		Exit(err.Error())
+		cmn.Exit(err.Error())
 	}
+	defer wsc.Stop()
 
 	// Read a bunch of responses
 	go func() {
 		for {
-			_, ok := <-ws.ResultsCh
+			_, ok := <-wsc.ResponsesCh
 			if !ok {
 				break
 			}
@@ -37,24 +34,14 @@ func main() {
 	for i := 0; ; i++ {
 		binary.BigEndian.PutUint64(buf, uint64(i))
 		//txBytes := hex.EncodeToString(buf[:n])
-		request, err := rpctypes.MapToRequest("fakeid",
-			"broadcast_tx",
-			map[string]interface{}{"tx": buf[:8]})
-		if err != nil {
-			Exit(err.Error())
-		}
-		reqBytes := wire.JSONBytes(request)
-		//fmt.Println("!!", string(reqBytes))
 		fmt.Print(".")
-		err = ws.WriteMessage(websocket.TextMessage, reqBytes)
+		err = wsc.Call(context.TODO(), "broadcast_tx", map[string]interface{}{"tx": buf[:8]})
 		if err != nil {
-			Exit(err.Error())
+			cmn.Exit(err.Error())
 		}
 		if i%1000 == 0 {
 			fmt.Println(i)
 		}
 		time.Sleep(time.Microsecond * 1000)
 	}
-
-	ws.Stop()
 }

blockchain/pool.go

@@ -6,16 +6,30 @@ import (
 	"time"
 
 	"github.com/tendermint/tendermint/types"
-	. "github.com/tendermint/tmlibs/common"
+	cmn "github.com/tendermint/tmlibs/common"
 	flow "github.com/tendermint/tmlibs/flowrate"
 	"github.com/tendermint/tmlibs/log"
 )
 
+/*
+
+eg, L = latency = 0.1s
+	P = num peers = 10
+	FN = num full nodes
+	BS = 1kB block size
+	CB = 1 Mbit/s = 128 kB/s
+	CB/P = 12.8 kB
+	B/S = CB/P/BS = 12.8 blocks/s
+
+	12.8 * 0.1 = 1.28 blocks on conn
+
+*/
+
 const (
-	requestIntervalMS         = 250
-	maxTotalRequesters        = 300
+	requestIntervalMS         = 100
+	maxTotalRequesters        = 1000
 	maxPendingRequests        = maxTotalRequesters
-	maxPendingRequestsPerPeer = 75
+	maxPendingRequestsPerPeer = 50
 	minRecvRate               = 10240 // 10Kb/s
 )
 
@@ -28,12 +42,12 @@ var peerTimeoutSeconds = time.Duration(15) // not const so we can override with
 	Every so often we ask peers what height they're on so we can keep going.
 
 	Requests are continuously made for blocks of higher heights until
-	the limits. If most of the requests have no available peers, and we
+	the limit is reached. If most of the requests have no available peers, and we
 	are not at peer limits, we can probably switch to consensus reactor
 */
 
 type BlockPool struct {
-	BaseService
+	cmn.BaseService
 	startTime time.Time
 
 	mtx sync.Mutex
@@ -42,7 +56,8 @@ type BlockPool struct {
 	height     int   // the lowest key in requesters.
 	numPending int32 // number of requests pending assignment or block response
 	// peers
-	peers map[string]*bpPeer
+	peers         map[string]*bpPeer
+	maxPeerHeight int
 
 	requestsCh chan<- BlockRequest
 	timeoutsCh chan<- string
@@ -59,7 +74,7 @@ func NewBlockPool(start int, requestsCh chan<- BlockRequest, timeoutsCh chan<- s
 		requestsCh: requestsCh,
 		timeoutsCh: timeoutsCh,
 	}
-	bp.BaseService = *NewBaseService(nil, "BlockPool", bp)
+	bp.BaseService = *cmn.NewBaseService(nil, "BlockPool", bp)
 	return bp
 }
 
@@ -69,16 +84,16 @@ func (pool *BlockPool) OnStart() error {
 	return nil
 }
 
-func (pool *BlockPool) OnStop() {
-	pool.BaseService.OnStop()
-}
+func (pool *BlockPool) OnStop() {}
 
 // Run spawns requesters as needed.
 func (pool *BlockPool) makeRequestersRoutine() {
+
 	for {
 		if !pool.IsRunning() {
 			break
 		}
+
 		_, numPending, lenRequesters := pool.GetStatus()
 		if numPending >= maxPendingRequests {
 			// sleep for a bit.
@@ -129,21 +144,16 @@ func (pool *BlockPool) IsCaughtUp() bool {
 	pool.mtx.Lock()
 	defer pool.mtx.Unlock()
 
-	height := pool.height
-
 	// Need at least 1 peer to be considered caught up.
 	if len(pool.peers) == 0 {
 		pool.Logger.Debug("Blockpool has no peers")
 		return false
 	}
 
-	maxPeerHeight := 0
-	for _, peer := range pool.peers {
-		maxPeerHeight = MaxInt(maxPeerHeight, peer.height)
-	}
-
-	isCaughtUp := (height > 0 || time.Now().Sub(pool.startTime) > 5*time.Second) && (maxPeerHeight == 0 || height >= maxPeerHeight)
-	pool.Logger.Info(Fmt("IsCaughtUp: %v", isCaughtUp), "height", height, "maxPeerHeight", maxPeerHeight)
+	// some conditions to determine if we're caught up
+	receivedBlockOrTimedOut := (pool.height > 0 || time.Since(pool.startTime) > 5*time.Second)
+	ourChainIsLongestAmongPeers := pool.maxPeerHeight == 0 || pool.height >= pool.maxPeerHeight
+	isCaughtUp := receivedBlockOrTimedOut && ourChainIsLongestAmongPeers
 	return isCaughtUp
 }
 
@@ -179,7 +189,7 @@ func (pool *BlockPool) PopRequest() {
 		delete(pool.requesters, pool.height)
 		pool.height++
 	} else {
-		PanicSanity(Fmt("Expected requester to pop, got nothing at height %v", pool.height))
+		cmn.PanicSanity(cmn.Fmt("Expected requester to pop, got nothing at height %v", pool.height))
 	}
 }
 
@@ -187,15 +197,16 @@ func (pool *BlockPool) PopRequest() {
 // Remove the peer and redo request from others.
 func (pool *BlockPool) RedoRequest(height int) {
 	pool.mtx.Lock()
+	defer pool.mtx.Unlock()
+
 	request := pool.requesters[height]
-	pool.mtx.Unlock()
 
 	if request.block == nil {
-		PanicSanity("Expected block to be non-nil")
+		cmn.PanicSanity("Expected block to be non-nil")
 	}
 	// RemovePeer will redo all requesters associated with this peer.
 	// TODO: record this malfeasance
-	pool.RemovePeer(request.peerID)
+	pool.removePeer(request.peerID)
 }
 
 // TODO: ensure that blocks come in order for each peer.
@@ -205,18 +216,29 @@ func (pool *BlockPool) AddBlock(peerID string, block *types.Block, blockSize int
 
 	requester := pool.requesters[block.Height]
 	if requester == nil {
+		// a block we didn't expect.
+		// TODO:if height is too far ahead, punish peer
 		return
 	}
 
 	if requester.setBlock(block, peerID) {
 		pool.numPending--
 		peer := pool.peers[peerID]
-		peer.decrPending(blockSize)
+		if peer != nil {
+			peer.decrPending(blockSize)
+		}
 	} else {
 		// Bad peer?
 	}
 }
 
+// MaxPeerHeight returns the heighest height reported by a peer
+func (pool *BlockPool) MaxPeerHeight() int {
+	pool.mtx.Lock()
+	defer pool.mtx.Unlock()
+	return pool.maxPeerHeight
+}
+
 // Sets the peer's alleged blockchain height.
 func (pool *BlockPool) SetPeerHeight(peerID string, height int) {
 	pool.mtx.Lock()
@@ -230,6 +252,10 @@ func (pool *BlockPool) SetPeerHeight(peerID string, height int) {
 		peer.setLogger(pool.Logger.With("peer", peerID))
 		pool.peers[peerID] = peer
 	}
+
+	if height > pool.maxPeerHeight {
+		pool.maxPeerHeight = height
+	}
 }
 
 func (pool *BlockPool) RemovePeer(peerID string) {
@@ -261,7 +287,6 @@ func (pool *BlockPool) pickIncrAvailablePeer(minHeight int) *bpPeer {
 		if peer.didTimeout {
 			pool.removePeer(peer.id)
 			continue
-		} else {
 		}
 		if peer.numPending >= maxPendingRequestsPerPeer {
 			continue
@@ -281,7 +306,7 @@ func (pool *BlockPool) makeNextRequester() {
 
 	nextHeight := pool.height + len(pool.requesters)
 	request := newBPRequester(pool, nextHeight)
-	request.SetLogger(pool.Logger.With("height", nextHeight))
+	// request.SetLogger(pool.Logger.With("height", nextHeight))
 
 	pool.requesters[nextHeight] = request
 	pool.numPending++
@@ -303,6 +328,7 @@ func (pool *BlockPool) sendTimeout(peerID string) {
 	pool.timeoutsCh <- peerID
 }
 
+// unused by tendermint; left for debugging purposes
 func (pool *BlockPool) debug() string {
 	pool.mtx.Lock() // Lock
 	defer pool.mtx.Unlock()
@@ -310,10 +336,10 @@ func (pool *BlockPool) debug() string {
 	str := ""
 	for h := pool.height; h < pool.height+len(pool.requesters); h++ {
 		if pool.requesters[h] == nil {
-			str += Fmt("H(%v):X ", h)
+			str += cmn.Fmt("H(%v):X ", h)
 		} else {
-			str += Fmt("H(%v):", h)
-			str += Fmt("B?(%v) ", pool.requesters[h].block != nil)
+			str += cmn.Fmt("H(%v):", h)
+			str += cmn.Fmt("B?(%v) ", pool.requesters[h].block != nil)
 		}
 	}
 	return str
@@ -326,7 +352,6 @@ type bpPeer struct {
 	id          string
 	recvMonitor *flow.Monitor
 
-	mtx        sync.Mutex
 	height     int
 	numPending int32
 	timeout    *time.Timer
@@ -352,7 +377,7 @@ func (peer *bpPeer) setLogger(l log.Logger) {
 
 func (peer *bpPeer) resetMonitor() {
 	peer.recvMonitor = flow.New(time.Second, time.Second*40)
-	var initialValue = float64(minRecvRate) * math.E
+	initialValue := float64(minRecvRate) * math.E
 	peer.recvMonitor.SetREMA(initialValue)
 }
 
@@ -394,7 +419,7 @@ func (peer *bpPeer) onTimeout() {
 //-------------------------------------
 
 type bpRequester struct {
-	BaseService
+	cmn.BaseService
 	pool       *BlockPool
 	height     int
 	gotBlockCh chan struct{}
@@ -415,7 +440,7 @@ func newBPRequester(pool *BlockPool, height int) *bpRequester {
 		peerID: "",
 		block:  nil,
 	}
-	bpr.BaseService = *NewBaseService(nil, "bpRequester", bpr)
+	bpr.BaseService = *cmn.NewBaseService(nil, "bpRequester", bpr)
 	return bpr
 }
 

blockchain/pool_test.go

@@ -6,7 +6,7 @@ import (
 	"time"
 
 	"github.com/tendermint/tendermint/types"
-	. "github.com/tendermint/tmlibs/common"
+	cmn "github.com/tendermint/tmlibs/common"
 	"github.com/tendermint/tmlibs/log"
 )
 
@@ -22,7 +22,7 @@ type testPeer struct {
 func makePeers(numPeers int, minHeight, maxHeight int) map[string]testPeer {
 	peers := make(map[string]testPeer, numPeers)
 	for i := 0; i < numPeers; i++ {
-		peerID := RandStr(12)
+		peerID := cmn.RandStr(12)
 		height := minHeight + rand.Intn(maxHeight-minHeight)
 		peers[peerID] = testPeer{peerID, height}
 	}

blockchain/reactor.go

@@ -12,15 +12,15 @@ import (
 	sm "github.com/tendermint/tendermint/state"
 	"github.com/tendermint/tendermint/types"
 	cmn "github.com/tendermint/tmlibs/common"
+	"github.com/tendermint/tmlibs/log"
 )
 
 const (
 	// BlockchainChannel is a channel for blocks and status updates (`BlockStore` height)
 	BlockchainChannel = byte(0x40)
 
-	defaultChannelCapacity = 100
-	defaultSleepIntervalMS = 500
-	trySyncIntervalMS      = 100
+	defaultChannelCapacity = 1000
+	trySyncIntervalMS      = 50
 	// stop syncing when last block's time is
 	// within this much of the system time.
 	// stopSyncingDurationMinutes = 10
@@ -29,13 +29,12 @@ const (
 	statusUpdateIntervalSeconds = 10
 	// check if we should switch to consensus reactor
 	switchToConsensusIntervalSeconds = 1
-	maxBlockchainResponseSize        = types.MaxBlockSize + 2
 )
 
 type consensusReactor interface {
 	// for when we switch from blockchain reactor and fast sync to
 	// the consensus machine
-	SwitchToConsensus(*sm.State)
+	SwitchToConsensus(*sm.State, int)
 }
 
 // BlockchainReactor handles long-term catchup syncing.
@@ -49,7 +48,6 @@ type BlockchainReactor struct {
 	fastSync     bool
 	requestsCh   chan BlockRequest
 	timeoutsCh   chan string
-	lastBlock    *types.Block
 
 	evsw types.EventSwitch
 }
@@ -82,7 +80,13 @@ func NewBlockchainReactor(state *sm.State, proxyAppConn proxy.AppConnConsensus,
 	return bcR
 }
 
-// OnStart implements BaseService
+// SetLogger implements cmn.Service by setting the logger on reactor and pool.
+func (bcR *BlockchainReactor) SetLogger(l log.Logger) {
+	bcR.BaseService.Logger = l
+	bcR.pool.Logger = l
+}
+
+// OnStart implements cmn.Service.
 func (bcR *BlockchainReactor) OnStart() error {
 	bcR.BaseReactor.OnStart()
 	if bcR.fastSync {
@@ -95,7 +99,7 @@ func (bcR *BlockchainReactor) OnStart() error {
 	return nil
 }
 
-// OnStop implements BaseService
+// OnStop implements cmn.Service.
 func (bcR *BlockchainReactor) OnStop() {
 	bcR.BaseReactor.OnStop()
 	bcR.pool.Stop()
@@ -106,50 +110,61 @@ func (bcR *BlockchainReactor) GetChannels() []*p2p.ChannelDescriptor {
 	return []*p2p.ChannelDescriptor{
 		&p2p.ChannelDescriptor{
 			ID:                BlockchainChannel,
-			Priority:          5,
-			SendQueueCapacity: 100,
+			Priority:          10,
+			SendQueueCapacity: 1000,
 		},
 	}
 }
 
 // AddPeer implements Reactor by sending our state to peer.
-func (bcR *BlockchainReactor) AddPeer(peer *p2p.Peer) {
+func (bcR *BlockchainReactor) AddPeer(peer p2p.Peer) {
 	if !peer.Send(BlockchainChannel, struct{ BlockchainMessage }{&bcStatusResponseMessage{bcR.store.Height()}}) {
 		// doing nothing, will try later in `poolRoutine`
 	}
 }
 
 // RemovePeer implements Reactor by removing peer from the pool.
-func (bcR *BlockchainReactor) RemovePeer(peer *p2p.Peer, reason interface{}) {
-	bcR.pool.RemovePeer(peer.Key)
+func (bcR *BlockchainReactor) RemovePeer(peer p2p.Peer, reason interface{}) {
+	bcR.pool.RemovePeer(peer.Key())
+}
+
+// respondToPeer loads a block and sends it to the requesting peer,
+// if we have it. Otherwise, we'll respond saying we don't have it.
+// According to the Tendermint spec, if all nodes are honest,
+// no node should be requesting for a block that's non-existent.
+func (bcR *BlockchainReactor) respondToPeer(msg *bcBlockRequestMessage, src p2p.Peer) (queued bool) {
+	block := bcR.store.LoadBlock(msg.Height)
+	if block != nil {
+		msg := &bcBlockResponseMessage{Block: block}
+		return src.TrySend(BlockchainChannel, struct{ BlockchainMessage }{msg})
+	}
+
+	bcR.Logger.Info("Peer asking for a block we don't have", "src", src, "height", msg.Height)
+
+	return src.TrySend(BlockchainChannel, struct{ BlockchainMessage }{
+		&bcNoBlockResponseMessage{Height: msg.Height},
+	})
 }
 
 // Receive implements Reactor by handling 4 types of messages (look below).
-func (bcR *BlockchainReactor) Receive(chID byte, src *p2p.Peer, msgBytes []byte) {
-	_, msg, err := DecodeMessage(msgBytes)
+func (bcR *BlockchainReactor) Receive(chID byte, src p2p.Peer, msgBytes []byte) {
+	_, msg, err := DecodeMessage(msgBytes, bcR.maxMsgSize())
 	if err != nil {
-		bcR.Logger.Error("Error decoding message", "error", err)
+		bcR.Logger.Error("Error decoding message", "err", err)
 		return
 	}
 
 	bcR.Logger.Debug("Receive", "src", src, "chID", chID, "msg", msg)
 
+	// TODO: improve logic to satisfy megacheck
 	switch msg := msg.(type) {
 	case *bcBlockRequestMessage:
-		// Got a request for a block. Respond with block if we have it.
-		block := bcR.store.LoadBlock(msg.Height)
-		if block != nil {
-			msg := &bcBlockResponseMessage{Block: block}
-			queued := src.TrySend(BlockchainChannel, struct{ BlockchainMessage }{msg})
-			if !queued {
-				// queue is full, just ignore.
-			}
-		} else {
-			// TODO peer is asking for things we don't have.
+		if queued := bcR.respondToPeer(msg, src); !queued {
+			// Unfortunately not queued since the queue is full.
 		}
 	case *bcBlockResponseMessage:
 		// Got a block.
-		bcR.pool.AddBlock(src.Key, msg.Block, len(msgBytes))
+		bcR.pool.AddBlock(src.Key(), msg.Block, len(msgBytes))
 	case *bcStatusRequestMessage:
 		// Send peer our state.
 		queued := src.TrySend(BlockchainChannel, struct{ BlockchainMessage }{&bcStatusResponseMessage{bcR.store.Height()}})
@@ -158,12 +173,18 @@ func (bcR *BlockchainReactor) Receive(chID byte, src *p2p.Peer, msgBytes []byte)
 		}
 	case *bcStatusResponseMessage:
 		// Got a peer status. Unverified.
-		bcR.pool.SetPeerHeight(src.Key, msg.Height)
+		bcR.pool.SetPeerHeight(src.Key(), msg.Height)
 	default:
 		bcR.Logger.Error(cmn.Fmt("Unknown message type %v", reflect.TypeOf(msg)))
 	}
 }
 
+// maxMsgSize returns the maximum allowable size of a
+// message on the blockchain reactor.
+func (bcR *BlockchainReactor) maxMsgSize() int {
+	return bcR.state.Params.BlockSizeParams.MaxBytes + 2
+}
+
 // Handle messages from the poolReactor telling the reactor what to do.
 // NOTE: Don't sleep in the FOR_LOOP or otherwise slow it down!
 // (Except for the SYNC_LOOP, which is the primary purpose and must be synchronous.)
@@ -173,6 +194,13 @@ func (bcR *BlockchainReactor) poolRoutine() {
 	statusUpdateTicker := time.NewTicker(statusUpdateIntervalSeconds * time.Second)
 	switchToConsensusTicker := time.NewTicker(switchToConsensusIntervalSeconds * time.Second)
 
+	blocksSynced := 0
+
+	chainID := bcR.state.ChainID
+
+	lastHundred := time.Now()
+	lastRate := 0.0
+
 FOR_LOOP:
 	for {
 		select {
@@ -194,24 +222,24 @@ FOR_LOOP:
 			if peer != nil {
 				bcR.Switch.StopPeerForError(peer, errors.New("BlockchainReactor Timeout"))
 			}
-		case _ = <-statusUpdateTicker.C:
+		case <-statusUpdateTicker.C:
 			// ask for status updates
 			go bcR.BroadcastStatusRequest()
-		case _ = <-switchToConsensusTicker.C:
-			height, numPending, _ := bcR.pool.GetStatus()
+		case <-switchToConsensusTicker.C:
+			height, numPending, lenRequesters := bcR.pool.GetStatus()
 			outbound, inbound, _ := bcR.Switch.NumPeers()
-			bcR.Logger.Info("Consensus ticker", "numPending", numPending, "total", len(bcR.pool.requesters),
+			bcR.Logger.Debug("Consensus ticker", "numPending", numPending, "total", lenRequesters,
 				"outbound", outbound, "inbound", inbound)
 			if bcR.pool.IsCaughtUp() {
 				bcR.Logger.Info("Time to switch to consensus reactor!", "height", height)
 				bcR.pool.Stop()
 
 				conR := bcR.Switch.Reactor("CONSENSUS").(consensusReactor)
-				conR.SwitchToConsensus(bcR.state)
+				conR.SwitchToConsensus(bcR.state, blocksSynced)
 
 				break FOR_LOOP
 			}
-		case _ = <-trySyncTicker.C: // chan time
+		case <-trySyncTicker.C: // chan time
 			// This loop can be slow as long as it's doing syncing work.
 		SYNC_LOOP:
 			for i := 0; i < 10; i++ {
@@ -222,16 +250,16 @@ FOR_LOOP:
 					// We need both to sync the first block.
 					break SYNC_LOOP
 				}
-				firstParts := first.MakePartSet(types.DefaultBlockPartSize)
+				firstParts := first.MakePartSet(bcR.state.Params.BlockPartSizeBytes)
 				firstPartsHeader := firstParts.Header()
 				// Finally, verify the first block using the second's commit
 				// NOTE: we can probably make this more efficient, but note that calling
 				// first.Hash() doesn't verify the tx contents, so MakePartSet() is
 				// currently necessary.
 				err := bcR.state.Validators.VerifyCommit(
-					bcR.state.ChainID, types.BlockID{first.Hash(), firstPartsHeader}, first.Height, second.LastCommit)
+					chainID, types.BlockID{first.Hash(), firstPartsHeader}, first.Height, second.LastCommit)
 				if err != nil {
-					bcR.Logger.Info("error in validation", "error", err)
+					bcR.Logger.Error("Error in validation", "err", err)
 					bcR.pool.RedoRequest(first.Height)
 					break SYNC_LOOP
 				} else {
@@ -248,6 +276,14 @@ FOR_LOOP:
 						// TODO This is bad, are we zombie?
 						cmn.PanicQ(cmn.Fmt("Failed to process committed block (%d:%X): %v", first.Height, first.Hash(), err))
 					}
+					blocksSynced += 1
+
+					if blocksSynced%100 == 0 {
+						lastRate = 0.9*lastRate + 0.1*(100/time.Since(lastHundred).Seconds())
+						bcR.Logger.Info("Fast Sync Rate", "height", bcR.pool.height,
+							"max_peer_height", bcR.pool.MaxPeerHeight(), "blocks/s", lastRate)
+						lastHundred = time.Now()
+					}
 				}
 			}
 			continue FOR_LOOP
@@ -272,10 +308,11 @@ func (bcR *BlockchainReactor) SetEventSwitch(evsw types.EventSwitch) {
 // Messages
 
 const (
-	msgTypeBlockRequest   = byte(0x10)
-	msgTypeBlockResponse  = byte(0x11)
-	msgTypeStatusResponse = byte(0x20)
-	msgTypeStatusRequest  = byte(0x21)
+	msgTypeBlockRequest    = byte(0x10)
+	msgTypeBlockResponse   = byte(0x11)
+	msgTypeNoBlockResponse = byte(0x12)
+	msgTypeStatusResponse  = byte(0x20)
+	msgTypeStatusRequest   = byte(0x21)
 )
 
 // BlockchainMessage is a generic message for this reactor.
@@ -285,17 +322,18 @@ var _ = wire.RegisterInterface(
 	struct{ BlockchainMessage }{},
 	wire.ConcreteType{&bcBlockRequestMessage{}, msgTypeBlockRequest},
 	wire.ConcreteType{&bcBlockResponseMessage{}, msgTypeBlockResponse},
+	wire.ConcreteType{&bcNoBlockResponseMessage{}, msgTypeNoBlockResponse},
 	wire.ConcreteType{&bcStatusResponseMessage{}, msgTypeStatusResponse},
 	wire.ConcreteType{&bcStatusRequestMessage{}, msgTypeStatusRequest},
 )
 
 // DecodeMessage decodes BlockchainMessage.
 // TODO: ensure that bz is completely read.
-func DecodeMessage(bz []byte) (msgType byte, msg BlockchainMessage, err error) {
+func DecodeMessage(bz []byte, maxSize int) (msgType byte, msg BlockchainMessage, err error) {
 	msgType = bz[0]
 	n := int(0)
 	r := bytes.NewReader(bz)
-	msg = wire.ReadBinary(struct{ BlockchainMessage }{}, r, maxBlockchainResponseSize, &n, &err).(struct{ BlockchainMessage }).BlockchainMessage
+	msg = wire.ReadBinary(struct{ BlockchainMessage }{}, r, maxSize, &n, &err).(struct{ BlockchainMessage }).BlockchainMessage
 	if err != nil && n != len(bz) {
 		err = errors.New("DecodeMessage() had bytes left over")
 	}
@@ -312,6 +350,14 @@ func (m *bcBlockRequestMessage) String() string {
 	return cmn.Fmt("[bcBlockRequestMessage %v]", m.Height)
 }
 
+type bcNoBlockResponseMessage struct {
+	Height int
+}
+
+func (brm *bcNoBlockResponseMessage) String() string {
+	return cmn.Fmt("[bcNoBlockResponseMessage %d]", brm.Height)
+}
+
 //-------------------------------------
 
 // NOTE: keep up-to-date with maxBlockchainResponseSize

blockchain/reactor_test.go

@@ -0,0 +1,151 @@
+package blockchain
+
+import (
+	"testing"
+
+	wire "github.com/tendermint/go-wire"
+	cmn "github.com/tendermint/tmlibs/common"
+	dbm "github.com/tendermint/tmlibs/db"
+	"github.com/tendermint/tmlibs/log"
+
+	cfg "github.com/tendermint/tendermint/config"
+	"github.com/tendermint/tendermint/p2p"
+	sm "github.com/tendermint/tendermint/state"
+	"github.com/tendermint/tendermint/types"
+)
+
+func newBlockchainReactor(maxBlockHeight int) *BlockchainReactor {
+	logger := log.TestingLogger()
+	config := cfg.ResetTestRoot("blockchain_reactor_test")
+
+	blockStore := NewBlockStore(dbm.NewMemDB())
+
+	// Get State
+	state, _ := sm.GetState(dbm.NewMemDB(), config.GenesisFile())
+	state.SetLogger(logger.With("module", "state"))
+	state.Save()
+
+	// Make the blockchainReactor itself
+	fastSync := true
+	bcReactor := NewBlockchainReactor(state.Copy(), nil, blockStore, fastSync)
+	bcReactor.SetLogger(logger.With("module", "blockchain"))
+
+	// Next: we need to set a switch in order for peers to be added in
+	bcReactor.Switch = p2p.NewSwitch(cfg.DefaultP2PConfig())
+
+	// Lastly: let's add some blocks in
+	for blockHeight := 1; blockHeight <= maxBlockHeight; blockHeight++ {
+		firstBlock := makeBlock(blockHeight, state)
+		secondBlock := makeBlock(blockHeight+1, state)
+		firstParts := firstBlock.MakePartSet(state.Params.BlockGossipParams.BlockPartSizeBytes)
+		blockStore.SaveBlock(firstBlock, firstParts, secondBlock.LastCommit)
+	}
+
+	return bcReactor
+}
+
+func TestNoBlockMessageResponse(t *testing.T) {
+	maxBlockHeight := 20
+
+	bcr := newBlockchainReactor(maxBlockHeight)
+	bcr.Start()
+	defer bcr.Stop()
+
+	// Add some peers in
+	peer := newbcrTestPeer(cmn.RandStr(12))
+	bcr.AddPeer(peer)
+
+	chID := byte(0x01)
+
+	tests := []struct {
+		height   int
+		existent bool
+	}{
+		{maxBlockHeight + 2, false},
+		{10, true},
+		{1, true},
+		{100, false},
+	}
+
+	for _, tt := range tests {
+		reqBlockMsg := &bcBlockRequestMessage{tt.height}
+		reqBlockBytes := wire.BinaryBytes(struct{ BlockchainMessage }{reqBlockMsg})
+		bcr.Receive(chID, peer, reqBlockBytes)
+		value := peer.lastValue()
+		msg := value.(struct{ BlockchainMessage }).BlockchainMessage
+
+		if tt.existent {
+			if blockMsg, ok := msg.(*bcBlockResponseMessage); !ok {
+				t.Fatalf("Expected to receive a block response for height %d", tt.height)
+			} else if blockMsg.Block.Height != tt.height {
+				t.Fatalf("Expected response to be for height %d, got %d", tt.height, blockMsg.Block.Height)
+			}
+		} else {
+			if noBlockMsg, ok := msg.(*bcNoBlockResponseMessage); !ok {
+				t.Fatalf("Expected to receive a no block response for height %d", tt.height)
+			} else if noBlockMsg.Height != tt.height {
+				t.Fatalf("Expected response to be for height %d, got %d", tt.height, noBlockMsg.Height)
+			}
+		}
+	}
+}
+
+//----------------------------------------------
+// utility funcs
+
+func makeTxs(blockNumber int) (txs []types.Tx) {
+	for i := 0; i < 10; i++ {
+		txs = append(txs, types.Tx([]byte{byte(blockNumber), byte(i)}))
+	}
+	return txs
+}
+
+func makeBlock(blockNumber int, state *sm.State) *types.Block {
+	prevHash := state.LastBlockID.Hash
+	prevParts := types.PartSetHeader{}
+	valHash := state.Validators.Hash()
+	prevBlockID := types.BlockID{prevHash, prevParts}
+	block, _ := types.MakeBlock(blockNumber, "test_chain", makeTxs(blockNumber),
+		new(types.Commit), prevBlockID, valHash, state.AppHash, state.Params.BlockGossipParams.BlockPartSizeBytes)
+	return block
+}
+
+// The Test peer
+type bcrTestPeer struct {
+	cmn.Service
+	key string
+	ch  chan interface{}
+}
+
+var _ p2p.Peer = (*bcrTestPeer)(nil)
+
+func newbcrTestPeer(key string) *bcrTestPeer {
+	return &bcrTestPeer{
+		Service: cmn.NewBaseService(nil, "bcrTestPeer", nil),
+		key:     key,
+		ch:      make(chan interface{}, 2),
+	}
+}
+
+func (tp *bcrTestPeer) lastValue() interface{} { return <-tp.ch }
+
+func (tp *bcrTestPeer) TrySend(chID byte, value interface{}) bool {
+	if _, ok := value.(struct{ BlockchainMessage }).BlockchainMessage.(*bcStatusResponseMessage); ok {
+		// Discard status response messages since they skew our results
+		// We only want to deal with:
+		// + bcBlockResponseMessage
+		// + bcNoBlockResponseMessage
+	} else {
+		tp.ch <- value
+	}
+	return true
+}
+
+func (tp *bcrTestPeer) Send(chID byte, data interface{}) bool { return tp.TrySend(chID, data) }
+func (tp *bcrTestPeer) NodeInfo() *p2p.NodeInfo               { return nil }
+func (tp *bcrTestPeer) Status() p2p.ConnectionStatus          { return p2p.ConnectionStatus{} }
+func (tp *bcrTestPeer) Key() string                           { return tp.key }
+func (tp *bcrTestPeer) IsOutbound() bool                      { return false }
+func (tp *bcrTestPeer) IsPersistent() bool                    { return true }
+func (tp *bcrTestPeer) Get(s string) interface{}              { return s }
+func (tp *bcrTestPeer) Set(string, interface{})               {}

blockchain/store.go

@@ -7,10 +7,10 @@ import (
 	"io"
 	"sync"
 
+	wire "github.com/tendermint/go-wire"
+	"github.com/tendermint/tendermint/types"
 	. "github.com/tendermint/tmlibs/common"
 	dbm "github.com/tendermint/tmlibs/db"
-	"github.com/tendermint/go-wire"
-	"github.com/tendermint/tendermint/types"
 )
 
 /*
@@ -25,7 +25,8 @@ Currently the precommit signatures are duplicated in the Block parts as
 well as the Commit.  In the future this may change, perhaps by moving
 the Commit data outside the Block.
 
-Panics indicate probable corruption in the data
+// NOTE: BlockStore methods will panic if they encounter errors
+// deserializing loaded data, indicating probable corruption on disk.
 */
 type BlockStore struct {
 	db dbm.DB

certifiers/client/main_test.go

@@ -0,0 +1,25 @@
+package client_test
+
+import (
+	"os"
+	"testing"
+
+	"github.com/tendermint/abci/example/dummy"
+
+	nm "github.com/tendermint/tendermint/node"
+	rpctest "github.com/tendermint/tendermint/rpc/test"
+)
+
+var node *nm.Node
+
+func TestMain(m *testing.M) {
+	// start a tendermint node (and merkleeyes) in the background to test against
+	app := dummy.NewDummyApplication()
+	node = rpctest.StartTendermint(app)
+	code := m.Run()
+
+	// and shut down proper at the end
+	node.Stop()
+	node.Wait()
+	os.Exit(code)
+}

certifiers/client/provider.go

@@ -0,0 +1,133 @@
+/*
+Package client defines a provider that uses a rpcclient
+to get information, which is used to get new headers
+and validators directly from a node.
+*/
+package client
+
+import (
+	"bytes"
+
+	rpcclient "github.com/tendermint/tendermint/rpc/client"
+	ctypes "github.com/tendermint/tendermint/rpc/core/types"
+	"github.com/tendermint/tendermint/types"
+
+	"github.com/tendermint/tendermint/certifiers"
+	certerr "github.com/tendermint/tendermint/certifiers/errors"
+)
+
+type SignStatusClient interface {
+	rpcclient.SignClient
+	rpcclient.StatusClient
+}
+
+type provider struct {
+	node       SignStatusClient
+	lastHeight int
+}
+
+// NewProvider can wrap any rpcclient to expose it as
+// a read-only provider.
+func NewProvider(node SignStatusClient) certifiers.Provider {
+	return &provider{node: node}
+}
+
+// NewProvider can connects to a tendermint json-rpc endpoint
+// at the given url, and uses that as a read-only provider.
+func NewHTTPProvider(remote string) certifiers.Provider {
+	return &provider{
+		node: rpcclient.NewHTTP(remote, "/websocket"),
+	}
+}
+
+// StoreCommit is a noop, as clients can only read from the chain...
+func (p *provider) StoreCommit(_ certifiers.FullCommit) error { return nil }
+
+// GetHash gets the most recent validator and sees if it matches
+//
+// TODO: improve when the rpc interface supports more functionality
+func (p *provider) GetByHash(hash []byte) (certifiers.FullCommit, error) {
+	var fc certifiers.FullCommit
+	vals, err := p.node.Validators(nil)
+	// if we get no validators, or a different height, return an error
+	if err != nil {
+		return fc, err
+	}
+	p.updateHeight(vals.BlockHeight)
+	vhash := types.NewValidatorSet(vals.Validators).Hash()
+	if !bytes.Equal(hash, vhash) {
+		return fc, certerr.ErrCommitNotFound()
+	}
+	return p.seedFromVals(vals)
+}
+
+// GetByHeight gets the validator set by height
+func (p *provider) GetByHeight(h int) (fc certifiers.FullCommit, err error) {
+	commit, err := p.node.Commit(&h)
+	if err != nil {
+		return fc, err
+	}
+	return p.seedFromCommit(commit)
+}
+
+func (p *provider) LatestCommit() (fc certifiers.FullCommit, err error) {
+	commit, err := p.GetLatestCommit()
+	if err != nil {
+		return fc, err
+	}
+	return p.seedFromCommit(commit)
+}
+
+// GetLatestCommit should return the most recent commit there is,
+// which handles queries for future heights as per the semantics
+// of GetByHeight.
+func (p *provider) GetLatestCommit() (*ctypes.ResultCommit, error) {
+	status, err := p.node.Status()
+	if err != nil {
+		return nil, err
+	}
+	return p.node.Commit(&status.LatestBlockHeight)
+}
+
+func CommitFromResult(result *ctypes.ResultCommit) certifiers.Commit {
+	return (certifiers.Commit)(result.SignedHeader)
+}
+
+func (p *provider) seedFromVals(vals *ctypes.ResultValidators) (certifiers.FullCommit, error) {
+	// now get the commits and build a full commit
+	commit, err := p.node.Commit(&vals.BlockHeight)
+	if err != nil {
+		return certifiers.FullCommit{}, err
+	}
+	fc := certifiers.NewFullCommit(
+		CommitFromResult(commit),
+		types.NewValidatorSet(vals.Validators),
+	)
+	return fc, nil
+}
+
+func (p *provider) seedFromCommit(commit *ctypes.ResultCommit) (fc certifiers.FullCommit, err error) {
+	fc.Commit = CommitFromResult(commit)
+
+	// now get the proper validators
+	vals, err := p.node.Validators(&commit.Header.Height)
+	if err != nil {
+		return fc, err
+	}
+
+	// make sure they match the commit (as we cannot enforce height)
+	vset := types.NewValidatorSet(vals.Validators)
+	if !bytes.Equal(vset.Hash(), commit.Header.ValidatorsHash) {
+		return fc, certerr.ErrValidatorsChanged()
+	}
+
+	p.updateHeight(commit.Header.Height)
+	fc.Validators = vset
+	return fc, nil
+}
+
+func (p *provider) updateHeight(h int) {
+	if h > p.lastHeight {
+		p.lastHeight = h
+	}
+}

certifiers/client/provider_test.go

@@ -0,0 +1,62 @@
+package client_test
+
+import (
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+
+	rpctest "github.com/tendermint/tendermint/rpc/test"
+
+	"github.com/tendermint/tendermint/certifiers"
+	"github.com/tendermint/tendermint/certifiers/client"
+	certerr "github.com/tendermint/tendermint/certifiers/errors"
+)
+
+func TestProvider(t *testing.T) {
+	assert, require := assert.New(t), require.New(t)
+
+	cfg := rpctest.GetConfig()
+	rpcAddr := cfg.RPC.ListenAddress
+	chainID := cfg.ChainID
+	p := client.NewHTTPProvider(rpcAddr)
+	require.NotNil(t, p)
+
+	// let it produce some blocks
+	time.Sleep(500 * time.Millisecond)
+
+	// let's get the highest block
+	seed, err := p.LatestCommit()
+
+	require.Nil(err, "%+v", err)
+	sh := seed.Height()
+	vhash := seed.Header.ValidatorsHash
+	assert.True(sh < 5000)
+
+	// let's check this is valid somehow
+	assert.Nil(seed.ValidateBasic(chainID))
+	cert := certifiers.NewStatic(chainID, seed.Validators)
+
+	// historical queries now work :)
+	lower := sh - 5
+	seed, err = p.GetByHeight(lower)
+	assert.Nil(err, "%+v", err)
+	assert.Equal(lower, seed.Height())
+
+	// also get by hash (given the match)
+	seed, err = p.GetByHash(vhash)
+	require.Nil(err, "%+v", err)
+	require.Equal(vhash, seed.Header.ValidatorsHash)
+	err = cert.Certify(seed.Commit)
+	assert.Nil(err, "%+v", err)
+
+	// get by hash fails without match
+	seed, err = p.GetByHash([]byte("foobar"))
+	assert.NotNil(err)
+	assert.True(certerr.IsCommitNotFoundErr(err))
+
+	// storing the seed silently ignored
+	err = p.StoreCommit(seed)
+	assert.Nil(err, "%+v", err)
+}

certifiers/commit.go

@@ -0,0 +1,96 @@
+package certifiers
+
+import (
+	"bytes"
+
+	"github.com/pkg/errors"
+
+	"github.com/tendermint/tendermint/types"
+
+	certerr "github.com/tendermint/tendermint/certifiers/errors"
+)
+
+// Certifier checks the votes to make sure the block really is signed properly.
+// Certifier must know the current set of validitors by some other means.
+type Certifier interface {
+	Certify(check Commit) error
+	ChainID() string
+}
+
+// Commit is basically the rpc /commit response, but extended
+//
+// This is the basepoint for proving anything on the blockchain. It contains
+// a signed header.  If the signatures are valid and > 2/3 of the known set,
+// we can store this checkpoint and use it to prove any number of aspects of
+// the system: such as txs, abci state, validator sets, etc...
+type Commit types.SignedHeader
+
+// FullCommit is a commit and the actual validator set,
+// the base info you need to update to a given point,
+// assuming knowledge of some previous validator set
+type FullCommit struct {
+	Commit     `json:"commit"`
+	Validators *types.ValidatorSet `json:"validator_set"`
+}
+
+func NewFullCommit(commit Commit, vals *types.ValidatorSet) FullCommit {
+	return FullCommit{
+		Commit:     commit,
+		Validators: vals,
+	}
+}
+
+func (c Commit) Height() int {
+	if c.Header == nil {
+		return 0
+	}
+	return c.Header.Height
+}
+
+func (c Commit) ValidatorsHash() []byte {
+	if c.Header == nil {
+		return nil
+	}
+	return c.Header.ValidatorsHash
+}
+
+// ValidateBasic does basic consistency checks and makes sure the headers
+// and commits are all consistent and refer to our chain.
+//
+// Make sure to use a Verifier to validate the signatures actually provide
+// a significantly strong proof for this header's validity.
+func (c Commit) ValidateBasic(chainID string) error {
+	// make sure the header is reasonable
+	if c.Header == nil {
+		return errors.New("Commit missing header")
+	}
+	if c.Header.ChainID != chainID {
+		return errors.Errorf("Header belongs to another chain '%s' not '%s'",
+			c.Header.ChainID, chainID)
+	}
+
+	if c.Commit == nil {
+		return errors.New("Commit missing signatures")
+	}
+
+	// make sure the header and commit match (height and hash)
+	if c.Commit.Height() != c.Header.Height {
+		return certerr.ErrHeightMismatch(c.Commit.Height(), c.Header.Height)
+	}
+	hhash := c.Header.Hash()
+	chash := c.Commit.BlockID.Hash
+	if !bytes.Equal(hhash, chash) {
+		return errors.Errorf("Commits sign block %X header is block %X",
+			chash, hhash)
+	}
+
+	// make sure the commit is reasonable
+	err := c.Commit.ValidateBasic()
+	if err != nil {
+		return errors.WithStack(err)
+	}
+
+	// looks good, we just need to make sure the signatures are really from
+	// empowered validators
+	return nil
+}

certifiers/doc.go

@@ -0,0 +1,133 @@
+/*
+Package certifiers allows you to securely validate headers
+without a full node.
+
+This library pulls together all the crypto and algorithms,
+so given a relatively recent (< unbonding period) known
+validator set, one can get indisputable proof that data is in
+the chain (current state) or detect if the node is lying to
+the client.
+
+Tendermint RPC exposes a lot of info, but a malicious node
+could return any data it wants to queries, or even to block
+headers, even making up fake signatures from non-existent
+validators to justify it. This is a lot of logic to get
+right, to be contained in a small, easy to use library,
+that does this for you, so you can just build nice UI.
+
+We design for clients who have no strong trust relationship
+with any tendermint node, just the validator set as a whole.
+Beyond building nice mobile or desktop applications, the
+cosmos hub is another important example of a client,
+that needs undeniable proof without syncing the full chain,
+in order to efficiently implement IBC.
+
+Commits
+
+There are two main data structures that we pass around - Commit
+and FullCommit. Both of them mirror what information is
+exposed in tendermint rpc.
+
+Commit is a block header along with enough validator signatures
+to prove its validity (> 2/3 of the voting power). A FullCommit
+is a Commit along with the full validator set. When the
+validator set doesn't change, the Commit is enough, but since
+the block header only has a hash, we need the FullCommit to
+follow any changes to the validator set.
+
+Certifiers
+
+A Certifier validates a new Commit given the currently known
+state. There are three different types of Certifiers exposed,
+each one building on the last one, with additional complexity.
+
+Static - given the validator set upon initialization. Verifies
+all signatures against that set and if the validator set
+changes, it will reject all headers.
+
+Dynamic - This wraps Static and has the same Certify
+method. However, it adds an Update method, which can be called
+with a FullCommit when the validator set changes. If it can
+prove this is a valid transition, it will update the validator
+set.
+
+Inquiring - this wraps Dynamic and implements an auto-update
+strategy on top of the Dynamic update. If a call to
+Certify fails as the validator set has changed, then it
+attempts to find a FullCommit and Update to that header.
+To get these FullCommits, it makes use of a Provider.
+
+Providers
+
+A Provider allows us to store and retrieve the FullCommits,
+to provide memory to the Inquiring Certifier.
+
+NewMemStoreProvider - in-memory cache.
+
+files.NewProvider - disk backed storage.
+
+client.NewHTTPProvider - query tendermint rpc.
+
+NewCacheProvider - combine multiple providers.
+
+The suggested use for local light clients is
+client.NewHTTPProvider for getting new data (Source),
+and NewCacheProvider(NewMemStoreProvider(),
+files.NewProvider()) to store confirmed headers (Trusted)
+
+How We Track Validators
+
+Unless you want to blindly trust the node you talk with, you
+need to trace every response back to a hash in a block header
+and validate the commit signatures of that block header match
+the proper validator set.  If there is a contant validator
+set, you store it locally upon initialization of the client,
+and check against that every time.
+
+Once there is a dynamic validator set, the issue of
+verifying a block becomes a bit more tricky. There is
+background information in a
+github issue (https://github.com/tendermint/tendermint/issues/377).
+
+In short, if there is a block at height H with a known
+(trusted) validator set V, and another block at height H'
+(H' > H) with validator set V' != V, then we want a way to
+safely update it.
+
+First, get the new (unconfirmed) validator set V' and
+verify H' is internally consistent and properly signed by
+this V'. Assuming it is a valid block, we check that at
+least 2/3 of the validators in V also signed it, meaning
+it would also be valid under our old assumptions.
+That should be enough, but we can also check that the
+V counts for at least 2/3 of the total votes in H'
+for extra safety (we can have a discussion if this is
+strictly required). If we can verify all this,
+then we can accept H' and V' as valid and use that to
+validate all blocks X > H'.
+
+If we cannot update directly from H -> H' because there was
+too much change to the validator set, then we can look for
+some Hm (H < Hm < H') with a validator set Vm.  Then we try
+to update H -> Hm and Hm -> H' in two separate steps.
+If one of these steps doesn't work, then we continue
+bisecting, until we eventually have to externally
+validate the valdiator set changes at every block.
+
+Since we never trust any server in this protocol, only the
+signatures themselves, it doesn't matter if the seed comes
+from a (possibly malicious) node or a (possibly malicious) user.
+We can accept it or reject it based only on our trusted
+validator set and cryptographic proofs. This makes it
+extremely important to verify that you have the proper
+validator set when initializing the client, as that is the
+root of all trust.
+
+Or course, this assumes that the known block is within the
+unbonding period to avoid the "nothing at stake" problem.
+If you haven't seen the state in a few months, you will need
+to manually verify the new validator set hash using off-chain
+means (the same as getting the initial hash).
+
+*/
+package certifiers

certifiers/dynamic.go

@@ -0,0 +1,89 @@
+package certifiers
+
+import (
+	"github.com/tendermint/tendermint/types"
+
+	certerr "github.com/tendermint/tendermint/certifiers/errors"
+)
+
+var _ Certifier = &Dynamic{}
+
+// Dynamic uses a Static for Certify, but adds an
+// Update method to allow for a change of validators.
+//
+// You can pass in a FullCommit with another validator set,
+// and if this is a provably secure transition (< 1/3 change,
+// sufficient signatures), then it will update the
+// validator set for the next Certify call.
+// For security, it will only follow validator set changes
+// going forward.
+type Dynamic struct {
+	cert       *Static
+	lastHeight int
+}
+
+func NewDynamic(chainID string, vals *types.ValidatorSet, height int) *Dynamic {
+	return &Dynamic{
+		cert:       NewStatic(chainID, vals),
+		lastHeight: height,
+	}
+}
+
+func (c *Dynamic) ChainID() string {
+	return c.cert.ChainID()
+}
+
+func (c *Dynamic) Validators() *types.ValidatorSet {
+	return c.cert.vSet
+}
+
+func (c *Dynamic) Hash() []byte {
+	return c.cert.Hash()
+}
+
+func (c *Dynamic) LastHeight() int {
+	return c.lastHeight
+}
+
+// Certify handles this with
+func (c *Dynamic) Certify(check Commit) error {
+	err := c.cert.Certify(check)
+	if err == nil {
+		// update last seen height if input is valid
+		c.lastHeight = check.Height()
+	}
+	return err
+}
+
+// Update will verify if this is a valid change and update
+// the certifying validator set if safe to do so.
+//
+// Returns an error if update is impossible (invalid proof or IsTooMuchChangeErr)
+func (c *Dynamic) Update(fc FullCommit) error {
+	// ignore all checkpoints in the past -> only to the future
+	h := fc.Height()
+	if h <= c.lastHeight {
+		return certerr.ErrPastTime()
+	}
+
+	// first, verify if the input is self-consistent....
+	err := fc.ValidateBasic(c.ChainID())
+	if err != nil {
+		return err
+	}
+
+	// now, make sure not too much change... meaning this commit
+	// would be approved by the currently known validator set
+	// as well as the new set
+	commit := fc.Commit.Commit
+	err = c.Validators().VerifyCommitAny(fc.Validators, c.ChainID(),
+		commit.BlockID, h, commit)
+	if err != nil {
+		return certerr.ErrTooMuchChange()
+	}
+
+	// looks good, we can update
+	c.cert = NewStatic(c.ChainID(), fc.Validators)
+	c.lastHeight = h
+	return nil
+}

certifiers/dynamic_test.go

@@ -0,0 +1,130 @@
+package certifiers_test
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+
+	"github.com/tendermint/tendermint/types"
+
+	"github.com/tendermint/tendermint/certifiers"
+	"github.com/tendermint/tendermint/certifiers/errors"
+)
+
+// TestDynamicCert just makes sure it still works like StaticCert
+func TestDynamicCert(t *testing.T) {
+	// assert, require := assert.New(t), require.New(t)
+	assert := assert.New(t)
+	// require := require.New(t)
+
+	keys := certifiers.GenValKeys(4)
+	// 20, 30, 40, 50 - the first 3 don't have 2/3, the last 3 do!
+	vals := keys.ToValidators(20, 10)
+	// and a certifier based on our known set
+	chainID := "test-dyno"
+	cert := certifiers.NewDynamic(chainID, vals, 0)
+
+	cases := []struct {
+		keys        certifiers.ValKeys
+		vals        *types.ValidatorSet
+		height      int
+		first, last int  // who actually signs
+		proper      bool // true -> expect no error
+		changed     bool // true -> expect validator change error
+	}{
+		// perfect, signed by everyone
+		{keys, vals, 1, 0, len(keys), true, false},
+		// skip little guy is okay
+		{keys, vals, 2, 1, len(keys), true, false},
+		// but not the big guy
+		{keys, vals, 3, 0, len(keys) - 1, false, false},
+		// even changing the power a little bit breaks the static validator
+		// the sigs are enough, but the validator hash is unknown
+		{keys, keys.ToValidators(20, 11), 4, 0, len(keys), false, true},
+	}
+
+	for _, tc := range cases {
+		check := tc.keys.GenCommit(chainID, tc.height, nil, tc.vals,
+			[]byte("bar"), tc.first, tc.last)
+		err := cert.Certify(check)
+		if tc.proper {
+			assert.Nil(err, "%+v", err)
+			assert.Equal(cert.LastHeight(), tc.height)
+		} else {
+			assert.NotNil(err)
+			if tc.changed {
+				assert.True(errors.IsValidatorsChangedErr(err), "%+v", err)
+			}
+		}
+	}
+}
+
+// TestDynamicUpdate makes sure we update safely and sanely
+func TestDynamicUpdate(t *testing.T) {
+	assert, require := assert.New(t), require.New(t)
+
+	chainID := "test-dyno-up"
+	keys := certifiers.GenValKeys(5)
+	vals := keys.ToValidators(20, 0)
+	cert := certifiers.NewDynamic(chainID, vals, 40)
+
+	// one valid block to give us a sense of time
+	h := 100
+	good := keys.GenCommit(chainID, h, nil, vals, []byte("foo"), 0, len(keys))
+	err := cert.Certify(good)
+	require.Nil(err, "%+v", err)
+
+	// some new sets to try later
+	keys2 := keys.Extend(2)
+	keys3 := keys2.Extend(4)
+
+	// we try to update with some blocks
+	cases := []struct {
+		keys        certifiers.ValKeys
+		vals        *types.ValidatorSet
+		height      int
+		first, last int  // who actually signs
+		proper      bool // true -> expect no error
+		changed     bool // true -> expect too much change error
+	}{
+		// same validator set, well signed, of course it is okay
+		{keys, vals, h + 10, 0, len(keys), true, false},
+		// same validator set, poorly signed, fails
+		{keys, vals, h + 20, 2, len(keys), false, false},
+
+		// shift the power a little, works if properly signed
+		{keys, keys.ToValidators(10, 0), h + 30, 1, len(keys), true, false},
+		// but not on a poor signature
+		{keys, keys.ToValidators(10, 0), h + 40, 2, len(keys), false, false},
+		// and not if it was in the past
+		{keys, keys.ToValidators(10, 0), h + 25, 0, len(keys), false, false},
+
+		// let's try to adjust to a whole new validator set (we have 5/7 of the votes)
+		{keys2, keys2.ToValidators(10, 0), h + 33, 0, len(keys2), true, false},
+
+		// properly signed but too much change, not allowed (only 7/11 validators known)
+		{keys3, keys3.ToValidators(10, 0), h + 50, 0, len(keys3), false, true},
+	}
+
+	for _, tc := range cases {
+		fc := tc.keys.GenFullCommit(chainID, tc.height, nil, tc.vals,
+			[]byte("bar"), tc.first, tc.last)
+		err := cert.Update(fc)
+		if tc.proper {
+			assert.Nil(err, "%d: %+v", tc.height, err)
+			// we update last seen height
+			assert.Equal(cert.LastHeight(), tc.height)
+			// and we update the proper validators
+			assert.EqualValues(fc.Header.ValidatorsHash, cert.Hash())
+		} else {
+			assert.NotNil(err, "%d", tc.height)
+			// we don't update the height
+			assert.NotEqual(cert.LastHeight(), tc.height)
+			if tc.changed {
+				assert.True(errors.IsTooMuchChangeErr(err),
+					"%d: %+v", tc.height, err)
+			}
+		}
+	}
+}

certifiers/errors/errors.go

@@ -0,0 +1,86 @@
+package errors
+
+import (
+	"fmt"
+
+	"github.com/pkg/errors"
+)
+
+var (
+	errValidatorsChanged = fmt.Errorf("Validators differ between header and certifier")
+	errCommitNotFound    = fmt.Errorf("Commit not found by provider")
+	errTooMuchChange     = fmt.Errorf("Validators change too much to safely update")
+	errPastTime          = fmt.Errorf("Update older than certifier height")
+	errNoPathFound       = fmt.Errorf("Cannot find a path of validators")
+)
+
+// IsCommitNotFoundErr checks whether an error is due to missing data
+func IsCommitNotFoundErr(err error) bool {
+	return err != nil && (errors.Cause(err) == errCommitNotFound)
+}
+
+func ErrCommitNotFound() error {
+	return errors.WithStack(errCommitNotFound)
+}
+
+// IsValidatorsChangedErr checks whether an error is due
+// to a differing validator set
+func IsValidatorsChangedErr(err error) bool {
+	return err != nil && (errors.Cause(err) == errValidatorsChanged)
+}
+
+func ErrValidatorsChanged() error {
+	return errors.WithStack(errValidatorsChanged)
+}
+
+// IsTooMuchChangeErr checks whether an error is due to too much change
+// between these validators sets
+func IsTooMuchChangeErr(err error) bool {
+	return err != nil && (errors.Cause(err) == errTooMuchChange)
+}
+
+func ErrTooMuchChange() error {
+	return errors.WithStack(errTooMuchChange)
+}
+
+func IsPastTimeErr(err error) bool {
+	return err != nil && (errors.Cause(err) == errPastTime)
+}
+
+func ErrPastTime() error {
+	return errors.WithStack(errPastTime)
+}
+
+// IsNoPathFoundErr checks whether an error is due to no path of
+// validators in provider from where we are to where we want to be
+func IsNoPathFoundErr(err error) bool {
+	return err != nil && (errors.Cause(err) == errNoPathFound)
+}
+
+func ErrNoPathFound() error {
+	return errors.WithStack(errNoPathFound)
+}
+
+//--------------------------------------------
+
+type errHeightMismatch struct {
+	h1, h2 int
+}
+
+func (e errHeightMismatch) Error() string {
+	return fmt.Sprintf("Blocks don't match - %d vs %d", e.h1, e.h2)
+}
+
+// IsHeightMismatchErr checks whether an error is due to data from different blocks
+func IsHeightMismatchErr(err error) bool {
+	if err == nil {
+		return false
+	}
+	_, ok := errors.Cause(err).(errHeightMismatch)
+	return ok
+}
+
+// ErrHeightMismatch returns an mismatch error with stack-trace
+func ErrHeightMismatch(h1, h2 int) error {
+	return errors.WithStack(errHeightMismatch{h1, h2})
+}

certifiers/errors/errors_test.go

@@ -0,0 +1,18 @@
+package errors
+
+import (
+	"errors"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+)
+
+func TestErrorHeight(t *testing.T) {
+	e1 := ErrHeightMismatch(2, 3)
+	e1.Error()
+	assert.True(t, IsHeightMismatchErr(e1))
+
+	e2 := errors.New("foobar")
+	assert.False(t, IsHeightMismatchErr(e2))
+	assert.False(t, IsHeightMismatchErr(nil))
+}

certifiers/files/commit.go

@@ -0,0 +1,77 @@
+package files
+
+import (
+	"encoding/json"
+	"os"
+
+	"github.com/pkg/errors"
+
+	wire "github.com/tendermint/go-wire"
+
+	"github.com/tendermint/tendermint/certifiers"
+	certerr "github.com/tendermint/tendermint/certifiers/errors"
+)
+
+const (
+	// MaxFullCommitSize is the maximum number of bytes we will
+	// read in for a full commit to avoid excessive allocations
+	// in the deserializer
+	MaxFullCommitSize = 1024 * 1024
+)
+
+// SaveFullCommit exports the seed in binary / go-wire style
+func SaveFullCommit(fc certifiers.FullCommit, path string) error {
+	f, err := os.Create(path)
+	if err != nil {
+		return errors.WithStack(err)
+	}
+	defer f.Close()
+
+	var n int
+	wire.WriteBinary(fc, f, &n, &err)
+	return errors.WithStack(err)
+}
+
+// SaveFullCommitJSON exports the seed in a json format
+func SaveFullCommitJSON(fc certifiers.FullCommit, path string) error {
+	f, err := os.Create(path)
+	if err != nil {
+		return errors.WithStack(err)
+	}
+	defer f.Close()
+	stream := json.NewEncoder(f)
+	err = stream.Encode(fc)
+	return errors.WithStack(err)
+}
+
+func LoadFullCommit(path string) (certifiers.FullCommit, error) {
+	var fc certifiers.FullCommit
+	f, err := os.Open(path)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return fc, certerr.ErrCommitNotFound()
+		}
+		return fc, errors.WithStack(err)
+	}
+	defer f.Close()
+
+	var n int
+	wire.ReadBinaryPtr(&fc, f, MaxFullCommitSize, &n, &err)
+	return fc, errors.WithStack(err)
+}
+
+func LoadFullCommitJSON(path string) (certifiers.FullCommit, error) {
+	var fc certifiers.FullCommit
+	f, err := os.Open(path)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return fc, certerr.ErrCommitNotFound()
+		}
+		return fc, errors.WithStack(err)
+	}
+	defer f.Close()
+
+	stream := json.NewDecoder(f)
+	err = stream.Decode(&fc)
+	return fc, errors.WithStack(err)
+}

certifiers/files/commit_test.go

@@ -0,0 +1,66 @@
+package files
+
+import (
+	"os"
+	"path/filepath"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+
+	cmn "github.com/tendermint/tmlibs/common"
+
+	"github.com/tendermint/tendermint/certifiers"
+)
+
+func tmpFile() string {
+	suffix := cmn.RandStr(16)
+	return filepath.Join(os.TempDir(), "fc-test-"+suffix)
+}
+
+func TestSerializeFullCommits(t *testing.T) {
+	assert, require := assert.New(t), require.New(t)
+
+	// some constants
+	appHash := []byte("some crazy thing")
+	chainID := "ser-ial"
+	h := 25
+
+	// build a fc
+	keys := certifiers.GenValKeys(5)
+	vals := keys.ToValidators(10, 0)
+	fc := keys.GenFullCommit(chainID, h, nil, vals, appHash, 0, 5)
+
+	require.Equal(h, fc.Height())
+	require.Equal(vals.Hash(), fc.ValidatorsHash())
+
+	// try read/write with json
+	jfile := tmpFile()
+	defer os.Remove(jfile)
+	jseed, err := LoadFullCommitJSON(jfile)
+	assert.NotNil(err)
+	err = SaveFullCommitJSON(fc, jfile)
+	require.Nil(err)
+	jseed, err = LoadFullCommitJSON(jfile)
+	assert.Nil(err, "%+v", err)
+	assert.Equal(h, jseed.Height())
+	assert.Equal(vals.Hash(), jseed.ValidatorsHash())
+
+	// try read/write with binary
+	bfile := tmpFile()
+	defer os.Remove(bfile)
+	bseed, err := LoadFullCommit(bfile)
+	assert.NotNil(err)
+	err = SaveFullCommit(fc, bfile)
+	require.Nil(err)
+	bseed, err = LoadFullCommit(bfile)
+	assert.Nil(err, "%+v", err)
+	assert.Equal(h, bseed.Height())
+	assert.Equal(vals.Hash(), bseed.ValidatorsHash())
+
+	// make sure they don't read the other format (different)
+	_, err = LoadFullCommit(jfile)
+	assert.NotNil(err)
+	_, err = LoadFullCommitJSON(bfile)
+	assert.NotNil(err)
+}

certifiers/files/provider.go

@@ -0,0 +1,134 @@
+/*
+Package files defines a Provider that stores all data in the filesystem
+
+We assume the same validator hash may be reused by many different
+headers/Commits, and thus store it separately. This leaves us
+with three issues:
+
+  1. Given a validator hash, retrieve the validator set if previously stored
+  2. Given a block height, find the Commit with the highest height <= h
+  3. Given a FullCommit, store it quickly to satisfy 1 and 2
+
+Note that we do not worry about caching, as that can be achieved by
+pairing this with a MemStoreProvider and CacheProvider from certifiers
+*/
+package files
+
+import (
+	"encoding/hex"
+	"fmt"
+	"math"
+	"os"
+	"path/filepath"
+	"sort"
+
+	"github.com/pkg/errors"
+
+	"github.com/tendermint/tendermint/certifiers"
+	certerr "github.com/tendermint/tendermint/certifiers/errors"
+)
+
+const (
+	Ext      = ".tsd"
+	ValDir   = "validators"
+	CheckDir = "checkpoints"
+	dirPerm  = os.FileMode(0755)
+	filePerm = os.FileMode(0644)
+)
+
+type provider struct {
+	valDir   string
+	checkDir string
+}
+
+// NewProvider creates the parent dir and subdirs
+// for validators and checkpoints as needed
+func NewProvider(dir string) certifiers.Provider {
+	valDir := filepath.Join(dir, ValDir)
+	checkDir := filepath.Join(dir, CheckDir)
+	for _, d := range []string{valDir, checkDir} {
+		err := os.MkdirAll(d, dirPerm)
+		if err != nil {
+			panic(err)
+		}
+	}
+	return &provider{valDir: valDir, checkDir: checkDir}
+}
+
+func (p *provider) encodeHash(hash []byte) string {
+	return hex.EncodeToString(hash) + Ext
+}
+
+func (p *provider) encodeHeight(h int) string {
+	// pad up to 10^12 for height...
+	return fmt.Sprintf("%012d%s", h, Ext)
+}
+
+func (p *provider) StoreCommit(fc certifiers.FullCommit) error {
+	// make sure the fc is self-consistent before saving
+	err := fc.ValidateBasic(fc.Commit.Header.ChainID)
+	if err != nil {
+		return err
+	}
+
+	paths := []string{
+		filepath.Join(p.checkDir, p.encodeHeight(fc.Height())),
+		filepath.Join(p.valDir, p.encodeHash(fc.Header.ValidatorsHash)),
+	}
+	for _, path := range paths {
+		err := SaveFullCommit(fc, path)
+		// unknown error in creating or writing immediately breaks
+		if err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+func (p *provider) GetByHeight(h int) (certifiers.FullCommit, error) {
+	// first we look for exact match, then search...
+	path := filepath.Join(p.checkDir, p.encodeHeight(h))
+	fc, err := LoadFullCommit(path)
+	if certerr.IsCommitNotFoundErr(err) {
+		path, err = p.searchForHeight(h)
+		if err == nil {
+			fc, err = LoadFullCommit(path)
+		}
+	}
+	return fc, err
+}
+
+func (p *provider) LatestCommit() (fc certifiers.FullCommit, err error) {
+	// Note to future: please update by 2077 to avoid rollover
+	return p.GetByHeight(math.MaxInt32 - 1)
+}
+
+// search for height, looks for a file with highest height < h
+// return certifiers.ErrCommitNotFound() if not there...
+func (p *provider) searchForHeight(h int) (string, error) {
+	d, err := os.Open(p.checkDir)
+	if err != nil {
+		return "", errors.WithStack(err)
+	}
+	files, err := d.Readdirnames(0)
+
+	d.Close()
+	if err != nil {
+		return "", errors.WithStack(err)
+	}
+
+	desired := p.encodeHeight(h)
+	sort.Strings(files)
+	i := sort.SearchStrings(files, desired)
+	if i == 0 {
+		return "", certerr.ErrCommitNotFound()
+	}
+	found := files[i-1]
+	path := filepath.Join(p.checkDir, found)
+	return path, errors.WithStack(err)
+}
+
+func (p *provider) GetByHash(hash []byte) (certifiers.FullCommit, error) {
+	path := filepath.Join(p.valDir, p.encodeHash(hash))
+	return LoadFullCommit(path)
+}

certifiers/files/provider_test.go

@@ -0,0 +1,96 @@
+package files_test
+
+import (
+	"bytes"
+	"errors"
+	"io/ioutil"
+	"os"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+
+	"github.com/tendermint/tendermint/certifiers"
+	certerr "github.com/tendermint/tendermint/certifiers/errors"
+	"github.com/tendermint/tendermint/certifiers/files"
+)
+
+func checkEqual(stored, loaded certifiers.FullCommit, chainID string) error {
+	err := loaded.ValidateBasic(chainID)
+	if err != nil {
+		return err
+	}
+	if !bytes.Equal(stored.ValidatorsHash(), loaded.ValidatorsHash()) {
+		return errors.New("Different block hashes")
+	}
+	return nil
+}
+
+func TestFileProvider(t *testing.T) {
+	assert, require := assert.New(t), require.New(t)
+
+	dir, err := ioutil.TempDir("", "fileprovider-test")
+	assert.Nil(err)
+	defer os.RemoveAll(dir)
+	p := files.NewProvider(dir)
+
+	chainID := "test-files"
+	appHash := []byte("some-data")
+	keys := certifiers.GenValKeys(5)
+	count := 10
+
+	// make a bunch of seeds...
+	seeds := make([]certifiers.FullCommit, count)
+	for i := 0; i < count; i++ {
+		// two seeds for each validator, to check how we handle dups
+		// (10, 0), (10, 1), (10, 1), (10, 2), (10, 2), ...
+		vals := keys.ToValidators(10, int64(count/2))
+		h := 20 + 10*i
+		check := keys.GenCommit(chainID, h, nil, vals, appHash, 0, 5)
+		seeds[i] = certifiers.NewFullCommit(check, vals)
+	}
+
+	// check provider is empty
+	seed, err := p.GetByHeight(20)
+	require.NotNil(err)
+	assert.True(certerr.IsCommitNotFoundErr(err))
+
+	seed, err = p.GetByHash(seeds[3].ValidatorsHash())
+	require.NotNil(err)
+	assert.True(certerr.IsCommitNotFoundErr(err))
+
+	// now add them all to the provider
+	for _, s := range seeds {
+		err = p.StoreCommit(s)
+		require.Nil(err)
+		// and make sure we can get it back
+		s2, err := p.GetByHash(s.ValidatorsHash())
+		assert.Nil(err)
+		err = checkEqual(s, s2, chainID)
+		assert.Nil(err)
+		// by height as well
+		s2, err = p.GetByHeight(s.Height())
+		err = checkEqual(s, s2, chainID)
+		assert.Nil(err)
+	}
+
+	// make sure we get the last hash if we overstep
+	seed, err = p.GetByHeight(5000)
+	if assert.Nil(err, "%+v", err) {
+		assert.Equal(seeds[count-1].Height(), seed.Height())
+		err = checkEqual(seeds[count-1], seed, chainID)
+		assert.Nil(err)
+	}
+
+	// and middle ones as well
+	seed, err = p.GetByHeight(47)
+	if assert.Nil(err, "%+v", err) {
+		// we only step by 10, so 40 must be the one below this
+		assert.Equal(40, seed.Height())
+	}
+
+	// and proper error for too low
+	_, err = p.GetByHeight(5)
+	assert.NotNil(err)
+	assert.True(certerr.IsCommitNotFoundErr(err))
+}

certifiers/helper.go

@@ -0,0 +1,147 @@
+package certifiers
+
+import (
+	"time"
+
+	crypto "github.com/tendermint/go-crypto"
+
+	"github.com/tendermint/tendermint/types"
+)
+
+// ValKeys is a helper for testing.
+//
+// It lets us simulate signing with many keys, either ed25519 or secp256k1.
+// The main use case is to create a set, and call GenCommit
+// to get propely signed header for testing.
+//
+// You can set different weights of validators each time you call
+// ToValidators, and can optionally extend the validator set later
+// with Extend or ExtendSecp
+type ValKeys []crypto.PrivKey
+
+// GenValKeys produces an array of private keys to generate commits
+func GenValKeys(n int) ValKeys {
+	res := make(ValKeys, n)
+	for i := range res {
+		res[i] = crypto.GenPrivKeyEd25519().Wrap()
+	}
+	return res
+}
+
+// Change replaces the key at index i
+func (v ValKeys) Change(i int) ValKeys {
+	res := make(ValKeys, len(v))
+	copy(res, v)
+	res[i] = crypto.GenPrivKeyEd25519().Wrap()
+	return res
+}
+
+// Extend adds n more keys (to remove, just take a slice)
+func (v ValKeys) Extend(n int) ValKeys {
+	extra := GenValKeys(n)
+	return append(v, extra...)
+}
+
+// GenSecpValKeys produces an array of secp256k1 private keys to generate commits
+func GenSecpValKeys(n int) ValKeys {
+	res := make(ValKeys, n)
+	for i := range res {
+		res[i] = crypto.GenPrivKeySecp256k1().Wrap()
+	}
+	return res
+}
+
+// ExtendSecp adds n more secp256k1 keys (to remove, just take a slice)
+func (v ValKeys) ExtendSecp(n int) ValKeys {
+	extra := GenSecpValKeys(n)
+	return append(v, extra...)
+}
+
+// ToValidators produces a list of validators from the set of keys
+// The first key has weight `init` and it increases by `inc` every step
+// so we can have all the same weight, or a simple linear distribution
+// (should be enough for testing)
+func (v ValKeys) ToValidators(init, inc int64) *types.ValidatorSet {
+	res := make([]*types.Validator, len(v))
+	for i, k := range v {
+		res[i] = types.NewValidator(k.PubKey(), init+int64(i)*inc)
+	}
+	return types.NewValidatorSet(res)
+}
+
+// signHeader properly signs the header with all keys from first to last exclusive
+func (v ValKeys) signHeader(header *types.Header, first, last int) *types.Commit {
+	votes := make([]*types.Vote, len(v))
+
+	// we need this list to keep the ordering...
+	vset := v.ToValidators(1, 0)
+
+	// fill in the votes we want
+	for i := first; i < last; i++ {
+		vote := makeVote(header, vset, v[i])
+		votes[vote.ValidatorIndex] = vote
+	}
+
+	res := &types.Commit{
+		BlockID:    types.BlockID{Hash: header.Hash()},
+		Precommits: votes,
+	}
+	return res
+}
+
+func makeVote(header *types.Header, vals *types.ValidatorSet, key crypto.PrivKey) *types.Vote {
+	addr := key.PubKey().Address()
+	idx, _ := vals.GetByAddress(addr)
+	vote := &types.Vote{
+		ValidatorAddress: addr,
+		ValidatorIndex:   idx,
+		Height:           header.Height,
+		Round:            1,
+		Type:             types.VoteTypePrecommit,
+		BlockID:          types.BlockID{Hash: header.Hash()},
+	}
+	// Sign it
+	signBytes := types.SignBytes(header.ChainID, vote)
+	vote.Signature = key.Sign(signBytes)
+	return vote
+}
+
+func genHeader(chainID string, height int, txs types.Txs,
+	vals *types.ValidatorSet, appHash []byte) *types.Header {
+
+	return &types.Header{
+		ChainID: chainID,
+		Height:  height,
+		Time:    time.Now(),
+		NumTxs:  len(txs),
+		// LastBlockID
+		// LastCommitHash
+		ValidatorsHash: vals.Hash(),
+		DataHash:       txs.Hash(),
+		AppHash:        appHash,
+	}
+}
+
+// GenCommit calls genHeader and signHeader and combines them into a Commit
+func (v ValKeys) GenCommit(chainID string, height int, txs types.Txs,
+	vals *types.ValidatorSet, appHash []byte, first, last int) Commit {
+
+	header := genHeader(chainID, height, txs, vals, appHash)
+	check := Commit{
+		Header: header,
+		Commit: v.signHeader(header, first, last),
+	}
+	return check
+}
+
+// GenFullCommit calls genHeader and signHeader and combines them into a Commit
+func (v ValKeys) GenFullCommit(chainID string, height int, txs types.Txs,
+	vals *types.ValidatorSet, appHash []byte, first, last int) FullCommit {
+
+	header := genHeader(chainID, height, txs, vals, appHash)
+	commit := Commit{
+		Header: header,
+		Commit: v.signHeader(header, first, last),
+	}
+	return NewFullCommit(commit, vals)
+}

certifiers/inquirer.go

@@ -0,0 +1,142 @@
+package certifiers
+
+import (
+	"github.com/tendermint/tendermint/types"
+
+	certerr "github.com/tendermint/tendermint/certifiers/errors"
+)
+
+type Inquiring struct {
+	cert *Dynamic
+	// These are only properly validated data, from local system
+	trusted Provider
+	// This is a source of new info, like a node rpc, or other import method
+	Source Provider
+}
+
+func NewInquiring(chainID string, fc FullCommit, trusted Provider, source Provider) *Inquiring {
+	// store the data in trusted
+	trusted.StoreCommit(fc)
+
+	return &Inquiring{
+		cert:    NewDynamic(chainID, fc.Validators, fc.Height()),
+		trusted: trusted,
+		Source:  source,
+	}
+}
+
+func (c *Inquiring) ChainID() string {
+	return c.cert.ChainID()
+}
+
+func (c *Inquiring) Validators() *types.ValidatorSet {
+	return c.cert.cert.vSet
+}
+
+func (c *Inquiring) LastHeight() int {
+	return c.cert.lastHeight
+}
+
+// Certify makes sure this is checkpoint is valid.
+//
+// If the validators have changed since the last know time, it looks
+// for a path to prove the new validators.
+//
+// On success, it will store the checkpoint in the store for later viewing
+func (c *Inquiring) Certify(commit Commit) error {
+	err := c.useClosestTrust(commit.Height())
+	if err != nil {
+		return err
+	}
+
+	err = c.cert.Certify(commit)
+	if !certerr.IsValidatorsChangedErr(err) {
+		return err
+	}
+	err = c.updateToHash(commit.Header.ValidatorsHash)
+	if err != nil {
+		return err
+	}
+
+	err = c.cert.Certify(commit)
+	if err != nil {
+		return err
+	}
+
+	// store the new checkpoint
+	c.trusted.StoreCommit(
+		NewFullCommit(commit, c.Validators()))
+	return nil
+}
+
+func (c *Inquiring) Update(fc FullCommit) error {
+	err := c.useClosestTrust(fc.Height())
+	if err != nil {
+		return err
+	}
+
+	err = c.cert.Update(fc)
+	if err == nil {
+		c.trusted.StoreCommit(fc)
+	}
+	return err
+}
+
+func (c *Inquiring) useClosestTrust(h int) error {
+	closest, err := c.trusted.GetByHeight(h)
+	if err != nil {
+		return err
+	}
+
+	// if the best seed is not the one we currently use,
+	// let's just reset the dynamic validator
+	if closest.Height() != c.LastHeight() {
+		c.cert = NewDynamic(c.ChainID(), closest.Validators, closest.Height())
+	}
+	return nil
+}
+
+// updateToHash gets the validator hash we want to update to
+// if IsTooMuchChangeErr, we try to find a path by binary search over height
+func (c *Inquiring) updateToHash(vhash []byte) error {
+	// try to get the match, and update
+	fc, err := c.Source.GetByHash(vhash)
+	if err != nil {
+		return err
+	}
+	err = c.cert.Update(fc)
+	// handle IsTooMuchChangeErr by using divide and conquer
+	if certerr.IsTooMuchChangeErr(err) {
+		err = c.updateToHeight(fc.Height())
+	}
+	return err
+}
+
+// updateToHeight will use divide-and-conquer to find a path to h
+func (c *Inquiring) updateToHeight(h int) error {
+	// try to update to this height (with checks)
+	fc, err := c.Source.GetByHeight(h)
+	if err != nil {
+		return err
+	}
+	start, end := c.LastHeight(), fc.Height()
+	if end <= start {
+		return certerr.ErrNoPathFound()
+	}
+	err = c.Update(fc)
+
+	// we can handle IsTooMuchChangeErr specially
+	if !certerr.IsTooMuchChangeErr(err) {
+		return err
+	}
+
+	// try to update to mid
+	mid := (start + end) / 2
+	err = c.updateToHeight(mid)
+	if err != nil {
+		return err
+	}
+
+	// if we made it to mid, we recurse
+	return c.updateToHeight(h)
+}

certifiers/inquirer_test.go

@@ -0,0 +1,165 @@
+package certifiers_test
+
+import (
+	"fmt"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+
+	"github.com/tendermint/tendermint/certifiers"
+)
+
+func TestInquirerValidPath(t *testing.T) {
+	assert, require := assert.New(t), require.New(t)
+	trust := certifiers.NewMemStoreProvider()
+	source := certifiers.NewMemStoreProvider()
+
+	// set up the validators to generate test blocks
+	var vote int64 = 10
+	keys := certifiers.GenValKeys(5)
+	vals := keys.ToValidators(vote, 0)
+
+	// construct a bunch of commits, each with one more height than the last
+	chainID := "inquiry-test"
+	count := 50
+	commits := make([]certifiers.FullCommit, count)
+	for i := 0; i < count; i++ {
+		// extend the keys by 1 each time
+		keys = keys.Extend(1)
+		vals = keys.ToValidators(vote, 0)
+		h := 20 + 10*i
+		appHash := []byte(fmt.Sprintf("h=%d", h))
+		commits[i] = keys.GenFullCommit(chainID, h, nil, vals, appHash, 0, len(keys))
+	}
+
+	// initialize a certifier with the initial state
+	cert := certifiers.NewInquiring(chainID, commits[0], trust, source)
+
+	// this should fail validation....
+	commit := commits[count-1].Commit
+	err := cert.Certify(commit)
+	require.NotNil(err)
+
+	// add a few seed in the middle should be insufficient
+	for i := 10; i < 13; i++ {
+		err := source.StoreCommit(commits[i])
+		require.Nil(err)
+	}
+	err = cert.Certify(commit)
+	assert.NotNil(err)
+
+	// with more info, we succeed
+	for i := 0; i < count; i++ {
+		err := source.StoreCommit(commits[i])
+		require.Nil(err)
+	}
+	err = cert.Certify(commit)
+	assert.Nil(err, "%+v", err)
+}
+
+func TestInquirerMinimalPath(t *testing.T) {
+	assert, require := assert.New(t), require.New(t)
+	trust := certifiers.NewMemStoreProvider()
+	source := certifiers.NewMemStoreProvider()
+
+	// set up the validators to generate test blocks
+	var vote int64 = 10
+	keys := certifiers.GenValKeys(5)
+	vals := keys.ToValidators(vote, 0)
+
+	// construct a bunch of commits, each with one more height than the last
+	chainID := "minimal-path"
+	count := 12
+	commits := make([]certifiers.FullCommit, count)
+	for i := 0; i < count; i++ {
+		// extend the validators, so we are just below 2/3
+		keys = keys.Extend(len(keys)/2 - 1)
+		vals = keys.ToValidators(vote, 0)
+		h := 5 + 10*i
+		appHash := []byte(fmt.Sprintf("h=%d", h))
+		commits[i] = keys.GenFullCommit(chainID, h, nil, vals, appHash, 0, len(keys))
+	}
+
+	// initialize a certifier with the initial state
+	cert := certifiers.NewInquiring(chainID, commits[0], trust, source)
+
+	// this should fail validation....
+	commit := commits[count-1].Commit
+	err := cert.Certify(commit)
+	require.NotNil(err)
+
+	// add a few seed in the middle should be insufficient
+	for i := 5; i < 8; i++ {
+		err := source.StoreCommit(commits[i])
+		require.Nil(err)
+	}
+	err = cert.Certify(commit)
+	assert.NotNil(err)
+
+	// with more info, we succeed
+	for i := 0; i < count; i++ {
+		err := source.StoreCommit(commits[i])
+		require.Nil(err)
+	}
+	err = cert.Certify(commit)
+	assert.Nil(err, "%+v", err)
+}
+
+func TestInquirerVerifyHistorical(t *testing.T) {
+	assert, require := assert.New(t), require.New(t)
+	trust := certifiers.NewMemStoreProvider()
+	source := certifiers.NewMemStoreProvider()
+
+	// set up the validators to generate test blocks
+	var vote int64 = 10
+	keys := certifiers.GenValKeys(5)
+	vals := keys.ToValidators(vote, 0)
+
+	// construct a bunch of commits, each with one more height than the last
+	chainID := "inquiry-test"
+	count := 10
+	commits := make([]certifiers.FullCommit, count)
+	for i := 0; i < count; i++ {
+		// extend the keys by 1 each time
+		keys = keys.Extend(1)
+		vals = keys.ToValidators(vote, 0)
+		h := 20 + 10*i
+		appHash := []byte(fmt.Sprintf("h=%d", h))
+		commits[i] = keys.GenFullCommit(chainID, h, nil, vals, appHash, 0, len(keys))
+	}
+
+	// initialize a certifier with the initial state
+	cert := certifiers.NewInquiring(chainID, commits[0], trust, source)
+
+	// store a few commits as trust
+	for _, i := range []int{2, 5} {
+		trust.StoreCommit(commits[i])
+	}
+
+	// let's see if we can jump forward using trusted commits
+	err := source.StoreCommit(commits[7])
+	require.Nil(err, "%+v", err)
+	check := commits[7].Commit
+	err = cert.Certify(check)
+	require.Nil(err, "%+v", err)
+	assert.Equal(check.Height(), cert.LastHeight())
+
+	// add access to all commits via untrusted source
+	for i := 0; i < count; i++ {
+		err := source.StoreCommit(commits[i])
+		require.Nil(err)
+	}
+
+	// try to check an unknown seed in the past
+	mid := commits[3].Commit
+	err = cert.Certify(mid)
+	require.Nil(err, "%+v", err)
+	assert.Equal(mid.Height(), cert.LastHeight())
+
+	// and jump all the way forward again
+	end := commits[count-1].Commit
+	err = cert.Certify(end)
+	require.Nil(err, "%+v", err)
+	assert.Equal(end.Height(), cert.LastHeight())
+}

certifiers/memprovider.go

@@ -0,0 +1,78 @@
+package certifiers
+
+import (
+	"encoding/hex"
+	"sort"
+
+	certerr "github.com/tendermint/tendermint/certifiers/errors"
+)
+
+type memStoreProvider struct {
+	// byHeight is always sorted by Height... need to support range search (nil, h]
+	// btree would be more efficient for larger sets
+	byHeight fullCommits
+	byHash   map[string]FullCommit
+}
+
+// fullCommits just exists to allow easy sorting
+type fullCommits []FullCommit
+
+func (s fullCommits) Len() int      { return len(s) }
+func (s fullCommits) Swap(i, j int) { s[i], s[j] = s[j], s[i] }
+func (s fullCommits) Less(i, j int) bool {
+	return s[i].Height() < s[j].Height()
+}
+
+func NewMemStoreProvider() Provider {
+	return &memStoreProvider{
+		byHeight: fullCommits{},
+		byHash:   map[string]FullCommit{},
+	}
+}
+
+func (m *memStoreProvider) encodeHash(hash []byte) string {
+	return hex.EncodeToString(hash)
+}
+
+func (m *memStoreProvider) StoreCommit(fc FullCommit) error {
+	// make sure the fc is self-consistent before saving
+	err := fc.ValidateBasic(fc.Commit.Header.ChainID)
+	if err != nil {
+		return err
+	}
+
+	// store the valid fc
+	key := m.encodeHash(fc.ValidatorsHash())
+	m.byHash[key] = fc
+	m.byHeight = append(m.byHeight, fc)
+	sort.Sort(m.byHeight)
+	return nil
+}
+
+func (m *memStoreProvider) GetByHeight(h int) (FullCommit, error) {
+	// search from highest to lowest
+	for i := len(m.byHeight) - 1; i >= 0; i-- {
+		fc := m.byHeight[i]
+		if fc.Height() <= h {
+			return fc, nil
+		}
+	}
+	return FullCommit{}, certerr.ErrCommitNotFound()
+}
+
+func (m *memStoreProvider) GetByHash(hash []byte) (FullCommit, error) {
+	var err error
+	fc, ok := m.byHash[m.encodeHash(hash)]
+	if !ok {
+		err = certerr.ErrCommitNotFound()
+	}
+	return fc, err
+}
+
+func (m *memStoreProvider) LatestCommit() (FullCommit, error) {
+	l := len(m.byHeight)
+	if l == 0 {
+		return FullCommit{}, certerr.ErrCommitNotFound()
+	}
+	return m.byHeight[l-1], nil
+}

certifiers/performance_test.go

@@ -0,0 +1,116 @@
+package certifiers_test
+
+import (
+	"fmt"
+	"testing"
+
+	"github.com/tendermint/tendermint/certifiers"
+)
+
+func BenchmarkGenCommit20(b *testing.B) {
+	keys := certifiers.GenValKeys(20)
+	benchmarkGenCommit(b, keys)
+}
+
+func BenchmarkGenCommit100(b *testing.B) {
+	keys := certifiers.GenValKeys(100)
+	benchmarkGenCommit(b, keys)
+}
+
+func BenchmarkGenCommitSec20(b *testing.B) {
+	keys := certifiers.GenSecpValKeys(20)
+	benchmarkGenCommit(b, keys)
+}
+
+func BenchmarkGenCommitSec100(b *testing.B) {
+	keys := certifiers.GenSecpValKeys(100)
+	benchmarkGenCommit(b, keys)
+}
+
+func benchmarkGenCommit(b *testing.B, keys certifiers.ValKeys) {
+	chainID := fmt.Sprintf("bench-%d", len(keys))
+	vals := keys.ToValidators(20, 10)
+	for i := 0; i < b.N; i++ {
+		h := 1 + i
+		appHash := []byte(fmt.Sprintf("h=%d", h))
+		keys.GenCommit(chainID, h, nil, vals, appHash, 0, len(keys))
+	}
+}
+
+// this benchmarks generating one key
+func BenchmarkGenValKeys(b *testing.B) {
+	keys := certifiers.GenValKeys(20)
+	for i := 0; i < b.N; i++ {
+		keys = keys.Extend(1)
+	}
+}
+
+// this benchmarks generating one key
+func BenchmarkGenSecpValKeys(b *testing.B) {
+	keys := certifiers.GenSecpValKeys(20)
+	for i := 0; i < b.N; i++ {
+		keys = keys.Extend(1)
+	}
+}
+
+func BenchmarkToValidators20(b *testing.B) {
+	benchmarkToValidators(b, 20)
+}
+
+func BenchmarkToValidators100(b *testing.B) {
+	benchmarkToValidators(b, 100)
+}
+
+// this benchmarks constructing the validator set (.PubKey() * nodes)
+func benchmarkToValidators(b *testing.B, nodes int) {
+	keys := certifiers.GenValKeys(nodes)
+	for i := 1; i <= b.N; i++ {
+		keys.ToValidators(int64(2*i), int64(i))
+	}
+}
+
+func BenchmarkToValidatorsSec100(b *testing.B) {
+	benchmarkToValidatorsSec(b, 100)
+}
+
+// this benchmarks constructing the validator set (.PubKey() * nodes)
+func benchmarkToValidatorsSec(b *testing.B, nodes int) {
+	keys := certifiers.GenSecpValKeys(nodes)
+	for i := 1; i <= b.N; i++ {
+		keys.ToValidators(int64(2*i), int64(i))
+	}
+}
+
+func BenchmarkCertifyCommit20(b *testing.B) {
+	keys := certifiers.GenValKeys(20)
+	benchmarkCertifyCommit(b, keys)
+}
+
+func BenchmarkCertifyCommit100(b *testing.B) {
+	keys := certifiers.GenValKeys(100)
+	benchmarkCertifyCommit(b, keys)
+}
+
+func BenchmarkCertifyCommitSec20(b *testing.B) {
+	keys := certifiers.GenSecpValKeys(20)
+	benchmarkCertifyCommit(b, keys)
+}
+
+func BenchmarkCertifyCommitSec100(b *testing.B) {
+	keys := certifiers.GenSecpValKeys(100)
+	benchmarkCertifyCommit(b, keys)
+}
+
+func benchmarkCertifyCommit(b *testing.B, keys certifiers.ValKeys) {
+	chainID := "bench-certify"
+	vals := keys.ToValidators(20, 10)
+	cert := certifiers.NewStatic(chainID, vals)
+	check := keys.GenCommit(chainID, 123, nil, vals, []byte("foo"), 0, len(keys))
+	for i := 0; i < b.N; i++ {
+		err := cert.Certify(check)
+		if err != nil {
+			panic(err)
+		}
+	}
+
+}

certifiers/provider.go

@@ -0,0 +1,125 @@
+package certifiers
+
+import (
+	certerr "github.com/tendermint/tendermint/certifiers/errors"
+)
+
+// Provider is used to get more validators by other means
+//
+// Examples: MemProvider, files.Provider, client.Provider....
+type Provider interface {
+	// StoreCommit saves a FullCommit after we have verified it,
+	// so we can query for it later. Important for updating our
+	// store of trusted commits
+	StoreCommit(fc FullCommit) error
+	// GetByHeight returns the closest commit with height <= h
+	GetByHeight(h int) (FullCommit, error)
+	// GetByHash returns a commit exactly matching this validator hash
+	GetByHash(hash []byte) (FullCommit, error)
+	// LatestCommit returns the newest commit stored
+	LatestCommit() (FullCommit, error)
+}
+
+// cacheProvider allows you to place one or more caches in front of a source
+// Provider.  It runs through them in order until a match is found.
+// So you can keep a local cache, and check with the network if
+// no data is there.
+type cacheProvider struct {
+	Providers []Provider
+}
+
+func NewCacheProvider(providers ...Provider) Provider {
+	return cacheProvider{
+		Providers: providers,
+	}
+}
+
+// StoreCommit tries to add the seed to all providers.
+//
+// Aborts on first error it encounters (closest provider)
+func (c cacheProvider) StoreCommit(fc FullCommit) (err error) {
+	for _, p := range c.Providers {
+		err = p.StoreCommit(fc)
+		if err != nil {
+			break
+		}
+	}
+	return err
+}
+
+/*
+GetByHeight should return the closest possible match from all providers.
+
+The Cache is usually organized in order from cheapest call (memory)
+to most expensive calls (disk/network). However, since GetByHeight returns
+a FullCommit at h' <= h, if the memory has a seed at h-10, but the network would
+give us the exact match, a naive "stop at first non-error" would hide
+the actual desired results.
+
+Thus, we query each provider in order until we find an exact match
+or we finished querying them all.  If at least one returned a non-error,
+then this returns the best match (minimum h-h').
+*/
+func (c cacheProvider) GetByHeight(h int) (fc FullCommit, err error) {
+	for _, p := range c.Providers {
+		var tfc FullCommit
+		tfc, err = p.GetByHeight(h)
+		if err == nil {
+			if tfc.Height() > fc.Height() {
+				fc = tfc
+			}
+			if tfc.Height() == h {
+				break
+			}
+		}
+	}
+	// even if the last one had an error, if any was a match, this is good
+	if fc.Height() > 0 {
+		err = nil
+	}
+	return fc, err
+}
+
+func (c cacheProvider) GetByHash(hash []byte) (fc FullCommit, err error) {
+	for _, p := range c.Providers {
+		fc, err = p.GetByHash(hash)
+		if err == nil {
+			break
+		}
+	}
+	return fc, err
+}
+
+func (c cacheProvider) LatestCommit() (fc FullCommit, err error) {
+	for _, p := range c.Providers {
+		var tfc FullCommit
+		tfc, err = p.LatestCommit()
+		if err == nil && tfc.Height() > fc.Height() {
+			fc = tfc
+		}
+	}
+	// even if the last one had an error, if any was a match, this is good
+	if fc.Height() > 0 {
+		err = nil
+	}
+	return fc, err
+}
+
+// missingProvider doens't store anything, always a miss
+// Designed as a mock for testing
+type missingProvider struct{}
+
+func NewMissingProvider() Provider {
+	return missingProvider{}
+}
+
+func (missingProvider) StoreCommit(_ FullCommit) error { return nil }
+func (missingProvider) GetByHeight(_ int) (FullCommit, error) {
+	return FullCommit{}, certerr.ErrCommitNotFound()
+}
+func (missingProvider) GetByHash(_ []byte) (FullCommit, error) {
+	return FullCommit{}, certerr.ErrCommitNotFound()
+}
+func (missingProvider) LatestCommit() (FullCommit, error) {
+	return FullCommit{}, certerr.ErrCommitNotFound()
+}

certifiers/provider_test.go

@@ -0,0 +1,128 @@
+package certifiers_test
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+
+	"github.com/tendermint/tendermint/certifiers"
+	"github.com/tendermint/tendermint/certifiers/errors"
+)
+
+func TestMemProvider(t *testing.T) {
+	p := certifiers.NewMemStoreProvider()
+	checkProvider(t, p, "test-mem", "empty")
+}
+
+func TestCacheProvider(t *testing.T) {
+	p := certifiers.NewCacheProvider(
+		certifiers.NewMissingProvider(),
+		certifiers.NewMemStoreProvider(),
+		certifiers.NewMissingProvider(),
+	)
+	checkProvider(t, p, "test-cache", "kjfhekfhkewhgit")
+}
+
+func checkProvider(t *testing.T, p certifiers.Provider, chainID, app string) {
+	assert, require := assert.New(t), require.New(t)
+	appHash := []byte(app)
+	keys := certifiers.GenValKeys(5)
+	count := 10
+
+	// make a bunch of commits...
+	commits := make([]certifiers.FullCommit, count)
+	for i := 0; i < count; i++ {
+		// two commits for each validator, to check how we handle dups
+		// (10, 0), (10, 1), (10, 1), (10, 2), (10, 2), ...
+		vals := keys.ToValidators(10, int64(count/2))
+		h := 20 + 10*i
+		commits[i] = keys.GenFullCommit(chainID, h, nil, vals, appHash, 0, 5)
+	}
+
+	// check provider is empty
+	fc, err := p.GetByHeight(20)
+	require.NotNil(err)
+	assert.True(errors.IsCommitNotFoundErr(err))
+
+	fc, err = p.GetByHash(commits[3].ValidatorsHash())
+	require.NotNil(err)
+	assert.True(errors.IsCommitNotFoundErr(err))
+
+	// now add them all to the provider
+	for _, s := range commits {
+		err = p.StoreCommit(s)
+		require.Nil(err)
+		// and make sure we can get it back
+		s2, err := p.GetByHash(s.ValidatorsHash())
+		assert.Nil(err)
+		assert.Equal(s, s2)
+		// by height as well
+		s2, err = p.GetByHeight(s.Height())
+		assert.Nil(err)
+		assert.Equal(s, s2)
+	}
+
+	// make sure we get the last hash if we overstep
+	fc, err = p.GetByHeight(5000)
+	if assert.Nil(err) {
+		assert.Equal(commits[count-1].Height(), fc.Height())
+		assert.Equal(commits[count-1], fc)
+	}
+
+	// and middle ones as well
+	fc, err = p.GetByHeight(47)
+	if assert.Nil(err) {
+		// we only step by 10, so 40 must be the one below this
+		assert.Equal(40, fc.Height())
+	}
+
+}
+
+// this will make a get height, and if it is good, set the data as well
+func checkGetHeight(t *testing.T, p certifiers.Provider, ask, expect int) {
+	fc, err := p.GetByHeight(ask)
+	require.Nil(t, err, "%+v", err)
+	if assert.Equal(t, expect, fc.Height()) {
+		err = p.StoreCommit(fc)
+		require.Nil(t, err, "%+v", err)
+	}
+}
+
+func TestCacheGetsBestHeight(t *testing.T) {
+	// assert, require := assert.New(t), require.New(t)
+	require := require.New(t)
+
+	// we will write data to the second level of the cache (p2),
+	// and see what gets cached, stored in
+	p := certifiers.NewMemStoreProvider()
+	p2 := certifiers.NewMemStoreProvider()
+	cp := certifiers.NewCacheProvider(p, p2)
+
+	chainID := "cache-best-height"
+	appHash := []byte("01234567")
+	keys := certifiers.GenValKeys(5)
+	count := 10
+
+	// set a bunch of commits
+	for i := 0; i < count; i++ {
+		vals := keys.ToValidators(10, int64(count/2))
+		h := 10 * (i + 1)
+		fc := keys.GenFullCommit(chainID, h, nil, vals, appHash, 0, 5)
+		err := p2.StoreCommit(fc)
+		require.NoError(err)
+	}
+
+	// let's get a few heights from the cache and set them proper
+	checkGetHeight(t, cp, 57, 50)
+	checkGetHeight(t, cp, 33, 30)
+
+	// make sure they are set in p as well (but nothing else)
+	checkGetHeight(t, p, 44, 30)
+	checkGetHeight(t, p, 50, 50)
+	checkGetHeight(t, p, 99, 50)
+
+	// now, query the cache for a higher value
+	checkGetHeight(t, p2, 99, 90)
+	checkGetHeight(t, cp, 99, 90)
+}

certifiers/static.go

@@ -0,0 +1,66 @@
+package certifiers
+
+import (
+	"bytes"
+
+	"github.com/pkg/errors"
+
+	"github.com/tendermint/tendermint/types"
+
+	certerr "github.com/tendermint/tendermint/certifiers/errors"
+)
+
+var _ Certifier = &Static{}
+
+// Static assumes a static set of validators, set on
+// initilization and checks against them.
+// The signatures on every header is checked for > 2/3 votes
+// against the known validator set upon Certify
+//
+// Good for testing or really simple chains.  Building block
+// to support real-world functionality.
+type Static struct {
+	chainID string
+	vSet    *types.ValidatorSet
+	vhash   []byte
+}
+
+func NewStatic(chainID string, vals *types.ValidatorSet) *Static {
+	return &Static{
+		chainID: chainID,
+		vSet:    vals,
+	}
+}
+
+func (c *Static) ChainID() string {
+	return c.chainID
+}
+
+func (c *Static) Validators() *types.ValidatorSet {
+	return c.vSet
+}
+
+func (c *Static) Hash() []byte {
+	if len(c.vhash) == 0 {
+		c.vhash = c.vSet.Hash()
+	}
+	return c.vhash
+}
+
+func (c *Static) Certify(commit Commit) error {
+	// do basic sanity checks
+	err := commit.ValidateBasic(c.chainID)
+	if err != nil {
+		return err
+	}
+
+	// make sure it has the same validator set we have (static means static)
+	if !bytes.Equal(c.Hash(), commit.Header.ValidatorsHash) {
+		return certerr.ErrValidatorsChanged()
+	}
+
+	// then make sure we have the proper signatures for this
+	err = c.vSet.VerifyCommit(c.chainID, commit.Commit.BlockID,
+		commit.Header.Height, commit.Commit)
+	return errors.WithStack(err)
+}

certifiers/static_test.go

@@ -0,0 +1,59 @@
+package certifiers_test
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+
+	"github.com/tendermint/tendermint/types"
+
+	"github.com/tendermint/tendermint/certifiers"
+	errors "github.com/tendermint/tendermint/certifiers/errors"
+)
+
+func TestStaticCert(t *testing.T) {
+	// assert, require := assert.New(t), require.New(t)
+	assert := assert.New(t)
+	// require := require.New(t)
+
+	keys := certifiers.GenValKeys(4)
+	// 20, 30, 40, 50 - the first 3 don't have 2/3, the last 3 do!
+	vals := keys.ToValidators(20, 10)
+	// and a certifier based on our known set
+	chainID := "test-static"
+	cert := certifiers.NewStatic(chainID, vals)
+
+	cases := []struct {
+		keys        certifiers.ValKeys
+		vals        *types.ValidatorSet
+		height      int
+		first, last int  // who actually signs
+		proper      bool // true -> expect no error
+		changed     bool // true -> expect validator change error
+	}{
+		// perfect, signed by everyone
+		{keys, vals, 1, 0, len(keys), true, false},
+		// skip little guy is okay
+		{keys, vals, 2, 1, len(keys), true, false},
+		// but not the big guy
+		{keys, vals, 3, 0, len(keys) - 1, false, false},
+		// even changing the power a little bit breaks the static validator
+		// the sigs are enough, but the validator hash is unknown
+		{keys, keys.ToValidators(20, 11), 4, 0, len(keys), false, true},
+	}
+
+	for _, tc := range cases {
+		check := tc.keys.GenCommit(chainID, tc.height, nil, tc.vals,
+			[]byte("foo"), tc.first, tc.last)
+		err := cert.Certify(check)
+		if tc.proper {
+			assert.Nil(err, "%+v", err)
+		} else {
+			assert.NotNil(err)
+			if tc.changed {
+				assert.True(errors.IsValidatorsChangedErr(err), "%+v", err)
+			}
+		}
+	}
+
+}

cmd/tendermint/commands/flags/log_level.go

@@ -1,87 +0,0 @@
-package flags
-
-import (
-	"fmt"
-	"strings"
-
-	"github.com/pkg/errors"
-
-	cfg "github.com/tendermint/tendermint/config"
-	"github.com/tendermint/tmlibs/log"
-)
-
-const (
-	defaultLogLevelKey = "*"
-)
-
-// ParseLogLevel parses complex log level - comma-separated
-// list of module:level pairs with an optional *:level pair (* means
-// all other modules).
-//
-// Example:
-//		ParseLogLevel("consensus:debug,mempool:debug,*:error", log.NewTMLogger(os.Stdout))
-func ParseLogLevel(lvl string, logger log.Logger) (log.Logger, error) {
-	if lvl == "" {
-		return nil, errors.New("Empty log level")
-	}
-
-	l := lvl
-
-	// prefix simple one word levels (e.g. "info") with "*"
-	if !strings.Contains(l, ":") {
-		l = defaultLogLevelKey + ":" + l
-	}
-
-	options := make([]log.Option, 0)
-
-	isDefaultLogLevelSet := false
-	var option log.Option
-	var err error
-
-	list := strings.Split(l, ",")
-	for _, item := range list {
-		moduleAndLevel := strings.Split(item, ":")
-
-		if len(moduleAndLevel) != 2 {
-			return nil, fmt.Errorf("Expected list in a form of \"module:level\" pairs, given pair %s, list %s", item, list)
-		}
-
-		module := moduleAndLevel[0]
-		level := moduleAndLevel[1]
-
-		if module == defaultLogLevelKey {
-			option, err = log.AllowLevel(level)
-			if err != nil {
-				return nil, errors.Wrap(err, fmt.Sprintf("Failed to parse default log level (pair %s, list %s)", item, l))
-			}
-			options = append(options, option)
-			isDefaultLogLevelSet = true
-		} else {
-			switch level {
-			case "debug":
-				option = log.AllowDebugWith("module", module)
-			case "info":
-				option = log.AllowInfoWith("module", module)
-			case "error":
-				option = log.AllowErrorWith("module", module)
-			case "none":
-				option = log.AllowNoneWith("module", module)
-			default:
-				return nil, fmt.Errorf("Expected either \"info\", \"debug\", \"error\" or \"none\" log level, given %s (pair %s, list %s)", level, item, list)
-			}
-			options = append(options, option)
-
-		}
-	}
-
-	// if "*" is not provided, set default global level
-	if !isDefaultLogLevelSet {
-		option, err = log.AllowLevel(cfg.DefaultLogLevel())
-		if err != nil {
-			return nil, err
-		}
-		options = append(options, option)
-	}
-
-	return log.NewFilter(logger, options...), nil
-}

cmd/tendermint/commands/flags/log_level_test.go

@@ -1,64 +0,0 @@
-package flags_test
-
-import (
-	"bytes"
-	"strings"
-	"testing"
-
-	tmflags "github.com/tendermint/tendermint/cmd/tendermint/commands/flags"
-	"github.com/tendermint/tmlibs/log"
-)
-
-func TestParseLogLevel(t *testing.T) {
-	var buf bytes.Buffer
-	jsonLogger := log.NewTMJSONLogger(&buf)
-
-	correctLogLevels := []struct {
-		lvl              string
-		expectedLogLines []string
-	}{
-		{"mempool:error", []string{``, ``, `{"_msg":"Mesmero","level":"error","module":"mempool"}`}},
-		{"mempool:error,*:debug", []string{``, ``, `{"_msg":"Mesmero","level":"error","module":"mempool"}`}},
-		{"*:debug,wire:none", []string{
-			`{"_msg":"Kingpin","level":"debug","module":"mempool"}`,
-			`{"_msg":"Kitty Pryde","level":"info","module":"mempool"}`,
-			`{"_msg":"Mesmero","level":"error","module":"mempool"}`}},
-	}
-
-	for _, c := range correctLogLevels {
-		logger, err := tmflags.ParseLogLevel(c.lvl, jsonLogger)
-		if err != nil {
-			t.Fatal(err)
-		}
-
-		logger = logger.With("module", "mempool")
-
-		buf.Reset()
-
-		logger.Debug("Kingpin")
-		if have := strings.TrimSpace(buf.String()); c.expectedLogLines[0] != have {
-			t.Errorf("\nwant '%s'\nhave '%s'\nlevel '%s'", c.expectedLogLines[0], have, c.lvl)
-		}
-
-		buf.Reset()
-
-		logger.Info("Kitty Pryde")
-		if have := strings.TrimSpace(buf.String()); c.expectedLogLines[1] != have {
-			t.Errorf("\nwant '%s'\nhave '%s'\nlevel '%s'", c.expectedLogLines[1], have, c.lvl)
-		}
-
-		buf.Reset()
-
-		logger.Error("Mesmero")
-		if have := strings.TrimSpace(buf.String()); c.expectedLogLines[2] != have {
-			t.Errorf("\nwant '%s'\nhave '%s'\nlevel '%s'", c.expectedLogLines[2], have, c.lvl)
-		}
-	}
-
-	incorrectLogLevel := []string{"some", "mempool:some", "*:some,mempool:error"}
-	for _, lvl := range incorrectLogLevel {
-		if _, err := tmflags.ParseLogLevel(lvl, jsonLogger); err == nil {
-			t.Fatalf("Expected %s to produce error", lvl)
-		}
-	}
-}

cmd/tendermint/commands/gen_validator.go

@@ -9,18 +9,16 @@ import (
 	"github.com/tendermint/tendermint/types"
 )
 
-var genValidatorCmd = &cobra.Command{
+// GenValidatorCmd allows the generation of a keypair for a
+// validator.
+var GenValidatorCmd = &cobra.Command{
 	Use:   "gen_validator",
 	Short: "Generate new validator keypair",
 	Run:   genValidator,
 }
 
-func init() {
-	RootCmd.AddCommand(genValidatorCmd)
-}
-
 func genValidator(cmd *cobra.Command, args []string) {
-	privValidator := types.GenPrivValidator()
+	privValidator := types.GenPrivValidatorFS("")
 	privValidatorJSONBytes, _ := json.MarshalIndent(privValidator, "", "\t")
 	fmt.Printf(`%v
 `, string(privValidatorJSONBytes))

cmd/tendermint/commands/init.go

@@ -9,21 +9,17 @@ import (
 	cmn "github.com/tendermint/tmlibs/common"
 )
 
-var initFilesCmd = &cobra.Command{
+// InitFilesCmd initialises a fresh Tendermint Core instance.
+var InitFilesCmd = &cobra.Command{
 	Use:   "init",
 	Short: "Initialize Tendermint",
 	Run:   initFiles,
 }
 
-func init() {
-	RootCmd.AddCommand(initFilesCmd)
-}
-
 func initFiles(cmd *cobra.Command, args []string) {
 	privValFile := config.PrivValidatorFile()
 	if _, err := os.Stat(privValFile); os.IsNotExist(err) {
-		privValidator := types.GenPrivValidator()
-		privValidator.SetFile(privValFile)
+		privValidator := types.GenPrivValidatorFS(privValFile)
 		privValidator.Save()
 
 		genFile := config.GenesisFile()
@@ -33,8 +29,8 @@ func initFiles(cmd *cobra.Command, args []string) {
 				ChainID: cmn.Fmt("test-chain-%v", cmn.RandStr(6)),
 			}
 			genDoc.Validators = []types.GenesisValidator{types.GenesisValidator{
-				PubKey: privValidator.PubKey,
-				Amount: 10,
+				PubKey: privValidator.GetPubKey(),
+				Power:  10,
 			}}
 
 			genDoc.SaveAs(genFile)

cmd/tendermint/commands/probe_upnp.go

@@ -9,16 +9,13 @@ import (
 	"github.com/tendermint/tendermint/p2p/upnp"
 )
 
-var probeUpnpCmd = &cobra.Command{
+// ProbeUpnpCmd adds capabilities to test the UPnP functionality.
+var ProbeUpnpCmd = &cobra.Command{
 	Use:   "probe_upnp",
 	Short: "Test UPnP functionality",
 	RunE:  probeUpnp,
 }
 
-func init() {
-	RootCmd.AddCommand(probeUpnpCmd)
-}
-
 func probeUpnp(cmd *cobra.Command, args []string) error {
 	capabilities, err := upnp.Probe(logger)
 	if err != nil {

cmd/tendermint/commands/replay.go

@@ -6,7 +6,8 @@ import (
 	"github.com/tendermint/tendermint/consensus"
 )
 
-var replayCmd = &cobra.Command{
+// ReplayCmd allows replaying of messages from the WAL.
+var ReplayCmd = &cobra.Command{
 	Use:   "replay",
 	Short: "Replay messages from WAL",
 	Run: func(cmd *cobra.Command, args []string) {
@@ -14,15 +15,12 @@ var replayCmd = &cobra.Command{
 	},
 }
 
-var replayConsoleCmd = &cobra.Command{
+// ReplayConsoleCmd allows replaying of messages from the WAL in a
+// console.
+var ReplayConsoleCmd = &cobra.Command{
 	Use:   "replay_console",
 	Short: "Replay messages from WAL in a console",
 	Run: func(cmd *cobra.Command, args []string) {
 		consensus.RunReplayFile(config.BaseConfig, config.Consensus, true)
 	},
 }
-
-func init() {
-	RootCmd.AddCommand(replayCmd)
-	RootCmd.AddCommand(replayConsoleCmd)
-}

cmd/tendermint/commands/reset_priv_validator.go

@@ -9,21 +9,27 @@ import (
 	"github.com/tendermint/tmlibs/log"
 )
 
-var resetAllCmd = &cobra.Command{
+// ResetAllCmd removes the database of this Tendermint core
+// instance.
+var ResetAllCmd = &cobra.Command{
 	Use:   "unsafe_reset_all",
 	Short: "(unsafe) Remove all the data and WAL, reset this node's validator",
 	Run:   resetAll,
 }
 
-var resetPrivValidatorCmd = &cobra.Command{
+// ResetPrivValidatorCmd resets the private validator files.
+var ResetPrivValidatorCmd = &cobra.Command{
 	Use:   "unsafe_reset_priv_validator",
 	Short: "(unsafe) Reset this node's validator",
 	Run:   resetPrivValidator,
 }
 
-func init() {
-	RootCmd.AddCommand(resetAllCmd)
-	RootCmd.AddCommand(resetPrivValidatorCmd)
+// ResetAll removes the privValidator files.
+// Exported so other CLI tools can use  it
+func ResetAll(dbDir, privValFile string, logger log.Logger) {
+	resetPrivValidatorFS(privValFile, logger)
+	os.RemoveAll(dbDir)
+	logger.Info("Removed all data", "dir", dbDir)
 }
 
 // XXX: this is totally unsafe.
@@ -35,26 +41,17 @@ func resetAll(cmd *cobra.Command, args []string) {
 // XXX: this is totally unsafe.
 // it's only suitable for testnets.
 func resetPrivValidator(cmd *cobra.Command, args []string) {
-	resetPrivValidatorLocal(config.PrivValidatorFile(), logger)
+	resetPrivValidatorFS(config.PrivValidatorFile(), logger)
 }
 
-// Exported so other CLI tools can use  it
-func ResetAll(dbDir, privValFile string, logger log.Logger) {
-	resetPrivValidatorLocal(privValFile, logger)
-	os.RemoveAll(dbDir)
-	logger.Info("Removed all data", "dir", dbDir)
-}
-
-func resetPrivValidatorLocal(privValFile string, logger log.Logger) {
+func resetPrivValidatorFS(privValFile string, logger log.Logger) {
 	// Get PrivValidator
-	var privValidator *types.PrivValidator
 	if _, err := os.Stat(privValFile); err == nil {
-		privValidator = types.LoadPrivValidator(privValFile)
+		privValidator := types.LoadPrivValidatorFS(privValFile)
 		privValidator.Reset()
 		logger.Info("Reset PrivValidator", "file", privValFile)
 	} else {
-		privValidator = types.GenPrivValidator()
-		privValidator.SetFile(privValFile)
+		privValidator := types.GenPrivValidatorFS(privValFile)
 		privValidator.Save()
 		logger.Info("Generated PrivValidator", "file", privValFile)
 	}

cmd/tendermint/commands/root.go

@@ -6,8 +6,9 @@ import (
 	"github.com/spf13/cobra"
 	"github.com/spf13/viper"
 
-	tmflags "github.com/tendermint/tendermint/cmd/tendermint/commands/flags"
 	cfg "github.com/tendermint/tendermint/config"
+	"github.com/tendermint/tmlibs/cli"
+	tmflags "github.com/tendermint/tmlibs/cli/flags"
 	"github.com/tendermint/tmlibs/log"
 )
 
@@ -20,20 +21,38 @@ func init() {
 	RootCmd.PersistentFlags().String("log_level", config.LogLevel, "Log level")
 }
 
+// ParseConfig retrieves the default environment configuration,
+// sets up the Tendermint root and ensures that the root exists
+func ParseConfig() (*cfg.Config, error) {
+	conf := cfg.DefaultConfig()
+	err := viper.Unmarshal(conf)
+	if err != nil {
+		return nil, err
+	}
+	conf.SetRoot(conf.RootDir)
+	cfg.EnsureRoot(conf.RootDir)
+	return conf, err
+}
+
+// RootCmd is the root command for Tendermint core.
 var RootCmd = &cobra.Command{
 	Use:   "tendermint",
 	Short: "Tendermint Core (BFT Consensus) in Go",
-	PersistentPreRunE: func(cmd *cobra.Command, args []string) error {
-		err := viper.Unmarshal(config)
+	PersistentPreRunE: func(cmd *cobra.Command, args []string) (err error) {
+		if cmd.Name() == VersionCmd.Name() {
+			return nil
+		}
+		config, err = ParseConfig()
 		if err != nil {
 			return err
 		}
-		config.SetRoot(config.RootDir)
-		cfg.EnsureRoot(config.RootDir)
-		logger, err = tmflags.ParseLogLevel(config.LogLevel, logger)
+		logger, err = tmflags.ParseLogLevel(config.LogLevel, logger, cfg.DefaultLogLevel())
 		if err != nil {
 			return err
 		}
+		if viper.GetBool(cli.TraceFlag) {
+			logger = log.NewTracingLogger(logger)
+		}
 		return nil
 	},
 }

cmd/tendermint/commands/root_test.go

@@ -3,15 +3,15 @@ package commands
 import (
 	"os"
 	"strconv"
+	"testing"
 
 	"github.com/spf13/cobra"
 	"github.com/spf13/viper"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
+
 	cfg "github.com/tendermint/tendermint/config"
 	"github.com/tendermint/tmlibs/cli"
-
-	"testing"
 )
 
 var (
@@ -23,8 +23,12 @@ const (
 )
 
 // isolate provides a clean setup and returns a copy of RootCmd you can
-// modify in the test cases
+// modify in the test cases.
+// NOTE: it unsets all TM* env variables.
 func isolate(cmds ...*cobra.Command) cli.Executable {
+	os.Unsetenv("TMHOME")
+	os.Unsetenv("TM_HOME")
+
 	viper.Reset()
 	config = cfg.DefaultConfig()
 	r := &cobra.Command{
@@ -64,7 +68,7 @@ func TestRootConfig(t *testing.T) {
 		{nil, nil, defaultRoot, defaults.Moniker, defaults.FastSync, dmax},
 		// try multiple ways of setting root (two flags, cli vs. env)
 		{[]string{"--home", conf}, nil, conf, cvals["moniker"], cfast, dmax},
-		{nil, map[string]string{"TMROOT": conf}, conf, cvals["moniker"], cfast, dmax},
+		{nil, map[string]string{"TMHOME": conf}, conf, cvals["moniker"], cfast, dmax},
 		// check setting p2p subflags two different ways
 		{[]string{"--p2p.max_num_peers", "420"}, nil, defaultRoot, defaults.Moniker, defaults.FastSync, 420},
 		{nil, map[string]string{"TM_P2P_MAX_NUM_PEERS": "17"}, defaultRoot, defaults.Moniker, defaults.FastSync, 17},

cmd/tendermint/commands/run_node.go

@@ -2,92 +2,66 @@ package commands
 
 import (
 	"fmt"
-	"io/ioutil"
-	"time"
 
 	"github.com/spf13/cobra"
 
-	"github.com/tendermint/tendermint/node"
-	"github.com/tendermint/tendermint/types"
-	cmn "github.com/tendermint/tmlibs/common"
+	nm "github.com/tendermint/tendermint/node"
 )
 
-var runNodeCmd = &cobra.Command{
-	Use:   "node",
-	Short: "Run the tendermint node",
-	RunE:  runNode,
-}
-
-func init() {
+// AddNodeFlags exposes some common configuration options on the command-line
+// These are exposed for convenience of commands embedding a tendermint node
+func AddNodeFlags(cmd *cobra.Command) {
 	// bind flags
-	runNodeCmd.Flags().String("moniker", config.Moniker, "Node Name")
+	cmd.Flags().String("moniker", config.Moniker, "Node Name")
 
 	// node flags
-	runNodeCmd.Flags().Bool("fast_sync", config.FastSync, "Fast blockchain syncing")
+	cmd.Flags().Bool("fast_sync", config.FastSync, "Fast blockchain syncing")
 
 	// abci flags
-	runNodeCmd.Flags().String("proxy_app", config.ProxyApp, "Proxy app address, or 'nilapp' or 'dummy' for local testing.")
-	runNodeCmd.Flags().String("abci", config.ABCI, "Specify abci transport (socket | grpc)")
+	cmd.Flags().String("proxy_app", config.ProxyApp, "Proxy app address, or 'nilapp' or 'dummy' for local testing.")
+	cmd.Flags().String("abci", config.ABCI, "Specify abci transport (socket | grpc)")
 
 	// rpc flags
-	runNodeCmd.Flags().String("rpc.laddr", config.RPC.ListenAddress, "RPC listen address. Port required")
-	runNodeCmd.Flags().String("rpc.grpc_laddr", config.RPC.GRPCListenAddress, "GRPC listen address (BroadcastTx only). Port required")
-	runNodeCmd.Flags().Bool("rpc.unsafe", config.RPC.Unsafe, "Enabled unsafe rpc methods")
+	cmd.Flags().String("rpc.laddr", config.RPC.ListenAddress, "RPC listen address. Port required")
+	cmd.Flags().String("rpc.grpc_laddr", config.RPC.GRPCListenAddress, "GRPC listen address (BroadcastTx only). Port required")
+	cmd.Flags().Bool("rpc.unsafe", config.RPC.Unsafe, "Enabled unsafe rpc methods")
 
 	// p2p flags
-	runNodeCmd.Flags().String("p2p.laddr", config.P2P.ListenAddress, "Node listen address. (0.0.0.0:0 means any interface, any port)")
-	runNodeCmd.Flags().String("p2p.seeds", config.P2P.Seeds, "Comma delimited host:port seed nodes")
-	runNodeCmd.Flags().Bool("p2p.skip_upnp", config.P2P.SkipUPNP, "Skip UPNP configuration")
-	runNodeCmd.Flags().Bool("p2p.pex", config.P2P.PexReactor, "Enable Peer-Exchange (dev feature)")
+	cmd.Flags().String("p2p.laddr", config.P2P.ListenAddress, "Node listen address. (0.0.0.0:0 means any interface, any port)")
+	cmd.Flags().String("p2p.seeds", config.P2P.Seeds, "Comma delimited host:port seed nodes")
+	cmd.Flags().Bool("p2p.skip_upnp", config.P2P.SkipUPNP, "Skip UPNP configuration")
+	cmd.Flags().Bool("p2p.pex", config.P2P.PexReactor, "Enable Peer-Exchange (dev feature)")
 
-	RootCmd.AddCommand(runNodeCmd)
+	// consensus flags
+	cmd.Flags().Bool("consensus.create_empty_blocks", config.Consensus.CreateEmptyBlocks, "Set this to false to only produce blocks when there are txs or when the AppHash changes")
 }
 
-// Users wishing to:
-//	* Use an external signer for their validators
-//	* Supply an in-proc abci app
-// should import github.com/tendermint/tendermint/node and implement
-// their own run_node to call node.NewNode (instead of node.NewNodeDefault)
-// with their custom priv validator and/or custom proxy.ClientCreator
-func runNode(cmd *cobra.Command, args []string) error {
+// NewRunNodeCmd returns the command that allows the CLI to start a
+// node. It can be used with a custom PrivValidator and in-process ABCI application.
+func NewRunNodeCmd(nodeProvider nm.NodeProvider) *cobra.Command {
+	cmd := &cobra.Command{
+		Use:   "node",
+		Short: "Run the tendermint node",
+		RunE: func(cmd *cobra.Command, args []string) error {
+			// Create & start node
+			n, err := nodeProvider(config, logger)
+			if err != nil {
+				return fmt.Errorf("Failed to create node: %v", err)
+			}
 
-	// Wait until the genesis doc becomes available
-	// This is for Mintnet compatibility.
-	// TODO: If Mintnet gets deprecated or genesis_file is
-	// always available, remove.
-	genDocFile := config.GenesisFile()
-	if !cmn.FileExists(genDocFile) {
-		logger.Info(cmn.Fmt("Waiting for genesis file %v...", genDocFile))
-		for {
-			time.Sleep(time.Second)
-			if !cmn.FileExists(genDocFile) {
-				continue
+			if _, err := n.Start(); err != nil {
+				return fmt.Errorf("Failed to start node: %v", err)
+			} else {
+				logger.Info("Started node", "nodeInfo", n.Switch().NodeInfo())
 			}
-			jsonBlob, err := ioutil.ReadFile(genDocFile)
-			if err != nil {
-				return fmt.Errorf("Couldn't read GenesisDoc file: %v", err)
-			}
-			genDoc, err := types.GenesisDocFromJSON(jsonBlob)
-			if err != nil {
-				return fmt.Errorf("Error reading GenesisDoc: %v", err)
-			}
-			if genDoc.ChainID == "" {
-				return fmt.Errorf("Genesis doc %v must include non-empty chain_id", genDocFile)
-			}
-			config.ChainID = genDoc.ChainID
-		}
+
+			// Trap signal, run forever.
+			n.RunForever()
+
+			return nil
+		},
 	}
 
-	// Create & start node
-	n := node.NewNodeDefault(config, logger.With("module", "node"))
-	if _, err := n.Start(); err != nil {
-		return fmt.Errorf("Failed to start node: %v", err)
-	} else {
-		logger.Info("Started node", "nodeInfo", n.Switch().NodeInfo())
-	}
-
-	// Trap signal, run forever.
-	n.RunForever()
-
-	return nil
+	AddNodeFlags(cmd)
+	return cmd
 }

cmd/tendermint/commands/show_validator.go

@@ -9,18 +9,15 @@ import (
 	"github.com/tendermint/tendermint/types"
 )
 
-var showValidatorCmd = &cobra.Command{
+// ShowValidatorCmd adds capabilities for showing the validator info.
+var ShowValidatorCmd = &cobra.Command{
 	Use:   "show_validator",
 	Short: "Show this node's validator info",
 	Run:   showValidator,
 }
 
-func init() {
-	RootCmd.AddCommand(showValidatorCmd)
-}
-
 func showValidator(cmd *cobra.Command, args []string) {
-	privValidator := types.LoadOrGenPrivValidator(config.PrivValidatorFile(), logger)
+	privValidator := types.LoadOrGenPrivValidatorFS(config.PrivValidatorFile())
 	pubKeyJSONBytes, _ := data.ToJSON(privValidator.PubKey)
 	fmt.Println(string(pubKeyJSONBytes))
 }

cmd/tendermint/commands/testnet.go

@@ -7,16 +7,10 @@ import (
 
 	"github.com/spf13/cobra"
 
-	cmn "github.com/tendermint/tmlibs/common"
 	"github.com/tendermint/tendermint/types"
+	cmn "github.com/tendermint/tmlibs/common"
 )
 
-var testnetFilesCmd = &cobra.Command{
-	Use:   "testnet",
-	Short: "Initialize files for a Tendermint testnet",
-	Run:   testnetFiles,
-}
-
 //flags
 var (
 	nValidators int
@@ -24,12 +18,18 @@ var (
 )
 
 func init() {
-	testnetFilesCmd.Flags().IntVar(&nValidators, "n", 4,
+	TestnetFilesCmd.Flags().IntVar(&nValidators, "n", 4,
 		"Number of validators to initialize the testnet with")
-	testnetFilesCmd.Flags().StringVar(&dataDir, "dir", "mytestnet",
+	TestnetFilesCmd.Flags().StringVar(&dataDir, "dir", "mytestnet",
 		"Directory to store initialization data for the testnet")
+}
 
-	RootCmd.AddCommand(testnetFilesCmd)
+// TestnetFilesCmd allows initialisation of files for a
+// Tendermint testnet.
+var TestnetFilesCmd = &cobra.Command{
+	Use:   "testnet",
+	Short: "Initialize files for a Tendermint testnet",
+	Run:   testnetFiles,
 }
 
 func testnetFiles(cmd *cobra.Command, args []string) {
@@ -45,10 +45,10 @@ func testnetFiles(cmd *cobra.Command, args []string) {
 		}
 		// Read priv_validator.json to populate vals
 		privValFile := path.Join(dataDir, mach, "priv_validator.json")
-		privVal := types.LoadPrivValidator(privValFile)
+		privVal := types.LoadPrivValidatorFS(privValFile)
 		genVals[i] = types.GenesisValidator{
-			PubKey: privVal.PubKey,
-			Amount: 1,
+			PubKey: privVal.GetPubKey(),
+			Power:  1,
 			Name:   mach,
 		}
 	}
@@ -87,7 +87,6 @@ func ensurePrivValidator(file string) {
 	if cmn.FileExists(file) {
 		return
 	}
-	privValidator := types.GenPrivValidator()
-	privValidator.SetFile(file)
+	privValidator := types.GenPrivValidatorFS(file)
 	privValidator.Save()
 }

cmd/tendermint/commands/version.go

@@ -8,14 +8,11 @@ import (
 	"github.com/tendermint/tendermint/version"
 )
 
-var versionCmd = &cobra.Command{
+// VersionCmd ...
+var VersionCmd = &cobra.Command{
 	Use:   "version",
 	Short: "Show version info",
 	Run: func(cmd *cobra.Command, args []string) {
 		fmt.Println(version.Version)
 	},
 }
-
-func init() {
-	RootCmd.AddCommand(versionCmd)
-}

cmd/tendermint/main.go

@@ -3,11 +3,39 @@ package main
 import (
 	"os"
 
-	"github.com/tendermint/tendermint/cmd/tendermint/commands"
 	"github.com/tendermint/tmlibs/cli"
+
+	cmd "github.com/tendermint/tendermint/cmd/tendermint/commands"
+	nm "github.com/tendermint/tendermint/node"
 )
 
 func main() {
-	cmd := cli.PrepareBaseCmd(commands.RootCmd, "TM", os.ExpandEnv("$HOME/.tendermint"))
+	rootCmd := cmd.RootCmd
+	rootCmd.AddCommand(
+		cmd.GenValidatorCmd,
+		cmd.InitFilesCmd,
+		cmd.ProbeUpnpCmd,
+		cmd.ReplayCmd,
+		cmd.ReplayConsoleCmd,
+		cmd.ResetAllCmd,
+		cmd.ResetPrivValidatorCmd,
+		cmd.ShowValidatorCmd,
+		cmd.TestnetFilesCmd,
+		cmd.VersionCmd)
+
+	// NOTE:
+	// Users wishing to:
+	//	* Use an external signer for their validators
+	//	* Supply an in-proc abci app
+	//	* Supply a genesis doc file from another source
+	//	* Provide their own DB implementation
+	// can copy this file and use something other than the
+	// DefaultNewNode function
+	nodeFunc := nm.DefaultNewNode
+
+	// Create & start node
+	rootCmd.AddCommand(cmd.NewRunNodeCmd(nodeFunc))
+
+	cmd := cli.PrepareBaseCmd(rootCmd, "TM", os.ExpandEnv("$HOME/.tendermint"))
 	cmd.Execute()
 }

config/config.go

@@ -4,10 +4,9 @@ import (
 	"fmt"
 	"path/filepath"
 	"time"
-
-	"github.com/tendermint/tendermint/types"
 )
 
+// Config defines the top level configuration for a Tendermint node
 type Config struct {
 	// Top level options use an anonymous struct
 	BaseConfig `mapstructure:",squash"`
@@ -19,6 +18,7 @@ type Config struct {
 	Consensus *ConsensusConfig `mapstructure:"consensus"`
 }
 
+// DefaultConfig returns a default configuration for a Tendermint node
 func DefaultConfig() *Config {
 	return &Config{
 		BaseConfig: DefaultBaseConfig(),
@@ -29,6 +29,7 @@ func DefaultConfig() *Config {
 	}
 }
 
+// TestConfig returns a configuration that can be used for testing
 func TestConfig() *Config {
 	return &Config{
 		BaseConfig: TestBaseConfig(),
@@ -39,7 +40,7 @@ func TestConfig() *Config {
 	}
 }
 
-// Set the RootDir for all Config structs
+// SetRoot sets the RootDir for all Config structs
 func (cfg *Config) SetRoot(root string) *Config {
 	cfg.BaseConfig.RootDir = root
 	cfg.RPC.RootDir = root
@@ -52,7 +53,7 @@ func (cfg *Config) SetRoot(root string) *Config {
 //-----------------------------------------------------------------------------
 // BaseConfig
 
-// BaseConfig struct for a Tendermint node
+// BaseConfig defines the base configuration for a Tendermint node
 type BaseConfig struct {
 	// The root directory for all data.
 	// This should be set in viper so it can unmarshal into this struct
@@ -102,6 +103,7 @@ type BaseConfig struct {
 	DBPath string `mapstructure:"db_dir"`
 }
 
+// DefaultBaseConfig returns a default base configuration for a Tendermint node
 func DefaultBaseConfig() BaseConfig {
 	return BaseConfig{
 		Genesis:           "genesis.json",
@@ -119,6 +121,7 @@ func DefaultBaseConfig() BaseConfig {
 	}
 }
 
+// TestBaseConfig returns a base configuration for testing a Tendermint node
 func TestBaseConfig() BaseConfig {
 	conf := DefaultBaseConfig()
 	conf.ChainID = "tendermint_test"
@@ -128,22 +131,27 @@ func TestBaseConfig() BaseConfig {
 	return conf
 }
 
+// GenesisFile returns the full path to the genesis.json file
 func (b BaseConfig) GenesisFile() string {
 	return rootify(b.Genesis, b.RootDir)
 }
 
+// PrivValidatorFile returns the full path to the priv_validator.json file
 func (b BaseConfig) PrivValidatorFile() string {
 	return rootify(b.PrivValidator, b.RootDir)
 }
 
+// DBDir returns the full path to the database directory
 func (b BaseConfig) DBDir() string {
 	return rootify(b.DBPath, b.RootDir)
 }
 
+// DefaultLogLevel returns a default log level of "error"
 func DefaultLogLevel() string {
 	return "error"
 }
 
+// DefaultPackageLogLevels returns a default log level setting so all packages log at "error", while the `state` package logs at "info"
 func DefaultPackageLogLevels() string {
 	return fmt.Sprintf("state:info,*:%s", DefaultLogLevel())
 }
@@ -151,6 +159,7 @@ func DefaultPackageLogLevels() string {
 //-----------------------------------------------------------------------------
 // RPCConfig
 
+// RPCConfig defines the configuration options for the Tendermint RPC server
 type RPCConfig struct {
 	RootDir string `mapstructure:"home"`
 
@@ -165,6 +174,7 @@ type RPCConfig struct {
 	Unsafe bool `mapstructure:"unsafe"`
 }
 
+// DefaultRPCConfig returns a default configuration for the RPC server
 func DefaultRPCConfig() *RPCConfig {
 	return &RPCConfig{
 		ListenAddress:     "tcp://0.0.0.0:46657",
@@ -173,6 +183,7 @@ func DefaultRPCConfig() *RPCConfig {
 	}
 }
 
+// TestRPCConfig returns a configuration for testing the RPC server
 func TestRPCConfig() *RPCConfig {
 	conf := DefaultRPCConfig()
 	conf.ListenAddress = "tcp://0.0.0.0:36657"
@@ -184,26 +195,59 @@ func TestRPCConfig() *RPCConfig {
 //-----------------------------------------------------------------------------
 // P2PConfig
 
+// P2PConfig defines the configuration options for the Tendermint peer-to-peer networking layer
 type P2PConfig struct {
-	RootDir        string `mapstructure:"home"`
-	ListenAddress  string `mapstructure:"laddr"`
-	Seeds          string `mapstructure:"seeds"`
-	SkipUPNP       bool   `mapstructure:"skip_upnp"`
-	AddrBook       string `mapstructure:"addr_book_file"`
-	AddrBookStrict bool   `mapstructure:"addr_book_strict"`
-	PexReactor     bool   `mapstructure:"pex"`
-	MaxNumPeers    int    `mapstructure:"max_num_peers"`
+	RootDir string `mapstructure:"home"`
+
+	// Address to listen for incoming connections
+	ListenAddress string `mapstructure:"laddr"`
+
+	// Comma separated list of seed nodes to connect to
+	Seeds string `mapstructure:"seeds"`
+
+	// Skip UPNP port forwarding
+	SkipUPNP bool `mapstructure:"skip_upnp"`
+
+	// Path to address book
+	AddrBook string `mapstructure:"addr_book_file"`
+
+	// Set true for strict address routability rules
+	AddrBookStrict bool `mapstructure:"addr_book_strict"`
+
+	// Set true to enable the peer-exchange reactor
+	PexReactor bool `mapstructure:"pex"`
+
+	// Maximum number of peers to connect to
+	MaxNumPeers int `mapstructure:"max_num_peers"`
+
+	// Time to wait before flushing messages out on the connection, in ms
+	FlushThrottleTimeout int `mapstructure:"flush_throttle_timeout"`
+
+	// Maximum size of a message packet payload, in bytes
+	MaxMsgPacketPayloadSize int `mapstructure:"max_msg_packet_payload_size"`
+
+	// Rate at which packets can be sent, in bytes/second
+	SendRate int64 `mapstructure:"send_rate"`
+
+	// Rate at which packets can be received, in bytes/second
+	RecvRate int64 `mapstructure:"recv_rate"`
 }
 
+// DefaultP2PConfig returns a default configuration for the peer-to-peer layer
 func DefaultP2PConfig() *P2PConfig {
 	return &P2PConfig{
-		ListenAddress:  "tcp://0.0.0.0:46656",
-		AddrBook:       "addrbook.json",
-		AddrBookStrict: true,
-		MaxNumPeers:    50,
+		ListenAddress:           "tcp://0.0.0.0:46656",
+		AddrBook:                "addrbook.json",
+		AddrBookStrict:          true,
+		MaxNumPeers:             50,
+		FlushThrottleTimeout:    100,
+		MaxMsgPacketPayloadSize: 1024,   // 1 kB
+		SendRate:                512000, // 500 kB/s
+		RecvRate:                512000, // 500 kB/s
 	}
 }
 
+// TestP2PConfig returns a configuration for testing the peer-to-peer layer
 func TestP2PConfig() *P2PConfig {
 	conf := DefaultP2PConfig()
 	conf.ListenAddress = "tcp://0.0.0.0:36656"
@@ -211,6 +255,7 @@ func TestP2PConfig() *P2PConfig {
 	return conf
 }
 
+// AddrBookFile returns the full path to the address bool
 func (p *P2PConfig) AddrBookFile() string {
 	return rootify(p.AddrBook, p.RootDir)
 }
@@ -218,6 +263,7 @@ func (p *P2PConfig) AddrBookFile() string {
 //-----------------------------------------------------------------------------
 // MempoolConfig
 
+// MempoolConfig defines the configuration options for the Tendermint mempool
 type MempoolConfig struct {
 	RootDir      string `mapstructure:"home"`
 	Recheck      bool   `mapstructure:"recheck"`
@@ -226,6 +272,7 @@ type MempoolConfig struct {
 	WalPath      string `mapstructure:"wal_dir"`
 }
 
+// DefaultMempoolConfig returns a default configuration for the Tendermint mempool
 func DefaultMempoolConfig() *MempoolConfig {
 	return &MempoolConfig{
 		Recheck:      true,
@@ -235,6 +282,7 @@ func DefaultMempoolConfig() *MempoolConfig {
 	}
 }
 
+// WalDir returns the full path to the mempool's write-ahead log
 func (m *MempoolConfig) WalDir() string {
 	return rootify(m.WalPath, m.RootDir)
 }
@@ -242,8 +290,8 @@ func (m *MempoolConfig) WalDir() string {
 //-----------------------------------------------------------------------------
 // ConsensusConfig
 
-// ConsensusConfig holds timeouts and details about the WAL, the block structure,
-// and timeouts in the consensus protocol.
+// ConsensusConfig defines the confuguration for the Tendermint consensus service,
+// including timeouts and details about the WAL and the block structure.
 type ConsensusConfig struct {
 	RootDir  string `mapstructure:"home"`
 	WalPath  string `mapstructure:"wal_file"`
@@ -266,49 +314,78 @@ type ConsensusConfig struct {
 	MaxBlockSizeTxs   int `mapstructure:"max_block_size_txs"`
 	MaxBlockSizeBytes int `mapstructure:"max_block_size_bytes"`
 
-	// TODO: This probably shouldn't be exposed but it makes it
-	// easy to write tests for the wal/replay
-	BlockPartSize int `mapstructure:"block_part_size"`
+	// EmptyBlocks mode and possible interval between empty blocks in seconds
+	CreateEmptyBlocks         bool `mapstructure:"create_empty_blocks"`
+	CreateEmptyBlocksInterval int  `mapstructure:"create_empty_blocks_interval"`
+
+	// Reactor sleep duration parameters are in ms
+	PeerGossipSleepDuration     int `mapstructure:"peer_gossip_sleep_duration"`
+	PeerQueryMaj23SleepDuration int `mapstructure:"peer_query_maj23_sleep_duration"`
 }
 
-// Wait this long for a proposal
+// WaitForTxs returns true if the consensus should wait for transactions before entering the propose step
+func (cfg *ConsensusConfig) WaitForTxs() bool {
+	return !cfg.CreateEmptyBlocks || cfg.CreateEmptyBlocksInterval > 0
+}
+
+// EmptyBlocks returns the amount of time to wait before proposing an empty block or starting the propose timer if there are no txs available
+func (cfg *ConsensusConfig) EmptyBlocksInterval() time.Duration {
+	return time.Duration(cfg.CreateEmptyBlocksInterval) * time.Second
+}
+
+// Propose returns the amount of time to wait for a proposal
 func (cfg *ConsensusConfig) Propose(round int) time.Duration {
 	return time.Duration(cfg.TimeoutPropose+cfg.TimeoutProposeDelta*round) * time.Millisecond
 }
 
-// After receiving any +2/3 prevote, wait this long for stragglers
+// Prevote returns the amount of time to wait for straggler votes after receiving any +2/3 prevotes
 func (cfg *ConsensusConfig) Prevote(round int) time.Duration {
 	return time.Duration(cfg.TimeoutPrevote+cfg.TimeoutPrevoteDelta*round) * time.Millisecond
 }
 
-// After receiving any +2/3 precommits, wait this long for stragglers
+// Precommit returns the amount of time to wait for straggler votes after receiving any +2/3 precommits
 func (cfg *ConsensusConfig) Precommit(round int) time.Duration {
 	return time.Duration(cfg.TimeoutPrecommit+cfg.TimeoutPrecommitDelta*round) * time.Millisecond
 }
 
-// After receiving +2/3 precommits for a single block (a commit), wait this long for stragglers in the next height's RoundStepNewHeight
+// Commit returns the amount of time to wait for straggler votes after receiving +2/3 precommits for a single block (ie. a commit).
 func (cfg *ConsensusConfig) Commit(t time.Time) time.Time {
 	return t.Add(time.Duration(cfg.TimeoutCommit) * time.Millisecond)
 }
 
+// PeerGossipSleep returns the amount of time to sleep if there is nothing to send from the ConsensusReactor
+func (cfg *ConsensusConfig) PeerGossipSleep() time.Duration {
+	return time.Duration(cfg.PeerGossipSleepDuration) * time.Millisecond
+}
+
+// PeerQueryMaj23Sleep returns the amount of time to sleep after each VoteSetMaj23Message is sent in the ConsensusReactor
+func (cfg *ConsensusConfig) PeerQueryMaj23Sleep() time.Duration {
+	return time.Duration(cfg.PeerQueryMaj23SleepDuration) * time.Millisecond
+}
+
+// DefaultConsensusConfig returns a default configuration for the consensus service
 func DefaultConsensusConfig() *ConsensusConfig {
 	return &ConsensusConfig{
-		WalPath:               "data/cs.wal/wal",
-		WalLight:              false,
-		TimeoutPropose:        3000,
-		TimeoutProposeDelta:   500,
-		TimeoutPrevote:        1000,
-		TimeoutPrevoteDelta:   500,
-		TimeoutPrecommit:      1000,
-		TimeoutPrecommitDelta: 500,
-		TimeoutCommit:         1000,
-		SkipTimeoutCommit:     false,
-		MaxBlockSizeTxs:       10000,
-		MaxBlockSizeBytes:     1,                          // TODO
-		BlockPartSize:         types.DefaultBlockPartSize, // TODO: we shouldnt be importing types
+		WalPath:                     "data/cs.wal/wal",
+		WalLight:                    false,
+		TimeoutPropose:              3000,
+		TimeoutProposeDelta:         500,
+		TimeoutPrevote:              1000,
+		TimeoutPrevoteDelta:         500,
+		TimeoutPrecommit:            1000,
+		TimeoutPrecommitDelta:       500,
+		TimeoutCommit:               1000,
+		SkipTimeoutCommit:           false,
+		MaxBlockSizeTxs:             10000,
+		MaxBlockSizeBytes:           1, // TODO
+		CreateEmptyBlocks:           true,
+		CreateEmptyBlocksInterval:   0,
+		PeerGossipSleepDuration:     100,
+		PeerQueryMaj23SleepDuration: 2000,
 	}
 }
 
+// TestConsensusConfig returns a configuration for testing the consensus service
 func TestConsensusConfig() *ConsensusConfig {
 	config := DefaultConsensusConfig()
 	config.TimeoutPropose = 2000
@@ -322,6 +399,7 @@ func TestConsensusConfig() *ConsensusConfig {
 	return config
 }
 
+// WalFile returns the full path to the write-ahead log file
 func (c *ConsensusConfig) WalFile() string {
 	if c.walFile != "" {
 		return c.walFile
@@ -329,6 +407,7 @@ func (c *ConsensusConfig) WalFile() string {
 	return rootify(c.WalPath, c.RootDir)
 }
 
+// SetWalFile sets the path to the write-ahead log file
 func (c *ConsensusConfig) SetWalFile(walFile string) {
 	c.walFile = walFile
 }

config/toml.go

@@ -119,7 +119,7 @@ var testGenesis = `{
         "type": "ed25519",
         "data":"3B3069C422E19688B45CBFAE7BB009FC0FA1B1EA86593519318B7214853803C8"
       },
-      "amount": 10,
+      "power": 10,
       "name": ""
     }
   ],

consensus/byzantine_test.go

@@ -5,9 +5,11 @@ import (
 	"testing"
 	"time"
 
+	crypto "github.com/tendermint/go-crypto"
+	data "github.com/tendermint/go-wire/data"
 	"github.com/tendermint/tendermint/p2p"
 	"github.com/tendermint/tendermint/types"
-	. "github.com/tendermint/tmlibs/common"
+	cmn "github.com/tendermint/tmlibs/common"
 	"github.com/tendermint/tmlibs/events"
 )
 
@@ -53,7 +55,7 @@ func TestByzantine(t *testing.T) {
 	eventLogger := logger.With("module", "events")
 	for i := 0; i < N; i++ {
 		if i == 0 {
-			css[i].privValidator = NewByzantinePrivValidator(css[i].privValidator.(*types.PrivValidator))
+			css[i].privValidator = NewByzantinePrivValidator(css[i].privValidator)
 			// make byzantine
 			css[i].decideProposal = func(j int) func(int, int) {
 				return func(height, round int) {
@@ -77,6 +79,7 @@ func TestByzantine(t *testing.T) {
 
 		var conRI p2p.Reactor
 		conRI = conR
+
 		if i == 0 {
 			conRI = NewByzantineReactor(conR)
 		}
@@ -98,10 +101,10 @@ func TestByzantine(t *testing.T) {
 	// start the state machines
 	byzR := reactors[0].(*ByzantineReactor)
 	s := byzR.reactor.conS.GetState()
-	byzR.reactor.SwitchToConsensus(s)
+	byzR.reactor.SwitchToConsensus(s, 0)
 	for i := 1; i < N; i++ {
 		cr := reactors[i].(*ConsensusReactor)
-		cr.SwitchToConsensus(cr.conS.GetState())
+		cr.SwitchToConsensus(cr.conS.GetState(), 0)
 	}
 
 	// byz proposer sends one block to peers[0]
@@ -116,10 +119,7 @@ func TestByzantine(t *testing.T) {
 	p2p.Connect2Switches(switches, ind1, ind2)
 
 	// wait for someone in the big partition to make a block
-
-	select {
-	case <-eventChans[ind2]:
-	}
+	<-eventChans[ind2]
 
 	t.Log("A block has been committed. Healing partition")
 
@@ -149,8 +149,8 @@ func TestByzantine(t *testing.T) {
 	case <-done:
 	case <-tick.C:
 		for i, reactor := range reactors {
-			t.Log(Fmt("Consensus Reactor %v", i))
-			t.Log(Fmt("%v", reactor))
+			t.Log(cmn.Fmt("Consensus Reactor %v", i))
+			t.Log(cmn.Fmt("%v", reactor))
 		}
 		t.Fatalf("Timed out waiting for all validators to commit first block")
 	}
@@ -190,7 +190,7 @@ func byzantineDecideProposalFunc(t *testing.T, height, round int, cs *ConsensusS
 	}
 }
 
-func sendProposalAndParts(height, round int, cs *ConsensusState, peer *p2p.Peer, proposal *types.Proposal, blockHash []byte, parts *types.PartSet) {
+func sendProposalAndParts(height, round int, cs *ConsensusState, peer p2p.Peer, proposal *types.Proposal, blockHash []byte, parts *types.PartSet) {
 	// proposal
 	msg := &ProposalMessage{Proposal: proposal}
 	peer.Send(DataChannel, struct{ ConsensusMessage }{msg})
@@ -220,7 +220,7 @@ func sendProposalAndParts(height, round int, cs *ConsensusState, peer *p2p.Peer,
 // byzantine consensus reactor
 
 type ByzantineReactor struct {
-	Service
+	cmn.Service
 	reactor *ConsensusReactor
 }
 
@@ -233,14 +233,14 @@ func NewByzantineReactor(conR *ConsensusReactor) *ByzantineReactor {
 
 func (br *ByzantineReactor) SetSwitch(s *p2p.Switch)               { br.reactor.SetSwitch(s) }
 func (br *ByzantineReactor) GetChannels() []*p2p.ChannelDescriptor { return br.reactor.GetChannels() }
-func (br *ByzantineReactor) AddPeer(peer *p2p.Peer) {
+func (br *ByzantineReactor) AddPeer(peer p2p.Peer) {
 	if !br.reactor.IsRunning() {
 		return
 	}
 
 	// Create peerState for peer
-	peerState := NewPeerState(peer)
-	peer.Data.Set(types.PeerStateKey, peerState)
+	peerState := NewPeerState(peer).SetLogger(br.reactor.Logger)
+	peer.Set(types.PeerStateKey, peerState)
 
 	// Send our state to peer.
 	// If we're fast_syncing, broadcast a RoundStepMessage later upon SwitchToConsensus().
@@ -248,10 +248,10 @@ func (br *ByzantineReactor) AddPeer(peer *p2p.Peer) {
 		br.reactor.sendNewRoundStepMessages(peer)
 	}
 }
-func (br *ByzantineReactor) RemovePeer(peer *p2p.Peer, reason interface{}) {
+func (br *ByzantineReactor) RemovePeer(peer p2p.Peer, reason interface{}) {
 	br.reactor.RemovePeer(peer, reason)
 }
-func (br *ByzantineReactor) Receive(chID byte, peer *p2p.Peer, msgBytes []byte) {
+func (br *ByzantineReactor) Receive(chID byte, peer p2p.Peer, msgBytes []byte) {
 	br.reactor.Receive(chID, peer, msgBytes)
 }
 
@@ -259,42 +259,42 @@ func (br *ByzantineReactor) Receive(chID byte, peer *p2p.Peer, msgBytes []byte)
 // byzantine privValidator
 
 type ByzantinePrivValidator struct {
-	Address      []byte `json:"address"`
-	types.Signer `json:"-"`
+	types.Signer
 
-	mtx sync.Mutex
+	pv types.PrivValidator
 }
 
 // Return a priv validator that will sign anything
-func NewByzantinePrivValidator(pv *types.PrivValidator) *ByzantinePrivValidator {
+func NewByzantinePrivValidator(pv types.PrivValidator) *ByzantinePrivValidator {
 	return &ByzantinePrivValidator{
-		Address: pv.Address,
-		Signer:  pv.Signer,
+		Signer: pv.(*types.PrivValidatorFS).Signer,
+		pv:     pv,
 	}
 }
 
-func (privVal *ByzantinePrivValidator) GetAddress() []byte {
-	return privVal.Address
+func (privVal *ByzantinePrivValidator) GetAddress() data.Bytes {
+	return privVal.pv.GetAddress()
 }
 
-func (privVal *ByzantinePrivValidator) SignVote(chainID string, vote *types.Vote) error {
-	privVal.mtx.Lock()
-	defer privVal.mtx.Unlock()
+func (privVal *ByzantinePrivValidator) GetPubKey() crypto.PubKey {
+	return privVal.pv.GetPubKey()
+}
 
-	// Sign
-	vote.Signature = privVal.Sign(types.SignBytes(chainID, vote))
+func (privVal *ByzantinePrivValidator) SignVote(chainID string, vote *types.Vote) (err error) {
+	vote.Signature, err = privVal.Sign(types.SignBytes(chainID, vote))
+	return err
+}
+
+func (privVal *ByzantinePrivValidator) SignProposal(chainID string, proposal *types.Proposal) (err error) {
+	proposal.Signature, err = privVal.Sign(types.SignBytes(chainID, proposal))
 	return nil
 }
 
-func (privVal *ByzantinePrivValidator) SignProposal(chainID string, proposal *types.Proposal) error {
-	privVal.mtx.Lock()
-	defer privVal.mtx.Unlock()
-
-	// Sign
-	proposal.Signature = privVal.Sign(types.SignBytes(chainID, proposal))
+func (privVal *ByzantinePrivValidator) SignHeartbeat(chainID string, heartbeat *types.Heartbeat) (err error) {
+	heartbeat.Signature, err = privVal.Sign(types.SignBytes(chainID, heartbeat))
 	return nil
 }
 
 func (privVal *ByzantinePrivValidator) String() string {
-	return Fmt("PrivValidator{%X}", privVal.Address)
+	return cmn.Fmt("PrivValidator{%X}", privVal.GetAddress())
 }

consensus/common.go

@@ -27,3 +27,9 @@ func subscribeToEventRespond(evsw types.EventSwitch, receiver, eventID string) c
 	})
 	return ch
 }
+
+func discardFromChan(ch chan interface{}, n int) {
+	for i := 0; i < n; i++ {
+		<-ch
+	}
+}

consensus/common_test.go

@@ -15,11 +15,12 @@ import (
 	abci "github.com/tendermint/abci/types"
 	bc "github.com/tendermint/tendermint/blockchain"
 	cfg "github.com/tendermint/tendermint/config"
+	cstypes "github.com/tendermint/tendermint/consensus/types"
 	mempl "github.com/tendermint/tendermint/mempool"
 	"github.com/tendermint/tendermint/p2p"
 	sm "github.com/tendermint/tendermint/state"
 	"github.com/tendermint/tendermint/types"
-	. "github.com/tendermint/tmlibs/common"
+	cmn "github.com/tendermint/tmlibs/common"
 	dbm "github.com/tendermint/tmlibs/db"
 	"github.com/tendermint/tmlibs/log"
 
@@ -31,10 +32,10 @@ import (
 
 // genesis, chain_id, priv_val
 var config *cfg.Config // NOTE: must be reset for each _test.go file
-var ensureTimeout = time.Duration(2)
+var ensureTimeout = time.Second * 2
 
 func ensureDir(dir string, mode os.FileMode) {
-	if err := EnsureDir(dir, mode); err != nil {
+	if err := cmn.EnsureDir(dir, mode); err != nil {
 		panic(err)
 	}
 }
@@ -50,12 +51,12 @@ type validatorStub struct {
 	Index  int // Validator index. NOTE: we don't assume validator set changes.
 	Height int
 	Round  int
-	*types.PrivValidator
+	types.PrivValidator
 }
 
 var testMinPower = 10
 
-func NewValidatorStub(privValidator *types.PrivValidator, valIndex int) *validatorStub {
+func NewValidatorStub(privValidator types.PrivValidator, valIndex int) *validatorStub {
 	return &validatorStub{
 		Index:         valIndex,
 		PrivValidator: privValidator,
@@ -65,7 +66,7 @@ func NewValidatorStub(privValidator *types.PrivValidator, valIndex int) *validat
 func (vs *validatorStub) signVote(voteType byte, hash []byte, header types.PartSetHeader) (*types.Vote, error) {
 	vote := &types.Vote{
 		ValidatorIndex:   vs.Index,
-		ValidatorAddress: vs.PrivValidator.Address,
+		ValidatorAddress: vs.PrivValidator.GetAddress(),
 		Height:           vs.Height,
 		Round:            vs.Round,
 		Type:             voteType,
@@ -142,7 +143,7 @@ func signAddVotes(to *ConsensusState, voteType byte, hash []byte, header types.P
 func validatePrevote(t *testing.T, cs *ConsensusState, round int, privVal *validatorStub, blockHash []byte) {
 	prevotes := cs.Votes.Prevotes(round)
 	var vote *types.Vote
-	if vote = prevotes.GetByAddress(privVal.Address); vote == nil {
+	if vote = prevotes.GetByAddress(privVal.GetAddress()); vote == nil {
 		panic("Failed to find prevote from validator")
 	}
 	if blockHash == nil {
@@ -159,7 +160,7 @@ func validatePrevote(t *testing.T, cs *ConsensusState, round int, privVal *valid
 func validateLastPrecommit(t *testing.T, cs *ConsensusState, privVal *validatorStub, blockHash []byte) {
 	votes := cs.LastCommit
 	var vote *types.Vote
-	if vote = votes.GetByAddress(privVal.Address); vote == nil {
+	if vote = votes.GetByAddress(privVal.GetAddress()); vote == nil {
 		panic("Failed to find precommit from validator")
 	}
 	if !bytes.Equal(vote.BlockID.Hash, blockHash) {
@@ -170,7 +171,7 @@ func validateLastPrecommit(t *testing.T, cs *ConsensusState, privVal *validatorS
 func validatePrecommit(t *testing.T, cs *ConsensusState, thisRound, lockRound int, privVal *validatorStub, votedBlockHash, lockedBlockHash []byte) {
 	precommits := cs.Votes.Precommits(thisRound)
 	var vote *types.Vote
-	if vote = precommits.GetByAddress(privVal.Address); vote == nil {
+	if vote = precommits.GetByAddress(privVal.GetAddress()); vote == nil {
 		panic("Failed to find precommit from validator")
 	}
 
@@ -222,25 +223,14 @@ func subscribeToVoter(cs *ConsensusState, addr []byte) chan interface{} {
 	return voteCh
 }
 
-func readVotes(ch chan interface{}, reads int) chan struct{} {
-	wg := make(chan struct{})
-	go func() {
-		for i := 0; i < reads; i++ {
-			<-ch // read the precommit event
-		}
-		close(wg)
-	}()
-	return wg
-}
-
 //-------------------------------------------------------------------------------
 // consensus states
 
-func newConsensusState(state *sm.State, pv *types.PrivValidator, app abci.Application) *ConsensusState {
+func newConsensusState(state *sm.State, pv types.PrivValidator, app abci.Application) *ConsensusState {
 	return newConsensusStateWithConfig(config, state, pv, app)
 }
 
-func newConsensusStateWithConfig(thisConfig *cfg.Config, state *sm.State, pv *types.PrivValidator, app abci.Application) *ConsensusState {
+func newConsensusStateWithConfig(thisConfig *cfg.Config, state *sm.State, pv types.PrivValidator, app abci.Application) *ConsensusState {
 	// Get BlockStore
 	blockDB := dbm.NewMemDB()
 	blockStore := bc.NewBlockStore(blockDB)
@@ -251,8 +241,11 @@ func newConsensusStateWithConfig(thisConfig *cfg.Config, state *sm.State, pv *ty
 	proxyAppConnCon := abcicli.NewLocalClient(mtx, app)
 
 	// Make Mempool
-	mempool := mempl.NewMempool(thisConfig.Mempool, proxyAppConnMem)
+	mempool := mempl.NewMempool(thisConfig.Mempool, proxyAppConnMem, 0)
 	mempool.SetLogger(log.TestingLogger().With("module", "mempool"))
+	if thisConfig.Consensus.WaitForTxs() {
+		mempool.EnableTxsAvailable()
+	}
 
 	// Make ConsensusReactor
 	cs := NewConsensusState(thisConfig.Consensus, state, proxyAppConnCon, blockStore, mempool)
@@ -266,27 +259,17 @@ func newConsensusStateWithConfig(thisConfig *cfg.Config, state *sm.State, pv *ty
 	return cs
 }
 
-func loadPrivValidator(config *cfg.Config) *types.PrivValidator {
+func loadPrivValidator(config *cfg.Config) *types.PrivValidatorFS {
 	privValidatorFile := config.PrivValidatorFile()
 	ensureDir(path.Dir(privValidatorFile), 0700)
-	privValidator := types.LoadOrGenPrivValidator(privValidatorFile, log.TestingLogger())
+	privValidator := types.LoadOrGenPrivValidatorFS(privValidatorFile)
 	privValidator.Reset()
 	return privValidator
 }
 
-func fixedConsensusState() *ConsensusState {
-	stateDB := dbm.NewMemDB()
-	state := sm.MakeGenesisStateFromFile(stateDB, config.GenesisFile())
-	state.SetLogger(log.TestingLogger().With("module", "state"))
-	privValidator := loadPrivValidator(config)
-	cs := newConsensusState(state, privValidator, counter.NewCounterApplication(true))
-	cs.SetLogger(log.TestingLogger())
-	return cs
-}
-
 func fixedConsensusStateDummy() *ConsensusState {
 	stateDB := dbm.NewMemDB()
-	state := sm.MakeGenesisStateFromFile(stateDB, config.GenesisFile())
+	state, _ := sm.MakeGenesisStateFromFile(stateDB, config.GenesisFile())
 	state.SetLogger(log.TestingLogger().With("module", "state"))
 	privValidator := loadPrivValidator(config)
 	cs := newConsensusState(state, privValidator, dummy.NewDummyApplication())
@@ -315,12 +298,22 @@ func randConsensusState(nValidators int) (*ConsensusState, []*validatorStub) {
 //-------------------------------------------------------------------------------
 
 func ensureNoNewStep(stepCh chan interface{}) {
-	timeout := time.NewTicker(ensureTimeout * time.Second)
+	timer := time.NewTimer(ensureTimeout)
 	select {
-	case <-timeout.C:
+	case <-timer.C:
 		break
 	case <-stepCh:
-		panic("We should be stuck waiting for more votes, not moving to the next step")
+		panic("We should be stuck waiting, not moving to the next step")
+	}
+}
+
+func ensureNewStep(stepCh chan interface{}) {
+	timer := time.NewTimer(ensureTimeout)
+	select {
+	case <-timer.C:
+		panic("We shouldnt be stuck waiting")
+	case <-stepCh:
+		break
 	}
 }
 
@@ -340,18 +333,25 @@ func consensusLogger() log.Logger {
 	})
 }
 
-func randConsensusNet(nValidators int, testName string, tickerFunc func() TimeoutTicker, appFunc func() abci.Application) []*ConsensusState {
+func randConsensusNet(nValidators int, testName string, tickerFunc func() TimeoutTicker, appFunc func() abci.Application, configOpts ...func(*cfg.Config)) []*ConsensusState {
 	genDoc, privVals := randGenesisDoc(nValidators, false, 10)
 	css := make([]*ConsensusState, nValidators)
 	logger := consensusLogger()
 	for i := 0; i < nValidators; i++ {
 		db := dbm.NewMemDB() // each state needs its own db
-		state := sm.MakeGenesisState(db, genDoc)
+		state, _ := sm.MakeGenesisState(db, genDoc)
 		state.SetLogger(logger.With("module", "state", "validator", i))
 		state.Save()
-		thisConfig := ResetConfig(Fmt("%s_%d", testName, i))
+		thisConfig := ResetConfig(cmn.Fmt("%s_%d", testName, i))
+		for _, opt := range configOpts {
+			opt(thisConfig)
+		}
 		ensureDir(path.Dir(thisConfig.Consensus.WalFile()), 0700) // dir for wal
-		css[i] = newConsensusStateWithConfig(thisConfig, state, privVals[i], appFunc())
+		app := appFunc()
+		vals := types.TM2PB.Validators(state.Validators)
+		app.InitChain(abci.RequestInitChain{Validators: vals})
+
+		css[i] = newConsensusStateWithConfig(thisConfig, state, privVals[i], app)
 		css[i].SetLogger(logger.With("validator", i))
 		css[i].SetTimeoutTicker(tickerFunc())
 	}
@@ -364,30 +364,33 @@ func randConsensusNetWithPeers(nValidators, nPeers int, testName string, tickerF
 	css := make([]*ConsensusState, nPeers)
 	for i := 0; i < nPeers; i++ {
 		db := dbm.NewMemDB() // each state needs its own db
-		state := sm.MakeGenesisState(db, genDoc)
+		state, _ := sm.MakeGenesisState(db, genDoc)
 		state.SetLogger(log.TestingLogger().With("module", "state"))
 		state.Save()
-		thisConfig := ResetConfig(Fmt("%s_%d", testName, i))
+		thisConfig := ResetConfig(cmn.Fmt("%s_%d", testName, i))
 		ensureDir(path.Dir(thisConfig.Consensus.WalFile()), 0700) // dir for wal
-		var privVal *types.PrivValidator
+		var privVal types.PrivValidator
 		if i < nValidators {
 			privVal = privVals[i]
 		} else {
-			privVal = types.GenPrivValidator()
-			_, tempFilePath := Tempfile("priv_validator_")
-			privVal.SetFile(tempFilePath)
+			_, tempFilePath := cmn.Tempfile("priv_validator_")
+			privVal = types.GenPrivValidatorFS(tempFilePath)
 		}
 
-		css[i] = newConsensusStateWithConfig(thisConfig, state, privVal, appFunc())
+		app := appFunc()
+		vals := types.TM2PB.Validators(state.Validators)
+		app.InitChain(abci.RequestInitChain{Validators: vals})
+
+		css[i] = newConsensusStateWithConfig(thisConfig, state, privVal, app)
 		css[i].SetLogger(log.TestingLogger())
 		css[i].SetTimeoutTicker(tickerFunc())
 	}
 	return css
 }
 
-func getSwitchIndex(switches []*p2p.Switch, peer *p2p.Peer) int {
+func getSwitchIndex(switches []*p2p.Switch, peer p2p.Peer) int {
 	for i, s := range switches {
-		if bytes.Equal(peer.NodeInfo.PubKey.Address(), s.NodeInfo().PubKey.Address()) {
+		if bytes.Equal(peer.NodeInfo().PubKey.Address(), s.NodeInfo().PubKey.Address()) {
 			return i
 		}
 	}
@@ -398,14 +401,14 @@ func getSwitchIndex(switches []*p2p.Switch, peer *p2p.Peer) int {
 //-------------------------------------------------------------------------------
 // genesis
 
-func randGenesisDoc(numValidators int, randPower bool, minPower int64) (*types.GenesisDoc, []*types.PrivValidator) {
+func randGenesisDoc(numValidators int, randPower bool, minPower int64) (*types.GenesisDoc, []*types.PrivValidatorFS) {
 	validators := make([]types.GenesisValidator, numValidators)
-	privValidators := make([]*types.PrivValidator, numValidators)
+	privValidators := make([]*types.PrivValidatorFS, numValidators)
 	for i := 0; i < numValidators; i++ {
 		val, privVal := types.RandValidator(randPower, minPower)
 		validators[i] = types.GenesisValidator{
 			PubKey: val.PubKey,
-			Amount: val.VotingPower,
+			Power:  val.VotingPower,
 		}
 		privValidators[i] = privVal
 	}
@@ -417,10 +420,10 @@ func randGenesisDoc(numValidators int, randPower bool, minPower int64) (*types.G
 	}, privValidators
 }
 
-func randGenesisState(numValidators int, randPower bool, minPower int64) (*sm.State, []*types.PrivValidator) {
+func randGenesisState(numValidators int, randPower bool, minPower int64) (*sm.State, []*types.PrivValidatorFS) {
 	genDoc, privValidators := randGenesisDoc(numValidators, randPower, minPower)
 	db := dbm.NewMemDB()
-	s0 := sm.MakeGenesisState(db, genDoc)
+	s0, _ := sm.MakeGenesisState(db, genDoc)
 	s0.SetLogger(log.TestingLogger().With("module", "state"))
 	s0.Save()
 	return s0, privValidators
@@ -462,7 +465,7 @@ func (m *mockTicker) ScheduleTimeout(ti timeoutInfo) {
 	if m.onlyOnce && m.fired {
 		return
 	}
-	if ti.Step == RoundStepNewHeight {
+	if ti.Step == cstypes.RoundStepNewHeight {
 		m.c <- ti
 		m.fired = true
 	}

consensus/mempool_test.go

@@ -8,13 +8,89 @@ import (
 	abci "github.com/tendermint/abci/types"
 	"github.com/tendermint/tendermint/types"
 
-	. "github.com/tendermint/tmlibs/common"
+	cmn "github.com/tendermint/tmlibs/common"
 )
 
 func init() {
 	config = ResetConfig("consensus_mempool_test")
 }
 
+func TestNoProgressUntilTxsAvailable(t *testing.T) {
+	config := ResetConfig("consensus_mempool_txs_available_test")
+	config.Consensus.CreateEmptyBlocks = false
+	state, privVals := randGenesisState(1, false, 10)
+	cs := newConsensusStateWithConfig(config, state, privVals[0], NewCounterApplication())
+	cs.mempool.EnableTxsAvailable()
+	height, round := cs.Height, cs.Round
+	newBlockCh := subscribeToEvent(cs.evsw, "tester", types.EventStringNewBlock(), 1)
+	startTestRound(cs, height, round)
+
+	ensureNewStep(newBlockCh) // first block gets committed
+	ensureNoNewStep(newBlockCh)
+	deliverTxsRange(cs, 0, 2)
+	ensureNewStep(newBlockCh) // commit txs
+	ensureNewStep(newBlockCh) // commit updated app hash
+	ensureNoNewStep(newBlockCh)
+
+}
+
+func TestProgressAfterCreateEmptyBlocksInterval(t *testing.T) {
+	config := ResetConfig("consensus_mempool_txs_available_test")
+	config.Consensus.CreateEmptyBlocksInterval = int(ensureTimeout.Seconds())
+	state, privVals := randGenesisState(1, false, 10)
+	cs := newConsensusStateWithConfig(config, state, privVals[0], NewCounterApplication())
+	cs.mempool.EnableTxsAvailable()
+	height, round := cs.Height, cs.Round
+	newBlockCh := subscribeToEvent(cs.evsw, "tester", types.EventStringNewBlock(), 1)
+	startTestRound(cs, height, round)
+
+	ensureNewStep(newBlockCh)   // first block gets committed
+	ensureNoNewStep(newBlockCh) // then we dont make a block ...
+	ensureNewStep(newBlockCh)   // until the CreateEmptyBlocksInterval has passed
+}
+
+func TestProgressInHigherRound(t *testing.T) {
+	config := ResetConfig("consensus_mempool_txs_available_test")
+	config.Consensus.CreateEmptyBlocks = false
+	state, privVals := randGenesisState(1, false, 10)
+	cs := newConsensusStateWithConfig(config, state, privVals[0], NewCounterApplication())
+	cs.mempool.EnableTxsAvailable()
+	height, round := cs.Height, cs.Round
+	newBlockCh := subscribeToEvent(cs.evsw, "tester", types.EventStringNewBlock(), 1)
+	newRoundCh := subscribeToEvent(cs.evsw, "tester", types.EventStringNewRound(), 1)
+	timeoutCh := subscribeToEvent(cs.evsw, "tester", types.EventStringTimeoutPropose(), 1)
+	cs.setProposal = func(proposal *types.Proposal) error {
+		if cs.Height == 2 && cs.Round == 0 {
+			// dont set the proposal in round 0 so we timeout and
+			// go to next round
+			cs.Logger.Info("Ignoring set proposal at height 2, round 0")
+			return nil
+		}
+		return cs.defaultSetProposal(proposal)
+	}
+	startTestRound(cs, height, round)
+
+	ensureNewStep(newRoundCh) // first round at first height
+	ensureNewStep(newBlockCh) // first block gets committed
+	ensureNewStep(newRoundCh) // first round at next height
+	deliverTxsRange(cs, 0, 2) // we deliver txs, but dont set a proposal so we get the next round
+	<-timeoutCh
+	ensureNewStep(newRoundCh) // wait for the next round
+	ensureNewStep(newBlockCh) // now we can commit the block
+}
+
+func deliverTxsRange(cs *ConsensusState, start, end int) {
+	// Deliver some txs.
+	for i := start; i < end; i++ {
+		txBytes := make([]byte, 8)
+		binary.BigEndian.PutUint64(txBytes, uint64(i))
+		err := cs.mempool.CheckTx(txBytes, nil)
+		if err != nil {
+			panic(cmn.Fmt("Error after CheckTx: %v", err))
+		}
+	}
+}
+
 func TestTxConcurrentWithCommit(t *testing.T) {
 
 	state, privVals := randGenesisState(1, false, 10)
@@ -22,21 +98,8 @@ func TestTxConcurrentWithCommit(t *testing.T) {
 	height, round := cs.Height, cs.Round
 	newBlockCh := subscribeToEvent(cs.evsw, "tester", types.EventStringNewBlock(), 1)
 
-	deliverTxsRange := func(start, end int) {
-		// Deliver some txs.
-		for i := start; i < end; i++ {
-			txBytes := make([]byte, 8)
-			binary.BigEndian.PutUint64(txBytes, uint64(i))
-			err := cs.mempool.CheckTx(txBytes, nil)
-			if err != nil {
-				panic(Fmt("Error after CheckTx: %v", err))
-			}
-			//	time.Sleep(time.Microsecond * time.Duration(rand.Int63n(3000)))
-		}
-	}
-
 	NTxs := 10000
-	go deliverTxsRange(0, NTxs)
+	go deliverTxsRange(cs, 0, NTxs)
 
 	startTestRound(cs, height, round)
 	ticker := time.NewTicker(time.Second * 20)
@@ -120,8 +183,8 @@ func NewCounterApplication() *CounterApplication {
 	return &CounterApplication{}
 }
 
-func (app *CounterApplication) Info() abci.ResponseInfo {
-	return abci.ResponseInfo{Data: Fmt("txs:%v", app.txCount)}
+func (app *CounterApplication) Info(req abci.RequestInfo) abci.ResponseInfo {
+	return abci.ResponseInfo{Data: cmn.Fmt("txs:%v", app.txCount)}
 }
 
 func (app *CounterApplication) DeliverTx(tx []byte) abci.Result {
@@ -138,7 +201,7 @@ func runTx(tx []byte, countPtr *int) abci.Result {
 	copy(tx8[len(tx8)-len(tx):], tx)
 	txValue := binary.BigEndian.Uint64(tx8)
 	if txValue != uint64(count) {
-		return abci.ErrBadNonce.AppendLog(Fmt("Invalid nonce. Expected %v, got %v", count, txValue))
+		return abci.ErrBadNonce.AppendLog(cmn.Fmt("Invalid nonce. Expected %v, got %v", count, txValue))
 	}
 	*countPtr += 1
 	return abci.OK

consensus/reactor_test.go

@@ -7,9 +7,11 @@ import (
 	"time"
 
 	"github.com/tendermint/abci/example/dummy"
+	"github.com/tendermint/tmlibs/events"
+
+	cfg "github.com/tendermint/tendermint/config"
 	"github.com/tendermint/tendermint/p2p"
 	"github.com/tendermint/tendermint/types"
-	"github.com/tendermint/tmlibs/events"
 )
 
 func init() {
@@ -52,7 +54,7 @@ func startConsensusNet(t *testing.T, css []*ConsensusState, N int, subscribeEven
 	// we'd block when the cs fires NewBlockEvent and the peers are trying to start their reactors
 	for i := 0; i < N; i++ {
 		s := reactors[i].conS.GetState()
-		reactors[i].SwitchToConsensus(s)
+		reactors[i].SwitchToConsensus(s, 0)
 	}
 	return reactors, eventChans
 }
@@ -76,6 +78,35 @@ func TestReactor(t *testing.T) {
 	}, css)
 }
 
+// Ensure a testnet sends proposal heartbeats and makes blocks when there are txs
+func TestReactorProposalHeartbeats(t *testing.T) {
+	N := 4
+	css := randConsensusNet(N, "consensus_reactor_test", newMockTickerFunc(true), newCounter,
+		func(c *cfg.Config) {
+			c.Consensus.CreateEmptyBlocks = false
+		})
+	reactors, eventChans := startConsensusNet(t, css, N, false)
+	defer stopConsensusNet(reactors)
+	heartbeatChans := make([]chan interface{}, N)
+	for i := 0; i < N; i++ {
+		heartbeatChans[i] = subscribeToEvent(css[i].evsw, "tester", types.EventStringProposalHeartbeat(), 1)
+	}
+	// wait till everyone sends a proposal heartbeat
+	timeoutWaitGroup(t, N, func(wg *sync.WaitGroup, j int) {
+		<-heartbeatChans[j]
+		wg.Done()
+	}, css)
+
+	// send a tx
+	css[3].mempool.CheckTx([]byte{1, 2, 3}, nil)
+
+	// wait till everyone makes the first new block
+	timeoutWaitGroup(t, N, func(wg *sync.WaitGroup, j int) {
+		<-eventChans[j]
+		wg.Done()
+	}, css)
+}
+
 //-------------------------------------------------------------
 // ensure we can make blocks despite cycling a validator set
 
@@ -101,7 +132,7 @@ func TestVotingPowerChange(t *testing.T) {
 	//---------------------------------------------------------------------------
 	t.Log("---------------------------- Testing changing the voting power of one validator a few times")
 
-	val1PubKey := css[0].privValidator.(*types.PrivValidator).PubKey
+	val1PubKey := css[0].privValidator.GetPubKey()
 	updateValidatorTx := dummy.MakeValSetChangeTx(val1PubKey.Bytes(), 25)
 	previousTotalVotingPower := css[0].GetRoundState().LastValidators.TotalVotingPower()
 
@@ -162,7 +193,7 @@ func TestValidatorSetChanges(t *testing.T) {
 	//---------------------------------------------------------------------------
 	t.Log("---------------------------- Testing adding one validator")
 
-	newValidatorPubKey1 := css[nVals].privValidator.(*types.PrivValidator).PubKey
+	newValidatorPubKey1 := css[nVals].privValidator.GetPubKey()
 	newValidatorTx1 := dummy.MakeValSetChangeTx(newValidatorPubKey1.Bytes(), uint64(testMinPower))
 
 	// wait till everyone makes block 2
@@ -188,7 +219,7 @@ func TestValidatorSetChanges(t *testing.T) {
 	//---------------------------------------------------------------------------
 	t.Log("---------------------------- Testing changing the voting power of one validator")
 
-	updateValidatorPubKey1 := css[nVals].privValidator.(*types.PrivValidator).PubKey
+	updateValidatorPubKey1 := css[nVals].privValidator.GetPubKey()
 	updateValidatorTx1 := dummy.MakeValSetChangeTx(updateValidatorPubKey1.Bytes(), 25)
 	previousTotalVotingPower := css[nVals].GetRoundState().LastValidators.TotalVotingPower()
 
@@ -204,10 +235,10 @@ func TestValidatorSetChanges(t *testing.T) {
 	//---------------------------------------------------------------------------
 	t.Log("---------------------------- Testing adding two validators at once")
 
-	newValidatorPubKey2 := css[nVals+1].privValidator.(*types.PrivValidator).PubKey
+	newValidatorPubKey2 := css[nVals+1].privValidator.GetPubKey()
 	newValidatorTx2 := dummy.MakeValSetChangeTx(newValidatorPubKey2.Bytes(), uint64(testMinPower))
 
-	newValidatorPubKey3 := css[nVals+2].privValidator.(*types.PrivValidator).PubKey
+	newValidatorPubKey3 := css[nVals+2].privValidator.GetPubKey()
 	newValidatorTx3 := dummy.MakeValSetChangeTx(newValidatorPubKey3.Bytes(), uint64(testMinPower))
 
 	waitForAndValidateBlock(t, nPeers, activeVals, eventChans, css, newValidatorTx2, newValidatorTx3)

consensus/replay.go

@@ -4,6 +4,7 @@ import (
 	"bytes"
 	"errors"
 	"fmt"
+	"hash/crc32"
 	"io"
 	"reflect"
 	"strconv"
@@ -11,7 +12,6 @@ import (
 	"time"
 
 	abci "github.com/tendermint/abci/types"
-	wire "github.com/tendermint/go-wire"
 	auto "github.com/tendermint/tmlibs/autofile"
 	cmn "github.com/tendermint/tmlibs/common"
 	"github.com/tendermint/tmlibs/log"
@@ -19,8 +19,11 @@ import (
 	"github.com/tendermint/tendermint/proxy"
 	sm "github.com/tendermint/tendermint/state"
 	"github.com/tendermint/tendermint/types"
+	"github.com/tendermint/tendermint/version"
 )
 
+var crc32c = crc32.MakeTable(crc32.Castagnoli)
+
 // Functionality to replay blocks and messages on recovery from a crash.
 // There are two general failure scenarios: failure during consensus, and failure while applying the block.
 // The former is handled by the WAL, the latter by the proxyApp Handshake on restart,
@@ -34,18 +37,11 @@ import (
 // as if it were received in receiveRoutine
 // Lines that start with "#" are ignored.
 // NOTE: receiveRoutine should not be running
-func (cs *ConsensusState) readReplayMessage(msgBytes []byte, newStepCh chan interface{}) error {
-	// Skip over empty and meta lines
-	if len(msgBytes) == 0 || msgBytes[0] == '#' {
+func (cs *ConsensusState) readReplayMessage(msg *TimedWALMessage, newStepCh chan interface{}) error {
+	// skip meta messages
+	if _, ok := msg.Msg.(EndHeightMessage); ok {
 		return nil
 	}
-	var err error
-	var msg TimedWALMessage
-	wire.ReadJSON(&msg, msgBytes, &err)
-	if err != nil {
-		fmt.Println("MsgBytes:", msgBytes, string(msgBytes))
-		return fmt.Errorf("Error reading json data: %v", err)
-	}
 
 	// for logging
 	switch m := msg.Msg.(type) {
@@ -82,7 +78,7 @@ func (cs *ConsensusState) readReplayMessage(msgBytes []byte, newStepCh chan inte
 				"blockID", v.BlockID, "peer", peerKey)
 		}
 
-		cs.handleMsg(m, cs.RoundState)
+		cs.handleMsg(m)
 	case timeoutInfo:
 		cs.Logger.Info("Replay: Timeout", "height", m.Height, "round", m.Round, "step", m.Step, "dur", m.Duration)
 		cs.handleTimeout(m, cs.RoundState)
@@ -103,7 +99,7 @@ func (cs *ConsensusState) catchupReplay(csHeight int) error {
 	// Ensure that ENDHEIGHT for this height doesn't exist
 	// NOTE: This is just a sanity check. As far as we know things work fine without it,
 	// and Handshake could reuse ConsensusState if it weren't for this check (since we can crash after writing ENDHEIGHT).
-	gr, found, err := cs.wal.group.Search("#ENDHEIGHT: ", makeHeightSearchFunc(csHeight))
+	gr, found, err := cs.wal.SearchForEndHeight(uint64(csHeight))
 	if gr != nil {
 		gr.Close()
 	}
@@ -112,55 +108,33 @@ func (cs *ConsensusState) catchupReplay(csHeight int) error {
 	}
 
 	// Search for last height marker
-	gr, found, err = cs.wal.group.Search("#ENDHEIGHT: ", makeHeightSearchFunc(csHeight-1))
+	gr, found, err = cs.wal.SearchForEndHeight(uint64(csHeight - 1))
 	if err == io.EOF {
 		cs.Logger.Error("Replay: wal.group.Search returned EOF", "#ENDHEIGHT", csHeight-1)
-		// if we upgraded from 0.9 to 0.9.1, we may have #HEIGHT instead
-		// TODO (0.10.0): remove this
-		gr, found, err = cs.wal.group.Search("#HEIGHT: ", makeHeightSearchFunc(csHeight))
-		if err == io.EOF {
-			cs.Logger.Error("Replay: wal.group.Search returned EOF", "#HEIGHT", csHeight)
-			return nil
-		} else if err != nil {
-			return err
-		}
 	} else if err != nil {
 		return err
-	} else {
-		defer gr.Close()
 	}
 	if !found {
-		// if we upgraded from 0.9 to 0.9.1, we may have #HEIGHT instead
-		// TODO (0.10.0): remove this
-		gr, found, err = cs.wal.group.Search("#HEIGHT: ", makeHeightSearchFunc(csHeight))
-		if err == io.EOF {
-			cs.Logger.Error("Replay: wal.group.Search returned EOF", "#HEIGHT", csHeight)
-			return nil
-		} else if err != nil {
-			return err
-		} else {
-			defer gr.Close()
-		}
-
-		// TODO (0.10.0): uncomment
-		// return errors.New(cmn.Fmt("Cannot replay height %d. WAL does not contain #ENDHEIGHT for %d.", csHeight, csHeight-1))
+		return errors.New(cmn.Fmt("Cannot replay height %d. WAL does not contain #ENDHEIGHT for %d.", csHeight, csHeight-1))
 	}
+	defer gr.Close()
 
 	cs.Logger.Info("Catchup by replaying consensus messages", "height", csHeight)
 
+	var msg *TimedWALMessage
+	dec := WALDecoder{gr}
+
 	for {
-		line, err := gr.ReadLine()
-		if err != nil {
-			if err == io.EOF {
-				break
-			} else {
-				return err
-			}
+		msg, err = dec.Decode()
+		if err == io.EOF {
+			break
+		} else if err != nil {
+			return err
 		}
 		// NOTE: since the priv key is set when the msgs are received
 		// it will attempt to eg double sign but we can just ignore it
 		// since the votes will be replayed and we'll get to the next step
-		if err := cs.readReplayMessage([]byte(line), nil); err != nil {
+		if err := cs.readReplayMessage(msg, nil); err != nil {
 			return err
 		}
 	}
@@ -221,7 +195,7 @@ func (h *Handshaker) NBlocks() int {
 // TODO: retry the handshake/replay if it fails ?
 func (h *Handshaker) Handshake(proxyApp proxy.AppConns) error {
 	// handshake is done via info request on the query conn
-	res, err := proxyApp.Query().InfoSync()
+	res, err := proxyApp.Query().InfoSync(abci.RequestInfo{version.Version})
 	if err != nil {
 		return errors.New(cmn.Fmt("Error calling Info: %v", err))
 	}
@@ -257,7 +231,7 @@ func (h *Handshaker) ReplayBlocks(appHash []byte, appBlockHeight int, proxyApp p
 	// If appBlockHeight == 0 it means that we are at genesis and hence should send InitChain
 	if appBlockHeight == 0 {
 		validators := types.TM2PB.Validators(h.state.Validators)
-		proxyApp.Consensus().InitChainSync(validators)
+		proxyApp.Consensus().InitChainSync(abci.RequestInitChain{validators})
 	}
 
 	// First handle edge cases and constraints on the storeBlockHeight
@@ -324,8 +298,11 @@ func (h *Handshaker) ReplayBlocks(appHash []byte, appBlockHeight int, proxyApp p
 func (h *Handshaker) replayBlocks(proxyApp proxy.AppConns, appBlockHeight, storeBlockHeight int, mutateState bool) ([]byte, error) {
 	// App is further behind than it should be, so we need to replay blocks.
 	// We replay all blocks from appBlockHeight+1.
+	//
 	// Note that we don't have an old version of the state,
 	// so we by-pass state validation/mutation using sm.ExecCommitBlock.
+	// This also means we won't be saving validator sets if they change during this period.
+	//
 	// If mutateState == true, the final block is replayed with h.replayBlock()
 
 	var appHash []byte

consensus/replay_file.go

@@ -4,6 +4,7 @@ import (
 	"bufio"
 	"errors"
 	"fmt"
+	"io"
 	"os"
 	"strconv"
 	"strings"
@@ -53,12 +54,20 @@ func (cs *ConsensusState) ReplayFile(file string, console bool) error {
 	defer pb.fp.Close()
 
 	var nextN int // apply N msgs in a row
-	for pb.scanner.Scan() {
+	var msg *TimedWALMessage
+	for {
 		if nextN == 0 && console {
 			nextN = pb.replayConsoleLoop()
 		}
 
-		if err := pb.cs.readReplayMessage(pb.scanner.Bytes(), newStepCh); err != nil {
+		msg, err = pb.dec.Decode()
+		if err == io.EOF {
+			return nil
+		} else if err != nil {
+			return err
+		}
+
+		if err := pb.cs.readReplayMessage(msg, newStepCh); err != nil {
 			return err
 		}
 
@@ -76,9 +85,9 @@ func (cs *ConsensusState) ReplayFile(file string, console bool) error {
 type playback struct {
 	cs *ConsensusState
 
-	fp      *os.File
-	scanner *bufio.Scanner
-	count   int // how many lines/msgs into the file are we
+	fp    *os.File
+	dec   *WALDecoder
+	count int // how many lines/msgs into the file are we
 
 	// replays can be reset to beginning
 	fileName     string    // so we can close/reopen the file
@@ -91,7 +100,7 @@ func newPlayback(fileName string, fp *os.File, cs *ConsensusState, genState *sm.
 		fp:           fp,
 		fileName:     fileName,
 		genesisState: genState,
-		scanner:      bufio.NewScanner(fp),
+		dec:          NewWALDecoder(fp),
 	}
 }
 
@@ -111,13 +120,20 @@ func (pb *playback) replayReset(count int, newStepCh chan interface{}) error {
 		return err
 	}
 	pb.fp = fp
-	pb.scanner = bufio.NewScanner(fp)
+	pb.dec = NewWALDecoder(fp)
 	count = pb.count - count
 	fmt.Printf("Reseting from %d to %d\n", pb.count, count)
 	pb.count = 0
 	pb.cs = newCS
-	for i := 0; pb.scanner.Scan() && i < count; i++ {
-		if err := pb.cs.readReplayMessage(pb.scanner.Bytes(), newStepCh); err != nil {
+	var msg *TimedWALMessage
+	for i := 0; i < count; i++ {
+		msg, err = pb.dec.Decode()
+		if err == io.EOF {
+			return nil
+		} else if err != nil {
+			return err
+		}
+		if err := pb.cs.readReplayMessage(msg, newStepCh); err != nil {
 			return err
 		}
 		pb.count += 1
@@ -241,12 +257,15 @@ func newConsensusStateForReplay(config cfg.BaseConfig, csConfig *cfg.ConsensusCo
 
 	// Get State
 	stateDB := dbm.NewDB("state", config.DBBackend, config.DBDir())
-	state := sm.MakeGenesisStateFromFile(stateDB, config.GenesisFile())
+	state, err := sm.MakeGenesisStateFromFile(stateDB, config.GenesisFile())
+	if err != nil {
+		cmn.Exit(err.Error())
+	}
 
 	// Create proxyAppConn connection (consensus, mempool, query)
 	clientCreator := proxy.DefaultClientCreator(config.ProxyApp, config.ABCI, config.DBDir())
 	proxyApp := proxy.NewAppConns(clientCreator, NewHandshaker(state, blockStore))
-	_, err := proxyApp.Start()
+	_, err = proxyApp.Start()
 	if err != nil {
 		cmn.Exit(cmn.Fmt("Error starting proxy app conns: %v", err))
 	}

consensus/replay_test.go

@@ -8,11 +8,11 @@ import (
 	"io/ioutil"
 	"os"
 	"path"
-	"strings"
 	"testing"
 	"time"
 
 	"github.com/tendermint/abci/example/dummy"
+	abci "github.com/tendermint/abci/types"
 	crypto "github.com/tendermint/go-crypto"
 	wire "github.com/tendermint/go-wire"
 	cmn "github.com/tendermint/tmlibs/common"
@@ -43,7 +43,7 @@ func init() {
 // after running it (eg. sometimes small_block2 will have 5 block parts, sometimes 6).
 // It should only have to be re-run if there is some breaking change to the consensus data structures (eg. blocks, votes)
 // or to the behaviour of the app (eg. computes app hash differently)
-var data_dir = path.Join(cmn.GoPath, "src/github.com/tendermint/tendermint/consensus", "test_data")
+var data_dir = path.Join(cmn.GoPath(), "src/github.com/tendermint/tendermint/consensus", "test_data")
 
 //------------------------------------------------------------------------------------------
 // WAL Tests
@@ -59,12 +59,12 @@ var baseStepChanges = []int{3, 6, 8}
 var testCases = []*testCase{
 	newTestCase("empty_block", baseStepChanges),   // empty block (has 1 block part)
 	newTestCase("small_block1", baseStepChanges),  // small block with txs in 1 block part
-	newTestCase("small_block2", []int{3, 11, 13}), // small block with txs across 6 smaller block parts
+	newTestCase("small_block2", []int{3, 12, 14}), // small block with txs across 6 smaller block parts
 }
 
 type testCase struct {
 	name    string
-	log     string       //full cs wal
+	log     []byte       //full cs wal
 	stepMap map[int]int8 // map lines of log to privval step
 
 	proposeLine   int
@@ -99,29 +99,27 @@ func newMapFromChanges(changes []int) map[int]int8 {
 	return m
 }
 
-func readWAL(p string) string {
+func readWAL(p string) []byte {
 	b, err := ioutil.ReadFile(p)
 	if err != nil {
 		panic(err)
 	}
-	return string(b)
+	return b
 }
 
-func writeWAL(walMsgs string) string {
-	tempDir := os.TempDir()
-	walDir := path.Join(tempDir, "/wal"+cmn.RandStr(12))
-	walFile := path.Join(walDir, "wal")
-	// Create WAL directory
-	err := cmn.EnsureDir(walDir, 0700)
+func writeWAL(walMsgs []byte) string {
+	walFile, err := ioutil.TempFile("", "wal")
 	if err != nil {
-		panic(err)
+		panic(fmt.Errorf("failed to create temp WAL file: %v", err))
 	}
-	// Write the needed WAL to file
-	err = cmn.WriteFile(walFile, []byte(walMsgs), 0600)
+	_, err = walFile.Write(walMsgs)
 	if err != nil {
-		panic(err)
+		panic(fmt.Errorf("failed to write to temp WAL file: %v", err))
 	}
-	return walFile
+	if err := walFile.Close(); err != nil {
+		panic(fmt.Errorf("failed to close temp WAL file: %v", err))
+	}
+	return walFile.Name()
 }
 
 func waitForBlock(newBlockCh chan interface{}, thisCase *testCase, i int) {
@@ -162,11 +160,11 @@ LOOP:
 	cs.Wait()
 }
 
-func toPV(pv PrivValidator) *types.PrivValidator {
-	return pv.(*types.PrivValidator)
+func toPV(pv types.PrivValidator) *types.PrivValidatorFS {
+	return pv.(*types.PrivValidatorFS)
 }
 
-func setupReplayTest(t *testing.T, thisCase *testCase, nLines int, crashAfter bool) (*ConsensusState, chan interface{}, string, string) {
+func setupReplayTest(t *testing.T, thisCase *testCase, nLines int, crashAfter bool) (*ConsensusState, chan interface{}, []byte, string) {
 	t.Log("-------------------------------------")
 	t.Logf("Starting replay test %v (of %d lines of WAL). Crash after = %v", thisCase.name, nLines, crashAfter)
 
@@ -175,11 +173,13 @@ func setupReplayTest(t *testing.T, thisCase *testCase, nLines int, crashAfter bo
 		lineStep -= 1
 	}
 
-	split := strings.Split(thisCase.log, "\n")
+	split := bytes.Split(thisCase.log, walSeparator)
 	lastMsg := split[nLines]
 
 	// we write those lines up to (not including) one with the signature
-	walFile := writeWAL(strings.Join(split[:nLines], "\n") + "\n")
+	b := bytes.Join(split[:nLines], walSeparator)
+	b = append(b, walSeparator...)
+	walFile := writeWAL(b)
 
 	cs := fixedConsensusStateDummy()
 
@@ -194,14 +194,19 @@ func setupReplayTest(t *testing.T, thisCase *testCase, nLines int, crashAfter bo
 	return cs, newBlockCh, lastMsg, walFile
 }
 
-func readTimedWALMessage(t *testing.T, walMsg string) TimedWALMessage {
-	var err error
-	var msg TimedWALMessage
-	wire.ReadJSON(&msg, []byte(walMsg), &err)
+func readTimedWALMessage(t *testing.T, rawMsg []byte) TimedWALMessage {
+	b := bytes.NewBuffer(rawMsg)
+	// because rawMsg does not contain a separator and WALDecoder#Decode expects it
+	_, err := b.Write(walSeparator)
+	if err != nil {
+		t.Fatal(err)
+	}
+	dec := NewWALDecoder(b)
+	msg, err := dec.Decode()
 	if err != nil {
 		t.Fatalf("Error reading json data: %v", err)
 	}
-	return msg
+	return *msg
 }
 
 //-----------------------------------------------
@@ -210,10 +215,15 @@ func readTimedWALMessage(t *testing.T, walMsg string) TimedWALMessage {
 
 func TestWALCrashAfterWrite(t *testing.T) {
 	for _, thisCase := range testCases {
-		split := strings.Split(thisCase.log, "\n")
-		for i := 0; i < len(split)-1; i++ {
-			cs, newBlockCh, _, walFile := setupReplayTest(t, thisCase, i+1, true)
-			runReplayTest(t, cs, walFile, newBlockCh, thisCase, i+1)
+		splitSize := bytes.Count(thisCase.log, walSeparator)
+		for i := 0; i < splitSize-1; i++ {
+			t.Run(fmt.Sprintf("%s:%d", thisCase.name, i), func(t *testing.T) {
+				cs, newBlockCh, _, walFile := setupReplayTest(t, thisCase, i+1, true)
+				cs.config.TimeoutPropose = 100
+				runReplayTest(t, cs, walFile, newBlockCh, thisCase, i+1)
+				// cleanup
+				os.Remove(walFile)
+			})
 		}
 	}
 }
@@ -225,14 +235,19 @@ func TestWALCrashAfterWrite(t *testing.T) {
 func TestWALCrashBeforeWritePropose(t *testing.T) {
 	for _, thisCase := range testCases {
 		lineNum := thisCase.proposeLine
-		// setup replay test where last message is a proposal
-		cs, newBlockCh, proposalMsg, walFile := setupReplayTest(t, thisCase, lineNum, false)
-		msg := readTimedWALMessage(t, proposalMsg)
-		proposal := msg.Msg.(msgInfo).Msg.(*ProposalMessage)
-		// Set LastSig
-		toPV(cs.privValidator).LastSignBytes = types.SignBytes(cs.state.ChainID, proposal.Proposal)
-		toPV(cs.privValidator).LastSignature = proposal.Proposal.Signature
-		runReplayTest(t, cs, walFile, newBlockCh, thisCase, lineNum)
+		t.Run(fmt.Sprintf("%s:%d", thisCase.name, lineNum), func(t *testing.T) {
+			// setup replay test where last message is a proposal
+			cs, newBlockCh, proposalMsg, walFile := setupReplayTest(t, thisCase, lineNum, false)
+			cs.config.TimeoutPropose = 100
+			msg := readTimedWALMessage(t, proposalMsg)
+			proposal := msg.Msg.(msgInfo).Msg.(*ProposalMessage)
+			// Set LastSig
+			toPV(cs.privValidator).LastSignBytes = types.SignBytes(cs.state.ChainID, proposal.Proposal)
+			toPV(cs.privValidator).LastSignature = proposal.Proposal.Signature
+			runReplayTest(t, cs, walFile, newBlockCh, thisCase, lineNum)
+			// cleanup
+			os.Remove(walFile)
+		})
 	}
 }
 
@@ -267,8 +282,6 @@ func testReplayCrashBeforeWriteVote(t *testing.T, thisCase *testCase, lineNum in
 var (
 	NUM_BLOCKS = 6 // number of blocks in the test_data/many_blocks.cswal
 	mempool    = types.MockMempool{}
-
-	testPartSize int
 )
 
 //---------------------------------------
@@ -316,11 +329,10 @@ func testHandshakeReplay(t *testing.T, nBlocks int, mode uint) {
 	if err != nil {
 		t.Fatal(err)
 	}
-	walFile := writeWAL(string(walBody))
+	walFile := writeWAL(walBody)
 	config.Consensus.SetWalFile(walFile)
 
-	privVal := types.LoadPrivValidator(config.PrivValidatorFile())
-	testPartSize = config.Consensus.BlockPartSize
+	privVal := types.LoadPrivValidatorFS(config.PrivValidatorFile())
 
 	wal, err := NewWAL(walFile, false)
 	if err != nil {
@@ -335,7 +347,7 @@ func testHandshakeReplay(t *testing.T, nBlocks int, mode uint) {
 		t.Fatalf(err.Error())
 	}
 
-	state, store := stateAndStore(config, privVal.PubKey)
+	state, store := stateAndStore(config, privVal.GetPubKey())
 	store.chain = chain
 	store.commits = commits
 
@@ -349,7 +361,7 @@ func testHandshakeReplay(t *testing.T, nBlocks int, mode uint) {
 		// run nBlocks against a new client to build up the app state.
 		// use a throwaway tendermint state
 		proxyApp := proxy.NewAppConns(clientCreator2, nil)
-		state, _ := stateAndStore(config, privVal.PubKey)
+		state, _ := stateAndStore(config, privVal.GetPubKey())
 		buildAppStateFromChain(proxyApp, state, chain, nBlocks, mode)
 	}
 
@@ -361,7 +373,7 @@ func testHandshakeReplay(t *testing.T, nBlocks int, mode uint) {
 	}
 
 	// get the latest app hash from the app
-	res, err := proxyApp.Query().InfoSync()
+	res, err := proxyApp.Query().InfoSync(abci.RequestInfo{""})
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -384,6 +396,7 @@ func testHandshakeReplay(t *testing.T, nBlocks int, mode uint) {
 }
 
 func applyBlock(st *sm.State, blk *types.Block, proxyApp proxy.AppConns) {
+	testPartSize := st.Params.BlockPartSizeBytes
 	err := st.ApplyBlock(nil, proxyApp.Consensus(), blk, blk.MakePartSet(testPartSize).Header(), mempool)
 	if err != nil {
 		panic(err)
@@ -398,7 +411,7 @@ func buildAppStateFromChain(proxyApp proxy.AppConns,
 	}
 
 	validators := types.TM2PB.Validators(state.Validators)
-	proxyApp.Consensus().InitChainSync(validators)
+	proxyApp.Consensus().InitChainSync(abci.RequestInitChain{validators})
 
 	defer proxyApp.Stop()
 	switch mode {
@@ -432,7 +445,7 @@ func buildTMStateFromChain(config *cfg.Config, state *sm.State, chain []*types.B
 	defer proxyApp.Stop()
 
 	validators := types.TM2PB.Validators(state.Validators)
-	proxyApp.Consensus().InitChainSync(validators)
+	proxyApp.Consensus().InitChainSync(abci.RequestInitChain{validators})
 
 	var latestAppHash []byte
 
@@ -466,7 +479,7 @@ func buildTMStateFromChain(config *cfg.Config, state *sm.State, chain []*types.B
 
 func makeBlockchainFromWAL(wal *WAL) ([]*types.Block, []*types.Commit, error) {
 	// Search for height marker
-	gr, found, err := wal.group.Search("#ENDHEIGHT: ", makeHeightSearchFunc(0))
+	gr, found, err := wal.SearchForEndHeight(0)
 	if err != nil {
 		return nil, nil, err
 	}
@@ -480,20 +493,17 @@ func makeBlockchainFromWAL(wal *WAL) ([]*types.Block, []*types.Commit, error) {
 	var blockParts *types.PartSet
 	var blocks []*types.Block
 	var commits []*types.Commit
-	for {
-		line, err := gr.ReadLine()
-		if err != nil {
-			if err == io.EOF {
-				break
-			} else {
-				return nil, nil, err
-			}
-		}
 
-		piece, err := readPieceFromWAL([]byte(line))
-		if err != nil {
+	dec := NewWALDecoder(gr)
+	for {
+		msg, err := dec.Decode()
+		if err == io.EOF {
+			break
+		} else if err != nil {
 			return nil, nil, err
 		}
+
+		piece := readPieceFromWAL(msg)
 		if piece == nil {
 			continue
 		}
@@ -503,7 +513,7 @@ func makeBlockchainFromWAL(wal *WAL) ([]*types.Block, []*types.Commit, error) {
 			// if its not the first one, we have a full block
 			if blockParts != nil {
 				var n int
-				block := wire.ReadBinary(&types.Block{}, blockParts.GetReader(), types.MaxBlockSize, &n, &err).(*types.Block)
+				block := wire.ReadBinary(&types.Block{}, blockParts.GetReader(), 0, &n, &err).(*types.Block)
 				blocks = append(blocks, block)
 			}
 			blockParts = types.NewPartSetFromHeader(*p)
@@ -524,22 +534,15 @@ func makeBlockchainFromWAL(wal *WAL) ([]*types.Block, []*types.Commit, error) {
 	}
 	// grab the last block too
 	var n int
-	block := wire.ReadBinary(&types.Block{}, blockParts.GetReader(), types.MaxBlockSize, &n, &err).(*types.Block)
+	block := wire.ReadBinary(&types.Block{}, blockParts.GetReader(), 0, &n, &err).(*types.Block)
 	blocks = append(blocks, block)
 	return blocks, commits, nil
 }
 
-func readPieceFromWAL(msgBytes []byte) (interface{}, error) {
-	// Skip over empty and meta lines
-	if len(msgBytes) == 0 || msgBytes[0] == '#' {
-		return nil, nil
-	}
-	var err error
-	var msg TimedWALMessage
-	wire.ReadJSON(&msg, msgBytes, &err)
-	if err != nil {
-		fmt.Println("MsgBytes:", msgBytes, string(msgBytes))
-		return nil, fmt.Errorf("Error reading json data: %v", err)
+func readPieceFromWAL(msg *TimedWALMessage) interface{} {
+	// skip meta messages
+	if _, ok := msg.Msg.(EndHeightMessage); ok {
+		return nil
 	}
 
 	// for logging
@@ -547,84 +550,24 @@ func readPieceFromWAL(msgBytes []byte) (interface{}, error) {
 	case msgInfo:
 		switch msg := m.Msg.(type) {
 		case *ProposalMessage:
-			return &msg.Proposal.BlockPartsHeader, nil
+			return &msg.Proposal.BlockPartsHeader
 		case *BlockPartMessage:
-			return msg.Part, nil
+			return msg.Part
 		case *VoteMessage:
-			return msg.Vote, nil
+			return msg.Vote
 		}
 	}
-	return nil, nil
-}
 
-// make some bogus txs
-func txsFunc(blockNum int) (txs []types.Tx) {
-	for i := 0; i < 10; i++ {
-		txs = append(txs, types.Tx([]byte{byte(blockNum), byte(i)}))
-	}
-	return txs
-}
-
-// sign a commit vote
-func signCommit(chainID string, privVal *types.PrivValidator, height, round int, hash []byte, header types.PartSetHeader) *types.Vote {
-	vote := &types.Vote{
-		ValidatorIndex:   0,
-		ValidatorAddress: privVal.Address,
-		Height:           height,
-		Round:            round,
-		Type:             types.VoteTypePrecommit,
-		BlockID:          types.BlockID{hash, header},
-	}
-
-	sig := privVal.Sign(types.SignBytes(chainID, vote))
-	vote.Signature = sig
-	return vote
-}
-
-// make a blockchain with one validator
-func makeBlockchain(t *testing.T, chainID string, nBlocks int, privVal *types.PrivValidator, proxyApp proxy.AppConns, state *sm.State) (blockchain []*types.Block, commits []*types.Commit) {
-
-	prevHash := state.LastBlockID.Hash
-	lastCommit := new(types.Commit)
-	prevParts := types.PartSetHeader{}
-	valHash := state.Validators.Hash()
-	prevBlockID := types.BlockID{prevHash, prevParts}
-
-	for i := 1; i < nBlocks+1; i++ {
-		block, parts := types.MakeBlock(i, chainID, txsFunc(i), lastCommit,
-			prevBlockID, valHash, state.AppHash, testPartSize)
-		fmt.Println(i)
-		fmt.Println(block.LastBlockID)
-		err := state.ApplyBlock(nil, proxyApp.Consensus(), block, block.MakePartSet(testPartSize).Header(), mempool)
-		if err != nil {
-			t.Fatal(i, err)
-		}
-
-		voteSet := types.NewVoteSet(chainID, i, 0, types.VoteTypePrecommit, state.Validators)
-		vote := signCommit(chainID, privVal, i, 0, block.Hash(), parts.Header())
-		_, err = voteSet.AddVote(vote)
-		if err != nil {
-			t.Fatal(err)
-		}
-
-		prevHash = block.Hash()
-		prevParts = parts.Header()
-		lastCommit = voteSet.MakeCommit()
-		prevBlockID = types.BlockID{prevHash, prevParts}
-
-		blockchain = append(blockchain, block)
-		commits = append(commits, lastCommit)
-	}
-	return blockchain, commits
+	return nil
 }
 
 // fresh state and mock store
 func stateAndStore(config *cfg.Config, pubKey crypto.PubKey) (*sm.State, *mockBlockStore) {
 	stateDB := dbm.NewMemDB()
-	state := sm.MakeGenesisStateFromFile(stateDB, config.GenesisFile())
+	state, _ := sm.MakeGenesisStateFromFile(stateDB, config.GenesisFile())
 	state.SetLogger(log.TestingLogger().With("module", "state"))
 
-	store := NewMockBlockStore(config)
+	store := NewMockBlockStore(config, state.Params)
 	return state, store
 }
 
@@ -633,13 +576,14 @@ func stateAndStore(config *cfg.Config, pubKey crypto.PubKey) (*sm.State, *mockBl
 
 type mockBlockStore struct {
 	config  *cfg.Config
+	params  types.ConsensusParams
 	chain   []*types.Block
 	commits []*types.Commit
 }
 
 // TODO: NewBlockStore(db.NewMemDB) ...
-func NewMockBlockStore(config *cfg.Config) *mockBlockStore {
-	return &mockBlockStore{config, nil, nil}
+func NewMockBlockStore(config *cfg.Config, params types.ConsensusParams) *mockBlockStore {
+	return &mockBlockStore{config, params, nil, nil}
 }
 
 func (bs *mockBlockStore) Height() int                       { return len(bs.chain) }
@@ -647,7 +591,7 @@ func (bs *mockBlockStore) LoadBlock(height int) *types.Block { return bs.chain[h
 func (bs *mockBlockStore) LoadBlockMeta(height int) *types.BlockMeta {
 	block := bs.chain[height-1]
 	return &types.BlockMeta{
-		BlockID: types.BlockID{block.Hash(), block.MakePartSet(bs.config.Consensus.BlockPartSize).Header()},
+		BlockID: types.BlockID{block.Hash(), block.MakePartSet(bs.params.BlockPartSizeBytes).Header()},
 		Header:  block.Header,
 	}
 }

consensus/state.go

@@ -4,24 +4,32 @@ import (
 	"bytes"
 	"errors"
 	"fmt"
-	"path"
+	"path/filepath"
 	"reflect"
+	"runtime/debug"
 	"sync"
 	"time"
 
 	fail "github.com/ebuchman/fail-test"
+
 	wire "github.com/tendermint/go-wire"
+	cmn "github.com/tendermint/tmlibs/common"
+	"github.com/tendermint/tmlibs/log"
+
 	cfg "github.com/tendermint/tendermint/config"
+	cstypes "github.com/tendermint/tendermint/consensus/types"
 	"github.com/tendermint/tendermint/proxy"
 	sm "github.com/tendermint/tendermint/state"
 	"github.com/tendermint/tendermint/types"
-	cmn "github.com/tendermint/tmlibs/common"
-	"github.com/tendermint/tmlibs/log"
 )
 
 //-----------------------------------------------------------------------------
 // Config
 
+const (
+	proposalHeartbeatIntervalSeconds = 2
+)
+
 //-----------------------------------------------------------------------------
 // Errors
 
@@ -32,117 +40,6 @@ var (
 	ErrVoteHeightMismatch       = errors.New("Error vote height mismatch")
 )
 
-//-----------------------------------------------------------------------------
-// RoundStepType enum type
-
-type RoundStepType uint8 // These must be numeric, ordered.
-
-const (
-	RoundStepNewHeight     = RoundStepType(0x01) // Wait til CommitTime + timeoutCommit
-	RoundStepNewRound      = RoundStepType(0x02) // Setup new round and go to RoundStepPropose
-	RoundStepPropose       = RoundStepType(0x03) // Did propose, gossip proposal
-	RoundStepPrevote       = RoundStepType(0x04) // Did prevote, gossip prevotes
-	RoundStepPrevoteWait   = RoundStepType(0x05) // Did receive any +2/3 prevotes, start timeout
-	RoundStepPrecommit     = RoundStepType(0x06) // Did precommit, gossip precommits
-	RoundStepPrecommitWait = RoundStepType(0x07) // Did receive any +2/3 precommits, start timeout
-	RoundStepCommit        = RoundStepType(0x08) // Entered commit state machine
-	// NOTE: RoundStepNewHeight acts as RoundStepCommitWait.
-)
-
-func (rs RoundStepType) String() string {
-	switch rs {
-	case RoundStepNewHeight:
-		return "RoundStepNewHeight"
-	case RoundStepNewRound:
-		return "RoundStepNewRound"
-	case RoundStepPropose:
-		return "RoundStepPropose"
-	case RoundStepPrevote:
-		return "RoundStepPrevote"
-	case RoundStepPrevoteWait:
-		return "RoundStepPrevoteWait"
-	case RoundStepPrecommit:
-		return "RoundStepPrecommit"
-	case RoundStepPrecommitWait:
-		return "RoundStepPrecommitWait"
-	case RoundStepCommit:
-		return "RoundStepCommit"
-	default:
-		return "RoundStepUnknown" // Cannot panic.
-	}
-}
-
-//-----------------------------------------------------------------------------
-
-// Immutable when returned from ConsensusState.GetRoundState()
-// TODO: Actually, only the top pointer is copied,
-// so access to field pointers is still racey
-type RoundState struct {
-	Height             int // Height we are working on
-	Round              int
-	Step               RoundStepType
-	StartTime          time.Time
-	CommitTime         time.Time // Subjective time when +2/3 precommits for Block at Round were found
-	Validators         *types.ValidatorSet
-	Proposal           *types.Proposal
-	ProposalBlock      *types.Block
-	ProposalBlockParts *types.PartSet
-	LockedRound        int
-	LockedBlock        *types.Block
-	LockedBlockParts   *types.PartSet
-	Votes              *HeightVoteSet
-	CommitRound        int            //
-	LastCommit         *types.VoteSet // Last precommits at Height-1
-	LastValidators     *types.ValidatorSet
-}
-
-func (rs *RoundState) RoundStateEvent() types.EventDataRoundState {
-	edrs := types.EventDataRoundState{
-		Height:     rs.Height,
-		Round:      rs.Round,
-		Step:       rs.Step.String(),
-		RoundState: rs,
-	}
-	return edrs
-}
-
-func (rs *RoundState) String() string {
-	return rs.StringIndented("")
-}
-
-func (rs *RoundState) StringIndented(indent string) string {
-	return fmt.Sprintf(`RoundState{
-%s  H:%v R:%v S:%v
-%s  StartTime:     %v
-%s  CommitTime:    %v
-%s  Validators:    %v
-%s  Proposal:      %v
-%s  ProposalBlock: %v %v
-%s  LockedRound:   %v
-%s  LockedBlock:   %v %v
-%s  Votes:         %v
-%s  LastCommit: %v
-%s  LastValidators:    %v
-%s}`,
-		indent, rs.Height, rs.Round, rs.Step,
-		indent, rs.StartTime,
-		indent, rs.CommitTime,
-		indent, rs.Validators.StringIndented(indent+"    "),
-		indent, rs.Proposal,
-		indent, rs.ProposalBlockParts.StringShort(), rs.ProposalBlock.StringShort(),
-		indent, rs.LockedRound,
-		indent, rs.LockedBlockParts.StringShort(), rs.LockedBlock.StringShort(),
-		indent, rs.Votes.StringIndented(indent+"    "),
-		indent, rs.LastCommit.StringShort(),
-		indent, rs.LastValidators.StringIndented(indent+"    "),
-		indent)
-}
-
-func (rs *RoundState) StringShort() string {
-	return fmt.Sprintf(`RoundState{H:%v R:%v S:%v ST:%v}`,
-		rs.Height, rs.Round, rs.Step, rs.StartTime)
-}
-
 //-----------------------------------------------------------------------------
 
 var (
@@ -157,29 +54,26 @@ type msgInfo struct {
 
 // internally generated messages which may update the state
 type timeoutInfo struct {
-	Duration time.Duration `json:"duration"`
-	Height   int           `json:"height"`
-	Round    int           `json:"round"`
-	Step     RoundStepType `json:"step"`
+	Duration time.Duration         `json:"duration"`
+	Height   int                   `json:"height"`
+	Round    int                   `json:"round"`
+	Step     cstypes.RoundStepType `json:"step"`
 }
 
 func (ti *timeoutInfo) String() string {
 	return fmt.Sprintf("%v ; %d/%d %v", ti.Duration, ti.Height, ti.Round, ti.Step)
 }
 
-type PrivValidator interface {
-	GetAddress() []byte
-	SignVote(chainID string, vote *types.Vote) error
-	SignProposal(chainID string, proposal *types.Proposal) error
-}
-
-// Tracks consensus state across block heights and rounds.
+// ConsensusState handles execution of the consensus algorithm.
+// It processes votes and proposals, and upon reaching agreement,
+// commits blocks to the chain and executes them against the application.
+// The internal state machine receives input from peers, the internal validator, and from a timer.
 type ConsensusState struct {
 	cmn.BaseService
 
 	// config details
 	config        *cfg.ConsensusConfig
-	privValidator PrivValidator // for signing votes
+	privValidator types.PrivValidator // for signing votes
 
 	// services for creating and executing blocks
 	proxyAppConn proxy.AppConnConsensus
@@ -188,7 +82,7 @@ type ConsensusState struct {
 
 	// internal state
 	mtx sync.Mutex
-	RoundState
+	cstypes.RoundState
 	state *sm.State // State until height-1.
 
 	// state changes may be triggered by msgs from peers,
@@ -203,8 +97,9 @@ type ConsensusState struct {
 
 	// a Write-Ahead Log ensures we can recover from any kind of crash
 	// and helps us avoid signing conflicting votes
-	wal        *WAL
-	replayMode bool // so we don't log signing errors during replay
+	wal          *WAL
+	replayMode   bool // so we don't log signing errors during replay
+	doWALCatchup bool // determines if we even try to do the catchup
 
 	// for tests where we want to limit the number of transitions the state makes
 	nSteps int
@@ -218,6 +113,7 @@ type ConsensusState struct {
 	done chan struct{}
 }
 
+// NewConsensusState returns a new ConsensusState.
 func NewConsensusState(config *cfg.ConsensusConfig, state *sm.State, proxyAppConn proxy.AppConnConsensus, blockStore types.BlockStore, mempool types.Mempool) *ConsensusState {
 	cs := &ConsensusState{
 		config:           config,
@@ -228,6 +124,7 @@ func NewConsensusState(config *cfg.ConsensusConfig, state *sm.State, proxyAppCon
 		internalMsgQueue: make(chan msgInfo, msgQueueSize),
 		timeoutTicker:    NewTimeoutTicker(),
 		done:             make(chan struct{}),
+		doWALCatchup:     true,
 	}
 	// set function defaults (may be overwritten before calling Start)
 	cs.decideProposal = cs.defaultDecideProposal
@@ -256,48 +153,53 @@ func (cs *ConsensusState) SetEventSwitch(evsw types.EventSwitch) {
 	cs.evsw = evsw
 }
 
+// String returns a string.
 func (cs *ConsensusState) String() string {
 	// better not to access shared variables
 	return cmn.Fmt("ConsensusState") //(H:%v R:%v S:%v", cs.Height, cs.Round, cs.Step)
 }
 
+// GetState returns a copy of the chain state.
 func (cs *ConsensusState) GetState() *sm.State {
 	cs.mtx.Lock()
 	defer cs.mtx.Unlock()
 	return cs.state.Copy()
 }
 
-func (cs *ConsensusState) GetRoundState() *RoundState {
+// GetRoundState returns a copy of the internal consensus state.
+func (cs *ConsensusState) GetRoundState() *cstypes.RoundState {
 	cs.mtx.Lock()
 	defer cs.mtx.Unlock()
 	return cs.getRoundState()
 }
 
-func (cs *ConsensusState) getRoundState() *RoundState {
+func (cs *ConsensusState) getRoundState() *cstypes.RoundState {
 	rs := cs.RoundState // copy
 	return &rs
 }
 
+// GetValidators returns a copy of the current validators.
 func (cs *ConsensusState) GetValidators() (int, []*types.Validator) {
 	cs.mtx.Lock()
 	defer cs.mtx.Unlock()
 	return cs.state.LastBlockHeight, cs.state.Validators.Copy().Validators
 }
 
-// Sets our private validator account for signing votes.
-func (cs *ConsensusState) SetPrivValidator(priv PrivValidator) {
+// SetPrivValidator sets the private validator account for signing votes.
+func (cs *ConsensusState) SetPrivValidator(priv types.PrivValidator) {
 	cs.mtx.Lock()
 	defer cs.mtx.Unlock()
 	cs.privValidator = priv
 }
 
-// Set the local timer
+// SetTimeoutTicker sets the local timer. It may be useful to overwrite for testing.
 func (cs *ConsensusState) SetTimeoutTicker(timeoutTicker TimeoutTicker) {
 	cs.mtx.Lock()
 	defer cs.mtx.Unlock()
 	cs.timeoutTicker = timeoutTicker
 }
 
+// LoadCommit loads the commit for a given height.
 func (cs *ConsensusState) LoadCommit(height int) *types.Commit {
 	cs.mtx.Lock()
 	defer cs.mtx.Unlock()
@@ -307,11 +209,13 @@ func (cs *ConsensusState) LoadCommit(height int) *types.Commit {
 	return cs.blockStore.LoadBlockCommit(height)
 }
 
+// OnStart implements cmn.Service.
+// It loads the latest state via the WAL, and starts the timeout and receive routines.
 func (cs *ConsensusState) OnStart() error {
 
 	walFile := cs.config.WalFile()
 	if err := cs.OpenWAL(walFile); err != nil {
-		cs.Logger.Error("Error loading ConsensusState wal", "error", err.Error())
+		cs.Logger.Error("Error loading ConsensusState wal", "err", err.Error())
 		return err
 	}
 
@@ -324,10 +228,12 @@ func (cs *ConsensusState) OnStart() error {
 
 	// we may have lost some votes if the process crashed
 	// reload from consensus log to catchup
-	if err := cs.catchupReplay(cs.Height); err != nil {
-		cs.Logger.Error("Error on catchup replay. Proceeding to start ConsensusState anyway", "error", err.Error())
-		// NOTE: if we ever do return an error here,
-		// make sure to stop the timeoutTicker
+	if cs.doWALCatchup {
+		if err := cs.catchupReplay(cs.Height); err != nil {
+			cs.Logger.Error("Error on catchup replay. Proceeding to start ConsensusState anyway", "err", err.Error())
+			// NOTE: if we ever do return an error here,
+			// make sure to stop the timeoutTicker
+		}
 	}
 
 	// now start the receiveRoutine
@@ -347,6 +253,7 @@ func (cs *ConsensusState) startRoutines(maxSteps int) {
 	go cs.receiveRoutine(maxSteps)
 }
 
+// OnStop implements cmn.Service. It stops all routines and waits for the WAL to finish.
 func (cs *ConsensusState) OnStop() {
 	cs.BaseService.OnStop()
 
@@ -358,17 +265,18 @@ func (cs *ConsensusState) OnStop() {
 	}
 }
 
+// Wait waits for the the main routine to return.
 // NOTE: be sure to Stop() the event switch and drain
 // any event channels or this may deadlock
 func (cs *ConsensusState) Wait() {
 	<-cs.done
 }
 
-// Open file to log all consensus messages and timeouts for deterministic accountability
+// OpenWAL opens a file to log all consensus messages and timeouts for deterministic accountability
 func (cs *ConsensusState) OpenWAL(walFile string) (err error) {
-	err = cmn.EnsureDir(path.Dir(walFile), 0700)
+	err = cmn.EnsureDir(filepath.Dir(walFile), 0700)
 	if err != nil {
-		cs.Logger.Error("Error ensuring ConsensusState wal dir", "error", err.Error())
+		cs.Logger.Error("Error ensuring ConsensusState wal dir", "err", err.Error())
 		return err
 	}
 
@@ -387,11 +295,13 @@ func (cs *ConsensusState) OpenWAL(walFile string) (err error) {
 }
 
 //------------------------------------------------------------
-// Public interface for passing messages into the consensus state,
-// possibly causing a state transition
+// Public interface for passing messages into the consensus state, possibly causing a state transition.
+// If peerKey == "", the msg is considered internal.
+// Messages are added to the appropriate queue (peer or internal).
+// If the queue is full, the function may block.
 // TODO: should these return anything or let callers just use events?
 
-// May block on send if queue is full.
+// AddVote inputs a vote.
 func (cs *ConsensusState) AddVote(vote *types.Vote, peerKey string) (added bool, err error) {
 	if peerKey == "" {
 		cs.internalMsgQueue <- msgInfo{&VoteMessage{vote}, ""}
@@ -403,7 +313,7 @@ func (cs *ConsensusState) AddVote(vote *types.Vote, peerKey string) (added bool,
 	return false, nil
 }
 
-// May block on send if queue is full.
+// SetProposal inputs a proposal.
 func (cs *ConsensusState) SetProposal(proposal *types.Proposal, peerKey string) error {
 
 	if peerKey == "" {
@@ -416,7 +326,7 @@ func (cs *ConsensusState) SetProposal(proposal *types.Proposal, peerKey string)
 	return nil
 }
 
-// May block on send if queue is full.
+// AddProposalBlockPart inputs a part of the proposal block.
 func (cs *ConsensusState) AddProposalBlockPart(height, round int, part *types.Part, peerKey string) error {
 
 	if peerKey == "" {
@@ -429,7 +339,7 @@ func (cs *ConsensusState) AddProposalBlockPart(height, round int, part *types.Pa
 	return nil
 }
 
-// May block on send if queue is full.
+// SetProposalAndBlock inputs the proposal and all block parts.
 func (cs *ConsensusState) SetProposalAndBlock(proposal *types.Proposal, block *types.Block, parts *types.PartSet, peerKey string) error {
 	cs.SetProposal(proposal, peerKey)
 	for i := 0; i < parts.Total(); i++ {
@@ -446,20 +356,20 @@ func (cs *ConsensusState) updateHeight(height int) {
 	cs.Height = height
 }
 
-func (cs *ConsensusState) updateRoundStep(round int, step RoundStepType) {
+func (cs *ConsensusState) updateRoundStep(round int, step cstypes.RoundStepType) {
 	cs.Round = round
 	cs.Step = step
 }
 
 // enterNewRound(height, 0) at cs.StartTime.
-func (cs *ConsensusState) scheduleRound0(rs *RoundState) {
+func (cs *ConsensusState) scheduleRound0(rs *cstypes.RoundState) {
 	//cs.Logger.Info("scheduleRound0", "now", time.Now(), "startTime", cs.StartTime)
 	sleepDuration := rs.StartTime.Sub(time.Now())
-	cs.scheduleTimeout(sleepDuration, rs.Height, 0, RoundStepNewHeight)
+	cs.scheduleTimeout(sleepDuration, rs.Height, 0, cstypes.RoundStepNewHeight)
 }
 
 // Attempt to schedule a timeout (by sending timeoutInfo on the tickChan)
-func (cs *ConsensusState) scheduleTimeout(duration time.Duration, height, round int, step RoundStepType) {
+func (cs *ConsensusState) scheduleTimeout(duration time.Duration, height, round int, step cstypes.RoundStepType) {
 	cs.timeoutTicker.ScheduleTimeout(timeoutInfo{duration, height, round, step})
 }
 
@@ -484,7 +394,7 @@ func (cs *ConsensusState) reconstructLastCommit(state *sm.State) {
 		return
 	}
 	seenCommit := cs.blockStore.LoadSeenCommit(state.LastBlockHeight)
-	lastPrecommits := types.NewVoteSet(cs.state.ChainID, state.LastBlockHeight, seenCommit.Round(), types.VoteTypePrecommit, state.LastValidators)
+	lastPrecommits := types.NewVoteSet(state.ChainID, state.LastBlockHeight, seenCommit.Round(), types.VoteTypePrecommit, state.LastValidators)
 	for _, precommit := range seenCommit.Precommits {
 		if precommit == nil {
 			continue
@@ -501,7 +411,7 @@ func (cs *ConsensusState) reconstructLastCommit(state *sm.State) {
 }
 
 // Updates ConsensusState and increments height to match that of state.
-// The round becomes 0 and cs.Step becomes RoundStepNewHeight.
+// The round becomes 0 and cs.Step becomes cstypes.RoundStepNewHeight.
 func (cs *ConsensusState) updateToState(state *sm.State) {
 	if cs.CommitRound > -1 && 0 < cs.Height && cs.Height != state.LastBlockHeight {
 		cmn.PanicSanity(cmn.Fmt("updateToState() expected state height of %v but found %v",
@@ -537,7 +447,7 @@ func (cs *ConsensusState) updateToState(state *sm.State) {
 
 	// RoundState fields
 	cs.updateHeight(height)
-	cs.updateRoundStep(0, RoundStepNewHeight)
+	cs.updateRoundStep(0, cstypes.RoundStepNewHeight)
 	if cs.CommitTime.IsZero() {
 		// "Now" makes it easier to sync up dev nodes.
 		// We add timeoutCommit to allow transactions
@@ -555,7 +465,7 @@ func (cs *ConsensusState) updateToState(state *sm.State) {
 	cs.LockedRound = 0
 	cs.LockedBlock = nil
 	cs.LockedBlockParts = nil
-	cs.Votes = NewHeightVoteSet(state.ChainID, height, validators)
+	cs.Votes = cstypes.NewHeightVoteSet(state.ChainID, height, validators)
 	cs.CommitRound = -1
 	cs.LastCommit = lastPrecommits
 	cs.LastValidators = state.LastValidators
@@ -582,8 +492,15 @@ func (cs *ConsensusState) newStep() {
 // receiveRoutine handles messages which may cause state transitions.
 // it's argument (n) is the number of messages to process before exiting - use 0 to run forever
 // It keeps the RoundState and is the only thing that updates it.
-// Updates (state transitions) happen on timeouts, complete proposals, and 2/3 majorities
+// Updates (state transitions) happen on timeouts, complete proposals, and 2/3 majorities.
+// ConsensusState must be locked before any internal state is updated.
 func (cs *ConsensusState) receiveRoutine(maxSteps int) {
+	defer func() {
+		if r := recover(); r != nil {
+			cs.Logger.Error("CONSENSUS FAILURE!!!", "err", r, "stack", string(debug.Stack()))
+		}
+	}()
+
 	for {
 		if maxSteps > 0 {
 			if cs.nSteps >= maxSteps {
@@ -596,15 +513,17 @@ func (cs *ConsensusState) receiveRoutine(maxSteps int) {
 		var mi msgInfo
 
 		select {
+		case height := <-cs.mempool.TxsAvailable():
+			cs.handleTxsAvailable(height)
 		case mi = <-cs.peerMsgQueue:
 			cs.wal.Save(mi)
 			// handles proposals, block parts, votes
 			// may generate internal events (votes, complete proposals, 2/3 majorities)
-			cs.handleMsg(mi, rs)
+			cs.handleMsg(mi)
 		case mi = <-cs.internalMsgQueue:
 			cs.wal.Save(mi)
 			// handles proposals, block parts, votes
-			cs.handleMsg(mi, rs)
+			cs.handleMsg(mi)
 		case ti := <-cs.timeoutTicker.Chan(): // tockChan:
 			cs.wal.Save(ti)
 			// if the timeout is relevant to the rs
@@ -628,7 +547,7 @@ func (cs *ConsensusState) receiveRoutine(maxSteps int) {
 }
 
 // state transitions on complete-proposal, 2/3-any, 2/3-one
-func (cs *ConsensusState) handleMsg(mi msgInfo, rs RoundState) {
+func (cs *ConsensusState) handleMsg(mi msgInfo) {
 	cs.mtx.Lock()
 	defer cs.mtx.Unlock()
 
@@ -663,11 +582,11 @@ func (cs *ConsensusState) handleMsg(mi msgInfo, rs RoundState) {
 		cs.Logger.Error("Unknown msg type", reflect.TypeOf(msg))
 	}
 	if err != nil {
-		cs.Logger.Error("Error with msg", "type", reflect.TypeOf(msg), "peer", peerKey, "error", err, "msg", msg)
+		cs.Logger.Error("Error with msg", "type", reflect.TypeOf(msg), "peer", peerKey, "err", err, "msg", msg)
 	}
 }
 
-func (cs *ConsensusState) handleTimeout(ti timeoutInfo, rs RoundState) {
+func (cs *ConsensusState) handleTimeout(ti timeoutInfo, rs cstypes.RoundState) {
 	cs.Logger.Debug("Received tock", "timeout", ti.Duration, "height", ti.Height, "round", ti.Round, "step", ti.Step)
 
 	// timeouts must be for current height, round, step
@@ -681,17 +600,19 @@ func (cs *ConsensusState) handleTimeout(ti timeoutInfo, rs RoundState) {
 	defer cs.mtx.Unlock()
 
 	switch ti.Step {
-	case RoundStepNewHeight:
+	case cstypes.RoundStepNewHeight:
 		// NewRound event fired from enterNewRound.
 		// XXX: should we fire timeout here (for timeout commit)?
 		cs.enterNewRound(ti.Height, 0)
-	case RoundStepPropose:
+	case cstypes.RoundStepNewRound:
+		cs.enterPropose(ti.Height, 0)
+	case cstypes.RoundStepPropose:
 		types.FireEventTimeoutPropose(cs.evsw, cs.RoundStateEvent())
 		cs.enterPrevote(ti.Height, ti.Round)
-	case RoundStepPrevoteWait:
+	case cstypes.RoundStepPrevoteWait:
 		types.FireEventTimeoutWait(cs.evsw, cs.RoundStateEvent())
 		cs.enterPrecommit(ti.Height, ti.Round)
-	case RoundStepPrecommitWait:
+	case cstypes.RoundStepPrecommitWait:
 		types.FireEventTimeoutWait(cs.evsw, cs.RoundStateEvent())
 		cs.enterNewRound(ti.Height, ti.Round+1)
 	default:
@@ -700,16 +621,25 @@ func (cs *ConsensusState) handleTimeout(ti timeoutInfo, rs RoundState) {
 
 }
 
+func (cs *ConsensusState) handleTxsAvailable(height int) {
+	cs.mtx.Lock()
+	defer cs.mtx.Unlock()
+	// we only need to do this for round 0
+	cs.enterPropose(height, 0)
+}
+
 //-----------------------------------------------------------------------------
 // State functions
 // Used internally by handleTimeout and handleMsg to make state transitions
 
-// Enter: +2/3 precommits for nil at (height,round-1)
+// Enter: `timeoutNewHeight` by startTime (commitTime+timeoutCommit),
+// 	or, if SkipTimeout==true, after receiving all precommits from (height,round-1)
 // Enter: `timeoutPrecommits` after any +2/3 precommits from (height,round-1)
-// Enter: `startTime = commitTime+timeoutCommit` from NewHeight(height)
+// Enter: +2/3 precommits for nil at (height,round-1)
+// Enter: +2/3 prevotes any or +2/3 precommits for block or any from (height, round)
 // NOTE: cs.StartTime was already set for height.
 func (cs *ConsensusState) enterNewRound(height int, round int) {
-	if cs.Height != height || round < cs.Round || (cs.Round == round && cs.Step != RoundStepNewHeight) {
+	if cs.Height != height || round < cs.Round || (cs.Round == round && cs.Step != cstypes.RoundStepNewHeight) {
 		cs.Logger.Debug(cmn.Fmt("enterNewRound(%v/%v): Invalid args. Current step: %v/%v/%v", height, round, cs.Height, cs.Round, cs.Step))
 		return
 	}
@@ -730,7 +660,7 @@ func (cs *ConsensusState) enterNewRound(height int, round int) {
 	// Setup new round
 	// we don't fire newStep for this step,
 	// but we fire an event, so update the round step first
-	cs.updateRoundStep(round, RoundStepNewRound)
+	cs.updateRoundStep(round, cstypes.RoundStepNewRound)
 	cs.Validators = validators
 	if round == 0 {
 		// We've already reset these upon new height,
@@ -745,13 +675,69 @@ func (cs *ConsensusState) enterNewRound(height int, round int) {
 
 	types.FireEventNewRound(cs.evsw, cs.RoundStateEvent())
 
-	// Immediately go to enterPropose.
-	cs.enterPropose(height, round)
+	// Wait for txs to be available in the mempool
+	// before we enterPropose in round 0. If the last block changed the app hash,
+	// we may need an empty "proof" block, and enterPropose immediately.
+	waitForTxs := cs.config.WaitForTxs() && round == 0 && !cs.needProofBlock(height)
+	if waitForTxs {
+		if cs.config.CreateEmptyBlocksInterval > 0 {
+			cs.scheduleTimeout(cs.config.EmptyBlocksInterval(), height, round, cstypes.RoundStepNewRound)
+		}
+		go cs.proposalHeartbeat(height, round)
+	} else {
+		cs.enterPropose(height, round)
+	}
 }
 
-// Enter: from NewRound(height,round).
+// needProofBlock returns true on the first height (so the genesis app hash is signed right away)
+// and where the last block (height-1) caused the app hash to change
+func (cs *ConsensusState) needProofBlock(height int) bool {
+	if height == 1 {
+		return true
+	}
+
+	lastBlockMeta := cs.blockStore.LoadBlockMeta(height - 1)
+	if !bytes.Equal(cs.state.AppHash, lastBlockMeta.Header.AppHash) {
+		return true
+	}
+	return false
+}
+
+func (cs *ConsensusState) proposalHeartbeat(height, round int) {
+	counter := 0
+	addr := cs.privValidator.GetAddress()
+	valIndex, v := cs.Validators.GetByAddress(addr)
+	if v == nil {
+		// not a validator
+		valIndex = -1
+	}
+	chainID := cs.state.ChainID
+	for {
+		rs := cs.GetRoundState()
+		// if we've already moved on, no need to send more heartbeats
+		if rs.Step > cstypes.RoundStepNewRound || rs.Round > round || rs.Height > height {
+			return
+		}
+		heartbeat := &types.Heartbeat{
+			Height:           rs.Height,
+			Round:            rs.Round,
+			Sequence:         counter,
+			ValidatorAddress: addr,
+			ValidatorIndex:   valIndex,
+		}
+		cs.privValidator.SignHeartbeat(chainID, heartbeat)
+		heartbeatEvent := types.EventDataProposalHeartbeat{heartbeat}
+		types.FireEventProposalHeartbeat(cs.evsw, heartbeatEvent)
+		counter += 1
+		time.Sleep(proposalHeartbeatIntervalSeconds * time.Second)
+	}
+}
+
+// Enter (CreateEmptyBlocks): from enterNewRound(height,round)
+// Enter (CreateEmptyBlocks, CreateEmptyBlocksInterval > 0 ): after enterNewRound(height,round), after timeout of CreateEmptyBlocksInterval
+// Enter (!CreateEmptyBlocks) : after enterNewRound(height,round), once txs are in the mempool
 func (cs *ConsensusState) enterPropose(height int, round int) {
-	if cs.Height != height || round < cs.Round || (cs.Round == round && RoundStepPropose <= cs.Step) {
+	if cs.Height != height || round < cs.Round || (cs.Round == round && cstypes.RoundStepPropose <= cs.Step) {
 		cs.Logger.Debug(cmn.Fmt("enterPropose(%v/%v): Invalid args. Current step: %v/%v/%v", height, round, cs.Height, cs.Round, cs.Step))
 		return
 	}
@@ -759,7 +745,7 @@ func (cs *ConsensusState) enterPropose(height int, round int) {
 
 	defer func() {
 		// Done enterPropose:
-		cs.updateRoundStep(round, RoundStepPropose)
+		cs.updateRoundStep(round, cstypes.RoundStepPropose)
 		cs.newStep()
 
 		// If we have the whole proposal + POL, then goto Prevote now.
@@ -771,7 +757,7 @@ func (cs *ConsensusState) enterPropose(height int, round int) {
 	}()
 
 	// If we don't get the proposal and all block parts quick enough, enterPrevote
-	cs.scheduleTimeout(cs.config.Propose(round), height, round, RoundStepPropose)
+	cs.scheduleTimeout(cs.config.Propose(round), height, round, cstypes.RoundStepPropose)
 
 	// Nothing more to do if we're not a validator
 	if cs.privValidator == nil {
@@ -779,7 +765,7 @@ func (cs *ConsensusState) enterPropose(height int, round int) {
 		return
 	}
 
-	if !bytes.Equal(cs.Validators.GetProposer().Address, cs.privValidator.GetAddress()) {
+	if !cs.isProposer() {
 		cs.Logger.Info("enterPropose: Not our turn to propose", "proposer", cs.Validators.GetProposer().Address, "privValidator", cs.privValidator)
 		if cs.Validators.HasAddress(cs.privValidator.GetAddress()) {
 			cs.Logger.Debug("This node is a validator")
@@ -793,6 +779,10 @@ func (cs *ConsensusState) enterPropose(height int, round int) {
 	}
 }
 
+func (cs *ConsensusState) isProposer() bool {
+	return bytes.Equal(cs.Validators.GetProposer().Address, cs.privValidator.GetAddress())
+}
+
 func (cs *ConsensusState) defaultDecideProposal(height, round int) {
 	var block *types.Block
 	var blockParts *types.PartSet
@@ -812,8 +802,7 @@ func (cs *ConsensusState) defaultDecideProposal(height, round int) {
 	// Make proposal
 	polRound, polBlockID := cs.Votes.POLInfo()
 	proposal := types.NewProposal(height, round, blockParts.Header(), polRound, polBlockID)
-	err := cs.privValidator.SignProposal(cs.state.ChainID, proposal)
-	if err == nil {
+	if err := cs.privValidator.SignProposal(cs.state.ChainID, proposal); err == nil {
 		// Set fields
 		/*  fields set by setProposal and addBlockPart
 		cs.Proposal = proposal
@@ -831,7 +820,7 @@ func (cs *ConsensusState) defaultDecideProposal(height, round int) {
 		cs.Logger.Debug(cmn.Fmt("Signed proposal block: %v", block))
 	} else {
 		if !cs.replayMode {
-			cs.Logger.Error("enterPropose: Error signing proposal", "height", height, "round", round, "error", err)
+			cs.Logger.Error("enterPropose: Error signing proposal", "height", height, "round", round, "err", err)
 		}
 	}
 }
@@ -872,9 +861,9 @@ func (cs *ConsensusState) createProposalBlock() (block *types.Block, blockParts
 
 	// Mempool validated transactions
 	txs := cs.mempool.Reap(cs.config.MaxBlockSizeTxs)
-
 	return types.MakeBlock(cs.Height, cs.state.ChainID, txs, commit,
-		cs.state.LastBlockID, cs.state.Validators.Hash(), cs.state.AppHash, cs.config.BlockPartSize)
+		cs.state.LastBlockID, cs.state.Validators.Hash(),
+		cs.state.AppHash, cs.state.Params.BlockPartSizeBytes)
 }
 
 // Enter: `timeoutPropose` after entering Propose.
@@ -883,14 +872,14 @@ func (cs *ConsensusState) createProposalBlock() (block *types.Block, blockParts
 // Prevote for LockedBlock if we're locked, or ProposalBlock if valid.
 // Otherwise vote nil.
 func (cs *ConsensusState) enterPrevote(height int, round int) {
-	if cs.Height != height || round < cs.Round || (cs.Round == round && RoundStepPrevote <= cs.Step) {
+	if cs.Height != height || round < cs.Round || (cs.Round == round && cstypes.RoundStepPrevote <= cs.Step) {
 		cs.Logger.Debug(cmn.Fmt("enterPrevote(%v/%v): Invalid args. Current step: %v/%v/%v", height, round, cs.Height, cs.Round, cs.Step))
 		return
 	}
 
 	defer func() {
 		// Done enterPrevote:
-		cs.updateRoundStep(round, RoundStepPrevote)
+		cs.updateRoundStep(round, cstypes.RoundStepPrevote)
 		cs.newStep()
 	}()
 
@@ -912,25 +901,26 @@ func (cs *ConsensusState) enterPrevote(height int, round int) {
 }
 
 func (cs *ConsensusState) defaultDoPrevote(height int, round int) {
+	logger := cs.Logger.With("height", height, "round", round)
 	// If a block is locked, prevote that.
 	if cs.LockedBlock != nil {
-		cs.Logger.Info("enterPrevote: Block was locked")
+		logger.Info("enterPrevote: Block was locked")
 		cs.signAddVote(types.VoteTypePrevote, cs.LockedBlock.Hash(), cs.LockedBlockParts.Header())
 		return
 	}
 
 	// If ProposalBlock is nil, prevote nil.
 	if cs.ProposalBlock == nil {
-		cs.Logger.Info("enterPrevote: ProposalBlock is nil")
+		logger.Info("enterPrevote: ProposalBlock is nil")
 		cs.signAddVote(types.VoteTypePrevote, nil, types.PartSetHeader{})
 		return
 	}
 
-	// Valdiate proposal block
+	// Validate proposal block
 	err := cs.state.ValidateBlock(cs.ProposalBlock)
 	if err != nil {
 		// ProposalBlock is invalid, prevote nil.
-		cs.Logger.Error("enterPrevote: ProposalBlock is invalid", "error", err)
+		logger.Error("enterPrevote: ProposalBlock is invalid", "err", err)
 		cs.signAddVote(types.VoteTypePrevote, nil, types.PartSetHeader{})
 		return
 	}
@@ -938,13 +928,13 @@ func (cs *ConsensusState) defaultDoPrevote(height int, round int) {
 	// Prevote cs.ProposalBlock
 	// NOTE: the proposal signature is validated when it is received,
 	// and the proposal block parts are validated as they are received (against the merkle hash in the proposal)
+	logger.Info("enterPrevote: ProposalBlock is valid")
 	cs.signAddVote(types.VoteTypePrevote, cs.ProposalBlock.Hash(), cs.ProposalBlockParts.Header())
-	return
 }
 
 // Enter: any +2/3 prevotes at next round.
 func (cs *ConsensusState) enterPrevoteWait(height int, round int) {
-	if cs.Height != height || round < cs.Round || (cs.Round == round && RoundStepPrevoteWait <= cs.Step) {
+	if cs.Height != height || round < cs.Round || (cs.Round == round && cstypes.RoundStepPrevoteWait <= cs.Step) {
 		cs.Logger.Debug(cmn.Fmt("enterPrevoteWait(%v/%v): Invalid args. Current step: %v/%v/%v", height, round, cs.Height, cs.Round, cs.Step))
 		return
 	}
@@ -955,22 +945,22 @@ func (cs *ConsensusState) enterPrevoteWait(height int, round int) {
 
 	defer func() {
 		// Done enterPrevoteWait:
-		cs.updateRoundStep(round, RoundStepPrevoteWait)
+		cs.updateRoundStep(round, cstypes.RoundStepPrevoteWait)
 		cs.newStep()
 	}()
 
 	// Wait for some more prevotes; enterPrecommit
-	cs.scheduleTimeout(cs.config.Prevote(round), height, round, RoundStepPrevoteWait)
+	cs.scheduleTimeout(cs.config.Prevote(round), height, round, cstypes.RoundStepPrevoteWait)
 }
 
-// Enter: +2/3 precomits for block or nil.
 // Enter: `timeoutPrevote` after any +2/3 prevotes.
+// Enter: +2/3 precomits for block or nil.
 // Enter: any +2/3 precommits for next round.
 // Lock & precommit the ProposalBlock if we have enough prevotes for it (a POL in this round)
 // else, unlock an existing lock and precommit nil if +2/3 of prevotes were nil,
 // else, precommit nil otherwise.
 func (cs *ConsensusState) enterPrecommit(height int, round int) {
-	if cs.Height != height || round < cs.Round || (cs.Round == round && RoundStepPrecommit <= cs.Step) {
+	if cs.Height != height || round < cs.Round || (cs.Round == round && cstypes.RoundStepPrecommit <= cs.Step) {
 		cs.Logger.Debug(cmn.Fmt("enterPrecommit(%v/%v): Invalid args. Current step: %v/%v/%v", height, round, cs.Height, cs.Round, cs.Step))
 		return
 	}
@@ -979,7 +969,7 @@ func (cs *ConsensusState) enterPrecommit(height int, round int) {
 
 	defer func() {
 		// Done enterPrecommit:
-		cs.updateRoundStep(round, RoundStepPrecommit)
+		cs.updateRoundStep(round, cstypes.RoundStepPrecommit)
 		cs.newStep()
 	}()
 
@@ -1059,12 +1049,11 @@ func (cs *ConsensusState) enterPrecommit(height int, round int) {
 	}
 	types.FireEventUnlock(cs.evsw, cs.RoundStateEvent())
 	cs.signAddVote(types.VoteTypePrecommit, nil, types.PartSetHeader{})
-	return
 }
 
 // Enter: any +2/3 precommits for next round.
 func (cs *ConsensusState) enterPrecommitWait(height int, round int) {
-	if cs.Height != height || round < cs.Round || (cs.Round == round && RoundStepPrecommitWait <= cs.Step) {
+	if cs.Height != height || round < cs.Round || (cs.Round == round && cstypes.RoundStepPrecommitWait <= cs.Step) {
 		cs.Logger.Debug(cmn.Fmt("enterPrecommitWait(%v/%v): Invalid args. Current step: %v/%v/%v", height, round, cs.Height, cs.Round, cs.Step))
 		return
 	}
@@ -1075,18 +1064,18 @@ func (cs *ConsensusState) enterPrecommitWait(height int, round int) {
 
 	defer func() {
 		// Done enterPrecommitWait:
-		cs.updateRoundStep(round, RoundStepPrecommitWait)
+		cs.updateRoundStep(round, cstypes.RoundStepPrecommitWait)
 		cs.newStep()
 	}()
 
 	// Wait for some more precommits; enterNewRound
-	cs.scheduleTimeout(cs.config.Precommit(round), height, round, RoundStepPrecommitWait)
+	cs.scheduleTimeout(cs.config.Precommit(round), height, round, cstypes.RoundStepPrecommitWait)
 
 }
 
 // Enter: +2/3 precommits for block
 func (cs *ConsensusState) enterCommit(height int, commitRound int) {
-	if cs.Height != height || RoundStepCommit <= cs.Step {
+	if cs.Height != height || cstypes.RoundStepCommit <= cs.Step {
 		cs.Logger.Debug(cmn.Fmt("enterCommit(%v/%v): Invalid args. Current step: %v/%v/%v", height, commitRound, cs.Height, cs.Round, cs.Step))
 		return
 	}
@@ -1095,7 +1084,7 @@ func (cs *ConsensusState) enterCommit(height int, commitRound int) {
 	defer func() {
 		// Done enterCommit:
 		// keep cs.Round the same, commitRound points to the right Precommits set.
-		cs.updateRoundStep(cs.Round, RoundStepCommit)
+		cs.updateRoundStep(cs.Round, cstypes.RoundStepCommit)
 		cs.CommitRound = commitRound
 		cs.CommitTime = time.Now()
 		cs.newStep()
@@ -1144,7 +1133,7 @@ func (cs *ConsensusState) tryFinalizeCommit(height int) {
 	if !cs.ProposalBlock.HashesTo(blockID.Hash) {
 		// TODO: this happens every time if we're not a validator (ugly logs)
 		// TODO: ^^ wait, why does it matter that we're a validator?
-		cs.Logger.Error("Attempt to finalize failed. We don't have the commit block.", "height", height, "proposal-block", cs.ProposalBlock.Hash(), "commit-block", blockID.Hash)
+		cs.Logger.Info("Attempt to finalize failed. We don't have the commit block.", "height", height, "proposal-block", cs.ProposalBlock.Hash(), "commit-block", blockID.Hash)
 		return
 	}
 
@@ -1152,9 +1141,9 @@ func (cs *ConsensusState) tryFinalizeCommit(height int) {
 	cs.finalizeCommit(height)
 }
 
-// Increment height and goto RoundStepNewHeight
+// Increment height and goto cstypes.RoundStepNewHeight
 func (cs *ConsensusState) finalizeCommit(height int) {
-	if cs.Height != height || cs.Step != RoundStepCommit {
+	if cs.Height != height || cs.Step != cstypes.RoundStepCommit {
 		cs.Logger.Debug(cmn.Fmt("finalizeCommit(%v): Invalid args. Current step: %v/%v/%v", height, cs.Height, cs.Round, cs.Step))
 		return
 	}
@@ -1203,7 +1192,7 @@ func (cs *ConsensusState) finalizeCommit(height int) {
 	// As is, ConsensusState should not be started again
 	// until we successfully call ApplyBlock (ie. here or in Handshake after restart)
 	if cs.wal != nil {
-		cs.wal.writeEndHeight(height)
+		cs.wal.Save(EndHeightMessage{uint64(height)})
 	}
 
 	fail.Fail() // XXX
@@ -1218,7 +1207,7 @@ func (cs *ConsensusState) finalizeCommit(height int) {
 	// NOTE: the block.AppHash wont reflect these txs until the next block
 	err := stateCopy.ApplyBlock(eventCache, cs.proxyAppConn, block, blockParts.Header(), cs.mempool)
 	if err != nil {
-		cs.Logger.Error("Error on ApplyBlock. Did the application crash? Please restart tendermint", "error", err)
+		cs.Logger.Error("Error on ApplyBlock. Did the application crash? Please restart tendermint", "err", err)
 		return
 	}
 
@@ -1248,9 +1237,8 @@ func (cs *ConsensusState) finalizeCommit(height int) {
 
 	// By here,
 	// * cs.Height has been increment to height+1
-	// * cs.Step is now RoundStepNewHeight
+	// * cs.Step is now cstypes.RoundStepNewHeight
 	// * cs.StartTime is set to when we will start round0.
-	return
 }
 
 //-----------------------------------------------------------------------------
@@ -1267,8 +1255,8 @@ func (cs *ConsensusState) defaultSetProposal(proposal *types.Proposal) error {
 		return nil
 	}
 
-	// We don't care about the proposal if we're already in RoundStepCommit.
-	if RoundStepCommit <= cs.Step {
+	// We don't care about the proposal if we're already in cstypes.RoundStepCommit.
+	if cstypes.RoundStepCommit <= cs.Step {
 		return nil
 	}
 
@@ -1309,13 +1297,14 @@ func (cs *ConsensusState) addProposalBlockPart(height int, part *types.Part, ver
 		// Added and completed!
 		var n int
 		var err error
-		cs.ProposalBlock = wire.ReadBinary(&types.Block{}, cs.ProposalBlockParts.GetReader(), types.MaxBlockSize, &n, &err).(*types.Block)
+		cs.ProposalBlock = wire.ReadBinary(&types.Block{}, cs.ProposalBlockParts.GetReader(),
+			cs.state.Params.BlockSizeParams.MaxBytes, &n, &err).(*types.Block)
 		// NOTE: it's possible to receive complete proposal blocks for future rounds without having the proposal
 		cs.Logger.Info("Received complete proposal block", "height", cs.ProposalBlock.Height, "hash", cs.ProposalBlock.Hash())
-		if cs.Step == RoundStepPropose && cs.isProposalComplete() {
+		if cs.Step == cstypes.RoundStepPropose && cs.isProposalComplete() {
 			// Move onto the next step
 			cs.enterPrevote(height, cs.Round)
-		} else if cs.Step == RoundStepCommit {
+		} else if cs.Step == cstypes.RoundStepCommit {
 			// If we're waiting on the proposal block...
 			cs.tryFinalizeCommit(height)
 		}
@@ -1334,23 +1323,18 @@ func (cs *ConsensusState) tryAddVote(vote *types.Vote, peerKey string) error {
 		if err == ErrVoteHeightMismatch {
 			return err
 		} else if _, ok := err.(*types.ErrVoteConflictingVotes); ok {
-			if peerKey == "" {
+			if bytes.Equal(vote.ValidatorAddress, cs.privValidator.GetAddress()) {
 				cs.Logger.Error("Found conflicting vote from ourselves. Did you unsafe_reset a validator?", "height", vote.Height, "round", vote.Round, "type", vote.Type)
 				return err
 			}
-			cs.Logger.Error("Found conflicting vote. Publish evidence (TODO)")
-			/* TODO
-			evidenceTx := &types.DupeoutTx{
-				Address: address,
-				VoteA:   *errDupe.VoteA,
-				VoteB:   *errDupe.VoteB,
-			}
-			cs.mempool.BroadcastTx(struct{???}{evidenceTx}) // shouldn't need to check returned err
-			*/
+			cs.Logger.Error("Found conflicting vote. Publish evidence (TODO)", "height", vote.Height, "round", vote.Round, "type", vote.Type, "valAddr", vote.ValidatorAddress, "valIndex", vote.ValidatorIndex)
+
+			// TODO: track evidence for inclusion in a block
+
 			return err
 		} else {
 			// Probably an invalid signature. Bad peer.
-			cs.Logger.Error("Error attempting to add vote", "error", err)
+			cs.Logger.Error("Error attempting to add vote", "err", err)
 			return ErrAddingVote
 		}
 	}
@@ -1360,12 +1344,12 @@ func (cs *ConsensusState) tryAddVote(vote *types.Vote, peerKey string) error {
 //-----------------------------------------------------------------------------
 
 func (cs *ConsensusState) addVote(vote *types.Vote, peerKey string) (added bool, err error) {
-	cs.Logger.Debug("addVote", "voteHeight", vote.Height, "voteType", vote.Type, "csHeight", cs.Height)
+	cs.Logger.Debug("addVote", "voteHeight", vote.Height, "voteType", vote.Type, "valIndex", vote.ValidatorIndex, "csHeight", cs.Height)
 
 	// A precommit for the previous height?
 	// These come in while we wait timeoutCommit
 	if vote.Height+1 == cs.Height {
-		if !(cs.Step == RoundStepNewHeight && vote.Type == types.VoteTypePrecommit) {
+		if !(cs.Step == cstypes.RoundStepNewHeight && vote.Type == types.VoteTypePrecommit) {
 			// TODO: give the reason ..
 			// fmt.Errorf("tryAddVote: Wrong height, not a LastCommit straggler commit.")
 			return added, ErrVoteHeightMismatch
@@ -1378,7 +1362,7 @@ func (cs *ConsensusState) addVote(vote *types.Vote, peerKey string) (added bool,
 			// if we can skip timeoutCommit and have all the votes now,
 			if cs.config.SkipTimeoutCommit && cs.LastCommit.HasAll() {
 				// go straight to new round (skip timeout commit)
-				// cs.scheduleTimeout(time.Duration(0), cs.Height, 0, RoundStepNewHeight)
+				// cs.scheduleTimeout(time.Duration(0), cs.Height, 0, cstypes.RoundStepNewHeight)
 				cs.enterNewRound(cs.Height, 0)
 			}
 		}
@@ -1442,7 +1426,7 @@ func (cs *ConsensusState) addVote(vote *types.Vote, peerKey string) (added bool,
 						if cs.config.SkipTimeoutCommit && precommits.HasAll() {
 							// if we have all the votes now,
 							// go straight to new round (skip timeout commit)
-							// cs.scheduleTimeout(time.Duration(0), cs.Height, 0, RoundStepNewHeight)
+							// cs.scheduleTimeout(time.Duration(0), cs.Height, 0, cstypes.RoundStepNewHeight)
 							cs.enterNewRound(cs.Height, 0)
 						}
 
@@ -1491,11 +1475,11 @@ func (cs *ConsensusState) signAddVote(type_ byte, hash []byte, header types.Part
 	vote, err := cs.signVote(type_, hash, header)
 	if err == nil {
 		cs.sendInternalMessage(msgInfo{&VoteMessage{vote}, ""})
-		cs.Logger.Info("Signed and pushed vote", "height", cs.Height, "round", cs.Round, "vote", vote, "error", err)
+		cs.Logger.Info("Signed and pushed vote", "height", cs.Height, "round", cs.Round, "vote", vote, "err", err)
 		return vote
 	} else {
 		//if !cs.replayMode {
-		cs.Logger.Error("Error signing vote", "height", cs.Height, "round", cs.Round, "vote", vote, "error", err)
+		cs.Logger.Error("Error signing vote", "height", cs.Height, "round", cs.Round, "vote", vote, "err", err)
 		//}
 		return nil
 	}
@@ -1503,7 +1487,7 @@ func (cs *ConsensusState) signAddVote(type_ byte, hash []byte, header types.Part
 
 //---------------------------------------------------------
 
-func CompareHRS(h1, r1 int, s1 RoundStepType, h2, r2 int, s2 RoundStepType) int {
+func CompareHRS(h1, r1 int, s1 cstypes.RoundStepType, h2, r2 int, s2 cstypes.RoundStepType) int {
 	if h1 < h2 {
 		return -1
 	} else if h1 > h2 {

consensus/state_test.go

@@ -6,8 +6,9 @@ import (
 	"testing"
 	"time"
 
+	cstypes "github.com/tendermint/tendermint/consensus/types"
 	"github.com/tendermint/tendermint/types"
-	. "github.com/tendermint/tmlibs/common"
+	cmn "github.com/tendermint/tmlibs/common"
 )
 
 func init() {
@@ -79,8 +80,8 @@ func TestProposerSelection0(t *testing.T) {
 	<-newRoundCh
 
 	prop = cs1.GetRoundState().Validators.GetProposer()
-	if !bytes.Equal(prop.Address, vss[1].Address) {
-		panic(Fmt("expected proposer to be validator %d. Got %X", 1, prop.Address))
+	if !bytes.Equal(prop.Address, vss[1].GetAddress()) {
+		panic(cmn.Fmt("expected proposer to be validator %d. Got %X", 1, prop.Address))
 	}
 }
 
@@ -100,8 +101,8 @@ func TestProposerSelection2(t *testing.T) {
 	// everyone just votes nil. we get a new proposer each round
 	for i := 0; i < len(vss); i++ {
 		prop := cs1.GetRoundState().Validators.GetProposer()
-		if !bytes.Equal(prop.Address, vss[(i+2)%len(vss)].Address) {
-			panic(Fmt("expected proposer to be validator %d. Got %X", (i+2)%len(vss), prop.Address))
+		if !bytes.Equal(prop.Address, vss[(i+2)%len(vss)].GetAddress()) {
+			panic(cmn.Fmt("expected proposer to be validator %d. Got %X", (i+2)%len(vss), prop.Address))
 		}
 
 		rs := cs1.GetRoundState()
@@ -180,7 +181,7 @@ func TestBadProposal(t *testing.T) {
 	height, round := cs1.Height, cs1.Round
 	vs2 := vss[1]
 
-	partSize := config.Consensus.BlockPartSize
+	partSize := cs1.state.Params.BlockPartSizeBytes
 
 	proposalCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringCompleteProposal(), 1)
 	voteCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringVote(), 1)
@@ -247,7 +248,7 @@ func TestFullRound1(t *testing.T) {
 
 	// grab proposal
 	re := <-propCh
-	propBlockHash := re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*RoundState).ProposalBlock.Hash()
+	propBlockHash := re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*cstypes.RoundState).ProposalBlock.Hash()
 
 	<-voteCh // wait for prevote
 	// NOTE: voteChan cap of 0 ensures we can complete this
@@ -327,7 +328,7 @@ func TestLockNoPOL(t *testing.T) {
 	vs2 := vss[1]
 	height := cs1.Height
 
-	partSize := config.Consensus.BlockPartSize
+	partSize := cs1.state.Params.BlockPartSizeBytes
 
 	timeoutProposeCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringTimeoutPropose(), 1)
 	timeoutWaitCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringTimeoutWait(), 1)
@@ -344,7 +345,7 @@ func TestLockNoPOL(t *testing.T) {
 	cs1.startRoutines(0)
 
 	re := <-proposalCh
-	rs := re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*RoundState)
+	rs := re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*cstypes.RoundState)
 	theBlockHash := rs.ProposalBlock.Hash()
 
 	<-voteCh // prevote
@@ -384,7 +385,7 @@ func TestLockNoPOL(t *testing.T) {
 
 	// now we're on a new round and not the proposer, so wait for timeout
 	re = <-timeoutProposeCh
-	rs = re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*RoundState)
+	rs = re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*cstypes.RoundState)
 
 	if rs.ProposalBlock != nil {
 		panic("Expected proposal block to be nil")
@@ -428,11 +429,11 @@ func TestLockNoPOL(t *testing.T) {
 	incrementRound(vs2)
 
 	re = <-proposalCh
-	rs = re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*RoundState)
+	rs = re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*cstypes.RoundState)
 
 	// now we're on a new round and are the proposer
 	if !bytes.Equal(rs.ProposalBlock.Hash(), rs.LockedBlock.Hash()) {
-		panic(Fmt("Expected proposal block to be locked block. Got %v, Expected %v", rs.ProposalBlock, rs.LockedBlock))
+		panic(cmn.Fmt("Expected proposal block to be locked block. Got %v, Expected %v", rs.ProposalBlock, rs.LockedBlock))
 	}
 
 	<-voteCh // prevote
@@ -493,7 +494,7 @@ func TestLockPOLRelock(t *testing.T) {
 	cs1, vss := randConsensusState(4)
 	vs2, vs3, vs4 := vss[1], vss[2], vss[3]
 
-	partSize := config.Consensus.BlockPartSize
+	partSize := cs1.state.Params.BlockPartSizeBytes
 
 	timeoutProposeCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringTimeoutPropose(), 1)
 	timeoutWaitCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringTimeoutWait(), 1)
@@ -502,8 +503,6 @@ func TestLockPOLRelock(t *testing.T) {
 	newRoundCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringNewRound(), 1)
 	newBlockCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringNewBlockHeader(), 1)
 
-	t.Logf("vs2 last round %v", vs2.PrivValidator.LastRound)
-
 	// everything done from perspective of cs1
 
 	/*
@@ -517,13 +516,14 @@ func TestLockPOLRelock(t *testing.T) {
 
 	<-newRoundCh
 	re := <-proposalCh
-	rs := re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*RoundState)
+	rs := re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*cstypes.RoundState)
 	theBlockHash := rs.ProposalBlock.Hash()
 
 	<-voteCh // prevote
 
 	signAddVotes(cs1, types.VoteTypePrevote, cs1.ProposalBlock.Hash(), cs1.ProposalBlockParts.Header(), vs2, vs3, vs4)
-	_, _, _ = <-voteCh, <-voteCh, <-voteCh // prevotes
+	// prevotes
+	discardFromChan(voteCh, 3)
 
 	<-voteCh // our precommit
 	// the proposed block should now be locked and our precommit added
@@ -532,7 +532,8 @@ func TestLockPOLRelock(t *testing.T) {
 	// add precommits from the rest
 	signAddVotes(cs1, types.VoteTypePrecommit, nil, types.PartSetHeader{}, vs2, vs4)
 	signAddVotes(cs1, types.VoteTypePrecommit, cs1.ProposalBlock.Hash(), cs1.ProposalBlockParts.Header(), vs3)
-	_, _, _ = <-voteCh, <-voteCh, <-voteCh // precommits
+	// precommites
+	discardFromChan(voteCh, 3)
 
 	// before we timeout to the new round set the new proposal
 	prop, propBlock := decideProposal(cs1, vs2, vs2.Height, vs2.Round+1)
@@ -544,7 +545,7 @@ func TestLockPOLRelock(t *testing.T) {
 	// timeout to new round
 	<-timeoutWaitCh
 
-	//XXX: this isnt gauranteed to get there before the timeoutPropose ...
+	//XXX: this isnt guaranteed to get there before the timeoutPropose ...
 	cs1.SetProposalAndBlock(prop, propBlock, propBlockParts, "some peer")
 
 	<-newRoundCh
@@ -570,7 +571,8 @@ func TestLockPOLRelock(t *testing.T) {
 
 	// now lets add prevotes from everyone else for the new block
 	signAddVotes(cs1, types.VoteTypePrevote, propBlockHash, propBlockParts.Header(), vs2, vs3, vs4)
-	_, _, _ = <-voteCh, <-voteCh, <-voteCh // prevotes
+	// prevotes
+	discardFromChan(voteCh, 3)
 
 	// now either we go to PrevoteWait or Precommit
 	select {
@@ -585,12 +587,12 @@ func TestLockPOLRelock(t *testing.T) {
 	validatePrecommit(t, cs1, 1, 1, vss[0], propBlockHash, propBlockHash)
 
 	signAddVotes(cs1, types.VoteTypePrecommit, propBlockHash, propBlockParts.Header(), vs2, vs3)
-	_, _ = <-voteCh, <-voteCh
+	discardFromChan(voteCh, 2)
 
 	be := <-newBlockCh
 	b := be.(types.TMEventData).Unwrap().(types.EventDataNewBlockHeader)
 	re = <-newRoundCh
-	rs = re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*RoundState)
+	rs = re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*cstypes.RoundState)
 	if rs.Height != 2 {
 		panic("Expected height to increment")
 	}
@@ -605,7 +607,7 @@ func TestLockPOLUnlock(t *testing.T) {
 	cs1, vss := randConsensusState(4)
 	vs2, vs3, vs4 := vss[1], vss[2], vss[3]
 
-	partSize := config.Consensus.BlockPartSize
+	partSize := cs1.state.Params.BlockPartSizeBytes
 
 	proposalCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringCompleteProposal(), 1)
 	timeoutProposeCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringTimeoutPropose(), 1)
@@ -626,7 +628,7 @@ func TestLockPOLUnlock(t *testing.T) {
 	startTestRound(cs1, cs1.Height, 0)
 	<-newRoundCh
 	re := <-proposalCh
-	rs := re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*RoundState)
+	rs := re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*cstypes.RoundState)
 	theBlockHash := rs.ProposalBlock.Hash()
 
 	<-voteCh // prevote
@@ -652,10 +654,10 @@ func TestLockPOLUnlock(t *testing.T) {
 
 	// timeout to new round
 	re = <-timeoutWaitCh
-	rs = re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*RoundState)
+	rs = re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*cstypes.RoundState)
 	lockedBlockHash := rs.LockedBlock.Hash()
 
-	//XXX: this isnt gauranteed to get there before the timeoutPropose ...
+	//XXX: this isnt guaranteed to get there before the timeoutPropose ...
 	cs1.SetProposalAndBlock(prop, propBlock, propBlockParts, "some peer")
 
 	<-newRoundCh
@@ -700,7 +702,7 @@ func TestLockPOLSafety1(t *testing.T) {
 	cs1, vss := randConsensusState(4)
 	vs2, vs3, vs4 := vss[1], vss[2], vss[3]
 
-	partSize := config.Consensus.BlockPartSize
+	partSize := cs1.state.Params.BlockPartSizeBytes
 
 	proposalCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringCompleteProposal(), 1)
 	timeoutProposeCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringTimeoutPropose(), 1)
@@ -712,7 +714,7 @@ func TestLockPOLSafety1(t *testing.T) {
 	startTestRound(cs1, cs1.Height, 0)
 	<-newRoundCh
 	re := <-proposalCh
-	rs := re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*RoundState)
+	rs := re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*cstypes.RoundState)
 	propBlock := rs.ProposalBlock
 
 	<-voteCh // prevote
@@ -742,7 +744,7 @@ func TestLockPOLSafety1(t *testing.T) {
 
 	incrementRound(vs2, vs3, vs4)
 
-	//XXX: this isnt gauranteed to get there before the timeoutPropose ...
+	//XXX: this isnt guaranteed to get there before the timeoutPropose ...
 	cs1.SetProposalAndBlock(prop, propBlock, propBlockParts, "some peer")
 
 	<-newRoundCh
@@ -760,7 +762,7 @@ func TestLockPOLSafety1(t *testing.T) {
 		re = <-proposalCh
 	}
 
-	rs = re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*RoundState)
+	rs = re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*cstypes.RoundState)
 
 	if rs.LockedBlock != nil {
 		panic("we should not be locked!")
@@ -821,7 +823,7 @@ func TestLockPOLSafety2(t *testing.T) {
 	cs1, vss := randConsensusState(4)
 	vs2, vs3, vs4 := vss[1], vss[2], vss[3]
 
-	partSize := config.Consensus.BlockPartSize
+	partSize := cs1.state.Params.BlockPartSizeBytes
 
 	proposalCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringCompleteProposal(), 1)
 	timeoutProposeCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringTimeoutPropose(), 1)
@@ -847,7 +849,7 @@ func TestLockPOLSafety2(t *testing.T) {
 
 	incrementRound(vs2, vs3, vs4)
 
-	cs1.updateRoundStep(0, RoundStepPrecommitWait)
+	cs1.updateRoundStep(0, cstypes.RoundStepPrecommitWait)
 
 	t.Log("### ONTO Round 1")
 	// jump in at round 1
@@ -928,7 +930,7 @@ func TestSlashingPrevotes(t *testing.T) {
 	re := <-proposalCh
 	<-voteCh // prevote
 
-	rs := re.(types.EventDataRoundState).RoundState.(*RoundState)
+	rs := re.(types.EventDataRoundState).RoundState.(*cstypes.RoundState)
 
 	// we should now be stuck in limbo forever, waiting for more prevotes
 	// add one for a different block should cause us to go into prevote wait
@@ -996,7 +998,7 @@ func TestHalt1(t *testing.T) {
 	cs1, vss := randConsensusState(4)
 	vs2, vs3, vs4 := vss[1], vss[2], vss[3]
 
-	partSize := config.Consensus.BlockPartSize
+	partSize := cs1.state.Params.BlockPartSizeBytes
 
 	proposalCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringCompleteProposal(), 1)
 	timeoutWaitCh := subscribeToEvent(cs1.evsw, "tester", types.EventStringTimeoutWait(), 1)
@@ -1008,7 +1010,7 @@ func TestHalt1(t *testing.T) {
 	startTestRound(cs1, cs1.Height, 0)
 	<-newRoundCh
 	re := <-proposalCh
-	rs := re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*RoundState)
+	rs := re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*cstypes.RoundState)
 	propBlock := rs.ProposalBlock
 	propBlockParts := propBlock.MakePartSet(partSize)
 
@@ -1031,7 +1033,7 @@ func TestHalt1(t *testing.T) {
 	// timeout to new round
 	<-timeoutWaitCh
 	re = <-newRoundCh
-	rs = re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*RoundState)
+	rs = re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*cstypes.RoundState)
 
 	t.Log("### ONTO ROUND 1")
 	/*Round2
@@ -1049,7 +1051,7 @@ func TestHalt1(t *testing.T) {
 	// receiving that precommit should take us straight to commit
 	<-newBlockCh
 	re = <-newRoundCh
-	rs = re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*RoundState)
+	rs = re.(types.TMEventData).Unwrap().(types.EventDataRoundState).RoundState.(*cstypes.RoundState)
 
 	if rs.Height != 2 {
 		panic("expected height to increment")

consensus/test_data/build.sh

@@ -1,88 +1,125 @@
 #!/usr/bin/env bash
 
-# XXX: removes tendermint dir
+# Requires: killall command and jq JSON processor.
 
-cd "$GOPATH/src/github.com/tendermint/tendermint" || exit 1
+# Get the parent directory of where this script is.
+SOURCE="${BASH_SOURCE[0]}"
+while [ -h "$SOURCE" ] ; do SOURCE="$(readlink "$SOURCE")"; done
+DIR="$( cd -P "$( dirname "$SOURCE" )/../.." && pwd )"
+
+# Change into that dir because we expect that.
+cd "$DIR" || exit 1
 
 # Make sure we have a tendermint command.
 if ! hash tendermint 2>/dev/null; then
 	make install
 fi
 
-# specify a dir to copy
+# Make sure we have a cutWALUntil binary.
+cutWALUntil=./scripts/cutWALUntil/cutWALUntil
+cutWALUntilDir=$(dirname $cutWALUntil)
+if ! hash $cutWALUntil 2>/dev/null; then
+	cd "$cutWALUntilDir" && go build && cd - || exit 1
+fi
+
+TMHOME=$(mktemp -d)
+export TMHOME="$TMHOME"
+
+if [[ ! -d "$TMHOME" ]]; then
+	echo "Could not create temp directory"
+  exit 1
+else
+	echo "TMHOME: ${TMHOME}"
+fi
+
 # TODO: eventually we should replace with `tendermint init --test`
 DIR_TO_COPY=$HOME/.tendermint_test/consensus_state_test
+if [ ! -d "$DIR_TO_COPY" ]; then
+    echo "$DIR_TO_COPY does not exist. Please run: go test ./consensus"
+    exit 1
+fi
+echo "==> Copying ${DIR_TO_COPY} to ${TMHOME} directory..."
+cp -r "$DIR_TO_COPY"/* "$TMHOME"
 
-TMHOME="$HOME/.tendermint"
-rm -rf "$TMHOME"
-cp -r "$DIR_TO_COPY" "$TMHOME"
-cp $TMHOME/config.toml $TMHOME/config.toml.bak
+# preserve original genesis file because later it will be modified (see small_block2)
+cp "$TMHOME/genesis.json" "$TMHOME/genesis.json.bak"
 
 function reset(){
+	echo "==> Resetting tendermint..."
 	tendermint unsafe_reset_all
-	cp $TMHOME/config.toml.bak $TMHOME/config.toml
+	cp "$TMHOME/genesis.json.bak" "$TMHOME/genesis.json"
 }
 
 reset
 
-# empty block
 function empty_block(){
-tendermint node --proxy_app=persistent_dummy &> /dev/null &
-sleep 5
-killall tendermint
+	echo "==> Starting tendermint..."
+	tendermint node --proxy_app=persistent_dummy &> /dev/null &
+	sleep 5
+	echo "==> Killing tendermint..."
+	killall tendermint
 
-# /q would print up to and including the match, then quit.
-# /Q doesn't include the match.
-# http://unix.stackexchange.com/questions/11305/grep-show-all-the-file-up-to-the-match
-sed '/ENDHEIGHT: 1/Q' ~/.tendermint/data/cs.wal/wal  > consensus/test_data/empty_block.cswal
+	echo "==> Copying WAL log..."
+	$cutWALUntil "$TMHOME/data/cs.wal/wal" 1 consensus/test_data/new_empty_block.cswal
+	mv consensus/test_data/new_empty_block.cswal consensus/test_data/empty_block.cswal
 
-reset
+	reset
 }
 
-# many blocks
 function many_blocks(){
-bash scripts/txs/random.sh 1000 36657 &> /dev/null &
-PID=$!
-tendermint node --proxy_app=persistent_dummy &> /dev/null &
-sleep 7
-killall tendermint
-kill -9 $PID
+	bash scripts/txs/random.sh 1000 36657 &> /dev/null &
+	PID=$!
+	echo "==> Starting tendermint..."
+	tendermint node --proxy_app=persistent_dummy &> /dev/null &
+	sleep 10
+	echo "==> Killing tendermint..."
+	kill -9 $PID
+	killall tendermint
 
-sed '/ENDHEIGHT: 6/Q' ~/.tendermint/data/cs.wal/wal  > consensus/test_data/many_blocks.cswal
+	echo "==> Copying WAL log..."
+	$cutWALUntil "$TMHOME/data/cs.wal/wal" 6 consensus/test_data/new_many_blocks.cswal
+	mv consensus/test_data/new_many_blocks.cswal consensus/test_data/many_blocks.cswal
 
-reset
+	reset
 }
 
 
-# small block 1
 function small_block1(){
-bash scripts/txs/random.sh 1000 36657 &> /dev/null &
-PID=$!
-tendermint node --proxy_app=persistent_dummy &> /dev/null &
-sleep 10
-killall tendermint
-kill -9 $PID
+	bash scripts/txs/random.sh 1000 36657 &> /dev/null &
+	PID=$!
+	echo "==> Starting tendermint..."
+	tendermint node --proxy_app=persistent_dummy &> /dev/null &
+	sleep 10
+	echo "==> Killing tendermint..."
+	kill -9 $PID
+	killall tendermint
 
-sed '/ENDHEIGHT: 1/Q' ~/.tendermint/data/cs.wal/wal  > consensus/test_data/small_block1.cswal
+	echo "==> Copying WAL log..."
+	$cutWALUntil "$TMHOME/data/cs.wal/wal" 1 consensus/test_data/new_small_block1.cswal
+	mv consensus/test_data/new_small_block1.cswal consensus/test_data/small_block1.cswal
 
-reset
+	reset
 }
 
 
-# small block 2 (part size = 512)
+# block part size = 512
 function small_block2(){
-echo "" >> ~/.tendermint/config.toml
-echo "block_part_size = 512" >> ~/.tendermint/config.toml
-bash scripts/txs/random.sh 1000 36657 &> /dev/null &
-PID=$!
-tendermint node --proxy_app=persistent_dummy &> /dev/null &
-sleep 5
-killall tendermint
-kill -9 $PID
+	cat "$TMHOME/genesis.json" | jq '. + {consensus_params: {block_size_params: {max_bytes: 22020096}, block_gossip_params: {block_part_size_bytes: 512}}}' > "$TMHOME/new_genesis.json"
+	mv "$TMHOME/new_genesis.json" "$TMHOME/genesis.json"
+	bash scripts/txs/random.sh 1000 36657 &> /dev/null &
+	PID=$!
+	echo "==> Starting tendermint..."
+	tendermint node --proxy_app=persistent_dummy &> /dev/null &
+	sleep 5
+	echo "==> Killing tendermint..."
+	kill -9 $PID
+	killall tendermint
 
-sed '/ENDHEIGHT: 1/Q' ~/.tendermint/data/cs.wal/wal  > consensus/test_data/small_block2.cswal
+	echo "==> Copying WAL log..."
+	$cutWALUntil "$TMHOME/data/cs.wal/wal" 1 consensus/test_data/new_small_block2.cswal
+	mv consensus/test_data/new_small_block2.cswal consensus/test_data/small_block2.cswal
 
-reset
+	reset
 }
 
 
@@ -107,4 +144,5 @@ case "$1" in
 		many_blocks
 esac
 
-
+echo "==> Cleaning up..."
+rm -rf "$TMHOME"

consensus/ticker.go

@@ -3,7 +3,7 @@ package consensus
 import (
 	"time"
 
-	. "github.com/tendermint/tmlibs/common"
+	cmn "github.com/tendermint/tmlibs/common"
 	"github.com/tendermint/tmlibs/log"
 )
 
@@ -29,24 +29,26 @@ type TimeoutTicker interface {
 // Timeouts are scheduled along the tickChan,
 // and fired on the tockChan.
 type timeoutTicker struct {
-	BaseService
+	cmn.BaseService
 
 	timer    *time.Timer
-	tickChan chan timeoutInfo
-	tockChan chan timeoutInfo
+	tickChan chan timeoutInfo // for scheduling timeouts
+	tockChan chan timeoutInfo // for notifying about them
 }
 
+// NewTimeoutTicker returns a new TimeoutTicker.
 func NewTimeoutTicker() TimeoutTicker {
 	tt := &timeoutTicker{
 		timer:    time.NewTimer(0),
 		tickChan: make(chan timeoutInfo, tickTockBufferSize),
 		tockChan: make(chan timeoutInfo, tickTockBufferSize),
 	}
-	tt.BaseService = *NewBaseService(nil, "TimeoutTicker", tt)
+	tt.BaseService = *cmn.NewBaseService(nil, "TimeoutTicker", tt)
 	tt.stopTimer() // don't want to fire until the first scheduled timeout
 	return tt
 }
 
+// OnStart implements cmn.Service. It starts the timeout routine.
 func (t *timeoutTicker) OnStart() error {
 
 	go t.timeoutRoutine()
@@ -54,16 +56,19 @@ func (t *timeoutTicker) OnStart() error {
 	return nil
 }
 
+// OnStop implements cmn.Service. It stops the timeout routine.
 func (t *timeoutTicker) OnStop() {
 	t.BaseService.OnStop()
 	t.stopTimer()
 }
 
+// Chan returns a channel on which timeouts are sent.
 func (t *timeoutTicker) Chan() <-chan timeoutInfo {
 	return t.tockChan
 }
 
-// The timeoutRoutine is alwaya available to read from tickChan (it won't block).
+// ScheduleTimeout schedules a new timeout by sending on the internal tickChan.
+// The timeoutRoutine is alwaya available to read from tickChan, so this won't block.
 // The scheduling may fail if the timeoutRoutine has already scheduled a timeout for a later height/round/step.
 func (t *timeoutTicker) ScheduleTimeout(ti timeoutInfo) {
 	t.tickChan <- ti
@@ -117,7 +122,7 @@ func (t *timeoutTicker) timeoutRoutine() {
 			t.Logger.Debug("Scheduled timeout", "dur", ti.Duration, "height", ti.Height, "round", ti.Round, "step", ti.Step)
 		case <-t.timer.C:
 			t.Logger.Info("Timed out", "dur", ti.Duration, "height", ti.Height, "round", ti.Round, "step", ti.Step)
-			// go routine here gaurantees timeoutRoutine doesn't block.
+			// go routine here guarantees timeoutRoutine doesn't block.
 			// Determinism comes from playback in the receiveRoutine.
 			// We can eliminate it by merging the timeoutRoutine into receiveRoutine
 			//  and managing the timeouts ourselves with a millisecond ticker

consensus/types/height_vote_set.go

@@ -1,11 +1,11 @@
-package consensus
+package types
 
 import (
 	"strings"
 	"sync"
 
 	"github.com/tendermint/tendermint/types"
-	. "github.com/tendermint/tmlibs/common"
+	cmn "github.com/tendermint/tmlibs/common"
 )
 
 type RoundVoteSet struct {
@@ -76,7 +76,7 @@ func (hvs *HeightVoteSet) SetRound(round int) {
 	hvs.mtx.Lock()
 	defer hvs.mtx.Unlock()
 	if hvs.round != 0 && (round < hvs.round+1) {
-		PanicSanity("SetRound() must increment hvs.round")
+		cmn.PanicSanity("SetRound() must increment hvs.round")
 	}
 	for r := hvs.round + 1; r <= round; r++ {
 		if _, ok := hvs.roundVoteSets[r]; ok {
@@ -89,7 +89,7 @@ func (hvs *HeightVoteSet) SetRound(round int) {
 
 func (hvs *HeightVoteSet) addRound(round int) {
 	if _, ok := hvs.roundVoteSets[round]; ok {
-		PanicSanity("addRound() for an existing round")
+		cmn.PanicSanity("addRound() for an existing round")
 	}
 	// log.Debug("addRound(round)", "round", round)
 	prevotes := types.NewVoteSet(hvs.chainID, hvs.height, round, types.VoteTypePrevote, hvs.valSet)
@@ -164,7 +164,7 @@ func (hvs *HeightVoteSet) getVoteSet(round int, type_ byte) *types.VoteSet {
 	case types.VoteTypePrecommit:
 		return rvs.Precommits
 	default:
-		PanicSanity(Fmt("Unexpected vote type %X", type_))
+		cmn.PanicSanity(cmn.Fmt("Unexpected vote type %X", type_))
 		return nil
 	}
 }
@@ -194,7 +194,7 @@ func (hvs *HeightVoteSet) StringIndented(indent string) string {
 		voteSetString = roundVoteSet.Precommits.StringShort()
 		vsStrings = append(vsStrings, voteSetString)
 	}
-	return Fmt(`HeightVoteSet{H:%v R:0~%v
+	return cmn.Fmt(`HeightVoteSet{H:%v R:0~%v
 %s  %v
 %s}`,
 		hvs.height, hvs.round,

consensus/types/height_vote_set_test.go

@@ -1,14 +1,17 @@
-package consensus
+package types
 
 import (
 	"testing"
 
+	cfg "github.com/tendermint/tendermint/config"
 	"github.com/tendermint/tendermint/types"
-	. "github.com/tendermint/tmlibs/common"
+	cmn "github.com/tendermint/tmlibs/common"
 )
 
+var config *cfg.Config // NOTE: must be reset for each _test.go file
+
 func init() {
-	config = ResetConfig("consensus_height_vote_set_test")
+	config = cfg.ResetTestRoot("consensus_height_vote_set_test")
 }
 
 func TestPeerCatchupRounds(t *testing.T) {
@@ -30,6 +33,9 @@ func TestPeerCatchupRounds(t *testing.T) {
 
 	vote1001_0 := makeVoteHR(t, 1, 1001, privVals, 0)
 	added, err = hvs.AddVote(vote1001_0, "peer1")
+	if err != nil {
+		t.Error("AddVote error", err)
+	}
 	if added {
 		t.Error("Expected to *not* add vote from peer, too many catchup rounds.")
 	}
@@ -41,10 +47,10 @@ func TestPeerCatchupRounds(t *testing.T) {
 
 }
 
-func makeVoteHR(t *testing.T, height, round int, privVals []*types.PrivValidator, valIndex int) *types.Vote {
+func makeVoteHR(t *testing.T, height, round int, privVals []*types.PrivValidatorFS, valIndex int) *types.Vote {
 	privVal := privVals[valIndex]
 	vote := &types.Vote{
-		ValidatorAddress: privVal.Address,
+		ValidatorAddress: privVal.GetAddress(),
 		ValidatorIndex:   valIndex,
 		Height:           height,
 		Round:            round,
@@ -54,7 +60,7 @@ func makeVoteHR(t *testing.T, height, round int, privVals []*types.PrivValidator
 	chainID := config.ChainID
 	err := privVal.SignVote(chainID, vote)
 	if err != nil {
-		panic(Fmt("Error signing vote: %v", err))
+		panic(cmn.Fmt("Error signing vote: %v", err))
 		return nil
 	}
 	return vote

consensus/types/reactor.go

@@ -0,0 +1,57 @@
+package types
+
+import (
+	"fmt"
+	"time"
+
+	"github.com/tendermint/tendermint/types"
+	cmn "github.com/tendermint/tmlibs/common"
+)
+
+//-----------------------------------------------------------------------------
+
+// PeerRoundState contains the known state of a peer.
+// NOTE: Read-only when returned by PeerState.GetRoundState().
+type PeerRoundState struct {
+	Height                   int                 // Height peer is at
+	Round                    int                 // Round peer is at, -1 if unknown.
+	Step                     RoundStepType       // Step peer is at
+	StartTime                time.Time           // Estimated start of round 0 at this height
+	Proposal                 bool                // True if peer has proposal for this round
+	ProposalBlockPartsHeader types.PartSetHeader //
+	ProposalBlockParts       *cmn.BitArray       //
+	ProposalPOLRound         int                 // Proposal's POL round. -1 if none.
+	ProposalPOL              *cmn.BitArray       // nil until ProposalPOLMessage received.
+	Prevotes                 *cmn.BitArray       // All votes peer has for this round
+	Precommits               *cmn.BitArray       // All precommits peer has for this round
+	LastCommitRound          int                 // Round of commit for last height. -1 if none.
+	LastCommit               *cmn.BitArray       // All commit precommits of commit for last height.
+	CatchupCommitRound       int                 // Round that we have commit for. Not necessarily unique. -1 if none.
+	CatchupCommit            *cmn.BitArray       // All commit precommits peer has for this height & CatchupCommitRound
+}
+
+// String returns a string representation of the PeerRoundState
+func (prs PeerRoundState) String() string {
+	return prs.StringIndented("")
+}
+
+// StringIndented returns a string representation of the PeerRoundState
+func (prs PeerRoundState) StringIndented(indent string) string {
+	return fmt.Sprintf(`PeerRoundState{
+%s  %v/%v/%v @%v
+%s  Proposal %v -> %v
+%s  POL      %v (round %v)
+%s  Prevotes   %v
+%s  Precommits %v
+%s  LastCommit %v (round %v)
+%s  Catchup    %v (round %v)
+%s}`,
+		indent, prs.Height, prs.Round, prs.Step, prs.StartTime,
+		indent, prs.ProposalBlockPartsHeader, prs.ProposalBlockParts,
+		indent, prs.ProposalPOL, prs.ProposalPOLRound,
+		indent, prs.Prevotes,
+		indent, prs.Precommits,
+		indent, prs.LastCommit, prs.LastCommitRound,
+		indent, prs.CatchupCommit, prs.CatchupCommitRound,
+		indent)
+}

consensus/types/state.go

@@ -0,0 +1,126 @@
+package types
+
+import (
+	"fmt"
+	"time"
+
+	"github.com/tendermint/tendermint/types"
+)
+
+//-----------------------------------------------------------------------------
+// RoundStepType enum type
+
+// RoundStepType enumerates the state of the consensus state machine
+type RoundStepType uint8 // These must be numeric, ordered.
+
+const (
+	RoundStepNewHeight     = RoundStepType(0x01) // Wait til CommitTime + timeoutCommit
+	RoundStepNewRound      = RoundStepType(0x02) // Setup new round and go to RoundStepPropose
+	RoundStepPropose       = RoundStepType(0x03) // Did propose, gossip proposal
+	RoundStepPrevote       = RoundStepType(0x04) // Did prevote, gossip prevotes
+	RoundStepPrevoteWait   = RoundStepType(0x05) // Did receive any +2/3 prevotes, start timeout
+	RoundStepPrecommit     = RoundStepType(0x06) // Did precommit, gossip precommits
+	RoundStepPrecommitWait = RoundStepType(0x07) // Did receive any +2/3 precommits, start timeout
+	RoundStepCommit        = RoundStepType(0x08) // Entered commit state machine
+	// NOTE: RoundStepNewHeight acts as RoundStepCommitWait.
+)
+
+// String returns a string
+func (rs RoundStepType) String() string {
+	switch rs {
+	case RoundStepNewHeight:
+		return "RoundStepNewHeight"
+	case RoundStepNewRound:
+		return "RoundStepNewRound"
+	case RoundStepPropose:
+		return "RoundStepPropose"
+	case RoundStepPrevote:
+		return "RoundStepPrevote"
+	case RoundStepPrevoteWait:
+		return "RoundStepPrevoteWait"
+	case RoundStepPrecommit:
+		return "RoundStepPrecommit"
+	case RoundStepPrecommitWait:
+		return "RoundStepPrecommitWait"
+	case RoundStepCommit:
+		return "RoundStepCommit"
+	default:
+		return "RoundStepUnknown" // Cannot panic.
+	}
+}
+
+//-----------------------------------------------------------------------------
+
+// RoundState defines the internal consensus state.
+// It is Immutable when returned from ConsensusState.GetRoundState()
+// TODO: Actually, only the top pointer is copied,
+// so access to field pointers is still racey
+type RoundState struct {
+	Height             int // Height we are working on
+	Round              int
+	Step               RoundStepType
+	StartTime          time.Time
+	CommitTime         time.Time // Subjective time when +2/3 precommits for Block at Round were found
+	Validators         *types.ValidatorSet
+	Proposal           *types.Proposal
+	ProposalBlock      *types.Block
+	ProposalBlockParts *types.PartSet
+	LockedRound        int
+	LockedBlock        *types.Block
+	LockedBlockParts   *types.PartSet
+	Votes              *HeightVoteSet
+	CommitRound        int            //
+	LastCommit         *types.VoteSet // Last precommits at Height-1
+	LastValidators     *types.ValidatorSet
+}
+
+// RoundStateEvent returns the H/R/S of the RoundState as an event.
+func (rs *RoundState) RoundStateEvent() types.EventDataRoundState {
+	edrs := types.EventDataRoundState{
+		Height:     rs.Height,
+		Round:      rs.Round,
+		Step:       rs.Step.String(),
+		RoundState: rs,
+	}
+	return edrs
+}
+
+// String returns a string
+func (rs *RoundState) String() string {
+	return rs.StringIndented("")
+}
+
+// StringIndented returns a string
+func (rs *RoundState) StringIndented(indent string) string {
+	return fmt.Sprintf(`RoundState{
+%s  H:%v R:%v S:%v
+%s  StartTime:     %v
+%s  CommitTime:    %v
+%s  Validators:    %v
+%s  Proposal:      %v
+%s  ProposalBlock: %v %v
+%s  LockedRound:   %v
+%s  LockedBlock:   %v %v
+%s  Votes:         %v
+%s  LastCommit: %v
+%s  LastValidators:    %v
+%s}`,
+		indent, rs.Height, rs.Round, rs.Step,
+		indent, rs.StartTime,
+		indent, rs.CommitTime,
+		indent, rs.Validators.StringIndented(indent+"    "),
+		indent, rs.Proposal,
+		indent, rs.ProposalBlockParts.StringShort(), rs.ProposalBlock.StringShort(),
+		indent, rs.LockedRound,
+		indent, rs.LockedBlockParts.StringShort(), rs.LockedBlock.StringShort(),
+		indent, rs.Votes.StringIndented(indent+"    "),
+		indent, rs.LastCommit.StringShort(),
+		indent, rs.LastValidators.StringIndented(indent+"    "),
+		indent)
+}
+
+// StringShort returns a string
+func (rs *RoundState) StringShort() string {
+	return fmt.Sprintf(`RoundState{H:%v R:%v S:%v ST:%v}`,
+		rs.Height, rs.Round, rs.Step, rs.StartTime)
+}

consensus/version.go

@@ -1,7 +1,7 @@
 package consensus
 
 import (
-	. "github.com/tendermint/tmlibs/common"
+	cmn "github.com/tendermint/tmlibs/common"
 )
 
 // kind of arbitrary
@@ -10,4 +10,4 @@ var Major = "0"    //
 var Minor = "2"    // replay refactor
 var Revision = "2" // validation -> commit
 
-var Version = Fmt("v%s/%s.%s.%s", Spec, Major, Minor, Revision)
+var Version = cmn.Fmt("v%s/%s.%s.%s", Spec, Major, Minor, Revision)

consensus/wal.go

@@ -1,22 +1,37 @@
 package consensus
 
 import (
+	"bytes"
+	"encoding/binary"
+	"fmt"
+	"hash/crc32"
+	"io"
 	"time"
 
 	wire "github.com/tendermint/go-wire"
 	"github.com/tendermint/tendermint/types"
 	auto "github.com/tendermint/tmlibs/autofile"
-	. "github.com/tendermint/tmlibs/common"
+	cmn "github.com/tendermint/tmlibs/common"
 )
 
 //--------------------------------------------------------
 // types and functions for savings consensus messages
 
+var (
+	walSeparator = []byte{55, 127, 6, 130} // 0x377f0682 - magic number
+)
+
 type TimedWALMessage struct {
-	Time time.Time  `json:"time"`
+	Time time.Time  `json:"time"` // for debugging purposes
 	Msg  WALMessage `json:"msg"`
 }
 
+// EndHeightMessage marks the end of the given height inside WAL.
+// @internal used by scripts/cutWALUntil util.
+type EndHeightMessage struct {
+	Height uint64 `json:"height"`
+}
+
 type WALMessage interface{}
 
 var _ = wire.RegisterInterface(
@@ -24,6 +39,7 @@ var _ = wire.RegisterInterface(
 	wire.ConcreteType{types.EventDataRoundState{}, 0x01},
 	wire.ConcreteType{msgInfo{}, 0x02},
 	wire.ConcreteType{timeoutInfo{}, 0x03},
+	wire.ConcreteType{EndHeightMessage{}, 0x04},
 )
 
 //--------------------------------------------------------
@@ -34,10 +50,12 @@ var _ = wire.RegisterInterface(
 // TODO: currently the wal is overwritten during replay catchup
 //   give it a mode so it's either reading or appending - must read to end to start appending again
 type WAL struct {
-	BaseService
+	cmn.BaseService
 
 	group *auto.Group
 	light bool // ignore block parts
+
+	enc *WALEncoder
 }
 
 func NewWAL(walFile string, light bool) (*WAL, error) {
@@ -48,8 +66,9 @@ func NewWAL(walFile string, light bool) (*WAL, error) {
 	wal := &WAL{
 		group: group,
 		light: light,
+		enc:   NewWALEncoder(group),
 	}
-	wal.BaseService = *NewBaseService(nil, "WAL", wal)
+	wal.BaseService = *cmn.NewBaseService(nil, "WAL", wal)
 	return wal, nil
 }
 
@@ -58,7 +77,7 @@ func (wal *WAL) OnStart() error {
 	if err != nil {
 		return err
 	} else if size == 0 {
-		wal.writeEndHeight(0)
+		wal.Save(EndHeightMessage{0})
 	}
 	_, err = wal.group.Start()
 	return err
@@ -70,35 +89,191 @@ func (wal *WAL) OnStop() {
 }
 
 // called in newStep and for each pass in receiveRoutine
-func (wal *WAL) Save(wmsg WALMessage) {
+func (wal *WAL) Save(msg WALMessage) {
 	if wal == nil {
 		return
 	}
+
 	if wal.light {
 		// in light mode we only write new steps, timeouts, and our own votes (no proposals, block parts)
-		if mi, ok := wmsg.(msgInfo); ok {
+		if mi, ok := msg.(msgInfo); ok {
 			if mi.PeerKey != "" {
 				return
 			}
 		}
 	}
+
 	// Write the wal message
-	var wmsgBytes = wire.JSONBytes(TimedWALMessage{time.Now(), wmsg})
-	err := wal.group.WriteLine(string(wmsgBytes))
+	if err := wal.enc.Encode(&TimedWALMessage{time.Now(), msg}); err != nil {
+		cmn.PanicQ(cmn.Fmt("Error writing msg to consensus wal: %v \n\nMessage: %v", err, msg))
+	}
+
+	// TODO: only flush when necessary
+	if err := wal.group.Flush(); err != nil {
+		cmn.PanicQ(cmn.Fmt("Error flushing consensus wal buf to file. Error: %v \n", err))
+	}
+}
+
+// SearchForEndHeight searches for the EndHeightMessage with the height and
+// returns an auto.GroupReader, whenever it was found or not and an error.
+// Group reader will be nil if found equals false.
+//
+// CONTRACT: caller must close group reader.
+func (wal *WAL) SearchForEndHeight(height uint64) (gr *auto.GroupReader, found bool, err error) {
+	var msg *TimedWALMessage
+
+	// NOTE: starting from the last file in the group because we're usually
+	// searching for the last height. See replay.go
+	min, max := wal.group.MinIndex(), wal.group.MaxIndex()
+	wal.Logger.Debug("Searching for height", "height", height, "min", min, "max", max)
+	for index := max; index >= min; index-- {
+		gr, err = wal.group.NewReader(index)
+		if err != nil {
+			return nil, false, err
+		}
+
+		dec := NewWALDecoder(gr)
+		for {
+			msg, err = dec.Decode()
+			if err == io.EOF {
+				// check next file
+				break
+			}
+			if err != nil {
+				gr.Close()
+				return nil, false, err
+			}
+
+			if m, ok := msg.Msg.(EndHeightMessage); ok {
+				if m.Height == height { // found
+					wal.Logger.Debug("Found", "height", height, "index", index)
+					return gr, true, nil
+				}
+			}
+		}
+
+		gr.Close()
+	}
+
+	return nil, false, nil
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+// A WALEncoder writes custom-encoded WAL messages to an output stream.
+//
+// Format: 4 bytes CRC sum + 4 bytes length + arbitrary-length value (go-wire encoded)
+type WALEncoder struct {
+	wr io.Writer
+}
+
+// NewWALEncoder returns a new encoder that writes to wr.
+func NewWALEncoder(wr io.Writer) *WALEncoder {
+	return &WALEncoder{wr}
+}
+
+// Encode writes the custom encoding of v to the stream.
+func (enc *WALEncoder) Encode(v interface{}) error {
+	data := wire.BinaryBytes(v)
+
+	crc := crc32.Checksum(data, crc32c)
+	length := uint32(len(data))
+	totalLength := 8 + int(length)
+
+	msg := make([]byte, totalLength)
+	binary.BigEndian.PutUint32(msg[0:4], crc)
+	binary.BigEndian.PutUint32(msg[4:8], length)
+	copy(msg[8:], data)
+
+	_, err := enc.wr.Write(msg)
+
+	if err == nil {
+		// TODO [Anton Kaliaev 23 Oct 2017]: remove separator
+		_, err = enc.wr.Write(walSeparator)
+	}
+
+	return err
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+// A WALDecoder reads and decodes custom-encoded WAL messages from an input
+// stream. See WALEncoder for the format used.
+//
+// It will also compare the checksums and make sure data size is equal to the
+// length from the header. If that is not the case, error will be returned.
+type WALDecoder struct {
+	rd io.Reader
+}
+
+// NewWALDecoder returns a new decoder that reads from rd.
+func NewWALDecoder(rd io.Reader) *WALDecoder {
+	return &WALDecoder{rd}
+}
+
+// Decode reads the next custom-encoded value from its reader and returns it.
+func (dec *WALDecoder) Decode() (*TimedWALMessage, error) {
+	b := make([]byte, 4)
+
+	n, err := dec.rd.Read(b)
+	if err == io.EOF {
+		return nil, err
+	}
 	if err != nil {
-		PanicQ(Fmt("Error writing msg to consensus wal. Error: %v \n\nMessage: %v", err, wmsg))
+		return nil, fmt.Errorf("failed to read checksum: %v", err)
 	}
-	// TODO: only flush when necessary
-	if err := wal.group.Flush(); err != nil {
-		PanicQ(Fmt("Error flushing consensus wal buf to file. Error: %v \n", err))
+	crc := binary.BigEndian.Uint32(b)
+
+	b = make([]byte, 4)
+	n, err = dec.rd.Read(b)
+	if err == io.EOF {
+		return nil, err
 	}
+	if err != nil {
+		return nil, fmt.Errorf("failed to read length: %v", err)
+	}
+	length := binary.BigEndian.Uint32(b)
+
+	data := make([]byte, length)
+	n, err = dec.rd.Read(data)
+	if err == io.EOF {
+		return nil, err
+	}
+	if err != nil {
+		return nil, fmt.Errorf("not enough bytes for data: %v (want: %d, read: %v)", err, length, n)
+	}
+
+	// check checksum before decoding data
+	actualCRC := crc32.Checksum(data, crc32c)
+	if actualCRC != crc {
+		return nil, fmt.Errorf("checksums do not match: (read: %v, actual: %v)", crc, actualCRC)
+	}
+
+	var nn int
+	var res *TimedWALMessage
+	res = wire.ReadBinary(&TimedWALMessage{}, bytes.NewBuffer(data), int(length), &nn, &err).(*TimedWALMessage)
+	if err != nil {
+		return nil, fmt.Errorf("failed to decode data: %v", err)
+	}
+
+	// TODO [Anton Kaliaev 23 Oct 2017]: remove separator
+	if err = readSeparator(dec.rd); err != nil {
+		return nil, err
+	}
+
+	return res, err
 }
 
-func (wal *WAL) writeEndHeight(height int) {
-	wal.group.WriteLine(Fmt("#ENDHEIGHT: %v", height))
-
-	// TODO: only flush when necessary
-	if err := wal.group.Flush(); err != nil {
-		PanicQ(Fmt("Error flushing consensus wal buf to file. Error: %v \n", err))
+// readSeparator reads a separator from r. It returns any error from underlying
+// reader or if it's not a separator.
+func readSeparator(r io.Reader) error {
+	b := make([]byte, len(walSeparator))
+	_, err := r.Read(b)
+	if err != nil {
+		return fmt.Errorf("failed to read separator: %v", err)
 	}
+	if !bytes.Equal(b, walSeparator) {
+		return fmt.Errorf("not a separator: %v", b)
+	}
+	return nil
 }

consensus/wal_test.go

@@ -0,0 +1,62 @@
+package consensus
+
+import (
+	"bytes"
+	"path"
+	"testing"
+	"time"
+
+	"github.com/tendermint/tendermint/consensus/types"
+	tmtypes "github.com/tendermint/tendermint/types"
+	cmn "github.com/tendermint/tmlibs/common"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func TestWALEncoderDecoder(t *testing.T) {
+	now := time.Now()
+	msgs := []TimedWALMessage{
+		TimedWALMessage{Time: now, Msg: EndHeightMessage{0}},
+		TimedWALMessage{Time: now, Msg: timeoutInfo{Duration: time.Second, Height: 1, Round: 1, Step: types.RoundStepPropose}},
+	}
+
+	b := new(bytes.Buffer)
+
+	for _, msg := range msgs {
+		b.Reset()
+
+		enc := NewWALEncoder(b)
+		err := enc.Encode(&msg)
+		require.NoError(t, err)
+
+		dec := NewWALDecoder(b)
+		decoded, err := dec.Decode()
+		require.NoError(t, err)
+
+		assert.Equal(t, msg.Time.Truncate(time.Millisecond), decoded.Time)
+		assert.Equal(t, msg.Msg, decoded.Msg)
+	}
+}
+
+func TestSearchForEndHeight(t *testing.T) {
+	wal, err := NewWAL(path.Join(data_dir, "many_blocks.cswal"), false)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	h := 3
+	gr, found, err := wal.SearchForEndHeight(uint64(h))
+	assert.NoError(t, err, cmn.Fmt("expected not to err on height %d", h))
+	assert.True(t, found, cmn.Fmt("expected to find end height for %d", h))
+	assert.NotNil(t, gr, "expected group not to be nil")
+	defer gr.Close()
+
+	dec := NewWALDecoder(gr)
+	msg, err := dec.Decode()
+	assert.NoError(t, err, "expected to decode a message")
+	rs, ok := msg.Msg.(tmtypes.EventDataRoundState)
+	assert.True(t, ok, "expected message of type EventDataRoundState")
+	assert.Equal(t, rs.Height, h+1, cmn.Fmt("wrong height"))
+
+}

docs/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line.
+SPHINXOPTS    =
+SPHINXBUILD   = python -msphinx
+SPHINXPROJ    = Tendermint
+SOURCEDIR     = .
+BUILDDIR      = _build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/README.md

@@ -0,0 +1,14 @@
+Here lies our documentation. After making edits, run:
+
+```
+pip install -r requirements.txt
+make html
+```
+
+to build the docs locally then open the file `_build/html/index.html` in your browser.
+
+**WARNING:** This documentation is intended to be viewed at:
+
+https://tendermint.readthedocs.io
+
+and may contain broken internal links when viewed from Github.

docs/_static/custom_collapsible_code.css

@@ -0,0 +1,17 @@
+.toggle {
+    padding-bottom: 1em ;
+}
+
+.toggle .header {
+    display: block;
+    clear: both;
+    cursor: pointer;
+}
+
+.toggle .header:after {
+    content: " ▼";
+}
+
+.toggle .header.open:after {
+    content: " ▲";
+}

docs/_static/custom_collapsible_code.js

@@ -0,0 +1,10 @@
+let makeCodeBlocksCollapsible = function() {
+    $(".toggle > *").hide();
+    $(".toggle .header").show();
+    $(".toggle .header").click(function() {
+        $(this).parent().children().not(".header").toggle({"duration": 400});
+        $(this).parent().children(".header").toggleClass("open");
+    });
+};
+// we could use the }(); way if we would have access to jQuery in HEAD, i.e. we would need to force the theme
+// to load jQuery before our custom scripts

docs/_templates/layout.html

@@ -0,0 +1,20 @@
+{% extends "!layout.html" %}
+
+{% set css_files = css_files + ["_static/custom_collapsible_code.css"] %}
+
+# sadly, I didn't find a css style way to add custom JS to a list that is automagically added to head like CSS (above) #}
+{% block extrahead %}
+<script type="text/javascript" src="_static/custom_collapsible_code.js"></script>
+{% endblock %}
+
+{% block footer %}
+<script type="text/javascript">
+    $(document).ready(function() {
+        // using this approach as we don't have access to the jQuery selectors
+        // when executing the function on load in HEAD
+        makeCodeBlocksCollapsible();
+    });
+</script>
+{% endblock %}
+
+

docs/abci-cli.rst

@@ -0,0 +1,292 @@
+Using ABCI-CLI
+==============
+
+To facilitate testing and debugging of ABCI servers and simple apps, we
+built a CLI, the ``abci-cli``, for sending ABCI messages from the
+command line.
+
+Install
+-------
+
+Make sure you `have Go installed <https://golang.org/doc/install>`__.
+
+Next, install the ``abci-cli`` tool and example applications:
+
+::
+
+    go get -u github.com/tendermint/abci/cmd/abci-cli
+
+If this fails, you may need to use ``glide`` to get vendored
+dependencies:
+
+::
+
+    go get github.com/Masterminds/glide
+    cd $GOPATH/src/github.com/tendermint/abci
+    glide install
+    go install ./cmd/abci-cli
+
+Now run ``abci-cli`` to see the list of commands:
+
+::
+
+    Usage:
+      abci-cli [command]
+
+    Available Commands:
+      batch       Run a batch of abci commands against an application
+      check_tx    Validate a tx
+      commit      Commit the application state and return the Merkle root hash
+      console     Start an interactive abci console for multiple commands
+      counter     ABCI demo example
+      deliver_tx  Deliver a new tx to the application
+      dummy       ABCI demo example
+      echo        Have the application echo a message
+      help        Help about any command
+      info        Get some info about the application
+      query       Query the application state
+      set_option  Set an options on the application
+
+    Flags:
+          --abci string      socket or grpc (default "socket")
+          --address string   address of application socket (default "tcp://127.0.0.1:46658")
+      -h, --help             help for abci-cli
+      -v, --verbose          print the command and results as if it were a console session
+
+                                          Use "abci-cli [command] --help" for more information about a command.
+
+
+Dummy - First Example
+---------------------
+
+The ``abci-cli`` tool lets us send ABCI messages to our application, to
+help build and debug them.
+
+The most important messages are ``deliver_tx``, ``check_tx``, and
+``commit``, but there are others for convenience, configuration, and
+information purposes.
+
+Let's start a dummy application, which was installed at the same time as
+``abci-cli`` above. The dummy just stores transactions in a merkle tree:
+
+::
+
+    abci-cli dummy
+
+In another terminal, run
+
+::
+
+    abci-cli echo hello
+    abci-cli info
+
+You'll see something like:
+
+::
+
+    -> data: hello
+    -> data.hex: 68656C6C6F
+
+and:
+
+::
+
+    -> data: {"size":0}
+    -> data.hex: 7B2273697A65223A307D
+
+An ABCI application must provide two things:
+
+-  a socket server
+-  a handler for ABCI messages
+
+When we run the ``abci-cli`` tool we open a new connection to the
+application's socket server, send the given ABCI message, and wait for a
+response.
+
+The server may be generic for a particular language, and we provide a
+`reference implementation in
+Golang <https://github.com/tendermint/abci/tree/master/server>`__. See
+the `list of other ABCI
+implementations <./ecosystem.html>`__ for servers in
+other languages.
+
+The handler is specific to the application, and may be arbitrary, so
+long as it is deterministic and conforms to the ABCI interface
+specification.
+
+So when we run ``abci-cli info``, we open a new connection to the ABCI
+server, which calls the ``Info()`` method on the application, which
+tells us the number of transactions in our Merkle tree.
+
+Now, since every command opens a new connection, we provide the
+``abci-cli console`` and ``abci-cli batch`` commands, to allow multiple
+ABCI messages to be sent over a single connection.
+
+Running ``abci-cli console`` should drop you in an interactive console
+for speaking ABCI messages to your application.
+
+Try running these commands:
+
+::
+
+    > echo hello
+    -> code: OK
+    -> data: hello
+    -> data.hex: 0x68656C6C6F
+    
+    > info
+    -> code: OK
+    -> data: {"size":0}
+    -> data.hex: 0x7B2273697A65223A307D
+    
+    > commit
+    -> code: OK
+    
+    > deliver_tx "abc"
+    -> code: OK
+    
+    > info
+    -> code: OK
+    -> data: {"size":1}
+    -> data.hex: 0x7B2273697A65223A317D
+    
+    > commit
+    -> code: OK
+    -> data.hex: 0x49DFD15CCDACDEAE9728CB01FBB5E8688CA58B91
+    
+    > query "abc"
+    -> code: OK
+    -> log: exists
+    -> height: 0
+    -> value: abc
+    -> value.hex: 616263
+    
+    > deliver_tx "def=xyz"
+    -> code: OK
+    
+    > commit
+    -> code: OK
+    -> data.hex: 0x70102DB32280373FBF3F9F89DA2A20CE2CD62B0B
+    
+    > query "def"
+    -> code: OK
+    -> log: exists
+    -> height: 0
+    -> value: xyz
+    -> value.hex: 78797A
+
+Note that if we do ``deliver_tx "abc"`` it will store ``(abc, abc)``,
+but if we do ``deliver_tx "abc=efg"`` it will store ``(abc, efg)``.
+
+Similarly, you could put the commands in a file and run
+``abci-cli --verbose batch < myfile``.
+
+Counter - Another Example
+-------------------------
+
+Now that we've got the hang of it, let's try another application, the
+"counter" app.
+
+The counter app doesn't use a Merkle tree, it just counts how many times
+we've sent a transaction, asked for a hash, or committed the state. The
+result of ``commit`` is just the number of transactions sent.
+
+This application has two modes: ``serial=off`` and ``serial=on``.
+
+When ``serial=on``, transactions must be a big-endian encoded
+incrementing integer, starting at 0.
+
+If ``serial=off``, there are no restrictions on transactions.
+
+We can toggle the value of ``serial`` using the ``set_option`` ABCI
+message.
+
+When ``serial=on``, some transactions are invalid. In a live blockchain,
+transactions collect in memory before they are committed into blocks. To
+avoid wasting resources on invalid transactions, ABCI provides the
+``check_tx`` message, which application developers can use to accept or
+reject transactions, before they are stored in memory or gossipped to
+other peers.
+
+In this instance of the counter app, ``check_tx`` only allows
+transactions whose integer is greater than the last committed one.
+
+Let's kill the console and the dummy application, and start the counter
+app:
+
+::
+
+    abci-cli counter
+
+In another window, start the ``abci-cli console``:
+
+::
+
+    > set_option serial on
+    -> code: OK
+    
+    > check_tx 0x00
+    -> code: OK
+    
+    > check_tx 0xff
+    -> code: OK
+    
+    > deliver_tx 0x00
+    -> code: OK
+    
+    > check_tx 0x00
+    -> code: BadNonce
+    -> log: Invalid nonce. Expected >= 1, got 0
+    
+    > deliver_tx 0x01
+    -> code: OK
+    
+    > deliver_tx 0x04
+    -> code: BadNonce
+    -> log: Invalid nonce. Expected 2, got 4
+    
+    > info
+    -> code: OK
+    -> data: {"hashes":0,"txs":2}
+    -> data.hex: 0x7B22686173686573223A302C22747873223A327D
+
+This is a very simple application, but between ``counter`` and
+``dummy``, its easy to see how you can build out arbitrary application
+states on top of the ABCI. `Hyperledger's
+Burrow <https://github.com/hyperledger/burrow>`__ also runs atop ABCI,
+bringing with it Ethereum-like accounts, the Ethereum virtual-machine,
+Monax's permissioning scheme, and native contracts extensions.
+
+But the ultimate flexibility comes from being able to write the
+application easily in any language.
+
+We have implemented the counter in a number of languages (see the
+example directory).
+
+To run the Node JS version, ``cd`` to ``example/js`` and run
+
+::
+
+    node app.js
+
+(you'll have to kill the other counter application process). In another
+window, run the console and those previous ABCI commands. You should get
+the same results as for the Go version.
+
+Bounties
+--------
+
+Want to write the counter app in your favorite language?! We'd be happy
+to add you to our `ecosystem <https://tendermint.com/ecosystem>`__!
+We're also offering `bounties <https://tendermint.com/bounties>`__ for
+implementations in new languages!
+
+The ``abci-cli`` is designed strictly for testing and debugging. In a
+real deployment, the role of sending messages is taken by Tendermint,
+which connects to the app using three separate connections, each with
+its own pattern of messages.
+
+For more information, see the `application developers
+guide <./app-development.html>`__. For examples of running an ABCI
+app with Tendermint, see the `getting started
+guide <./getting-started.html>`__.

docs/app-architecture.rst

@@ -0,0 +1,125 @@
+Application Architecture Guide
+==============================
+
+Overview
+--------
+
+A blockchain application is more than the consensus engine and the
+transaction logic (eg. smart contracts, business logic) as implemented
+in the ABCI app. There are also (mobile, web, desktop) clients that will
+need to connect and make use of the app. We will assume for now that you
+have a well designed transactions and database model, but maybe this
+will be the topic of another article. This article is more interested in
+various ways of setting up the "plumbing" and connecting these pieces,
+and demonstrating some evolving best practices.
+
+Security
+--------
+
+A very important aspect when constructing a blockchain is security. The
+consensus model can be DoSed (no consensus possible) by corrupting 1/3
+of the validators and exploited (writing arbitrary blocks) by corrupting
+2/3 of the validators. So, while the security is not that of the
+"weakest link", you should take care that the "average link" is
+sufficiently hardened.
+
+One big attack surface on the validators is the communication between
+the ABCI app and the tendermint core. This should be highly protected.
+Ideally, the app and the core are running on the same machine, so no
+external agent can target the communication channel. You can use unix
+sockets (with permissions preventing access from other users), or even
+compile the two apps into one binary if the ABCI app is also writen in
+go. If you are unable to do that due to language support, then the ABCI
+app should bind a TCP connection to localhost (127.0.0.1), which is less
+efficient and secure, but still not reachable from outside. If you must
+run the ABCI app and tendermint core on separate machines, make sure you
+have a secure communication channel (ssh tunnel?)
+
+Now assuming, you have linked together your app and the core securely,
+you must also make sure no one can get on the machine it is hosted on.
+At this point it is basic network security. Run on a secure operating
+system (SELinux?). Limit who has access to the machine (user accounts,
+but also where the physical machine is hosted). Turn off all services
+except for ssh, which should only be accessible by some well-guarded
+public/private key pairs (no password). And maybe even firewall off
+access to the ports used by the validators, so only known validators can
+connect.
+
+There was also a suggestion on slack from @jhon about compiling
+everything together with a unikernel for more security, such as
+`Mirage <https://mirage.io>`__ or
+`UNIK <https://github.com/emc-advanced-dev/unik>`__.
+
+Connecting your client to the blockchain
+----------------------------------------
+
+Tendermint Core RPC
+~~~~~~~~~~~~~~~~~~~
+
+The concept is that the ABCI app is completely hidden from the outside
+world and only communicated through a tested and secured `interface
+exposed by the tendermint core <./specification/rpc.html>`__. This interface
+exposes a lot of data on the block header and consensus process, which
+is quite useful for externally verifying the system. It also includes
+3(!) methods to broadcast a transaction (propose it for the blockchain,
+and possibly await a response). And one method to query app-specific
+data from the ABCI application.
+
+Pros:
+* Server code already written
+* Access to block headers to validate merkle proofs (nice for light clients)
+* Basic read/write functionality is supported
+
+Cons:
+* Limited interface to app. All queries must be serialized into
+[]byte (less expressive than JSON over HTTP) and there is no way to push
+data from ABCI app to the client (eg. notify me if account X receives a
+transaction)
+
+Custom ABCI server
+~~~~~~~~~~~~~~~~~~
+
+This was proposed by @wolfposd on slack and demonstrated by
+`TMChat <https://github.com/wolfposd/TMChat>`__, a sample app. The
+concept is to write a custom server for your app (with typical REST
+API/websockets/etc for easy use by a mobile app). This custom server is
+in the same binary as the ABCI app and data store, so can easily react
+to complex events there that involve understanding the data format (send
+a message if my balance drops below 500). All "writes" sent to this
+server are proxied via websocket/JSON-RPC to tendermint core. When they
+come back as deliver\_tx over ABCI, they will be written to the data
+store. For "reads", we can do any queries we wish that are supported by
+our architecture, using any web technology that is useful. The general
+architecture is shown in the following diagram:
+
+Pros: \* Separates application logic from blockchain logic \* Allows
+much richer, more flexible client-facing API \* Allows pub-sub, watching
+certain fields, etc.
+
+Cons: \* Access to ABCI app can be dangerous (be VERY careful not to
+write unless it comes from the validator node) \* No direct access to
+the blockchain headers to verify tx \* You must write your own API (but
+maybe that's a pro...)
+
+Hybrid solutions
+~~~~~~~~~~~~~~~~
+
+Likely the least secure but most versatile. The client can access both
+the tendermint node for all blockchain info, as well as a custom app
+server, for complex queries and pub-sub on the abci app.
+
+Pros: All from both above solutions
+
+Cons: Even more complexity; even more attack vectors (less
+security)
+
+Scalability
+-----------
+
+Read replica using non-validating nodes? They could forward transactions
+to the validators (fewer connections, more security), and locally allow
+all queries in any of the above configurations. Thus, while
+transaction-processing speed is limited by the speed of the abci app and
+the number of validators, one should be able to scale our read
+performance to quite an extent (until the replication process drains too
+many resources from the validator nodes).

docs/app-development.rst

@@ -0,0 +1,628 @@
+Application Development Guide
+=============================
+
+ABCI Design
+-----------
+
+The purpose of ABCI is to provide a clean interface between state
+transition machines on one computer and the mechanics of their
+replication across multiple computers. The former we call 'application
+logic' and the latter the 'consensus engine'. Application logic
+validates transactions and optionally executes transactions against some
+persistent state. A consensus engine ensures all transactions are
+replicated in the same order on every machine. We call each machine in a
+consensus engine a 'validator', and each validator runs the same
+transactions through the same application logic. In particular, we are
+interested in blockchain-style consensus engines, where transactions are
+committed in hash-linked blocks.
+
+The ABCI design has a few distinct components:
+
+-  message protocol
+
+   -  pairs of request and response messages
+   -  consensus makes requests, application responds
+   -  defined using protobuf
+
+-  server/client
+
+   -  consensus engine runs the client
+   -  application runs the server
+   -  two implementations:
+
+      -  async raw bytes
+      -  grpc
+
+-  blockchain protocol
+
+   -  abci is connection oriented
+   -  Tendermint Core maintains three connections:
+
+      -  `mempool connection <#mempool-connection>`__: for checking if
+         transactions should be relayed before they are committed; only
+         uses ``CheckTx``
+      -  `consensus connection <#consensus-connection>`__: for executing
+         transactions that have been committed. Message sequence is -
+         for every block -
+         ``BeginBlock, [DeliverTx, ...], EndBlock, Commit``
+      -  `query connection <#query-connection>`__: for querying the
+         application state; only uses Query and Info
+
+The mempool and consensus logic act as clients, and each maintains an
+open ABCI connection with the application, which hosts an ABCI server.
+Shown are the request and response types sent on each connection.
+
+Message Protocol
+----------------
+
+The message protocol consists of pairs of requests and responses. Some
+messages have no fields, while others may include byte-arrays, strings,
+or integers. See the ``message Request`` and ``message Response``
+definitions in `the protobuf definition
+file <https://github.com/tendermint/abci/blob/master/types/types.proto>`__,
+and the `protobuf
+documentation <https://developers.google.com/protocol-buffers/docs/overview>`__
+for more details.
+
+For each request, a server should respond with the corresponding
+response, where order of requests is preserved in the order of
+responses.
+
+Server
+------
+
+To use ABCI in your programming language of choice, there must be a ABCI
+server in that language. Tendermint supports two kinds of implementation
+of the server:
+
+-  Asynchronous, raw socket server (Tendermint Socket Protocol, also
+   known as TSP or Teaspoon)
+-  GRPC
+
+Both can be tested using the ``abci-cli`` by setting the ``--abci`` flag
+appropriately (ie. to ``socket`` or ``grpc``).
+
+See examples, in various stages of maintenance, in
+`Go <https://github.com/tendermint/abci/tree/master/server>`__,
+`JavaScript <https://github.com/tendermint/js-abci>`__,
+`Python <https://github.com/tendermint/abci/tree/master/example/python3/abci>`__,
+`C++ <https://github.com/mdyring/cpp-tmsp>`__, and
+`Java <https://github.com/jTendermint/jabci>`__.
+
+GRPC
+~~~~
+
+If GRPC is available in your language, this is the easiest approach,
+though it will have significant performance overhead.
+
+To get started with GRPC, copy in the `protobuf
+file <https://github.com/tendermint/abci/blob/master/types/types.proto>`__
+and compile it using the GRPC plugin for your language. For instance,
+for golang, the command is
+``protoc --go_out=plugins=grpc:. types.proto``. See the `grpc
+documentation for more details <http://www.grpc.io/docs/>`__. ``protoc``
+will autogenerate all the necessary code for ABCI client and server in
+your language, including whatever interface your application must
+satisfy to be used by the ABCI server for handling requests.
+
+TSP
+~~~
+
+If GRPC is not available in your language, or you require higher
+performance, or otherwise enjoy programming, you may implement your own
+ABCI server using the Tendermint Socket Protocol, known affectionately
+as Teaspoon. The first step is still to auto-generate the relevant data
+types and codec in your language using ``protoc``. Messages coming over
+the socket are Protobuf3 encoded, but additionally length-prefixed to
+facilitate use as a streaming protocol. Protobuf3 doesn't have an
+official length-prefix standard, so we use our own. The first byte in
+the prefix represents the length of the Big Endian encoded length. The
+remaining bytes in the prefix are the Big Endian encoded length.
+
+For example, if the Protobuf3 encoded ABCI message is 0xDEADBEEF (4
+bytes), the length-prefixed message is 0x0104DEADBEEF. If the Protobuf3
+encoded ABCI message is 65535 bytes long, the length-prefixed message
+would be like 0x02FFFF....
+
+Note this prefixing does not apply for grpc.
+
+An ABCI server must also be able to support multiple connections, as
+Tendermint uses three connections.
+
+Client
+------
+
+There are currently two use-cases for an ABCI client. One is a testing
+tool, as in the ``abci-cli``, which allows ABCI requests to be sent via
+command line. The other is a consensus engine, such as Tendermint Core,
+which makes requests to the application every time a new transaction is
+received or a block is committed.
+
+It is unlikely that you will need to implement a client. For details of
+our client, see
+`here <https://github.com/tendermint/abci/tree/master/client>`__.
+
+Most of the examples below are from `dummy application
+<https://github.com/tendermint/abci/blob/master/example/dummy/dummy.go>`__,
+which is a part of the abci repo. `persistent_dummy application
+<https://github.com/tendermint/abci/blob/master/example/dummy/persistent_dummy.go>`__
+is used to show ``BeginBlock``, ``EndBlock`` and ``InitChain``
+example implementations.
+
+Blockchain Protocol
+-------------------
+
+In ABCI, a transaction is simply an arbitrary length byte-array. It is
+the application's responsibility to define the transaction codec as they
+please, and to use it for both CheckTx and DeliverTx.
+
+Note that there are two distinct means for running transactions,
+corresponding to stages of 'awareness' of the transaction in the
+network. The first stage is when a transaction is received by a
+validator from a client into the so-called mempool or transaction pool -
+this is where we use CheckTx. The second is when the transaction is
+successfully committed on more than 2/3 of validators - where we use
+DeliverTx. In the former case, it may not be necessary to run all the
+state transitions associated with the transaction, as the transaction
+may not ultimately be committed until some much later time, when the
+result of its execution will be different. For instance, an Ethereum
+ABCI app would check signatures and amounts in CheckTx, but would not
+actually execute any contract code until the DeliverTx, so as to avoid
+executing state transitions that have not been finalized.
+
+To formalize the distinction further, two explicit ABCI connections are
+made between Tendermint Core and the application: the mempool connection
+and the consensus connection. We also make a third connection, the query
+connection, to query the local state of the app.
+
+Mempool Connection
+~~~~~~~~~~~~~~~~~~
+
+The mempool connection is used *only* for CheckTx requests. Transactions
+are run using CheckTx in the same order they were received by the
+validator. If the CheckTx returns ``OK``, the transaction is kept in
+memory and relayed to other peers in the same order it was received.
+Otherwise, it is discarded.
+
+CheckTx requests run concurrently with block processing; so they should
+run against a copy of the main application state which is reset after
+every block. This copy is necessary to track transitions made by a
+sequence of CheckTx requests before they are included in a block. When a
+block is committed, the application must ensure to reset the mempool
+state to the latest committed state. Tendermint Core will then filter
+through all transactions in the mempool, removing any that were included
+in the block, and re-run the rest using CheckTx against the post-Commit
+mempool state.
+
+.. container:: toggle
+
+    .. container:: header
+
+        **Show/Hide Go Example**
+
+    .. code-block:: go
+
+        func (app *DummyApplication) CheckTx(tx []byte) types.Result {
+          return types.OK
+        }
+
+.. container:: toggle
+
+    .. container:: header
+
+        **Show/Hide Java Example**
+
+    .. code-block:: java
+
+        ResponseCheckTx requestCheckTx(RequestCheckTx req) {
+            byte[] transaction = req.getTx().toByteArray();
+
+            // validate transaction
+
+            if (notValid) {
+                return ResponseCheckTx.newBuilder().setCode(CodeType.BadNonce).setLog("invalid tx").build();
+            } else {
+                return ResponseCheckTx.newBuilder().setCode(CodeType.OK).build();
+            }
+        }
+
+Consensus Connection
+~~~~~~~~~~~~~~~~~~~~
+
+The consensus connection is used only when a new block is committed, and
+communicates all information from the block in a series of requests:
+``BeginBlock, [DeliverTx, ...], EndBlock, Commit``. That is, when a
+block is committed in the consensus, we send a list of DeliverTx
+requests (one for each transaction) sandwiched by BeginBlock and
+EndBlock requests, and followed by a Commit.
+
+DeliverTx
+^^^^^^^^^
+
+DeliverTx is the workhorse of the blockchain. Tendermint sends the
+DeliverTx requests asynchronously but in order, and relies on the
+underlying socket protocol (ie. TCP) to ensure they are received by the
+app in order. They have already been ordered in the global consensus by
+the Tendermint protocol.
+
+DeliverTx returns a abci.Result, which includes a Code, Data, and Log.
+The code may be non-zero (non-OK), meaning the corresponding transaction
+should have been rejected by the mempool, but may have been included in
+a block by a Byzantine proposer.
+
+The block header will be updated (TODO) to include some commitment to
+the results of DeliverTx, be it a bitarray of non-OK transactions, or a
+merkle root of the data returned by the DeliverTx requests, or both.
+
+.. container:: toggle
+
+    .. container:: header
+
+        **Show/Hide Go Example**
+
+    .. code-block:: go
+
+        // tx is either "key=value" or just arbitrary bytes
+        func (app *DummyApplication) DeliverTx(tx []byte) types.Result {
+          parts := strings.Split(string(tx), "=")
+          if len(parts) == 2 {
+            app.state.Set([]byte(parts[0]), []byte(parts[1]))
+          } else {
+            app.state.Set(tx, tx)
+          }
+          return types.OK
+        }
+
+.. container:: toggle
+
+    .. container:: header
+
+        **Show/Hide Java Example**
+
+    .. code-block:: java
+
+        /**
+         * Using Protobuf types from the protoc compiler, we always start with a byte[]
+         */
+        ResponseDeliverTx deliverTx(RequestDeliverTx request) {
+            byte[] transaction  = request.getTx().toByteArray();
+
+            // validate your transaction
+
+            if (notValid) {
+                return ResponseDeliverTx.newBuilder().setCode(CodeType.BadNonce).setLog("transaction was invalid").build();
+            } else {
+                ResponseDeliverTx.newBuilder().setCode(CodeType.OK).build();
+            }
+
+        }
+
+Commit
+^^^^^^
+
+Once all processing of the block is complete, Tendermint sends the
+Commit request and blocks waiting for a response. While the mempool may
+run concurrently with block processing (the BeginBlock, DeliverTxs, and
+EndBlock), it is locked for the Commit request so that its state can be
+safely reset during Commit. This means the app *MUST NOT* do any
+blocking communication with the mempool (ie. broadcast\_tx) during
+Commit, or there will be deadlock. Note also that all remaining
+transactions in the mempool are replayed on the mempool connection
+(CheckTx) following a commit.
+
+The app should respond to the Commit request with a byte array, which is the deterministic
+state root of the application. It is included in the header of the next
+block. It can be used to provide easily verified Merkle-proofs of the
+state of the application.
+
+It is expected that the app will persist state to disk on Commit. The
+option to have all transactions replayed from some previous block is the
+job of the `Handshake <#handshake>`__.
+
+.. container:: toggle
+
+    .. container:: header
+
+        **Show/Hide Go Example**
+
+    .. code-block:: go
+
+        func (app *DummyApplication) Commit() types.Result {
+          hash := app.state.Hash()
+          return types.NewResultOK(hash, "")
+        }
+
+.. container:: toggle
+
+    .. container:: header
+
+        **Show/Hide Java Example**
+
+    .. code-block:: java
+
+        ResponseCommit requestCommit(RequestCommit requestCommit) {
+
+            // update the internal app-state
+            byte[] newAppState = calculateAppState();
+
+            // and return it to the node
+            return ResponseCommit.newBuilder().setCode(CodeType.OK).setData(ByteString.copyFrom(newAppState)).build();
+        }
+
+BeginBlock
+^^^^^^^^^^
+
+The BeginBlock request can be used to run some code at the beginning of
+every block. It also allows Tendermint to send the current block hash
+and header to the application, before it sends any of the transactions.
+
+The app should remember the latest height and header (ie. from which it
+has run a successful Commit) so that it can tell Tendermint where to
+pick up from when it restarts. See information on the Handshake, below.
+
+.. container:: toggle
+
+    .. container:: header
+
+        **Show/Hide Go Example**
+
+    .. code-block:: go
+
+        // Track the block hash and header information
+        func (app *PersistentDummyApplication) BeginBlock(params types.RequestBeginBlock) {
+          // update latest block info
+          app.blockHeader = params.Header
+
+          // reset valset changes
+          app.changes = make([]*types.Validator, 0)
+        }
+
+.. container:: toggle
+
+    .. container:: header
+
+        **Show/Hide Java Example**
+
+    .. code-block:: java
+
+        /*
+         * all types come from protobuf definition
+         */
+        ResponseBeginBlock requestBeginBlock(RequestBeginBlock req) {
+
+            Header header = req.getHeader();
+            byte[] prevAppHash = header.getAppHash().toByteArray();
+            long prevHeight = header.getHeight();
+            long numTxs = header.getNumTxs();
+
+            // run your pre-block logic. Maybe prepare a state snapshot, message components, etc
+
+            return ResponseBeginBlock.newBuilder().build();
+        }
+
+EndBlock
+^^^^^^^^
+
+The EndBlock request can be used to run some code at the end of every
+block. Additionally, the response may contain a list of validators,
+which can be used to update the validator set. To add a new validator or
+update an existing one, simply include them in the list returned in the
+EndBlock response. To remove one, include it in the list with a
+``power`` equal to ``0``. Tendermint core will take care of updating the
+validator set. Note validator set changes are only available in v0.8.0
+and up.
+
+.. container:: toggle
+
+    .. container:: header
+
+        **Show/Hide Go Example**
+
+    .. code-block:: go
+
+        // Update the validator set
+        func (app *PersistentDummyApplication) EndBlock(height uint64) (resEndBlock types.ResponseEndBlock) {
+          return types.ResponseEndBlock{Diffs: app.changes}
+        }
+
+.. container:: toggle
+
+    .. container:: header
+
+        **Show/Hide Java Example**
+
+    .. code-block:: java
+
+        /*
+         * Assume that one validator changes. The new validator has a power of 10
+         */
+        ResponseEndBlock requestEndBlock(RequestEndBlock req) {
+            final long currentHeight = req.getHeight();
+            final byte[] validatorPubKey = getValPubKey();
+
+            ResponseEndBlock.Builder builder = ResponseEndBlock.newBuilder();
+            builder.addDiffs(1, Types.Validator.newBuilder().setPower(10L).setPubKey(ByteString.copyFrom(validatorPubKey)).build());
+
+            return builder.build();
+        }
+
+Query Connection
+~~~~~~~~~~~~~~~~
+
+This connection is used to query the application without engaging
+consensus. It's exposed over the tendermint core rpc, so clients can
+query the app without exposing a server on the app itself, but they must
+serialize each query as a single byte array. Additionally, certain
+"standardized" queries may be used to inform local decisions, for
+instance about which peers to connect to.
+
+Tendermint Core currently uses the Query connection to filter peers upon
+connecting, according to IP address or public key. For instance,
+returning non-OK ABCI response to either of the following queries will
+cause Tendermint to not connect to the corresponding peer:
+
+-  ``p2p/filter/addr/<addr>``, where ``<addr>`` is an IP address.
+-  ``p2p/filter/pubkey/<pubkey>``, where ``<pubkey>`` is the hex-encoded
+   ED25519 key of the node (not it's validator key)
+
+Note: these query formats are subject to change!
+
+.. container:: toggle
+
+    .. container:: header
+
+        **Show/Hide Go Example**
+
+    .. code-block:: go
+
+        func (app *DummyApplication) Query(reqQuery types.RequestQuery) (resQuery types.ResponseQuery) {
+          if reqQuery.Prove {
+            value, proof, exists := app.state.Proof(reqQuery.Data)
+            resQuery.Index = -1 // TODO make Proof return index
+            resQuery.Key = reqQuery.Data
+            resQuery.Value = value
+            resQuery.Proof = proof
+            if exists {
+              resQuery.Log = "exists"
+            } else {
+              resQuery.Log = "does not exist"
+            }
+            return
+          } else {
+            index, value, exists := app.state.Get(reqQuery.Data)
+            resQuery.Index = int64(index)
+            resQuery.Value = value
+            if exists {
+              resQuery.Log = "exists"
+            } else {
+              resQuery.Log = "does not exist"
+            }
+            return
+          }
+        }
+
+.. container:: toggle
+
+    .. container:: header
+
+        **Show/Hide Java Example**
+
+    .. code-block:: java
+
+        ResponseQuery requestQuery(RequestQuery req) {
+            final boolean isProveQuery = req.getProve();
+            final ResponseQuery.Builder responseBuilder = ResponseQuery.newBuilder();
+
+            if (isProveQuery) {
+                com.app.example.ProofResult proofResult = generateProof(req.getData().toByteArray());
+                final byte[] proofAsByteArray = proofResult.getAsByteArray();
+
+                responseBuilder.setProof(ByteString.copyFrom(proofAsByteArray));
+                responseBuilder.setKey(req.getData());
+                responseBuilder.setValue(ByteString.copyFrom(proofResult.getData()));
+                responseBuilder.setLog(result.getLogValue());
+            } else {
+                byte[] queryData = req.getData().toByteArray();
+
+                final com.app.example.QueryResult result = generateQueryResult(queryData);
+
+                responseBuilder.setIndex(result.getIndex());
+                responseBuilder.setValue(ByteString.copyFrom(result.getValue()));
+                responseBuilder.setLog(result.getLogValue());
+            }
+
+            return responseBuilder.build();
+        }
+
+Handshake
+~~~~~~~~~
+
+When the app or tendermint restarts, they need to sync to a common
+height. When an ABCI connection is first established, Tendermint will
+call ``Info`` on the Query connection. The response should contain the
+LastBlockHeight and LastBlockAppHash - the former is the last block for
+which the app ran Commit successfully, the latter is the response
+from that Commit.
+
+Using this information, Tendermint will determine what needs to be
+replayed, if anything, against the app, to ensure both Tendermint and
+the app are synced to the latest block height.
+
+If the app returns a LastBlockHeight of 0, Tendermint will just replay
+all blocks.
+
+.. container:: toggle
+
+    .. container:: header
+
+        **Show/Hide Go Example**
+
+    .. code-block:: go
+
+        func (app *DummyApplication) Info(req types.RequestInfo) (resInfo types.ResponseInfo) {
+          return types.ResponseInfo{Data: cmn.Fmt("{\"size\":%v}", app.state.Size())}
+        }
+
+.. container:: toggle
+
+    .. container:: header
+
+        **Show/Hide Java Example**
+
+    .. code-block:: java
+
+        ResponseInfo requestInfo(RequestInfo req) {
+            final byte[] lastAppHash = getLastAppHash();
+            final long lastHeight = getLastHeight();
+            return ResponseInfo.newBuilder().setLastBlockAppHash(ByteString.copyFrom(lastAppHash)).setLastBlockHeight(lastHeight).build();
+        }
+
+Genesis
+~~~~~~~
+
+``InitChain`` will be called once upon the genesis. ``params`` includes the
+initial validator set. Later on, it may be extended to take parts of the
+consensus params.
+
+.. container:: toggle
+
+    .. container:: header
+
+        **Show/Hide Go Example**
+
+    .. code-block:: go
+
+        // Save the validators in the merkle tree
+        func (app *PersistentDummyApplication) InitChain(params types.RequestInitChain) {
+          for _, v := range params.Validators {
+            r := app.updateValidator(v)
+            if r.IsErr() {
+              app.logger.Error("Error updating validators", "r", r)
+            }
+          }
+        }
+
+.. container:: toggle
+
+    .. container:: header
+
+        **Show/Hide Java Example**
+
+    .. code-block:: java
+
+        /*
+         * all types come from protobuf definition
+         */
+        ResponseInitChain requestInitChain(RequestInitChain req) {
+            final int validatorsCount = req.getValidatorsCount();
+            final List<Types.Validator> validatorsList = req.getValidatorsList();
+
+            validatorsList.forEach((validator) -> {
+                long power = validator.getPower();
+                byte[] validatorPubKey = validator.getPubKey().toByteArray();
+
+                // do somehing for validator setup in app
+            });
+
+            return ResponseInitChain.newBuilder().build();
+        }

docs/architecture/ABCI.md

@@ -1,16 +0,0 @@
-# ABCI
-
-ABCI is an interface between the consensus/blockchain engine known as tendermint, and the application-specific business logic, known as an ABCi app.
-
-The tendermint core should run unchanged for all apps.  Each app can customize it, the supported transactions, queries, even the validator sets and how to handle staking / slashing stake. This customization is achieved by implementing the ABCi app to send the proper information to the tendermint engine to perform as directed.
-
-To understand this decision better, think of the design of the tendermint engine.
-
-* A blockchain is simply consensus on a unique global ordering of events.
-* This consensus can efficiently be implemented using BFT and PoS
-* This code can be generalized to easily support a large number of blockchains
-* The block-chain specific code, the interpretation of the individual events, can be implemented by a 3rd party app without touching the consensus engine core
-* Use an efficient, language-agnostic layer to implement this (ABCi)
-
-
-Bucky, please make this doc real.

docs/architecture/README.md

@@ -2,15 +2,4 @@
 
 This is a location to record all high-level architecture decisions in the tendermint project.  Not the implementation details, but the reasoning that happened.  This should be refered to for guidance of the "right way" to extend the application.  And if we notice that the original decisions were lacking, we should have another open discussion, record the new decisions here, and then modify the code to match.
 
-This is like our guide and mentor when Jae and Bucky are offline.... The concept comes from a [blog post](https://product.reverb.com/documenting-architecture-decisions-the-reverb-way-a3563bb24bd0#.78xhdix6t) that resonated among the team when Anton shared it.
-
-Each section of the code can have it's own markdown file in this directory, and please add a link to the readme.
-
-## Sections
-
-* [ABCI](./ABCI.md)
-* [go-merkle / merkleeyes](./merkle.md)
-* [Frey's thoughts on the data store](./merkle-frey.md)
-* basecoin
-* tendermint core (multiple sections)
-* ???
+Read up on the concept in this [blog post](https://product.reverb.com/documenting-architecture-decisions-the-reverb-way-a3563bb24bd0#.78xhdix6t).

docs/architecture/adr-002-event-subscription.md

@@ -0,0 +1,90 @@
+# ADR 2: Event Subscription
+
+## Context
+
+In the light client (or any other client), the user may want to **subscribe to
+a subset of transactions** (rather than all of them) using `/subscribe?event=X`. For
+example, I want to subscribe for all transactions associated with a particular
+account. Same for fetching. The user may want to **fetch transactions based on
+some filter** (rather than fetching all the blocks). For example, I want to get
+all transactions for a particular account in the last two weeks (`tx's block
+time >= '2017-06-05'`).
+
+Now you can't even subscribe to "all txs" in Tendermint.
+
+The goal is a simple and easy to use API for doing that.
+
+![Tx Send Flow Diagram](img/tags1.png)
+
+## Decision
+
+ABCI app return tags with a `DeliverTx` response inside the `data` field (_for
+now, later we may create a separate field_). Tags is a list of key-value pairs,
+protobuf encoded.
+
+Example data:
+
+```json
+{
+  "abci.account.name": "Igor",
+  "abci.account.address": "0xdeadbeef",
+  "tx.gas": 7
+}
+```
+
+### Subscribing for transactions events
+
+If the user wants to receive only a subset of transactions, ABCI-app must
+return a list of tags with a `DeliverTx` response. These tags will be parsed and
+matched with the current queries (subscribers). If the query matches the tags,
+subscriber will get the transaction event.
+
+```
+/subscribe?query="tm.event = Tx AND tx.hash = AB0023433CF0334223212243BDD AND abci.account.invoice.number = 22"
+```
+
+A new package must be developed to replace the current `events` package. It
+will allow clients to subscribe to a different types of events in the future:
+
+```
+/subscribe?query="abci.account.invoice.number = 22"
+/subscribe?query="abci.account.invoice.owner CONTAINS Igor"
+```
+
+### Fetching transactions
+
+This is a bit tricky because a) we want to support a number of indexers, all of
+which have a different API b) we don't know whenever tags will be sufficient
+for the most apps (I guess we'll see).
+
+```
+/txs/search?query="tx.hash = AB0023433CF0334223212243BDD AND abci.account.owner CONTAINS Igor"
+/txs/search?query="abci.account.owner = Igor"
+```
+
+For historic queries we will need a indexing storage (Postgres, SQLite, ...).
+
+### Issues
+
+- https://github.com/tendermint/basecoin/issues/91
+- https://github.com/tendermint/tendermint/issues/376
+- https://github.com/tendermint/tendermint/issues/287
+- https://github.com/tendermint/tendermint/issues/525 (related)
+
+## Status
+
+proposed
+
+## Consequences
+
+### Positive
+
+- same format for event notifications and search APIs
+- powerful enough query
+
+### Negative
+
+- performance of the `match` function (where we have too many queries / subscribers)
+- there is an issue where there are too many txs in the DB
+
+### Neutral