json2wal: increase reader's buffer size (#3147)

```
panic: failed to unmarshal json: unexpected end of JSON input

goroutine 1 [running]:
main.main()
	/root/gelgo/src/github.com/tendermint/tendermint/scripts/json2wal/main.go:66 +0x73f
```

Closes #3146

CHANGELOG.md

@@ -1,5 +1,60 @@
 # Changelog
 
+## v0.28.0
+
+*January 16th, 2019*
+
+Special thanks to external contributors on this release:
+@fmauricios, @gianfelipe93, @husio, @needkane, @srmo, @yutianwu
+
+This release is primarily about upgrades to the `privval` system -
+separating the `priv_validator.json` into distinct config and data files, and
+refactoring the socket validator to support reconnections.
+
+**Note:** Please backup your existing `priv_validator.json` before using this
+version.
+
+See [UPGRADING.md](UPGRADING.md) for more details.
+
+### BREAKING CHANGES:
+
+* CLI/RPC/Config
+  - [cli] Removed `--proxy_app=dummy` option. Use `kvstore` (`persistent_kvstore`) instead.
+  - [cli] Renamed `--proxy_app=nilapp` to `--proxy_app=noop`.
+  - [config] [\#2992](https://github.com/tendermint/tendermint/issues/2992) `allow_duplicate_ip` is now set to false
+  - [privval] [\#1181](https://github.com/tendermint/tendermint/issues/1181) Split `priv_validator.json` into immutable (`config/priv_validator_key.json`) and mutable (`data/priv_validator_state.json`) parts (@yutianwu)
+  - [privval] [\#2926](https://github.com/tendermint/tendermint/issues/2926) Split up `PubKeyMsg` into `PubKeyRequest` and `PubKeyResponse` to be consistent with other message types
+  - [privval] [\#2923](https://github.com/tendermint/tendermint/issues/2923) Listen for unix socket connections instead of dialing them
+
+* Apps
+
+* Go API
+  - [types] [\#2981](https://github.com/tendermint/tendermint/issues/2981) Remove `PrivValidator.GetAddress()`
+
+* Blockchain Protocol
+
+* P2P Protocol
+
+### FEATURES:
+- [rpc] [\#3052](https://github.com/tendermint/tendermint/issues/3052) Include peer's remote IP in `/net_info`
+
+### IMPROVEMENTS:
+- [consensus] [\#3086](https://github.com/tendermint/tendermint/issues/3086) Log peerID on ignored votes (@srmo)
+- [docs] [\#3061](https://github.com/tendermint/tendermint/issues/3061) Added specification for signing consensus msgs at
+  ./docs/spec/consensus/signing.md
+- [privval] [\#2948](https://github.com/tendermint/tendermint/issues/2948) Memoize pubkey so it's only requested once on startup
+- [privval] [\#2923](https://github.com/tendermint/tendermint/issues/2923) Retry RemoteSigner connections on error
+
+### BUG FIXES:
+
+- [build] [\#3085](https://github.com/tendermint/tendermint/issues/3085) Fix `Version` field in build scripts (@husio)
+- [crypto/multisig] [\#3102](https://github.com/tendermint/tendermint/issues/3102) Fix multisig keys address length
+- [crypto/encoding] [\#3101](https://github.com/tendermint/tendermint/issues/3101) Fix `PubKeyMultisigThreshold` unmarshalling into `crypto.PubKey` interface
+- [p2p/conn] [\#3111](https://github.com/tendermint/tendermint/issues/3111) Make SecretConnection thread safe
+- [rpc] [\#3053](https://github.com/tendermint/tendermint/issues/3053) Fix internal error in `/tx_search` when results are empty
+  (@gianfelipe93)
+- [types] [\#2926](https://github.com/tendermint/tendermint/issues/2926) Do not panic if retrieving the privval's public key fails
+
 ## v0.27.4
 
 *December 21st, 2018*
@@ -17,9 +72,8 @@
 ### BREAKING CHANGES:
 
 * Go API
-
-- [dep] [\#3027](https://github.com/tendermint/tendermint/issues/3027) Revert to mainline Go crypto library, eliminating the modified
-  `bcrypt.GenerateFromPassword`
+  - [dep] [\#3027](https://github.com/tendermint/tendermint/issues/3027) Revert to mainline Go crypto library, eliminating the modified
+    `bcrypt.GenerateFromPassword`
 
 ## v0.27.2
 

CHANGELOG_PENDING.md

@@ -1,4 +1,4 @@
-## v0.28.0
+## v0.29.0
 
 *TBD*
 
@@ -7,34 +7,23 @@ Special thanks to external contributors on this release:
 ### BREAKING CHANGES:
 
 * CLI/RPC/Config
-- [cli] Removed `node` `--proxy_app=dummy` option. Use `kvstore` (`persistent_kvstore`) instead.
-- [cli] Renamed `node` `--proxy_app=nilapp` to `--proxy_app=noop`.
-- [config] \#2992 `allow_duplicate_ip` is now set to false
-- [privval] \#2926 split up `PubKeyMsg` into `PubKeyRequest` and `PubKeyResponse` to be consistent with other message types
-- [privval] \#2923 listen for unix socket connections instead of dialing them
+- [types] consistent field order of `CanonicalVote` and `CanonicalProposal`
 
 * Apps
 
 * Go API
-- [types] \#2926 memoize consensus public key on initialization of remote signer and return the memoized key on
-`PrivValidator.GetPubKey()` instead of requesting it again
-- [types] \#2981 Remove `PrivValidator.GetAddress()`
+- [node] \#3082 MetricsProvider now requires you to pass a chain ID
 
 * Blockchain Protocol
+  * [merkle] \#2713 Merkle trees now match the RFC 6962 specification
 
 * P2P Protocol
 
 ### FEATURES:
-- [privval] \#1181 Split immutable and mutable parts of `priv_validator.json`
 
 ### IMPROVEMENTS:
-- [p2p/conn] \#3111 make SecretConnection thread safe
-- [privval] \#2923 retry RemoteSigner connections on error
-- [rpc] \#3047 Include peer's remote IP in `/net_info`
+- [rpc] \#3065 return maxPerPage (100), not defaultPerPage (30) if `per_page` is greater than the max 100.
+- [instrumentation] \#3082 add 'chain_id' label for all metrics
 
 ### BUG FIXES:
-
-- [types] \#2926 do not panic if retrieving the private validator's public key fails
-- [rpc] \#3080 check if the variable "skipCount" is bigger than zero. If it is not, we set it to 0. If it, we do not do anything.
-- [crypto/multisig] \#3102 fix multisig keys address length
-- [crypto/encoding] \#3101 Fix `PubKeyMultisigThreshold` unmarshalling into `crypto.PubKey` interface
+- [log] \#3060 fix year format

Makefile

@@ -292,9 +292,7 @@ build-linux:
 	GOOS=linux GOARCH=amd64 $(MAKE) build
 
 build-docker-localnode:
-	cd networks/local
-	make
-	cd -
+	@cd networks/local && make
 
 # Run a 4-node testnet locally
 localnet-start: localnet-stop

UPGRADING.md

@@ -3,6 +3,55 @@
 This guide provides steps to be followed when you upgrade your applications to
 a newer version of Tendermint Core.
 
+## v0.28.0
+
+This release breaks the format for the `priv_validator.json` file
+and the protocol used for the external validator process.
+It is compatible with v0.27.0 blockchains (neither the BlockProtocol nor the
+P2PProtocol have changed).
+
+Please read carefully for details about upgrading.
+
+**Note:** Backup your `config/priv_validator.json`
+before proceeding.
+
+### `priv_validator.json`
+
+The `config/priv_validator.json` is now two files:
+`config/priv_validator_key.json` and `data/priv_validator_state.json`.
+The former contains the key material, the later contains the details on the last
+message signed.
+
+When running v0.28.0 for the first time, it will back up any pre-existing
+`priv_validator.json` file and proceed to split it into the two new files.
+Upgrading should happen automatically without problem.
+
+To upgrade manually, use the provided `privValUpgrade.go` script, with exact paths for the old
+`priv_validator.json` and the locations for the two new files. It's recomended
+to use the default paths, of `config/priv_validator_key.json` and
+`data/priv_validator_state.json`, respectively:
+
+```
+go run scripts/privValUpgrade.go <old-path> <new-key-path> <new-state-path>
+```
+
+### External validator signers
+
+The Unix and TCP implementations of the remote signing validator
+have been consolidated into a single implementation.
+Thus in both cases, the external process is expected to dial
+Tendermint. This is different from how Unix sockets used to work, where
+Tendermint dialed the external process.
+
+The `PubKeyMsg` was also split into separate `Request` and `Response` types
+for consistency with other messages.
+
+Note that the TCP sockets don't yet use a persistent key,
+so while they're encrypted, they can't yet be properly authenticated.
+See [#3105](https://github.com/tendermint/tendermint/issues/3105).
+Note the Unix socket has neither encryption nor authentication, but will
+add a shared-secret in [#3099](https://github.com/tendermint/tendermint/issues/3099).
+
 ## v0.27.0
 
 This release contains some breaking changes to the block and p2p protocols,

blockchain/store_test.go

@@ -311,7 +311,7 @@ func TestLoadBlockPart(t *testing.T) {
 	gotPart, _, panicErr := doFn(loadPart)
 	require.Nil(t, panicErr, "an existent and proper block should not panic")
 	require.Nil(t, res, "a properly saved block should return a proper block")
-	require.Equal(t, gotPart.(*types.Part).Hash(), part1.Hash(),
+	require.Equal(t, gotPart.(*types.Part), part1,
 		"expecting successful retrieval of previously saved block")
 }
 

cmd/priv_val_server/main.go

@@ -45,7 +45,7 @@ func main() {
 		dialer = privval.DialTCPFn(address, connTimeout, ed25519.GenPrivKey())
 	default:
 		logger.Error("Unknown protocol", "protocol", protocol)
-		return
+		os.Exit(1)
 	}
 
 	rs := privval.NewRemoteSigner(logger, *chainID, pv, dialer)

consensus/metrics.go

@@ -8,7 +8,11 @@ import (
 	stdprometheus "github.com/prometheus/client_golang/prometheus"
 )
 
-const MetricsSubsystem = "consensus"
+const (
+	// MetricsSubsystem is a subsystem shared by all metrics exposed by this
+	// package.
+	MetricsSubsystem = "consensus"
+)
 
 // Metrics contains metrics exposed by this package.
 type Metrics struct {
@@ -50,101 +54,107 @@ type Metrics struct {
 }
 
 // PrometheusMetrics returns Metrics build using Prometheus client library.
-func PrometheusMetrics(namespace string) *Metrics {
+// Optionally, labels can be provided along with their values ("foo",
+// "fooValue").
+func PrometheusMetrics(namespace string, labelsAndValues ...string) *Metrics {
+	labels := []string{}
+	for i := 0; i < len(labelsAndValues); i += 2 {
+		labels = append(labels, labelsAndValues[i])
+	}
 	return &Metrics{
 		Height: prometheus.NewGaugeFrom(stdprometheus.GaugeOpts{
 			Namespace: namespace,
 			Subsystem: MetricsSubsystem,
 			Name:      "height",
 			Help:      "Height of the chain.",
-		}, []string{}),
+		}, labels).With(labelsAndValues...),
 		Rounds: prometheus.NewGaugeFrom(stdprometheus.GaugeOpts{
 			Namespace: namespace,
 			Subsystem: MetricsSubsystem,
 			Name:      "rounds",
 			Help:      "Number of rounds.",
-		}, []string{}),
+		}, labels).With(labelsAndValues...),
 
 		Validators: prometheus.NewGaugeFrom(stdprometheus.GaugeOpts{
 			Namespace: namespace,
 			Subsystem: MetricsSubsystem,
 			Name:      "validators",
 			Help:      "Number of validators.",
-		}, []string{}),
+		}, labels).With(labelsAndValues...),
 		ValidatorsPower: prometheus.NewGaugeFrom(stdprometheus.GaugeOpts{
 			Namespace: namespace,
 			Subsystem: MetricsSubsystem,
 			Name:      "validators_power",
 			Help:      "Total power of all validators.",
-		}, []string{}),
+		}, labels).With(labelsAndValues...),
 		MissingValidators: prometheus.NewGaugeFrom(stdprometheus.GaugeOpts{
 			Namespace: namespace,
 			Subsystem: MetricsSubsystem,
 			Name:      "missing_validators",
 			Help:      "Number of validators who did not sign.",
-		}, []string{}),
+		}, labels).With(labelsAndValues...),
 		MissingValidatorsPower: prometheus.NewGaugeFrom(stdprometheus.GaugeOpts{
 			Namespace: namespace,
 			Subsystem: MetricsSubsystem,
 			Name:      "missing_validators_power",
 			Help:      "Total power of the missing validators.",
-		}, []string{}),
+		}, labels).With(labelsAndValues...),
 		ByzantineValidators: prometheus.NewGaugeFrom(stdprometheus.GaugeOpts{
 			Namespace: namespace,
 			Subsystem: MetricsSubsystem,
 			Name:      "byzantine_validators",
 			Help:      "Number of validators who tried to double sign.",
-		}, []string{}),
+		}, labels).With(labelsAndValues...),
 		ByzantineValidatorsPower: prometheus.NewGaugeFrom(stdprometheus.GaugeOpts{
 			Namespace: namespace,
 			Subsystem: MetricsSubsystem,
 			Name:      "byzantine_validators_power",
 			Help:      "Total power of the byzantine validators.",
-		}, []string{}),
+		}, labels).With(labelsAndValues...),
 
 		BlockIntervalSeconds: prometheus.NewGaugeFrom(stdprometheus.GaugeOpts{
 			Namespace: namespace,
 			Subsystem: MetricsSubsystem,
 			Name:      "block_interval_seconds",
 			Help:      "Time between this and the last block.",
-		}, []string{}),
+		}, labels).With(labelsAndValues...),
 
 		NumTxs: prometheus.NewGaugeFrom(stdprometheus.GaugeOpts{
 			Namespace: namespace,
 			Subsystem: MetricsSubsystem,
 			Name:      "num_txs",
 			Help:      "Number of transactions.",
-		}, []string{}),
+		}, labels).With(labelsAndValues...),
 		BlockSizeBytes: prometheus.NewGaugeFrom(stdprometheus.GaugeOpts{
 			Namespace: namespace,
 			Subsystem: MetricsSubsystem,
 			Name:      "block_size_bytes",
 			Help:      "Size of the block.",
-		}, []string{}),
+		}, labels).With(labelsAndValues...),
 		TotalTxs: prometheus.NewGaugeFrom(stdprometheus.GaugeOpts{
 			Namespace: namespace,
 			Subsystem: MetricsSubsystem,
 			Name:      "total_txs",
 			Help:      "Total number of transactions.",
-		}, []string{}),
+		}, labels).With(labelsAndValues...),
 		CommittedHeight: prometheus.NewGaugeFrom(stdprometheus.GaugeOpts{
 			Namespace: namespace,
 			Subsystem: MetricsSubsystem,
 			Name:      "latest_block_height",
 			Help:      "The latest block height.",
-		}, []string{}),
+		}, labels).With(labelsAndValues...),
 		FastSyncing: prometheus.NewGaugeFrom(stdprometheus.GaugeOpts{
 			Namespace: namespace,
 			Subsystem: MetricsSubsystem,
 			Name:      "fast_syncing",
 			Help:      "Whether or not a node is fast syncing. 1 if yes, 0 if no.",
-		}, []string{}),
+		}, labels).With(labelsAndValues...),
 		BlockParts: prometheus.NewCounterFrom(stdprometheus.CounterOpts{
 			Namespace: namespace,
 			Subsystem: MetricsSubsystem,
 			Name:      "block_parts",
 			Help:      "Number of blockparts transmitted by peer.",
-		}, []string{"peer_id"}),
+		}, append(labels, "peer_id")).With(labelsAndValues...),
 	}
 }
 

crypto/merkle/hash.go

@@ -0,0 +1,21 @@
+package merkle
+
+import (
+	"github.com/tendermint/tendermint/crypto/tmhash"
+)
+
+// TODO: make these have a large predefined capacity
+var (
+	leafPrefix  = []byte{0}
+	innerPrefix = []byte{1}
+)
+
+// returns tmhash(0x00 || leaf)
+func leafHash(leaf []byte) []byte {
+	return tmhash.Sum(append(leafPrefix, leaf...))
+}
+
+// returns tmhash(0x01 || left || right)
+func innerHash(left []byte, right []byte) []byte {
+	return tmhash.Sum(append(innerPrefix, append(left, right...)...))
+}

crypto/merkle/proof_simple_value.go

@@ -71,11 +71,11 @@ func (op SimpleValueOp) Run(args [][]byte) ([][]byte, error) {
 	hasher.Write(value) // does not error
 	vhash := hasher.Sum(nil)
 
+	bz := new(bytes.Buffer)
 	// Wrap <op.Key, vhash> to hash the KVPair.
-	hasher = tmhash.New()
-	encodeByteSlice(hasher, []byte(op.key)) // does not error
-	encodeByteSlice(hasher, []byte(vhash))  // does not error
-	kvhash := hasher.Sum(nil)
+	encodeByteSlice(bz, []byte(op.key)) // does not error
+	encodeByteSlice(bz, []byte(vhash))  // does not error
+	kvhash := leafHash(bz.Bytes())
 
 	if !bytes.Equal(kvhash, op.Proof.LeafHash) {
 		return nil, cmn.NewError("leaf hash mismatch: want %X got %X", op.Proof.LeafHash, kvhash)

crypto/merkle/rfc6962_test.go

@@ -0,0 +1,97 @@
+package merkle
+
+// Copyright 2016 Google Inc. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+// These tests were taken from https://github.com/google/trillian/blob/master/merkle/rfc6962/rfc6962_test.go,
+// and consequently fall under the above license.
+import (
+	"bytes"
+	"encoding/hex"
+	"testing"
+
+	"github.com/tendermint/tendermint/crypto/tmhash"
+)
+
+func TestRFC6962Hasher(t *testing.T) {
+	_, leafHashTrail := trailsFromByteSlices([][]byte{[]byte("L123456")})
+	leafHash := leafHashTrail.Hash
+	_, leafHashTrail = trailsFromByteSlices([][]byte{[]byte{}})
+	emptyLeafHash := leafHashTrail.Hash
+	for _, tc := range []struct {
+		desc string
+		got  []byte
+		want string
+	}{
+		// Since creating a merkle tree of no leaves is unsupported here, we skip
+		// the corresponding trillian test vector.
+
+		// Check that the empty hash is not the same as the hash of an empty leaf.
+		// echo -n 00 | xxd -r -p | sha256sum
+		{
+			desc: "RFC6962 Empty Leaf",
+			want: "6e340b9cffb37a989ca544e6bb780a2c78901d3fb33738768511a30617afa01d"[:tmhash.Size*2],
+			got:  emptyLeafHash,
+		},
+		// echo -n 004C313233343536 | xxd -r -p | sha256sum
+		{
+			desc: "RFC6962 Leaf",
+			want: "395aa064aa4c29f7010acfe3f25db9485bbd4b91897b6ad7ad547639252b4d56"[:tmhash.Size*2],
+			got:  leafHash,
+		},
+		// echo -n 014E3132334E343536 | xxd -r -p | sha256sum
+		{
+			desc: "RFC6962 Node",
+			want: "aa217fe888e47007fa15edab33c2b492a722cb106c64667fc2b044444de66bbb"[:tmhash.Size*2],
+			got:  innerHash([]byte("N123"), []byte("N456")),
+		},
+	} {
+		t.Run(tc.desc, func(t *testing.T) {
+			wantBytes, err := hex.DecodeString(tc.want)
+			if err != nil {
+				t.Fatalf("hex.DecodeString(%x): %v", tc.want, err)
+			}
+			if got, want := tc.got, wantBytes; !bytes.Equal(got, want) {
+				t.Errorf("got %x, want %x", got, want)
+			}
+		})
+	}
+}
+
+func TestRFC6962HasherCollisions(t *testing.T) {
+	// Check that different leaves have different hashes.
+	leaf1, leaf2 := []byte("Hello"), []byte("World")
+	_, leafHashTrail := trailsFromByteSlices([][]byte{leaf1})
+	hash1 := leafHashTrail.Hash
+	_, leafHashTrail = trailsFromByteSlices([][]byte{leaf2})
+	hash2 := leafHashTrail.Hash
+	if bytes.Equal(hash1, hash2) {
+		t.Errorf("Leaf hashes should differ, but both are %x", hash1)
+	}
+	// Compute an intermediate subtree hash.
+	_, subHash1Trail := trailsFromByteSlices([][]byte{hash1, hash2})
+	subHash1 := subHash1Trail.Hash
+	// Check that this is not the same as a leaf hash of their concatenation.
+	preimage := append(hash1, hash2...)
+	_, forgedHashTrail := trailsFromByteSlices([][]byte{preimage})
+	forgedHash := forgedHashTrail.Hash
+	if bytes.Equal(subHash1, forgedHash) {
+		t.Errorf("Hasher is not second-preimage resistant")
+	}
+	// Swap the order of nodes and check that the hash is different.
+	_, subHash2Trail := trailsFromByteSlices([][]byte{hash2, hash1})
+	subHash2 := subHash2Trail.Hash
+	if bytes.Equal(subHash1, subHash2) {
+		t.Errorf("Subtree hash does not depend on the order of leaves")
+	}
+}

crypto/merkle/simple_map_test.go

@@ -13,14 +13,14 @@ func TestSimpleMap(t *testing.T) {
 		values []string // each string gets converted to []byte in test
 		want   string
 	}{
-		{[]string{"key1"}, []string{"value1"}, "321d150de16dceb51c72981b432b115045383259b1a550adf8dc80f927508967"},
-		{[]string{"key1"}, []string{"value2"}, "2a9e4baf321eac99f6eecc3406603c14bc5e85bb7b80483cbfc75b3382d24a2f"},
+		{[]string{"key1"}, []string{"value1"}, "a44d3cc7daba1a4600b00a2434b30f8b970652169810d6dfa9fb1793a2189324"},
+		{[]string{"key1"}, []string{"value2"}, "0638e99b3445caec9d95c05e1a3fc1487b4ddec6a952ff337080360b0dcc078c"},
 		// swap order with 2 keys
-		{[]string{"key1", "key2"}, []string{"value1", "value2"}, "c4d8913ab543ba26aa970646d4c99a150fd641298e3367cf68ca45fb45a49881"},
-		{[]string{"key2", "key1"}, []string{"value2", "value1"}, "c4d8913ab543ba26aa970646d4c99a150fd641298e3367cf68ca45fb45a49881"},
+		{[]string{"key1", "key2"}, []string{"value1", "value2"}, "8fd19b19e7bb3f2b3ee0574027d8a5a4cec370464ea2db2fbfa5c7d35bb0cff3"},
+		{[]string{"key2", "key1"}, []string{"value2", "value1"}, "8fd19b19e7bb3f2b3ee0574027d8a5a4cec370464ea2db2fbfa5c7d35bb0cff3"},
 		// swap order with 3 keys
-		{[]string{"key1", "key2", "key3"}, []string{"value1", "value2", "value3"}, "b23cef00eda5af4548a213a43793f2752d8d9013b3f2b64bc0523a4791196268"},
-		{[]string{"key1", "key3", "key2"}, []string{"value1", "value3", "value2"}, "b23cef00eda5af4548a213a43793f2752d8d9013b3f2b64bc0523a4791196268"},
+		{[]string{"key1", "key2", "key3"}, []string{"value1", "value2", "value3"}, "1dd674ec6782a0d586a903c9c63326a41cbe56b3bba33ed6ff5b527af6efb3dc"},
+		{[]string{"key1", "key3", "key2"}, []string{"value1", "value3", "value2"}, "1dd674ec6782a0d586a903c9c63326a41cbe56b3bba33ed6ff5b527af6efb3dc"},
 	}
 	for i, tc := range tests {
 		db := newSimpleMap()

crypto/merkle/simple_proof.go

@@ -5,7 +5,6 @@ import (
 	"errors"
 	"fmt"
 
-	"github.com/tendermint/tendermint/crypto/tmhash"
 	cmn "github.com/tendermint/tendermint/libs/common"
 )
 
@@ -67,7 +66,8 @@ func SimpleProofsFromMap(m map[string][]byte) (rootHash []byte, proofs map[strin
 
 // Verify that the SimpleProof proves the root hash.
 // Check sp.Index/sp.Total manually if needed
-func (sp *SimpleProof) Verify(rootHash []byte, leafHash []byte) error {
+func (sp *SimpleProof) Verify(rootHash []byte, leaf []byte) error {
+	leafHash := leafHash(leaf)
 	if sp.Total < 0 {
 		return errors.New("Proof total must be positive")
 	}
@@ -128,19 +128,19 @@ func computeHashFromAunts(index int, total int, leafHash []byte, innerHashes [][
 		if len(innerHashes) == 0 {
 			return nil
 		}
-		numLeft := (total + 1) / 2
+		numLeft := getSplitPoint(total)
 		if index < numLeft {
 			leftHash := computeHashFromAunts(index, numLeft, leafHash, innerHashes[:len(innerHashes)-1])
 			if leftHash == nil {
 				return nil
 			}
-			return simpleHashFromTwoHashes(leftHash, innerHashes[len(innerHashes)-1])
+			return innerHash(leftHash, innerHashes[len(innerHashes)-1])
 		}
 		rightHash := computeHashFromAunts(index-numLeft, total-numLeft, leafHash, innerHashes[:len(innerHashes)-1])
 		if rightHash == nil {
 			return nil
 		}
-		return simpleHashFromTwoHashes(innerHashes[len(innerHashes)-1], rightHash)
+		return innerHash(innerHashes[len(innerHashes)-1], rightHash)
 	}
 }
 
@@ -182,12 +182,13 @@ func trailsFromByteSlices(items [][]byte) (trails []*SimpleProofNode, root *Simp
 	case 0:
 		return nil, nil
 	case 1:
-		trail := &SimpleProofNode{tmhash.Sum(items[0]), nil, nil, nil}
+		trail := &SimpleProofNode{leafHash(items[0]), nil, nil, nil}
 		return []*SimpleProofNode{trail}, trail
 	default:
-		lefts, leftRoot := trailsFromByteSlices(items[:(len(items)+1)/2])
-		rights, rightRoot := trailsFromByteSlices(items[(len(items)+1)/2:])
-		rootHash := simpleHashFromTwoHashes(leftRoot.Hash, rightRoot.Hash)
+		k := getSplitPoint(len(items))
+		lefts, leftRoot := trailsFromByteSlices(items[:k])
+		rights, rightRoot := trailsFromByteSlices(items[k:])
+		rootHash := innerHash(leftRoot.Hash, rightRoot.Hash)
 		root := &SimpleProofNode{rootHash, nil, nil, nil}
 		leftRoot.Parent = root
 		leftRoot.Right = rightRoot

crypto/merkle/simple_tree.go

@@ -1,23 +1,9 @@
 package merkle
 
 import (
-	"github.com/tendermint/tendermint/crypto/tmhash"
+	"math/bits"
 )
 
-// simpleHashFromTwoHashes is the basic operation of the Merkle tree: Hash(left | right).
-func simpleHashFromTwoHashes(left, right []byte) []byte {
-	var hasher = tmhash.New()
-	err := encodeByteSlice(hasher, left)
-	if err != nil {
-		panic(err)
-	}
-	err = encodeByteSlice(hasher, right)
-	if err != nil {
-		panic(err)
-	}
-	return hasher.Sum(nil)
-}
-
 // SimpleHashFromByteSlices computes a Merkle tree where the leaves are the byte slice,
 // in the provided order.
 func SimpleHashFromByteSlices(items [][]byte) []byte {
@@ -25,11 +11,12 @@ func SimpleHashFromByteSlices(items [][]byte) []byte {
 	case 0:
 		return nil
 	case 1:
-		return tmhash.Sum(items[0])
+		return leafHash(items[0])
 	default:
-		left := SimpleHashFromByteSlices(items[:(len(items)+1)/2])
-		right := SimpleHashFromByteSlices(items[(len(items)+1)/2:])
-		return simpleHashFromTwoHashes(left, right)
+		k := getSplitPoint(len(items))
+		left := SimpleHashFromByteSlices(items[:k])
+		right := SimpleHashFromByteSlices(items[k:])
+		return innerHash(left, right)
 	}
 }
 
@@ -44,3 +31,17 @@ func SimpleHashFromMap(m map[string][]byte) []byte {
 	}
 	return sm.Hash()
 }
+
+// getSplitPoint returns the largest power of 2 less than length
+func getSplitPoint(length int) int {
+	if length < 1 {
+		panic("Trying to split a tree with size < 1")
+	}
+	uLength := uint(length)
+	bitlen := bits.Len(uLength)
+	k := 1 << uint(bitlen-1)
+	if k == length {
+		k >>= 1
+	}
+	return k
+}

crypto/merkle/simple_tree_test.go

@@ -34,7 +34,6 @@ func TestSimpleProof(t *testing.T) {
 
 	// For each item, check the trail.
 	for i, item := range items {
-		itemHash := tmhash.Sum(item)
 		proof := proofs[i]
 
 		// Check total/index
@@ -43,30 +42,53 @@ func TestSimpleProof(t *testing.T) {
 		require.Equal(t, proof.Total, total, "Unmatched totals: %d vs %d", proof.Total, total)
 
 		// Verify success
-		err := proof.Verify(rootHash, itemHash)
-		require.NoError(t, err, "Verificatior failed: %v.", err)
+		err := proof.Verify(rootHash, item)
+		require.NoError(t, err, "Verification failed: %v.", err)
 
 		// Trail too long should make it fail
 		origAunts := proof.Aunts
 		proof.Aunts = append(proof.Aunts, cmn.RandBytes(32))
-		err = proof.Verify(rootHash, itemHash)
+		err = proof.Verify(rootHash, item)
 		require.Error(t, err, "Expected verification to fail for wrong trail length")
 
 		proof.Aunts = origAunts
 
 		// Trail too short should make it fail
 		proof.Aunts = proof.Aunts[0 : len(proof.Aunts)-1]
-		err = proof.Verify(rootHash, itemHash)
+		err = proof.Verify(rootHash, item)
 		require.Error(t, err, "Expected verification to fail for wrong trail length")
 
 		proof.Aunts = origAunts
 
 		// Mutating the itemHash should make it fail.
-		err = proof.Verify(rootHash, MutateByteSlice(itemHash))
+		err = proof.Verify(rootHash, MutateByteSlice(item))
 		require.Error(t, err, "Expected verification to fail for mutated leaf hash")
 
 		// Mutating the rootHash should make it fail.
-		err = proof.Verify(MutateByteSlice(rootHash), itemHash)
+		err = proof.Verify(MutateByteSlice(rootHash), item)
 		require.Error(t, err, "Expected verification to fail for mutated root hash")
 	}
 }
+
+func Test_getSplitPoint(t *testing.T) {
+	tests := []struct {
+		length int
+		want   int
+	}{
+		{1, 0},
+		{2, 1},
+		{3, 2},
+		{4, 2},
+		{5, 4},
+		{10, 8},
+		{20, 16},
+		{100, 64},
+		{255, 128},
+		{256, 128},
+		{257, 256},
+	}
+	for _, tt := range tests {
+		got := getSplitPoint(tt.length)
+		require.Equal(t, tt.want, got, "getSplitPoint(%d) = %v, want %v", tt.length, got, tt.want)
+	}
+}

docs/app-dev/app-architecture.md

@@ -45,6 +45,6 @@ Tendermint.
 See the following for more extensive documentation:
 
 - [Interchain Standard for the Light-Client REST API](https://github.com/cosmos/cosmos-sdk/pull/1028)
-- [Tendermint RPC Docs](https://tendermint.github.io/slate/)
+- [Tendermint RPC Docs](https://tendermint.com/rpc/)
 - [Tendermint in Production](../tendermint-core/running-in-production.md)
 - [ABCI spec](./abci-spec.md)

docs/app-dev/ecosystem.json

@@ -63,6 +63,13 @@
       "author": "Zach Balder",
       "description": "Public service reporting and tracking"
     },
+    {
+      "name": "ParadigmCore",
+      "url": "https://github.com/ParadigmFoundation/ParadigmCore",
+      "language": "TypeScript",
+      "author": "Paradigm Labs",
+      "description": "Reference implementation of the Paradigm Protocol, and OrderStream network client."
+    },
     {
       "name": "Passchain",
       "url": "https://github.com/trusch/passchain",

docs/app-dev/indexing-transactions.md

@@ -78,7 +78,7 @@ endpoint:
 curl "localhost:26657/tx_search?query=\"account.name='igor'\"&prove=true"
 ```
 
-Check out [API docs](https://tendermint.github.io/slate/?shell#txsearch)
+Check out [API docs](https://tendermint.com/rpc/#txsearch)
 for more information on query syntax and other options.
 
 ## Subscribing to transactions
@@ -97,5 +97,5 @@ by providing a query to `/subscribe` RPC endpoint.
 }
 ```
 
-Check out [API docs](https://tendermint.github.io/slate/#subscribe) for
+Check out [API docs](https://tendermint.com/rpc/#subscribe) for
 more information on query syntax and other options.

docs/architecture/adr-020-block-size.md

@@ -7,6 +7,7 @@
 28-08-2018: Third version after Ethan's comments
 30-08-2018: AminoOverheadForBlock => MaxAminoOverheadForBlock
 31-08-2018: Bounding evidence and chain ID
+13-01-2019: Add section on MaxBytes vs MaxDataBytes
 
 ## Context
 
@@ -20,6 +21,32 @@ We should just remove MaxTxs all together and stick with MaxBytes, and have a
 But we can't just reap BlockSize.MaxBytes, since MaxBytes is for the entire block,
 not for the txs inside the block. There's extra amino overhead + the actual
 headers on top of the actual transactions + evidence + last commit.
+We could also consider using a MaxDataBytes instead of or in addition to MaxBytes.
+
+## MaxBytes vs MaxDataBytes
+
+The [PR #3045](https://github.com/tendermint/tendermint/pull/3045) suggested
+additional clarity/justification was necessary here, wither respect to the use
+of MaxDataBytes in addition to, or instead of, MaxBytes.
+
+MaxBytes provides a clear limit on the total size of a block that requires no
+additional calculation if you want to use it to bound resource usage, and there
+has been considerable discussions about optimizing tendermint around 1MB blocks.
+Regardless, we need some maximum on the size of a block so we can avoid
+unmarshalling blocks that are too big during the consensus, and it seems more
+straightforward to provide a single fixed number for this rather than a
+computation of "MaxDataBytes + everything else you need to make room for
+(signatures, evidence, header)". MaxBytes provides a simple bound so we can
+always say "blocks are less than X MB".
+
+Having both MaxBytes and MaxDataBytes feels like unnecessary complexity. It's
+not particularly surprising for MaxBytes to imply the maximum size of the
+entire block (not just txs), one just has to know that a block includes header,
+txs, evidence, votes. For more fine grained control over the txs included in the
+block, there is the MaxGas. In practice, the MaxGas may be expected to do most of
+the tx throttling, and the MaxBytes to just serve as an upper bound on the total
+size. Applications can use MaxGas as a MaxDataBytes by just taking the gas for
+every tx to be its size in bytes.
 
 ## Proposed solution
 
@@ -61,7 +88,7 @@ MaxXXX stayed the same.
 
 ## Status
 
-Proposed.
+Accepted.
 
 ## Consequences
 

docs/networks/docker-compose.md

@@ -4,7 +4,7 @@ With Docker Compose, you can spin up local testnets with a single command.
 
 ## Requirements
 
-1. [Install tendermint](/docs/install.md)
+1. [Install tendermint](/docs/introduction/install.md)
 2. [Install docker](https://docs.docker.com/engine/installation/)
 3. [Install docker-compose](https://docs.docker.com/compose/install/)
 

docs/spec/blockchain/blockchain.md

@@ -109,10 +109,6 @@ Tendermint uses the
 [Google.Protobuf.WellKnownTypes.Timestamp](https://developers.google.com/protocol-buffers/docs/reference/csharp/class/google/protobuf/well-known-types/timestamp)
 format, which uses two integers, one for Seconds and for Nanoseconds.
 
-NOTE: there is currently a small divergence between Tendermint and the
-Google.Protobuf.WellKnownTypes.Timestamp that should be resolved. See [this
-issue](https://github.com/tendermint/go-amino/issues/223) for details.
-
 ## Data
 
 Data is just a wrapper for a list of transactions, where transactions are

docs/spec/blockchain/encoding.md

@@ -144,12 +144,17 @@ func MakeParts(obj interface{}, partSize int) []Part
 For an overview of Merkle trees, see
 [wikipedia](https://en.wikipedia.org/wiki/Merkle_tree)
 
-A Simple Tree is a simple compact binary tree for a static list of items. Simple Merkle trees are used in numerous places in Tendermint to compute a cryptographic digest of a data structure. In a Simple Tree, the transactions and validation signatures of a block are hashed using this simple merkle tree logic.
+We use the RFC 6962 specification of a merkle tree, instantiated with sha256 as the hash function.
+Merkle trees are used throughout Tendermint to compute a cryptographic digest of a data structure.
+The differences between RFC 6962 and the simplest form a merkle tree are that:
 
-If the number of items is not a power of two, the tree will not be full
-and some leaf nodes will be at different levels. Simple Tree tries to
-keep both sides of the tree the same size, but the left side may be one
-greater, for example:
+1) leaf nodes and inner nodes have different hashes. 
+   This is to prevent a proof to an inner node, claiming that it is the hash of the leaf. 
+   The leaf nodes are `SHA256(0x00 || leaf_data)`, and inner nodes are `SHA256(0x01 || left_hash || right_hash)`.
+
+2) When the number of items isn't a power of two, the left half of the tree is as big as it could be.
+   (The smallest power of two less than the number of items) This allows new leaves to be added with less
+   recomputation. For example:
 
 ```
    Simple Tree with 6 items           Simple Tree with 7 items
@@ -163,48 +168,31 @@ greater, for example:
      / \             / \                / \             / \
     /   \           /   \              /   \           /   \
    /     \         /     \            /     \         /     \
-  *       h2      *       h5         *       *       *       h6
- / \             / \                / \     / \     / \
-h0  h1          h3  h4             h0  h1  h2  h3  h4  h5
-```
-
-Tendermint always uses the `TMHASH` hash function, which is equivalent to
-SHA256:
-
-```
-func TMHASH(bz []byte) []byte {
-    return SHA256(bz)
-}
+  *       *       h4     h5          *       *       *       h6
+ / \     / \                        / \     / \     / \
+h0  h1  h2 h3                      h0  h1  h2  h3  h4  h5
 ```
 
 ### Simple Merkle Root
 
-The function `SimpleMerkleRoot` is a simple recursive function defined as follows:
+The function `MerkleRoot` is a simple recursive function defined as follows:
 
 ```go
-func SimpleMerkleRoot(hashes [][]byte) []byte{
-    switch len(hashes) {
-    case 0:
-        return nil
-    case 1:
-        return hashes[0]
-    default:
-        left := SimpleMerkleRoot(hashes[:(len(hashes)+1)/2])
-        right := SimpleMerkleRoot(hashes[(len(hashes)+1)/2:])
-        return SimpleConcatHash(left, right)
-    }
-}
-
-func SimpleConcatHash(left, right []byte) []byte{
-    left = encodeByteSlice(left)
-    right = encodeByteSlice(right)
-    return TMHASH(append(left, right))
+func MerkleRootFromLeafs(leafs [][]byte) []byte{
+	switch len(items) {
+	case 0:
+		return nil
+	case 1:
+		return leafHash(leafs[0])      // SHA256(0x00 || leafs[0])
+	default:
+		k := getSplitPoint(len(items)) // largest power of two smaller than items
+		left := MerkleRootFromLeafs(items[:k])
+		right := MerkleRootFromLeafs(items[k:])
+		return innerHash(left, right)  // SHA256(0x01 || left || right)
+	}
 }
 ```
 
-Note that the leaves are Amino encoded as byte-arrays (ie. simple Uvarint length
-prefix) before being concatenated together and hashed.
-
 Note: we will abuse notion and invoke `SimpleMerkleRoot` with arguments of type `struct` or type `[]struct`.
 For `struct` arguments, we compute a `[][]byte` containing the hash of each
 field in the struct, in the same order the fields appear in the struct.
@@ -214,7 +202,7 @@ For `[]struct` arguments, we compute a `[][]byte` by hashing the individual `str
 
 Proof that a leaf is in a Merkle tree consists of a simple structure:
 
-```
+```golang
 type SimpleProof struct {
         Aunts [][]byte
 }
@@ -222,7 +210,7 @@ type SimpleProof struct {
 
 Which is verified using the following:
 
-```
+```golang
 func (proof SimpleProof) Verify(index, total int, leafHash, rootHash []byte) bool {
 	computedHash := computeHashFromAunts(index, total, leafHash, proof.Aunts)
     return computedHash == rootHash
@@ -238,7 +226,7 @@ func computeHashFromAunts(index, total int, leafHash []byte, innerHashes [][]byt
 
 	assert(len(innerHashes) > 0)
 
-	numLeft := (total + 1) / 2
+	numLeft := getSplitPoint(total) // largest power of 2 less than total
 	if index < numLeft {
 		leftHash := computeHashFromAunts(index, numLeft, leafHash, innerHashes[:len(innerHashes)-1])
 		assert(leftHash != nil)
@@ -301,8 +289,8 @@ type CanonicalVote struct {
 	Type      byte
 	Height    int64            `binary:"fixed64"`
 	Round     int64            `binary:"fixed64"`
-	Timestamp time.Time
 	BlockID   CanonicalBlockID
+	Timestamp time.Time
 	ChainID   string
 }
 ```

docs/spec/consensus/signing.md

@@ -59,9 +59,9 @@ type PartSetHeader struct {
 ```
 
 To be included in a valid vote or proposal, BlockID must either represent a `nil` block, or a complete one.
-We introduce two methods, `BlockID.IsNil()` and `BlockID.IsComplete()` for these cases, respectively.
+We introduce two methods, `BlockID.IsZero()` and `BlockID.IsComplete()` for these cases, respectively.
 
-`BlockID.IsNil()` returns true for BlockID `b` if each of the following
+`BlockID.IsZero()` returns true for BlockID `b` if each of the following
 are true:
 
 ```
@@ -81,7 +81,7 @@ len(b.PartsHeader.Hash) == 32
 
 ## Proposals
 
-The structure of a propsal for signing looks like:
+The structure of a proposal for signing looks like:
 
 ```
 type CanonicalProposal struct {
@@ -118,8 +118,8 @@ type CanonicalVote struct {
 	Type      SignedMsgType // type alias for byte
 	Height    int64         `binary:"fixed64"`
 	Round     int64         `binary:"fixed64"`
-	Timestamp time.Time
 	BlockID   BlockID
+	Timestamp time.Time
 	ChainID   string
 }
 ```
@@ -130,7 +130,7 @@ A vote is valid if each of the following lines evaluates to true for vote `v`:
 v.Type == 0x1 || v.Type == 0x2
 v.Height > 0
 v.Round >= 0
-v.BlockID.IsNil() || v.BlockID.IsValid()
+v.BlockID.IsZero() || v.BlockID.IsComplete()
 ```
 
 In other words, a vote is valid for signing if it contains the type of a Prevote

libs/log/tmfmt_logger.go

@@ -90,7 +90,7 @@ func (l tmfmtLogger) Log(keyvals ...interface{}) error {
 	//     D										- first character of the level, uppercase (ASCII only)
 	//     [2016-05-02|11:06:44.322]    - our time format (see https://golang.org/src/time/format.go)
 	//     Stopping ...					- message
-	enc.buf.WriteString(fmt.Sprintf("%c[%s] %-44s ", lvl[0]-32, time.Now().Format("2016-01-02|15:04:05.000"), msg))
+	enc.buf.WriteString(fmt.Sprintf("%c[%s] %-44s ", lvl[0]-32, time.Now().Format("2006-01-02|15:04:05.000"), msg))
 
 	if module != unknown {
 		enc.buf.WriteString("module=" + module + " ")

lite/proxy/query_test.go

@@ -10,6 +10,7 @@ import (
 	"github.com/stretchr/testify/require"
 
 	"github.com/tendermint/tendermint/abci/example/kvstore"
+	"github.com/tendermint/tendermint/crypto/merkle"
 	"github.com/tendermint/tendermint/lite"
 	certclient "github.com/tendermint/tendermint/lite/client"
 	nm "github.com/tendermint/tendermint/node"
@@ -143,12 +144,13 @@ func TestTxProofs(t *testing.T) {
 	require.NotNil(err)
 	require.Contains(err.Error(), "not found")
 
-	// Now let's check with the real tx hash.
+	// Now let's check with the real tx root hash.
 	key = types.Tx(tx).Hash()
 	res, err = cl.Tx(key, true)
 	require.NoError(err, "%#v", err)
 	require.NotNil(res)
-	err = res.Proof.Validate(key)
+	keyHash := merkle.SimpleHashFromByteSlices([][]byte{key})
+	err = res.Proof.Validate(keyHash)
 	assert.NoError(err, "%#v", err)
 
 	commit, err := GetCertifiedCommit(br.Height, cl, cert)

mempool/metrics.go

@@ -7,7 +7,11 @@ import (
 	stdprometheus "github.com/prometheus/client_golang/prometheus"
 )
 
-const MetricsSubsytem = "mempool"
+const (
+	// MetricsSubsystem is a subsystem shared by all metrics exposed by this
+	// package.
+	MetricsSubsystem = "mempool"
+)
 
 // Metrics contains metrics exposed by this package.
 // see MetricsProvider for descriptions.
@@ -23,33 +27,39 @@ type Metrics struct {
 }
 
 // PrometheusMetrics returns Metrics build using Prometheus client library.
-func PrometheusMetrics(namespace string) *Metrics {
+// Optionally, labels can be provided along with their values ("foo",
+// "fooValue").
+func PrometheusMetrics(namespace string, labelsAndValues ...string) *Metrics {
+	labels := []string{}
+	for i := 0; i < len(labelsAndValues); i += 2 {
+		labels = append(labels, labelsAndValues[i])
+	}
 	return &Metrics{
 		Size: prometheus.NewGaugeFrom(stdprometheus.GaugeOpts{
 			Namespace: namespace,
-			Subsystem: MetricsSubsytem,
+			Subsystem: MetricsSubsystem,
 			Name:      "size",
 			Help:      "Size of the mempool (number of uncommitted transactions).",
-		}, []string{}),
+		}, labels).With(labelsAndValues...),
 		TxSizeBytes: prometheus.NewHistogramFrom(stdprometheus.HistogramOpts{
 			Namespace: namespace,
-			Subsystem: MetricsSubsytem,
+			Subsystem: MetricsSubsystem,
 			Name:      "tx_size_bytes",
 			Help:      "Transaction sizes in bytes.",
 			Buckets:   stdprometheus.ExponentialBuckets(1, 3, 17),
-		}, []string{}),
+		}, labels).With(labelsAndValues...),
 		FailedTxs: prometheus.NewCounterFrom(stdprometheus.CounterOpts{
 			Namespace: namespace,
-			Subsystem: MetricsSubsytem,
+			Subsystem: MetricsSubsystem,
 			Name:      "failed_txs",
 			Help:      "Number of failed transactions.",
-		}, []string{}),
+		}, labels).With(labelsAndValues...),
 		RecheckTimes: prometheus.NewCounterFrom(stdprometheus.CounterOpts{
 			Namespace: namespace,
-			Subsystem: MetricsSubsytem,
+			Subsystem: MetricsSubsystem,
 			Name:      "recheck_times",
 			Help:      "Number of times transactions are rechecked in the mempool.",
-		}, []string{}),
+		}, labels).With(labelsAndValues...),
 	}
 }
 

node/node.go

@@ -117,15 +117,17 @@ func DefaultNewNode(config *cfg.Config, logger log.Logger) (*Node, error) {
 }
 
 // MetricsProvider returns a consensus, p2p and mempool Metrics.
-type MetricsProvider func() (*cs.Metrics, *p2p.Metrics, *mempl.Metrics, *sm.Metrics)
+type MetricsProvider func(chainID string) (*cs.Metrics, *p2p.Metrics, *mempl.Metrics, *sm.Metrics)
 
 // DefaultMetricsProvider returns Metrics build using Prometheus client library
 // if Prometheus is enabled. Otherwise, it returns no-op Metrics.
 func DefaultMetricsProvider(config *cfg.InstrumentationConfig) MetricsProvider {
-	return func() (*cs.Metrics, *p2p.Metrics, *mempl.Metrics, *sm.Metrics) {
+	return func(chainID string) (*cs.Metrics, *p2p.Metrics, *mempl.Metrics, *sm.Metrics) {
 		if config.Prometheus {
-			return cs.PrometheusMetrics(config.Namespace), p2p.PrometheusMetrics(config.Namespace),
-				mempl.PrometheusMetrics(config.Namespace), sm.PrometheusMetrics(config.Namespace)
+			return cs.PrometheusMetrics(config.Namespace, "chain_id", chainID),
+				p2p.PrometheusMetrics(config.Namespace, "chain_id", chainID),
+				mempl.PrometheusMetrics(config.Namespace, "chain_id", chainID),
+				sm.PrometheusMetrics(config.Namespace, "chain_id", chainID)
 		}
 		return cs.NopMetrics(), p2p.NopMetrics(), mempl.NopMetrics(), sm.NopMetrics()
 	}
@@ -274,7 +276,7 @@ func NewNode(config *cfg.Config,
 		consensusLogger.Info("This node is not a validator", "addr", addr, "pubKey", pubKey)
 	}
 
-	csMetrics, p2pMetrics, memplMetrics, smMetrics := metricsProvider()
+	csMetrics, p2pMetrics, memplMetrics, smMetrics := metricsProvider(genDoc.ChainID)
 
 	// Make MempoolReactor
 	mempool := mempl.NewMempool(
@@ -901,7 +903,7 @@ func createAndStartPrivValidatorSocketClient(
 
 	pvsc := privval.NewSocketVal(logger.With("module", "privval"), listener)
 	if err := pvsc.Start(); err != nil {
-		return nil, errors.Wrap(err, "failed to start")
+		return nil, errors.Wrap(err, "failed to start private validator")
 	}
 
 	return pvsc, nil

p2p/metrics.go

@@ -7,7 +7,11 @@ import (
 	stdprometheus "github.com/prometheus/client_golang/prometheus"
 )
 
-const MetricsSubsystem = "p2p"
+const (
+	// MetricsSubsystem is a subsystem shared by all metrics exposed by this
+	// package.
+	MetricsSubsystem = "p2p"
+)
 
 // Metrics contains metrics exposed by this package.
 type Metrics struct {
@@ -24,38 +28,44 @@ type Metrics struct {
 }
 
 // PrometheusMetrics returns Metrics build using Prometheus client library.
-func PrometheusMetrics(namespace string) *Metrics {
+// Optionally, labels can be provided along with their values ("foo",
+// "fooValue").
+func PrometheusMetrics(namespace string, labelsAndValues ...string) *Metrics {
+	labels := []string{}
+	for i := 0; i < len(labelsAndValues); i += 2 {
+		labels = append(labels, labelsAndValues[i])
+	}
 	return &Metrics{
 		Peers: prometheus.NewGaugeFrom(stdprometheus.GaugeOpts{
 			Namespace: namespace,
 			Subsystem: MetricsSubsystem,
 			Name:      "peers",
 			Help:      "Number of peers.",
-		}, []string{}),
+		}, labels).With(labelsAndValues...),
 		PeerReceiveBytesTotal: prometheus.NewCounterFrom(stdprometheus.CounterOpts{
 			Namespace: namespace,
 			Subsystem: MetricsSubsystem,
 			Name:      "peer_receive_bytes_total",
 			Help:      "Number of bytes received from a given peer.",
-		}, []string{"peer_id"}),
+		}, append(labels, "peer_id")).With(labelsAndValues...),
 		PeerSendBytesTotal: prometheus.NewCounterFrom(stdprometheus.CounterOpts{
 			Namespace: namespace,
 			Subsystem: MetricsSubsystem,
 			Name:      "peer_send_bytes_total",
 			Help:      "Number of bytes sent to a given peer.",
-		}, []string{"peer_id"}),
+		}, append(labels, "peer_id")).With(labelsAndValues...),
 		PeerPendingSendBytes: prometheus.NewGaugeFrom(stdprometheus.GaugeOpts{
 			Namespace: namespace,
 			Subsystem: MetricsSubsystem,
 			Name:      "peer_pending_send_bytes",
 			Help:      "Number of pending bytes to be sent to a given peer.",
-		}, []string{"peer_id"}),
+		}, append(labels, "peer_id")).With(labelsAndValues...),
 		NumTxs: prometheus.NewGaugeFrom(stdprometheus.GaugeOpts{
 			Namespace: namespace,
 			Subsystem: MetricsSubsystem,
 			Name:      "num_txs",
 			Help:      "Number of transactions submitted by each peer.",
-		}, []string{"peer_id"}),
+		}, append(labels, "peer_id")).With(labelsAndValues...),
 	}
 }
 

privval/client.go

@@ -191,19 +191,19 @@ func (sc *SocketVal) OnStop() {
 // connection is closed in OnStop.
 // returns true if the listener is closed
 // (ie. it returns a nil conn).
-func (sc *SocketVal) reset() (bool, error) {
+func (sc *SocketVal) reset() (closed bool, err error) {
 	sc.mtx.Lock()
 	defer sc.mtx.Unlock()
 
 	// first check if the conn already exists and close it.
 	if sc.signer != nil {
 		if err := sc.signer.Close(); err != nil {
-			sc.Logger.Error("error closing connection", "err", err)
+			sc.Logger.Error("error closing socket val connection during reset", "err", err)
 		}
 	}
 
 	// wait for a new conn
-	conn, err := sc.waitConnection()
+	conn, err := sc.acceptConnection()
 	if err != nil {
 		return false, err
 	}
@@ -224,6 +224,8 @@ func (sc *SocketVal) reset() (bool, error) {
 	return false, nil
 }
 
+// Attempt to accept a connection.
+// Times out after the listener's acceptDeadline
 func (sc *SocketVal) acceptConnection() (net.Conn, error) {
 	conn, err := sc.listener.Accept()
 	if err != nil {
@@ -231,33 +233,6 @@ func (sc *SocketVal) acceptConnection() (net.Conn, error) {
 			return nil, nil // Ignore error from listener closing.
 		}
 		return nil, err
-
 	}
 	return conn, nil
 }
-
-// waitConnection uses the configured wait timeout to error if no external
-// process connects in the time period.
-func (sc *SocketVal) waitConnection() (net.Conn, error) {
-	var (
-		connc = make(chan net.Conn, 1)
-		errc  = make(chan error, 1)
-	)
-
-	go func(connc chan<- net.Conn, errc chan<- error) {
-		conn, err := sc.acceptConnection()
-		if err != nil {
-			errc <- err
-			return
-		}
-
-		connc <- conn
-	}(connc, errc)
-
-	select {
-	case conn := <-connc:
-		return conn, nil
-	case err := <-errc:
-		return nil, err
-	}
-}

privval/client_test.go

@@ -13,7 +13,6 @@ import (
 	cmn "github.com/tendermint/tendermint/libs/common"
 	"github.com/tendermint/tendermint/libs/log"
 
-	p2pconn "github.com/tendermint/tendermint/p2p/conn"
 	"github.com/tendermint/tendermint/types"
 )
 
@@ -27,363 +26,386 @@ var (
 	testHeartbeatTimeout3o2 = 6 * time.Millisecond // 3/2 of the other one
 )
 
+type socketTestCase struct {
+	addr   string
+	dialer Dialer
+}
+
+func socketTestCases(t *testing.T) []socketTestCase {
+	tcpAddr := fmt.Sprintf("tcp://%s", testFreeTCPAddr(t))
+	unixFilePath, err := testUnixAddr()
+	require.NoError(t, err)
+	unixAddr := fmt.Sprintf("unix://%s", unixFilePath)
+	return []socketTestCase{
+		socketTestCase{
+			addr:   tcpAddr,
+			dialer: DialTCPFn(tcpAddr, testConnDeadline, ed25519.GenPrivKey()),
+		},
+		socketTestCase{
+			addr:   unixAddr,
+			dialer: DialUnixFn(unixFilePath),
+		},
+	}
+}
+
 func TestSocketPVAddress(t *testing.T) {
-	var (
-		chainID = cmn.RandStr(12)
-		sc, rs  = testSetupSocketPair(t, chainID, types.NewMockPV())
-	)
-	defer sc.Stop()
-	defer rs.Stop()
+	for _, tc := range socketTestCases(t) {
+		// Execute the test within a closure to ensure the deferred statements
+		// are called between each for loop iteration, for isolated test cases.
+		func() {
+			var (
+				chainID = cmn.RandStr(12)
+				sc, rs  = testSetupSocketPair(t, chainID, types.NewMockPV(), tc.addr, tc.dialer)
+			)
+			defer sc.Stop()
+			defer rs.Stop()
 
-	serverAddr := rs.privVal.GetPubKey().Address()
-	clientAddr := sc.GetPubKey().Address()
+			serverAddr := rs.privVal.GetPubKey().Address()
+			clientAddr := sc.GetPubKey().Address()
 
-	assert.Equal(t, serverAddr, clientAddr)
+			assert.Equal(t, serverAddr, clientAddr)
+		}()
+	}
 }
 
 func TestSocketPVPubKey(t *testing.T) {
-	var (
-		chainID = cmn.RandStr(12)
-		sc, rs  = testSetupSocketPair(t, chainID, types.NewMockPV())
-	)
-	defer sc.Stop()
-	defer rs.Stop()
+	for _, tc := range socketTestCases(t) {
+		func() {
+			var (
+				chainID = cmn.RandStr(12)
+				sc, rs  = testSetupSocketPair(t, chainID, types.NewMockPV(), tc.addr, tc.dialer)
+			)
+			defer sc.Stop()
+			defer rs.Stop()
 
-	clientKey := sc.GetPubKey()
+			clientKey := sc.GetPubKey()
 
-	privvalPubKey := rs.privVal.GetPubKey()
+			privvalPubKey := rs.privVal.GetPubKey()
 
-	assert.Equal(t, privvalPubKey, clientKey)
+			assert.Equal(t, privvalPubKey, clientKey)
+		}()
+	}
 }
 
 func TestSocketPVProposal(t *testing.T) {
-	var (
-		chainID = cmn.RandStr(12)
-		sc, rs  = testSetupSocketPair(t, chainID, types.NewMockPV())
+	for _, tc := range socketTestCases(t) {
+		func() {
+			var (
+				chainID = cmn.RandStr(12)
+				sc, rs  = testSetupSocketPair(t, chainID, types.NewMockPV(), tc.addr, tc.dialer)
 
-		ts             = time.Now()
-		privProposal   = &types.Proposal{Timestamp: ts}
-		clientProposal = &types.Proposal{Timestamp: ts}
-	)
-	defer sc.Stop()
-	defer rs.Stop()
+				ts             = time.Now()
+				privProposal   = &types.Proposal{Timestamp: ts}
+				clientProposal = &types.Proposal{Timestamp: ts}
+			)
+			defer sc.Stop()
+			defer rs.Stop()
 
-	require.NoError(t, rs.privVal.SignProposal(chainID, privProposal))
-	require.NoError(t, sc.SignProposal(chainID, clientProposal))
-	assert.Equal(t, privProposal.Signature, clientProposal.Signature)
+			require.NoError(t, rs.privVal.SignProposal(chainID, privProposal))
+			require.NoError(t, sc.SignProposal(chainID, clientProposal))
+			assert.Equal(t, privProposal.Signature, clientProposal.Signature)
+		}()
+	}
 }
 
 func TestSocketPVVote(t *testing.T) {
-	var (
-		chainID = cmn.RandStr(12)
-		sc, rs  = testSetupSocketPair(t, chainID, types.NewMockPV())
+	for _, tc := range socketTestCases(t) {
+		func() {
+			var (
+				chainID = cmn.RandStr(12)
+				sc, rs  = testSetupSocketPair(t, chainID, types.NewMockPV(), tc.addr, tc.dialer)
 
-		ts    = time.Now()
-		vType = types.PrecommitType
-		want  = &types.Vote{Timestamp: ts, Type: vType}
-		have  = &types.Vote{Timestamp: ts, Type: vType}
-	)
-	defer sc.Stop()
-	defer rs.Stop()
+				ts    = time.Now()
+				vType = types.PrecommitType
+				want  = &types.Vote{Timestamp: ts, Type: vType}
+				have  = &types.Vote{Timestamp: ts, Type: vType}
+			)
+			defer sc.Stop()
+			defer rs.Stop()
 
-	require.NoError(t, rs.privVal.SignVote(chainID, want))
-	require.NoError(t, sc.SignVote(chainID, have))
-	assert.Equal(t, want.Signature, have.Signature)
+			require.NoError(t, rs.privVal.SignVote(chainID, want))
+			require.NoError(t, sc.SignVote(chainID, have))
+			assert.Equal(t, want.Signature, have.Signature)
+		}()
+	}
 }
 
 func TestSocketPVVoteResetDeadline(t *testing.T) {
-	var (
-		chainID = cmn.RandStr(12)
-		sc, rs  = testSetupSocketPair(t, chainID, types.NewMockPV())
+	for _, tc := range socketTestCases(t) {
+		func() {
+			var (
+				chainID = cmn.RandStr(12)
+				sc, rs  = testSetupSocketPair(t, chainID, types.NewMockPV(), tc.addr, tc.dialer)
 
-		ts    = time.Now()
-		vType = types.PrecommitType
-		want  = &types.Vote{Timestamp: ts, Type: vType}
-		have  = &types.Vote{Timestamp: ts, Type: vType}
-	)
-	defer sc.Stop()
-	defer rs.Stop()
+				ts    = time.Now()
+				vType = types.PrecommitType
+				want  = &types.Vote{Timestamp: ts, Type: vType}
+				have  = &types.Vote{Timestamp: ts, Type: vType}
+			)
+			defer sc.Stop()
+			defer rs.Stop()
 
-	time.Sleep(testConnDeadline2o3)
+			time.Sleep(testConnDeadline2o3)
 
-	require.NoError(t, rs.privVal.SignVote(chainID, want))
-	require.NoError(t, sc.SignVote(chainID, have))
-	assert.Equal(t, want.Signature, have.Signature)
+			require.NoError(t, rs.privVal.SignVote(chainID, want))
+			require.NoError(t, sc.SignVote(chainID, have))
+			assert.Equal(t, want.Signature, have.Signature)
 
-	// This would exceed the deadline if it was not extended by the previous message
-	time.Sleep(testConnDeadline2o3)
+			// This would exceed the deadline if it was not extended by the previous message
+			time.Sleep(testConnDeadline2o3)
 
-	require.NoError(t, rs.privVal.SignVote(chainID, want))
-	require.NoError(t, sc.SignVote(chainID, have))
-	assert.Equal(t, want.Signature, have.Signature)
+			require.NoError(t, rs.privVal.SignVote(chainID, want))
+			require.NoError(t, sc.SignVote(chainID, have))
+			assert.Equal(t, want.Signature, have.Signature)
+		}()
+	}
 }
 
 func TestSocketPVVoteKeepalive(t *testing.T) {
-	var (
-		chainID = cmn.RandStr(12)
-		sc, rs  = testSetupSocketPair(t, chainID, types.NewMockPV())
+	for _, tc := range socketTestCases(t) {
+		func() {
+			var (
+				chainID = cmn.RandStr(12)
+				sc, rs  = testSetupSocketPair(t, chainID, types.NewMockPV(), tc.addr, tc.dialer)
 
-		ts    = time.Now()
-		vType = types.PrecommitType
-		want  = &types.Vote{Timestamp: ts, Type: vType}
-		have  = &types.Vote{Timestamp: ts, Type: vType}
-	)
-	defer sc.Stop()
-	defer rs.Stop()
+				ts    = time.Now()
+				vType = types.PrecommitType
+				want  = &types.Vote{Timestamp: ts, Type: vType}
+				have  = &types.Vote{Timestamp: ts, Type: vType}
+			)
+			defer sc.Stop()
+			defer rs.Stop()
 
-	time.Sleep(testConnDeadline * 2)
+			time.Sleep(testConnDeadline * 2)
 
-	require.NoError(t, rs.privVal.SignVote(chainID, want))
-	require.NoError(t, sc.SignVote(chainID, have))
-	assert.Equal(t, want.Signature, have.Signature)
+			require.NoError(t, rs.privVal.SignVote(chainID, want))
+			require.NoError(t, sc.SignVote(chainID, have))
+			assert.Equal(t, want.Signature, have.Signature)
+		}()
+	}
 }
 
 func TestSocketPVDeadline(t *testing.T) {
-	var (
-		addr            = testFreeAddr(t)
-		listenc         = make(chan struct{})
-		thisConnTimeout = 100 * time.Millisecond
-		sc              = newSocketVal(log.TestingLogger(), addr, thisConnTimeout)
-	)
+	for _, tc := range socketTestCases(t) {
+		func() {
+			var (
+				listenc         = make(chan struct{})
+				thisConnTimeout = 100 * time.Millisecond
+				sc              = newSocketVal(log.TestingLogger(), tc.addr, thisConnTimeout)
+			)
 
-	go func(sc *SocketVal) {
-		defer close(listenc)
+			go func(sc *SocketVal) {
+				defer close(listenc)
 
-		assert.Equal(t, sc.Start().(cmn.Error).Data(), ErrConnTimeout)
+				// Note: the TCP connection times out at the accept() phase,
+				// whereas the Unix domain sockets connection times out while
+				// attempting to fetch the remote signer's public key.
+				assert.True(t, IsConnTimeout(sc.Start()))
 
-		assert.False(t, sc.IsRunning())
-	}(sc)
+				assert.False(t, sc.IsRunning())
+			}(sc)
 
-	for {
-		conn, err := cmn.Connect(addr)
-		if err != nil {
-			continue
-		}
-
-		_, err = p2pconn.MakeSecretConnection(
-			conn,
-			ed25519.GenPrivKey(),
-		)
-		if err == nil {
-			break
-		}
-	}
-
-	<-listenc
-}
-
-func TestRemoteSignerRetry(t *testing.T) {
-	var (
-		attemptc = make(chan int)
-		retries  = 2
-	)
-
-	ln, err := net.Listen("tcp", "127.0.0.1:0")
-	require.NoError(t, err)
-
-	go func(ln net.Listener, attemptc chan<- int) {
-		attempts := 0
-
-		for {
-			conn, err := ln.Accept()
-			require.NoError(t, err)
-
-			err = conn.Close()
-			require.NoError(t, err)
-
-			attempts++
-
-			if attempts == retries {
-				attemptc <- attempts
-				break
+			for {
+				_, err := cmn.Connect(tc.addr)
+				if err == nil {
+					break
+				}
 			}
-		}
-	}(ln, attemptc)
 
-	rs := NewRemoteSigner(
-		log.TestingLogger(),
-		cmn.RandStr(12),
-		types.NewMockPV(),
-		DialTCPFn(ln.Addr().String(), testConnDeadline, ed25519.GenPrivKey()),
-	)
-	defer rs.Stop()
-
-	RemoteSignerConnDeadline(time.Millisecond)(rs)
-	RemoteSignerConnRetries(retries)(rs)
-
-	assert.Equal(t, rs.Start(), ErrDialRetryMax)
-
-	select {
-	case attempts := <-attemptc:
-		assert.Equal(t, retries, attempts)
-	case <-time.After(100 * time.Millisecond):
-		t.Error("expected remote to observe connection attempts")
+			<-listenc
+		}()
 	}
 }
 
 func TestRemoteSignVoteErrors(t *testing.T) {
-	var (
-		chainID = cmn.RandStr(12)
-		sc, rs  = testSetupSocketPair(t, chainID, types.NewErroringMockPV())
+	for _, tc := range socketTestCases(t) {
+		func() {
+			var (
+				chainID = cmn.RandStr(12)
+				sc, rs  = testSetupSocketPair(t, chainID, types.NewErroringMockPV(), tc.addr, tc.dialer)
 
-		ts    = time.Now()
-		vType = types.PrecommitType
-		vote  = &types.Vote{Timestamp: ts, Type: vType}
-	)
-	defer sc.Stop()
-	defer rs.Stop()
+				ts    = time.Now()
+				vType = types.PrecommitType
+				vote  = &types.Vote{Timestamp: ts, Type: vType}
+			)
+			defer sc.Stop()
+			defer rs.Stop()
 
-	err := sc.SignVote("", vote)
-	require.Equal(t, err.(*RemoteSignerError).Description, types.ErroringMockPVErr.Error())
+			err := sc.SignVote("", vote)
+			require.Equal(t, err.(*RemoteSignerError).Description, types.ErroringMockPVErr.Error())
 
-	err = rs.privVal.SignVote(chainID, vote)
-	require.Error(t, err)
-	err = sc.SignVote(chainID, vote)
-	require.Error(t, err)
+			err = rs.privVal.SignVote(chainID, vote)
+			require.Error(t, err)
+			err = sc.SignVote(chainID, vote)
+			require.Error(t, err)
+		}()
+	}
 }
 
 func TestRemoteSignProposalErrors(t *testing.T) {
-	var (
-		chainID = cmn.RandStr(12)
-		sc, rs  = testSetupSocketPair(t, chainID, types.NewErroringMockPV())
+	for _, tc := range socketTestCases(t) {
+		func() {
+			var (
+				chainID = cmn.RandStr(12)
+				sc, rs  = testSetupSocketPair(t, chainID, types.NewErroringMockPV(), tc.addr, tc.dialer)
 
-		ts       = time.Now()
-		proposal = &types.Proposal{Timestamp: ts}
-	)
-	defer sc.Stop()
-	defer rs.Stop()
+				ts       = time.Now()
+				proposal = &types.Proposal{Timestamp: ts}
+			)
+			defer sc.Stop()
+			defer rs.Stop()
 
-	err := sc.SignProposal("", proposal)
-	require.Equal(t, err.(*RemoteSignerError).Description, types.ErroringMockPVErr.Error())
+			err := sc.SignProposal("", proposal)
+			require.Equal(t, err.(*RemoteSignerError).Description, types.ErroringMockPVErr.Error())
 
-	err = rs.privVal.SignProposal(chainID, proposal)
-	require.Error(t, err)
+			err = rs.privVal.SignProposal(chainID, proposal)
+			require.Error(t, err)
 
-	err = sc.SignProposal(chainID, proposal)
-	require.Error(t, err)
+			err = sc.SignProposal(chainID, proposal)
+			require.Error(t, err)
+		}()
+	}
 }
 
 func TestErrUnexpectedResponse(t *testing.T) {
-	var (
-		addr    = testFreeAddr(t)
-		logger  = log.TestingLogger()
-		chainID = cmn.RandStr(12)
-		readyc  = make(chan struct{})
-		errc    = make(chan error, 1)
+	for _, tc := range socketTestCases(t) {
+		func() {
+			var (
+				logger  = log.TestingLogger()
+				chainID = cmn.RandStr(12)
+				readyc  = make(chan struct{})
+				errc    = make(chan error, 1)
 
-		rs = NewRemoteSigner(
-			logger,
-			chainID,
-			types.NewMockPV(),
-			DialTCPFn(addr, testConnDeadline, ed25519.GenPrivKey()),
-		)
-		sc = newSocketVal(logger, addr, testConnDeadline)
-	)
+				rs = NewRemoteSigner(
+					logger,
+					chainID,
+					types.NewMockPV(),
+					tc.dialer,
+				)
+				sc = newSocketVal(logger, tc.addr, testConnDeadline)
+			)
 
-	testStartSocketPV(t, readyc, sc)
-	defer sc.Stop()
-	RemoteSignerConnDeadline(time.Millisecond)(rs)
-	RemoteSignerConnRetries(100)(rs)
-	// we do not want to Start() the remote signer here and instead use the connection to
-	// reply with intentionally wrong replies below:
-	rsConn, err := rs.connect()
-	defer rsConn.Close()
-	require.NoError(t, err)
-	require.NotNil(t, rsConn)
-	// send over public key to get the remote signer running:
-	go testReadWriteResponse(t, &PubKeyResponse{}, rsConn)
-	<-readyc
+			testStartSocketPV(t, readyc, sc)
+			defer sc.Stop()
+			RemoteSignerConnDeadline(time.Millisecond)(rs)
+			RemoteSignerConnRetries(100)(rs)
+			// we do not want to Start() the remote signer here and instead use the connection to
+			// reply with intentionally wrong replies below:
+			rsConn, err := rs.connect()
+			defer rsConn.Close()
+			require.NoError(t, err)
+			require.NotNil(t, rsConn)
+			// send over public key to get the remote signer running:
+			go testReadWriteResponse(t, &PubKeyResponse{}, rsConn)
+			<-readyc
 
-	// Proposal:
-	go func(errc chan error) {
-		errc <- sc.SignProposal(chainID, &types.Proposal{})
-	}(errc)
-	// read request and write wrong response:
-	go testReadWriteResponse(t, &SignedVoteResponse{}, rsConn)
-	err = <-errc
-	require.Error(t, err)
-	require.Equal(t, err, ErrUnexpectedResponse)
+			// Proposal:
+			go func(errc chan error) {
+				errc <- sc.SignProposal(chainID, &types.Proposal{})
+			}(errc)
+			// read request and write wrong response:
+			go testReadWriteResponse(t, &SignedVoteResponse{}, rsConn)
+			err = <-errc
+			require.Error(t, err)
+			require.Equal(t, err, ErrUnexpectedResponse)
 
-	// Vote:
-	go func(errc chan error) {
-		errc <- sc.SignVote(chainID, &types.Vote{})
-	}(errc)
-	// read request and write wrong response:
-	go testReadWriteResponse(t, &SignedProposalResponse{}, rsConn)
-	err = <-errc
-	require.Error(t, err)
-	require.Equal(t, err, ErrUnexpectedResponse)
+			// Vote:
+			go func(errc chan error) {
+				errc <- sc.SignVote(chainID, &types.Vote{})
+			}(errc)
+			// read request and write wrong response:
+			go testReadWriteResponse(t, &SignedProposalResponse{}, rsConn)
+			err = <-errc
+			require.Error(t, err)
+			require.Equal(t, err, ErrUnexpectedResponse)
+		}()
+	}
 }
 
-func TestRetryTCPConnToRemoteSigner(t *testing.T) {
-	var (
-		addr    = testFreeAddr(t)
-		logger  = log.TestingLogger()
-		chainID = cmn.RandStr(12)
-		readyc  = make(chan struct{})
+func TestRetryConnToRemoteSigner(t *testing.T) {
+	for _, tc := range socketTestCases(t) {
+		func() {
+			var (
+				logger  = log.TestingLogger()
+				chainID = cmn.RandStr(12)
+				readyc  = make(chan struct{})
 
-		rs = NewRemoteSigner(
-			logger,
-			chainID,
-			types.NewMockPV(),
-			DialTCPFn(addr, testConnDeadline, ed25519.GenPrivKey()),
-		)
-		thisConnTimeout = testConnDeadline
-		sc              = newSocketVal(logger, addr, thisConnTimeout)
-	)
-	// Ping every:
-	SocketValHeartbeat(testHeartbeatTimeout)(sc)
+				rs = NewRemoteSigner(
+					logger,
+					chainID,
+					types.NewMockPV(),
+					tc.dialer,
+				)
+				thisConnTimeout = testConnDeadline
+				sc              = newSocketVal(logger, tc.addr, thisConnTimeout)
+			)
+			// Ping every:
+			SocketValHeartbeat(testHeartbeatTimeout)(sc)
 
-	RemoteSignerConnDeadline(testConnDeadline)(rs)
-	RemoteSignerConnRetries(10)(rs)
+			RemoteSignerConnDeadline(testConnDeadline)(rs)
+			RemoteSignerConnRetries(10)(rs)
 
-	testStartSocketPV(t, readyc, sc)
-	defer sc.Stop()
-	require.NoError(t, rs.Start())
-	assert.True(t, rs.IsRunning())
+			testStartSocketPV(t, readyc, sc)
+			defer sc.Stop()
+			require.NoError(t, rs.Start())
+			assert.True(t, rs.IsRunning())
 
-	<-readyc
-	time.Sleep(testHeartbeatTimeout * 2)
+			<-readyc
+			time.Sleep(testHeartbeatTimeout * 2)
 
-	rs.Stop()
-	rs2 := NewRemoteSigner(
-		logger,
-		chainID,
-		types.NewMockPV(),
-		DialTCPFn(addr, testConnDeadline, ed25519.GenPrivKey()),
-	)
-	// let some pings pass
-	time.Sleep(testHeartbeatTimeout3o2)
-	require.NoError(t, rs2.Start())
-	assert.True(t, rs2.IsRunning())
-	defer rs2.Stop()
+			rs.Stop()
+			rs2 := NewRemoteSigner(
+				logger,
+				chainID,
+				types.NewMockPV(),
+				tc.dialer,
+			)
+			// let some pings pass
+			time.Sleep(testHeartbeatTimeout3o2)
+			require.NoError(t, rs2.Start())
+			assert.True(t, rs2.IsRunning())
+			defer rs2.Stop()
 
-	// give the client some time to re-establish the conn to the remote signer
-	// should see sth like this in the logs:
-	//
-	// E[10016-01-10|17:12:46.128] Ping                                         err="remote signer timed out"
-	// I[10016-01-10|17:16:42.447] Re-created connection to remote signer       impl=SocketVal
-	time.Sleep(testConnDeadline * 2)
+			// give the client some time to re-establish the conn to the remote signer
+			// should see sth like this in the logs:
+			//
+			// E[10016-01-10|17:12:46.128] Ping                                         err="remote signer timed out"
+			// I[10016-01-10|17:16:42.447] Re-created connection to remote signer       impl=SocketVal
+			time.Sleep(testConnDeadline * 2)
+		}()
+	}
 }
 
 func newSocketVal(logger log.Logger, addr string, connDeadline time.Duration) *SocketVal {
-	ln, err := net.Listen(cmn.ProtocolAndAddress(addr))
+	proto, address := cmn.ProtocolAndAddress(addr)
+	ln, err := net.Listen(proto, address)
+	logger.Info("Listening at", "proto", proto, "address", address)
 	if err != nil {
 		panic(err)
 	}
-	tcpLn := NewTCPListener(ln, ed25519.GenPrivKey())
-	TCPListenerAcceptDeadline(testAcceptDeadline)(tcpLn)
-	TCPListenerConnDeadline(testConnDeadline)(tcpLn)
-	return NewSocketVal(logger, tcpLn)
+	var svln net.Listener
+	if proto == "unix" {
+		unixLn := NewUnixListener(ln)
+		UnixListenerAcceptDeadline(testAcceptDeadline)(unixLn)
+		UnixListenerConnDeadline(connDeadline)(unixLn)
+		svln = unixLn
+	} else {
+		tcpLn := NewTCPListener(ln, ed25519.GenPrivKey())
+		TCPListenerAcceptDeadline(testAcceptDeadline)(tcpLn)
+		TCPListenerConnDeadline(connDeadline)(tcpLn)
+		svln = tcpLn
+	}
+	return NewSocketVal(logger, svln)
 }
 
 func testSetupSocketPair(
 	t *testing.T,
 	chainID string,
 	privValidator types.PrivValidator,
+	addr string,
+	dialer Dialer,
 ) (*SocketVal, *RemoteSigner) {
 	var (
-		addr    = testFreeAddr(t)
 		logger  = log.TestingLogger()
 		privVal = privValidator
 		readyc  = make(chan struct{})
@@ -391,7 +413,7 @@ func testSetupSocketPair(
 			logger,
 			chainID,
 			privVal,
-			DialTCPFn(addr, testConnDeadline, ed25519.GenPrivKey()),
+			dialer,
 		)
 
 		thisConnTimeout = testConnDeadline
@@ -429,8 +451,8 @@ func testStartSocketPV(t *testing.T, readyc chan struct{}, sc *SocketVal) {
 	}(sc)
 }
 
-// testFreeAddr claims a free port so we don't block on listener being ready.
-func testFreeAddr(t *testing.T) string {
+// testFreeTCPAddr claims a free port so we don't block on listener being ready.
+func testFreeTCPAddr(t *testing.T) string {
 	ln, err := net.Listen("tcp", "127.0.0.1:0")
 	require.NoError(t, err)
 	defer ln.Close()

privval/remote_signer.go

@@ -258,3 +258,18 @@ func handleRequest(req RemoteSignerMsg, chainID string, privVal types.PrivValida
 
 	return res, err
 }
+
+// IsConnTimeout returns a boolean indicating whether the error is known to
+// report that a connection timeout occurred. This detects both fundamental
+// network timeouts, as well as ErrConnTimeout errors.
+func IsConnTimeout(err error) bool {
+	if cmnErr, ok := err.(cmn.Error); ok {
+		if cmnErr.Data() == ErrConnTimeout {
+			return true
+		}
+	}
+	if _, ok := err.(timeoutError); ok {
+		return true
+	}
+	return false
+}

privval/remote_signer_test.go

@@ -0,0 +1,90 @@
+package privval
+
+import (
+	"net"
+	"testing"
+	"time"
+
+	"github.com/pkg/errors"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"github.com/tendermint/tendermint/crypto/ed25519"
+	cmn "github.com/tendermint/tendermint/libs/common"
+	"github.com/tendermint/tendermint/libs/log"
+	"github.com/tendermint/tendermint/types"
+)
+
+// TestRemoteSignerRetryTCPOnly will test connection retry attempts over TCP. We
+// don't need this for Unix sockets because the OS instantly knows the state of
+// both ends of the socket connection. This basically causes the
+// RemoteSigner.dialer() call inside RemoteSigner.connect() to return
+// successfully immediately, putting an instant stop to any retry attempts.
+func TestRemoteSignerRetryTCPOnly(t *testing.T) {
+	var (
+		attemptc = make(chan int)
+		retries  = 2
+	)
+
+	ln, err := net.Listen("tcp", "127.0.0.1:0")
+	require.NoError(t, err)
+
+	go func(ln net.Listener, attemptc chan<- int) {
+		attempts := 0
+
+		for {
+			conn, err := ln.Accept()
+			require.NoError(t, err)
+
+			err = conn.Close()
+			require.NoError(t, err)
+
+			attempts++
+
+			if attempts == retries {
+				attemptc <- attempts
+				break
+			}
+		}
+	}(ln, attemptc)
+
+	rs := NewRemoteSigner(
+		log.TestingLogger(),
+		cmn.RandStr(12),
+		types.NewMockPV(),
+		DialTCPFn(ln.Addr().String(), testConnDeadline, ed25519.GenPrivKey()),
+	)
+	defer rs.Stop()
+
+	RemoteSignerConnDeadline(time.Millisecond)(rs)
+	RemoteSignerConnRetries(retries)(rs)
+
+	assert.Equal(t, rs.Start(), ErrDialRetryMax)
+
+	select {
+	case attempts := <-attemptc:
+		assert.Equal(t, retries, attempts)
+	case <-time.After(100 * time.Millisecond):
+		t.Error("expected remote to observe connection attempts")
+	}
+}
+
+func TestIsConnTimeoutForFundamentalTimeouts(t *testing.T) {
+	// Generate a networking timeout
+	dialer := DialTCPFn(testFreeTCPAddr(t), time.Millisecond, ed25519.GenPrivKey())
+	_, err := dialer()
+	assert.Error(t, err)
+	assert.True(t, IsConnTimeout(err))
+}
+
+func TestIsConnTimeoutForWrappedConnTimeouts(t *testing.T) {
+	dialer := DialTCPFn(testFreeTCPAddr(t), time.Millisecond, ed25519.GenPrivKey())
+	_, err := dialer()
+	assert.Error(t, err)
+	err = cmn.ErrorWrap(ErrConnTimeout, err.Error())
+	assert.True(t, IsConnTimeout(err))
+}
+
+func TestIsConnTimeoutForNonTimeoutErrors(t *testing.T) {
+	assert.False(t, IsConnTimeout(cmn.ErrorWrap(ErrDialRetryMax, "max retries exceeded")))
+	assert.False(t, IsConnTimeout(errors.New("completely irrelevant error")))
+}

privval/socket.go

@@ -157,7 +157,7 @@ type timeoutConn struct {
 	connDeadline time.Duration
 }
 
-// newTimeoutConn returns an instance of newTCPTimeoutConn.
+// newTimeoutConn returns an instance of timeoutConn.
 func newTimeoutConn(
 	conn net.Conn,
 	connDeadline time.Duration) *timeoutConn {

privval/socket_test.go

@@ -1,7 +1,9 @@
 package privval
 
 import (
+	"io/ioutil"
 	"net"
+	"os"
 	"testing"
 	"time"
 
@@ -18,67 +20,114 @@ func newPrivKey() ed25519.PrivKeyEd25519 {
 //-------------------------------------------
 // tests
 
-func TestTCPListenerAcceptDeadline(t *testing.T) {
+type listenerTestCase struct {
+	description string // For test reporting purposes.
+	listener    net.Listener
+	dialer      Dialer
+}
+
+// testUnixAddr will attempt to obtain a platform-independent temporary file
+// name for a Unix socket
+func testUnixAddr() (string, error) {
+	f, err := ioutil.TempFile("", "tendermint-privval-test-*")
+	if err != nil {
+		return "", err
+	}
+	addr := f.Name()
+	f.Close()
+	os.Remove(addr)
+	return addr, nil
+}
+
+func tcpListenerTestCase(t *testing.T, acceptDeadline, connectDeadline time.Duration) listenerTestCase {
 	ln, err := net.Listen("tcp", "127.0.0.1:0")
 	if err != nil {
 		t.Fatal(err)
 	}
 
 	tcpLn := NewTCPListener(ln, newPrivKey())
-	TCPListenerAcceptDeadline(time.Millisecond)(tcpLn)
-	TCPListenerConnDeadline(time.Second)(tcpLn)
-
-	_, err = tcpLn.Accept()
-	opErr, ok := err.(*net.OpError)
-	if !ok {
-		t.Fatalf("have %v, want *net.OpError", err)
-	}
-
-	if have, want := opErr.Op, "accept"; have != want {
-		t.Errorf("have %v, want %v", have, want)
+	TCPListenerAcceptDeadline(acceptDeadline)(tcpLn)
+	TCPListenerConnDeadline(connectDeadline)(tcpLn)
+	return listenerTestCase{
+		description: "TCP",
+		listener:    tcpLn,
+		dialer:      DialTCPFn(ln.Addr().String(), testConnDeadline, newPrivKey()),
 	}
 }
 
-func TestTCPListenerConnDeadline(t *testing.T) {
-	ln, err := net.Listen("tcp", "127.0.0.1:0")
+func unixListenerTestCase(t *testing.T, acceptDeadline, connectDeadline time.Duration) listenerTestCase {
+	addr, err := testUnixAddr()
+	if err != nil {
+		t.Fatal(err)
+	}
+	ln, err := net.Listen("unix", addr)
 	if err != nil {
 		t.Fatal(err)
 	}
 
-	tcpLn := NewTCPListener(ln, newPrivKey())
-	TCPListenerAcceptDeadline(time.Second)(tcpLn)
-	TCPListenerConnDeadline(time.Millisecond)(tcpLn)
+	unixLn := NewUnixListener(ln)
+	UnixListenerAcceptDeadline(acceptDeadline)(unixLn)
+	UnixListenerConnDeadline(connectDeadline)(unixLn)
+	return listenerTestCase{
+		description: "Unix",
+		listener:    unixLn,
+		dialer:      DialUnixFn(addr),
+	}
+}
 
-	readyc := make(chan struct{})
-	donec := make(chan struct{})
-	go func(ln net.Listener) {
-		defer close(donec)
+func listenerTestCases(t *testing.T, acceptDeadline, connectDeadline time.Duration) []listenerTestCase {
+	return []listenerTestCase{
+		tcpListenerTestCase(t, acceptDeadline, connectDeadline),
+		unixListenerTestCase(t, acceptDeadline, connectDeadline),
+	}
+}
 
-		c, err := ln.Accept()
+func TestListenerAcceptDeadlines(t *testing.T) {
+	for _, tc := range listenerTestCases(t, time.Millisecond, time.Second) {
+		_, err := tc.listener.Accept()
+		opErr, ok := err.(*net.OpError)
+		if !ok {
+			t.Fatalf("for %s listener, have %v, want *net.OpError", tc.description, err)
+		}
+
+		if have, want := opErr.Op, "accept"; have != want {
+			t.Errorf("for %s listener,  have %v, want %v", tc.description, have, want)
+		}
+	}
+}
+
+func TestListenerConnectDeadlines(t *testing.T) {
+	for _, tc := range listenerTestCases(t, time.Second, time.Millisecond) {
+		readyc := make(chan struct{})
+		donec := make(chan struct{})
+		go func(ln net.Listener) {
+			defer close(donec)
+
+			c, err := ln.Accept()
+			if err != nil {
+				t.Fatal(err)
+			}
+			<-readyc
+
+			time.Sleep(2 * time.Millisecond)
+
+			msg := make([]byte, 200)
+			_, err = c.Read(msg)
+			opErr, ok := err.(*net.OpError)
+			if !ok {
+				t.Fatalf("for %s listener, have %v, want *net.OpError", tc.description, err)
+			}
+
+			if have, want := opErr.Op, "read"; have != want {
+				t.Errorf("for %s listener, have %v, want %v", tc.description, have, want)
+			}
+		}(tc.listener)
+
+		_, err := tc.dialer()
 		if err != nil {
 			t.Fatal(err)
 		}
-		<-readyc
-
-		time.Sleep(2 * time.Millisecond)
-
-		msg := make([]byte, 200)
-		_, err = c.Read(msg)
-		opErr, ok := err.(*net.OpError)
-		if !ok {
-			t.Fatalf("have %v, want *net.OpError", err)
-		}
-
-		if have, want := opErr.Op, "read"; have != want {
-			t.Errorf("have %v, want %v", have, want)
-		}
-	}(tcpLn)
-
-	dialer := DialTCPFn(ln.Addr().String(), testConnDeadline, newPrivKey())
-	_, err = dialer()
-	if err != nil {
-		t.Fatal(err)
+		close(readyc)
+		<-donec
 	}
-	close(readyc)
-	<-donec
 }

rpc/core/pipe.go

@@ -149,8 +149,10 @@ func validatePage(page, perPage, totalCount int) int {
 }
 
 func validatePerPage(perPage int) int {
-	if perPage < 1 || perPage > maxPerPage {
+	if perPage < 1 {
 		return defaultPerPage
+	} else if perPage > maxPerPage {
+		return maxPerPage
 	}
 	return perPage
 }

rpc/core/pipe_test.go

@@ -47,7 +47,6 @@ func TestPaginationPage(t *testing.T) {
 }
 
 func TestPaginationPerPage(t *testing.T) {
-
 	cases := []struct {
 		totalCount int
 		perPage    int
@@ -59,7 +58,7 @@ func TestPaginationPerPage(t *testing.T) {
 		{5, defaultPerPage, defaultPerPage},
 		{5, maxPerPage - 1, maxPerPage - 1},
 		{5, maxPerPage, maxPerPage},
-		{5, maxPerPage + 1, defaultPerPage},
+		{5, maxPerPage + 1, maxPerPage},
 	}
 
 	for _, c := range cases {

scripts/json2wal/main.go

@@ -45,7 +45,10 @@ func main() {
 	}
 	defer walFile.Close()
 
-	br := bufio.NewReader(f)
+	// the length of tendermint/wal/MsgInfo in the wal.json may exceed the defaultBufSize(4096) of bufio
+	// because of the byte array in BlockPart
+	// leading to unmarshal error: unexpected end of JSON input
+	br := bufio.NewReaderSize(f, 2*types.BlockPartSizeBytes)
 	dec := cs.NewWALEncoder(walFile)
 
 	for {

state/metrics.go

@@ -7,14 +7,26 @@ import (
 	stdprometheus "github.com/prometheus/client_golang/prometheus"
 )
 
-const MetricsSubsystem = "state"
+const (
+	// MetricsSubsystem is a subsystem shared by all metrics exposed by this
+	// package.
+	MetricsSubsystem = "state"
+)
 
+// Metrics contains metrics exposed by this package.
 type Metrics struct {
 	// Time between BeginBlock and EndBlock.
 	BlockProcessingTime metrics.Histogram
 }
 
-func PrometheusMetrics(namespace string) *Metrics {
+// PrometheusMetrics returns Metrics build using Prometheus client library.
+// Optionally, labels can be provided along with their values ("foo",
+// "fooValue").
+func PrometheusMetrics(namespace string, labelsAndValues ...string) *Metrics {
+	labels := []string{}
+	for i := 0; i < len(labelsAndValues); i += 2 {
+		labels = append(labels, labelsAndValues[i])
+	}
 	return &Metrics{
 		BlockProcessingTime: prometheus.NewHistogramFrom(stdprometheus.HistogramOpts{
 			Namespace: namespace,
@@ -22,10 +34,11 @@ func PrometheusMetrics(namespace string) *Metrics {
 			Name:      "block_processing_time",
 			Help:      "Time between BeginBlock and EndBlock in ms.",
 			Buckets:   stdprometheus.LinearBuckets(1, 10, 10),
-		}, []string{}),
+		}, labels).With(labelsAndValues...),
 	}
 }
 
+// NopMetrics returns no-op Metrics.
 func NopMetrics() *Metrics {
 	return &Metrics{
 		BlockProcessingTime: discard.NewHistogram(),

state/state_test.go

@@ -3,6 +3,7 @@ package state
 import (
 	"bytes"
 	"fmt"
+	"math/big"
 	"testing"
 
 	"github.com/stretchr/testify/assert"
@@ -263,6 +264,346 @@ func TestOneValidatorChangesSaveLoad(t *testing.T) {
 	}
 }
 
+func TestProposerPriorityDoesNotGetResetToZero(t *testing.T) {
+	// assert that we preserve accum when calling updateState:
+	// https://github.com/tendermint/tendermint/issues/2718
+	tearDown, _, state := setupTestCase(t)
+	defer tearDown(t)
+	origVotingPower := int64(10)
+	val1PubKey := ed25519.GenPrivKey().PubKey()
+	val1 := &types.Validator{Address: val1PubKey.Address(), PubKey: val1PubKey, VotingPower: origVotingPower}
+
+	state.Validators = types.NewValidatorSet([]*types.Validator{val1})
+	state.NextValidators = state.Validators
+
+	// NewValidatorSet calls IncrementProposerPriority but uses on a copy of val1
+	assert.EqualValues(t, 0, val1.ProposerPriority)
+
+	block := makeBlock(state, state.LastBlockHeight+1)
+	blockID := types.BlockID{block.Hash(), block.MakePartSet(testPartSize).Header()}
+	abciResponses := &ABCIResponses{
+		EndBlock: &abci.ResponseEndBlock{ValidatorUpdates: nil},
+	}
+	validatorUpdates, err := types.PB2TM.ValidatorUpdates(abciResponses.EndBlock.ValidatorUpdates)
+	require.NoError(t, err)
+	updatedState, err := updateState(state, blockID, &block.Header, abciResponses, validatorUpdates)
+	assert.NoError(t, err)
+
+	assert.Equal(t, -origVotingPower, updatedState.NextValidators.Validators[0].ProposerPriority)
+
+	// add a validator
+	val2PubKey := ed25519.GenPrivKey().PubKey()
+	val2VotingPower := int64(100)
+	updateAddVal := abci.ValidatorUpdate{PubKey: types.TM2PB.PubKey(val2PubKey), Power: val2VotingPower}
+	validatorUpdates, err = types.PB2TM.ValidatorUpdates([]abci.ValidatorUpdate{updateAddVal})
+	assert.NoError(t, err)
+	updatedState2, err := updateState(updatedState, blockID, &block.Header, abciResponses, validatorUpdates)
+	assert.NoError(t, err)
+
+	require.Equal(t, len(updatedState2.NextValidators.Validators), 2)
+	_, addedVal2 := updatedState2.NextValidators.GetByAddress(val2PubKey.Address())
+	// adding a validator should not lead to a ProposerPriority equal to zero (unless the combination of averaging and
+	// incrementing would cause so; which is not the case here)
+	totalPowerBefore2 := origVotingPower                                                      // 10
+	wantVal2ProposerPrio := -(totalPowerBefore2 + (totalPowerBefore2 >> 3)) + val2VotingPower // 89
+	avg := (0 + wantVal2ProposerPrio) / 2                                                     // 44
+	wantVal2ProposerPrio -= avg                                                               // 45
+	totalPowerAfter := origVotingPower + val2VotingPower                                      // 110
+	wantVal2ProposerPrio -= totalPowerAfter                                                   // -65
+	assert.Equal(t, wantVal2ProposerPrio, addedVal2.ProposerPriority)                         // not zero == -65
+
+	// Updating a validator does not reset the ProposerPriority to zero:
+	updatedVotingPowVal2 := int64(1)
+	updateVal := abci.ValidatorUpdate{PubKey: types.TM2PB.PubKey(val2PubKey), Power: updatedVotingPowVal2}
+	validatorUpdates, err = types.PB2TM.ValidatorUpdates([]abci.ValidatorUpdate{updateVal})
+	assert.NoError(t, err)
+	updatedState3, err := updateState(updatedState2, blockID, &block.Header, abciResponses, validatorUpdates)
+	assert.NoError(t, err)
+
+	require.Equal(t, len(updatedState3.NextValidators.Validators), 2)
+	_, prevVal1 := updatedState3.Validators.GetByAddress(val1PubKey.Address())
+	_, updatedVal2 := updatedState3.NextValidators.GetByAddress(val2PubKey.Address())
+
+	expectedVal1PrioBeforeAvg := prevVal1.ProposerPriority + prevVal1.VotingPower // -44 + 10 == -34
+	wantVal2ProposerPrio = wantVal2ProposerPrio + updatedVotingPowVal2            // -64
+	avg = (wantVal2ProposerPrio + expectedVal1PrioBeforeAvg) / 2                  // (-64-34)/2 == -49
+	wantVal2ProposerPrio = wantVal2ProposerPrio - avg                             // -15
+	assert.Equal(t, wantVal2ProposerPrio, updatedVal2.ProposerPriority)           // -15
+}
+
+func TestProposerPriorityProposerAlternates(t *testing.T) {
+	// Regression test that would fail if the inner workings of
+	// IncrementProposerPriority change.
+	// Additionally, make sure that same power validators alternate if both
+	// have the same voting power (and the 2nd was added later).
+	tearDown, _, state := setupTestCase(t)
+	defer tearDown(t)
+	origVotinPower := int64(10)
+	val1PubKey := ed25519.GenPrivKey().PubKey()
+	val1 := &types.Validator{Address: val1PubKey.Address(), PubKey: val1PubKey, VotingPower: origVotinPower}
+
+	// reset state validators to above validator
+	state.Validators = types.NewValidatorSet([]*types.Validator{val1})
+	state.NextValidators = state.Validators
+	// we only have one validator:
+	assert.Equal(t, val1PubKey.Address(), state.Validators.Proposer.Address)
+
+	block := makeBlock(state, state.LastBlockHeight+1)
+	blockID := types.BlockID{block.Hash(), block.MakePartSet(testPartSize).Header()}
+	// no updates:
+	abciResponses := &ABCIResponses{
+		EndBlock: &abci.ResponseEndBlock{ValidatorUpdates: nil},
+	}
+	validatorUpdates, err := types.PB2TM.ValidatorUpdates(abciResponses.EndBlock.ValidatorUpdates)
+	require.NoError(t, err)
+
+	updatedState, err := updateState(state, blockID, &block.Header, abciResponses, validatorUpdates)
+	assert.NoError(t, err)
+
+	// 0 + 10 (initial prio) - 10 (avg) - 10 (mostest - total) = -10
+	assert.Equal(t, -origVotinPower, updatedState.NextValidators.Validators[0].ProposerPriority)
+	assert.Equal(t, val1PubKey.Address(), updatedState.NextValidators.Proposer.Address)
+
+	// add a validator with the same voting power as the first
+	val2PubKey := ed25519.GenPrivKey().PubKey()
+	updateAddVal := abci.ValidatorUpdate{PubKey: types.TM2PB.PubKey(val2PubKey), Power: origVotinPower}
+	validatorUpdates, err = types.PB2TM.ValidatorUpdates([]abci.ValidatorUpdate{updateAddVal})
+	assert.NoError(t, err)
+
+	updatedState2, err := updateState(updatedState, blockID, &block.Header, abciResponses, validatorUpdates)
+	assert.NoError(t, err)
+
+	require.Equal(t, len(updatedState2.NextValidators.Validators), 2)
+	assert.Equal(t, updatedState2.Validators, updatedState.NextValidators)
+
+	// val1 will still be proposer as val2 just got added:
+	assert.Equal(t, val1PubKey.Address(), updatedState.NextValidators.Proposer.Address)
+	assert.Equal(t, updatedState2.Validators.Proposer.Address, updatedState2.NextValidators.Proposer.Address)
+	assert.Equal(t, updatedState2.Validators.Proposer.Address, val1PubKey.Address())
+	assert.Equal(t, updatedState2.NextValidators.Proposer.Address, val1PubKey.Address())
+
+	_, updatedVal1 := updatedState2.NextValidators.GetByAddress(val1PubKey.Address())
+	_, oldVal1 := updatedState2.Validators.GetByAddress(val1PubKey.Address())
+	_, updatedVal2 := updatedState2.NextValidators.GetByAddress(val2PubKey.Address())
+
+	totalPower := origVotinPower
+	v2PrioWhenAddedVal2 := -(totalPower + (totalPower >> 3))
+	v2PrioWhenAddedVal2 = v2PrioWhenAddedVal2 + origVotinPower       // -11 + 10 == -1
+	v1PrioWhenAddedVal2 := oldVal1.ProposerPriority + origVotinPower // -10 + 10 == 0
+	// have to express the AVG in big.Ints as -1/2 == -1 in big.Int while -1/2 == 0 in int64
+	avgSum := big.NewInt(0).Add(big.NewInt(v2PrioWhenAddedVal2), big.NewInt(v1PrioWhenAddedVal2))
+	avg := avgSum.Div(avgSum, big.NewInt(2))
+	expectedVal2Prio := v2PrioWhenAddedVal2 - avg.Int64()
+	totalPower = 2 * origVotinPower // 10 + 10
+	expectedVal1Prio := oldVal1.ProposerPriority + origVotinPower - avg.Int64() - totalPower
+	// val1's ProposerPriority story: -10 (see above) + 10 (voting pow) - (-1) (avg) - 20 (total) == -19
+	assert.EqualValues(t, expectedVal1Prio, updatedVal1.ProposerPriority)
+	// val2 prio when added: -(totalVotingPower + (totalVotingPower >> 3)) == -11
+	// -> -11 + 10  (voting power) - (-1) (avg) == 0
+	assert.EqualValues(t, expectedVal2Prio, updatedVal2.ProposerPriority, "unexpected proposer priority for validator: %v", updatedVal2)
+
+	validatorUpdates, err = types.PB2TM.ValidatorUpdates(abciResponses.EndBlock.ValidatorUpdates)
+	require.NoError(t, err)
+
+	updatedState3, err := updateState(updatedState2, blockID, &block.Header, abciResponses, validatorUpdates)
+	assert.NoError(t, err)
+
+	// proposer changes from now on (every iteration) as long as there are no changes in the validator set:
+	assert.NotEqual(t, updatedState3.Validators.Proposer.Address, updatedState3.NextValidators.Proposer.Address)
+
+	assert.Equal(t, updatedState3.Validators, updatedState2.NextValidators)
+	_, updatedVal1 = updatedState3.NextValidators.GetByAddress(val1PubKey.Address())
+	_, oldVal1 = updatedState3.Validators.GetByAddress(val1PubKey.Address())
+	_, updatedVal2 = updatedState3.NextValidators.GetByAddress(val2PubKey.Address())
+	_, oldVal2 := updatedState3.Validators.GetByAddress(val2PubKey.Address())
+
+	// val2 will be proposer:
+	assert.Equal(t, val2PubKey.Address(), updatedState3.NextValidators.Proposer.Address)
+	// check if expected proposer prio is matched:
+
+	avgSum = big.NewInt(oldVal1.ProposerPriority + origVotinPower + oldVal2.ProposerPriority + origVotinPower)
+	avg = avgSum.Div(avgSum, big.NewInt(2))
+	expectedVal1Prio2 := oldVal1.ProposerPriority + origVotinPower - avg.Int64()
+	expectedVal2Prio2 := oldVal2.ProposerPriority + origVotinPower - avg.Int64() - totalPower
+
+	// -19 + 10 - 0 (avg) == -9
+	assert.EqualValues(t, expectedVal1Prio2, updatedVal1.ProposerPriority, "unexpected proposer priority for validator: %v", updatedVal2)
+	// 0 + 10 - 0 (avg) - 20 (total) == -10
+	assert.EqualValues(t, expectedVal2Prio2, updatedVal2.ProposerPriority, "unexpected proposer priority for validator: %v", updatedVal2)
+
+	// no changes in voting power and both validators have same voting power
+	// -> proposers should alternate:
+	oldState := updatedState3
+	for i := 0; i < 1000; i++ {
+		// no validator updates:
+		abciResponses := &ABCIResponses{
+			EndBlock: &abci.ResponseEndBlock{ValidatorUpdates: nil},
+		}
+		validatorUpdates, err = types.PB2TM.ValidatorUpdates(abciResponses.EndBlock.ValidatorUpdates)
+		require.NoError(t, err)
+
+		updatedState, err := updateState(oldState, blockID, &block.Header, abciResponses, validatorUpdates)
+		assert.NoError(t, err)
+		// alternate (and cyclic priorities):
+		assert.NotEqual(t, updatedState.Validators.Proposer.Address, updatedState.NextValidators.Proposer.Address, "iter: %v", i)
+		assert.Equal(t, oldState.Validators.Proposer.Address, updatedState.NextValidators.Proposer.Address, "iter: %v", i)
+
+		_, updatedVal1 = updatedState.NextValidators.GetByAddress(val1PubKey.Address())
+		_, updatedVal2 = updatedState.NextValidators.GetByAddress(val2PubKey.Address())
+
+		if i%2 == 0 {
+			assert.Equal(t, updatedState.Validators.Proposer.Address, val2PubKey.Address())
+			assert.Equal(t, expectedVal1Prio, updatedVal1.ProposerPriority) // -19
+			assert.Equal(t, expectedVal2Prio, updatedVal2.ProposerPriority) // 0
+		} else {
+			assert.Equal(t, updatedState.Validators.Proposer.Address, val1PubKey.Address())
+			assert.Equal(t, expectedVal1Prio2, updatedVal1.ProposerPriority) // -9
+			assert.Equal(t, expectedVal2Prio2, updatedVal2.ProposerPriority) // -10
+		}
+		// update for next iteration:
+		oldState = updatedState
+	}
+}
+
+func TestLargeGenesisValidator(t *testing.T) {
+	tearDown, _, state := setupTestCase(t)
+	defer tearDown(t)
+	// TODO: increase genesis voting power to sth. more  close to MaxTotalVotingPower with changes that
+	// fix with tendermint/issues/2960; currently, the last iteration would take forever though
+	genesisVotingPower := int64(types.MaxTotalVotingPower / 100000000000000)
+	genesisPubKey := ed25519.GenPrivKey().PubKey()
+	// fmt.Println("genesis addr: ", genesisPubKey.Address())
+	genesisVal := &types.Validator{Address: genesisPubKey.Address(), PubKey: genesisPubKey, VotingPower: genesisVotingPower}
+	// reset state validators to above validator
+	state.Validators = types.NewValidatorSet([]*types.Validator{genesisVal})
+	state.NextValidators = state.Validators
+	require.True(t, len(state.Validators.Validators) == 1)
+
+	// update state a few times with no validator updates
+	// asserts that the single validator's ProposerPrio stays the same
+	oldState := state
+	for i := 0; i < 10; i++ {
+		// no updates:
+		abciResponses := &ABCIResponses{
+			EndBlock: &abci.ResponseEndBlock{ValidatorUpdates: nil},
+		}
+		validatorUpdates, err := types.PB2TM.ValidatorUpdates(abciResponses.EndBlock.ValidatorUpdates)
+		require.NoError(t, err)
+
+		block := makeBlock(oldState, oldState.LastBlockHeight+1)
+		blockID := types.BlockID{block.Hash(), block.MakePartSet(testPartSize).Header()}
+
+		updatedState, err := updateState(oldState, blockID, &block.Header, abciResponses, validatorUpdates)
+		// no changes in voting power (ProposerPrio += VotingPower == 0 in 1st round; than shiftByAvg == no-op,
+		// than -Total == -Voting)
+		// -> no change in ProposerPrio (stays -Total == -VotingPower):
+		assert.EqualValues(t, oldState.NextValidators, updatedState.NextValidators)
+		assert.EqualValues(t, -genesisVotingPower, updatedState.NextValidators.Proposer.ProposerPriority)
+
+		oldState = updatedState
+	}
+	// add another validator, do a few iterations (create blocks),
+	// add more validators with same voting power as the 2nd
+	// let the genesis validator "unbond",
+	// see how long it takes until the effect wears off and both begin to alternate
+	// see: https://github.com/tendermint/tendermint/issues/2960
+	firstAddedValPubKey := ed25519.GenPrivKey().PubKey()
+	// fmt.Println("first added addr: ", firstAddedValPubKey.Address())
+	firstAddedValVotingPower := int64(10)
+	firstAddedVal := abci.ValidatorUpdate{PubKey: types.TM2PB.PubKey(firstAddedValPubKey), Power: firstAddedValVotingPower}
+	validatorUpdates, err := types.PB2TM.ValidatorUpdates([]abci.ValidatorUpdate{firstAddedVal})
+	assert.NoError(t, err)
+	abciResponses := &ABCIResponses{
+		EndBlock: &abci.ResponseEndBlock{ValidatorUpdates: []abci.ValidatorUpdate{firstAddedVal}},
+	}
+	block := makeBlock(oldState, oldState.LastBlockHeight+1)
+	blockID := types.BlockID{block.Hash(), block.MakePartSet(testPartSize).Header()}
+	updatedState, err := updateState(oldState, blockID, &block.Header, abciResponses, validatorUpdates)
+
+	lastState := updatedState
+	for i := 0; i < 200; i++ {
+		// no updates:
+		abciResponses := &ABCIResponses{
+			EndBlock: &abci.ResponseEndBlock{ValidatorUpdates: nil},
+		}
+		validatorUpdates, err := types.PB2TM.ValidatorUpdates(abciResponses.EndBlock.ValidatorUpdates)
+		require.NoError(t, err)
+
+		block := makeBlock(lastState, lastState.LastBlockHeight+1)
+		blockID := types.BlockID{block.Hash(), block.MakePartSet(testPartSize).Header()}
+
+		updatedStateInner, err := updateState(lastState, blockID, &block.Header, abciResponses, validatorUpdates)
+		lastState = updatedStateInner
+	}
+	// set state to last state of above iteration
+	state = lastState
+
+	// set oldState to state before above iteration
+	oldState = updatedState
+	_, oldGenesisVal := oldState.NextValidators.GetByAddress(genesisVal.Address)
+	_, newGenesisVal := state.NextValidators.GetByAddress(genesisVal.Address)
+	_, addedOldVal := oldState.NextValidators.GetByAddress(firstAddedValPubKey.Address())
+	_, addedNewVal := state.NextValidators.GetByAddress(firstAddedValPubKey.Address())
+	// expect large negative proposer priority for both (genesis validator decreased, 2nd validator increased):
+	assert.True(t, oldGenesisVal.ProposerPriority > newGenesisVal.ProposerPriority)
+	assert.True(t, addedOldVal.ProposerPriority < addedNewVal.ProposerPriority)
+
+	// add 10 validators with the same voting power as the one added directly after genesis:
+	for i := 0; i < 10; i++ {
+		addedPubKey := ed25519.GenPrivKey().PubKey()
+
+		addedVal := abci.ValidatorUpdate{PubKey: types.TM2PB.PubKey(addedPubKey), Power: firstAddedValVotingPower}
+		validatorUpdates, err := types.PB2TM.ValidatorUpdates([]abci.ValidatorUpdate{addedVal})
+		assert.NoError(t, err)
+
+		abciResponses := &ABCIResponses{
+			EndBlock: &abci.ResponseEndBlock{ValidatorUpdates: []abci.ValidatorUpdate{addedVal}},
+		}
+		block := makeBlock(oldState, oldState.LastBlockHeight+1)
+		blockID := types.BlockID{block.Hash(), block.MakePartSet(testPartSize).Header()}
+		state, err = updateState(state, blockID, &block.Header, abciResponses, validatorUpdates)
+	}
+	require.Equal(t, 10+2, len(state.NextValidators.Validators))
+
+	// remove genesis validator:
+	removeGenesisVal := abci.ValidatorUpdate{PubKey: types.TM2PB.PubKey(genesisPubKey), Power: 0}
+	abciResponses = &ABCIResponses{
+		EndBlock: &abci.ResponseEndBlock{ValidatorUpdates: []abci.ValidatorUpdate{removeGenesisVal}},
+	}
+	block = makeBlock(oldState, oldState.LastBlockHeight+1)
+	blockID = types.BlockID{block.Hash(), block.MakePartSet(testPartSize).Header()}
+	validatorUpdates, err = types.PB2TM.ValidatorUpdates(abciResponses.EndBlock.ValidatorUpdates)
+	require.NoError(t, err)
+	updatedState, err = updateState(state, blockID, &block.Header, abciResponses, validatorUpdates)
+	require.NoError(t, err)
+	// only the first added val (not the genesis val) should be left
+	assert.Equal(t, 11, len(updatedState.NextValidators.Validators))
+
+	// call update state until the effect for the 3rd added validator
+	// being proposer for a long time after the genesis validator left wears off:
+	curState := updatedState
+	count := 0
+	isProposerUnchanged := true
+	for isProposerUnchanged {
+		abciResponses := &ABCIResponses{
+			EndBlock: &abci.ResponseEndBlock{ValidatorUpdates: nil},
+		}
+		validatorUpdates, err = types.PB2TM.ValidatorUpdates(abciResponses.EndBlock.ValidatorUpdates)
+		require.NoError(t, err)
+		block = makeBlock(curState, curState.LastBlockHeight+1)
+		blockID = types.BlockID{block.Hash(), block.MakePartSet(testPartSize).Header()}
+		curState, err = updateState(curState, blockID, &block.Header, abciResponses, validatorUpdates)
+		if !bytes.Equal(curState.Validators.Proposer.Address, curState.NextValidators.Proposer.Address) {
+			isProposerUnchanged = false
+		}
+		count++
+	}
+	// first proposer change happens after this many iters; we probably want to lower this number:
+	// TODO: change with https://github.com/tendermint/tendermint/issues/2960
+	firstProposerChangeExpectedAfter := 438
+	assert.Equal(t, firstProposerChangeExpectedAfter, count)
+}
+
 func TestStoreLoadValidatorsIncrementsProposerPriority(t *testing.T) {
 	const valSetSize = 2
 	tearDown, stateDB, state := setupTestCase(t)

types/block.go

@@ -11,6 +11,7 @@ import (
 
 	"github.com/tendermint/tendermint/crypto"
 	"github.com/tendermint/tendermint/crypto/merkle"
+	"github.com/tendermint/tendermint/crypto/tmhash"
 	cmn "github.com/tendermint/tendermint/libs/common"
 	"github.com/tendermint/tendermint/version"
 )
@@ -788,11 +789,6 @@ type BlockID struct {
 	PartsHeader PartSetHeader `json:"parts"`
 }
 
-// IsZero returns true if this is the BlockID for a nil-block
-func (blockID BlockID) IsZero() bool {
-	return len(blockID.Hash) == 0 && blockID.PartsHeader.IsZero()
-}
-
 // Equals returns true if the BlockID matches the given BlockID
 func (blockID BlockID) Equals(other BlockID) bool {
 	return bytes.Equal(blockID.Hash, other.Hash) &&
@@ -820,6 +816,19 @@ func (blockID BlockID) ValidateBasic() error {
 	return nil
 }
 
+// IsZero returns true if this is the BlockID of a nil block.
+func (blockID BlockID) IsZero() bool {
+	return len(blockID.Hash) == 0 &&
+		blockID.PartsHeader.IsZero()
+}
+
+// IsComplete returns true if this is a valid BlockID of a non-nil block.
+func (blockID BlockID) IsComplete() bool {
+	return len(blockID.Hash) == tmhash.Size &&
+		blockID.PartsHeader.Total > 0 &&
+		len(blockID.PartsHeader.Hash) == tmhash.Size
+}
+
 // String returns a human readable string representation of the BlockID
 func (blockID BlockID) String() string {
 	return fmt.Sprintf(`%v:%v`, blockID.Hash, blockID.PartsHeader)

types/canonical.go

@@ -36,8 +36,8 @@ type CanonicalVote struct {
 	Type      SignedMsgType // type alias for byte
 	Height    int64         `binary:"fixed64"`
 	Round     int64         `binary:"fixed64"`
-	Timestamp time.Time
 	BlockID   CanonicalBlockID
+	Timestamp time.Time
 	ChainID   string
 }
 
@@ -75,8 +75,8 @@ func CanonicalizeVote(chainID string, vote *Vote) CanonicalVote {
 		Type:      vote.Type,
 		Height:    vote.Height,
 		Round:     int64(vote.Round), // cast int->int64 to make amino encode it fixed64 (does not work for int)
-		Timestamp: vote.Timestamp,
 		BlockID:   CanonicalizeBlockID(vote.BlockID),
+		Timestamp: vote.Timestamp,
 		ChainID:   chainID,
 	}
 }

types/part_set.go

@@ -9,7 +9,6 @@ import (
 	"github.com/pkg/errors"
 
 	"github.com/tendermint/tendermint/crypto/merkle"
-	"github.com/tendermint/tendermint/crypto/tmhash"
 	cmn "github.com/tendermint/tendermint/libs/common"
 )
 
@@ -27,16 +26,6 @@ type Part struct {
 	hash []byte
 }
 
-func (part *Part) Hash() []byte {
-	if part.hash != nil {
-		return part.hash
-	}
-	hasher := tmhash.New()
-	hasher.Write(part.Bytes) // nolint: errcheck, gas
-	part.hash = hasher.Sum(nil)
-	return part.hash
-}
-
 // ValidateBasic performs basic validation.
 func (part *Part) ValidateBasic() error {
 	if part.Index < 0 {
@@ -75,7 +64,7 @@ func (psh PartSetHeader) String() string {
 }
 
 func (psh PartSetHeader) IsZero() bool {
-	return psh.Total == 0
+	return psh.Total == 0 && len(psh.Hash) == 0
 }
 
 func (psh PartSetHeader) Equals(other PartSetHeader) bool {
@@ -217,7 +206,7 @@ func (ps *PartSet) AddPart(part *Part) (bool, error) {
 	}
 
 	// Check hash proof
-	if part.Proof.Verify(ps.Hash(), part.Hash()) != nil {
+	if part.Proof.Verify(ps.Hash(), part.Bytes) != nil {
 		return false, ErrPartSetInvalidProof
 	}
 

types/proposal.go

@@ -60,6 +60,10 @@ func (p *Proposal) ValidateBasic() error {
 	if err := p.BlockID.ValidateBasic(); err != nil {
 		return fmt.Errorf("Wrong BlockID: %v", err)
 	}
+	// ValidateBasic above would pass even if the BlockID was empty:
+	if !p.BlockID.IsComplete() {
+		return fmt.Errorf("Expected a complete, non-empty BlockID, got: %v", p.BlockID)
+	}
 
 	// NOTE: Timestamp validation is subtle and handled elsewhere.
 

types/results_test.go

@@ -38,7 +38,7 @@ func TestABCIResults(t *testing.T) {
 
 	for i, res := range results {
 		proof := results.ProveResult(i)
-		valid := proof.Verify(root, res.Hash())
+		valid := proof.Verify(root, res.Bytes())
 		assert.NoError(t, valid, "%d", i)
 	}
 }

types/tx.go

@@ -31,13 +31,14 @@ func (tx Tx) String() string {
 // Txs is a slice of Tx.
 type Txs []Tx
 
-// Hash returns the simple Merkle root hash of the transactions.
+// Hash returns the Merkle root hash of the transaction hashes.
+// i.e. the leaves of the tree are the hashes of the txs.
 func (txs Txs) Hash() []byte {
 	// These allocations will be removed once Txs is switched to [][]byte,
 	// ref #2603. This is because golang does not allow type casting slices without unsafe
 	txBzs := make([][]byte, len(txs))
 	for i := 0; i < len(txs); i++ {
-		txBzs[i] = txs[i]
+		txBzs[i] = txs[i].Hash()
 	}
 	return merkle.SimpleHashFromByteSlices(txBzs)
 }
@@ -69,7 +70,7 @@ func (txs Txs) Proof(i int) TxProof {
 	l := len(txs)
 	bzs := make([][]byte, l)
 	for i := 0; i < l; i++ {
-		bzs[i] = txs[i]
+		bzs[i] = txs[i].Hash()
 	}
 	root, proofs := merkle.SimpleProofsFromByteSlices(bzs)
 
@@ -87,8 +88,8 @@ type TxProof struct {
 	Proof    merkle.SimpleProof
 }
 
-// LeadHash returns the hash of the this proof refers to.
-func (tp TxProof) LeafHash() []byte {
+// Leaf returns the hash(tx), which is the leaf in the merkle tree which this proof refers to.
+func (tp TxProof) Leaf() []byte {
 	return tp.Data.Hash()
 }
 
@@ -104,7 +105,7 @@ func (tp TxProof) Validate(dataHash []byte) error {
 	if tp.Proof.Total <= 0 {
 		return errors.New("Proof total must be positive")
 	}
-	valid := tp.Proof.Verify(tp.RootHash, tp.LeafHash())
+	valid := tp.Proof.Verify(tp.RootHash, tp.Leaf())
 	if valid != nil {
 		return errors.New("Proof is not internally consistent")
 	}

types/tx_test.go

@@ -66,14 +66,13 @@ func TestValidTxProof(t *testing.T) {
 		root := txs.Hash()
 		// make sure valid proof for every tx
 		for i := range txs {
-			leaf := txs[i]
-			leafHash := leaf.Hash()
+			tx := []byte(txs[i])
 			proof := txs.Proof(i)
 			assert.Equal(t, i, proof.Proof.Index, "%d: %d", h, i)
 			assert.Equal(t, len(txs), proof.Proof.Total, "%d: %d", h, i)
 			assert.EqualValues(t, root, proof.RootHash, "%d: %d", h, i)
-			assert.EqualValues(t, leaf, proof.Data, "%d: %d", h, i)
-			assert.EqualValues(t, leafHash, proof.LeafHash(), "%d: %d", h, i)
+			assert.EqualValues(t, tx, proof.Data, "%d: %d", h, i)
+			assert.EqualValues(t, txs[i].Hash(), proof.Leaf(), "%d: %d", h, i)
 			assert.Nil(t, proof.Validate(root), "%d: %d", h, i)
 			assert.NotNil(t, proof.Validate([]byte("foobar")), "%d: %d", h, i)
 

types/vote.go

@@ -52,8 +52,8 @@ type Vote struct {
 	Type             SignedMsgType `json:"type"`
 	Height           int64         `json:"height"`
 	Round            int           `json:"round"`
-	Timestamp        time.Time     `json:"timestamp"`
 	BlockID          BlockID       `json:"block_id"` // zero if vote is nil.
+	Timestamp        time.Time     `json:"timestamp"`
 	ValidatorAddress Address       `json:"validator_address"`
 	ValidatorIndex   int           `json:"validator_index"`
 	Signature        []byte        `json:"signature"`
@@ -127,6 +127,11 @@ func (vote *Vote) ValidateBasic() error {
 	if err := vote.BlockID.ValidateBasic(); err != nil {
 		return fmt.Errorf("Wrong BlockID: %v", err)
 	}
+	// BlockID.ValidateBasic would not err if we for instance have an empty hash but a
+	// non-empty PartsSetHeader:
+	if !vote.BlockID.IsZero() && !vote.BlockID.IsComplete() {
+		return fmt.Errorf("BlockID must be either empty or complete, got: %v", vote.BlockID)
+	}
 	if len(vote.ValidatorAddress) != crypto.AddressSize {
 		return fmt.Errorf("Expected ValidatorAddress size to be %d bytes, got %d bytes",
 			crypto.AddressSize,

types/vote_test.go

@@ -63,8 +63,8 @@ func TestVoteSignableTestVectors(t *testing.T) {
 		{
 			CanonicalizeVote("", &Vote{}),
 			// NOTE: Height and Round are skipped here. This case needs to be considered while parsing.
-			// []byte{0x22, 0x9, 0x9, 0x0, 0x9, 0x6e, 0x88, 0xf1, 0xff, 0xff, 0xff},
-			[]byte{0x22, 0xb, 0x8, 0x80, 0x92, 0xb8, 0xc3, 0x98, 0xfe, 0xff, 0xff, 0xff, 0x1},
+			// []byte{0x2a, 0x9, 0x9, 0x0, 0x9, 0x6e, 0x88, 0xf1, 0xff, 0xff, 0xff},
+			[]byte{0x2a, 0xb, 0x8, 0x80, 0x92, 0xb8, 0xc3, 0x98, 0xfe, 0xff, 0xff, 0xff, 0x1},
 		},
 		// with proper (fixed size) height and round (PreCommit):
 		{
@@ -76,7 +76,7 @@ func TestVoteSignableTestVectors(t *testing.T) {
 				0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, // height
 				0x19,                                   // (field_number << 3) | wire_type
 				0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, // round
-				0x22, // (field_number << 3) | wire_type
+				0x2a, // (field_number << 3) | wire_type
 				// remaining fields (timestamp):
 				0xb, 0x8, 0x80, 0x92, 0xb8, 0xc3, 0x98, 0xfe, 0xff, 0xff, 0xff, 0x1},
 		},
@@ -90,7 +90,7 @@ func TestVoteSignableTestVectors(t *testing.T) {
 				0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, // height
 				0x19,                                   // (field_number << 3) | wire_type
 				0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, // round
-				0x22, // (field_number << 3) | wire_type
+				0x2a, // (field_number << 3) | wire_type
 				// remaining fields (timestamp):
 				0xb, 0x8, 0x80, 0x92, 0xb8, 0xc3, 0x98, 0xfe, 0xff, 0xff, 0xff, 0x1},
 		},
@@ -102,7 +102,7 @@ func TestVoteSignableTestVectors(t *testing.T) {
 				0x19,                                   // (field_number << 3) | wire_type
 				0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, // round
 				// remaining fields (timestamp):
-				0x22,
+				0x2a,
 				0xb, 0x8, 0x80, 0x92, 0xb8, 0xc3, 0x98, 0xfe, 0xff, 0xff, 0xff, 0x1},
 		},
 		// containing non-empty chain_id:
@@ -114,7 +114,7 @@ func TestVoteSignableTestVectors(t *testing.T) {
 				0x19,                                   // (field_number << 3) | wire_type
 				0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, // round
 				// remaining fields:
-				0x22,                                                                // (field_number << 3) | wire_type
+				0x2a,                                                                // (field_number << 3) | wire_type
 				0xb, 0x8, 0x80, 0x92, 0xb8, 0xc3, 0x98, 0xfe, 0xff, 0xff, 0xff, 0x1, // timestamp
 				0x32,                                                                               // (field_number << 3) | wire_type
 				0xd, 0x74, 0x65, 0x73, 0x74, 0x5f, 0x63, 0x68, 0x61, 0x69, 0x6e, 0x5f, 0x69, 0x64}, // chainID

version/version.go

@@ -18,7 +18,7 @@ const (
 	// TMCoreSemVer is the current version of Tendermint Core.
 	// It's the Semantic Version of the software.
 	// Must be a string because scripts like dist.sh read this file.
-	TMCoreSemVer = "0.27.4"
+	TMCoreSemVer = "0.28.0"
 
 	// ABCISemVer is the semantic version of the ABCI library
 	ABCISemVer  = "0.15.0"