changelog and version (#3709)

Otherwise, we'll be trying to include them in each consecutive block.

The downside is that evil proposers will be able to drop valid
transactions (see #3322).

Reverts https://github.com/tendermint/tendermint/pull/3625

Fixes #3699

CHANGELOG.md

@@ -1,15 +1,30 @@
 # Changelog
 
+## v0.31.7
+
+*June 3, 2019*
+
+This releases fixes a regression in the mempool introduced in v0.31.6.
+The regression caused the invalid committed txs to be proposed in blocks over and
+over again.
+
+### BUG FIXES:
+- [mempool] \#3699 Remove all committed txs from the mempool.
+    This reverts the change from v0.31.6 where we only remove valid txs from the mempool.
+    Note this means malicious proposals can cause txs to be dropped from the
+    mempools of other nodes by including them in blocks before they are valid.
+    See \#3322.
+
 ## v0.31.6
 
 *May 31st, 2019*
 
-This release fixes a magnitude of issues in the consensus and p2p packages as
-well as one security issue in the mempool package. It also contains many
-improvements. See below for details.
+This release contains many fixes and improvements, primarily for p2p functionality.
+It also fixes a security issue in the mempool package.
 
-Tendermint now supports [boltdb](https://github.com/etcd-io/bbolt), although
-in experimental mode. Feel free to try and report any issues.
+With this release, Tendermint now supports [boltdb](https://github.com/etcd-io/bbolt), although
+in experimental mode. Feel free to try and report to us any findings/issues.
+Note also that the build tags for compiling CLevelDB have changed.
 
 Special thanks to external contributors on this release:
 @guagualvcha, @james-ray, @gregdhill, @climber73, @yutianwu,
@@ -18,22 +33,12 @@ Special thanks to external contributors on this release:
 ### BREAKING CHANGES:
 
 * Go API
-- [libs/common] Removed deprecated `PanicSanity`, `PanicCrisis`,
-  `PanicConsensus` and `PanicQ`
-- [mempool] [\#2659](https://github.com/tendermint/tendermint/issues/2659) `Mempool` now an interface
-  * old `Mempool` implementation renamed to `CListMempool`
-  * `NewMempool` renamed to `NewCListMempool`
-  * `Option` renamed to `CListOption`
-  * unexpose `MempoolReactor.Mempool`
-  * `MempoolReactor` renamed to `Reactor`
-  * `NewMempoolReactor` renamed to `NewReactor`
-  * unexpose `TxID` method
-  * `TxInfo.PeerID` renamed to `SenderID`
-- [node] Moved `GenesisDocProvider` and `DefaultGenesisDocProviderFunc` to `state` package
-- [p2p] [\#3346](https://github.com/tendermint/tendermint/issues/3346) `Reactor#InitPeer` method is added to `Reactor` interface
-- [state] [\#2659](https://github.com/tendermint/tendermint/issues/2659) `Mempool` interface moved to mempool package
-  * `MockMempool` moved to top-level mock package and renamed to `Mempool`
-- [types] [\#1648](https://github.com/tendermint/tendermint/issues/1648) `Commit#VoteSignBytes` signature was changed
+  - [libs/common] Removed deprecated `PanicSanity`, `PanicCrisis`,
+    `PanicConsensus` and `PanicQ`
+  - [mempool, state] [\#2659](https://github.com/tendermint/tendermint/issues/2659) `Mempool` now an interface that lives in the mempool package.
+    See issue and PR for more details.
+  - [p2p] [\#3346](https://github.com/tendermint/tendermint/issues/3346) `Reactor#InitPeer` method is added to `Reactor` interface
+  - [types] [\#1648](https://github.com/tendermint/tendermint/issues/1648) `Commit#VoteSignBytes` signature was changed
 
 ### FEATURES:
 - [node] [\#2659](https://github.com/tendermint/tendermint/issues/2659) Add `node.Mempool()` method, which allows you to access mempool
@@ -47,7 +52,7 @@ Special thanks to external contributors on this release:
 - [cli] [\#3661](https://github.com/tendermint/tendermint/pull/3661) Add
   `--hostname-suffix`, `--hostname` and `--random-monikers` options to `testnet`
   cmd for greater peer address/identity generation flexibility.
-- [crypto] [\#3672](https://github.com/tendermint/tendermint/issues/3672) Return more info in the AddSignatureFromPubKey error
+- [crypto] [\#3672](https://github.com/tendermint/tendermint/issues/3672) Return more info in the `AddSignatureFromPubKey` error
 - [cs/replay] [\#3460](https://github.com/tendermint/tendermint/issues/3460) Check appHash for each block
 - [libs/db] [\#3611](https://github.com/tendermint/tendermint/issues/3611) Conditional compilation
   * Use `cleveldb` tag instead of `gcc` to compile Tendermint with CLevelDB or
@@ -56,7 +61,7 @@ Special thanks to external contributors on this release:
   * Use `boltdb` tag to compile Tendermint with bolt db
 - [node] [\#3362](https://github.com/tendermint/tendermint/issues/3362) Return an error if `persistent_peers` list is invalid (except
   when IP lookup fails)
-- [p2p] [\#3463](https://github.com/tendermint/tendermint/pull/3463) Do not log "Can't add peer's address to addrbook" error for a private peer
+- [p2p] [\#3463](https://github.com/tendermint/tendermint/pull/3463) Do not log "Can't add peer's address to addrbook" error for a private peer (@guagualvcha)
 - [p2p] [\#3531](https://github.com/tendermint/tendermint/issues/3531) Terminate session on nonce wrapping (@climber73)
 - [pex] [\#3647](https://github.com/tendermint/tendermint/pull/3647) Dial seeds, if any, instead of crawling peers first (@defunctzombie)
 - [rpc] [\#3534](https://github.com/tendermint/tendermint/pull/3534) Add support for batched requests/responses in JSON RPC
@@ -64,23 +69,18 @@ Special thanks to external contributors on this release:
   incorrect (except when IP lookup fails)
 
 ### BUG FIXES:
-- [consensus] [\#3067](https://github.com/tendermint/tendermint/issues/3067) getBeginBlockValidatorInfo loads validators from stateDB
-  instead of state (@james-ray)
-- [consensus] [\#3346](https://github.com/tendermint/tendermint/issues/3346) Create a peer state in consensus reactor before the peer
+- [consensus] [\#3067](https://github.com/tendermint/tendermint/issues/3067) Fix replay from appHeight==0 with validator set changes (@james-ray)
+- [consensus] [\#3304](https://github.com/tendermint/tendermint/issues/3304) Create a peer state in consensus reactor before the peer
   is started (@guagualvcha)
 - [lite] [\#3669](https://github.com/tendermint/tendermint/issues/3669) Add context parameter to RPC Handlers in proxy routes (@yutianwu)
-- [mempool] [\#3322](https://github.com/tendermint/tendermint/issues/3322) Remove only valid (Code==0) txs on Update
-  * `Mempool#Update` and `BlockExecutor#Commit` now accept
-  `[]*abci.ResponseDeliverTx` - list of `DeliverTx` responses, which should
-  match `block.Txs`
-- [p2p] [\#3338](https://github.com/tendermint/tendermint/issues/3338) Ensure RemovePeer is always called before InitPeer (upon a peer
+- [mempool] [\#3322](https://github.com/tendermint/tendermint/issues/3322) When a block is committed, only remove committed txs from the mempool
+that were valid (ie. `ResponseDeliverTx.Code == 0`)
+- [p2p] [\#3338](https://github.com/tendermint/tendermint/issues/3338) Ensure `RemovePeer` is always called before `InitPeer` (upon a peer
   reconnecting to our node)
-- [p2p] [\#3532](https://github.com/tendermint/tendermint/issues/3532) limit the number of attempts to connect to a peer in seed mode
+- [p2p] [\#3532](https://github.com/tendermint/tendermint/issues/3532) Limit the number of attempts to connect to a peer in seed mode
   to 16 (as a result, the node will stop retrying after a 35 hours time window)
-- [p2p] [\#3362](https://github.com/tendermint/tendermint/issues/3362) make persistent prop independent of conn direction
-  * `Switch#DialPeersAsync` now only takes a list of peers
-  * `Switch#DialPeerWithAddress` now only takes an address
-- [pex] [\#3603](https://github.com/tendermint/tendermint/issues/3603) Dial seeds when addrbook needs more addresses (@defunctzombie)
+- [p2p] [\#3362](https://github.com/tendermint/tendermint/issues/3362) Allow inbound peers to be persistent, including for seed nodes.
+- [pex] [\#3603](https://github.com/tendermint/tendermint/pull/3603) Dial seeds when addrbook needs more addresses (@defunctzombie)
 
 ### OTHERS:
 - [networks] fixes ansible integration script (@carlosflrs)

CHANGELOG_PENDING.md

@@ -1,4 +1,4 @@
-## v0.31.7
+## v0.31.8
 
 **
 

mempool/clist_mempool.go

@@ -538,24 +538,23 @@ func (mem *CListMempool) Update(
 		if deliverTxResponses[i].Code == abci.CodeTypeOK {
 			// Add valid committed tx to the cache (if missing).
 			_ = mem.cache.Push(tx)
-
-			// Remove valid committed tx from the mempool.
-			if e, ok := mem.txsMap.Load(txKey(tx)); ok {
-				mem.removeTx(tx, e.(*clist.CElement), false)
-			}
 		} else {
 			// Allow invalid transactions to be resubmitted.
 			mem.cache.Remove(tx)
+		}
 
-			// Don't remove invalid tx from the mempool.
-			// Otherwise evil proposer can drop valid txs.
-			// Example:
-			//   100 -> 101 -> 102
-			// Block, proposed by evil proposer:
-			//   101 -> 102
-			// Mempool (if you remove txs):
-			//   100
-			// https://github.com/tendermint/tendermint/issues/3322.
+		// Remove committed tx from the mempool.
+		//
+		// Note an evil proposer can drop valid txs!
+		// Mempool before:
+		//   100 -> 101 -> 102
+		// Block, proposed by an evil proposer:
+		//   101 -> 102
+		// Mempool after:
+		//   100
+		// https://github.com/tendermint/tendermint/issues/3322.
+		if e, ok := mem.txsMap.Load(txKey(tx)); ok {
+			mem.removeTx(tx, e.(*clist.CElement), false)
 		}
 	}
 

mempool/clist_mempool_test.go

@@ -200,12 +200,15 @@ func TestMempoolUpdate(t *testing.T) {
 		assert.Zero(t, mempool.Size())
 	}
 
-	// 3. Removes invalid transactions from the cache, but leaves them in the mempool (if present)
+	// 3. Removes invalid transactions from the cache and the mempool (if present)
 	{
 		err := mempool.CheckTx([]byte{0x03}, nil)
 		require.NoError(t, err)
 		mempool.Update(1, []types.Tx{[]byte{0x03}}, abciResponses(1, 1), nil, nil)
-		assert.Equal(t, 1, mempool.Size())
+		assert.Zero(t, mempool.Size())
+
+		err = mempool.CheckTx([]byte{0x03}, nil)
+		assert.NoError(t, err)
 	}
 }
 

node/node.go

@@ -65,6 +65,19 @@ func DefaultDBProvider(ctx *DBContext) (dbm.DB, error) {
 	return dbm.NewDB(ctx.ID, dbType, ctx.Config.DBDir()), nil
 }
 
+// GenesisDocProvider returns a GenesisDoc.
+// It allows the GenesisDoc to be pulled from sources other than the
+// filesystem, for instance from a distributed key-value store cluster.
+type GenesisDocProvider func() (*types.GenesisDoc, error)
+
+// DefaultGenesisDocProviderFunc returns a GenesisDocProvider that loads
+// the GenesisDoc from the config.GenesisFile() on the filesystem.
+func DefaultGenesisDocProviderFunc(config *cfg.Config) GenesisDocProvider {
+	return func() (*types.GenesisDoc, error) {
+		return types.GenesisDocFromFile(config.GenesisFile())
+	}
+}
+
 // NodeProvider takes a config and a logger and returns a ready to go Node.
 type NodeProvider func(*cfg.Config, log.Logger) (*Node, error)
 
@@ -99,7 +112,7 @@ func DefaultNewNode(config *cfg.Config, logger log.Logger) (*Node, error) {
 		privval.LoadOrGenFilePV(newPrivValKey, newPrivValState),
 		nodeKey,
 		proxy.DefaultClientCreator(config.ProxyApp, config.ABCI, config.DBDir()),
-		sm.DefaultGenesisDocProviderFunc(config),
+		DefaultGenesisDocProviderFunc(config),
 		DefaultDBProvider,
 		DefaultMetricsProvider(config.Instrumentation),
 		logger,
@@ -466,7 +479,7 @@ func NewNode(config *cfg.Config,
 	privValidator types.PrivValidator,
 	nodeKey *p2p.NodeKey,
 	clientCreator proxy.ClientCreator,
-	genesisDocProvider sm.GenesisDocProvider,
+	genesisDocProvider GenesisDocProvider,
 	dbProvider DBProvider,
 	metricsProvider MetricsProvider,
 	logger log.Logger) (*Node, error) {
@@ -476,7 +489,7 @@ func NewNode(config *cfg.Config,
 		return nil, err
 	}
 
-	state, genDoc, err := sm.LoadStateFromDBOrGenesisDocProvider(stateDB, genesisDocProvider)
+	state, genDoc, err := LoadStateFromDBOrGenesisDocProvider(stateDB, genesisDocProvider)
 	if err != nil {
 		return nil, err
 	}
@@ -1002,6 +1015,56 @@ func makeNodeInfo(
 
 //------------------------------------------------------------------------------
 
+var (
+	genesisDocKey = []byte("genesisDoc")
+)
+
+// LoadStateFromDBOrGenesisDocProvider attempts to load the state from the
+// database, or creates one using the given genesisDocProvider and persists the
+// result to the database. On success this also returns the genesis doc loaded
+// through the given provider.
+func LoadStateFromDBOrGenesisDocProvider(stateDB dbm.DB, genesisDocProvider GenesisDocProvider) (sm.State, *types.GenesisDoc, error) {
+	// Get genesis doc
+	genDoc, err := loadGenesisDoc(stateDB)
+	if err != nil {
+		genDoc, err = genesisDocProvider()
+		if err != nil {
+			return sm.State{}, nil, err
+		}
+		// save genesis doc to prevent a certain class of user errors (e.g. when it
+		// was changed, accidentally or not). Also good for audit trail.
+		saveGenesisDoc(stateDB, genDoc)
+	}
+	state, err := sm.LoadStateFromDBOrGenesisDoc(stateDB, genDoc)
+	if err != nil {
+		return sm.State{}, nil, err
+	}
+	return state, genDoc, nil
+}
+
+// panics if failed to unmarshal bytes
+func loadGenesisDoc(db dbm.DB) (*types.GenesisDoc, error) {
+	b := db.Get(genesisDocKey)
+	if len(b) == 0 {
+		return nil, errors.New("Genesis doc not found")
+	}
+	var genDoc *types.GenesisDoc
+	err := cdc.UnmarshalJSON(b, &genDoc)
+	if err != nil {
+		panic(fmt.Sprintf("Failed to load genesis doc due to unmarshaling error: %v (bytes: %X)", err, b))
+	}
+	return genDoc, nil
+}
+
+// panics if failed to marshal the given genesis document
+func saveGenesisDoc(db dbm.DB, genDoc *types.GenesisDoc) {
+	b, err := cdc.MarshalJSON(genDoc)
+	if err != nil {
+		panic(fmt.Sprintf("Failed to save genesis doc due to marshaling error: %v", err))
+	}
+	db.SetSync(genesisDocKey, b)
+}
+
 func createAndStartPrivValidatorSocketClient(
 	listenAddr string,
 	logger log.Logger,

rpc/test/helpers.go

@@ -8,10 +8,8 @@ import (
 	"strings"
 	"time"
 
-	"github.com/tendermint/tendermint/libs/log"
-	"github.com/tendermint/tendermint/state"
-
 	abci "github.com/tendermint/tendermint/abci/types"
+	"github.com/tendermint/tendermint/libs/log"
 
 	cfg "github.com/tendermint/tendermint/config"
 	cmn "github.com/tendermint/tendermint/libs/common"
@@ -167,7 +165,7 @@ func NewTendermint(app abci.Application, opts *Options) *nm.Node {
 		panic(err)
 	}
 	node, err := nm.NewNode(config, pv, nodeKey, papp,
-		state.DefaultGenesisDocProviderFunc(config),
+		nm.DefaultGenesisDocProviderFunc(config),
 		nm.DefaultDBProvider,
 		nm.DefaultMetricsProvider(config.Instrumentation),
 		logger)

scripts/release_management/bump-semver.py

@@ -8,6 +8,7 @@
 
 import re
 import argparse
+import sys
 
 
 def semver(ver):
@@ -17,6 +18,18 @@ def semver(ver):
   return ver
 
 
+def get_tendermint_version():
+  """Extracts the current Tendermint version from version/version.go"""
+  pattern = re.compile(r"TMCoreSemVer = \"(?P<version>([0-9.]+)+)\"")
+  with open("version/version.go", "rt") as version_file:
+    for line in version_file:
+      m = pattern.search(line)
+      if m:
+        return m.group('version')
+
+  return None
+
+
 if __name__ == "__main__":
   parser = argparse.ArgumentParser()
   parser.add_argument("--version", help="Version number to bump, e.g.: v1.0.0", required=True, type=semver)
@@ -34,4 +47,16 @@ if __name__ == "__main__":
   else:
     patch = int(patch) + 1
 
-  print("{0}.{1}".format(majorminorprefix, patch))
+  expected_version = "{0}.{1}".format(majorminorprefix, patch)
+  # if we're doing a release
+  if expected_version != "v0.0.0":
+    cur_version = get_tendermint_version()
+    if not cur_version:
+      print("Failed to obtain Tendermint version from version/version.go")
+      sys.exit(1)
+    expected_version_noprefix = expected_version.lstrip("v")
+    if expected_version_noprefix != "0.0.0" and expected_version_noprefix != cur_version:
+      print("Expected version/version.go#TMCoreSemVer to be {0}, but was {1}".format(expected_version_noprefix, cur_version))
+      sys.exit(1)
+
+  print(expected_version)

state/store.go

@@ -1,11 +1,9 @@
 package state
 
 import (
-	"errors"
 	"fmt"
 
 	abci "github.com/tendermint/tendermint/abci/types"
-	cfg "github.com/tendermint/tendermint/config"
 	cmn "github.com/tendermint/tendermint/libs/common"
 	dbm "github.com/tendermint/tendermint/libs/db"
 	"github.com/tendermint/tendermint/types"
@@ -19,23 +17,6 @@ const (
 	valSetCheckpointInterval = 100000
 )
 
-var (
-	genesisDocKey = []byte("genesisDoc")
-)
-
-// GenesisDocProvider returns a GenesisDoc.
-// It allows the GenesisDoc to be pulled from sources other than the
-// filesystem, for instance from a distributed key-value store cluster.
-type GenesisDocProvider func() (*types.GenesisDoc, error)
-
-// DefaultGenesisDocProviderFunc returns a GenesisDocProvider that loads
-// the GenesisDoc from the config.GenesisFile() on the filesystem.
-func DefaultGenesisDocProviderFunc(config *cfg.Config) GenesisDocProvider {
-	return func() (*types.GenesisDoc, error) {
-		return types.GenesisDocFromFile(config.GenesisFile())
-	}
-}
-
 //------------------------------------------------------------------------
 
 func calcValidatorsKey(height int64) []byte {
@@ -84,52 +65,6 @@ func LoadStateFromDBOrGenesisDoc(stateDB dbm.DB, genesisDoc *types.GenesisDoc) (
 	return state, nil
 }
 
-// LoadStateFromDBOrGenesisDocProvider attempts to load the state from the
-// database, or creates one using the given genesisDocProvider and persists the
-// result to the database. On success this also returns the genesis doc loaded
-// through the given provider.
-func LoadStateFromDBOrGenesisDocProvider(stateDB dbm.DB, genesisDocProvider GenesisDocProvider) (State, *types.GenesisDoc, error) {
-	// Get genesis doc
-	genDoc, err := loadGenesisDoc(stateDB)
-	if err != nil {
-		genDoc, err = genesisDocProvider()
-		if err != nil {
-			return State{}, nil, err
-		}
-		// save genesis doc to prevent a certain class of user errors (e.g. when it
-		// was changed, accidentally or not). Also good for audit trail.
-		saveGenesisDoc(stateDB, genDoc)
-	}
-	state, err := LoadStateFromDBOrGenesisDoc(stateDB, genDoc)
-	if err != nil {
-		return State{}, nil, err
-	}
-	return state, genDoc, nil
-}
-
-// panics if failed to unmarshal bytes
-func loadGenesisDoc(db dbm.DB) (*types.GenesisDoc, error) {
-	bytes := db.Get(genesisDocKey)
-	if len(bytes) == 0 {
-		return nil, errors.New("Genesis doc not found")
-	}
-	var genDoc *types.GenesisDoc
-	err := cdc.UnmarshalJSON(bytes, &genDoc)
-	if err != nil {
-		panic(fmt.Sprintf("Failed to load genesis doc due to unmarshaling error: %v (bytes: %X)", err, bytes))
-	}
-	return genDoc, nil
-}
-
-// panics if failed to marshal the given genesis document
-func saveGenesisDoc(db dbm.DB, genDoc *types.GenesisDoc) {
-	bytes, err := cdc.MarshalJSON(genDoc)
-	if err != nil {
-		panic(fmt.Sprintf("Failed to save genesis doc due to marshaling error: %v", err))
-	}
-	db.SetSync(genesisDocKey, bytes)
-}
-
 // LoadState loads the State from the database.
 func LoadState(db dbm.DB) State {
 	return loadState(db, stateKey)

types/block.go

@@ -584,6 +584,7 @@ func (commit *Commit) VoteSignBytes(chainID string, valIdx int) []byte {
 
 // memoizeHeightRound memoizes the height and round of the commit using
 // the first non-nil vote.
+// Should be called before any attempt to access `commit.height` or `commit.round`.
 func (commit *Commit) memoizeHeightRound() {
 	if len(commit.Precommits) == 0 {
 		return

version/version.go

@@ -20,7 +20,7 @@ const (
 	// Must be a string because scripts like dist.sh read this file.
 	// XXX: Don't change the name of this variable or you will break
 	// automation :)
-	TMCoreSemVer = "0.31.5"
+	TMCoreSemVer = "0.31.7"
 
 	// ABCISemVer is the semantic version of the ABCI library
 	ABCISemVer  = "0.16.0"