139 Commits

Author SHA1 Message Date
Matthew Slipper
92343ef484 Add additional metrics (#2500)
* Add additional metrics
Continues addressing https://github.com/cosmos/cosmos-sdk/issues/2169.
* Add nop metrics to fix NPE
* Tweak buckets, code review
* Update buckets
* Update docs with new metrics
* Code review updates
2018-10-10 18:27:43 +02:00
Ethan Buchman
52e21cebcf
remove some xxx comments and the config.mempool.recheck_empty (#2505)
* remove some XXX

* config: remove Mempool.RecheckEmpty

* docs: remove recheck_empty
2018-09-30 13:28:34 -04:00
Ethan Buchman
f5824bc837
Update abci and app docs (#2470)
* mempool: update some comments

* make build_c

* docs: notes about databases and WAL files

* docs: determinism. closes #1279

* docs: small note about query paths. closes #2090

* docs: gas

* docs: abci consensus params
2018-09-23 01:14:05 -04:00
Dev Ojha
111e627037 mempool: Filter new txs if they have insufficient gas (#2385)
This also refactors the prior mempool to filter to be known as
"precheck filter" and this new filter is called "postcheck filter"

This PR also fixes a bug where the precheck filter previously didn't
account for the amino overhead, which could a maliciously sized tx to
halt blocks from getting any txs in them.

* Move maxGas outside of function definition to avoid race condition
* Type filter funcs and make public
* Use helper method for post check
* Remove superfluous Filter suffix
* Move default pre/post checks into package
* Fix broken references
* Fix typos
* Expand on examples for checks
2018-09-22 02:50:06 +02:00
Anton Kaliaev
8d50bb9dad conesnsu: follow up to removing some consensus params (#2427)
* follow up to removing some consensus params Refs #2382
* change args type to int64 in state#makeParams
* make valsCount and evidenceCount ints again
* MaxEvidenceBytesPerBlock: include magic number in godoc
* [spec] creating a proposal
* test state#TxFilter
* panic if MaxDataBytes is less than 0
* fixes after review
* use amino#UvarintSize to calculate overhead
0c74291f3b/encoder.go (L85-L90)
* avoid cyclic imports
* you can do better Go, come on
* remove testdouble package
2018-09-21 11:00:36 +02:00
bradyjoestar
26aa978456 Make mempool cache a proper LRU (#2407)
Closes #2399
2018-09-19 21:40:22 +04:00
Dev Ojha
1ea64fc27f Make mempool aware of MaxGas requirement (#2360)
* Make mempool aware of MaxGas requirement

* update spec

* Add tests

* Switch GasWanted from kv store to persistent kv store

* Fix typo in test name

* switch back to using kvstore, not persistent kv store
2018-09-12 16:41:19 -04:00
Anton Kaliaev
166fd82b70
max-bytes PR follow-up (#2318)
* ReapMaxTxs: return all txs if max is negative

this mirrors ReapMaxBytes behavior

See https://github.com/tendermint/tendermint/pull/2184#discussion_r214439950

* increase MaxAminoOverheadForBlock

tested with:

```
func TestMaxAminoOverheadForBlock(t *testing.T) {

        maxChainID := ""
        for i := 0; i < MaxChainIDLen; i++ {
                maxChainID += "𠜎"
        }

        h := Header{
                ChainID:            maxChainID,
                Height:             10,
                Time:               time.Now().UTC(),
                NumTxs:             100,
                TotalTxs:           200,
                LastBlockID:        makeBlockID(make([]byte, 20), 300, make([]byte, 20)),
                LastCommitHash:     tmhash.Sum([]byte("last_commit_hash")),
                DataHash:           tmhash.Sum([]byte("data_hash")),
                ValidatorsHash:     tmhash.Sum([]byte("validators_hash")),
                NextValidatorsHash: tmhash.Sum([]byte("next_validators_hash")),
                ConsensusHash:      tmhash.Sum([]byte("consensus_hash")),
                AppHash:            tmhash.Sum([]byte("app_hash")),
                LastResultsHash:    tmhash.Sum([]byte("last_results_hash")),
                EvidenceHash:       tmhash.Sum([]byte("evidence_hash")),
                ProposerAddress:    tmhash.Sum([]byte("proposer_address")),
        }
        b := Block{
                Header:     h,
                Data:       Data{Txs: makeTxs(10000, 100)},
                Evidence:   EvidenceData{},
                LastCommit: &Commit{},
        }

        bz, err := cdc.MarshalBinary(b)
        require.NoError(t, err)

        assert.Equal(t, MaxHeaderBytes+MaxAminoOverheadForBlock-2, len(bz)-1000000-20000-1)
}
```

* fix MaxYYY constants calculation

by using math.MaxInt64

See https://github.com/tendermint/tendermint/pull/2184#discussion_r214444244

* pass mempool filter as an option

See https://github.com/tendermint/tendermint/pull/2184#discussion_r214445869

* fixes after Dev's comments
2018-09-04 11:46:34 +04:00
Anton Kaliaev
e873fed815
calculate amino overhead on the fly 2018-08-31 16:01:22 +04:00
Anton Kaliaev
d73c5cbdb1
reap max bytes from the mempool & check transaction size
See ADR 020: Limiting txs size inside a block docs/architecture/adr-020-block-size.md

Refs #2035
2018-08-31 16:01:21 +04:00
Dev Ojha
bd531401a0 mempool: Store txs by hash inside of cache (#2234)
* mempool: Store txs by hash inside of cache

This allows for large cachesizes, without fear of the memory
consumption growing rapidly.

* (squash this) rename hashedTx -> txHash
2018-08-29 08:10:06 +04:00
Dev Ojha
8a1a79257e mempool: Keep cache hashmap and linked list in sync (#2188)
* mempool: Keep cache hashmap and linked list in sync

This removes bugs with the linked list being full, but hashmap empty

* address PR comments

* switch clist back to list
2018-08-13 14:42:33 +04:00
Dev Ojha
2756be5a59 libs: Remove usage of custom Fmt, in favor of fmt.Sprintf (#2199)
* libs: Remove usage of custom Fmt, in favor of fmt.Sprintf

Closes #2193

* Fix bug that was masked by custom Fmt!
2018-08-10 09:25:57 +04:00
Jun Kimura
e1b9bf7c81 set capacity of txsmap (#2166) 2018-08-07 20:50:10 +04:00
Ethan Buchman
15b112e669 mempool: chan bool -> chan struct{} 2018-07-23 21:06:47 -04:00
srmo
e36ce6f893 fix race condition on proposal height for published txs (#2021)
* #1920 try to fix race condition on proposal height for published txs

- related to create_empty_blocks=false
- published height for accepted tx can be wrong (too low)
- use the actual mempool height + 1 for the proposal
- expose Height() on mempool

* #1920 add initial test for mempool.Height()

- not sure how to test the lock
- can the mutex reference be of type Locker?
-- this way, we can use a "mock" of the mutex to test triggering

* #1920 use the ConsensusState height in favor of mempool

- gets rid of indirections
- doesn't need any "+1" magic

* #1920 cosmetic

- if we use cs.Height, it's enough to evaluate right before propose

* #1920 cleanup TODO and non-needed code

* #1920 add changelog entry
2018-07-23 15:34:45 +04:00
Ethan Buchman
d55243f0e6 fix import paths 2018-07-01 22:36:49 -04:00
Anton Kaliaev
aa20c45ae9 log total when adding new txs to the mempool 2018-06-27 18:14:45 +04:00
Ethan Buchman
fa3bd05d44 mempool: log hashes, not whole tx 2018-06-27 18:14:45 +04:00
Alexander Simmerl
a605b66c5a
Move abci imports 2018-06-22 07:28:07 +02:00
Ethan Buchman
9cd9f3338b Merge branch 'master' into develop 2018-06-21 12:57:09 -07:00
Anton Kaliaev
205d8b8062
fixes after @xla review
- move prometheus metrics into internal packages
- *Option structs
- misc. format changes
2018-06-20 12:40:25 +04:00
Anton Kaliaev
84812145cb
friendly apis for constructors 2018-06-20 12:40:25 +04:00
Anton Kaliaev
7efb73aa18
mempool size metric 2018-06-20 12:38:45 +04:00
Anton Kaliaev
4f5492c831
add nopTxCache (Nil Object Pattern)
to better handle zero cache size
2018-06-19 11:59:07 +04:00
Ethan Buchman
4b2348f697 mempool: fix cache_size==0. closes #1761 2018-06-18 18:21:19 -07:00
Ethan Buchman
a519825bf8 consensus: fixes #1754
* updateToState exits early if the state isn't new, which happens after
* fast syncing. This results in not sending a NewRoundStep message. The mempool
* reactor depends on PeerState, which is updated by NewRoundStep
* messages. If the peer never sends a NewRoundStep, the mempool reactor
* will think they're behind, and never forward transactions. Note this
* only happens when `create_empty_blocks = false`, because otherwise
* peers will move through the consensus state and send a NewRoundStep
* for a new step soon anyways. Simple fix is just to send the
* NewRoundStep message during updateToState even if exit early
2018-06-18 17:08:09 -07:00
Anton Kaliaev
3fa734ef5a
recheck only if there are txs left in the mempool (#1645) 2018-06-04 14:28:47 +04:00
Ethan Buchman
98b0c51b5f fix possible mempool deadlock 2018-06-03 16:11:52 -04:00
Anton Kaliaev
202a43a5af
remove TODO
no longer relevant, I guess. since ABCI only defines 0 (success) code.
2018-05-18 10:48:13 +04:00
Anton Kaliaev
c9001d5a11
bound the mempool
Refs #345
2018-05-18 10:45:51 +04:00
Anton Kaliaev
ae572b9038
remove extra call to Exists
Refs #345

https://github.com/tendermint/tendermint/issues/1539#issuecomment-387240606
2018-05-18 10:17:09 +04:00
Anton Kaliaev
bbe1355957
log only hash, not tx itself 2018-05-17 10:00:59 +04:00
Anton Kaliaev
7c14fa820d
do not log txs at info level
BEFORE:

```
./tm-bench -c 5 -r 1000 127.0.0.1:46657
Stats          Avg       StdDev     Max
Txs/sec        1826      843        2744
Blocks/sec     1.100     0.300      2
```

AFTER:

```
./tm-bench -T 30 -c 5 -r 1000 127.0.0.1:46657
Stats          Avg       StdDev     Max
Txs/sec        6120      1970       9776
Blocks/sec     1.000     0.000      1
```
2018-05-17 10:00:59 +04:00
Anton Kaliaev
11b68f1934
rewrite broadcastTxRoutine to use channels
https://play.golang.org/p/gN21yO9IRs3

```
func waitWithCancel(f func() *clist.CElement, ctx context.Context) *clist.CElement {
	el := make(chan *clist.CElement, 1)
	select {
	case el <- f():
```
will just run f() blockingly, so this doesn't change much in terms of behavior.
2018-02-05 16:36:26 +04:00
Anton Kaliaev
5f3048bd09
call FlushSync before calling CommitSync
if we call it after, we might receive a "fresh" transaction from
  `broadcast_tx_sync` before old transactions (which were not
  committed).

Refs #1091

```
Commit is called with a lock on the mempool, meaning no calls to CheckTx
can start. However, since CheckTx is called async in the mempool
connection, some CheckTx might have already "sailed", when the lock is
released in the mempool and Commit proceeds.

Then, that spurious CheckTx has not yet "begun" in the ABCI app (stuck
in transport?). Instead, ABCI app manages to start to process the
Commit. Next, the spurious, "sailed" CheckTx happens in the wrong place.
```
2018-01-23 16:56:14 +04:00
Ethan Buchman
f06cc6630b mempool: cfg.CacheSize and expose InitWAL 2018-01-19 01:03:03 -05:00
Ethan Buchman
c9be2b89f9 mempool: return error on cached txs 2017-12-02 01:15:11 -05:00
Anton Kaliaev
922af7c405
int64 height
uint64 is considered dangerous. the details will follow in a blog post.
2017-12-01 19:04:53 -06:00
Anton Kaliaev
86af889dfb remove unnecessary casts (Refs #911) 2017-12-01 17:17:22 -05:00
Anton Kaliaev
b3492356e6 uint64 height (Refs #911) 2017-12-01 17:17:22 -05:00
Emmanuel Odeke
3595b5931a
mempool: implement Mempool.CloseWAL
Fixes https://github.com/tendermint/tendermint/issues/890

Add a CloseWAL method to Mempool to close the underlying WAL file
and then discard it so that further writes to it will have no effect.
2017-11-27 21:37:25 -07:00
Zach Ramsay
8f0237610e linting errors: clean it all up 2017-11-27 22:39:11 +00:00
Zach Ramsay
b75d4f73e7 errcheck: PR comment fixes 2017-11-27 22:39:11 +00:00
Zach Ramsay
331857c9e6 linting: apply errcheck part2 2017-11-27 22:39:11 +00:00
Ethan Buchman
ac2ef9e0ea Merge pull request #750 from tendermint/feature/cleanup
Cleanup of code and code docs
2017-10-23 11:14:15 -04:00
Adrian Brink
819f81f702 Change NOTE to CONTRACT 2017-10-23 11:04:45 -04:00
Adrian Brink
782a836db0 Cleanup of code and code docs
This cleans up some of the code in the state package
2017-10-23 11:04:45 -04:00
Anton Kaliaev
f908dd0e55 only notify when there are some txs (Refs #753) 2017-10-23 10:31:00 -04:00
Anton Kaliaev
54f2cc9709
[docs] add how to read logs guide [ci skip] 2017-10-04 18:35:22 +04:00