mirror of
https://github.com/fluencelabs/tendermint
synced 2025-06-23 17:51:39 +00:00
sync to develop
This commit is contained in:
Anca Zamfir
@ -1,14 +1,18 @@
|
||||
# ADR 025 Commit
|
||||
|
||||
## Context
|
||||
|
||||
Currently the `Commit` structure contains a lot of potentially redundant or unnecessary data.
|
||||
In particular it contains an array of every precommit from the validators, which includes many copies of the same data. Such as `Height`, `Round`, `Type`, and `BlockID`. Also the `ValidatorIndex` could be derived from the vote's position in the array, and the `ValidatorAddress` could potentially be derived from runtime context. The only truely necessary data is the `Signature` and `Timestamp` associated with each `Vote`.
|
||||
It contains a list of precommits from every validator, where the precommit
|
||||
includes the whole `Vote` structure. Thus each of the commit height, round,
|
||||
type, and blockID are repeated for every validator, and could be deduplicated.
|
||||
|
||||
```
|
||||
type Commit struct {
|
||||
BlockID BlockID `json:"block_id"`
|
||||
Precommits []*Vote `json:"precommits"`
|
||||
}
|
||||
|
||||
type Vote struct {
|
||||
ValidatorAddress Address `json:"validator_address"`
|
||||
ValidatorIndex int `json:"validator_index"`
|
||||
@ -26,7 +30,9 @@ References:
|
||||
[#2226](https://github.com/tendermint/tendermint/issues/2226)
|
||||
|
||||
## Proposed Solution
|
||||
|
||||
We can improve efficiency by replacing the usage of the `Vote` struct with a subset of each vote, and by storing the constant values (`Height`, `Round`, `BlockID`) in the Commit itself.
|
||||
|
||||
```
|
||||
type Commit struct {
|
||||
Height int64
|
||||
@ -34,42 +40,56 @@ type Commit struct {
|
||||
BlockID BlockID `json:"block_id"`
|
||||
Precommits []*CommitSig `json:"precommits"`
|
||||
}
|
||||
|
||||
type CommitSig struct {
|
||||
BlockID BlockIDFlag
|
||||
ValidatorAddress Address
|
||||
Signature []byte
|
||||
Timestamp time.Time
|
||||
Signature []byte
|
||||
}
|
||||
|
||||
|
||||
// indicate which BlockID the signature is for
|
||||
type BlockIDFlag int
|
||||
|
||||
const (
|
||||
BlockIDFlagAbsent BlockIDFlag = iota // vote is not included in the Commit.Precommits
|
||||
BlockIDFlagCommit // voted for the Commit.BlockID
|
||||
BlockIDFlagNil // voted for nil
|
||||
)
|
||||
|
||||
```
|
||||
Continuing to store the `ValidatorAddress` in the `CommitSig` takes up extra space, but simplifies the process and allows for easier debugging.
|
||||
|
||||
Note the need for an extra byte to indicate whether the signature is for the BlockID or for nil.
|
||||
This byte can also be used to indicate an absent vote, rather than using a nil object like we currently do,
|
||||
which has been [problematic for compatibility between Amino and proto3](https://github.com/tendermint/go-amino/issues/260).
|
||||
|
||||
Note we also continue to store the `ValidatorAddress` in the `CommitSig`.
|
||||
While this still takes 20-bytes per signature, it ensures that the Commit has all
|
||||
information necessary to reconstruct Vote, which simplifies mapping between Commit and Vote objects
|
||||
and with debugging. It also may be necessary for the light-client to know which address a signature corresponds to if
|
||||
it is trying to verify a current commit with an older validtor set.
|
||||
|
||||
## Status
|
||||
|
||||
Proposed
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
The size of a `Commit` transmitted over the network goes from:
|
||||
|
||||
|BlockID| + n * (|Address| + |ValidatorIndex| + |Height| + |Round| + |Timestamp| + |Type| + |BlockID| + |Signature|)
|
||||
Removing the Type/Height/Round/Index and the BlockID saves roughly 80 bytes per precommit.
|
||||
It varies because some integers are varint. The BlockID contains two 32-byte hashes an integer,
|
||||
and the Height is 8-bytes.
|
||||
|
||||
to:
|
||||
For a chain with 100 validators, that's up to 8kB in savings per block!
|
||||
|
||||
|
||||
|BlockID|+|Height|+|Round| + n*(|Address| + |Signature| + |Timestamp|)
|
||||
|
||||
This saves:
|
||||
|
||||
n * (|BlockID| + |ValidatorIndex| + |Type|) + (n-1) * (Height + Round)
|
||||
|
||||
In the current context, this would concretely be:
|
||||
(assuming all ints are int64, and hashes are 32 bytes)
|
||||
|
||||
n *(72 + 8 + 1 + 8 + 8) - 16 = n * 97 - 16
|
||||
|
||||
With 100 validators this is a savings of almost 10KB on every block.
|
||||
|
||||
### Negative
|
||||
This would add some complexity to the processing and verification of blocks and commits, as votes would have to be reconstructed to be verified and gossiped. The reconstruction could be relatively straightforward, only requiring the copying of data from the `Commit` itself into the newly created `Vote`.
|
||||
|
||||
- Large breaking change to the block and commit structure
|
||||
- Requires differentiating in code between the Vote and CommitSig objects, which may add some complexity (votes need to be reconstructed to be verified and gossiped)
|
||||
|
||||
### Neutral
|
||||
This design leaves the `ValidatorAddress` in the `CommitSig` and in the `Vote`. These could be removed at some point for additional savings, but that would introduce more complexity, and make printing of `Commit` and `VoteSet` objects less informative, which could harm debugging efficiency and UI/UX.
|
||||
|
||||
- Commit.Precommits no longer contains nil values
|
||||
|
||||
100
docs/architecture/adr-037-deliver-block.md
Normal file
100
docs/architecture/adr-037-deliver-block.md
Normal file
@ -0,0 +1,100 @@
|
||||
# ADR 037: Deliver Block
|
||||
|
||||
Author: Daniil Lashin (@danil-lashin)
|
||||
|
||||
## Changelog
|
||||
|
||||
13-03-2019: Initial draft
|
||||
|
||||
## Context
|
||||
|
||||
Initial conversation: https://github.com/tendermint/tendermint/issues/2901
|
||||
|
||||
Some applications can handle transactions in parallel, or at least some
|
||||
part of tx processing can be parallelized. Now it is not possible for developer
|
||||
to execute txs in parallel because Tendermint delivers them consequentially.
|
||||
|
||||
## Decision
|
||||
|
||||
Now Tendermint have `BeginBlock`, `EndBlock`, `Commit`, `DeliverTx` steps
|
||||
while executing block. This doc proposes merging this steps into one `DeliverBlock`
|
||||
step. It will allow developers of applications to decide how they want to
|
||||
execute transactions (in parallel or consequentially). Also it will simplify and
|
||||
speed up communications between application and Tendermint.
|
||||
|
||||
As @jaekwon [mentioned](https://github.com/tendermint/tendermint/issues/2901#issuecomment-477746128)
|
||||
in discussion not all application will benefit from this solution. In some cases,
|
||||
when application handles transaction consequentially, it way slow down the blockchain,
|
||||
because it need to wait until full block is transmitted to application to start
|
||||
processing it. Also, in the case of complete change of ABCI, we need to force all the apps
|
||||
to change their implementation completely. That's why I propose to introduce one more ABCI
|
||||
type.
|
||||
|
||||
# Implementation Changes
|
||||
|
||||
In addition to default application interface which now have this structure
|
||||
|
||||
```go
|
||||
type Application interface {
|
||||
// Info and Mempool methods...
|
||||
|
||||
// Consensus Connection
|
||||
InitChain(RequestInitChain) ResponseInitChain // Initialize blockchain with validators and other info from TendermintCore
|
||||
BeginBlock(RequestBeginBlock) ResponseBeginBlock // Signals the beginning of a block
|
||||
DeliverTx(tx []byte) ResponseDeliverTx // Deliver a tx for full processing
|
||||
EndBlock(RequestEndBlock) ResponseEndBlock // Signals the end of a block, returns changes to the validator set
|
||||
Commit() ResponseCommit // Commit the state and return the application Merkle root hash
|
||||
}
|
||||
```
|
||||
|
||||
this doc proposes to add one more:
|
||||
|
||||
```go
|
||||
type Application interface {
|
||||
// Info and Mempool methods...
|
||||
|
||||
// Consensus Connection
|
||||
InitChain(RequestInitChain) ResponseInitChain // Initialize blockchain with validators and other info from TendermintCore
|
||||
DeliverBlock(RequestDeliverBlock) ResponseDeliverBlock // Deliver full block
|
||||
Commit() ResponseCommit // Commit the state and return the application Merkle root hash
|
||||
}
|
||||
|
||||
type RequestDeliverBlock struct {
|
||||
Hash []byte
|
||||
Header Header
|
||||
Txs Txs
|
||||
LastCommitInfo LastCommitInfo
|
||||
ByzantineValidators []Evidence
|
||||
}
|
||||
|
||||
type ResponseDeliverBlock struct {
|
||||
ValidatorUpdates []ValidatorUpdate
|
||||
ConsensusParamUpdates *ConsensusParams
|
||||
Tags []common.KVPair
|
||||
TxResults []ResponseDeliverTx
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
Also, we will need to add new config param, which will specify what kind of ABCI application uses.
|
||||
For example, it can be `abci_type`. Then we will have 2 types:
|
||||
- `advanced` - current ABCI
|
||||
- `simple` - proposed implementation
|
||||
|
||||
## Status
|
||||
|
||||
In review
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
|
||||
- much simpler introduction and tutorials for new developers (instead of implementing 5 methods whey
|
||||
will need to implement only 3)
|
||||
- txs can be handled in parallel
|
||||
- simpler interface
|
||||
- faster communications between Tendermint and application
|
||||
|
||||
### Negative
|
||||
|
||||
- Tendermint should now support 2 kinds of ABCI
|
||||
@ -1,7 +1,8 @@
|
||||
# ADR 037: Peer Behaviour Interface
|
||||
# ADR 039: Peer Behaviour Interface
|
||||
|
||||
## Changelog
|
||||
* 07-03-2019: Initial draft
|
||||
* 14-03-2019: Updates from feedback
|
||||
|
||||
## Context
|
||||
|
||||
@ -19,36 +20,46 @@ and ties up the reactors in a larger dependency graph when testing.
|
||||
|
||||
Introduce a `PeerBehaviour` interface and concrete implementations which
|
||||
provide methods for reactors to signal peer behaviour without direct
|
||||
coupling `p2p.Switch`. Introduce a ErrPeer to provide
|
||||
concrete reasons for stopping peers.
|
||||
coupling `p2p.Switch`. Introduce a ErrorBehaviourPeer to provide
|
||||
concrete reasons for stopping peers. Introduce GoodBehaviourPeer to provide
|
||||
concrete ways in which a peer contributes.
|
||||
|
||||
### Implementation Changes
|
||||
|
||||
PeerBehaviour then becomes an interface for signaling peer errors as well
|
||||
as for marking peers as `good`.
|
||||
|
||||
XXX: It might be better to pass p2p.ID instead of the whole peer but as
|
||||
a first draft maintain the underlying implementation as much as
|
||||
possible.
|
||||
|
||||
```go
|
||||
type PeerBehaviour interface {
|
||||
Errored(peer Peer, reason ErrPeer)
|
||||
MarkPeerAsGood(peer Peer)
|
||||
Behaved(peer Peer, reason GoodBehaviourPeer)
|
||||
Errored(peer Peer, reason ErrorBehaviourPeer)
|
||||
}
|
||||
```
|
||||
|
||||
Instead of signaling peers to stop with arbitrary reasons:
|
||||
`reason interface{}`
|
||||
|
||||
We introduce a concrete error type ErrPeer:
|
||||
We introduce a concrete error type ErrorBehaviourPeer:
|
||||
```go
|
||||
type ErrPeer int
|
||||
type ErrorBehaviourPeer int
|
||||
|
||||
const (
|
||||
ErrPeerUnknown = iota
|
||||
ErrPeerBadMessage
|
||||
ErrPeerMessageOutofOrder
|
||||
ErrorBehaviourUnknown = iota
|
||||
ErrorBehaviourBadMessage
|
||||
ErrorBehaviourMessageOutofOrder
|
||||
...
|
||||
)
|
||||
```
|
||||
|
||||
To provide additional information on the ways a peer contributed, we introduce
|
||||
the GoodBehaviourPeer type.
|
||||
|
||||
```go
|
||||
type GoodBehaviourPeer int
|
||||
|
||||
const (
|
||||
GoodBehaviourVote = iota
|
||||
GoodBehaviourBlockPart
|
||||
...
|
||||
)
|
||||
```
|
||||
@ -60,11 +71,11 @@ type SwitchedPeerBehaviour struct {
|
||||
sw *Switch
|
||||
}
|
||||
|
||||
func (spb *SwitchedPeerBehaviour) Errored(peer Peer, reason ErrPeer) {
|
||||
func (spb *SwitchedPeerBehaviour) Errored(peer Peer, reason ErrorBehaviourPeer) {
|
||||
spb.sw.StopPeerForError(peer, reason)
|
||||
}
|
||||
|
||||
func (spb *SwitchedPeerBehaviour) MarkPeerAsGood(peer Peer) {
|
||||
func (spb *SwitchedPeerBehaviour) Behaved(peer Peer, reason GoodBehaviourPeer) {
|
||||
spb.sw.MarkPeerAsGood(peer)
|
||||
}
|
||||
|
||||
@ -75,51 +86,54 @@ func NewSwitchedPeerBehaviour(sw *Switch) *SwitchedPeerBehaviour {
|
||||
}
|
||||
```
|
||||
|
||||
Reactors, which are often difficult to unit test[<sup>2</sup>](#references). could use an implementation which exposes the signals produced by the reactor in
|
||||
Reactors, which are often difficult to unit test[<sup>2</sup>](#references) could use an implementation which exposes the signals produced by the reactor in
|
||||
manufactured scenarios:
|
||||
|
||||
```go
|
||||
type PeerErrors map[Peer][]ErrPeer
|
||||
type GoodPeers map[Peer]bool
|
||||
type ErrorBehaviours map[Peer][]ErrorBehaviourPeer
|
||||
type GoodBehaviours map[Peer][]GoodBehaviourPeer
|
||||
|
||||
type StorePeerBehaviour struct {
|
||||
pe PeerErrors
|
||||
gp GoodPeers
|
||||
eb ErrorBehaviours
|
||||
gb GoodBehaviours
|
||||
}
|
||||
|
||||
func NewStorePeerBehaviour() *StorePeerBehaviour{
|
||||
return &StorePeerBehaviour{
|
||||
pe: make(PeerErrors),
|
||||
gp: GoodPeers{},
|
||||
eb: make(ErrorBehaviours),
|
||||
gb: make(GoodBehaviours),
|
||||
}
|
||||
}
|
||||
|
||||
func (spb StorePeerBehaviour) Errored(peer Peer, reason ErrPeer) {
|
||||
if _, ok := spb.pe[peer]; !ok {
|
||||
spb.pe[peer] = []ErrPeer{reason}
|
||||
func (spb StorePeerBehaviour) Errored(peer Peer, reason ErrorBehaviourPeer) {
|
||||
if _, ok := spb.eb[peer]; !ok {
|
||||
spb.eb[peer] = []ErrorBehaviours{reason}
|
||||
} else {
|
||||
spb.pe[peer] = append(spb.pe[peer], reason)
|
||||
spb.eb[peer] = append(spb.eb[peer], reason)
|
||||
}
|
||||
}
|
||||
|
||||
func (mpb *StorePeerBehaviour) GetPeerErrors() PeerErrors {
|
||||
return mpb.pe
|
||||
func (mpb *StorePeerBehaviour) GetErrored() ErrorBehaviours {
|
||||
return mpb.eb
|
||||
}
|
||||
|
||||
func (spb *StorePeerBehaviour) MarkPeerAsGood(peer Peer) {
|
||||
if _, ok := spb.gp[peer]; !ok {
|
||||
spb.gp[peer] = true
|
||||
|
||||
func (spb StorePeerBehaviour) Behaved(peer Peer, reason GoodBehaviourPeer) {
|
||||
if _, ok := spb.gb[peer]; !ok {
|
||||
spb.gb[peer] = []GoodBehaviourPeer{reason}
|
||||
} else {
|
||||
spb.gb[peer] = append(spb.gb[peer], reason)
|
||||
}
|
||||
}
|
||||
|
||||
func (spb *StorePeerBehaviour) GetGoodPeers() GoodPeers {
|
||||
return spb.gp
|
||||
func (spb *StorePeerBehaviour) GetBehaved() GoodBehaviours {
|
||||
return spb.gb
|
||||
}
|
||||
```
|
||||
|
||||
## Status
|
||||
|
||||
Proposed
|
||||
Accepted
|
||||
|
||||
## Consequences
|
||||
|
||||
534
docs/architecture/adr-040-blockchain-reactor-refactor.md
Normal file
534
docs/architecture/adr-040-blockchain-reactor-refactor.md
Normal file
@ -0,0 +1,534 @@
|
||||
# ADR 040: Blockchain Reactor Refactor
|
||||
|
||||
## Changelog
|
||||
|
||||
19-03-2019: Initial draft
|
||||
|
||||
## Context
|
||||
|
||||
The Blockchain Reactor's high level responsibility is to enable peers who are far behind the current state of the
|
||||
blockchain to quickly catch up by downloading many blocks in parallel from its peers, verifying block correctness, and
|
||||
executing them against the ABCI application. We call the protocol executed by the Blockchain Reactor `fast-sync`.
|
||||
The current architecture diagram of the blockchain reactor can be found here:
|
||||
|
||||
|
||||
|
||||
The current architecture consists of dozens of routines and it is tightly depending on the `Switch`, making writing
|
||||
unit tests almost impossible. Current tests require setting up complex dependency graphs and dealing with concurrency.
|
||||
Note that having dozens of routines is in this case overkill as most of the time routines sits idle waiting for
|
||||
something to happen (message to arrive or timeout to expire). Due to dependency on the `Switch`, testing relatively
|
||||
complex network scenarios and failures (for example adding and removing peers) is very complex tasks and frequently lead
|
||||
to complex tests with not deterministic behavior ([#3400]). Impossibility to write proper tests makes confidence in
|
||||
the code low and this resulted in several issues (some are fixed in the meantime and some are still open):
|
||||
[#3400], [#2897], [#2896], [#2699], [#2888], [#2457], [#2622], [#2026].
|
||||
|
||||
## Decision
|
||||
|
||||
To remedy these issues we plan a major refactor of the blockchain reactor. The proposed architecture is largely inspired
|
||||
by ADR-30 and is presented on the following diagram:
|
||||
|
||||
|
||||
We suggest a concurrency architecture where the core algorithm (we call it `Controller`) is extracted into a finite
|
||||
state machine. The active routine of the reactor is called `Executor` and is responsible for receiving and sending
|
||||
messages from/to peers and triggering timeouts. What messages should be sent and timeouts triggered is determined mostly
|
||||
by the `Controller`. The exception is `Peer Heartbeat` mechanism which is `Executor` responsibility. The heartbeat
|
||||
mechanism is used to remove slow and unresponsive peers from the peer list. Writing of unit tests is simpler with
|
||||
this architecture as most of the critical logic is part of the `Controller` function. We expect that simpler concurrency
|
||||
architecture will not have significant negative effect on the performance of this reactor (to be confirmed by
|
||||
experimental evaluation).
|
||||
|
||||
|
||||
### Implementation changes
|
||||
|
||||
We assume the following system model for "fast sync" protocol:
|
||||
|
||||
* a node is connected to a random subset of all nodes that represents its peer set. Some nodes are correct and some
|
||||
might be faulty. We don't make assumptions about ratio of faulty nodes, i.e., it is possible that all nodes in some
|
||||
peer set are faulty.
|
||||
* we assume that communication between correct nodes is synchronous, i.e., if a correct node `p` sends a message `m` to
|
||||
a correct node `q` at time `t`, then `q` will receive message the latest at time `t+Delta` where `Delta` is a system
|
||||
parameter that is known by network participants. `Delta` is normally chosen to be an order of magnitude higher than
|
||||
the real communication delay (maximum) between correct nodes. Therefore if a correct node `p` sends a request message
|
||||
to a correct node `q` at time `t` and there is no the corresponding reply at time `t + 2*Delta`, then `p` can assume
|
||||
that `q` is faulty. Note that the network assumptions for the consensus reactor are different (we assume partially
|
||||
synchronous model there).
|
||||
|
||||
The requirements for the "fast sync" protocol are formally specified as follows:
|
||||
|
||||
- `Correctness`: If a correct node `p` is connected to a correct node `q` for a long enough period of time, then `p`
|
||||
- will eventually download all requested blocks from `q`.
|
||||
- `Termination`: If a set of peers of a correct node `p` is stable (no new nodes are added to the peer set of `p`) for
|
||||
- a long enough period of time, then protocol eventually terminates.
|
||||
- `Fairness`: A correct node `p` sends requests for blocks to all peers from its peer set.
|
||||
|
||||
As explained above, the `Executor` is responsible for sending and receiving messages that are part of the `fast-sync`
|
||||
protocol. The following messages are exchanged as part of `fast-sync` protocol:
|
||||
|
||||
``` go
|
||||
type Message int
|
||||
const (
|
||||
MessageUnknown Message = iota
|
||||
MessageStatusRequest
|
||||
MessageStatusResponse
|
||||
MessageBlockRequest
|
||||
MessageBlockResponse
|
||||
)
|
||||
```
|
||||
`MessageStatusRequest` is sent periodically to all peers as a request for a peer to provide its current height. It is
|
||||
part of the `Peer Heartbeat` mechanism and a failure to respond timely to this message results in a peer being removed
|
||||
from the peer set. Note that the `Peer Heartbeat` mechanism is used only while a peer is in `fast-sync` mode. We assume
|
||||
here existence of a mechanism that gives node a possibility to inform its peers that it is in the `fast-sync` mode.
|
||||
|
||||
``` go
|
||||
type MessageStatusRequest struct {
|
||||
SeqNum int64 // sequence number of the request
|
||||
}
|
||||
```
|
||||
`MessageStatusResponse` is sent as a response to `MessageStatusRequest` to inform requester about the peer current
|
||||
height.
|
||||
|
||||
``` go
|
||||
type MessageStatusResponse struct {
|
||||
SeqNum int64 // sequence number of the corresponding request
|
||||
Height int64 // current peer height
|
||||
}
|
||||
```
|
||||
|
||||
`MessageBlockRequest` is used to make a request for a block and the corresponding commit certificate at a given height.
|
||||
|
||||
``` go
|
||||
type MessageBlockRequest struct {
|
||||
Height int64
|
||||
}
|
||||
```
|
||||
|
||||
`MessageBlockResponse` is a response for the corresponding block request. In addition to providing the block and the
|
||||
corresponding commit certificate, it contains also a current peer height.
|
||||
|
||||
``` go
|
||||
type MessageBlockResponse struct {
|
||||
Height int64
|
||||
Block Block
|
||||
Commit Commit
|
||||
PeerHeight int64
|
||||
}
|
||||
```
|
||||
|
||||
In addition to sending and receiving messages, and `HeartBeat` mechanism, controller is also managing timeouts
|
||||
that are triggered upon `Controller` request. `Controller` is then informed once a timeout expires.
|
||||
|
||||
``` go
|
||||
type TimeoutTrigger int
|
||||
const (
|
||||
TimeoutUnknown TimeoutTrigger = iota
|
||||
TimeoutResponseTrigger
|
||||
TimeoutTerminationTrigger
|
||||
)
|
||||
```
|
||||
|
||||
The `Controller` can be modelled as a function with clearly defined inputs:
|
||||
|
||||
* `State` - current state of the node. Contains data about connected peers and its behavior, pending requests,
|
||||
* received blocks, etc.
|
||||
* `Event` - significant events in the network.
|
||||
|
||||
producing clear outputs:
|
||||
|
||||
* `State` - updated state of the node,
|
||||
* `MessageToSend` - signal what message to send and to which peer
|
||||
* `TimeoutTrigger` - signal that timeout should be triggered.
|
||||
|
||||
|
||||
We consider the following `Event` types:
|
||||
|
||||
``` go
|
||||
type Event int
|
||||
const (
|
||||
EventUnknown Event = iota
|
||||
EventStatusReport
|
||||
EventBlockRequest
|
||||
EventBlockResponse
|
||||
EventRemovePeer
|
||||
EventTimeoutResponse
|
||||
EventTimeoutTermination
|
||||
)
|
||||
```
|
||||
|
||||
`EventStatusResponse` event is generated once `MessageStatusResponse` is received by the `Executor`.
|
||||
|
||||
``` go
|
||||
type EventStatusReport struct {
|
||||
PeerID ID
|
||||
Height int64
|
||||
}
|
||||
```
|
||||
|
||||
`EventBlockRequest` event is generated once `MessageBlockRequest` is received by the `Executor`.
|
||||
|
||||
``` go
|
||||
type EventBlockRequest struct {
|
||||
Height int64
|
||||
PeerID p2p.ID
|
||||
}
|
||||
```
|
||||
`EventBlockResponse` event is generated upon reception of `MessageBlockResponse` message by the `Executor`.
|
||||
|
||||
``` go
|
||||
type EventBlockResponse struct {
|
||||
Height int64
|
||||
Block Block
|
||||
Commit Commit
|
||||
PeerID ID
|
||||
PeerHeight int64
|
||||
}
|
||||
```
|
||||
`EventRemovePeer` is generated by `Executor` to signal that the connection to a peer is closed due to peer misbehavior.
|
||||
|
||||
``` go
|
||||
type EventRemovePeer struct {
|
||||
PeerID ID
|
||||
}
|
||||
```
|
||||
`EventTimeoutResponse` is generated by `Executor` to signal that a timeout triggered by `TimeoutResponseTrigger` has
|
||||
expired.
|
||||
|
||||
``` go
|
||||
type EventTimeoutResponse struct {
|
||||
PeerID ID
|
||||
Height int64
|
||||
}
|
||||
```
|
||||
`EventTimeoutTermination` is generated by `Executor` to signal that a timeout triggered by `TimeoutTerminationTrigger`
|
||||
has expired.
|
||||
|
||||
``` go
|
||||
type EventTimeoutTermination struct {
|
||||
Height int64
|
||||
}
|
||||
```
|
||||
|
||||
`MessageToSend` is just a wrapper around `Message` type that contains id of the peer to which message should be sent.
|
||||
|
||||
``` go
|
||||
type MessageToSend struct {
|
||||
PeerID ID
|
||||
Message Message
|
||||
}
|
||||
```
|
||||
|
||||
The Controller state machine can be in two modes: `ModeFastSync` when
|
||||
a node is trying to catch up with the network by downloading committed blocks,
|
||||
and `ModeConsensus` in which it executes Tendermint consensus protocol. We
|
||||
consider that `fast sync` mode terminates once the Controller switch to
|
||||
`ModeConsensus`.
|
||||
|
||||
``` go
|
||||
type Mode int
|
||||
const (
|
||||
ModeUnknown Mode = iota
|
||||
ModeFastSync
|
||||
ModeConsensus
|
||||
)
|
||||
```
|
||||
`Controller` is managing the following state:
|
||||
|
||||
``` go
|
||||
type ControllerState struct {
|
||||
Height int64 // the first block that is not committed
|
||||
Mode Mode // mode of operation
|
||||
PeerMap map[ID]PeerStats // map of peer IDs to peer statistics
|
||||
MaxRequestPending int64 // maximum height of the pending requests
|
||||
FailedRequests []int64 // list of failed block requests
|
||||
PendingRequestsNum int // total number of pending requests
|
||||
Store []BlockInfo // contains list of downloaded blocks
|
||||
Executor BlockExecutor // store, verify and executes blocks
|
||||
}
|
||||
```
|
||||
|
||||
`PeerStats` data structure keeps for every peer its current height and a list of pending requests for blocks.
|
||||
|
||||
``` go
|
||||
type PeerStats struct {
|
||||
Height int64
|
||||
PendingRequest int64 // a request sent to this peer
|
||||
}
|
||||
```
|
||||
|
||||
`BlockInfo` data structure is used to store information (as part of block store) about downloaded blocks: from what peer
|
||||
a block and the corresponding commit certificate are received.
|
||||
``` go
|
||||
type BlockInfo struct {
|
||||
Block Block
|
||||
Commit Commit
|
||||
PeerID ID // a peer from which we received the corresponding Block and Commit
|
||||
}
|
||||
```
|
||||
|
||||
The `Controller` is initialized by providing an initial height (`startHeight`) from which it will start downloading
|
||||
blocks from peers and the current state of the `BlockExecutor`.
|
||||
|
||||
``` go
|
||||
func NewControllerState(startHeight int64, executor BlockExecutor) ControllerState {
|
||||
state = ControllerState {}
|
||||
state.Height = startHeight
|
||||
state.Mode = ModeFastSync
|
||||
state.MaxRequestPending = startHeight - 1
|
||||
state.PendingRequestsNum = 0
|
||||
state.Executor = executor
|
||||
initialize state.PeerMap, state.FailedRequests and state.Store to empty data structures
|
||||
return state
|
||||
}
|
||||
```
|
||||
|
||||
The core protocol logic is given with the following function:
|
||||
|
||||
``` go
|
||||
func handleEvent(state ControllerState, event Event) (ControllerState, Message, TimeoutTrigger, Error) {
|
||||
msg = nil
|
||||
timeout = nil
|
||||
error = nil
|
||||
|
||||
switch state.Mode {
|
||||
case ModeConsensus:
|
||||
switch event := event.(type) {
|
||||
case EventBlockRequest:
|
||||
msg = createBlockResponseMessage(state, event)
|
||||
return state, msg, timeout, error
|
||||
default:
|
||||
error = "Only respond to BlockRequests while in ModeConsensus!"
|
||||
return state, msg, timeout, error
|
||||
}
|
||||
|
||||
case ModeFastSync:
|
||||
switch event := event.(type) {
|
||||
case EventBlockRequest:
|
||||
msg = createBlockResponseMessage(state, event)
|
||||
return state, msg, timeout, error
|
||||
|
||||
case EventStatusResponse:
|
||||
return handleEventStatusResponse(event, state)
|
||||
|
||||
case EventRemovePeer:
|
||||
return handleEventRemovePeer(event, state)
|
||||
|
||||
case EventBlockResponse:
|
||||
return handleEventBlockResponse(event, state)
|
||||
|
||||
case EventResponseTimeout:
|
||||
return handleEventResponseTimeout(event, state)
|
||||
|
||||
case EventTerminationTimeout:
|
||||
// Termination timeout is triggered in case of empty peer set and in case there are no pending requests.
|
||||
// If this timeout expires and in the meantime no new peers are added or new pending requests are made
|
||||
// then `fast-sync` mode terminates by switching to `ModeConsensus`.
|
||||
// Note that termination timeout should be higher than the response timeout.
|
||||
if state.Height == event.Height && state.PendingRequestsNum == 0 { state.State = ConsensusMode }
|
||||
return state, msg, timeout, error
|
||||
|
||||
default:
|
||||
error = "Received unknown event type!"
|
||||
return state, msg, timeout, error
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
``` go
|
||||
func createBlockResponseMessage(state ControllerState, event BlockRequest) MessageToSend {
|
||||
msgToSend = nil
|
||||
if _, ok := state.PeerMap[event.PeerID]; !ok { peerStats = PeerStats{-1, -1} }
|
||||
if state.Executor.ContainsBlockWithHeight(event.Height) && event.Height > peerStats.Height {
|
||||
peerStats = event.Height
|
||||
msg = BlockResponseMessage{
|
||||
Height: event.Height,
|
||||
Block: state.Executor.getBlock(eventHeight),
|
||||
Commit: state.Executor.getCommit(eventHeight),
|
||||
PeerID: event.PeerID,
|
||||
CurrentHeight: state.Height - 1,
|
||||
}
|
||||
msgToSend = MessageToSend { event.PeerID, msg }
|
||||
}
|
||||
state.PeerMap[event.PeerID] = peerStats
|
||||
return msgToSend
|
||||
}
|
||||
```
|
||||
|
||||
``` go
|
||||
func handleEventStatusResponse(event EventStatusResponse, state ControllerState) (ControllerState, MessageToSend, TimeoutTrigger, Error) {
|
||||
if _, ok := state.PeerMap[event.PeerID]; !ok {
|
||||
peerStats = PeerStats{ -1, -1 }
|
||||
} else {
|
||||
peerStats = state.PeerMap[event.PeerID]
|
||||
}
|
||||
|
||||
if event.Height > peerStats.Height { peerStats.Height = event.Height }
|
||||
// if there are no pending requests for this peer, try to send him a request for block
|
||||
if peerStats.PendingRequest == -1 {
|
||||
msg = createBlockRequestMessages(state, event.PeerID, peerStats.Height)
|
||||
// msg is nil if no request for block can be made to a peer at this point in time
|
||||
if msg != nil {
|
||||
peerStats.PendingRequests = msg.Height
|
||||
state.PendingRequestsNum++
|
||||
// when a request for a block is sent to a peer, a response timeout is triggered. If no corresponding block is sent by the peer
|
||||
// during response timeout period, then the peer is considered faulty and is removed from the peer set.
|
||||
timeout = ResponseTimeoutTrigger{ msg.PeerID, msg.Height, PeerTimeout }
|
||||
} else if state.PendingRequestsNum == 0 {
|
||||
// if there are no pending requests and no new request can be placed to the peer, termination timeout is triggered.
|
||||
// If termination timeout expires and we are still at the same height and there are no pending requests, the "fast-sync"
|
||||
// mode is finished and we switch to `ModeConsensus`.
|
||||
timeout = TerminationTimeoutTrigger{ state.Height, TerminationTimeout }
|
||||
}
|
||||
}
|
||||
state.PeerMap[event.PeerID] = peerStats
|
||||
return state, msg, timeout, error
|
||||
}
|
||||
```
|
||||
|
||||
``` go
|
||||
func handleEventRemovePeer(event EventRemovePeer, state ControllerState) (ControllerState, MessageToSend, TimeoutTrigger, Error) {
|
||||
if _, ok := state.PeerMap[event.PeerID]; ok {
|
||||
pendingRequest = state.PeerMap[event.PeerID].PendingRequest
|
||||
// if a peer is removed from the peer set, its pending request is declared failed and added to the `FailedRequests` list
|
||||
// so it can be retried.
|
||||
if pendingRequest != -1 {
|
||||
add(state.FailedRequests, pendingRequest)
|
||||
}
|
||||
state.PendingRequestsNum--
|
||||
delete(state.PeerMap, event.PeerID)
|
||||
// if the peer set is empty after removal of this peer then termination timeout is triggered.
|
||||
if state.PeerMap.isEmpty() {
|
||||
timeout = TerminationTimeoutTrigger{ state.Height, TerminationTimeout }
|
||||
}
|
||||
} else { error = "Removing unknown peer!" }
|
||||
return state, msg, timeout, error
|
||||
```
|
||||
|
||||
``` go
|
||||
func handleEventBlockResponse(event EventBlockResponse, state ControllerState) (ControllerState, MessageToSend, TimeoutTrigger, Error)
|
||||
if state.PeerMap[event.PeerID] {
|
||||
peerStats = state.PeerMap[event.PeerID]
|
||||
// when expected block arrives from a peer, it is added to the store so it can be verified and if correct executed after.
|
||||
if peerStats.PendingRequest == event.Height {
|
||||
peerStats.PendingRequest = -1
|
||||
state.PendingRequestsNum--
|
||||
if event.PeerHeight > peerStats.Height { peerStats.Height = event.PeerHeight }
|
||||
state.Store[event.Height] = BlockInfo{ event.Block, event.Commit, event.PeerID }
|
||||
// blocks are verified sequentially so adding a block to the store does not mean that it will be immediately verified
|
||||
// as some of the previous blocks might be missing.
|
||||
state = verifyBlocks(state) // it can lead to event.PeerID being removed from peer list
|
||||
if _, ok := state.PeerMap[event.PeerID]; ok {
|
||||
// we try to identify new request for a block that can be asked to the peer
|
||||
msg = createBlockRequestMessage(state, event.PeerID, peerStats.Height)
|
||||
if msg != nil {
|
||||
peerStats.PendingRequests = msg.Height
|
||||
state.PendingRequestsNum++
|
||||
// if request for block is made, response timeout is triggered
|
||||
timeout = ResponseTimeoutTrigger{ msg.PeerID, msg.Height, PeerTimeout }
|
||||
} else if state.PeerMap.isEmpty() || state.PendingRequestsNum == 0 {
|
||||
// if the peer map is empty (the peer can be removed as block verification failed) or there are no pending requests
|
||||
// termination timeout is triggered.
|
||||
timeout = TerminationTimeoutTrigger{ state.Height, TerminationTimeout }
|
||||
}
|
||||
}
|
||||
} else { error = "Received Block from wrong peer!" }
|
||||
} else { error = "Received Block from unknown peer!" }
|
||||
|
||||
state.PeerMap[event.PeerID] = peerStats
|
||||
return state, msg, timeout, error
|
||||
}
|
||||
```
|
||||
|
||||
``` go
|
||||
func handleEventResponseTimeout(event, state) {
|
||||
if _, ok := state.PeerMap[event.PeerID]; ok {
|
||||
peerStats = state.PeerMap[event.PeerID]
|
||||
// if a response timeout expires and the peer hasn't delivered the block, the peer is removed from the peer list and
|
||||
// the request is added to the `FailedRequests` so the block can be downloaded from other peer
|
||||
if peerStats.PendingRequest == event.Height {
|
||||
add(state.FailedRequests, pendingRequest)
|
||||
delete(state.PeerMap, event.PeerID)
|
||||
state.PendingRequestsNum--
|
||||
// if peer set is empty, then termination timeout is triggered
|
||||
if state.PeerMap.isEmpty() {
|
||||
timeout = TimeoutTrigger{ state.Height, TerminationTimeout }
|
||||
}
|
||||
}
|
||||
}
|
||||
return state, msg, timeout, error
|
||||
}
|
||||
```
|
||||
|
||||
``` go
|
||||
func createBlockRequestMessage(state ControllerState, peerID ID, peerHeight int64) MessageToSend {
|
||||
msg = nil
|
||||
blockHeight = -1
|
||||
r = find request in state.FailedRequests such that r <= peerHeight // returns `nil` if there are no such request
|
||||
// if there is a height in failed requests that can be downloaded from the peer send request to it
|
||||
if r != nil {
|
||||
blockNumber = r
|
||||
delete(state.FailedRequests, r)
|
||||
} else if state.MaxRequestPending < peerHeight {
|
||||
// if height of the maximum pending request is smaller than peer height, then ask peer for next block
|
||||
state.MaxRequestPending++
|
||||
blockHeight = state.MaxRequestPending // increment state.MaxRequestPending and then return the new value
|
||||
}
|
||||
|
||||
if blockHeight > -1 { msg = MessageToSend { peerID, MessageBlockRequest { blockHeight } }
|
||||
return msg
|
||||
}
|
||||
```
|
||||
|
||||
``` go
|
||||
func verifyBlocks(state State) State {
|
||||
done = false
|
||||
for !done {
|
||||
block = state.Store[height]
|
||||
if block != nil {
|
||||
verified = verify block.Block using block.Commit // return `true` is verification succeed, 'false` otherwise
|
||||
|
||||
if verified {
|
||||
block.Execute() // executing block is costly operation so it might make sense executing asynchronously
|
||||
state.Height++
|
||||
} else {
|
||||
// if block verification failed, then it is added to `FailedRequests` and the peer is removed from the peer set
|
||||
add(state.FailedRequests, height)
|
||||
state.Store[height] = nil
|
||||
if _, ok := state.PeerMap[block.PeerID]; ok {
|
||||
pendingRequest = state.PeerMap[block.PeerID].PendingRequest
|
||||
// if there is a pending request sent to the peer that is just to be removed from the peer set, add it to `FailedRequests`
|
||||
if pendingRequest != -1 {
|
||||
add(state.FailedRequests, pendingRequest)
|
||||
state.PendingRequestsNum--
|
||||
}
|
||||
delete(state.PeerMap, event.PeerID)
|
||||
}
|
||||
done = true
|
||||
}
|
||||
} else { done = true }
|
||||
}
|
||||
return state
|
||||
}
|
||||
```
|
||||
|
||||
In the proposed architecture `Controller` is not active task, i.e., it is being called by the `Executor`. Depending on
|
||||
the return values returned by `Controller`,`Executor` will send a message to some peer (`msg` != nil), trigger a
|
||||
timeout (`timeout` != nil) or deal with errors (`error` != nil).
|
||||
In case a timeout is triggered, it will provide as an input to `Controller` the corresponding timeout event once
|
||||
timeout expires.
|
||||
|
||||
|
||||
## Status
|
||||
|
||||
Draft.
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
|
||||
- isolated implementation of the algorithm
|
||||
- improved testability - simpler to prove correctness
|
||||
- clearer separation of concerns - easier to reason
|
||||
|
||||
### Negative
|
||||
|
||||
### Neutral
|
||||
29
docs/architecture/adr-041-proposer-selection-via-abci.md
Normal file
29
docs/architecture/adr-041-proposer-selection-via-abci.md
Normal file
@ -0,0 +1,29 @@
|
||||
# ADR 041: Application should be in charge of validator set
|
||||
|
||||
## Changelog
|
||||
|
||||
|
||||
## Context
|
||||
|
||||
Currently Tendermint is in charge of validator set and proposer selection. Application can only update the validator set changes at EndBlock time.
|
||||
To support Light Client, application should make sure at least 2/3 of validator are same at each round.
|
||||
|
||||
Application should have full control on validator set changes and proposer selection. In each round Application can provide the list of validators for next rounds in order with their power. The proposer is the first in the list, in case the proposer is offline, the next one can propose the proposal and so on.
|
||||
|
||||
## Decision
|
||||
|
||||
## Status
|
||||
|
||||
## Consequences
|
||||
|
||||
Tendermint is no more in charge of validator set and its changes. The Application should provide the correct information.
|
||||
However Tendermint can provide psedo-randomness algorithm to help application for selecting proposer in each round.
|
||||
|
||||
### Positive
|
||||
|
||||
### Negative
|
||||
|
||||
### Neutral
|
||||
|
||||
## References
|
||||
|
||||
BIN
docs/architecture/img/bc-reactor-refactor.png
Normal file
BIN
docs/architecture/img/bc-reactor-refactor.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 7.9 KiB |
BIN
docs/architecture/img/bc-reactor.png
Normal file
BIN
docs/architecture/img/bc-reactor.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 43 KiB |
Reference in New Issue
Block a user