8 Commits

Author SHA1 Message Date
Raúl Kripalani
72f9d4c2c0 add godoc on global vars. 2019-01-29 20:48:38 +00:00
Raúl Kripalani
83660414ce cleanup channels on context cancellation. 2019-01-29 17:11:40 +00:00
Raúl Kripalani
abacfe5fc9 refactor interface of Consume(). 2019-01-29 16:47:42 +00:00
Raúl Kripalani
7cd14f5d1e replace concurrency level error with panic. 2019-01-29 16:34:39 +00:00
Raúl Kripalani
24f0a2dec8 remove unneeded lk; group global vars. 2019-01-29 16:15:40 +00:00
Raúl Kripalani
95a0975dd7 enhance logging and import prefixes. 2019-01-29 00:56:12 +00:00
Raúl Kripalani
f044043c3d adjust godoc. 2019-01-29 00:56:12 +00:00
Raúl Kripalani
5e74c4a6f2 introduce adaptive queue for DHT dials.
This patch introduces an adaptive dial queue that spawns a dynamically sized
set of goroutines to preemptively stage dials for later handoff to the DHT
protocol for RPC. It identifies backpressure on both ends (dial consumers and
dial producers), and takes compensating action by adjusting the worker pool.

We start with `DialQueueMinParallelism` number of workers (6), and scale up
and down based on demand and supply of dialled peers.

The following events trigger scaling:
- we scale up when we can't immediately return a successful dial to a new
  consumer.
- we scale down when we've been idle for a while waiting for new dial
  attempts.
- we scale down when we complete a dial and realise nobody was waiting for it.

Dialler throttling (e.g. FD limit exceeded) is a concern, as we can easily
spin up more workers to compensate, and end up adding fuel to the fire. Since
we have no deterministic way to detect this for now, we hard-limit concurrency
to `DialQueueMaxParallelism` (20).
2019-01-29 00:56:12 +00:00