14 Commits

Author SHA1 Message Date
Matt Joiner
8b37385468 Rename log to logger 2019-02-03 13:14:35 +11:00
Raúl Kripalani
94682b4df2 fix a race in dial queue. 2019-02-01 19:20:32 +00:00
Raúl Kripalani
ebcfcd46a6 make dial queue parameters configurable. 2019-02-01 13:43:27 +11:00
Matt Joiner
7246a3b0f4 Fix races with DialQueue variables 2019-01-31 13:08:45 +11:00
Raúl Kripalani
74d22f3f5e park waiters in slice; revise closure logic. 2019-01-30 23:25:10 +00:00
Raúl Kripalani
f2df3ec540 fix shutdown logic; fix timer logic. 2019-01-29 20:49:04 +00:00
Raúl Kripalani
72f9d4c2c0 add godoc on global vars. 2019-01-29 20:48:38 +00:00
Raúl Kripalani
83660414ce cleanup channels on context cancellation. 2019-01-29 17:11:40 +00:00
Raúl Kripalani
abacfe5fc9 refactor interface of Consume(). 2019-01-29 16:47:42 +00:00
Raúl Kripalani
7cd14f5d1e replace concurrency level error with panic. 2019-01-29 16:34:39 +00:00
Raúl Kripalani
24f0a2dec8 remove unneeded lk; group global vars. 2019-01-29 16:15:40 +00:00
Raúl Kripalani
95a0975dd7 enhance logging and import prefixes. 2019-01-29 00:56:12 +00:00
Raúl Kripalani
f044043c3d adjust godoc. 2019-01-29 00:56:12 +00:00
Raúl Kripalani
5e74c4a6f2 introduce adaptive queue for DHT dials.
This patch introduces an adaptive dial queue that spawns a dynamically sized
set of goroutines to preemptively stage dials for later handoff to the DHT
protocol for RPC. It identifies backpressure on both ends (dial consumers and
dial producers), and takes compensating action by adjusting the worker pool.

We start with `DialQueueMinParallelism` number of workers (6), and scale up
and down based on demand and supply of dialled peers.

The following events trigger scaling:
- we scale up when we can't immediately return a successful dial to a new
  consumer.
- we scale down when we've been idle for a while waiting for new dial
  attempts.
- we scale down when we complete a dial and realise nobody was waiting for it.

Dialler throttling (e.g. FD limit exceeded) is a concern, as we can easily
spin up more workers to compensate, and end up adding fuel to the fire. Since
we have no deterministic way to detect this for now, we hard-limit concurrency
to `DialQueueMaxParallelism` (20).
2019-01-29 00:56:12 +00:00