feat: async peerstore backed by datastores (#1058)

We have a peerstore that keeps all data for all observed peers in memory with no eviction.

This is fine when you don't discover many peers but when using the DHT you encounter a significant number of peers so our peer storage grows and grows over time.

We have a persistent peer store, but it just periodically writes peers into the datastore to be read at startup, still keeping them in memory.

It also means a restart doesn't give you any temporary reprieve from the memory leak as the previously observed peer data is read into memory at startup.

This change refactors the peerstore to use a datastore by default, reading and writing peer info as it arrives.  It can be configured with a MemoryDatastore if desired.

It was necessary to change the peerstore and *book interfaces to be asynchronous since the datastore api is asynchronous.

BREAKING CHANGE: `libp2p.handle`, `libp2p.registrar.register` and the peerstore methods have become async
This commit is contained in:
Alex Potsides
2022-01-20 12:03:35 +00:00
committed by GitHub
parent 0a4dc54d08
commit 978eb3676f
94 changed files with 3263 additions and 4039 deletions

View File

@ -2,35 +2,28 @@
const path = require('path')
const execa = require('execa')
const pWaitFor = require('p-wait-for')
const { toString: uint8ArrayToString } = require('uint8arrays/to-string')
async function test() {
process.stdout.write('1.js\n')
const addrs = []
let foundIt = false
const proc = execa('node', [path.join(__dirname, '1.js')], {
cwd: path.resolve(__dirname),
all: true
})
let output = ''
proc.all.on('data', async (data) => {
process.stdout.write(data)
const line = uint8ArrayToString(data)
output += uint8ArrayToString(data)
// Discovered peer
if (!foundIt && line.includes('Found it, multiaddrs are:')) {
foundIt = true
// Discovered peers
if (output.includes('Found it, multiaddrs are:')) {
proc.kill()
}
addrs.push(line)
})
await pWaitFor(() => addrs.length === 2)
proc.kill()
}
module.exports = test