This uses the stdlib's method of creating a tempfile in our write
file atomimc method, with a few modifications. We use a 64 bit number
rather than 32 bit, and therefore a corresponding LCG. This is to
reduce collision probability. (Note we currently used 32 bytes previously,
so this is likely a concern)
We handle reseeding the LCG in such a way that multiple threads are
even less likely to reuse the same seed.
Previously we logged `Testing for i <i>` for all i in [0,100).
This was unnecessary. This changes it to just log the value for i on
error messages, to reduce the unnecessary verbosity in log files.
This now makes bit array functions which take in a second bit array, thread
safe. Previously there was a warning on bitarray.Update to be lock the
second parameter externally if thread safety wasrequired.
This was not done within the codebase, so it was fine to change here.
Closes#2080
We were computing these functions incorrectly.
I'm not sure what distribution these numbers are, but it isn't the
normal exponential distribution. (We're making the probability of
getting a number of a particular bitlength equal, but the number in
that bitlength thats gets chosen is uniformly chosen)
We weren't using these functions anywhere in our codebase, and they
had a nomenclature error. (There aren't exponentially distributed
integers, instead they would be geometrically distributed)
The godocs fell out of sync with the code here. Additionally we had
warning that these randomness functions weren't for cryptographic
use on every function. However these warnings are confusing, since
there was no implication that they would be secure there, and a
single warning on the actual Rand type would suffice. (This is what
is done in golang's math/rand godoc)
Additionally we indicated that rand.Bytes() was reading OS randomness
but in fact that had been changed.
This commit switches all usage of math/rand to cmn's rand. The only
exceptions are within the random file itself, the tools package, and the
crypto package. In tools you don't want it to lock between the go-routines.
The crypto package doesn't use it so the crypto package have no other
dependencies within tendermint/tendermint for easier portability.
Crypto/rand usage is unadjusted.
Closes#1343
Refs #1755
I started with writing a test for wsConnection (WebsocketManager) where
I:
- create a WS connection
- do a simple echo call
- close it
No leaking goroutines, nor any leaking memory were detected.
For useful shortcuts see my blog post
https://blog.cosmos.network/debugging-the-memory-leak-in-tendermint-210186711420
Then I went to the rpc tests to see if calling Subscribe results in
memory growth. It did.
I used a slightly modified version of TestHeaderEvents function:
```
func TestHeaderEvents(t *testing.T) {
// memory heap before
f, err := os.Create("/tmp/mem1.mprof")
if err != nil {
t.Fatal(err)
}
pprof.WriteHeapProfile(f)
f.Close()
for i := 0; i < 100; i++ {
c := getHTTPClient()
err = c.Start()
require.Nil(t, err)
evtTyp := types.EventNewBlockHeader
evt, err := client.WaitForOneEvent(c, evtTyp, waitForEventTimeout)
require.Nil(t, err)
_, ok := evt.(types.EventDataNewBlockHeader)
require.True(t, ok)
c.Stop()
c = nil
}
runtime.GC()
// memory heap before
f, err = os.Create("/tmp/mem2.mprof")
if err != nil {
t.Fatal(err)
}
pprof.WriteHeapProfile(f)
f.Close()
// dump all running goroutines
time.Sleep(10 * time.Second)
pprof.Lookup("goroutine").WriteTo(os.Stdout, 1)
}
```
```
Showing nodes accounting for 35159.16kB, 100% of 35159.16kB total
Showing top 10 nodes out of 48
flat flat% sum% cum cum%
32022.23kB 91.08% 91.08% 32022.23kB 91.08% github.com/tendermint/tendermint/libs/pubsub/query.(*QueryParser).Init
1056.33kB 3.00% 94.08% 1056.33kB 3.00% bufio.NewReaderSize
528.17kB 1.50% 95.58% 528.17kB 1.50% bufio.NewWriterSize
528.17kB 1.50% 97.09% 528.17kB 1.50% github.com/tendermint/tendermint/consensus.NewConsensusState
512.19kB 1.46% 98.54% 512.19kB 1.46% runtime.malg
512.08kB 1.46% 100% 512.08kB 1.46% syscall.ByteSliceFromString
0 0% 100% 512.08kB 1.46% github.com/tendermint/tendermint/consensus.(*ConsensusState).(github.com/tendermint/tendermint/consensus.defaultDecideProposal)-fm
0 0% 100% 512.08kB 1.46% github.com/tendermint/tendermint/consensus.(*ConsensusState).addVote
0 0% 100% 512.08kB 1.46% github.com/tendermint/tendermint/consensus.(*ConsensusState).defaultDecideProposal
0 0% 100% 512.08kB 1.46% github.com/tendermint/tendermint/consensus.(*ConsensusState).enterNewRound
```
100 subscriptions produce 32MB.
Again, no additional goroutines are running after the end of the test
(wsConnection readRoutine and writeRoutine both finishes). **It means
that some exiting goroutine or object is holding a reference to the
*Query objects, which are leaking.**
One of them is pubsub#loop. It's using state.queries to map queries to
clients and state.clients to map clients to queries.
Before this commit, we're not thoroughly cleaning state.queries, which
was the reason for memory leakage.
