Commit Graph

7160 Commits

Author SHA1 Message Date
b65ddfb16a Dynamic HZ: adapt cron frequency to number of clients. 2018-07-23 14:21:04 +02:00
e6ea603ad3 Dynamic HZ: separate hz from the configured hz.
This way we can remember what the user configured HZ is, but change the
actual HZ dynamically if needed in the dynamic HZ feature
implementation.
2018-07-23 14:13:58 +02:00
4ff47a0b9b Top comment clientsCron(). 2018-07-20 09:46:18 +02:00
aba6855282 Clarify that clientsCronTrackExpansiveClients() indexes may jump ahead. 2018-07-20 09:36:48 +02:00
be88c0b16a Rename INFO CLIENT max buffers field names for correctness.
They are actually delayed a few seconds, so let's call them "recent".
2018-07-19 17:38:20 +02:00
0cf3794e6e Fix wrong array index variable in getExpansiveClientsInfo(). 2018-07-19 17:34:15 +02:00
ea3a20c5d0 Change INFO CLIENTS sections to report pre-computed max/min client buffers. 2018-07-19 17:16:19 +02:00
8f7e496ba5 Rename var in clientsCronTrackExpansiveClients() for clarity. 2018-07-19 13:59:13 +02:00
8d617596f1 Implement a function to retrieve the expansive clients mem usage. 2018-07-19 13:58:04 +02:00
85a1b4f807 clientsCronTrackExpansiveClients() actual implementation. 2018-07-19 13:54:20 +02:00
d4c5fc57db clientsCronTrackExpansiveClients() skeleton and ideas. 2018-07-19 13:49:00 +02:00
1c95c07596 Make vars used only by INFO CLIENTS local to the block.
Related to #4727.
2018-07-19 12:23:05 +02:00
16b8d364cc Merge pull request #4727 from kingpeterpaule/redis-fix-info-cli
move getClientsMaxBuffers func into  info clients command
2018-07-19 12:21:35 +02:00
0aca977c47 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-07-18 17:41:37 +02:00
313b2240ae In addReplyErrorLength() only panic when replying to slave.
See #5135 for more context.
2018-07-18 17:41:16 +02:00
6183f0590d Refine comment in addReplyErrorLength() about replying to masters/slaves.
See #5135 for some context.
2018-07-18 17:40:07 +02:00
22e9321c3e Merge pull request #5138 from oranagra/improve_defrag_test
make active defrag test more stable
2018-07-18 11:05:03 +02:00
f89c93c8ad make active defrag test more stable
on slower machines, the active defrag test tended to fail.
although the fragmentation ratio was below the treshold, the defragger was
still in the middle of a scan cycle.

this commit changes:
- the defragger uses the current fragmentation state, rather than the cache one
  that is updated by server cron every 100ms. this actually fixes a bug of
  starting one excess scan cycle
- the test lets the defragger use more CPU cycles, in hope that the defrag
  will be faster, but also give it more time before we give up.
2018-07-18 10:16:33 +03:00
8213f64d64 Merge pull request #5122 from trevor211/allowWritesWhenAofDisabled
Accept write commands if persisting is disabled
2018-07-17 18:08:46 +02:00
46fd927800 Merge pull request #4237 from aspirewit/update-comment
Update the comment
2018-07-17 18:04:55 +02:00
6201f7b4e0 Streams: better error when $ is given with XREADGROUP. 2018-07-17 17:54:10 +02:00
4bff45c7a6 Merge pull request #5136 from 0xtonyxia/fix-xread-id-parse
Fix xreadgroup with '$' ID.
2018-07-17 17:50:31 +02:00
afc7e08a20 Panic when we are sending an error to our master/slave.
Related to #5135, see discussion there.
2018-07-17 17:42:30 +02:00
e03358c0d9 Merge pull request #5135 from oranagra/rare_repl_corruption
fix rare replication stream corruption with disk-based replication
2018-07-17 17:33:58 +02:00
846cf12ae5 Streams: remove meaningless if condition.
It's already checked if xreadgroup is set and groupname is NULL.
2018-07-17 18:23:47 +08:00
6501b6bb6d Streams: return an error message if using xreadgroup with '$' ID.
Redis will always return an empty result when '$' ID is specified
with xreadgroup command, it's meaningless.
2018-07-17 18:19:10 +08:00
d55598988b fix rare replication stream corruption with disk-based replication
The slave sends \n keepalive messages to the master while parsing the rdb,
and later sends REPLCONF ACK once a second. rarely, the master recives both
a linefeed char and a REPLCONF in the same read, \n*3\r\n$8\r\nREPLCONF\r\n...
and it tries to trim two chars (\r\n) from the query buffer,
trimming the '*' from *3\r\n$8\r\nREPLCONF\r\n...

then the master tries to process a command starting with '3' and replies to
the slave a bunch of -ERR and one +OK.
although the slave silently ignores these (prints a log message), this corrupts
the replication offset at the slave since the slave increases the replication
offset, and the master did not.

other than the fix in processInlineBuffer, i did several other improvments
while hunting this very rare bug.

- when redis replies with "unknown command" it includes a portion of the
  arguments, not just the command name. so it would be easier to understand
  what was recived, in my case, on the slave side,  it was -ERR, but
  the "arguments" were the interesting part (containing info on the error).
- about a year ago i added code in addReplyErrorLength to print the error to
  the log in case of a reply to master (since this string isn't actually
  trasmitted to the master), now changed that block to print a similar log
  message to indicate an error being sent from the master to the slave.
  note that the slave is marked as CLIENT_SLAVE only after PSYNC was received,
  so this will not cause any harm for REPLCONF, and will only indicate problems
  that are gonna corrupt the replication stream anyway.
- two places were c->reply was emptied, and i wanted to reset sentlen
  this is a precaution (i did not actually see such a problem), since a
  non-zero sentlen will cause corruption to be transmitted on the socket.
2018-07-17 12:51:49 +03:00
cefe21d28a dict.c: remove a few trailing spaces. 2018-07-17 10:39:47 +02:00
4fc2099235 Merge pull request #5128 from kingpeterpaule/remove-one-loop-in-freeMemoryIfNeeded
remove ineffective loop in dictGetSomeKeys.
2018-07-17 10:38:55 +02:00
9fbd49bbaf Merge pull request #5113 from 0xtonyxia/using-compare-func-instead
Streams: using streamCompareID() instead of direct compare.
2018-07-16 18:34:35 +02:00
cab396761e Merge pull request #5127 from oranagra/sds_req_type
bugfix in sdsReqType creating 64bit sds headers on 32bit systems
2018-07-16 18:32:14 +02:00
f9c84d6d39 Hopefully improve commenting of #5126.
Reading the PR gave me the opportunity to better specify what the code
was doing in places where I was not immediately sure about what was
going on. Moreover I documented the structure in server.h so that people
reading the header file will immediately understand what the structure
is useful for.
2018-07-16 17:56:54 +02:00
e22a1218bc Merge pull request #5126 from oranagra/slave_buf_memory_2
slave buffers were wasteful and incorrectly counted causing eviction
2018-07-16 17:45:50 +02:00
28dd8dd150 Merge pull request #5132 from soloestoy/propagate-xdel-correctly
Streams: correctly propagate xdel if needed
2018-07-16 16:04:32 +02:00
bf680b6f8c slave buffers were wasteful and incorrectly counted causing eviction
A) slave buffers didn't count internal fragmentation and sds unused space,
   this caused them to induce eviction although we didn't mean for it.

B) slave buffers were consuming about twice the memory of what they actually needed.
- this was mainly due to sdsMakeRoomFor growing to twice as much as needed each time
  but networking.c not storing more than 16k (partially fixed recently in 237a38737).
- besides it wasn't able to store half of the new string into one buffer and the
  other half into the next (so the above mentioned fix helped mainly for small items).
- lastly, the sds buffers had up to 30% internal fragmentation that was wasted,
  consumed but not used.

C) inefficient performance due to starting from a small string and reallocing many times.

what i changed:
- creating dedicated buffers for reply list, counting their size with zmalloc_size
- when creating a new reply node from, preallocate it to at least 16k.
- when appending a new reply to the buffer, first fill all the unused space of the
  previous node before starting a new one.

other changes:
- expose mem_not_counted_for_evict info field for the benefit of the test suite
- add a test to make sure slave buffers are counted correctly and that they don't cause eviction
2018-07-16 16:43:42 +03:00
73306c6fed Streams: correctly propagate xdel if needed 2018-07-16 20:48:07 +08:00
103c5a1a3c Add a few comments to streamIteratorRemoveEntry(). 2018-07-16 12:41:55 +02:00
a317f55d25 Merge pull request #5131 from soloestoy/optimize-xdel
Streams: free lp if all elements are deleted
2018-07-16 12:39:38 +02:00
185e0d9cd8 Modify XINFO field from last-id to last-generated-id.
Related to #5129.
2018-07-16 12:25:26 +02:00
4215e74be5 Merge pull request #5129 from soloestoy/xinfo-show-last-id
Streams: show last id for streams and groups
2018-07-16 12:24:14 +02:00
c9324f81a2 Streams: free lp if all elements are deleted 2018-07-16 15:57:41 +08:00
b6ce7d5ddc Update dict.c
change coding style.
2018-07-16 14:29:59 +08:00
b4ba5ac840 Streams: show last id for streams and groups 2018-07-16 13:24:54 +08:00
816fc6cbc6 remove one ineffective loop in dictGetSomeKeys. 2018-07-16 11:28:22 +08:00
b05a22e261 bugfix in sdsReqType creating 64bit sds headers on 32bit systems 2018-07-15 18:24:18 +03:00
491682a668 Streams: using streamCompareID() instead of direct compare in block.c. 2018-07-14 15:03:05 +08:00
a2177cd2b6 Streams: add streamCompareID() declaration in stream.h. 2018-07-14 15:02:24 +08:00
0484dbcf68 Streams: using streamCompareID() instead of direct compare. 2018-07-14 20:34:06 +08:00
2d4366c544 Accept write commands if persisting is disabled,
event if we do have problems persisting on disk
previously.
2018-07-14 09:06:23 +08:00
ab33bcd346 Merge pull request #5120 from andrewsensus/leap-year-comment-patch-1
update leap year comment
2018-07-13 18:18:45 +02:00