Commit Graph

5975 Commits

Author SHA1 Message Date
482785ac62 add malloc_usable_size for libc malloc
this reduces the extra 8 bytes we save before each pointer.
but more importantly maybe, it makes the valgrind runs to be more similiar
to our normal runs.

note: the change in malloc_stats struct in server.h is to eliminate an name conflict.
structs that are not typedefed are resolved from a separate name space.
2018-06-19 18:18:23 +03:00
4da296307c Merge pull request #5023 from FX-HAO/unstable
Fix update_zmalloc_stat_alloc in zrealloc
2018-06-19 16:50:22 +02:00
5f5e1199ef Merge pull request #5041 from oranagra/redis-rdb-check_rdbLoadMillisecondTime
fix redis-rdb-check to provide proper arguments to rdbLoadMillisecondTime
2018-06-19 16:06:11 +02:00
4848fbec8b Modules: convert hash to hash table for big objects. 2018-06-19 16:03:00 +02:00
f31b0405f0 fix redis-rdb-check to provide proper arguments to rdbLoadMillisecondTime
due to incorrect forward declaration, it didn't provide all arguments.
this lead to random value being read from the stack and return of incorrect time,
which in this case doesn't matter since no one uses it.
2018-06-19 16:54:22 +03:00
333c98c43a AOF: remove no longer used variable "now". 2018-06-19 15:54:05 +02:00
e94b2053c6 Modify clusterRedirectClient() to handle ZPOP and XREAD. 2018-06-19 15:53:32 +02:00
26229aa607 use safe macro (non empty) in memrev64ifbe to eliminate empty if warning 2018-06-19 16:46:41 +03:00
5cd3c9529d 64 bit RDB_OPCODE_RESIZEDB in rdb saving
this complication in the code is from times were rdbSaveLen didn't support 64 bits.
2018-06-19 16:43:12 +03:00
ba92b517b8 Remove AOF optimization to skip expired keys.
Basically we cannot be sure that if the key is expired while writing the
AOF, the main thread will surely find the key expired. There are
possible race conditions like the moment at which the "now" is sampled,
and the fact that time may jump backward.

Think about the following:

SET a 5
EXPIRE a 1

AOF rewrite starts after about 1 second. The child process finds the key
expired, while in the main thread instead an INCR command is called
against the key "a" immediately after a fork, and the scheduler was
faster to give execution time to the main thread, so "a" is yet not
expired.

The main thread will generate an INCR a command to the AOF log that will
be appended to the rewritten AOF file, but that INCR command will target
a non existin "a" key, so a new non volatile key "a" will be created.

Two observations:

A) In theory by computing "now" before the fork, we should be sure that
if a key is expired at that time, it will be expired later when the
main thread will try to access to such key. However this does not take
into account the fact that the computer time may jump backward.

B) Technically we may still make the process safe by using a monotonic
time source.

However there were other similar related bugs, and in general the new
"vision" is that Redis persistence files should represent the memory
state without trying to be too smart: this makes the design more
consistent, bugs less likely to arise from complex interactions, and in
the end what is to fix is the Redis expire process to have less expired
keys in RAM.

Thanks to Oran Agra and Guy Benoish for writing me an email outlining
this problem, after they conducted a Redis 5 code review.
2018-06-19 15:43:06 +02:00
44571088d8 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-06-18 17:09:16 +02:00
6967d0bd5a Revert fix #4976 just leaving the flush() part. 2018-06-18 17:09:00 +02:00
0ed0dc3c02 Fix incrDecrCommand() to create shared objects when needed.
See #5011.
2018-06-18 16:56:31 +02:00
bd92389c2d Refactor createObjectFromLongLong() to be suitable for value objects. 2018-06-18 16:55:16 +02:00
3518bb66d7 Merge pull request #5020 from shenlongxing/fix-config
Fix config_set_numerical_field() integer overflow.
2018-06-18 16:02:23 +02:00
2076660843 Streams: fix xreadGetKeys() for correctness.
The old version could not handle the fact that "STREAMS" is a valid key
name for streams. Now we really try to parse the command like the
command implementation would do.

Related to #5028 and 4857.
2018-06-18 14:06:06 +02:00
e670ccffea Merge pull request #4857 from youjiali1995/fix-command-getkeys
Fix core dump when using some commands with wrong arguments.
2018-06-18 13:54:38 +02:00
a0b27dae85 Streams: fix xreadGetKeys() buffer overflow.
The loop allocated a buffer for the right number of keys positions, then
overflowed it going past the limit.

Related to #4857 and cause of the memory violation seen in #5028.
2018-06-18 13:51:19 +02:00
62f9ac6f43 Streams: Change XADD MAXLEN handling of values <= 0.
Now a MAXLEN of 0 really does what it means: it will create a zero
entries stream. This is useful in order to make sure that the behavior
is identical to XTRIM, that must be able to reduce the stream to zero
elements when MAXLEN is given.

Also now MAXLEN with a count < 0 will return an error.
2018-06-18 10:05:21 +02:00
d4c4f20a45 Update sort.c
Redundant second if, and we may use "not" operation for bool, I suppose it's faster.
2018-06-17 21:25:51 +02:00
79a1c19ac2 XADD MAXLEN should return an error for values < 0. 2018-06-17 10:44:01 +02:00
2e0ab4a807 Merge pull request #4976 from trevor211/fixDebugLoadaof
Critical: Fix server crash and data inconsistency in some cases.
2018-06-16 11:05:04 +02:00
94658303e9 Merge pull request #4758 from soloestoy/rdb-save-incremental-fsync
Rdb save incremental fsync
2018-06-16 10:59:37 +02:00
6a66b93b18 Sentinel: add an option to deny online script reconfiguration.
The ability of "SENTINEL SET" to change the reconfiguration script at
runtime is a problem even in the security model of Redis: any client
inside the network may set any executable to be ran once a failover is
triggered.

This option adds protection for this problem: by default the two
SENTINEL SET subcommands modifying scripts paths are denied. However the
user is still able to rever that using the Sentinel configuration file
in order to allow such a feature.
2018-06-14 18:57:58 +02:00
d353023a83 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-06-14 18:11:04 +02:00
d6e8fe77af Fix infinite loop in dbRandomKey().
Thanks to @kevinmcgehee for signaling the issue and reasoning about the
consequences and potential fixes.

Issue #5015.
2018-06-14 18:08:21 +02:00
40d5df6547 Merge pull request #5008 from zwkno1/unstable
fix geohashEncode bug when step > 30
2018-06-14 14:01:40 +02:00
8bc3ffcb29 Merge pull request #5021 from soloestoy/fix-exists
fix exists command on slave
2018-06-14 13:58:30 +02:00
6c7847a183 Merge pull request #5018 from soloestoy/optimize-reply
optimize reply list memory usage
2018-06-14 13:32:00 +02:00
a4f658b2b5 Fix update_zmalloc_stat_alloc in zrealloc 2018-06-14 16:44:19 +08:00
2ffa533f85 fix exists command on slave 2018-06-14 01:30:07 +08:00
e4e5a670e7 Fix config_set_numerical_field() integer overflow. 2018-06-13 23:27:53 +08:00
963002d71e optimize reply list memory usage 2018-06-13 20:35:40 +08:00
ce17f76bbd Security: fix redis-cli buffer overflow.
Thanks to Fakhri Zulkifli for reporting it.

The fix switched to dynamic allocation, copying the final prompt in the
static buffer only at the end.
2018-06-13 12:40:33 +02:00
032ea657d7 RDB: Apply fix to rdbLoadMillisecondTime() only for new RDB versions.
This way we let big endian systems to still load old RDB versions.
However newver versions will be saved and loaded in a way that make RDB
expires cross-endian again. Thanks to @oranagra for the reporting and
the discussion about this problem, leading to this fix.
2018-06-12 18:21:39 +02:00
6b8a24a665 Streams: generate a few additional events.
Currently it does not look it's sensible to generate events for streams
consumer groups modification, being them metadata, however at least for
key-level events, like the creation or removal of a consumer group, I
added a few events here and there. Later we can evaluate if it makes
sense to add more. From the POV instead of WAIT (in Redis transaciton)
and signaling the key as modified, it looks like that the transaction
should not fail when a stream is modified, so no calls are made in
consumer groups related functions to signalModifiedKey().
2018-06-12 18:11:15 +02:00
b38682199b Fix rdbSaveKeyValuePair() integer overflow.
Again thanks to @oranagra. The object idle time does not fit into an int
sometimes: use the native type that the serialization function will get
as argument, which is uint64_t.
2018-06-12 17:31:04 +02:00
e534e9aa83 In scanDatabaseForReadyLists() now we need to handle ZSETs as well.
Since the introduction of ZPOP makes this needed. Thanks to @oranagra
for reporting.
2018-06-12 17:28:40 +02:00
f70e88c1f6 RDB: store times consistently in little endian.
I'm not sure how this escaped the attention of Redis users for years,
but finally @oranagra reported this issue... Thanks to Oran.
2018-06-12 17:22:03 +02:00
4774d61691 Streams: improve type correctness in t_stream.c. 2018-06-12 14:12:53 +02:00
6b675b9525 Applies addReplySubSyntaxError to stream commands 2018-06-12 14:52:50 +03:00
bcc42028c1 Fix XGROUP help missing space. 2018-06-12 13:20:46 +02:00
e916f4519c Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-06-12 13:14:01 +02:00
bcecbebb74 Merge pull request #5012 from shenlongxing/fix-config
fix active-defrag-threshold value error
2018-06-12 13:05:19 +02:00
13957c9d07 fix active-defrag-threshold value error
The active-defrag-threshold-lower/active-defrag-threshold-upper min/max  value in redis.conf should be consistent with 'config set' command.
2018-06-12 17:59:32 +08:00
7cc1312789 Streams: fix backward iteration when entry is not flagged SAMEFIELD.
See issue #5006. The comment in the code was also wrong and
was rectified as well.
2018-06-12 10:22:49 +02:00
7659619824 Update geohash.c
fix geohasEncode bug when step > 31
2018-06-12 15:28:28 +08:00
82661ba329 Merge pull request #5007 from leonchen83/patch-2
fix typo issue #5005
2018-06-12 09:28:26 +02:00
923e63e5ec Streams: increment dirty counter for XGROUP SETID/DESTROY.
See issue #5005 comments.
2018-06-12 09:27:40 +02:00
fac3e8aab5 fix typo
fix [#5005](https://github.com/antirez/redis/issues/5005)
2018-06-12 08:52:18 +08:00