Commit Graph

244 Commits

Author SHA1 Message Date
d3a9dff6b9 Merge pull request #6615 from soloestoy/wrap-also-propagate-as-multi
Wrap also propagate as multi
2019-12-19 09:24:52 +01:00
9a7b6a9f51 Merge pull request #5780 from soloestoy/lua-multi-more-clear
scripting: flag lua_client as CLIENT_MULTI after redis.replicate_command() immediately
2019-12-19 09:06:28 +01:00
12caffee61 Added a missed space in lua errors 2019-12-16 23:38:31 -08:00
576a08908b Split error message so dependandent callers give a useful result 2019-12-16 23:34:37 -08:00
44aa22c635 Added better exception handling around scripting and module 2019-12-16 23:33:53 -08:00
c73d70fb46 Propagation: propagate EXEC directly in lua script 2019-11-22 15:45:21 +08:00
37a10cef02 Propagation: wrap commands in also_propagate array with MULIT/EXEC
Random command like SPOP with count is replicated as
some SREM operations, and store them in also_propagate
array to propagate after the call, but this would break
atomicity.

To keep the command's atomicity, wrap also_propagate
array with MULTI/EXEC.
2019-11-22 15:42:49 +08:00
73841e8c49 scripting: a little refactor about EVAL and redis.call()
Change server.lua_client's flag in a more explicit way.
2019-11-22 11:58:03 +08:00
097c4a6536 scripting: flag lua_client as CLIENT_MULTI after redis.replicate_commands() immediately
To avoid nested MULTI/EXEC, we check the lua_caller's flag,
if we are in the MULTI context we flag the lua_client as
CLIENT_MULTI, but it's not enough we shoud flag lua_client
as CLIENT_MULTI after redis.replicate_commands() immediately
or the first write command after redis.replicate_commands()
cannot know it's in an transaction, I know the missing CLIENT_MULTI
doesn't have any effect now, but it's a real bug and we should fix
it, in case someday we allow some dangerous command like BLPOP.
2019-11-22 11:51:18 +08:00
0db3b0a0ff Merge remote-tracking branch 'upstream/unstable' into tls 2019-10-16 17:08:07 +03:00
b087dd1db6 TLS: Connections refactoring and TLS support.
* Introduce a connection abstraction layer for all socket operations and
integrate it across the code base.
* Provide an optional TLS connections implementation based on OpenSSL.
* Pull a newer version of hiredis with TLS support.
* Tests, redis-cli updates for TLS support.
2019-10-07 21:06:13 +03:00
ee1cef189f Minor aesthetic changes to #6419. 2019-10-04 12:00:41 +02:00
98426e9886 On LUA script timeout, print the script SHA to the log
since the slowlog and other means that can help you detect the bad script
are only exposed after the script is done. it might be a good idea to at least
print the script name (sha) to the log when it timeouts.
2019-10-02 08:40:35 +03:00
6129758558 Merge branch 'unstable' into modules_fork 2019-09-27 11:24:06 +02:00
0a146a8be4 Add useless break for uniformity / future protection. 2019-09-20 11:19:07 +02:00
30a3644e64 RESP3: Fix function redisProtocolToLuaType about RESP3->Lua bools. 2019-09-20 08:37:23 +08:00
8ea4bdd91d RESP3: Lua double -> RESP3 conversion. 2019-09-17 19:26:46 +02:00
89f929b12a RESP3: RESP3 double -> Lua conversion. 2019-09-17 19:20:30 +02:00
19aba4ac78 RESP3: double -> human readable conversion. 2019-09-17 19:08:33 +02:00
e8e30bc402 RESP3: bool and null values in RESP -> human readable conversion. 2019-09-17 18:57:24 +02:00
f01f0c02d1 RESP3: convert RESP3 null as Lua nil. Implement RESP3->Lua bools. 2019-09-16 18:36:16 +02:00
6931004969 RESP3: change behavior of Lua returning true/false for RESP3.
Here we introduce a change in the way we convert values from Lua to
Redis when RESP3 is selected: this is possible without breaking the fact
we can return directly what a command returned, because there is no
Redis command in RESP2 that returns true or false to Lua, so the
conversion in the case of RESP2 is totally arbitrary. When a script is
written selecting RESP3 from Lua, it totally makes sense to change such
behavior and return RESP3 true/false when Lua true/false is returned.
2019-09-16 18:18:17 +02:00
2cc4d0286c RESP3: implement new NULL representation parsing in Lua. 2019-09-16 17:49:47 +02:00
ca81d49006 RESP3: handle set Lua -> Redis conversion. 2019-09-16 12:19:19 +02:00
c792504fab RESP3: handle map Lua -> Redis conversion. 2019-09-16 12:15:39 +02:00
dad38c19c8 RESP3: report set/map as nested tables to Lua. 2019-09-16 11:49:42 +02:00
888efc1b36 RESP3: Lua parsing should depend on lua client, not lua caller.
We want all the scripts to run in RESP2 mode by default. It's up to the
caller to switch to V3 using redis.setresp() if it is really needed.
This way most scripts written for past Redis versions will continue to
work with Redis >= 6 even if the client is in RESP3 mode.
2019-09-13 19:38:39 +02:00
26b6c697d3 RESP3: Lua debugger support for printing sets and maps. 2019-09-13 19:19:10 +02:00
93c52ff5ff RESP3: implement lua.setresp(). 2019-09-13 19:01:39 +02:00
56258c6b7d Module API for Forking
* create module API for forking child processes.
* refactor duplicate code around creating and tracking forks by AOF and RDB.
* child processes listen to SIGUSR1 and dies exitFromChild in order to
  eliminate a valgrind warning of unhandled signal.
* note that BGSAVE error reply has changed.

valgrind error is:
  Process terminating with default action of signal 10 (SIGUSR1)
2019-07-17 16:40:24 +03:00
325fc1cb2e CommandFilter API: Support Lua and RM_call() flows. 2019-03-18 23:06:38 +02:00
4380423d40 ACL: enforce ACLs in Lua scripts as well. 2019-01-29 10:12:22 +01:00
317f8b9d38 RESP3: most null replies converted. 2019-01-09 17:00:29 +01:00
a1feda2388 RESP3: Scripting RESP3 mode set/map protocol -> Lua conversion. 2019-01-09 17:00:29 +01:00
9330bcc7ee RESP3: Fix API in scripting.c leaving Lua conversions RESP2. 2019-01-09 17:00:29 +01:00
929c686cce Actually use the protectClient() API where needed.
Related to #4804.
2018-10-09 13:18:52 +02:00
ecc48369ce Fix invalid use of sdsZmallocSize on an embedded string
sdsZmallocSize assumes a dynamically allocated SDS. When given a string
object created by createEmbeddedStringObject, it calls zmalloc_size on a
pointer that isn't the one returned by zmalloc
2018-09-30 11:32:48 +02:00
1051d93837 Slave removal: scripting.c logs and other stuff fixed. 2018-09-11 15:32:28 +02:00
7895835df6 Use commands (effects) replication by default in scripts.
See issue #5250 and issue #5292 for more info.
2018-09-05 19:33:56 +02:00
7e11825ef4 Safer script stop condition on OOM.
Here the idea is that we do not want freeMemoryIfNeeded() to propagate a
DEL command before the script and change what happens in the script
execution once it reaches the slave. For example see this potential
issue (in the words of @soloestoy):

On master, we run the following script:

    if redis.call('get','key')
    then
        redis.call('set','xxx','yyy')
    end
    redis.call('set','c','d')

Then when redis attempts to execute redis.call('set','xxx','yyy'), we call freeMemoryIfNeeded(), and the key may get deleted, and because redis.call('set','xxx','yyy') has already been executed on master, this script will be replicated to slave.

But the slave received "DEL key" before the script, and will ignore maxmemory, so after that master has xxx and c, slave has only one key c.

Note that this patch (and other related work) was authored collaboratively in
issue #5250 with the help of @soloestoy and @oranagra.

Related to issue #5250.
2018-09-05 15:48:08 +02:00
092e4de613 Propagate read-only scripts as SCRIPT LOAD.
See issue #5250 and the new comments added to the code in this commit
for details.
2018-09-05 15:44:33 +02:00
6c001bfc0d Unblocked clients API refactoring. See #4418. 2018-09-03 18:39:18 +02:00
2290c4bee1 if master is already unblocked, do not unblock it twice 2018-09-03 14:36:48 +08:00
7fa493912e After slave Lua script leaves busy state, re-process the master buffer.
Technically speaking we don't really need to put the master client in
the clients that need to be processed, since in practice the PING
commands from the master will take care, however it is conceptually more
sane to do so.
2018-08-31 16:45:02 +02:00
83af8ef1fd Allow scripts to timeout even if from the master instance.
However the master scripts will be impossible to kill.

Related to #5297.
2018-08-31 16:45:02 +02:00
f5b29c6444 Allow scripts to timeout on slaves as well.
See reasoning in #5297.
2018-08-31 16:45:01 +02:00
132be8aed5 Correct "did not received" -> "did not receive" typos/grammar. 2018-08-26 14:45:39 +02:00
db693be00d Refactoring: replace low-level checks with writeCommandsDeniedByDiskError(). 2018-07-31 13:16:43 +02:00
ac3c012a7f Merge pull request #5153 from trevor211/fixLuaScript
Consider aof write error as well as rdb in lua script.
2018-07-30 18:10:06 +02:00
445a2a2b1b Merge pull request #4883 from itamarhaber/lua_scripts-in-info-memory
Adds memory information about the scripts' cache to INFO
2018-07-23 18:43:05 +02:00