mirror of
https://github.com/fluencelabs/redis
synced 2025-06-23 05:51:34 +00:00
Allow an AOF rewrite buffer > 2GB (Fix for issue #504).
During the AOF rewrite process, the parent process needs to accumulate the new writes in an in-memory buffer: when the child will terminate the AOF rewriting process this buffer (that ist the difference between the dataset when the rewrite was started, and the current dataset) is flushed to the new AOF file. We used to implement this buffer using an sds.c string, but sds.c has a 2GB limit. Sometimes the dataset can be big enough, the amount of writes so high, and the rewrite process slow enough that we overflow the 2GB limit, causing a crash, documented on github by issue #504. In order to prevent this from happening, this commit introduces a new system to accumulate writes, implemented by a linked list of blocks of 10 MB each, so that we also avoid paying the reallocation cost. Note that theoretically modern operating systems may implement realloc() simply as a remaping of the old pages, thus with very good performances, see for instance the mremap() syscall on Linux. However this is not always true, and jemalloc by default avoids doing this because there are issues with the current implementation of mremap(). For this reason we are using a linked list of blocks instead of a single block that gets reallocated again and again. The changes in this commit lacks testing, that will be performed before merging into the unstable branch. This fix will not enter 2.4 because it is too invasive. However 2.4 will log a warning when the AOF rewrite buffer is near to the 2GB limit.
This commit is contained in:
@ -505,7 +505,7 @@ struct redisServer {
|
||||
off_t aof_current_size; /* AOF current size. */
|
||||
int aof_rewrite_scheduled; /* Rewrite once BGSAVE terminates. */
|
||||
pid_t aof_child_pid; /* PID if rewriting process */
|
||||
sds aof_rewrite_buf; /* buffer taken by parent during oppend only rewrite */
|
||||
list *aof_rewrite_buf_blocks; /* Hold changes during an AOF rewrite. */
|
||||
sds aof_buf; /* AOF buffer, written before entering the event loop */
|
||||
int aof_fd; /* File descriptor of currently selected AOF file */
|
||||
int aof_selected_db; /* Currently selected DB in AOF */
|
||||
@ -843,6 +843,8 @@ int loadAppendOnlyFile(char *filename);
|
||||
void stopAppendOnly(void);
|
||||
int startAppendOnly(void);
|
||||
void backgroundRewriteDoneHandler(int exitcode, int bysignal);
|
||||
void aofRewriteBufferReset(void);
|
||||
unsigned long aofRewriteBufferSize(void);
|
||||
|
||||
/* Sorted sets data type */
|
||||
|
||||
|
Reference in New Issue
Block a user