Redis is an in-memory database, the data stored in memory, reading and writing efficiency of the database stored on disk is much faster than traditional data. But once the process exits, Redis data will be lost.
To solve this problem, Redis provides RDB and AOF two kinds of persistence solution to save the data in memory to disk, to avoid data loss. RDB is described in this article “Redis RDB persistent explanation”, today we look at AOF related principles.
AOF (append only file) persisted in an independent manner the log records every time a write command and re-execute the AOF file command when Redis restart in order to achieve the purpose of data recovery. AOF main role is to solve real-time data persistence.
RDB and AOF
antirez in “Redis persistence decryption” in an article about the AOF RDB and their advantages and disadvantages:
RDB is a compact compressed binary file, on behalf of Redis data backup at some point in time. It is ideal for backup, the whole amount of replication scenarios. For example bgsave backup performed every 6 hours, and copy the file to the remote machine or RDB file system for disaster recovery.
Redis load RDB to recover data much faster than AOF way
RDB way no way to do real-time data persistence, and AOF ways to do it.
Here, we look at how to do real-time AOF is persistent.
AOF persistence to achieve
As shown above, to achieve AOF persistence function may be added into the command (the append), file write (Write), file synchronization (Sync), overwriting files (the rewrite) and restart the loading (load). Process is as follows:
All write command will be appended to the AOF buffer.
AOF buffer synchronous operation to the hard disk according to a corresponding policy.
With AOF files increases, the need for regular files for AOF rewrite, to achieve the purpose of compression.
When Redis reboot, you can load AOF file for data recovery.
When AOF persistence feature is turned on, Redis after the completion of the implementation of a write command, by agreement format (that is, RESP, namely communication protocol Redis client and server interaction) will write commands are executed appended to the Redis server AOF end of the buffer maintenance.
For example, SET mykey myvalue this command buffer AOF recorded in the following format.
Redis protocol format this will not repeat, AOF reason why direct agreement in text format, because all write commands append operation should be carried out directly using protocol format, to avoid secondary processing overhead.
Write and file synchronization
Redis before the end of each event loop, it will call flushAppendOnlyFile function, you need to determine whether the contents of AOF buffer is written to the AOF and synchronize files.
Behavior flushAppendOnlyFile function is determined by the value redis.conf configuration appendfsync options. This option has three optional values, which are always, everysec and no:
always: Redis must be written in each event cycle AOF all the contents of the buffer to AOF files, and synchronize files AOF, so the efficiency is always the option of three values among appendfsync the worst one, but from security to that is the most secure. When downtime occurs, AOF persistence you will only lose a command data generated by the event loop.
everysec: Redis cycle should AOF all the contents of the buffer is written in each event to AOF file, and every second is necessary for AOF files in a sub-thread synchronization. From an efficiency point of view, this mode is fast enough. When downtime occurs, it will only lose one second command data.
no: Redis must be written to the file AOF AOF all the contents of the buffer in each event loop. The operating system controls the synchronization of AOF documents. In this mode the fastest, but time synchronization interval is long, more data can be lost if it fails.
write operation will trigger the Linux system write latency (delayed write) mechanism. In the Linux kernel provides page buffer is used to provide hard disk IO performance. write operation returns directly after the write buffer system. Hard synchronization mechanism operating system dependent scheduling, for example: page buffer space is full or reaches a certain time period. Prior to synchronize files, if a system failure is down at this time, the buffer data will be lost.
The fsync for a single file operations, forcing them hard sync, fsync will block until written to disk is returned after completion to ensure data persistence.
Appendfsync three values representing three different calls fsync strategy. The more frequent calls fsync cycle, the worse the efficiency of reading and writing, but the corresponding higher security, the less data lost during downtime occurs.
For Linux, I / O system calls, and each action as shown in FIG. You can view details “to talk about Linux I / O” article.
AOF Data Recovery
AOF file inside contains all the commands necessary to rebuild Redis write data, so long as Redis read and re-run it again AOF files saved inside a write command, you can restore the previous state Redis closed.
AOF Redis read detailed steps and the reduction state of the database files as follows:
Create a fake client with no network connection, because Redis commands can only be executed in the client context, and the commands used to load the AOF file directly originate from the AOF file rather than the network connection, so the server uses a pseudo-client with no network connection to execute the file pieces saved write commands, pseudo A client executes a command exactly the same as a client with a network connection.
Analysis AOF file and remove a write command.
Use the pseudo-client to execute the read write command.
It has been implementing steps 2 and 3 until all AOF file write commands have been processed so far.
After completing the above steps, the database state saved by the AOF file will be fully restored.
Because AOF persistence Redis is to record the state by saving the write command is executed, so long as Redis running, the contents of the file AOF will be more and more, the file size will be growing, if not controlled, oversized AOF file is likely to affect the Redis even host computer.
In order to solve the problem AOF file volume expansion, Redis provides file AOF rewrite (rewrite) function. With this feature, Redis can create a new AOF AOF file to replace the existing file. Same old and new AOF files saved Redis state, but the new AOF file does not contain any command to honor a waste of space, so the volume of new AOF file is usually much smaller than the volume of a lot of old AOF documents.
As shown above, to be recorded before the overwriting state key called list, AOF of five commands to save the file, and after rewriting, only necessary to save one command.
AOF file does not need to rewrite existing file AOF any read or write operation analysis, but achieved by reading the current state of the database server. First, the key value read from the database now, and then use a command to record key-value pairs, recording multiple commands in place before this key-value pairs, this is the realization of the principle functions of the AOF rewrite.
In the actual process, in order to avoid client input buffer overflow, AOF rewrite when performing command when dealing with lists, hash tables, set and ordered set of four keys with multiple elements may be, it will key to check the number of elements contained, if the quantity exceeds REDISAOFREWRITEITEMSPER_CMD (typically 64) constant, then the recording command using the plurality of values of the key, rather than a command.
There are three main trigger mechanisms for rewrite:
Bgrewriteaof manually invoke the command, if you currently have rewrite child running processes, then this rewrite will postpone the execution, or directly trigger a rewrite.
Open AOF By configuring the instruction manual, if there is no RDB child under process conditions, will trigger a rewrite, the data is written to rewrite the current database file.
In Redis timer, if there is need to exit rewrite execution and no RDB or rewrite a running child process is triggered once the file size has been reached or AOF rewrite conditions configured automatically trigger once.
AOF rewrite the background
The AOF rewrite function does a lot of write operations and the thread calling the function will be blocked for a long time, so Redis performs an AOF rewrite operation in the child process.
During AOF rewriting, the Redis process can continue to process client command requests.
The child process has a copy of the in-memory data of the parent process, which can also guarantee the security of the data if the lock is not applied.
However, during the restart AOF conducted in the child, Redis receiving client orders, the state will modify an existing database, resulting in inconsistent data and the current state of the AOF rewrite files saved in the database state.
To do this, Redis sets up an AOF rewrite buffer, which is used after the server creates the child process, and when Redis finishes executing a write command, it sends the write command to both the AOF buffer and the AOF rewrite buffer.
When the child process to complete AOF rewrite work, it sends a signal to the parent process, the parent process after receiving the signal, a signal processing function calls, and perform the following tasks:
AOF rewrite all the contents of the buffer is written to the new AOF file, to ensure consistent database files saved in the new AOF status and the current state of the server.
AOF new rename file, to overwrite the existing file AOF atom, a complete replacement of the old and new files
Continue to process client requests command.
In the background throughout the AOF rewrite process, only the main process will Redis signal processing function is executed causing obstruction, at other times, AOF rewrite the background will not block the main process.
Follow-up will continue to learn Redis replication and cluster-related knowledge, we hope lasting attention.
Personal blog, welcome to play. address