Categories
Uncategorized

A thorough understanding of the text Redis serialization protocol, you can also write Redis client

premise

Netty want to learn recently when coding and decoding module based on a Redis service agreement, the way the process of reading the sequence of Redis service agreement RESP, combined with his understanding of the document translation and achieve a simple Java-based analytical RESP. JDK version used for the preparation of this paper [8+].

RESP Profile

Redis Redis client and server based on a protocol called RESP communicate, called the RESP Redis Serialization Protocol, i.e. Redis serialization protocol. Although RESP as Redis design, but it can also be applied in other client – server (Client-Server) software projects. RESP in the design trade-offs to consider the following points:

    Easy to implement.

    Quick resolution.

    High readability.

RESP can serialize different data types, such as integers, strings, arrays, there is a special type Error. Redis command to be executed will be packaged as a request is similar to a string array and then sent to the server by Redis Redis client. Redis server selects corresponding to a data type based on a particular reply command type (which is a translation, the original is: Redis replies with a command-specific data type).

RESP secure binary (binary-safe), and need not be transmitted from one process to another process batch process data at RESP, because it uses the prefix length (prefixed-length, will be analyzed later, that is, each data prefix block has a good definition of the number of data blocks, which is similar to the fixed-length coding Netty decoding) to transfer bulk data.

Note: The protocol outlined here only used in a client – server communication, Redis Cluster using different binary protocol messages exchanged between the nodes (i.e. does not use the RESP Redis communication between nodes in the cluster).

Network layer

Redis clients by creating a connection on TCP 6379 port, connect to the Redis server.

Although technically non-TCP RESP specific underlying communication protocol in, but in the context of Redis, RESP only for the TCP connection (or similar stream oriented connection, such as Unix sockets).

Request – response model

Redis server receives a command composed of different parameters, and the command is received after the reply is sent back process will Redis client. This is the simplest model, but there are two exceptions in the case of:

    Redis support pipe (Pipelining, assembly line, in most cases used to be called the pipeline) operations. The case of the pipeline, Redis client may send multiple commands once, and then wait for a one-time reply (the text of the reply are replies, understood as Redis server will return a one-time batch reply result).

    When a client subscription Redis Pub / Sub Channel, the agreement will change the semantics of the protocol and become a push (push protocol), that is to say, the client no longer needs to send commands, because the server will automatically Redis client (subscribed to change channel clients) send a new message (it is meant herein: in the publish / subscribe model, the service end message is Redis proactively pushed to subscribers of a particular channel Redis clients).

In addition to these two exceptions, Redis protocol is a simple request – response protocol.

RESP supported data types

RESP 1.2 introduced in Redis, in Redis 2.0, RESP became Redis standard protocol to communicate with the service end. That is, if you need to write Redis client, you have to implement this protocol in the client.

Is a sequence of protocol RESP nature, it supports the following data types: line string, the error message, an integer number, and a fixed-length string array RESP.

RESP Redis as the request – response protocol following manner:

    Redis client command to package an array type of RESP (array elements are fixed-length string type, attention to this point, is important) Redis sent to the server.

    Redis server reply command is implemented in accordance with one type of data selected corresponding to the RESP.

In the RESP, depending on the data type of the first byte of the data packet:

    The first byte of a line string is +.

    The first byte is the error message -.

    The first byte integer number is:

    The first byte fixed-length string is $.

    The first array is byte * RESP.

Further, in the RESP may be used in an array of fixed-length string or variants represent a special Null value, it will be mentioned later. Different parts of the RESP, the protocol always \ r \ n (CRLF) terminates.

A summary of the five data types currently in RESP is as follows:

type of data

This article name translation

Basic features

example

Single-line string

The first byte is+, the last two bytes are rn, the other bytes are string content

Error message

The first byte is -, the last two bytes are rn, the other bytes are the text content of the exception message

Integer number

The first byte is:, the last two bytes are \ r \ n, the other bytes of the text content of digital

Fixed-length string

The first byte is $, the next byte is the content string length rn, the last two bytes are rn, the other bytes are the string content

RESP array

* The first byte is followed is the number of bytes of elements \ r \ n, the last two bytes are \ r \ n, the other bytes of the contents of individual elements, each element may be any type of data

Simple String +OK\r\n
Error -ERR\r\n
Integer :100\r\n
Bulk String $4\r\ndoge\r\n
Array *2\r\n:100\r\n$4\r\ndoge\r\n

The following subsections provide a more detailed analysis of each data type.

RESP simple string -Simple String

Simple strings are encoded as follows:

    (1) The first byte is +.

    (2) This is followed by a string that cannot contain CR or LF characters.

    (3) Termination with CRLF.

Simple strings guarantee the transfer of non-binary safe strings with minimal overhead. For example, many Redis commands will need to reply to the OK string. At this point, a 5-byte datarram encoded by a simple string is as follows:

+OK\r\n

If you need to send a binary safe string, you need to use a fixed length string.

When the Redis server responds with a simple string, the Redis client library should return a string to the caller that is composed of characters after the+until the end of the string content (actually the content of part (2) mentioned above), excluding the final CRLF bytes.

RESP Error Message – Error

The error message type is a RESP-specific data type. In fact, the error message type is basically the same as the simple string type, except that its first byte is -. The biggest difference between the error message type and the simple string type is that the error message should be perceived as an exception for the client, and the string content in the error message should be perceived as an error message returned by the Redis server. The error message is encoded as follows:

    (1) The first byte is -.

    (2) This is followed by a string that cannot contain CR or LF characters.

    (3) Termination with CRLF.

A simple example is as follows:

-Error message\r\n

Redis server will only reply to error messages if there is a real error or perceived error, such as attempting to perform an operation on the wrong data type or the command does not exist. When a Redis client receives an error message, an exception should be triggered (usually thrown directly, which can be classified according to the contents of the error message). Here are some examples of error message response:

-ERR unknown command 'foobar'
-WRONGTYPE Operation against a key holding the wrong kind of value

– The content between the first word after to the first space or newline, representing the type of error returned. This is just a convention used by Redis, not part of the RESP error message format.

For example, ERR is a generic error, WIRTYPE is a more specific error, indicating that the client is trying to perform an operation on the wrong data type. This definition, called an error prefix, is a way to enable clients to understand the type of error returned by the server, without relying on the exact message definition given, which can change over time.

Client implementation can return different types of anomalies for different types of errors, or by the name of the wrong type as a string directly to the caller to provide a common method of capturing wrong.

However, the function of error message classification processing should not be considered critical as it does not have a huge effect, and some client implementations may simply return a specific value to block error messages as a generic exception handling, such as returning false directly.

RESP Integer Number – Integer

Integer numbers are encoded as follows:

    (1) The first byte is:.

    (2) Next is a string that can’t contain CR or LF characters, that is, the number should be converted to a sequence of characters, and eventually output to bytes.

    (3) Termination with CRLF.

For example:

:0\r\n
:1000\r\n

Many Redis commands return integer numbers, like INCR, LLEN, LASTSAVE commands and so on.

The integer number returned has no special meaning, like INCR returns the total amount of increments, while LASTSAVE is the UNIX timestamp. However, the Redis server guarantees that the returned integer numbers are in the range of signed 64-bit integers.

In some cases, integer number returned will refer to true or false. The command returns EXISTS or SISMEMBER represents true, 0 representatives false.

In some cases, the integer number returned indicates whether the command actually produced an effect. Command execution such as SADD, SREM and SETNX returns 1 for the command execution to take effect and 0 for the command execution to not take effect (equivalent to the command not executed).

The following set of commands are executed to return integer numbers: SETNX, DEL, EXPLE, INCRBY, DECR, DECRBY, DBSIZE, LASTSAVE, RENAMENX, MOVE, LLEN, SADD, SREM, SISCAMLER SCARD.

RESP fixed length string – Bulk String

A fixed length string is used to represent a binary safe string with a maximum length of 512MB (Bulk, which itself has a large volume meaning). The constant length string is encoded as follows:

    (1) The first byte is $.

    (2) Next is the length of bytes that make up the string (called prefixed length, that is, the prefix length), and the prefix length blocks are terminated in CRLF.

    (3) Then is a string that cannot contain CR or LF characters, that is, the number should be converted to a sequence of characters and eventually output to bytes.

    (4) Termination with CRLF.

For example, doge string using fixed-length encoded as follows:

The first byte

Prefix length

String content

CRLF CRLF

Fixed-length string

$ 4 \r\n doge \r\n ===> $4\r\ndoge\r\n

foobar string using fixed-length encoded as follows:

The first byte

Prefix length

String content

CRLF CRLF

Fixed-length string

$ 6 \r\n foobar \r\n ===> $6\r\nfoobar\r\n

It represents an empty string (Empty String, corresponding to the Java “”) when using a fixed-length string is encoded as follows:

The first byte

Prefix length

CRLF CRLF

Fixed-length string

$ 0 \r\n \r\n ===> $0\r\n\r\n

Fixed strings can also use special format represents a Null value refers to a value does not exist. In this particular format, prefix length is -1, and there is no data, so the use of fixed-length string Null values ​​are encoded as follows:

The first byte

Prefix length

CRLF

Fixed-length string

$ -1 \r\n ===> $-1\r\n

When the server returns Redis fixed-length encoded string Null value, the client should not return an empty string, but should return the corresponding programming language Null Object. E.g. Ruby corresponding nil, C language, corresponding to NULL, Java corresponding null, and so on.

RESP array -Array

RESP Redis client uses to send commands to the array Redis server. Similarly, some Redis server command is executed after completion of array types will need to use the RESP elements of the collection back to the client, such as returning a list of elements LRANGE command. RESP and cognitive array of arrays we are not exactly the same, its encoding format is as follows:

    (1) The first byte *.

    (2) followed by a number of array elements RESP (decimal number, but, ultimately, be converted to a sequence of bytes, such as 10 to 1 and 0 need to convert two adjacent bytes), the number of elements in the block CRLF terminated.

    Each element content (. 3) RESP array, each element may be any type of data RESP.

A code RESP empty array as follows:

*0\r\n

A fixed-length string element 2 are as follows content includes encoding RESP array foo and bar:

*2\r\n$3\r\nfoo\r\n$3\r\nbar\r\n

The general format is: * CRLF RESP array as a prefix part, while other types of data elements of the array just RESP one by one in series. E.g. encoding a RESP array elements comprises three types of integers as follows:

*3\r\n:1\r\n:2\r\n:3\r\n

RESP elements of the array are not necessarily the same type of data, it may contain mixed types of elements. The following is an example comprising four elements and an integer type fixed-length string type element (a total of five elements) encoding the RESP array (in order to see more clearly, coded division multiple rows, in fact, can not do that) :

# 元素个数
*5\r\n
# 第1个整型类型的元素
:1\r\n
# 第2个整型类型的元素
:2\r\n
# 第3个整型类型的元素
:3\r\n
# 第4个整型类型的元素
:4\r\n
# 定长字符串类型的元素
$6\r\n
foobar\r\n

Redis server response message first line * 5 \ r \ n defined immediately followed by the 5 response data, respectively, and each reply data item as elements constituting a multi-element transmission for a given length reply (Multi Bulk reply, feeling more difficult to translate, probably means here is that each row is a return item in the entire reply message).

Herein may be analogous to the Java ArrayList (generic erase), somewhat similar to the following pseudocode:

List encode = new ArrayList();
// 添加元素个数
encode.add(elementCount);
encode.add(CRLF);
// 添加第1个整型类型的元素 - 1
encode.add(':');
encode.add(1);
encode.add(CRLF);
// 添加第2个整型类型的元素 - 2
encode.add(':');
encode.add(2);
encode.add(CRLF);
// 添加第3个整型类型的元素 - 3
encode.add(':');
encode.add(3);
encode.add(CRLF);
// 添加第4个整型类型的元素 - 4
encode.add(':');
encode.add(4);
encode.add(CRLF);
// 添加定长字符串类型的元素
encode.add('$');
// 前缀长度
encode.add(6);
// 字符串内容
encode.add("foobar");
encode.add(CRLF);

RESP array concept also exists a Null value, hereinafter referred RESP Null Array. In historical reasons, the array uses RESP Another special encoding format defined Null value, different from the fixed length character string Null value string. For example, when the command execution BLPOP timeout, returns a response RESP Null Array type. RESP Null Array is coded as follows:

*-1\r\n

When Redis server response is RESP Null Array type, the client should return a Null object instead of an empty array or an empty list. This is important, it is a reply to distinguish between an empty array (that is, the command correctly executed, returns the result of normal) or other reasons (such as timeouts BLPOP command, etc.) of the key.

RESP elements of the array may be an array RESP, the following is a type 2 array RESP RESP array element contains, coded as follows (in order to see more clearly, coded division multiple rows, in fact, can not do so):

# 元素个数
*2\r\n
# 第1个RESP数组元素
*3\r\n
:1\r\n
:2\r\n
:3\r\n
# 第2个RESP数组元素
*2\r\n
+Foo\r\n
-Bar\r\n

The above RESP RESP array contains two types of array elements, the array element comprising a RESP integer 3 types of elements, the second array element comprising a RESP simple string type element and an error message types of elements.

RESP array elements Null

RESP single element in the array has the concept of a Null value, hereinafter referred to as Null element. Redis server reply RESP if it is an array type, and there are elements of RESP Null array, it means that the element is lost, must not be replaced by an empty string. The absence of specified key premise, when the option with GET mode, SORT command may happen.

The following is an example of RESP Null array element comprising (in order to see more clearly, coded division multiple rows, in fact, can not do so):

*3\r\n
$3\r\n
foo\r\n
$-1\r\n
$3\r\n
bar\r\n

An array of two elements RESP is Null elements, content client API eventual return should be:

# Ruby
["foo",nil,"bar"]
# Java
["foo",null,"bar"]

RESP other related content

mainly include:

    Example Redis to send commands to the server.

    Batch command and the pipe.

    In-command (Inline Commands).

In fact, there is a document written in C and high performance RESP parser, do not make the translation, because the mastery of the content of the RESP, you can write a parser for any language.

To send commands to the Redis server

If the RESP has been relatively familiar with the serialized format, then write Redis client libraries will become very easy. We can further specify the interaction between the client and the server:

    Redis Redis client sends the server contains only fixed-length string type element array RESP.

    Redis server can use any type of data RESP Redis reply to the client, depending on the specific type of data typically command type.

The following are typical examples of interaction: Redis client sends a command to obtain LLEN mylist mylist KEY as the length, the Redis server will reply integer type, as shown in the following example (C is client, S server), the following pseudocode :

C: *2\r\n
C: $4\r\n
C: LLEN\r\n
C: $6\r\n
C: mylist\r\n

S: :48293\r\n

For simplicity, we use line breaks to separate different portions of the protocol (referred to herein above code branch shown), but the actual interaction Redis client sending * 2 \ r \ n $ 4 \ r \ nLLEN \ r \ n $ 6 \ r \ nmylist \ r \ n is a whole when sent.

Batch command and the pipe

Redis client may send a command batch using the same connection. Redis supports the pipeline characteristics, so Redis client can send multiple commands write once, read and reply to the Redis server on a command before sending the next command do not need. After the batch command to send all replies can be (combined into one reply) in the final. More information can be viewed Using pipelining to speedup Redis queries.

Inline command

In some scenarios, we may be telnet commands can be used, under these conditions, we need to send commands to the Redis server. Although Redis protocol is easy to implement, but it is not ideal in an interactive session and redis-cli in some cases it may not be available. In this kind of reason, Redis command format that is designed specifically for human design, called inline command (Inline Command format.

The following is an example of a server / client using inline chat command (S on behalf of the server, C behalf of the client):

C: PING
S: +PONG

The following is another example of using an inline command returns an integer:

C: EXISTS somekey
S: :0

Basically just write parameters separated by spaces in the telnet session. In addition since no unified protocol request command begins with *, Redis this situation can be detected and parsed command input.

Based on high-performance parser written RESP

Automatic expansion because there is not necessary to switch write mode and so on native byte buffer java.nio.ByteBuffer JDK provided herein incorporated directly ByteBuf Netty Netty and use data provided by the RESP type resolution. Time of this writing (2019-10-09) The latest version of Netty is 4.1.42.Final. The introduction of dependence:


    io.netty
    netty-buffer
    4.1.42.Final

Decoder Interface definitions:

public interface RespDecoder{
    
    V decode(ByteBuf buffer);
}

Define constants:

public class RespConstants {

    public static final Charset ASCII = StandardCharsets.US_ASCII;
    public static final Charset UTF_8 = StandardCharsets.UTF_8;

    public static final byte DOLLAR_BYTE = '$';
    public static final byte ASTERISK_BYTE = '*';
    public static final byte PLUS_BYTE = '+';
    public static final byte MINUS_BYTE = '-';
    public static final byte COLON_BYTE = ':';

    public static final String EMPTY_STRING = "";
    public static final Long ZERO = 0L;
    public static final Long NEGATIVE_ONE = -1L;
    public static final byte CR = (byte) '\r';
    public static final byte LF = (byte) '\n';
    public static final byte[] CRLF = "\r\n".getBytes(ASCII);

    public enum ReplyType {

        SIMPLE_STRING,

        ERROR,

        INTEGER,

        BULK_STRING,

        RESP_ARRAY
    }
}

The following sections parsing module implemented ignore the first byte has been resolved, because the first byte of a specific data type is determined.

Simple string parsing

Single string type is a simple string, which is corresponding to the analysis result of Java String type. Decoder implemented as follows:

// 解析单行字符串
public class LineStringDecoder implements RespDecoder {

    @Override
    public String decode(ByteBuf buffer) {
        return CodecUtils.X.readLine(buffer);
    }
}

public enum CodecUtils {

    X;

    public int findLineEndIndex(ByteBuf buffer) {
        int index = buffer.forEachByte(ByteProcessor.FIND_LF);
        return (index > 0 && buffer.getByte(index - 1) == '\r') ? index : -1;
    }

    public String readLine(ByteBuf buffer) {
        int lineEndIndex = findLineEndIndex(buffer);
        if (lineEndIndex > -1) {
            int lineStartIndex = buffer.readerIndex();
            // 计算字节长度
            int size = lineEndIndex - lineStartIndex - 1;
            byte[] bytes = new byte[size];
            buffer.readBytes(bytes);
            // 重置读游标为\r\n之后的第一个字节
            buffer.readerIndex(lineEndIndex + 1);
            buffer.markReaderIndex();
            return new String(bytes, RespConstants.UTF_8);
        }
        return null;
    }
}

public class RespSimpleStringDecoder extends LineStringDecoder {
    
}

Here extracted LineStringDecoder class for parsing a line string, so that the error message when the parsing can be done once to inherit. have a test:

public static void main(String[] args) throws Exception {
    ByteBuf buffer = ByteBufAllocator.DEFAULT.buffer();
    // +OK\r\n
    buffer.writeBytes("+OK".getBytes(RespConstants.UTF_8));
    buffer.writeBytes(RespConstants.CRLF);
    String value = RespCodec.X.decode(buffer);
    log.info("Decode result:{}", value);
}
// Decode result:OK

Resolving Error Messages

The nature of the error message string is a single line, so that the implementation can be decoded and decoding of simple strings consistent. Error message data type of decoder is as follows:

public class RespErrorDecoder extends LineStringDecoder {

}

have a test:

public static void main(String[] args) throws Exception {
    ByteBuf buffer = ByteBufAllocator.DEFAULT.buffer();
    // -ERR unknown command 'foobar'\r\n
    buffer.writeBytes("-ERR unknown command 'foobar'".getBytes(RespConstants.UTF_8));
    buffer.writeBytes(RespConstants.CRLF);
    String value = RespCodec.X.decode(buffer);
    log.info("Decode result:{}", value);
}
// Decode result:ERR unknown command 'foobar'

Parsing integer numbers

Integer number type, essentially needs to be restored from the long sequence of bytes of 64bit signed integer, because it is signed, the type flag: the first byte after needs to judge whether a negative character – as is from left to right resolve, then every parse out a new position, the current numeric value to be multiplied by 10. Which realize the decoder as follows:

public class RespIntegerDecoder implements RespDecoder {

    @Override
    public Long decode(ByteBuf buffer) {
        int lineEndIndex = CodecUtils.X.findLineEndIndex(buffer);
        // 没有行尾,异常
        if (-1 == lineEndIndex) {
            return null;
        }
        long result = 0L;
        int lineStartIndex = buffer.readerIndex();
        boolean negative = false;
        byte firstByte = buffer.getByte(lineStartIndex);
        // 负数
        if (RespConstants.MINUS_BYTE == firstByte) {
            negative = true;
        } else {
            int digit = firstByte - '0';
            result = result * 10 + digit;
        }
        for (int i = lineStartIndex + 1; i < (lineEndIndex - 1); i++) {
            byte value = buffer.getByte(i);
            int digit = value - '0';
            result = result * 10 + digit;
        }
        if (negative) {
            result = -result;
        }
        // 重置读游标为\r\n之后的第一个字节
        buffer.readerIndex(lineEndIndex + 1);
        return result;
    }
}

Parsing integer number type is relatively complex, must pay attention to a negative determination. have a test:

public static void main(String[] args) throws Exception {
    ByteBuf buffer = ByteBufAllocator.DEFAULT.buffer();
    // :-1000\r\n
    buffer.writeBytes(":-1000".getBytes(RespConstants.UTF_8));
    buffer.writeBytes(RespConstants.CRLF);
    Long value = RespCodec.X.decode(buffer);
    log.info("Decode result:{}", value);
}
// Decode result:-1000

Fixed-length string parsing

The key type fixed-length string is parsed to read the first byte of the sequence block type identifier $ parsed into 64bit signed integer, used to determine the byte length need to be resolved later string content, and then and then read in accordance with the following byte length. Decoder which achieve the following:

public class RespBulkStringDecoder implements RespDecoder {

    @Override
    public String decode(ByteBuf buffer) {
        int lineEndIndex = CodecUtils.X.findLineEndIndex(buffer);
        if (-1 == lineEndIndex) {
            return null;
        }
        // 使用RespIntegerDecoder读取长度
        Long length = (Long) DefaultRespCodec.DECODERS.get(ReplyType.INTEGER).decode(buffer);
        if (null == length) {
            return null;
        }
        // Bulk Null String
        if (RespConstants.NEGATIVE_ONE.equals(length)) {
            return null;
        }
        // Bulk Empty String
        if (RespConstants.ZERO.equals(length)) {
            return RespConstants.EMPTY_STRING;
        }
        // 真实字节内容的长度
        int readLength = (int) length.longValue();
        if (buffer.readableBytes() > readLength) {
            byte[] bytes = new byte[readLength];
            buffer.readBytes(bytes);
            // 重置读游标为\r\n之后的第一个字节
            buffer.readerIndex(buffer.readerIndex() + 2);
            return new String(bytes, RespConstants.UTF_8);
        }
        return null;
    }
}

have a test:

public static void main(String[] args) throws Exception{
    ByteBuf buffer = ByteBufAllocator.DEFAULT.buffer();
    // $6\r\nthrowable\r\n
    buffer = ByteBufAllocator.DEFAULT.buffer();
    buffer.writeBytes("$9".getBytes(RespConstants.UTF_8));
    buffer.writeBytes(RespConstants.CRLF);
    buffer.writeBytes("throwable".getBytes(RespConstants.UTF_8));
    buffer.writeBytes(RespConstants.CRLF);
    String value = RespCodec.X.decode(buffer);
    log.info("Decode result:{}", value);
}
// Decode result:throwable

Array resolve RESP

RESP array type resolution of key:

    The first sequence of bytes to read the type identifier * 64bit block resolved into a signed integer, determining the number of elements in the array.

    Recursive resolution for each element.

Reference lot Redis protocol analysis framework, many of which are implemented by the stack, or state machine, where the first point with a simple recursive decoder code is as follows:

public class RespArrayDecoder implements RespDecoder {

    @Override
    public Object decode(ByteBuf buffer) {
        int lineEndIndex = CodecUtils.X.findLineEndIndex(buffer);
        if (-1 == lineEndIndex) {
            return null;
        }
        // 解析元素个数
        Long length = (Long) DefaultRespCodec.DECODERS.get(ReplyType.INTEGER).decode(buffer);
        if (null == length) {
            return null;
        }
        // Null Array
        if (RespConstants.NEGATIVE_ONE.equals(length)) {
            return null;
        }
        // Array Empty List
        if (RespConstants.ZERO.equals(length)) {
            return Lists.newArrayList();
        }
        List result = Lists.newArrayListWithCapacity((int) length.longValue());
        // 递归
        for (int i = 0; i < length; i++) {
            result.add(DefaultRespCodec.X.decode(buffer));
        }
        return result;
    }
}

have a test:

public static void main(String[] args) throws Exception {
    ByteBuf buffer = ByteBufAllocator.DEFAULT.buffer();
    //*2\r\n$3\r\nfoo\r\n$3\r\nbar\r\n
    buffer = ByteBufAllocator.DEFAULT.buffer();
    buffer.writeBytes("*2".getBytes(RespConstants.UTF_8));
    buffer.writeBytes(RespConstants.CRLF);
    buffer.writeBytes("$3".getBytes(RespConstants.UTF_8));
    buffer.writeBytes(RespConstants.CRLF);
    buffer.writeBytes("foo".getBytes(RespConstants.UTF_8));
    buffer.writeBytes(RespConstants.CRLF);
    buffer.writeBytes("$3".getBytes(RespConstants.UTF_8));
    buffer.writeBytes(RespConstants.CRLF);
    buffer.writeBytes("bar".getBytes(RespConstants.UTF_8));
    buffer.writeBytes(RespConstants.CRLF);
    List value = RespCodec.X.decode(buffer);
    log.info("Decode result:{}", value);
}
// Decode result:[foo, bar]

summary

There are relatively deep understanding of the content and process of RESP after its codec, you can write Netty based codec module Redis service as an entry Netty very meaningful example. Finally, one of only demonstrates the decoding section, the encoding module RESP and more details will be displayed in another article by Netty achieve Redis client article.

References:

  • Redis Protocol specification

link

I hope you can read this, then I found:

  • Github Page:http://www.throwable.club/2019/10/09/redis-serialization-protocol-decode-guide
  • Coding Page:http://throwable.coding.me/2019/10/09/redis-serialization-protocol-decode-guide

appendix

This article covered all of the code:

public class RespConstants {

    public static final Charset ASCII = StandardCharsets.US_ASCII;
    public static final Charset UTF_8 = StandardCharsets.UTF_8;

    public static final byte DOLLAR_BYTE = '$';
    public static final byte ASTERISK_BYTE = '*';
    public static final byte PLUS_BYTE = '+';
    public static final byte MINUS_BYTE = '-';
    public static final byte COLON_BYTE = ':';

    public static final String EMPTY_STRING = "";
    public static final Long ZERO = 0L;
    public static final Long NEGATIVE_ONE = -1L;
    public static final byte CR = (byte) '\r';
    public static final byte LF = (byte) '\n';
    public static final byte[] CRLF = "\r\n".getBytes(ASCII);

    public enum ReplyType {

        SIMPLE_STRING,

        ERROR,

        INTEGER,

        BULK_STRING,

        RESP_ARRAY
    }
}

public enum CodecUtils {

    X;

    public int findLineEndIndex(ByteBuf buffer) {
        int index = buffer.forEachByte(ByteProcessor.FIND_LF);
        return (index > 0 && buffer.getByte(index - 1) == '\r') ? index : -1;
    }

    public String readLine(ByteBuf buffer) {
        int lineEndIndex = findLineEndIndex(buffer);
        if (lineEndIndex > -1) {
            int lineStartIndex = buffer.readerIndex();
            // 计算字节长度
            int size = lineEndIndex - lineStartIndex - 1;
            byte[] bytes = new byte[size];
            buffer.readBytes(bytes);
            // 重置读游标为\r\n之后的第一个字节
            buffer.readerIndex(lineEndIndex + 1);
            buffer.markReaderIndex();
            return new String(bytes, RespConstants.UTF_8);
        }
        return null;
    }
}

public interface RespCodec {

    RespCodec X = DefaultRespCodec.X;

     OUT decode(ByteBuf buffer);

     ByteBuf encode(IN in);
}

public enum DefaultRespCodec implements RespCodec {

    X;

    static final Map DECODERS = Maps.newConcurrentMap();
    private static final RespDecoder DEFAULT_DECODER = new DefaultRespDecoder();

    static {
        DECODERS.put(ReplyType.SIMPLE_STRING, new RespSimpleStringDecoder());
        DECODERS.put(ReplyType.ERROR, new RespErrorDecoder());
        DECODERS.put(ReplyType.INTEGER, new RespIntegerDecoder());
        DECODERS.put(ReplyType.BULK_STRING, new RespBulkStringDecoder());
        DECODERS.put(ReplyType.RESP_ARRAY, new RespArrayDecoder());
    }

    @SuppressWarnings("unchecked")
    @Override
    public  OUT decode(ByteBuf buffer) {
        return (OUT) DECODERS.getOrDefault(determineReplyType(buffer), DEFAULT_DECODER).decode(buffer);
    }

    private ReplyType determineReplyType(ByteBuf buffer) {
        byte firstByte = buffer.readByte();
        ReplyType replyType;
        switch (firstByte) {
            case RespConstants.PLUS_BYTE:
                replyType = ReplyType.SIMPLE_STRING;
                break;
            case RespConstants.MINUS_BYTE:
                replyType = ReplyType.ERROR;
                break;
            case RespConstants.COLON_BYTE:
                replyType = ReplyType.INTEGER;
                break;
            case RespConstants.DOLLAR_BYTE:
                replyType = ReplyType.BULK_STRING;
                break;
            case RespConstants.ASTERISK_BYTE:
                replyType = ReplyType.RESP_ARRAY;
                break;
            default: {
                throw new IllegalArgumentException("first byte:" + firstByte);
            }
        }
        return replyType;
    }

    @Override
    public  ByteBuf encode(IN in) {
        // TODO
        throw new UnsupportedOperationException("encode");
    }
}

public interface RespDecoder {

    V decode(ByteBuf buffer);
}

public class DefaultRespDecoder implements RespDecoder {

    @Override
    public Object decode(ByteBuf buffer) {
        throw new IllegalStateException("decoder");
    }
}

public class LineStringDecoder implements RespDecoder {

    @Override
    public String decode(ByteBuf buffer) {
        return CodecUtils.X.readLine(buffer);
    }
}

public class RespSimpleStringDecoder extends LineStringDecoder {

}

public class RespErrorDecoder extends LineStringDecoder {

}

public class RespIntegerDecoder implements RespDecoder {

    @Override
    public Long decode(ByteBuf buffer) {
        int lineEndIndex = CodecUtils.X.findLineEndIndex(buffer);
        // 没有行尾,异常
        if (-1 == lineEndIndex) {
            return null;
        }
        long result = 0L;
        int lineStartIndex = buffer.readerIndex();
        boolean negative = false;
        byte firstByte = buffer.getByte(lineStartIndex);
        // 负数
        if (RespConstants.MINUS_BYTE == firstByte) {
            negative = true;
        } else {
            int digit = firstByte - '0';
            result = result * 10 + digit;
        }
        for (int i = lineStartIndex + 1; i < (lineEndIndex - 1); i++) {
            byte value = buffer.getByte(i);
            int digit = value - '0';
            result = result * 10 + digit;
        }
        if (negative) {
            result = -result;
        }
        // 重置读游标为\r\n之后的第一个字节
        buffer.readerIndex(lineEndIndex + 1);
        return result;
    }
}

public class RespBulkStringDecoder implements RespDecoder {

    @Override
    public String decode(ByteBuf buffer) {
        int lineEndIndex = CodecUtils.X.findLineEndIndex(buffer);
        if (-1 == lineEndIndex) {
            return null;
        }
        Long length = (Long) DefaultRespCodec.DECODERS.get(ReplyType.INTEGER).decode(buffer);
        if (null == length) {
            return null;
        }
        // Bulk Null String
        if (RespConstants.NEGATIVE_ONE.equals(length)) {
            return null;
        }
        // Bulk Empty String
        if (RespConstants.ZERO.equals(length)) {
            return RespConstants.EMPTY_STRING;
        }
        // 真实字节内容的长度
        int readLength = (int) length.longValue();
        if (buffer.readableBytes() > readLength) {
            byte[] bytes = new byte[readLength];
            buffer.readBytes(bytes);
            // 重置读游标为\r\n之后的第一个字节
            buffer.readerIndex(buffer.readerIndex() + 2);
            return new String(bytes, RespConstants.UTF_8);
        }
        return null;
    }
}

public class RespArrayDecoder implements RespDecoder {

    @Override
    public Object decode(ByteBuf buffer) {
        int lineEndIndex = CodecUtils.X.findLineEndIndex(buffer);
        if (-1 == lineEndIndex) {
            return null;
        }
        // 解析元素个数
        Long length = (Long) DefaultRespCodec.DECODERS.get(ReplyType.INTEGER).decode(buffer);
        if (null == length) {
            return null;
        }
        // Null Array
        if (RespConstants.NEGATIVE_ONE.equals(length)) {
            return null;
        }
        // Array Empty List
        if (RespConstants.ZERO.equals(length)) {
            return Lists.newArrayList();
        }
        List result = Lists.newArrayListWithCapacity((int) length.longValue());
        // 递归
        for (int i = 0; i < length; i++) {
            result.add(DefaultRespCodec.X.decode(buffer));
        }
        return result;
    }
}

(Herein End e-a-20191009 c-2-d)

Leave a Reply