The terms and application point of view of function kafka

kafka term (Terminology)

Topic and Consumer Group

Topic Kafka published to each cluster message has a category, the category is called Topic. (Topic different physically separate storage of messages, the message is logically a Topic although stored on one or more of the message broker, but the user to specify a Topic to production or consumption data without concern for where the data are stored in)

For apps, producers must specify a topic topic to publish messages. to determine where it was sent.

Consumer Group consumer group, each belonging to a specific Consumer Consumer Group (group name can be specified for each Consumer, if the group name specified in the Default group).

This concept is the corresponding and Topic.

A topic can have multiple groups of consumer spending.

Generally a service to create a consumer group.

As shown below

Using the above two can be represented this way:


Producer and Consumer

Producer responsible for issuing messages to Kafka broker

From the application point of view, is to write code with factory mode resume producer, basically guarantee that a service is a producer to send data to the queue.

In general, a server will play a producer. But not necessarily, a great God to write an infinite loop until the establishment of a bunch of producers out of memory also can be achieved.

Consumer news consumers, read a message to Kafka broker clients.

And producers corresponds to a server from a consumer. Similarly, the write cycle of death until the establishment of a bunch of consumers out of memory also can be achieved.


Partitions and Broker

Partitions physically, each Topic contains one or more Partitions. This service is designed so that the data is too big, and it is divided into small pieces. All the small pieces together to complete a function, here is a topic.

A broker cluster consists of one or more servers, which are called brokers.

For applications, producers send out consumption, they don’t care. Consumers slow down to consume at their own rate. This time there may be a lot of news generated, consumer pressure is still within a certain range. What is decoupling between producers and consumers is a caching service broker.

Above with a view showing as follows:

Application Scenarios

Basic Application

  1. For decoupling the production side and consumption side. For example, two teams need to share the same data, but the data can only be stored by the party. The other required data can consume the message as a message by a consumer.

  2. To increase the degree of concurrency. Http call a complex internal logic. This call can be divided into two stages of processing. The first phase calls check legitimacy. The check results and query number returned to the client. This stage is simple, can support a high degree of concurrency. If the verification message sent valid execution stage of the processing. Because MQ message consumption rate is relatively constant, not overwhelm services. The client can through a single query results.

Advanced Applications

Dead Letter

If the message queue of the message set the delay consumption, then the message will be blocked until the normal news behind this message is consumed. The solution is to use another queue to store these messages will be blocked. This is the dead letter queue.

Leave a Reply