1.

what is the consumer group in Kafka?

Answer»

Let's first understand the concept of the consumer in Kafka architecture. The consumers are the system or process which subscribe to topics created at the Kafka BROKER. The producer's system sends messages to topics and once messages are committed successfully then only subscribers systems are allowed to read the messages. The consumer group is tagging of consumers system in such a way to make it multi-threaded or multi-machine system. 

As we can see in the above diagram, two consumers 1 & 2 are being tagged in the same group. Also, we can see that individual customers reading DATA from different partition of topics. Some common characteristic of consumer groups are as follows:

  1. Consumers system can join the consumer group by having the same group.id.
  2. The consumer group supports multiple processing at the same time by endorsing parallelism, one can have a maximum number of consumers similar to several partitions. So each partition gets mapped to one INSTANCE of the consumer from the same group.
  3. The consumer is assigned to a single partition of the topic by Kafka broker to ensure only particular consumer consumes messages belonging to that partition.
  4. It also ensures that messages are read from a single consumer only.
  5. Messages are ordered in Kafka and it appears in the same order they are committed.

The recommendation for the consumer group suggests having a similar number of consumer instances in line with several partitions. In case if we will go with a greater number of consumers then it will result in excess customers sitting idle resulting in wasting resources. In the case of partitions numbers greater then it will result in the same consumers reading from more than one partition. This should not be an issue until the time ordering of messages is not IMPORTANT for the use case. Kafka does not have inbuilt SUPPORT for the ordering of messages across different partitions.

This is the reason why Kafka recommends to have the same number of consumers in line with partitions to maintain the ordering of messages.



Discussion

No Comment Found