InterviewSolution
| 1. |
What is an offset in Kafka? |
|
Answer» One of the critical information which is required for any producer/CONSUMER system to write /read messages in different partition. It is a unique identifier associated with each message in different partitions. This is a sequence number which goes on increasing as messages come in the partition. The ZooKeeper system keeps track of an offset associated with any specific TOPIC stored in a specific partition. There is a class ConsumerRecord having method offset which helps CONSUMERS to get offset details related to a specific topic in a given partition. Once consumer systems get to know about the offset, it helps to identify messages in topics. Kafka stores the messages published for some time depending upon configuration. Let's assume if the retention period is defined as 4 days then messages will be stored in Kafka for 4 days irrespective of messages read or not by consumer systems. The memory space will be freed up only after 4 days. The offset is very important for record and it is maintained by every customer reading the record. Once it reads the messages, it increases the offset linearly. But a consumer can read the messages in any order. It can go to some old offset or MOVE to new offset as per the requirement. Let's have a look at the below diagram which will give more clarity on offset CONCEPT:
|
|