How do we achieve FIFO behaviour in Kafka?

1.	How do we achieve FIFO behaviour in Kafka?
Answer» Kafka stores messages in topics which in turn gets stored in different partitions. The partition is an immutable sequence of ordered messages which is CONTINUOUSLY appended to. A message is uniquely identified in the partition by a sequential number called offset. The FIFO BEHAVIOUR can only be achieved inside the partitions. We can achieve FIFO behaviour by following the steps below : 1: We need to first SET enable auto-commit PROPERTY false : Set enable.auto.commit=false 2: Once messages get processed, we should not make a CALL to consumer.commitSync(); 3: We can then call to “subscribe” and ensure the registry of consumer system to the topic. 4: The Listener consumerRebalance should be implemented and within a listener, we should call consumer.seek(topicPartition, offset). 5: Once the message gets processed, the offset associated with the message should be stored along with the processed message. 6: We should also ensure idempotent as a safety measure.

Answer»

Kafka stores messages in topics which in turn gets stored in different partitions. The partition is an immutable sequence of ordered messages which is CONTINUOUSLY appended to. A message is uniquely identified in the partition by a sequential number called offset. The FIFO BEHAVIOUR can only be achieved inside the partitions. We can achieve FIFO behaviour by following the steps below :

1: We need to first SET enable auto-commit PROPERTY false :

Set enable.auto.commit=false

2: Once messages get processed, we should not make a CALL to consumer.commitSync();

3: We can then call to “subscribe” and ensure the registry of consumer system to the topic.

4: The Listener consumerRebalance should be implemented and within a listener, we should call consumer.seek(topicPartition, offset).

5: Once the message gets processed, the offset associated with the message should be stored along with the processed message.

6: We should also ensure idempotent as a safety measure.

How do we achieve FIFO behaviour in Kafka?

Discussion

No Comment Found

Related InterviewSolutions

Reply to Comment