How can Kafka producer maintain exactly once semantics?

1.	How can Kafka producer maintain exactly once semantics?
Answer» With Kafka messaging system, three DIFFERENT types of semantics can be achieved. At max once: Wherein a messaging system will never duplicate a MESSAGE but might miss out on some messages occasionally. At least once: Wherein a messaging system will never miss a message but might duplicate some messages occasionally. Exactly once: Where in it will deliver all the messages without any duplication. Kafka transactions help achieve exactly once semantic between Kafka brokers and clients. In order to achieve this we need to set below properties at producer end – enable.idempotence=true and transactional.ID=<some unique id>. We also need to call initTransaction to PREPARE the producer to use transactions. With these properties set, if the producer (characterized by producer id> accidentally sends the same message to Kafka more than once, then Kafka BROKER detects and de-duplicates it.

Answer»

With Kafka messaging system, three DIFFERENT types of semantics can be achieved.

At max once: Wherein a messaging system will never duplicate a MESSAGE but might miss out on some messages occasionally.
At least once: Wherein a messaging system will never miss a message but might duplicate some messages occasionally.
Exactly once: Where in it will deliver all the messages without any duplication.

Kafka transactions help achieve exactly once semantic between Kafka brokers and clients. In order to achieve this we need to set below properties at producer end – enable.idempotence=true and transactional.ID=<some unique id>. We also need to call initTransaction to PREPARE the producer to use transactions. With these properties set, if the producer (characterized by producer id> accidentally sends the same message to Kafka more than once, then Kafka BROKER detects and de-duplicates it.

How can Kafka producer maintain exactly once semantics?

Discussion

No Comment Found

Related InterviewSolutions

Reply to Comment