1.

What is a producer in Kafka? What are the different types of Kafka producer APIs? How does Kafka producer write data to a topic containing multiple partitions?

Answer»

A producer publishes messages to one or more Kafka topics. The message contains information related to what topic and partition should the message be published to.

There are three DIFFERENT types of producer APIs –

  1. Fire and forget – The simplest approach, it involves calling send() METHOD of producer API to send the message to the key. In this case, the application doesn’t care whether the message is successfully received by the broker or not.
  2. Synchronous producer – In this method, the calling application waits until it gets a response. In the case of success, we get a RecordMetadata object, and in the event of failure, we get an exception. However, note that this will limit your throughput because you are waiting for every message to get acknowledged.
  3. Asynchronous producer – A better and faster way of SENDING messages to Kafka, this involves providing a callback function to RECEIVE the acknowledgment. The application doesn’t WAIT for success/failure and the callback function is invoked when the message is successfully acknowledged or in case of a failure.

Kafka messages are key-value pairs. The key is used for partitioning messages being sent to the topic. When writing a message to a topic, the producer has an option to provide the message key. This key determines which partition of the topic the message goes to. If the key is not specified, then the messages are sent to partitions of the topic in round robin fashion.

Note that Kafka orders messages only inside a partition, hence choosing the right partition key is an important factor in application design.



Discussion

No Comment Found