1.

How do we send large messages with Kafka?

Answer»

Kafka system by default does not handle the large size of data. The data max size is 1 MB but there are ways to increase that size. We should also ensure to increase the network buffers as well for our consumers and producers system. We need to adjust a few PROPERTIES to achieve the same :

  1. The consumer system which read messages from a topic can read the largest size of a message-driven by property fetch.message.max.bytes. If we would LIKE the consumer system to read large size of data accordingly we can set the property.
  2. The Kafka broker has also one property replica .fetch.max.bytes which drive message sizes replicated across clusters. For the message to be replicated in the correct size, we need to make SURE we have a size not defined too small for this property otherwise messages will be not be committed successfully resulting in non-availability of messages for consumer systems.
  3. There is another property on broker side message.max.bytes which determines the maximum size of data that Kafka broker can receive from the producer system.
  4. The broker side has max .message.bytes property which validates the maximum size of message one can append to the topic. The size is pre-compression in size. This size applies to the topic only.

Now it is clear that if we would like to SEND large Kafka messages then it can be easily achieved by tweaking few properties explained above. The broker related config can be FOUND at $KAFKA_HOME/config/server.properties while consumer-related config found at $KAFKA_HOME/config/consumer.properties



Discussion

No Comment Found