Explore topic-wise InterviewSolutions in .

The offset is a UNIQUE identifier of a RECORD within a partition. It denotes the position of the consumer in the partition. Consumers can read messages starting from a specific offset and can read from any offset point they choose.

• Partition offset has a unique sequence id CALLED as offset.
• Each partition should have a partition offset. 

TOPIC can also have MULTIPLE partition logs like the click-topic has in the image to the right. This allows for multiple consumers to read from a topic in parallel.

KAFKA topic is shared into the partitions, which contains messages in an unmodifiable sequence.

• Partition is a logical grouping of data.
• Partitions allow you to parallelize a topic by splitting the data in a topic across multiple brokers.  
• There are ONE or more than one partition can GROUP in topic.
• Partition allow to parallelize the topic by splitting a data in a multiple topic across the multiple cluster. 
• Each partition has an identifier called offset.
• Each partition can be placed on a separate machine to allow for multiple CONSUMER to read the topic PARALLEL. 

Topic is a logical feed name to which RECORDS are published. Topics in KAFKA supports multi-subscriber model, so that topic can have zero, one, or many consumers that subscribe to the data written to it.

• Topic is a specific category which keep the stream of messages.
• Topic split into partition.
• For each Kafka, there at least one partition should be there.
• Each partition contains message or payload in a non-modified ordered sequence. 
• Each message with in a partition has an identifier, which is called as a offset.
• A topic has a name, and it MUST be unique across the cluster. 
• PRODUCER need topic to publish the payload. 
• Consumer pulled the same payload from the consumer. 
• For every Topic the cluster maintain the log LOOK like below.

Every partition has an ordered and immutable sequence of records which is continuously appended to—a structured commit log. The Kafka cluster durably persists all published records—whether they have been consumed—using a configurable retention period.

KAFKA PROCESS DIAGRAM comprises the below essential COMPONENT which is require to setup the messaging INFRASTRUCTURE. 

• Topic
• Broker
• Zookeeper
• Partition 
• Producer
• Consume

Communication between the clients and the servers is done with a simple, high-performance, language agnostic TCP protocol. This protocol is versioned and maintains backwards compatibility with older version

Let’s TALK about some modern source of data now a days which is a data—transactional data such as orders, inventory, and shopping carts — is being augmented with things such as clicking, likes, recommendations and searches on a web page. All this data is deeply important to analyze the consumers behaviors, and it can feed a set of predictive ANALYTICS engines that can be the differentiator for companies. 

• Support low latency message delivery.
• Handling the real time traffic.
• Assurance for fault tolerant.
• Easy to integrate with SPARK application to process a high volume of messaging data.
• Has an ability to create a cluster of messaging container which monitor and supervise by coordination server like Zookeeper.

So, when we need to handle this KIND of volume of data, we need Kafka to solve this PROBLEM. 

There is PLETHORA of use case, where Kafka fit into the REAL work application, however I listed below are the real work use case which is frequently using.

• Metrics: Use for monitoring operation data, which can use for analysis or doing statistical operation on gather the data from distributed system 
• Log Aggregation solution: can be used across an organization to collect logs from multiple services, which consume by consumer services to perform the analytical operation.
• Stream Processing: Kafka’s strong durability is also very useful in the context of stream processing.
• Asynchronous communication: In microservices, keeping this HUGE system synchronous is not desirable, because it can render the entire application unresponsive. Also, it can defeat the whole purpose of dividing into microservices in the first place. Hence, having Kafka at that time makes the whole data flow easier. Because it is distributed, highly fault-tolerant and it has constant monitoring of broker nodes through services like Zookeeper. So, it makes it efficient to work.
• CHAT bots: Chat bots is one of the POPULAR use cases when we require reliable messaging services for a smooth delivery. 
• Multi-tenant solution. Multi-tenancy is enabled by configuring which topics can produce or consume data. There are also operations support for quotas

Above are the use case where predominately require a Kafka framework, apart from that there are other cases which depends upon the requirement and design.

Now a days kafka is a key messaging framework, not because of its features EVEN for reliable transmission of messages from sender to RECEIVER, however, below are the key POINTS which should consider.

• Reliability − Kafka provides a reliable delivery from publisher to a subscriber with zero message loss..
• Scalability −Kafka achieve this ability by using clustering ALONG with the zookeeper coordination server
• Durability −By using distributed log, the messages can persist on disk.
• Performance − Kafka provides high throughput and low latency across the publish and subscribe APPLICATION. 

Considering the above features Kafka is one of the best options to use in Bigdata Technologies to handle the large volume of messages for a smooth delivery.

Kafka provide a RELIABLE delivery for messages from sender to RECEIVER apart from that it has other key features as well.

• Kafka is designed for achieving high throughput and fault TOLERANT messaging services.
• Kafka provides build in patriation called as a Topic.
• Also provide the feature of replication.
• Kafka provides a queue, which can handle the high volume of data and eventually transfer the MESSAGE from one sender to receiver. 
• Kafka also persisted the message in the disk along with has ability to replicate the messages across the cluster
• Kafka work with zookeeper for coordination and synchronization with other services.
• Kafka has GOOD inbuilt support Apache Spark.

To utilize all this key feature, we need to configure the Kafka cluster properly along with the zookeeper configuration.

Kafka is a messaging framework developed by apache foundation, which is to create the create the messaging SYSTEM along with can provide fault tolerant cluster along with the low latency system, to ensure end to end delivery.

Below are the bullet POINTS:

• Kafka is a messaging system, which has provided fault tolerant capability to prevent the MESSAGE loss. 
• Design on public-subscribe model.
• Kafka cab support both Java and Scala.
• Kafka was originated at LINKEDIN and later became an open sourced Apache project in 2011
• Work seamlessly with spark and other big data technology.
• Support cluster mode operation
• Kafka messaging system can be use for web service architecture or big data architecture. 
• Kafka ease to code and configure as compare to other messaging framework.

      Kafka required other component such as the zookeeper to create a cluster and act as a coordination server