Apache Kafka Tutorial

Key Takeaways

• Kafka stores replayable records in topics; consuming a record doesn't delete it.
• Ordering is per partition, not across an entire topic. Record keys influence partition selection.
• Consumer groups divide partitions among consumers, while committed offsets provide restart checkpoints.
• Kafka 4.x uses KRaft, not ZooKeeper. You'll format storage before starting your local server.
• A single broker with replication factor 1 is suitable for development or testing, not production.

Apache Kafka is a distributed event-streaming platform: producers write records to topic partitions, and consumers read them independently. This Apache Kafka tutorial explains that architecture, then shows you how to start Kafka locally, create a topic, publish a message, and consume and replay it.

What Is Kafka?

Think of Kafka as a durable event log rather than a mailbox that empties when someone reads it. An event, also called a record or message, describes something that happened: an order was placed, a payment completed, or a sensor reported a measurement.

A record contains a value and can include a key, timestamp, and headers. Different applications can read the same retained records independently, without the producer knowing which consumers exist.

For the messaging comparison, see JMS vs Kafka. If you're evaluating whether you need an event log at all, start with when to use Kafka.

Kafka Architecture

The basic flow is producer → topic partition → consumer. Brokers store the records; controllers coordinate cluster metadata and leadership.

Kafka Cluster

A cluster contains Kafka servers working together. KRaft controllers manage metadata, while brokers store and serve records; a development server can perform both roles.

Topic

A topic is a named stream, such as orders, divided into one or more partitions.

Partition

A partition is an ordered log. Each record has an offset identifying its position within that partition, not within the whole topic.

Broker

A broker stores assigned partition replicas and handles client requests. Partition placement and leader distribution affect load; having equal partition counts doesn't guarantee equal traffic.

Producer

A producer publishes records to topics. It selects a partition and sends each record to that partition's leader.

Consumer

A consumer reads records and tracks its position separately for each partition. Consumers can cooperate through a consumer group.

Replica

A replica is a copy of a partition. The replication factor counts all copies, including the leader; it is separate from the partition count.

Leader

The leader handles writes for its partition. Consumers normally read from leaders, although Kafka also supports configured follower reads.

Follower

A follower copies the leader's log and can lag behind. The in-sync replica set, or ISR, tracks replicas meeting Kafka's liveness and replication-lag requirements. These relationships are covered in Kafka's replication design documentation.

This is one possible multi-broker layout, not the single-node environment you'll run below:

Producer --key--> topic: orders
                 3 partitions, replication factor 2

             Broker A     Broker B     Broker C
Partition 0  leader       follower     -
Partition 1  -            leader       follower
Partition 2  follower     -            leader

Consumer group "billing":
  Consumer 1 <-- partitions 0 and 2
  Consumer 2 <-- partition 1

For a deeper walkthrough, see Kafka architecture explained.

Replication and Failure Guarantees

Failover requires an eligible replacement, not just any follower. Kafka prefers an in-sync replica; its Eligible Leader Replicas mechanism can also track replicas outside the current ISR that are safe to elect. Unclean leader election can sacrifice data to restore availability. With replication factor 1, there's no follower to promote.

Producer acknowledgments matter too. acks=all waits for all current in-sync replicas, not every configured replica. A common durability configuration combines replication factor 3, min.insync.replicas=2, and acks=all; writes using those acknowledgments fail if there aren't enough in-sync replicas. These settings reduce specific failure risks, not every possible cause of data loss.

Kafka Topics, Partitions, and Consumer Groups

Keys Determine Placement; Partitions Determine Ordering

When you don't explicitly select a partition, the default Java producer logic hashes a supplied key to choose one. With unchanged partition count and partitioning configuration, the same serialized key maps to the same partition. Without a key, the producer uses sticky, batching-oriented partition selection—not a promise to alternate every record evenly.

Three messages sent to a three-partition topic therefore need not occupy three different partitions. Kafka preserves the log order within each partition, but provides no total order across partitions. Changing the partition count can change where a key maps.

Consumer Groups Share Work

Consumers sharing a group.id form a consumer group. With group-managed subscriptions, a partition is assigned to at most one member of that group at a time; one member can handle multiple partitions. For a single three-partition topic, a fourth consumer may be idle. Another group reads the topic independently.

An offset identifies a record within a partition. The consumer's current position identifies what it will read next, while a committed offset is the saved checkpoint used after a restart. Membership changes can trigger partition reassignment, called a rebalance.

Retention Makes Replay Possible

Reading or committing an offset doesn't remove records. With the delete cleanup policy, Kafka removes old log segments according to time or size limits; compacted topics instead retain the latest value for each key, subject to compaction rules. Replay can only read history that is still retained. See the topic retention and durability settings.

Run Kafka Locally with KRaft

1. Install the Prerequisites and Download Kafka

This walkthrough targets Apache Kafka 4.3.1 and Java 25, which Kafka 4.3 supports. Use Bash for the shell commands, and open three terminals in the extracted Kafka directory as you work through the steps.

Install Java first, then check the runtime:

java --version

Download kafka_2.13-4.3.1.tgz, the binary archive rather than the source archive, from the official Kafka downloads page. From the directory containing the download, run:

tar -xzf kafka_2.13-4.3.1.tgz
cd kafka_2.13-4.3.1

2. Format Storage and Start Kafka

The supplied config/server.properties runs a combined broker/controller. Before starting it, replace its existing listeners line with the following loopback-only endpoints:

# Replace the existing listeners setting in config/server.properties.
listeners=PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093

These listeners are unencrypted and unauthenticated. Binding them to localhost limits network exposure; it does not add authentication.

Check log.dirs in that file. The supplied configuration uses /tmp/kraft-combined-logs; reserve it for this disposable exercise, or choose a new empty directory if you've used it for another cluster.

In terminal 1, generate a cluster ID, initialize the storage, and start Kafka. This follows the official KRaft quickstart:

KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"

bin/kafka-storage.sh format --standalone \
  -t "$KAFKA_CLUSTER_ID" \
  -c config/server.properties

bin/kafka-server-start.sh config/server.properties

Keep terminal 1 running. Formatting initializes a new cluster's storage; it is not a routine restart step. To restart this environment later, run only the server-start command with the same configuration and storage.

3. Create a Kafka Topic

In terminal 2, from the extracted Kafka directory, create the original tutorial's sample topic:

bin/kafka-topics.sh --create \
  --bootstrap-server localhost:9092 \
  --topic sample \
  --partitions 1 \
  --replication-factor 1

Inspect it before sending data:

bin/kafka-topics.sh --describe \
  --bootstrap-server localhost:9092 \
  --topic sample

The description should report one partition and replication factor 1. Those are separate choices: the partition count controls how the log is divided; the replication factor controls its number of copies.

4. Publish a Message

In terminal 2, start the console producer:

bin/kafka-console-producer.sh \
  --bootstrap-server localhost:9092 \
  --topic sample

Once the producer is running, type the following at its prompt and press Enter. This is producer input, not another shell command:

Hello Kafka!

Each input line becomes a separate record. Leave the producer running so you can send more messages.

5. Consume and Replay the Message

In terminal 3, start a consumer with a new group name:

bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic sample \
  --group sample-reader-v1 \
  --from-beginning

Expected record output:

Hello Kafka!

Type another line in the producer terminal to see it arrive in the consumer terminal.

--from-beginning selects the earliest retained offset when the consumer has no valid saved offset. It does not override an existing group's valid committed offsets. If you restart sample-reader-v1, it can resume from its checkpoint rather than replay everything.

Stop the consumer with Ctrl+C, then use another unused group name to replay the retained records:

bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic sample \
  --group sample-replay-v1 \
  --from-beginning

If two consumers run concurrently in the same group against this one-partition topic, only one can receive its partition assignment.

6. Try Keyed Records Across Three Partitions

Stop the interactive producer with Ctrl+C. In terminal 2, create a separate topic and pipe three keyed records into a producer:

bin/kafka-topics.sh --create \
  --bootstrap-server localhost:9092 \
  --topic orders-demo \
  --partitions 3 \
  --replication-factor 1

printf '%s\n' 'order-42:created' 'order-17:created' 'order-42:paid' |
  bin/kafka-console-producer.sh \
    --bootstrap-server localhost:9092 \
    --topic orders-demo \
    --reader-property parse.key=true \
    --reader-property key.separator=:

Here, the text before the colon is the key; the remainder is the value. Stop the previous consumer, then inspect keys, partitions, and offsets:

bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic orders-demo \
  --group orders-reader-v1 \
  --from-beginning \
  --formatter-property print.key=true \
  --formatter-property print.partition=true \
  --formatter-property print.offset=true

The two order-42 records should share a partition, with created before paid. Don't expect a particular partition number or an overall order relative to order-17; different keys can also land in the same partition. The reader properties configure record construction: parse.key=true and key.separator=: split each input line at the first colon into a key and value. Without key parsing, these input lines would become entire values with null keys. The formatter properties only change the displayed output, not the stored records.

These examples use --reader-property and --formatter-property, rather than their deprecated --property alternatives.

7. Stop the Local Environment

Stop any running console clients with Ctrl+C, then stop the server in terminal 1 with Ctrl+C. Keep the storage if you want to resume later; don't delete directories belonging to another Kafka environment.

What You Built

You've started a KRaft server, created topics, published records, and consumed retained history. The important boundaries are now visible: ordering belongs to partitions, progress belongs to consumer groups, retention controls replay, and replication settings determine which failures the system can tolerate.

To move from console commands into application code, continue with the Spring Boot Kafka consumer tutorial.

Before You Use Kafka in Production

• Separate availability from convenience. Use multiple brokers and replicated topics. Plan a redundant controller quorum; the KRaft deployment guidance advises against combined roles for critical deployments.
• Configure security explicitly. TLS encryption, authentication, and authorization require configuration. Follow the Kafka security documentation; localhost plaintext examples are not production security settings.
• Monitor failure and backlog signals. Track consumer lag, under-replicated partitions, disk capacity, and request failures. The Kafka monitoring reference explains the available metrics.
• Plan capacity and recovery together. Account for retention, replication, traffic, and consumer recovery time. Verify that consumers can catch up before required history expires.

FAQ

How do I run Apache Kafka locally without ZooKeeper?

Kafka 4.x uses KRaft and does not support ZooKeeper mode. For a new local environment, generate a cluster ID, format storage with the standalone KRaft configuration, and start the server before creating topics.

Does Kafka guarantee message ordering?

Kafka preserves record order within a partition, not across all partitions of a topic. With unchanged partition count and default key-based partitioning, the same serialized key maps to the same partition.

Does a Kafka consumer delete messages after reading them?

No. Reading records or committing offsets does not delete them; topic cleanup and retention policies determine which records remain available for replay.

Does Kafka require Java, and do Kafka clients need to use Java?

The Kafka broker requires a supported Java runtime; this walkthrough uses Java 25. Client applications do not have to use Java: Kafka clients are available for languages including Python, Go, and C/C++.

Is a single broker sufficient for a Kafka setup?

A single broker is sufficient for a disposable development or testing environment, but not for a production system requiring broker-failure redundancy. With replication factor 1, a partition has no follower copy on another broker.

Further Reading

• Kafka quickstart — local startup and console tooling.
• Kafka design — logs, replication, and delivery guarantees.
• Producer configuration — partition selection and acknowledgments.
• KafkaConsumer documentation — consumer groups, positions, and committed offsets.
• Topic configuration — retention, compaction, and minimum in-sync replicas.
• KRaft operations — controller roles and storage initialization.
• Supported Java versions — runtime compatibility for this release.
• Kafka security overview — encryption, authentication, and authorization.