When you should be using Kafka

Key Takeaways

• Use Kafka when retained event history, replay, and independent downstream consumers are requirements—not because your architecture has microservices.
• Conventional consumer groups share partitions internally and read independently across groups. Ordering is per partition, not across a topic.
• Compare Kafka with RabbitMQ and cloud queues by replay, routing, delivery guarantees, ordering, latency, and operational cost.
• Replication and exactly-once processing have configuration and integration requirements; neither is a blanket promise against data loss or duplicate business effects.

Use Kafka when your system needs a durable, replayable event stream that multiple applications can process independently. Its retained log lets downstream services consume live events and revisit available history without requiring producers to publish those events again.

If you only need to hand a task to an available worker, start by evaluating a queue. Kafka can support that workload, but its adoption by large companies isn't a reason to take on its partitioning, retention, and operational responsibilities.

What Is Kafka?

Apache Kafka is an event streaming platform built around partitioned, persistent logs. Producers write records to named topics, brokers store the topics' partitions, and consumers read those records. A topic is a named stream—not a database table with arbitrary queries and row updates.

How Does Kafka Work?

Each partition is an ordered sequence of records. An offset identifies a record's position within its partition; offsets aren't global identifiers across an entire topic. A record contains a key, which may be absent, a value, a timestamp, and optional metadata headers.

For example, an order event might use an order ID as its key and an order-status change as its value. Headers can carry tracing or format metadata. For the underlying components, see our Kafka architecture explanation.

Partition Keys Determine Your Ordering Boundary

Kafka preserves log order within a partition, not across all partitions or by event timestamp. Use a consistent partitioning strategy and an entity key, such as an order ID, when that entity's events must stay together. Your application must also preserve processing order rather than dispatching related records to workers that finish unpredictably.

Adding partitions can change where keyed records are routed, while existing records remain where they were. Treat partition expansion as an ordering-sensitive change, not just a capacity adjustment. Load-test skewed keys as well as evenly distributed traffic.

Replication Is a Configuration, Not an Automatic Guarantee

Operators choose a replication factor and replica placement. A replication factor of 1 provides no replica-based fault tolerance. Place replicas across appropriate failure domains rather than assuming separate processes provide independent protection.

Producer acks=all waits for the in-sync replicas to acknowledge a write. The topic's min.insync.replicas sets the minimum in-sync replica count required for those writes to succeed. A common starting design is replication factor 3, min.insync.replicas=2, and acks=all, but surviving failures still depends on replica health, placement, and recovery policy.

The Key Advantage of Kafka: Independent Consumers and Replay

Scalability and Multiple Subscribers

With conventional consumer groups, each group has its own committed offsets for each topic-partition. A committed offset records where processing should resume; committing it doesn't delete the underlying records. One group's progress doesn't advance another group's position.

Within a group, each partition is assigned to one consumer at a time. Three partitions therefore support at most three consumers with partition assignments for that topic in that group. More members don't add partition-level parallelism. This is load-sharing within a group and fan-out across groups—not broadcasting to every member.

Illustrative topic: orders
Configured replicas are omitted from this view.

P0: [offset 40, key A] -> [offset 41, key A]
P1: [offset 12, key B] -> [offset 13, key C]

Search group:
  worker S1 reads P0
  worker S2 reads P1

Analytics group:
  worker A1 reads P0 and P1

Each group maintains its own position for P0 and P1.

A new application can read available history without changing the existing groups. Configure its starting position explicitly: a new group name alone doesn't guarantee a full replay. The auto.offset.reset setting applies when there is no initial offset or the stored offset is no longer available.

Retention Versus Compaction

Kafka doesn't provide endless storage. With the delete cleanup policy, time- and size-based retention remove old log segments, independently of whether every consumer has finished reading them. Your replay window depends on retained data, storage capacity, and cost.

Compaction removes superseded records while retaining the latest value for each key, with deletion markers handled separately. Cleanup runs in the background, so multiple versions may remain temporarily. Compaction is useful for rebuilding current keyed state; it isn't a promise to preserve every historical event.

Kafka vs RabbitMQ: Compare Semantics, Not Slogans

RabbitMQ exchanges route messages to queues through bindings. Depending on exchange type, routing can use exact routing keys, routing-key patterns, or headers; fanout exchanges copy messages to their bound destinations. Conventional queues support acknowledgments and redelivery, so saying queues cannot deliver the same message twice is wrong.

Kafka, RabbitMQ Queues, and Cloud Queues

These distinctions follow the documented RabbitMQ queue semantics, SQS visibility and deletion model, and AWS's publish-subscribe guidance. SQS standard and FIFO queues have different ordering guarantees; don't treat them as interchangeable.

Push vs Pull

Kafka consumers poll for records and can pause fetching from assigned partitions. RabbitMQ commonly delivers asynchronously to registered consumers, but AMQP 0-9-1 also supports polling through basic.get. With manual acknowledgments, RabbitMQ prefetch limits outstanding unacknowledged deliveries. Neither model requires immediate business processing.

Push versus pull doesn't decide which system meets your latency target. Benchmark producer-to-business-effect latency with your acknowledgment settings, batching, network placement, failure handling, and realistic backlog. For the broader messaging-model distinction, see JMS vs Kafka.

Where a Database-Backed Outbox Fits

An outbox is a publishing pattern, not a competing broker. Write the business change and an event row in one database transaction, then have a relay publish committed events. The relay can target Kafka, RabbitMQ, or a cloud queue. This avoids the inconsistent dual-write case where a database commit succeeds but event publication fails.

A relay can still publish duplicates, so consumers need idempotency: processing an event again must not repeat its business effect. A small outbox relay and managed queue may be enough when you don't need a shared event history. See the transactional outbox pattern and our explanation of change data capture.

Apache Kafka Use Cases: What Makes Each a Fit?

Kafka's official use-case overview covers these categories. The category alone isn't the justification; the requirement behind it is.

• Messaging: distribute business events to independently deployed applications that need their own consumption progress.
• Website activity tracking: make activity events available to live processing and later reporting pipelines.
• Metrics: provide a shared operational-data feed when several downstream systems need it.
• Log aggregation: transport log events to multiple processors; use a separate destination for searchable logs.
• Stream processing: continuously transform, enrich, aggregate, or join event streams.
• Event sourcing: record state changes as events; ensure retention and schema choices preserve the history required to rebuild state.
• Commit log: distribute changes and support rebuilding downstream state from a log.

For example, an order-event stream could feed search indexing, fraud analysis, and a warehouse through separate groups. A new indexing implementation could backfill from retained events while the existing consumers continue processing. That's a replay-and-fan-out requirement, even if the event rate is modest.

By contrast, a service dispatching thumbnail-generation jobs to interchangeable workers may only need task delivery, retries, and failed-job handling. Evaluate a queue first. If the caller needs an immediate answer, also consider whether synchronous request/reply is the simpler contract.

Can Kafka Be Used as a Database?

Kafka stores data durably, and compacted topics can support rebuilding keyed state. That doesn't turn the broker into a general-purpose database: fetching records by partition and offset isn't the same interface as arbitrary SQL queries, secondary-index lookups, or updating individual rows.

Use Kafka as the event history and build queryable views in an appropriate database or search system. If you're considering Kafka as the source of truth, first specify which history must survive retention, how deletions work, and how long a complete rebuild may take.

Delivery Semantics and Exactly-Once Boundaries

Broker durability and successful business processing are different guarantees. Decide what happens if a consumer crashes between updating its destination and recording progress.

• At-most-once: committing progress before processing can skip work after a crash.
• At-least-once: processing before committing progress allows replay after a crash, so completed work may repeat.
• Exactly-once within Kafka: transactions can atomically commit output records and consumed offsets. Downstream consumers must use the appropriate committed-read isolation to exclude aborted transactional output.

Kafka transactions don't automatically include an external database update, payment request, or email. End-to-end correctness requires cooperation from the destination, such as transactional storage of output and progress, or deduplication using stable event IDs. Producer idempotence prevents duplicate writes caused by producer retries; it doesn't make arbitrary external side effects exactly-once.

Is Kafka Overkill?

Kafka is overkill when its retained-log model doesn't solve an actual requirement and your team still inherits the complexity. There is no useful universal threshold of “millions of events.” A low-volume system can justify Kafka through replay and independent consumers; a high-volume task dispatcher may still fit a queue.

Be skeptical when the justification is only “we have microservices,” “we might grow,” or “large companies use it.” Ask which failure, recovery, or data-distribution requirement would become materially harder without Kafka.

When to Use Kafka: A Practical Decision Framework

Choose Kafka when you can answer these questions concretely:

1. Replay: Which applications must reprocess past events, and how much history must remain available?
2. Fan-out: Which independent consumers need their own progress and deployment lifecycle?
3. Ordering: What entity must stay ordered, and does its partitioning strategy distribute load acceptably?
4. Delivery and routing: Do you need an ordered stream, record-by-record work distribution, broker-side routing, or synchronous replies?
5. Latency and recovery: What are the normal latency target, tolerated outage, and required catch-up time?
6. Ownership and cost: Who owns storage, replication, upgrades, schemas, security, monitoring, and incident response?

If retained history and independent stream consumers are central, Kafka deserves a serious evaluation. If the answers mostly describe task dispatch or routing, start with RabbitMQ or a managed queue. Once you've made that architectural choice, use our Apache Kafka tutorial for implementation fundamentals.

Production Checklist

Kafka 4.x uses KRaft for cluster metadata; ZooKeeper mode was removed in Kafka 4.0. Current deployment planning should account for KRaft controllers as well as brokers, rather than copying a ZooKeeper-era setup.

• Budget the whole system: include replicated storage, retention, network transfer, managed-service charges, and engineering/on-call time. Managed hosting doesn't replace application-level design.
• Size for recovery: test partition distribution, hot keys, peak writes, and catch-up capacity while live traffic continues.
• Monitor failure signals: track consumer lag, event age, replica health, disk capacity, and producer errors—not just broker uptime.
• Protect data: configure transport encryption, authentication, and read/write authorization. Set explicit retention and access policies for sensitive events.
• Test evolution and recovery: replay older event schemas, crash consumers around destination writes and commits, verify duplicate handling, and rehearse upgrades.

FAQ

When should you use Kafka instead of RabbitMQ?

Use Kafka when retained event history, replay, and independent consumer groups are central requirements. Prefer RabbitMQ queues when broker-side routing and individually acknowledged work items better match the workload; evaluate RabbitMQ Streams separately if you also need replay.

When is Kafka overkill?

Kafka is overkill when you only need straightforward task delivery or request/reply and cannot identify a requirement that benefits from its retained log. Event volume alone isn't a useful decision rule.

Can multiple Kafka consumer groups read the same topic?

Yes. Each conventional consumer group maintains independent offsets and can read the same retained topic records. Within a group, consumers divide partition assignments rather than each receiving every record.

Does Kafka guarantee ordering?

Kafka guarantees log order within each partition, not across an entire topic. Consistent key-based partitioning helps preserve entity-level ordering, but partition expansion and concurrent application processing need careful handling.

Can Kafka replace a database?

Kafka can store event history and support state reconstruction, but it doesn't provide a general-purpose database query interface. Most applications still need a database or another queryable view for serving current state.

Further Reading

• Apache Kafka introduction — records, topics, partitions, and event streaming.
• Kafka consumer documentation — group assignments, offsets, replay, and flow control.
• Kafka design documentation — delivery semantics, transactions, and compaction.
• Kafka topic configuration — retention, cleanup policies, and minimum in-sync replicas.
• RabbitMQ exchanges — bindings and routing behavior.
• RabbitMQ acknowledgments and publisher confirms — delivery handling and publishing guarantees.
• RabbitMQ Streams — retained logs, repeated reads, and differences from queues.
• AWS transactional outbox guidance — coordinating database changes with event publication.