Let’s say it’s Friday. Not party Friday, but Black Friday. You’re working on a busy e-commerce system that handles thousands of orders per minute. Suddenly, the service responsible for billing processing crashes. Until it recovers, new orders are piling up. How do you resume processing after the service restart?

Typically, you use the messaging system to accept incoming requests and then process them gradually. Messaging systems have durable storage capabilities to keep messages until they’re delivered. Kafka can even keep them longer with a defined retention policy.

If we’re using Kafka, when the service restarts, it can resubscribe to the topic. But which messages should we process?

One naive approach to ensure consistency might be reprocessing messages from the topic's earliest position. That might prevent missed events, but it could also lead to a massive backlog of replayed data—and a serious risk of double processing. Would you really want to read every message on the topic from the very beginning, risking duplicate charges and triggering actions that have already been handled? Wouldn’t it be better to pick up exactly where you left off, with minimal overhead and no guesswork about what you’ve already handled?

Not surprisingly, that’s what Kafka does: it has built-in offset tracking. What’s offset? Offsets let each consumer record its precise position in the messages stream (logical position in the topic portion). That ensures that restarts or redeployments don’t force you to re-ingest everything you’ve ever consumed.

Services that consume message streams can restart or be redeployed for countless reasons: you might update a container in Kubernetes, roll out a patch on a bare-metal server, or autoscale in a cloud environment.

In this article, we’ll look at how Kafka manages offsets under the hood, the failure scenarios you must prepare for, and how offsets help you keep your system consistent—even when services are constantly starting and stopping. We’ll also see how other technologies tackle a similar challenge.

As always, we take a specific tool and try to extend it to the wide architecture level.

Check also previous articles in these series:

• How Kafka Producer Works internally

• Kafka Consumers: Under the Hood of Message Processing

• Understanding Kafka's Consumer Protocol: A Deep Dive into How Consumers Talk to Brokers

Let’s do the thought experiment. Stop now for a moment and consider how you would implement offset storage for a messaging system. Remember your findings, or better yet, note them down.

Are you ready? Let’s now discuss various options we could use, and check how Kafka offset storage evolved and why.

1. Early Attempts at Storing Offsets

Database Storage. A straightforward way to track what a consumer has processed is to store the last processed message ID in a relational database or key-value store. This can be fine if you have a single consumer, but you run into trouble as soon as you introduce multiple consumers. Each consumer must update the same record, raising the risk of race conditions, distributed locking or expensive transactions. Plus, if one consumer crashes mid-update, another might not know which offset is the latest.

Local Files. Another idea is to write the current offset to a local file on the machine running the consumer. This might spare you the overhead of a database, but it quickly becomes unmanageable if the machine dies or you need to scale out. Each new consumer instance has its own file, and there’s no easy way to keep them all consistent.

As soon as you have more than one consumer—or if you want failover without losing track of progress these approaches break down. That’s why Kafka uses a different model.

2. From ZooKeeper to special topic

Historically, Kafka itself used Apache ZooKeeper to store consumer offsets.

ZooKeeper is a distributed key-value store with additional coordination features. It allows distributed systems to store and retrieve data across multiple servers. It acts like a centralized database that helps different parts of a system share and manage configuration information and state. Unlike a simple key-value store, it provides features like atomic writes, watches (notifications), and hierarchical naming, making it particularly useful for configuration management and synchronization in distributed computing environments.

Using Zookeeper to store offsets might have worked fine when you had only a few consumer groups or infrequent commits, but it wasn’t designed to handle the constant stream of updates in larger deployments. Every offset commit triggered writes to ZooKeeper, and as the number of consumer groups and partitions grew, those writes multiplied, creating performance bottlenecks and stability concerns. ZooKeeper was never intended for high-throughput offset tracking, so teams running sizable Kafka clusters began encountering scaling issues—ranging from slower commit latencies to potential coordination timeouts—when offset commits overloaded ZooKeeper.

To solve these problems, Kafka introduced the __consumer_offsets topic in version 0.9. This internal topic made offset management an integral part of Kafka rather than an external concern. By storing offset commits in a Kafka topic, you benefit from the following:

• Replication across brokers: If one broker fails, another has the offset data.

• Natural ordering of commits: Commits appear in the topic in the exact order they happen.

• Compaction: Kafka discards older offset commits for the same consumer-group-partition key, controlling the topic size.

Version 0.9 still allowed using Zookeeper through the offsets.storage=zookeeper setting but made it obsolete. Then, Zookeeper implementation was dropped. Let’s discuss in detail why the topic for offsets was superior design.

3. How offsets topic works

Although __consumer_offsets might look like any other Kafka topic from the outside, but it’s specifically optimized to track consumer positions. Let’s unpack its structure and flow:

Topic Structure and Partitioning

By default, it’s partitioned into a fixed number of partitions (often 50), with each partition managing offset commits for a subset of consumer groups. This design ensures that offset storage can scale horizontally—no single partition is overloaded by too many commits.

In this simplified flow, a consumer sends an offset commit to the Kafka broker, which appends a commit event to the appropriate partition in __consumer_offsets. That commit event is then replicated to follower brokers, and finally, the broker acknowledges success back to the consumer. Such replication across multiple brokers helps to protect against broker failures. When one broker dies, the cluster can still keep up with information on the others.

You should always consider how frequently you commit offsets to get the desired performance. Committing after every message assures minimal replays but burdens the broker with frequent writes. Committing in larger batches cuts down on overhead but means a crash could reprocess a bigger chunk of messages. Monitoring metrics like consumer lag, commit latency, and rebalance frequency helps you tune these factors for your workloads.

Message Format

Under the hood, each offset commit is stored as a small message. While the actual format is internal to Kafka, you can think of it in a simplified TypeScript-esque interface:

interface OffsetCommit {
  groupId: string;         // e.g. 'payment-service'
  topic: string;           // e.g. 'transactions'
  partition: number;       // e.g. 0
  offset: number;          // last processed offset
  metadata?: string;       // optional metadata
  commitTimestamp: number; // epoch time for commit
}...