Setting Up Kafka Right: Topic Design, Partitioning, and Idempotency

Failures in streaming systems rarely show up right away. Events processed twice, a single overloaded consumer, or processing that falls further and further behind often surface only when downstream data stops adding up. Most of these problems can be prevented when the Kafka topics are designed.

Key takeaways

• Every Kafka topic is a contract between producers and consumers and deserves deliberate decisions about naming, schema, and retention.
• The partition key controls ordering and load distribution at the same time, and a badly chosen key creates hot partitions.
• With at-least-once delivery, duplicates are unavoidable over time, so consumers have to be idempotent.
• Idempotent consumers turn replaying a topic into a non-event instead of a data-corruption risk.
• Consumer lag is the earliest warning signal of a bottleneck anywhere in the pipeline.

Treating Kafka topics as contracts

I treat every Kafka topic as a contract between whoever produces to it and whoever consumes from it. Being a name in a config file comes second. For every topic I make three decisions deliberately.

Naming conventions

A topic's name should tell you what domain event it carries and who owns it. Which service happens to write to it today is irrelevant for that.

Schema evolution with a schema registry

Schema evolution needs to be an explicit, versioned decision. It must not happen implicitly because a producer added a field. A schema registry with compatibility rules enforced at write time catches such changes before they become a consumer's runtime exception.

Retention

The third decision is retention: how long events need to live, and whether a topic is a transient message bus or a durable log that downstream systems might replay from the beginning.

I don't answer any of these three questions with a default. I decide them once, up front, because changing them later means coordinating every producer and consumer at once.

Partitioning: ordering and load distribution

Partition count and partition key are where most of the design work lives. They determine ordering guarantees and how evenly load spreads across consumers.

Choosing the right partition key

Kafka only guarantees ordering within a partition. Events that must be processed in order relative to each other therefore need the same partition key, typically an entity ID.

Get the key wrong and you either lose ordering you needed or, more commonly, create a hot partition. One key's volume then dwarfs the rest, and a single consumer becomes the bottleneck for the whole topic.

Sizing consumer groups

I size consumer groups to partition count deliberately. A group can never have more active consumers than partitions, and extra consumers just sit idle.

Idempotent consumers instead of duplicates

At-least-once delivery is the practical default for Kafka. Over a long enough time window, that makes duplicates a certainty, whether from producer retries or consumer rebalances.

Rather than trying to guarantee exactly-once delivery at the broker level, I design consumers to be idempotent. Every handler produces the same end state whether an event is processed once or twice. Usually that works through an idempotency key checked against a store before a side effect runs.

Replay without risk

That same property is what makes replay safe. If I need to reprocess a topic from an earlier offset to recover from a bug, idempotent consumers make that a non-event instead of a data-corruption risk.

Running Kafka: retention and consumer lag

Day to day, I watch two levers. The first is retention strategy: time-based, size-based, or compacted for topics that represent current state rather than a log of events. It directly affects storage cost and how far back a replay can reach.

The second is consumer lag, by far the most useful health signal a streaming system gives you. A consumer group falling behind its produced offsets is the earliest warning of a downstream bottleneck, well before anything looks broken from the outside.

What this means for your company

If your systems exchange data through Kafka, these design decisions have tangible consequences. An event processed twice can mean a duplicate booking or a wrong report. A bottleneck in the pipeline eventually shows up as late data in systems your business departments rely on. That is why consumer lag is a metric that belongs on the IT lead's dashboard, because it reports these problems before anyone in a business department notices them.

These failures can largely be avoided when topics, keys, and consumers are designed deliberately from the start. Fixing them later gets expensive, because every team involved then has to switch over at the same time. I describe how I plan event-driven systems and design them for growing load on the page about AWS costs and scaling.