Async and streaming / Intermediate / 9 min
Kafka: a log, not a queue
Partitions, keys, consumer groups, lag, rebalancing and replay, in seven guided steps.
Start the simulation → Change the load, break a part, watch what happens. Free, no sign-up.
In short
- What it is
- Kafka is a log split into partitions; producers append, and each consumer group keeps its own read position (offset).
- Why it matters
- Many systems can read the same events independently and replay them, at very high throughput.
- The trade-off
- Order is kept only inside one partition, and a group cannot have more busy consumers than there are partitions.
- Say it in an interview
- "I key events by order id so all events for one order land in one partition and stay in order."
Check yourself
A topic has 3 partitions and a consumer group has 5 consumers. How many consumers sit idle?
Answer: 2. Each partition is read by exactly one consumer in a group, so only 3 can work.
How does Kafka keep all events for one user in order?
Answer: Using the user ID as the message key so they go to one partition. Ordering is per partition, and the key decides the partition.
What is consumer lag?
Answer: How far a group’s committed offset is behind the end of the log. It is the count of messages written but not yet processed, and the key Kafka health metric.