Design interviews / Intermediate / 5 min
Design a logging system: ship, buffer, index, search
Agents ship log lines to an indexer. Start an incident and see errors lost without a Kafka buffer, and DEBUG logs flood storage and search.
Start the simulation → Change the load, break a part, watch what happens. Free, no sign-up.
In short
- What it is
- A logging system collects log lines from every service through agents, buffers them, indexes them and lets engineers search them.
- Why it matters
- When something breaks, logs are how you find out why, and they are needed most during the incident that floods them.
- The trade-off
- Shipping more levels and fields helps debugging but multiplies cost and lag; a buffer prevents loss but adds a moving part.
- Say it in an interview
- "Agents (Fluent Bit) to Kafka, consumers into Elasticsearch/OpenSearch or Loki, INFO and up by default, sampling for noisy logs, hot/warm/cold retention."
Check yourself
Why put a durable buffer like Kafka between log agents and the indexer?
Answer: It absorbs bursts, so lines wait instead of being dropped when the indexer is busy. Incidents create log spikes exactly when you need the logs; a buffer turns a spike into a short delay instead of data loss.
What happens if every service ships DEBUG logs in production?
Answer: Volume and storage cost multiply and the indexer falls far behind. DEBUG is often ten times the volume of INFO; ship it only for short, targeted debugging.
During an incident, why might searching for errors show fewer than really happened?
Answer: Some lines were dropped and others are still waiting to be indexed. Watch both drops and indexing lag; both make the picture incomplete when it matters most.