Async and streaming / Advanced / 6 min
Real-time analytics: windows and watermarks
Count clicks in 10-second windows as they stream in. The watermark trades complete counts against how fast results appear.
Start the simulation → Change the load, break a part, watch what happens. Free, no sign-up.
In short
- What it is
- A real-time analytics pipeline counts events (clicks, views, payments) in time windows as they stream in, for live dashboards and alerts.
- Why it matters
- Events arrive out of order and some very late, so the system must decide when a window is complete enough to publish.
- The trade-off
- A watermark with more allowed lateness gives more complete counts but publishes later; late events after that are dropped or sent as corrections.
- Say it in an interview
- "Kafka into Flink with event-time tumbling windows, a watermark of a few seconds, late data to a side output or an update, and results into a fast store like Druid or Redis for the dashboard."
Check yourself
Why count by event time rather than by arrival time?
Answer: A click belongs to the moment it happened, even if the network delivered it late. Event-time windows give the same answer however the network delays events; arrival-time counts shift with every hiccup.
What does the watermark tell the stream processor?
Answer: That events up to that time have (probably) all arrived, so earlier windows can be published. When the watermark passes the end of a window, the window is closed and its result emitted.
You raise the allowed lateness. What changes?
Answer: Counts get more complete, but results are published later. Waiting longer lets late events in, at the cost of latency; many systems also send a corrected result for very late data.