Watermarking

A watermark is a moving marker a stream processing system uses to track how far event time has progressed, a running estimate of "we don't expect to see any more events earlier than this timestamp." It lets the system decide when a time window, like "total sales between 2pm and 3pm", can be closed and its result emitted, without waiting forever for every possible straggling event.

Watermarking is the primary mechanism used to bound how much late-arriving data a pipeline waits for: a watermark set five minutes behind the latest observed event time keeps a window open for late records up to five minutes old, and treats anything later as too late for the normal result. This differs from using processing time, when the system saw the event, as the cutoff, since processing time can be delayed unpredictably in ways unrelated to when the event actually occurred.

Watermarking matters because it is the tradeoff knob between result latency and completeness: a tighter watermark produces faster results that may miss more late data, and a looser one waits longer but captures more of it, and it interacts with how a system delivers exactly-once processing guarantees around window boundaries. The common pitfall is setting a watermark based on typical delay rather than worst-case delay, so an unusually late batch of events routinely falls outside the window and is silently dropped.