Late-arriving data

Late-arriving data is a record whose event time, when it actually happened, falls before a window that a pipeline has already processed or closed, because it was delayed in transit: a mobile app that buffered events while offline, a network retry, a slow upstream system. By the time it arrives, the pipeline may have already computed and published results for that period without it.

The core distinction is between event time, when something happened, and processing time, when the pipeline actually saw it; a stream processing system has to decide how long to keep a window open waiting for stragglers before finalizing results, using techniques like watermarking to define an acceptable cutoff. Waiting longer catches more late data but delays results and increases data freshness lag; closing windows sooner produces faster, more current results, but risks under-counting periods that still have late records trickling in.

Handling late data well matters because silently dropping it produces systematically undercounted historical metrics, an underestimated hourly revenue figure that never gets corrected, while naively including it can double-count if a value was already emitted once. Common approaches include re-emitting a corrected result for the affected window when late data arrives, or routing very late records to a separate reconciliation process. A frequent pitfall is not distinguishing late data from a duplicate: they require different handling, and conflating them corrupts one metric or the other.