A streaming pipeline can be healthy even when the data isn't complete yet.

One thing worth separating in streaming systems is processing time from event time.

A message can arrive late because of a disconnected device, a network delay or an upstream system that buffers events. If downstream aggregations assume every event arrives in chronological order, the resulting windows can be misleading.

In event-stream processing, event-time windows and watermarks help define how long the system waits for late events before considering a window sufficiently complete. The exact behaviour depends on the processing engine and its configuration.

A few questions come up when designing these pipelines: How late can events realistically arrive? What should happen to events that arrive after the allowed lateness? Does the business require corrected historical aggregates? How do you monitor event-time lag separately from processing latency?

There is not one watermark setting that fits every workload. A short delay may reduce reporting latency but increase the number of late events that need separate handling.

How do you choose the lateness threshold for Azure streaming workloads, especially when the business expects both low latency and accurate historical results?