Fix Azure Stream Analytics Watermark Delay

Quick Answer

Watermark delay is Stream Analytics' way of telling you how far behind real time the job's processing has fallen, so when that delay keeps growing rather than staying flat, the job's compute capacity is not keeping pace with its workload. Streaming Units are the unit of compute and memory Stream Analytics allocates to a job, and a 5-minute hopping window with a 1-minute hop is relatively demanding, since it means the engine is maintaining and re-evaluating overlapping window state far more often than a simple tumbling window would require, on top of reading from all 10 Event Hubs partitions. When the SU allocation is too low for that workload, the job cannot process incoming events as fast as they arrive, so a backlog builds up, and that backlog is exactly what shows up as an ever-increasing watermark delay, since the system is always a little further behind than it was a moment ago. This is different from a one-time or stable delay, which might point to a transient spike; a delay that keeps climbing over time signals a sustained capacity shortfall rather than a temporary blip. Whenever a Stream Analytics scenario describes watermark delay or output lag that grows continuously, especially alongside a nontrivial windowing configuration or multiple input partitions, think first about whether the job has enough Streaming Units before looking at other causes.