PDE Designing Data Processing Systems Practice Question

A logistics company uses Cloud Dataflow to process a continuous stream of GPS events from delivery trucks. The pipeline must compute the distance traveled per truck per hour and write the results to BigQuery. The events are keyed by truck ID, and the pipeline uses windowing with a one-hour fixed window. The team notices that some trucks report events with timestamps that are several minutes late due to network delays. They want to ensure that late events are still included in the correct window and that the results are emitted only after a reasonable wait. What should they configure?

⚠ Common exam trap

The trap here is focusing only on triggers and forgetting that allowed lateness controls whether late data is accepted; without it, late events are discarded regardless of the trigger.

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

Set a trigger that fires when the watermark passes the end of the window, and set allowed lateness to a duration longer than the expected delay.

To include late events in the correct window, you must set allowed lateness to a value greater than the expected delay. The trigger on the watermark passing the window end ensures results are emitted after the window closes, and the allowed lateness keeps the window state alive to accept late data. This configuration balances timely results with completeness.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✗
Use a session window with a gap duration equal to the expected network delay.
Why it's wrong here
Session windows group events based on gaps of inactivity, which is not suitable for hourly distance calculations. The requirement is to compute distance per hour, so fixed windows are appropriate. Session windows would create unpredictable boundaries and could merge or split hours incorrectly, failing to produce consistent hourly results.
✗
Set a trigger that fires when the watermark passes the end of the window, and set allowed lateness to zero.
Why it's wrong here
Setting allowed lateness to zero means any event arriving after the watermark is discarded. Since the scenario explicitly mentions late events due to network delays, this would drop those events and produce incomplete results. The watermark passing the end of the window would not wait for the late data, so the requirement to include late events is not met.
✓
Set a trigger that fires when the watermark passes the end of the window, and set allowed lateness to a duration longer than the expected delay.
Why this is correct
Configuring allowed lateness to a duration greater than the maximum expected delay ensures that late events are still accepted and processed into the correct window. The trigger firing on the watermark passing the window end provides timely results, and the allowed lateness extends the window's lifetime to accommodate late data. This balances completeness and latency.
✗
Set a trigger that fires every minute, and set allowed lateness to zero.
Why it's wrong here
Firing every minute provides frequent updates but does not address late data because allowed lateness is zero. Late events would still be dropped. Additionally, firing every minute may produce many intermediate results that are later updated, which could be inefficient and may not align with the hourly aggregation requirement.

Go deeper

PDE Designing Data Processing Systems Practice Question

Why each option matters

Option-by-option breakdown

Related to this question