PDE Maintaining and Automating Data Workloads Practice Question

A media company runs a Cloud Composer environment whose DAGs trigger Dataflow batch jobs and BigQuery loads. The operations team reports that Composer costs are rising and that DAGs occasionally stall because workers are saturated. You review the environment and find that several tasks are long-running sensors that hold worker slots while waiting on external conditions. Which TWO changes should you make to reduce worker saturation and cost? (Choose two.)

⚠ Common exam trap

The trap here is treating worker saturation as a scheduling or concurrency problem, when the real cause is tasks that occupy worker slots for long durations while doing nothing but waiting.

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

Move the long-running Dataflow job monitoring out of the DAG by having the DAG submit the job and exit while a separate process tracks completion.

Worker saturation caused by long-blocking sensors and job-waiting tasks is best relieved by changing how those tasks consume resources rather than by adding concurrency limits or rescheduling. Deferrable sensors release slots during waits, and decoupling long Dataflow job monitoring from the worker keeps the pool available for other tasks, which lowers both stalls and cost.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✓
Move the long-running Dataflow job monitoring out of the DAG by having the DAG submit the job and exit while a separate process tracks completion.
Why this is correct
When a task blocks a worker for the entire duration of a Dataflow job, that slot is unavailable for other work. Submitting the job and returning, then checking status separately or through a deferrable waiter, reduces the time a worker is occupied. This shortens the critical path of worker occupancy and alleviates the saturation the team is seeing.
✗
Change the DAG schedule from a cron expression to a timedelta interval so runs are spaced further apart.
Why it's wrong here
Adjusting the schedule changes when runs start but not how much worker capacity each run consumes. If a single run already saturates workers because of slot-holding sensors, spacing runs apart does not prevent the stall. It also delays data delivery, which the business may not accept, without addressing the underlying resource contention.
✗
Set the Composer environment's worker count to the minimum and rely on autoscaling to add workers only when the queue is deep.
Why it's wrong here
Reducing workers while tasks are already saturating the pool would make stalls more frequent, not less. Autoscaling responds to queue depth with some lag and cannot compensate for tasks that hold slots for hours. The correct direction is to stop tasks from monopolizing slots, not to shrink the worker pool that is already under pressure.
✗
Increase the DAG's max_active_tasks parameter so more tasks can run in parallel on the same workers.
Why it's wrong here
max_active_tasks raises how many tasks may run concurrently, but it does not create more CPU or memory on the workers. If workers are already saturated, allowing more concurrent tasks worsens contention rather than relieving it. This parameter controls admission, not capacity, so it does not solve the reported problem.
✓
Switch the long-waiting sensors to deferrable mode so they release the worker slot while waiting and resume when the condition is met.
Why this is correct
Deferrable operators hand off the wait to a trigger process and free the worker slot, so many sensors can be pending simultaneously without occupying workers. This directly addresses the saturation described, because slot-holding sensors are the reported cause of stalled DAGs. It also lowers cost since fewer worker resources are idled during waits.

Go deeper

PDE Maintaining and Automating Data Workloads Practice Question

Why each option matters

Option-by-option breakdown

Related to this question