PDE Maintaining and Automating Data Workloads Practice Question

You are using Cloud Composer to orchestrate a data pipeline that runs a Dataproc job to process data, followed by a BigQuery load. You notice that the Dataproc job sometimes takes longer than expected, causing the BigQuery load to start before the Dataproc job finishes, resulting in incomplete data. Which Airflow feature should you use to ensure the BigQuery load only runs after the Dataproc job completes successfully?

⚠ Common exam trap

Many candidates confuse task dependencies with sensors or retries; only explicit dependencies guarantee that one task waits for another's successful completion.

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

Set a dependency between the Dataproc job task and the BigQuery load task using the >> operator.

The correct way to enforce that the BigQuery load runs only after the Dataproc job completes successfully is to define a task dependency using the >> operator. This creates a directed edge in the DAG, ensuring the downstream task waits for the upstream task to succeed. Other options either do not control ordering or do not enforce success. Task dependencies are the core mechanism in Airflow for sequencing tasks in a data pipeline.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✓
Set a dependency between the Dataproc job task and the BigQuery load task using the >> operator.
Why this is correct
In Airflow, task dependencies are defined using the bitshift operators, such as >>, to specify that one task must complete successfully before another starts. By setting the Dataproc job task upstream of the BigQuery load task, you ensure that the load only runs after the Dataproc job finishes successfully. This is the fundamental way to control execution order in a DAG and directly addresses the issue of premature BigQuery loads due to timing.
✗
Use a TimeSensor to wait for the Dataproc job to finish.
Why it's wrong here
A TimeSensor waits for a specific time or interval, not for another task's completion. It does not monitor the status of the Dataproc job. Using a TimeSensor would introduce arbitrary delays and still not guarantee that the Dataproc job has finished. The correct approach is to establish a task dependency, not to rely on time-based sensors, which are better suited for waiting on external events like file arrivals.
✗
Use the trigger_rule parameter to set the BigQuery load task to 'all_done'.
Why it's wrong here
The trigger_rule parameter determines when a task should run based on the states of its upstream tasks. Setting it to 'all_done' means the task runs after all upstream tasks have completed, regardless of success or failure. This could cause the BigQuery load to run even if the Dataproc job failed, which is undesirable. It does not enforce success and does not solve the ordering issue; it only changes the condition for running after dependencies.
✗
Set the Dataproc job task's retries to a high number.
Why it's wrong here
Increasing retries on the Dataproc job task only affects how many times the task is retried if it fails; it does not control the ordering of tasks. The BigQuery load task could still start immediately after the Dataproc task is queued, regardless of retries. Retries are for fault tolerance, not for sequencing. Therefore, this setting does not ensure that the BigQuery load waits for the Dataproc job to complete successfully.

Go deeper

PDE Maintaining and Automating Data Workloads Practice Question

Why each option matters

Option-by-option breakdown

Related to this question