PDE Maintaining and Automating Data Workloads Practice Question
Your company uses Cloud Composer to orchestrate a data pipeline that includes Dataproc Spark jobs and BigQuery load operations. You need to pass the output file path from the Spark job to the next BigQuery task in the DAG. Which two mechanisms can you use to share data between tasks? (Choose TWO.)
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Write the output path to a Cloud Storage object and read it in the next task.
Airflow XComs allow tasks to exchange small amounts of data (e.g., file paths) by pushing and pulling values. Cloud Storage can be used as an intermediate store: the Spark job writes output to GCS, and the BigQuery task reads from that location. BigQuery does not directly communicate with Dataproc. Cloud Composer variables are for global configuration, not task-to-task. Pub/Sub is not needed for simple file path sharing.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Store the output path as a Cloud Composer variable.
Why it's wrong here
Variables are global and not scoped to a specific DAG run; they are not intended for task-to-task data sharing.
- ✗
Publish the output path to a Pub/Sub topic and subscribe in the next task.
Why it's wrong here
Publishing to Pub/Sub introduces asynchronous, event-driven coupling between tasks, but Cloud Composer’s XCom mechanism is designed for synchronous, in-memory data exchange within the same DAG execution context. Pub/Sub would require the downstream task to poll or subscribe to a topic, adding latency and external dependency management that XCom avoids. This option is tempting because Pub/Sub is a standard choice for decoupling microservices or triggering workflows across separate systems, where tasks run in independent environments rather than within a single Airflow DAG.
- ✓
Write the output path to a Cloud Storage object and read it in the next task.
Why this is correct
Cloud Storage is a durable store that both tasks can access.
- ✗
Use BigQuery as an intermediary to store the output path.
Why it's wrong here
BigQuery is a data warehouse, not designed for passing small variables.
- ✓
Use Airflow XComs to push the output path from the Spark task and pull it in the BigQuery task.
Why this is correct
XComs are designed for passing small data between tasks.
Go deeper
Related to this question
About these practice questions
Courseiva writes every PDE question from scratch — 890 in total, each with an explanation and a wrong-answer breakdown. None are copied from real exams or dumps. Learn why practice questions differ from exam dumps →
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This PDE practice question is part of Courseiva's free Google Cloud certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the PDE exam.