mediumMultiple Choice
PDE Practice Question: A data engineer is designing a batch data…
A data engineer is designing a batch data pipeline that reads Avro files from Cloud Storage, transforms data using Apache Beam, and writes to BigQuery. The pipeline must handle daily runs and backfills. Which runner should they use?
⚠ Common exam trap
Watch out — candidates often confuse the runner with the execution engine, assuming that any distributed runner (Flink, Spark) is suitable for production, when the question specifically tests knowledge of Google Cloud-native services and the need for managed infrastructure for batch pipelines with backfills.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
DataflowRunner
DataflowRunner is the correct choice because it is the fully managed service runner for Apache Beam on Google Cloud, optimized for batch and streaming pipelines. It automatically handles scaling, resource management, and exactly-once processing semantics, which are essential for reliable daily runs and backfills with Avro files from Cloud Storage and BigQuery sinks.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
FlinkRunner
Why it's wrong here
FlinkRunner executes Apache Beam pipelines on a Flink cluster, which the scenario does not provide or require. The stem's Cloud Storage and BigQuery dependencies point to a managed Google Cloud runner. FlinkRunner suits organisations already operating Flink infrastructure needing low-latency streaming with custom cluster control.
- ✓
DataflowRunner
Why this is correct
DataflowRunner executes Apache Beam pipelines on Google Cloud, reading from Cloud Storage and writing to BigQuery with autoscaling. It handles both scheduled daily runs and historical backfills by replaying the same pipeline over specified date ranges without infrastructure management.
- ✗
SparkRunner
Why it's wrong here
SparkRunner executes pipelines on an Apache Spark cluster, which the scenario does not provide; DataflowRunner is the managed runner for Cloud Storage and BigQuery. SparkRunner suits organisations already operating Spark infrastructure for their Beam jobs.
- ✗
DirectRunner
Why it's wrong here
DirectRunner executes locally on the engineer's machine, so it cannot process Cloud Storage Avro at scale or meet daily and backfill throughput. It is designed for local development and testing of pipeline logic before deployment, where small in-memory datasets and immediate debugging matter more than distributed execution.
Go deeper
Related to this question
About these practice questions
One of 747 original PDE practice questions on Courseiva, each with a full explanation and wrong-answer analysis — not exam dumps or protected exam content. Learn why practice questions differ from exam dumps →
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This PDE practice question is part of Courseiva's free Google Cloud certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the PDE exam.