Quick Answer
The answer is Dataproc with PySpark for this batch ETL pipeline processing daily log files. This is the correct choice because Dataproc provides a fully managed Spark and Hadoop cluster that excels at distributed processing of large-scale data, while PySpark’s native integration with the BigQuery connector allows you to read from Cloud Storage, perform complex transformations, and write aggregated results directly to BigQuery without intermediate staging. On the Google Professional Data Engineer exam, this scenario tests your understanding of choosing the right compute service for batch workloads—common traps include selecting Dataflow (better for streaming or simpler pipelines) or BigQuery itself (which lacks the flexibility for custom PySpark transformations on raw log files). A key memory tip: think of Dataproc as your “Spark on GCP” for heavy-lifting batch ETL, especially when you need to leverage existing PySpark code or perform complex joins and aggregations on daily log dumps.
This PDE practice question tests your understanding of designing data processing systems. Compare every option against the stated constraints before choosing — the best answer satisfies all requirements, not just the most obvious one. After answering, compare your reasoning against the explanation and wrong-answer breakdown below. Once you have made your selection, read the full explanation to reinforce the concept and understand why each distractor is designed to mislead on exam day.
A data engineer needs to design a batch pipeline that processes daily log files from Cloud Storage and writes aggregated results to BigQuery. Which service is most appropriate for this ETL job?
Cloud Pub/Sub with Cloud Functions
Why wrong: Pub/Sub is for streaming; Cloud Functions are not suited for large file processing.
Cloud Composer
Why wrong: Composer is an orchestrator, not an execution engine.
Cloud Data Fusion
Why wrong: Data Fusion is a graphical ETL tool, better for codeless pipelines but less flexible for complex logic.
Dataproc with PySpark
Dataproc handles large batch processing efficiently with Spark.
Answer choices
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
Dataproc with PySpark
Dataproc with PySpark is the most appropriate choice because it provides a managed Spark/Hadoop environment that can efficiently process large daily log files stored in Cloud Storage using distributed computing. PySpark's native integration with BigQuery via the Spark BigQuery connector allows direct writing of aggregated results, making it ideal for batch ETL workloads that require complex transformations and high throughput.
Key principle: Answer the scenario, not the keyword: identify the specific constraint before choosing the most familiar-sounding option.
Answer analysis
For each option: why learners choose it and why it is or isn't the right answer here.
Cloud Pub/Sub with Cloud Functions
Why it's wrong here
Pub/Sub is for streaming; Cloud Functions are not suited for large file processing.
Cloud Composer
Why it's wrong here
Composer is an orchestrator, not an execution engine.
Cloud Data Fusion
Why it's wrong here
Data Fusion is a graphical ETL tool, better for codeless pipelines but less flexible for complex logic.
Dataproc with PySpark
Why this is correct
Dataproc handles large batch processing efficiently with Spark.
Related concept
Read the scenario before looking for a memorised answer.
Common exam traps
The trap here is that candidates often confuse orchestration (Cloud Composer) with execution, or assume serverless options like Cloud Functions can handle heavy batch ETL, but the question specifically requires a service that performs the ETL processing, not just schedules or triggers it.
Detailed technical explanation
Under the hood, Dataproc clusters can be configured with preemptible instances to reduce costs for batch jobs, and PySpark leverages RDDs and DataFrames to process data in parallel across nodes. The BigQuery connector uses the Storage API for high-throughput writes, and Dataproc's autoscaling can dynamically adjust cluster size based on workload, making it cost-efficient for daily batch runs that may have variable data volumes.
Key takeaway
Answer the scenario, not the keyword: identify the specific constraint before choosing the most familiar-sounding option.
Real-world example
A media company stores terabytes of video archives that are accessed once a year for audit purposes. Moving these objects to a cold storage tier (Azure Archive, S3 Glacier, or Google Nearline) costs a fraction of hot storage. Questions like this test whether you understand storage tiers, access frequency tradeoffs, and retrieval latency requirements.
What to study next
Identify which exam domain this question belongs to, review the core concept, then practise similar questions from the same domain.
Designing data processing systems — study guide chapter
Learn the concepts, then practise the questions
Designing data processing systems practice questions
Targeted practice on this topic area only
All PDE questions
499 questions across all exam domains
Google Professional Data Engineer study guide
Full concept coverage aligned to exam objectives
PDE practice test guide
How to use practice tests most effectively before exam day
Related practice questions
Use these pages to review the topic behind this question. This is how one missed question becomes focused revision.
Practise PDE questions linked to Designing data processing systems.
Practise PDE questions linked to Building and operationalizing data processing systems.
Practise PDE questions linked to Operationalizing machine learning models.
Practise PDE questions linked to Ensuring solution quality.
Practise PDE questions linked to PDE fundamentals.
Practise PDE questions linked to PDE scenario.
Practise PDE questions linked to PDE troubleshooting.
Practice this exam
Short sessions build daily habit. Longer sessions build exam-day stamina. Try a timed session to simulate real conditions.
FAQ
Designing data processing systems — This question tests Designing data processing systems — Read the scenario before looking for a memorised answer..
The correct answer is: Dataproc with PySpark — Dataproc with PySpark is the most appropriate choice because it provides a managed Spark/Hadoop environment that can efficiently process large daily log files stored in Cloud Storage using distributed computing. PySpark's native integration with BigQuery via the Spark BigQuery connector allows direct writing of aggregated results, making it ideal for batch ETL workloads that require complex transformations and high throughput.
Identify which exam domain this question belongs to, review the core concept, then practise similar questions from the same domain.
Read the scenario before looking for a memorised answer.
About these practice questions
Courseiva creates original exam-style practice questions with explanations and wrong-answer analysis. It does not publish real exam questions, exam dumps, or protected exam content. Learn why practice questions differ from exam dumps →
Last reviewed: Jun 24, 2026
This PDE practice question is part of Courseiva's free Google Cloud certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the PDE exam.
Question Discussion
Share a tip, memory trick, or ask about the reasoning behind this question. Do not post real exam questions, leaked content, braindumps, or copyrighted exam material. Comments are moderated and may be removed without notice.
Sign in to join the discussion.