- A
Use Pub/Sub with at-least-once delivery and deduplicate in BigQuery using a unique identifier.
Why wrong: At-least-once delivery requires manual deduplication, which is error-prone and not automatically ensured by Dataflow.
- B
Configure Dataflow with exactly-once sinks using file staging and deduplication.
Dataflow's exactly-once sink mechanism ensures each record is written exactly once, preventing duplicates.
- C
Use Cloud Functions to deduplicate messages before they enter the pipeline.
Why wrong: Deduplicating before the pipeline adds latency and complexity; it is not a built-in Dataflow feature.
- D
Enable idempotent writes in BigQuery.
Why wrong: BigQuery does not support idempotent writes; deduplication must be handled upstream.
Question 326 of 1,000
Ensure Exactly-Once Semantics in Dataflow Pipelines
This PDE practice question tests your understanding of ensuring solution quality. Read the scenario carefully and evaluate each option against the stated constraints before committing to an answer. After answering, compare your reasoning against the explanation and wrong-answer breakdown below. Once you have made your selection, read the full explanation to reinforce the concept and understand why each distractor is designed to mislead on exam day.
A data pipeline processes streaming data with Dataflow. The team notices occasional data duplication in BigQuery. What is the best approach to ensure exactly-once processing?
Clue words in this question
Noticing these words before you look at the options changes how you read each choice.
Clue:
"best"Why it matters: Signals that multiple options may be partially correct. Choose the option that most directly solves the exact problem described, not the one that sounds most complete.
Quick Answer
The answer is to configure Dataflow with exactly-once sinks using file staging and deduplication. This approach works because Dataflow’s exactly-once mechanism writes streaming data to temporary files before committing them to BigQuery, then uses a deduplication step to remove any duplicates that might arise from retries or failures. On the Google Professional Data Engineer exam, this question tests your understanding of how Dataflow guarantees end-to-end exactly-once processing, a critical concept for streaming pipelines. A common trap is assuming that BigQuery’s native write operations are idempotent, but they are not; without Dataflow’s built-in dedup, at-least-once delivery will produce duplicates. Remember the memory tip: “Stage, then dedup—no duplicates show up.”
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Configure Dataflow with exactly-once sinks using file staging and deduplication.
Option B is correct because Dataflow's exactly-once sinks use a two-phase commit protocol with file staging and deduplication to ensure that each record is written exactly once to the sink, even if the pipeline retries. This approach handles the inherent at-least-once delivery from Pub/Sub by staging output files and committing them atomically, preventing duplicates in BigQuery without relying on downstream deduplication.
Key principle: Answer the scenario, not the keyword: identify the specific constraint before choosing the most familiar-sounding option.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Use Pub/Sub with at-least-once delivery and deduplicate in BigQuery using a unique identifier.
Why it's wrong here
At-least-once delivery requires manual deduplication, which is error-prone and not automatically ensured by Dataflow.
- ✓
Configure Dataflow with exactly-once sinks using file staging and deduplication.
Why this is correct
Dataflow's exactly-once sink mechanism ensures each record is written exactly once, preventing duplicates.
Clue confirmation
The clue word "best" in the question point toward this answer.
Related concept
Read the scenario before looking for a memorised answer.
- ✗
Use Cloud Functions to deduplicate messages before they enter the pipeline.
Why it's wrong here
Deduplicating before the pipeline adds latency and complexity; it is not a built-in Dataflow feature.
- ✗
Enable idempotent writes in BigQuery.
Why it's wrong here
BigQuery does not support idempotent writes; deduplication must be handled upstream.
Common exam traps
Common exam trap: answer the scenario, not the keyword
The trap here is that candidates often assume deduplication at the destination (BigQuery) is sufficient, but the key insight is that exactly-once processing must be enforced at the pipeline level (Dataflow) using mechanisms like file staging and atomic commit to avoid race conditions and state inconsistencies across distributed workers.
Detailed technical explanation
How to think about this question
Dataflow's exactly-once sink mechanism leverages a shuffle service and a commit protocol that writes to a staging directory before atomically renaming files to the final output location. This ensures that even if a worker fails and retries, the staged output is either fully committed or discarded, preventing partial writes. In practice, this matters when processing high-volume streams where even a small duplication rate can lead to significant cost and data integrity issues in downstream analytics.
KKey Concepts to Remember
- Read the scenario before looking for a memorised answer.
- Find the constraint that changes the correct option.
- Eliminate answers that are true in general but not in this case.
TExam Day Tips
- Watch for words such as best, first, most likely and least administrative effort.
- Review why wrong options are wrong, not only why the correct option is correct.
Key takeaway
Answer the scenario, not the keyword: identify the specific constraint before choosing the most familiar-sounding option.
Real-world example
How this comes up in practice
A cloud solutions architect for a retail company is evaluating services for a new workload. The correct answer here reflects best practice for the specific scenario described — not a general cloud recommendation. Answer the scenario, not the keyword: identify the specific constraint before choosing the most familiar-sounding option. Cloud exam questions reward reading the constraint carefully: the same technology can be right or wrong depending on the use case.
What to study next
Got this wrong? Here's your next step.
Identify which exam domain this question belongs to, review the core concept, then practise similar questions from the same domain.
- →
Ensuring solution quality — study guide chapter
Learn the concepts, then practise the questions
- →
Ensuring solution quality practice questions
Targeted practice on this topic area only
- →
All PDE questions
1,000 questions across all exam domains
- →
Google Professional Data Engineer study guide
Full concept coverage aligned to exam objectives
- →
PDE practice test guide
How to use practice tests most effectively before exam day
Related practice questions
Related PDE practice-question pages
Use these pages to review the topic behind this question. This is how one missed question becomes focused revision.
Designing Data Processing Systems practice questions
Practise PDE questions linked to Designing Data Processing Systems.
Ingesting and Processing the Data practice questions
Practise PDE questions linked to Ingesting and Processing the Data.
Storing the Data practice questions
Practise PDE questions linked to Storing the Data.
Preparing and Using Data for Analysis practice questions
Practise PDE questions linked to Preparing and Using Data for Analysis.
Maintaining and Automating Data Workloads practice questions
Practise PDE questions linked to Maintaining and Automating Data Workloads.
Building and operationalizing data processing systems practice questions
Practise PDE questions linked to Building and operationalizing data processing systems.
Operationalizing machine learning models practice questions
Practise PDE questions linked to Operationalizing machine learning models.
Ensuring solution quality practice questions
Practise PDE questions linked to Ensuring solution quality.
PDE fundamentals practice questions
Practise PDE questions linked to PDE fundamentals.
PDE scenario practice questions
Practise PDE questions linked to PDE scenario.
PDE troubleshooting practice questions
Practise PDE questions linked to PDE troubleshooting.
Practice this exam
Start a free PDE practice session
Short sessions build daily habit. Longer sessions build exam-day stamina. Try a timed session to simulate real conditions.
FAQ
Questions learners often ask
What does this PDE question test?
Ensuring solution quality — This question tests Ensuring solution quality — Read the scenario before looking for a memorised answer..
What is the correct answer to this question?
The correct answer is: Configure Dataflow with exactly-once sinks using file staging and deduplication. — Option B is correct because Dataflow's exactly-once sinks use a two-phase commit protocol with file staging and deduplication to ensure that each record is written exactly once to the sink, even if the pipeline retries. This approach handles the inherent at-least-once delivery from Pub/Sub by staging output files and committing them atomically, preventing duplicates in BigQuery without relying on downstream deduplication.
What should I do if I get this PDE question wrong?
Identify which exam domain this question belongs to, review the core concept, then practise similar questions from the same domain.
Are there clue words in this question I should notice?
Yes — watch for: "best". Signals that multiple options may be partially correct. Choose the option that most directly solves the exact problem described, not the one that sounds most complete.
What is the key concept behind this question?
Read the scenario before looking for a memorised answer.
About these practice questions
Courseiva creates original exam-style practice questions with explanations and wrong-answer analysis. It does not publish real exam questions, exam dumps, or protected exam content. Learn why practice questions differ from exam dumps →
Same concept, more angles
1 more ways this is tested on PDE
These questions test the same concept from different angles. Work through them to make sure you can recognise it however the exam phrases it.
Variation 1. A financial services company operates a real-time fraud detection pipeline using Apache Beam running on Google Cloud Dataflow. The pipeline reads transactions from Pub/Sub, enriches them with customer data from Bigtable, runs a machine learning model with side inputs from a Redis cluster, and writes results to BigQuery for downstream reporting. The data must be processed with exactly-once semantics to avoid duplicate fraud alerts or missing transactions. The pipeline currently uses a global window with 5-minute accumulation, but the team is experiencing high latency and occasional duplicates when the model side input is updated (triggered every 15 minutes via a WatchTransform). Additionally, the pipeline has a dead letter queue that outputs failed records to a separate Pub/Sub topic, but these records are never reprocessed. The team needs to ensure high reliability and data quality. Which course of action should the team take to improve solution quality?
hard- A.Use fixed windows with a 10-minute duration and session gap of 2 minutes, disable side input caching, and log all dead letter records to Cloud Storage for manual inspection.
- B.Switch to a batch processing approach that runs every minute using Cloud Composer, with data loaded from Pub/Sub into BigQuery and then processed with Dataproc to run the model.
- ✓ C.Implement sliding windows of 5 minutes with a 2-minute allowed lateness, use side inputs with periodic refreshes using the .withUpdateFrequency transformation, and set up a Cloud Function to automatically replay dead letter records back to the main Pub/Sub topic after fixing the issue.
- D.Keep the global window but use a custom trigger with early firings every 30 seconds and a late-firing threshold of 1 minute, and configure the side input to be broadcast every 5 minutes using a Read transform.
Why C: Option C is correct because sliding windows with 2-minute allowed lateness handle late-arriving data without causing duplicates, and the .withUpdateFrequency transformation refreshes the side input every 15 minutes without triggering reprocessing, reducing latency. Replaying dead letter records via a Cloud Function ensures data completeness. Option A is incorrect because fixed windows with session gaps do not address side input latency and may lose late events. Option B is incorrect because batch processing is unsuitable for real-time fraud detection and introduces significant latency. Option D is incorrect because custom triggers with early firings can cause duplicates due to side input updates, and the Read transform does not efficiently handle periodic model refreshes.
Keep practising
More PDE practice questions
- A company wants to process large CSV files stored in Cloud Storage and load them into BigQuery. The files are generated…
- A company runs a Dataflow streaming pipeline that reads from Cloud Pub/Sub and writes to BigQuery. The pipeline uses a s…
- A company uses Cloud Dataproc for ephemeral clusters to run batch jobs. They want to ensure job reliability and data qua…
- Your company uses Vertex AI Pipelines to automate model retraining. The pipeline has three steps: data extraction from B…
- A company wants to use BigQuery to query data stored in Parquet files in Cloud Storage without loading the data into Big…
- A company has deployed a machine learning model to AI Platform Prediction. The model uses a custom container with a Tens…
Last reviewed: Jul 4, 2026
This PDE practice question is part of Courseiva's free Google Cloud certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the PDE exam.
Question Discussion
Share a tip, memory trick, or ask about the reasoning behind this question. Do not post real exam questions, leaked content, braindumps, or copyrighted exam material. Comments are moderated and may be removed without notice.
Sign in to join the discussion.