hardMultiple ChoiceObjective-mapped
End-to-End Exactly-Once Processing — Pub/Sub to Bigtable
You are designing a streaming data pipeline that must guarantee exactly-once processing semantics for financial transactions. The pipeline reads from Cloud Pub/Sub and writes to Cloud Bigtable. Each transaction has a unique transaction ID. Which features do you need to implement to ensure exactly-once semantics end-to-end?
Quick Answer
The answer is to use Dataflow with exactly-once processing and idempotent Bigtable mutations keyed by the transaction ID. This combination is correct because Dataflow’s built-in checkpointing and source-side deduplication ensure each record from Pub/Sub is processed exactly once, while Bigtable’s idempotent writes—where the same transaction ID always produces the same result—prevent duplicates even if a mutation is retried. On the Google Professional Data Engineer exam, this scenario tests your understanding of how to bridge streaming guarantees with a NoSQL sink; a common trap is assuming Pub/Sub’s at-least-once delivery alone suffices, forgetting that the sink must also be idempotent. The key insight is that exactly-once is a pipeline property, not a single service feature. Memory tip: think “Dataflow dedup + Bigtable idempotent = end-to-end exactly-once.”
⚠ Common exam trap
Google Cloud often tests the misconception that Pub/Sub's 'exactly-once delivery' feature exists or that manual offset management alone can achieve end-to-end exactly-once semantics, when in reality Pub/Sub only offers at-least-once delivery and requires a processing framework like Dataflow to achieve exactly-once end-to-end.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Use Dataflow with exactly-once processing, and ensure the Bigtable sink uses idempotent mutations based on the transaction ID.
Dataflow's exactly-once processing guarantees that each record is processed precisely once, and idempotent Bigtable mutations (keyed by transaction ID) ensure that even if a mutation is retried, the result is the same. This combination provides end-to-end exactly-once semantics: Dataflow handles source-side deduplication and checkpointing, while Bigtable's idempotent writes prevent duplicates at the sink.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Use Cloud Pub/Sub with synchronous pull and manually commit offsets after successfully writing to Bigtable.
Why it's wrong here
Manual offset tracking is possible but not native; Dataflow handles this automatically with exactly-once.
- ✓
Use Dataflow with exactly-once processing, and ensure the Bigtable sink uses idempotent mutations based on the transaction ID.
Why this is correct
Dataflow deduplicates records using unique identifiers; Bigtable idempotent writes (e.g., using CheckAndMutate) ensure that even if a mutation is retried, the result is the same.
- ✗
Use Dataflow with at-least-once processing and implement deduplication in a windowed transform using the transaction ID.
Why it's wrong here
At-least-once processing in Dataflow can still cause duplicates if not combined with exactly-once checkpointing.
- ✗
Use Cloud Pub/Sub with exactly-once delivery enabled, and write to Bigtable using single-row transactions.
Why it's wrong here
Cloud Pub/Sub does not support exactly-once delivery; it is at-least-once. Single-row transactions are insufficient for deduplication across multiple rows.
Go deeper
Related to this question
About these practice questions
Courseiva writes every PDE question from scratch — 890 in total, each with an explanation and a wrong-answer breakdown. None are copied from real exams or dumps. Learn why practice questions differ from exam dumps →
Same concept, more angles
1 more way this is tested on PDE
These questions test the same concept from different angles. Work through them to make sure you can recognise it however the exam phrases it.
Variation 1. A company is ingesting real-time sensor data from thousands of devices into Cloud Pub/Sub. They need to process this data with low latency (seconds) and exactly-once semantics. Which data processing service should they use?
easy- A.Cloud Run with Pub/Sub push
- B.Cloud Functions triggered by Pub/Sub
- ✓ C.Dataflow streaming with exactly-once processing
- D.Dataproc with Spark Streaming
Why C: Dataflow streaming with exactly-once processing is the correct choice because it provides exactly-once semantics for Pub/Sub sources via checkpointing and idempotent sinks, and it meets the low-latency (seconds) requirement through its streaming engine that minimizes per-element overhead. Cloud Dataflow's integration with Pub/Sub ensures that each message is processed exactly once, even in the presence of failures, by using snapshots and consistent state management.
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This PDE practice question is part of Courseiva's free Google Cloud certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the PDE exam.