PDE Ingesting and Processing the Data Practice Question
A data pipeline processes JSON files from Cloud Storage, transforms them using Apache Beam, and writes the output to BigQuery. Some records are malformed and cause the pipeline to fail. How should the engineer handle these errors to ensure the pipeline continues processing while preserving the malformed records for analysis?
⚠ Common exam trap
A common misconception in Google Cloud data pipelines is that error handling should involve retrying indefinitely or simply logging errors to Cloud Logging, but the correct approach is to isolate and persist malformed records to a durable sink like Cloud Storage for later analysis.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Catch exceptions in a DoFn and write the malformed records to a separate Cloud Storage bucket using a FileIO sink.
It allows the pipeline to continue processing by catching exceptions within a DoFn and writing malformed records to a separate Cloud Storage bucket using a FileIO sink. This preserves the malformed records for later analysis without blocking the main data flow, which is a standard pattern in Apache Beam for handling dead-letter records. The approach ensures fault tolerance while maintaining data integrity for debugging.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Set the pipeline to retry malformed records indefinitely until they succeed.
Why it's wrong here
Malformed records will never succeed, leading to infinite retries and pipeline stuck.
- ✗
Use a side input to send malformed records to a dead letter queue in Pub/Sub for later reprocessing.
Why it's wrong here
Using a side input to send malformed records to a dead letter queue is incorrect because side inputs provide supplementary, often static, data *to* a `PTransform` for enrichment or configuration, rather than acting as an output channel *for* processed elements. A dead letter queue requires a separate output `PCollection` for erroneous records, typically achieved by branching the pipeline based on validation. This option is tempting as dead letter queues are crucial for robust error handling, and side inputs are valuable for enriching main data streams with lookup tables or dynamic parameters.
- ✗
Log the malformed records to Stackdriver and skip them in the pipeline.
Why it's wrong here
Logging does not preserve the records in a structured, queryable format for analysis.
- ✓
Catch exceptions in a DoFn and write the malformed records to a separate Cloud Storage bucket using a FileIO sink.
Why this is correct
This follows the dead letter pattern: malformed records are written to a separate sink, allowing the pipeline to continue and enabling later analysis.
Go deeper
Related to this question
About these practice questions
This PDE question is part of Courseiva's 890-question bank — original exam-style content with full explanations and wrong-answer analysis, never real exam questions or exam dumps. Learn why practice questions differ from exam dumps →
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This PDE practice question is part of Courseiva's free Google Cloud certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the PDE exam.