PDE Ingesting and Processing the Data Practice Question

A data pipeline processes JSON files from Cloud Storage, transforms them using Apache Beam, and writes the output to BigQuery. Some records are malformed and cause the pipeline to fail. How should the engineer handle these errors to ensure the pipeline continues processing while preserving the malformed records for analysis?

⚠ Common exam trap

A common misconception in Google Cloud data pipelines is that error handling should involve retrying indefinitely or simply logging errors to Cloud Logging, but the correct approach is to isolate and persist malformed records to a durable sink like Cloud Storage for later analysis.

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

Catch exceptions in a DoFn and write the malformed records to a separate Cloud Storage bucket using a FileIO sink.

It allows the pipeline to continue processing by catching exceptions within a DoFn and writing malformed records to a separate Cloud Storage bucket using a FileIO sink. This preserves the malformed records for later analysis without blocking the main data flow, which is a standard pattern in Apache Beam for handling dead-letter records. The approach ensures fault tolerance while maintaining data integrity for debugging.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✗
Set the pipeline to retry malformed records indefinitely until they succeed.
Why it's wrong here
Malformed records will never parse, so retrying indefinitely blocks the pipeline and loses the records rather than preserving them. Dead-letter queues or side outputs exist precisely to route unparseable elements elsewhere; retries suit transient failures such as timeouts or quota errors, not structural JSON faults.
✗
Use a side input to send malformed records to a dead letter queue in Pub/Sub for later reprocessing.
Why it's wrong here
Using a side input to send malformed records to a dead letter queue is incorrect because side inputs provide supplementary, often static, data *to* a `PTransform` for enrichment or configuration, rather than acting as an output channel *for* processed elements. A dead letter queue requires a separate output `PCollection` for erroneous records, typically achieved by branching the pipeline based on validation. This option is tempting as dead letter queues are crucial for robust error handling, and side inputs are valuable for enriching main data streams with lookup tables or dynamic parameters.
✗
Log the malformed records to Stackdriver and skip them in the pipeline.
Why it's wrong here
Stackdriver logging discards the record's payload context and offers no durable, queryable store for later analysis, so malformed records are effectively lost. A BigQuery dead-letter table or Beam side output preserves them; logging suits operational diagnostics, not data recovery and reprocessing.
✓
Catch exceptions in a DoFn and write the malformed records to a separate Cloud Storage bucket using a FileIO sink.
Why this is correct
Wrapping parsing in a DoFn try/except lets the pipeline emit valid records downstream while routing malformed ones to a separate Cloud Storage bucket via FileIO. This satisfies both constraints: the pipeline continues processing, and the bad records are preserved for later analysis.

Go deeper

PDE Ingesting and Processing the Data Practice Question

Why each option matters

Option-by-option breakdown

Related to this question