PDE Ingesting and Processing the Data Practice Question

A company is building a data pipeline that ingests streaming data from Pub/Sub, transforms it with Dataflow, and loads it into BigQuery. They want to handle malformed messages that cannot be parsed. Which TWO actions should they implement for error handling? (Choose 2)

⚠ Common exam trap

Google Cloud often tests the misconception that raising an exception (Option D) is acceptable for error handling in streaming pipelines, but the correct approach is to isolate failures using a dead letter sink (Option C) while logging errors (Option E) to maintain pipeline continuity.

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

Use a dead letter sink to write malformed messages to Cloud Storage or Pub/Sub for later analysis

Option C is correct because a dead letter sink (often implemented in Dataflow via a tagged output or a separate Pub/Sub topic/Cloud Storage path) captures unparseable records so they can be inspected and reprocessed without losing data or halting the pipeline. Option E is correct because logging the parsing error and continuing lets the pipeline keep processing valid messages, which is essential for a resilient streaming pipeline where one bad record should not stop the flow. Option A is not appropriate because silently dropping malformed messages loses data and provides no visibility for debugging or remediation. Option B is not the right pattern because a side input is used to supply supplementary data to a DoFn, not to filter or quarantine malformed records. Option D is wrong because raising an exception in the DoFn would fail the pipeline immediately, causing unnecessary downtime and blocking valid messages from being processed.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✗
Configure the pipeline to drop malformed messages silently
Why it's wrong here
Silent dropping discards malformed messages with no record, log or alert, so failures become invisible and data loss is undetectable. The scenario requires handling, meaning capture and review. Dead-letter queues or error tables preserve the payload for later diagnosis and reprocessing.
✗
Use a side input to filter out malformed messages
Why it's wrong here
Side inputs supply supplementary lookup data to a DoFn's main processing, not a routing mechanism for unparseable records. Malformed messages need a dead-letter destination, such as a BigQuery error table or Pub/Sub topic, so they can be inspected and reprocessed rather than discarded during transformation.
✓
Use a dead letter sink to write malformed messages to Cloud Storage or Pub/Sub for later analysis
Why this is correct
A dead letter sink captures unparseable messages instead of failing the pipeline, satisfying the requirement to handle malformed records. Dataflow's dead letter pattern writes them to Cloud Storage or Pub/Sub, preserving them for later analysis while the main pipeline continues.
✗
Raise an exception in the DoFn to fail the pipeline immediately
Why it's wrong here
Raising an exception in the DoFn halts the entire streaming pipeline, so one bad message blocks all valid traffic. Error handling should isolate malformed records into a dead-letter sink while the pipeline continues. Failing fast suits batch validation where no output may proceed.
✓
Log the error and continue processing the next message
Why this is correct
Logging the parse failure and continuing lets the streaming pipeline keep processing valid messages rather than stalling on malformed ones. This satisfies the error-handling requirement by isolating bad records without halting ingestion, complementing a dead letter sink.

Go deeper

PDE Ingesting and Processing the Data Practice Question

Why each option matters

Option-by-option breakdown

Related to this question