PDE Ingesting and Processing the Data Practice Question

You are designing a Dataflow pipeline that reads from Pub/Sub and writes to BigQuery. Some incoming messages are malformed and fail to parse. How should you handle these messages to ensure the pipeline continues processing without data loss?

⚠ Common exam trap

PDE often tests the misconception that Pub/Sub retry or pipeline failure is an acceptable error-handling strategy — candidates forget that retries cannot fix deterministic parse failures and that failing the pipeline violates availability requirements.

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

Write malformed messages to a dead-letter sink (e.g., Pub/Sub topic or GCS) and continue processing

Writing malformed messages to a dead-letter sink (Pub/Sub topic or GCS) preserves the data for later inspection while allowing the pipeline to continue processing valid messages. This is the standard Dataflow/Apache Beam pattern for handling unparseable records without halting the stream or silently discarding data. It satisfies both the 'no data loss' and 'continue processing' requirements.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✗
Configure Pub/Sub to retry indefinitely until the message is processed
Why it's wrong here
Infinite retries cause the malformed message to be redelivered forever, blocking the subscription's progress and never resolving the parse failure. It is tempting because retries handle transient errors such as timeouts, which is the right approach when failures are temporary rather than caused by permanently invalid message content.
✗
Use a try-catch block in the pipeline and ignore malformed messages
Why it's wrong here
Catching and discarding malformed messages silently drops them, violating the no-data-loss requirement; they must be routed to a dead-letter destination for later inspection. It is tempting because try-catch keeps the pipeline running, which is the correct pattern when malformed records are genuinely disposable and no audit trail is required.
✓
Write malformed messages to a dead-letter sink (e.g., Pub/Sub topic or GCS) and continue processing
Why this is correct
Routing unparseable records to a dead-letter sink isolates them from the main pipeline, so a single malformed message cannot halt processing or be silently dropped. This satisfies the no-data-loss constraint by preserving failures for later inspection while valid messages continue to BigQuery.
✗
Set the pipeline to fail and alert the team via Cloud Monitoring
Why it's wrong here
Failing the pipeline halts all processing and leaves the malformed message unhandled, so throughput stops and no dead-letter record is produced. It is tempting because failing fast with Cloud Monitoring alerts surfaces problems immediately, which suits pipelines where any malformed input signals an upstream defect that must be fixed before continuing.

Go deeper

PDE Ingesting and Processing the Data Practice Question

Why each option matters

Option-by-option breakdown

Related to this question