The correct strategy is to use a stateful DoFn with an asynchronous HTTP client to batch external API calls. This approach improves reliability and performance in Dataflow streaming external API enrichment by decoupling the API requests from the main processing thread, allowing concurrent, non-blocking I/O that reduces latency and prevents pipeline backpressure. Stateful processing, via `@StateId` annotations, manages retry logic and deduplication across bundles, while batching minimizes the number of individual HTTP connections. On the Google Professional Data Engineer exam, this scenario tests your understanding of how to handle external dependencies in streaming pipelines without resorting to synchronous calls, which cause thread starvation and timeouts. A common trap is choosing a simple `ParDo` with synchronous HTTP, which blocks workers and fails under load. Memory tip: think “async + state = resilient enrichment” — the asynchronous client keeps data flowing, and stateful DoFn keeps track of what’s been sent or needs retrying.
PDE Practice Question: Building and operationalizing data processing systems
This PDE practice question tests your understanding of building and operationalizing data processing systems. Read the scenario carefully and evaluate each option against the stated constraints before committing to an answer. After answering, compare your reasoning against the explanation and wrong-answer breakdown below. Once you have made your selection, read the full explanation to reinforce the concept and understand why each distractor is designed to mislead on exam day.
A company is using Dataflow to stream data from Cloud Pub/Sub to BigQuery. The pipeline includes a custom ParDo transformation that enriches the data with external API calls. The pipeline is experiencing high latency and occasional failures due to API timeouts. What strategy should be employed to improve reliability and performance?
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Use a DoFn with stateful processing and batch API calls using asynchronous HTTP client.
Option C is correct because using a DoFn with stateful processing and an asynchronous HTTP client allows the pipeline to batch API calls and handle timeouts without blocking the main processing thread. This reduces latency by enabling concurrent requests and improves reliability through retry logic and state management, which is essential for external API enrichment in Dataflow.
Key principle: Answer the scenario, not the keyword: identify the specific constraint before choosing the most familiar-sounding option.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
✗
Remove the enrichment step and store raw data in BigQuery.
Why it's wrong here
This loses required data enrichment.
✗
Use a global window to accumulate all data before enrichment.
Why it's wrong here
Global window is inappropriate for streaming and introduces unbounded delay.
✓
Use a DoFn with stateful processing and batch API calls using asynchronous HTTP client.
Why this is correct
Batching and async calls reduce per-element latency and handle timeouts gracefully.
Related concept
Read the scenario before looking for a memorised answer.
✗
Increase the number of workers to parallelize API calls.
Why it's wrong here
This may increase load on the API but does not address timeouts.
Common exam traps
Common exam trap: answer the scenario, not the keyword
Google Cloud often tests the misconception that scaling workers (Option D) is a universal fix for performance issues, but the trap here is that API timeouts are often caused by the external service's capacity, not the pipeline's parallelism, and stateful batching with async calls is the correct architectural pattern.
Detailed technical explanation
How to think about this question
Stateful processing in Dataflow uses per-key state (e.g., ValueState or MapState) to buffer elements and batch API calls, reducing the number of external requests. An asynchronous HTTP client (e.g., using Apache Beam's Wait.on or a custom async handler) allows the DoFn to process other elements while waiting for responses, improving throughput. In real-world scenarios, this pattern is critical when the external API has rate limits or high latency, as it decouples the pipeline's processing speed from the API's response time.
KKey Concepts to Remember
Read the scenario before looking for a memorised answer.
Find the constraint that changes the correct option.
Eliminate answers that are true in general but not in this case.
TExam Day Tips
→Watch for words such as best, first, most likely and least administrative effort.
→Review why wrong options are wrong, not only why the correct option is correct.
Key takeaway
Answer the scenario, not the keyword: identify the specific constraint before choosing the most familiar-sounding option.
Real-world example
How this comes up in practice
A cloud solutions architect for a retail company is evaluating services for a new workload. The correct answer here reflects best practice for the specific scenario described — not a general cloud recommendation. Answer the scenario, not the keyword: identify the specific constraint before choosing the most familiar-sounding option. Cloud exam questions reward reading the constraint carefully: the same technology can be right or wrong depending on the use case.
What to study next
Got this wrong? Here's your next step.
Identify which exam domain this question belongs to, review the core concept, then practise similar questions from the same domain.
Building and operationalizing data processing systems — This question tests Building and operationalizing data processing systems — Read the scenario before looking for a memorised answer..
What is the correct answer to this question?
The correct answer is: Use a DoFn with stateful processing and batch API calls using asynchronous HTTP client. — Option C is correct because using a DoFn with stateful processing and an asynchronous HTTP client allows the pipeline to batch API calls and handle timeouts without blocking the main processing thread. This reduces latency by enabling concurrent requests and improves reliability through retry logic and state management, which is essential for external API enrichment in Dataflow.
What should I do if I get this PDE question wrong?
Identify which exam domain this question belongs to, review the core concept, then practise similar questions from the same domain.
What is the key concept behind this question?
Read the scenario before looking for a memorised answer.
About these practice questions
Courseiva creates original exam-style practice questions with explanations and wrong-answer analysis. It does not publish real exam questions, exam dumps, or protected exam content. Learn why practice questions differ from exam dumps →
Share a tip, memory trick, or ask about the reasoning behind this question. Do not post real exam questions, leaked content, braindumps, or copyrighted exam material. Comments are moderated and may be removed without notice.
This PDE practice question is part of Courseiva's free Google Cloud certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the PDE exam.
Question Discussion
Share a tip, memory trick, or ask about the reasoning behind this question. Do not post real exam questions, leaked content, braindumps, or copyrighted exam material. Comments are moderated and may be removed without notice.
Sign in to join the discussion.