A company runs a streaming data pipeline using Dataflow to process real-time data and insert into BigQuery. Recently, workers are frequently failing with out-of-memory errors and the pipeline latency is increasing. What should they do to resolve the issue?
Streaming Engine moves state to a backend service, reducing memory usage per worker.
Why this answer
Dataflow Streaming Engine offloads the streaming data processing state and shuffle data from worker memory to a backend service, reducing memory pressure on workers. This directly addresses out-of-memory errors and latency increases without requiring manual scaling or machine type changes. It is the recommended solution for streaming pipelines experiencing memory bottlenecks.
Exam trap
Google Cloud often tests the misconception that scaling up resources (more memory or more workers) is the primary fix for streaming pipeline memory issues, when the real solution is to offload state management using Streaming Engine.
How to eliminate wrong answers
Option A is wrong because simply increasing worker machine type and memory does not resolve the root cause of state management overhead in streaming pipelines; it only delays the failure and increases cost without optimizing data flow. Option B is wrong because adding Pub/Sub buffering does not fix the memory issue within Dataflow workers; it shifts the problem to a different layer and may introduce additional latency and complexity. Option C is wrong because enabling autoscaling and increasing max workers can help with throughput but does not reduce per-worker memory consumption; workers may still fail with OOM errors if the pipeline's state or shuffle data exceeds available memory.