hardMultiple Choice
Dataflow Autoscaling for Pub/Sub Backlog
A company processes IoT sensor data in near real-time. They ingest data via Cloud Pub/Sub, then a Dataflow streaming pipeline writes to Bigtable for low-latency queries. Recently, they observed increased Pub/Sub message backlog during traffic spikes. What is the most effective scaling strategy?
Quick Answer
The answer is to increase the Dataflow worker count and adjust the autoscaling configuration. This is correct because a growing Pub/Sub backlog during traffic spikes signals that the Dataflow pipeline is the bottleneck—it cannot consume messages as fast as Pub/Sub publishes them. Dataflow autoscaling for Pub/Sub backlog relies on horizontal scaling; by raising the maximum worker count and fine-tuning the autoscaling parameters, the pipeline dynamically adds more workers to match the incoming throughput, thereby draining the backlog. On the Google Professional Data Engineer exam, this scenario tests your understanding of streaming pipeline throughput and the consumer-producer relationship in Pub/Sub. A common trap is to assume Pub/Sub itself is the problem, but Pub/Sub is built for high ingestion rates—the real issue is always the consumer’s processing capacity. Memory tip: “Backlog means the consumer is slow, not the source.”
⚠ Common exam trap
The trap here is that candidates mistakenly think Pub/Sub's throughput is limited by partitions (like Kafka) or that throttling the publisher is a valid scaling strategy, when in fact the bottleneck is the streaming pipeline's processing capacity, which must be scaled horizontally.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Increase Dataflow worker count and adjust autoscaling configuration
The increased Pub/Sub backlog during traffic spikes indicates that the Dataflow pipeline is unable to consume messages as fast as they are being published. Increasing the Dataflow worker count and adjusting autoscaling configuration allows the pipeline to scale horizontally, processing more messages per second and reducing the backlog. Pub/Sub itself is designed to handle high throughput, so the bottleneck is the consumer (Dataflow), not the ingestion layer.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Increase Pub/Sub subscription throughput by increasing the number of partitions
Why it's wrong here
Pub/Sub subscriptions have no partitions; partitioning belongs to Kafka or Pub/Sub Lite, so this mechanism does not exist here. It is tempting because partition-based parallelism is the standard throughput fix, and it would be correct for scaling a Kafka consumer group or Pub/Sub Lite subscription.
- ✓
Increase Dataflow worker count and adjust autoscaling configuration
Why this is correct
Increasing Dataflow worker count with tuned autoscaling directly drains the Pub/Sub backlog, since backlog itself is the autoscaling signal for streaming pipelines. Horizontal worker scaling raises parallel processing throughput, matching the traffic spikes, whereas Bigtable or Pub/Sub-side changes would not accelerate consumption of the queued messages.
- ✗
Use a Cloud Scheduler to throttle Pub/Sub publishing
Why it's wrong here
Throttling publishing reduces input volume, so backlog persists or worsens while sensors keep producing. It is tempting as a protective measure against overload, and would be correct for smoothing bursty traffic where downstream capacity is fixed and data loss is acceptable, not for clearing backlog.
- ✗
Add a Cloud Function to pre-process messages before they are consumed by Dataflow
Why it's wrong here
Cloud Functions cannot reduce Pub/Sub backlog; Dataflow autoscaling of worker count based on pipeline throughput is the mechanism that drains messages faster. Pre-processing adds latency and another hop. Cloud Functions suit lightweight event-driven tasks, not scaling a streaming consumer.
Go deeper
Related to this question
About these practice questions
One of 747 original PDE practice questions on Courseiva, each with a full explanation and wrong-answer analysis — not exam dumps or protected exam content. Learn why practice questions differ from exam dumps →
Same concept, more angles
1 more way this is tested on PDE
These questions test the same concept from different angles. Work through them to make sure you can recognise it however the exam phrases it.
Variation 1. A gaming company uses Pub/Sub to ingest player events and Dataflow for real-time analytics. They notice that the Pub/Sub subscription backlog is growing despite the Dataflow pipeline running continuously. The pipeline has a 1-hour window for aggregations. What is the most effective way to reduce the backlog?
hard- ✓ A.Increase the Dataflow pipeline's worker count via autoscaling.
- B.Use a push subscription instead of pull.
- C.Decrease the window duration to 10 minutes.
- D.Enable Pub/Sub topic retention.
Why A: Increasing the Dataflow pipeline's worker count via autoscaling directly addresses the backlog by adding more parallel processing capacity to consume messages from the Pub/Sub subscription faster. Since the pipeline is continuously running but the backlog grows, the bottleneck is processing throughput, not pipeline availability. Autoscaling allows Dataflow to dynamically allocate more workers based on the backlog size, matching consumption rate to the incoming message rate.
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This PDE practice question is part of Courseiva's free Google Cloud certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the PDE exam.