A company processes IoT sensor data in near real-time. They ingest data via Cloud Pub/Sub, then a Dataflow streaming pipeline writes to Bigtable for low-latency queries. Recently, they observed increased Pub/Sub message backlog during traffic spikes. What is the most effective scaling strategy?
Dataflow autoscaling can handle backlogs if enough workers are provisioned; increasing the max number of workers allows the pipeline to catch up during spikes.
Why this answer
The correct answer is B because the increased Pub/Sub backlog during traffic spikes indicates that the Dataflow pipeline is unable to consume messages as fast as they are being published. Increasing the Dataflow worker count and adjusting autoscaling configuration allows the pipeline to scale horizontally, processing more messages per second and reducing the backlog. Pub/Sub itself is designed to handle high throughput, so the bottleneck is the consumer (Dataflow), not the ingestion layer.
Exam trap
The trap here is that candidates mistakenly think Pub/Sub's throughput is limited by partitions (like Kafka) or that throttling the publisher is a valid scaling strategy, when in fact the bottleneck is the streaming pipeline's processing capacity, which must be scaled horizontally.
How to eliminate wrong answers
Option A is wrong because Pub/Sub does not use partitions like Kafka; increasing partitions is not a valid concept for Pub/Sub subscriptions, and throughput is managed by the subscriber's ability to pull messages, not by partitioning. Option C is wrong because throttling Pub/Sub publishing with Cloud Scheduler would reduce the incoming data rate, but this is counterproductive for near real-time processing and does not address the root cause of insufficient consumer capacity. Option D is wrong because adding a Cloud Function to pre-process messages would introduce an additional processing step that could further increase latency and does not directly solve the Dataflow pipeline's inability to keep up with the message volume.