A company has a batch ETL job that runs daily using Cloud Dataflow. The job reads from Cloud Storage, transforms data, and writes to BigQuery. Recently, the job started failing with 'Resources have been exhausted' errors. What is the most likely cause?
Resource exhausted error indicates quota issue.
Why this answer
The 'Resources have been exhausted' error in Cloud Dataflow typically indicates that the project has reached its Dataflow API quota, such as the maximum number of concurrent jobs or API requests per minute. This is a common issue when multiple jobs run simultaneously or when the quota is set low by default. The error is distinct from resource exhaustion in the underlying compute or storage layers.
Exam trap
Google Cloud often tests the distinction between API quota exhaustion and resource exhaustion in the underlying infrastructure (e.g., Compute Engine CPU/memory), leading candidates to incorrectly attribute the error to increased data volume or schema changes.
How to eliminate wrong answers
Option A is wrong because deleting the Cloud Storage bucket would cause a 'bucket not found' or 'object not found' error, not a 'Resources have been exhausted' error. Option C is wrong because a significant increase in input data volume would lead to autoscaling limits or worker resource exhaustion (e.g., out of memory), but the specific 'Resources have been exhausted' message is tied to API quota limits, not data volume. Option D is wrong because a schema change in BigQuery would result in a schema mismatch or insertion error, not an API quota exhaustion error.