Google ACE Practice Question: Designing a data pipeline that reads from Cloud…

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

• ✓

  Configure a Cloud Storage Pub/Sub notification → Cloud Function that launches a Dataflow job for each new file.

  Why this is correct

  GCS object notifications to Pub/Sub trigger a Cloud Function that launches a Dataflow job. Dataflow handles files up to 5 GB within 10 minutes using parallel workers, and the event-driven architecture processes files exactly when they land.

• ✗

  Use a Cloud Scheduler cron job that scans Cloud Storage every minute and processes new files with Dataflow.

  Why it's wrong here

  A Cloud Scheduler cron job that scans Cloud Storage every minute introduces up to 60 seconds of inherent latency, because a file arriving just after a scan waits until the next poll. It is polling-based, not event-driven, and requires separate bookkeeping to track which blobs have already been processed to avoid duplicates or missing files. Repeatedly listing the bucket also incurs unnecessary API costs and compute time during idle periods, whereas Pub/Sub notifications fire instantly and cost nothing when no files land.

• ✗

  Use Cloud Functions triggered by GCS events to read and transform the 5 GB file directly.

  Why it's wrong here

  Processing a 5 GB file directly inside a Cloud Function triggered by a GCS event is not viable because of the execution environment's hard limits. Cloud Functions gen1 times out at 9 minutes and gen2 at 60 minutes, with a maximum of 8 GB memory—so loading the entire 5 GB file into memory for transformation is risky and often fails. More fundamentally, a single function instance processes the file serially, whereas Dataflow shards the work across many workers and can easily finish within the 10-minute window.

• ✗

  Use BigQuery Data Transfer Service to load files from Cloud Storage on a schedule.

  Why it's wrong here

  BigQuery Data Transfer Service is built specifically for scheduled, idempotent loads into BigQuery, not for running custom transformation pipelines. Its minimum schedule interval is 15 minutes, which fails the 10-minute processing requirement, and it cannot execute the custom transformation logic (e.g., data cleanup, enrichment, or schema changes) that the CSV requires. Even though it can read from Cloud Storage, it moves data directly into BigQuery tables and does not support a parallel, pipeline-based processing model like Dataflow.

Go deeper

Related to this question