Maintaining and Automating Data Workloads practice questions

BigQueryInsertJobOperator with wait_for_downstream=True

Why wrong: wait_for_downstream is not a parameter. Sensor and dependency are needed.

DataflowPythonOperator and GCSObjectExistenceSensor

Why wrong: DataflowPythonOperator is for Dataflow pipelines, not BigQuery load.

BigQueryOperator and FileSensor with downstream dependency

Why wrong: FileSensor is for local files, not GCS. Also downstream dependency is incorrect.

Pub/Sub subscription/num_undelivered_messages and Dataflow job/watermark_lag

Why wrong: num_undelivered_messages is a Pub/Sub metric; watermark_lag is similar to system lag but not standard. Better metrics exist.

Dataproc cluster/yarn_allocated_memory_percentage and Dataflow job/worker_cpu

Why wrong: Dataproc metrics are irrelevant for Dataflow.

BigQuery query/execution_times and Dataflow job/elapsed_time

Why wrong: BigQuery metrics irrelevant. elapsed_time is total job duration, not lag.

BigQuery table snapshots

Why wrong: Snapshots are immutable; if the source changes, the snapshot remains static. For ongoing sharing, clones are better.

BigQuery authorized views

Why wrong: Authorized views allow querying the latest data but do not provide a point-in-time snapshot.

BigQuery export to Cloud Storage

Why wrong: Exporting incurs costs and creates a separate copy, not a zero-copy clone.

Dataflow Shuffle

Why wrong: Dataflow Shuffle is a service for efficient data shuffling, not template creation.

Dataflow SQL

Why wrong: Dataflow SQL is for SQL-based pipelines, not reusable templates.

Dataflow Classic Templates

Why wrong: Classic Templates are limited to a single environment; Flex Templates are more flexible for custom containers.

Use committed use discounts for all VMs

Why wrong: Committed use discounts require 1-year commitment; preemptible VMs are cheaper without commitment.

Enable Dataproc auto-scaling

Why wrong: Auto-scaling helps but does not reduce cost as much as using preemptible VMs.

Use preemptible VMs for all nodes including master

Why wrong: Master node should not be preemptible to avoid cluster instability.

Metric: topic/send_message_operation_count; filter: topic_id

Why wrong: Topic metrics don't measure subscription backlog age.

Metric: subscription/ack_message_count; filter: subscription_id

Why wrong: ack_message_count is throughput, not age.

Metric: subscription/num_undelivered_messages; filter: subscription_id

Why wrong: num_undelivered_messages gives count, not age.

Dataplex Data Profiling

Why wrong: Profiling automatically generates statistics but does not enforce custom rules.

BigQuery stored procedures with scheduled queries

Why wrong: While possible, Dataplex provides native integration and reporting.

Cloud DLP inspection jobs

Why wrong: DLP focuses on sensitive data detection, not general quality rules.

Use bq ls to list table sizes

Why wrong: bq ls shows table metadata, not query cost.

Use BigQuery reservations to get cost estimate

Why wrong: Reservations are for flat-rate pricing, not estimation.

Use INFORMATION_SCHEMA.JOBS_BY_PROJECT to view past costs

Why wrong: This shows actual cost of past queries, not estimate for a new query.

A >> B; B >> D; A >> C; C >> D

Why wrong: This is equivalent but less idiomatic; still works but not the best practice.

A >> B >> C >> D

Why wrong: This runs sequentially: A then B then C then D.

A.set_downstream(B); B.set_upstream(C); C.set_downstream(D)

Why wrong: B.set_upstream(C) makes C run before B, which is incorrect.

Drain the pipeline first, then start a new one

Why wrong: Draining causes data loss; the requirement is to update without draining.

Use a different pipeline name and cancel the old one

Why wrong: This creates a new job, not an update.

Stop the job, update the code, and restart

Why wrong: Stopping loses state; update without stopping is possible.

Use Dataplex data quality rules with a custom SQL regex

Why wrong: Dataplex quality rules do not perform de-identification.

Use Cloud Data Loss Prevention API with Cloud Composer

Why wrong: While possible, Cloud Scheduler is simpler for scheduled jobs.

Use BigQuery column-level security with classification

Why wrong: Column-level security restricts access but does not mask data.

Use bq show to view job statistics

Why wrong: bq show shows metadata, not execution details.

Use bq query --format=prettyjson and look at statistics

Why wrong: Statistics provide summary but not per-stage breakdown.

Use Cloud Monitoring to view query execution graphs

Why wrong: Monitoring shows aggregated metrics, not per-query stage details.

Use ALTER TABLE RENAME COLUMN and then add new column

Why wrong: Renaming does not change mode.

Use bq update --schema to change the mode

Why wrong: bq update does not support changing column mode directly.

Use ALTER TABLE ALTER COLUMN SET DATA TYPE to change to NULLABLE

Why wrong: ALTER COLUMN only allows relaxing a column from REQUIRED to NULLABLE if using DDL with SET NOT NULL? Actually BigQuery only allows relaxing REQUIRED to NULLABLE via SQL using a query or manually recreating the table.

Use on-demand pricing for bursting

Why wrong: On-demand pricing is not part of flat-rate; flex slots are for bursting.

Set a maximum number of slots per query

Why wrong: This limits query resources but does not provide bursting capability.

Create a reservation with baseline + autoscaling slots

Why wrong: Autoscaling in reservations is for on-demand, not for flat-rate with committed use.

Operators (e.g., PythonOperator)

Why wrong: Operators perform actions; sensors are a type of operator but specifically for waiting.

SubDAGs for grouping tasks

Why wrong: SubDAGs are optional and not essential for this pattern.

extract >> [transform, load]

Why wrong: Incorrect: This makes transform and load run in parallel after extract, so load doesn't wait for transform.

transform >> extract >> load

Why wrong: Incorrect: This would start transform before extract, which violates the requirement.

extract >> load >> transform

Why wrong: Incorrect: This would start load before transform, which is not allowed.

Cloud Scheduler

Why wrong: Cloud Scheduler triggers a single target (e.g., a Cloud Function) but cannot orchestrate multiple steps within one workflow.

Cloud Composer

Why wrong: Cloud Composer requires a managed Airflow environment (minimum 3 nodes), which is more expensive and complex for a simple hourly workflow.

Dataflow

Why wrong: Dataflow is a data processing service, not an orchestration service. It is not designed for workflow orchestration.

Stop the pipeline, update the template, and restart with the same job name.

Why wrong: Stopping causes data loss unless using drain; restarting loses checkpointed state. The requirement is to update without draining.

Modify the original template and redeploy it as a new job with the same pipeline name.

Why wrong: Redeploying as a new job creates a separate pipeline; the original continues running. You cannot update a running job by launching a new one.

Use the gcloud dataflow jobs drain command, then restart with the new template.

Why wrong: Draining stops the pipeline gracefully but violates the requirement of updating without draining.

BigQuery write throughput is throttling the pipeline.

Why wrong: BigQuery throttling would cause backpressure, but system lag is more directly tied to processing capacity. However, this is less likely than worker underprovisioning.

The Pub/Sub subscription has too many unacknowledged messages.

Why wrong: High undelivered messages are a symptom, not a cause; the pipeline is not processing fast enough.

The pipeline is using a global window with late data handling.

Why wrong: Global windows with late data can cause watermark lag, but not necessarily increasing system lag; system lag is about processing delay.

Create a BigQuery table snapshot at the desired point in time and share it.

Why wrong: Table snapshots incur storage costs based on the size of the snapshot. Although they provide point-in-time reads, they are not as cost-effective as clones.

Export the table to Cloud Storage as Avro and share a signed URL.

Why wrong: Exporting incurs export costs and storage costs in GCS. The partner would need to load into BigQuery, adding more costs.

Grant the partner access to the original table with an authorized view.

Why wrong: This gives access to the live table, not a point-in-time snapshot. Also, the partner's queries incur analysis costs on the company's slot reservations if not using on-demand.