PDE Maintaining and Automating Data Workloads Practice Test 4 — 15 Questions

Question 1

You are using Cloud Composer 2 to orchestrate a data pipeline that includes tasks that interact with BigQuery and Cloud Storage. You notice that some tasks are failing due to insufficient permissions, and you want to ensure that the Composer environment has the necessary IAM roles to execute these tasks. Which two IAM roles should you grant to the Composer service account to allow it to manage BigQuery jobs and access Cloud Storage objects? (Choose two.)

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

roles/storage.objectAdmin. To allow Composer tasks to manage BigQuery data and access Cloud Storage objects, the service account needs roles that grant data modification in BigQuery and object-level control in Cloud Storage. The BigQuery Data Editor role provides the necessary BigQuery permissions, while the Storage Object Admin role provides the necessary Cloud Storage permissions. These roles together enable the pipeline to function without permission errors.

Answer

roles/composer.worker

Answer

roles/storage.bucketViewer

Answer

roles/bigquery.user

Question 2

A data engineer uses Cloud Composer to orchestrate a daily batch pipeline. A downstream task should only start after an upstream BigQuery load job finishes successfully and a specific file appears in Cloud Storage. Which combination of operators should the engineer use in the Airflow DAG?

Accepted Answer

BigQueryInsertJobOperator and GCSObjectExistenceSensor with upstream dependency. The engineer needs a BigQuery load job to finish successfully and a specific file to appear in Cloud Storage before a downstream task starts. The correct combination is BigQueryInsertJobOperator to run and wait for the BigQuery job, and GCSObjectExistenceSensor to check for the file, with the sensor set as an upstream dependency of the downstream task. This ensures both conditions are met before proceeding.

Answer

BigQueryInsertJobOperator with wait_for_downstream=True

Answer

DataflowPythonOperator and GCSObjectExistenceSensor

Answer

BigQueryOperator and FileSensor with downstream dependency

Question 3

An engineer needs to create a reusable Dataflow pipeline that can be executed with different parameters without modifying code. Which Dataflow feature should they use?

Accepted Answer

Dataflow Flex Templates. Dataflow Flex Templates allow packaging a pipeline as a Docker image with a metadata file, enabling reuse with different parameters at runtime without code changes. They support dynamic parameters and are the recommended approach for reusable pipelines.

Answer

Dataflow Shuffle

Answer

Dataflow SQL

Answer

Dataflow Classic Templates

Question 4

A data engineer needs to alert when Pub/Sub subscription has messages older than 1 hour. Which Cloud Monitoring metric and filter should they use?

Accepted Answer

Metric: subscription/oldest_unacked_message_age; filter: subscription_id. The metric subscription/oldest_unacked_message_age directly reports the age of the oldest unacknowledged message in a subscription, which is exactly what's needed to alert when messages are older than 1 hour. The filter subscription_id scopes the metric to the specific subscription. This metric is designed for monitoring message backlog age and is the correct choice for this requirement.

Answer

Metric: topic/send_message_operation_count; filter: topic_id

Answer

Metric: subscription/ack_message_count; filter: subscription_id

Answer

Metric: subscription/num_undelivered_messages; filter: subscription_id

Question 5

A team wants to enforce data quality rules on BigQuery tables using Dataplex. They need to run column-level checks for null values and row-level checks for value ranges on a schedule. Which Dataplex feature should they use?

Accepted Answer

Dataplex Data Quality Tasks. Dataplex Data Quality Tasks allow you to define and run data quality rules on BigQuery tables, including column-level checks (e.g., null checks) and row-level checks (e.g., value ranges). These tasks can be scheduled to run periodically, and they generate results that can be monitored. This is the native Dataplex feature for enforcing data quality.

Answer

Dataplex Data Profiling

Answer

BigQuery stored procedures with scheduled queries

Answer

Cloud DLP inspection jobs

Question 6

An organization uses BigQuery on-demand pricing. To control costs, they want to estimate the bytes processed by a query before running it. Which command or method should they use?

Accepted Answer

Use the bq query --dry_run command. The bq query --dry_run flag validates a query and returns the estimated bytes that would be processed without actually executing it or incurring charges. This is the standard method to estimate on-demand BigQuery costs before running a query, since on-demand pricing is based on bytes processed.

Answer

Use bq ls to list table sizes

Answer

Use BigQuery reservations to get cost estimate

Answer

Use INFORMATION_SCHEMA.JOBS_BY_PROJECT to view past costs

Question 7

A streaming Dataflow pipeline needs to be updated without draining the existing pipeline. Which update strategy should be used?

Accepted Answer

Replace the job with a new job using the same pipeline name. Dataflow supports replacing a running streaming job with an updated one using the same pipeline name via the 'Replace job' (or update) mechanism, which performs an in-place upgrade that preserves the pipeline's state and does not require draining. This is the supported path for updating streaming pipelines without stopping data ingestion. Draining, cancelling, or stopping the job all interrupt processing and are not the recommended no-drain update strategy.

Answer

Drain the pipeline first, then start a new one

Answer

Use a different pipeline name and cancel the old one

Answer

Stop the job, update the code, and restart

Question 8

A company wants to use Cloud DLP to inspect data in BigQuery for sensitive information and de-identify it by masking credit card numbers. They want to perform this on a schedule. Which approach should they take?

Accepted Answer

Use Cloud DLP inspect and de-identify jobs triggered by Cloud Scheduler. Cloud DLP can inspect BigQuery tables and de-identify using transforms like masking. Scheduling can be done via Cloud Scheduler.

Answer

Use Dataplex data quality rules with a custom SQL regex

Answer

Use Cloud Data Loss Prevention API with Cloud Composer

Answer

Use BigQuery column-level security with classification

Question 9

A data engineer is building a batch pipeline that runs daily using Cloud Composer. The pipeline has three tasks: extract data from Cloud Storage, transform data using Dataflow, and load the transformed data into BigQuery. The engineer wants to ensure that the Dataflow job only starts after the extraction task completes successfully, and the load task only starts after the Dataflow job finishes. How should the engineer define the task dependencies in the Airflow DAG?

Accepted Answer

extract >> transform >> load. Airflow uses the bitshift operator (>>) to define task dependencies in a linear sequence. The DAG must ensure that the extract task completes before the transform task starts, and the transform task completes before the load task starts. This is achieved by chaining the tasks in order: extract >> transform >> load, which enforces the required sequential execution.

Answer

extract >> [transform, load]

Answer

transform >> extract >> load

Answer

extract >> load >> transform

Question 10

You need to schedule a simple workflow that fetches data from an API every hour, transforms it using Cloud Functions, and writes the result to Cloud Storage. The workflow has no complex branching or retry logic beyond basic retries. Which orchestration service is the MOST cost-effective and simplest to implement?

Accepted Answer

Workflows. Workflows is a serverless, fully managed orchestration service that charges only per step executed, making it ideal for simple linear workflows with basic retries. It natively integrates with Cloud Functions and Cloud Storage via connectors, so the hourly fetch-transform-write pipeline can be defined in YAML/JSON without managing infrastructure. Cloud Scheduler alone can trigger jobs but cannot orchestrate multi-step logic or handle the transform step.

Answer

Cloud Scheduler

Answer

Cloud Composer

Answer

Dataflow

Question 11

A data engineer needs to inspect a BigQuery table for sensitive data such as credit card numbers and email addresses before sharing it with a third party. The engineer also wants to de-identify the data by masking the sensitive columns. Which Google Cloud service should be used?

Accepted Answer

Cloud DLP. Cloud DLP (Data Loss Prevention) is the correct service because it is specifically designed to inspect, classify, and de-identify sensitive data such as credit card numbers and email addresses. It provides built-in infoType detectors for over 150 types of sensitive data and supports masking, tokenization, and other de-identification techniques. The engineer can use Cloud DLP to scan BigQuery tables and then apply transformations to mask the sensitive columns before sharing.

Answer

Dataplex

Answer

BigQuery column-level security

Answer

Data Catalog

Question 12

A company runs a critical batch pipeline using Cloud Dataflow. The pipeline processes financial transactions and runs every hour. Recently, some runs have failed due to transient errors (e.g., network timeouts). The engineer wants to automatically retry failed runs without manual intervention. The pipeline is launched from a Cloud Composer DAG using DataflowPythonOperator. What is the BEST way to handle retries?

Accepted Answer

Set the 'retries' parameter in the DAG's default_args to a positive integer.. Cloud Composer (Apache Airflow) natively supports task-level retries via the 'retries' parameter in default_args. When a DataflowPythonOperator fails due to a transient error, Airflow automatically re-executes the task up to the specified number of retries, without requiring custom sensors or external triggers. This is the simplest and most reliable mechanism for handling transient failures in a DAG-driven pipeline.

Answer

Add a DataflowJobStatusSensor in the DAG that waits for job completion and retries if failed.

Answer

Configure the Dataflow pipeline to automatically retry on failure using the --numberOfWorkerHarnessThreads option.

Answer

Use a Cloud Function triggered by Cloud Scheduler to re-launch the pipeline if the Dataflow job fails.

Question 13

You are optimizing a BigQuery query that scans 1 TB of data every day. The query joins a large fact table (partitioned by date) with a small dimension table. You notice that the query always scans the entire fact table, even though you only need the last 7 days of data. Which optimization will MOST reduce the bytes scanned?

Accepted Answer

Add a WHERE clause that filters on the date column used for partitioning.. BigQuery partitioned tables allow partition pruning, where the query engine skips partitions that do not match the filter condition. Adding a WHERE clause on the partitioning column (date) restricts the scan to only the last 7 days' partitions, drastically reducing bytes scanned. This is the most direct and effective optimization for the described scenario.

Answer

Create a materialized view that pre-aggregates the data by day.

Answer

Cluster the fact table on the join key used in the query.

Answer

Change the table to use time-unit column partitioning with a 1-day partition interval.

Question 14

A company uses Cloud Pub/Sub for a real-time data pipeline. The subscription has a backlog of millions of messages that are not being processed quickly enough. In Cloud Monitoring, you observe that the 'subscription/num_undelivered_messages' metric is high and growing, while 'subscription/oldest_unacked_message_age' is also increasing. Which action is MOST likely to reduce the backlog?

Accepted Answer

Increase the number of subscribers or the throughput capacity of the existing subscribers.. The backlog indicates that subscribers cannot keep up with the message flow. Increasing the number of subscribers or scaling their throughput capacity directly addresses the processing bottleneck, allowing messages to be pulled and acknowledged faster. Cloud Pub/Sub scales horizontally, so adding more pull subscribers or increasing the resources of existing ones (e.g., more worker threads, higher CPU/memory) reduces the backlog.

Answer

Delete the subscription and recreate it with a larger message retention duration.

Answer

Reduce the acknowledgment deadline to force faster processing.

Answer

Change the subscription type from push to pull.

Question 15

A company is migrating their on-premises data warehouse to BigQuery. They have a mix of batch and streaming ingestion. The data team wants to optimize query costs. Which THREE practices should they adopt?

Accepted Answer

Use materialized views for frequently executed aggregations.. Option B is correct because materialized views precompute and store the results of frequently executed aggregations, so repeated queries read the smaller cached result set instead of rescanning the full base tables, directly reducing bytes processed and thus query cost. Option C is correct because partitioning a table by a DATE or TIMESTAMP column enables partition pruning, so queries with filters on that column scan only the relevant partitions rather than the entire table, cutting the data billed per query. Option E is correct because clustering on columns frequently used in filters and joins co-locates related data in storage blocks, allowing BigQuery to skip irrelevant blocks and reduce bytes scanned for those predicates. Option A is not correct because flat-rate pricing changes how slots are billed (a committed capacity model) rather than reducing the bytes processed by queries, so it does not itself optimize query cost for this migration. Option D is not correct because capping concurrent queries or slot capacity is a workload-management throttle, not a cost-optimization practice, and it can even slow throughput without lowering bytes-scanned charges.

Answer

Switch to flat-rate pricing to cap slot usage.

Answer

Limit the number of concurrent queries by setting a maximum slot capacity.