PDE Maintaining and Automating Data Workloads Practice Test 2 — 15 Questions

Question 1

A company runs BigQuery workloads with varying demand. They want to use flat-rate pricing with baseline slots and the ability to burst during peak times. Which TWO actions should they take? (Choose 2)

Accepted Answer

Use flex slots for short-term bursts. To achieve flat-rate pricing with baseline slots and bursting, purchase committed use reservations for the baseline (option D) to get a discount on consistent usage, and use flex slots for short-term bursts (option B) which are also flat-rate. Option E is incorrect because autoscaling with reservations can use idle committed slots for bursts, but it may still fall back to on-demand pricing if no idle slots are available, so it does not ensure that all bursts are covered by flat-rate pricing. Flex slots (option B) guarantee flat-rate bursting.

Answer

Use on-demand pricing for bursting

Answer

Set a maximum number of slots per query

Answer

Create a reservation with baseline + autoscaling slots

Question 2

A company uses Cloud Composer (Airflow) to orchestrate pipelines. They want to implement a pattern where a task polls for a file arrival in Cloud Storage and then triggers subsequent tasks. Which THREE Airflow concepts are essential? (Choose 3)

Accepted Answer

Sensors (e.g., GCSObjectExistenceSensor). A is correct because Sensors are a specialized type of operator designed to wait for a specific condition, such as file arrival in Cloud Storage. The `GCSObjectExistenceSensor` in Cloud Composer (Airflow) polls Google Cloud Storage at a configurable interval until the target file exists, making it the precise tool for this file-arrival polling pattern.

Answer

Operators (e.g., PythonOperator)

Answer

SubDAGs for grouping tasks

Question 3

A data engineer is building a batch pipeline that runs daily using Cloud Composer. The pipeline has three tasks: extract data from Cloud Storage, transform data using Dataflow, and load the transformed data into BigQuery. The engineer wants to ensure that the Dataflow job only starts after the extraction task completes successfully, and the load task only starts after the Dataflow job finishes. How should the engineer define the task dependencies in the Airflow DAG?

Accepted Answer

extract >> transform >> load. Airflow uses the bitshift operator (>>) to define task dependencies in a linear sequence. The DAG must ensure that the extract task completes before the transform task starts, and the transform task completes before the load task starts. This is achieved by chaining the tasks in order: extract >> transform >> load, which enforces the required sequential execution.

Answer

extract >> [transform, load]

Answer

transform >> extract >> load

Answer

extract >> load >> transform

Question 4

You need to schedule a simple workflow that fetches data from an API every hour, transforms it using Cloud Functions, and writes the result to Cloud Storage. The workflow has no complex branching or retry logic beyond basic retries. Which orchestration service is the MOST cost-effective and simplest to implement?

Accepted Answer

Workflows. Workflows is serverless, pay-per-execution, and defined in YAML/JSON. It integrates natively with Cloud Functions and Cloud Storage. Cloud Composer is overkill for simple linear workflows and incurs cluster costs. Cloud Scheduler alone cannot orchestrate multiple steps. Dataflow is for data processing, not orchestration.

Answer

Cloud Scheduler

Answer

Cloud Composer

Answer

Dataflow

Question 5

A company runs a streaming Dataflow pipeline that reads from Pub/Sub, enriches data with a side input from BigQuery, and writes to BigQuery. After updating the pipeline code (adding a new field to the output), the engineer notices that the new pipeline version is not picking up the updated code because the job was started from a template. The engineer wants to update the streaming pipeline without draining it. What should the engineer do?

Accepted Answer

Use the gcloud dataflow jobs update command with the new Flex Template.. The `gcloud dataflow jobs update` command allows you to update a running streaming Dataflow pipeline with a new Flex Template without draining or stopping the job. This command performs an in-place update, preserving the job's state and checkpointing, so the pipeline continues processing with the new code. Since the original job was started from a template, using this command with the new Flex Template ensures the updated code is picked up seamlessly.

Answer

Stop the pipeline, update the template, and restart with the same job name.

Answer

Modify the original template and redeploy it as a new job with the same pipeline name.

Answer

Use the gcloud dataflow jobs drain command, then restart with the new template.

Question 6

You are monitoring a streaming Dataflow pipeline that reads from Pub/Sub and writes to BigQuery. In Cloud Monitoring, you notice that the 'system_lag' metric is increasing over time and now exceeds 10 minutes. The 'data_watermark' metric shows a steady lag. What is the most likely cause of the increasing system lag?

Accepted Answer

The Dataflow pipeline is underprovisioned with workers, causing processing backlog.. An underprovisioned pipeline lacks sufficient worker resources to process incoming messages at the rate they arrive. This causes a growing backlog in the pipeline's internal buffers, which directly increases the 'system_lag' metric (the time between data ingestion and processing). The 'data_watermark' lag remaining steady indicates that the pipeline is still making progress on event-time processing, but the overall processing capacity is insufficient to keep up with the input rate.

Answer

BigQuery write throughput is throttling the pipeline.

Answer

The Pub/Sub subscription has too many unacknowledged messages.

Answer

The pipeline is using a global window with late data handling.

Question 7

A company wants to share a large BigQuery dataset with a partner for analysis. The partner needs read-only access to a specific snapshot of the data as of a certain point in time, and the company wants to avoid additional storage costs for the partner. What is the most cost-effective approach?

Accepted Answer

Create a BigQuery table snapshot at the desired point in time and share it.. BigQuery table snapshots are zero-copy, read-only, and share the underlying storage with the base table. They do not incur additional storage costs as long as the snapshot is within the time travel window (7 days default). Since the partner requires read-only access to a specific point-in-time snapshot and the company wants to avoid additional storage costs, a table snapshot is the most cost-effective approach. In contrast, table clones are writable and designed for development purposes, not read-only sharing; exporting to Cloud Storage incurs storage and egress costs; and authorized views do not provide a point-in-time snapshot and incur query costs.

Answer

Create a BigQuery table clone at the desired point in time and share it with the partner.

Answer

Export the table to Cloud Storage as Avro and share a signed URL.

Answer

Grant the partner access to the original table with an authorized view.

Question 8

Your organization has a BigQuery flat-rate reservation with 2000 slots. During peak hours, query performance degrades because concurrent queries exceed the available slots. You want to handle these bursts without changing the base reservation. What should you do?

Accepted Answer

Purchase flex slots during peak hours to add capacity temporarily.. Flex slots allow you to temporarily add capacity to a BigQuery flat-rate reservation without committing to a permanent increase. This handles burst workloads during peak hours by adding slots on demand, and you only pay for the time they are used. The base reservation of 2000 slots remains unchanged, meeting the requirement.

Answer

Enable autoscaling on the reservation to automatically add slots up to a maximum.

Answer

Purchase committed use discounts to increase the base reservation to 3000 slots.

Answer

Change the pricing model to on-demand to allow unlimited slots.

Question 9

A data engineer is designing a Dataflow pipeline that reads from a Kafka topic (using Pub/Sub for Kafka) and writes to BigQuery. The data schema may change over time, with new fields appearing. The engineer wants to handle schema drift automatically without failing the pipeline. Which approach should the engineer use?

Accepted Answer

Store the entire record as a single JSON string column in BigQuery and parse it later.. Storing the entire record as a single JSON string column in BigQuery allows the pipeline to accept any schema changes without requiring schema modifications at write time. This approach decouples the ingestion from schema evolution, enabling the data to be parsed later using BigQuery's JSON functions (e.g., JSON_EXTRACT) or by loading into a separate schema-on-read layer. It avoids pipeline failures caused by mismatched fields or type changes.

Answer

Use a Dataflow side input that reads the latest schema from a file and updates the BigQuery schema accordingly.

Answer

Define a UDF in Dataflow that dynamically adjusts the output schema.

Answer

Configure the Dataflow pipeline to use BigQuery's schema autodetect option for each insert.

Question 10

You are troubleshooting a Dataproc cluster that runs nightly Spark jobs. The jobs are failing with out-of-memory errors. You want to reduce costs while fixing the issue. Which combination of actions should you take? (Select the BEST answer.)

Accepted Answer

Increase the number of preemptible worker nodes and use standard machine types for the master.. Preemptible workers are cheaper and can be used for worker nodes, but not for master nodes. Adding more preemptible workers can increase parallelism and reduce memory pressure per worker, but may cause more preemptions. Using high-memory master nodes is not necessary for worker memory issues. Using SSDs for scratch storage can improve performance but does not directly address OOM. Reducing worker count would exacerbate the problem.

Answer

Use SSD persistent disks for all nodes to improve I/O performance.

Answer

Decrease the number of worker nodes and increase the size of each worker.

Answer

Switch to a high-memory machine type for the master node only.

Question 11

A data engineer needs to inspect a BigQuery table for sensitive data such as credit card numbers and email addresses before sharing it with a third party. The engineer also wants to de-identify the data by masking the sensitive columns. Which Google Cloud service should be used?

Accepted Answer

Cloud DLP. Cloud DLP (Data Loss Prevention) is the correct service because it is specifically designed to inspect, classify, and de-identify sensitive data such as credit card numbers and email addresses. It provides built-in infoType detectors for over 150 types of sensitive data and supports masking, tokenization, and other de-identification techniques. The engineer can use Cloud DLP to scan BigQuery tables and then apply transformations to mask the sensitive columns before sharing.

Answer

Dataplex

Answer

BigQuery column-level security

Answer

Data Catalog

Question 12

A company runs a critical batch pipeline using Cloud Dataflow. The pipeline processes financial transactions and runs every hour. Recently, some runs have failed due to transient errors (e.g., network timeouts). The engineer wants to automatically retry failed runs without manual intervention. The pipeline is launched from a Cloud Composer DAG using DataflowPythonOperator. What is the BEST way to handle retries?

Accepted Answer

Set the 'retries' parameter in the DAG's default_args to a positive integer.. Cloud Composer (Apache Airflow) natively supports task-level retries via the 'retries' parameter in default_args. When a DataflowPythonOperator fails due to a transient error, Airflow automatically re-executes the task up to the specified number of retries, without requiring custom sensors or external triggers. This is the simplest and most reliable mechanism for handling transient failures in a DAG-driven pipeline.

Answer

Add a DataflowJobStatusSensor in the DAG that waits for job completion and retries if failed.

Answer

Configure the Dataflow pipeline to automatically retry on failure using the --numberOfWorkerHarnessThreads option.

Answer

Use a Cloud Function triggered by Cloud Scheduler to re-launch the pipeline if the Dataflow job fails.

Question 13

You are optimizing a BigQuery query that scans 1 TB of data every day. The query joins a large fact table (partitioned by date) with a small dimension table. You notice that the query always scans the entire fact table, even though you only need the last 7 days of data. Which optimization will MOST reduce the bytes scanned?

Accepted Answer

Add a WHERE clause that filters on the date column used for partitioning.. Using a WHERE clause on the partitioning column (e.g., date) allows BigQuery to perform partition pruning, drastically reducing scanned bytes. Clustering on the join key can improve performance but does not reduce bytes scanned as much as partition pruning. Materialized views precompute aggregations but may not reduce scans if the base query still scans full table. Changing to a clustered table does not prune partitions automatically.

Answer

Create a materialized view that pre-aggregates the data by day.

Answer

Cluster the fact table on the join key used in the query.

Answer

Change the table to use time-unit column partitioning with a 1-day partition interval.

Question 14

A company uses Cloud Pub/Sub for a real-time data pipeline. The subscription has a backlog of millions of messages that are not being processed quickly enough. In Cloud Monitoring, you observe that the 'subscription/num_undelivered_messages' metric is high and growing, while 'subscription/oldest_unacked_message_age' is also increasing. Which action is MOST likely to reduce the backlog?

Accepted Answer

Increase the number of subscribers or the throughput capacity of the existing subscribers.. The backlog indicates that subscribers cannot keep up with the message flow. Increasing the number of subscribers or scaling their throughput capacity directly addresses the processing bottleneck, allowing messages to be pulled and acknowledged faster. Cloud Pub/Sub scales horizontally, so adding more pull subscribers or increasing the resources of existing ones (e.g., more worker threads, higher CPU/memory) reduces the backlog.

Answer

Delete the subscription and recreate it with a larger message retention duration.

Answer

Reduce the acknowledgment deadline to force faster processing.

Answer

Change the subscription type from push to pull.

Question 15

A data engineer needs to set up a Dataplex data quality scan to run weekly on a BigQuery table. The scan should check that: (1) the 'email' column is not null, (2) the 'age' column is between 0 and 120, and (3) the 'country_code' column matches a list of valid ISO codes. Which TWO Dataplex features should the engineer use?

Accepted Answer

Schedule the data quality scan using Dataplex scheduling options. The engineer needs to run the scan weekly, so scheduling (A) is required. For the data quality checks, column rules can be used for both the 'age' range check and the 'country_code' set check, as they apply to all values in the column. The 'email' not null check can also be done as a column rule, making a separate row rule unnecessary. Therefore, only A and C are needed.

Answer

Use Cloud DLP to inspect the table for data quality issues

Answer

Create a Dataplex asset and add a tag template to enforce constraints

Answer

Create a row rule for the 'email' column using SQL condition 'email IS NOT NULL'