PDE Maintaining and Automating Data Workloads Practice Test 2 — 15 Questions

Question 1

A data engineer is building a Cloud Workflows workflow that orchestrates multiple Cloud Functions and API calls. The workflow should handle transient failures with retries and send a notification to a Pub/Sub topic if the workflow ultimately fails. Which THREE steps should the engineer include in the workflow definition?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use the 'googleapis.pubsub.v1.projects.topics.publish' connector to send a failure notification.. Option B is correct because Cloud Workflows provides built-in connectors such as 'googleapis.pubsub.v1.projects.topics.publish' that let the workflow publish a message to a Pub/Sub topic when the workflow ultimately fails, satisfying the notification requirement. Option C is correct because a 'try'/'catch' block is the Workflows construct for catching exceptions raised by steps and routing execution to a failure-handling step, which is exactly how the workflow detects ultimate failure before notifying Pub/Sub. Option E is correct because the 'retry' block (with max_retries and backoff parameters such as initial_delay and multiplier) is the native Workflows mechanism for handling transient failures on API call steps. Option A is not correct because a 'for' loop is a general iteration construct, not the retry mechanism, and manually looping does not provide the built-in backoff and retry semantics of the 'retry' block. Option D is not correct because a 'switch' step only performs conditional branching on expressions; it does not itself detect or handle exceptions, so it is not the mechanism for routing on failure.

Answer

Use a 'for' loop to iterate over retries.

Answer

Use a 'try' / 'catch' block to handle exceptions and route to a failure step.

Question 2

A company uses BigQuery flat-rate pricing with 500 slots purchased as a committed use discount. During peak hours, they need additional capacity but do not want to buy more committed slots. They have a secondary project used for ad-hoc queries by analysts. How can they provide burst capacity to the primary project during peak times without increasing committed spend?

Accepted Answer

Create flex slots in the secondary project, create a reservation in the secondary project, and assign the reservation to the primary project.. BigQuery flat-rate pricing allows purchasing committed slots. To provide burst capacity without increasing committed spend, you can use flex slots, which are a flexible, short-term commitment. You create flex slots in a secondary project, create a reservation there, and then assign that reservation to the primary project. This allows the primary project to use the additional slots during peak times.

Answer

Enable autoscaling slot management in the primary project's reservation, allowing slots to scale up based on demand.

Answer

Upgrade the primary project's edition to Enterprise Plus to allow bursting.

Answer

Purchase additional committed use slots in the primary project and apply them to the reservation.

Question 3

Your team uses Cloud Composer to run Apache Airflow DAGs. One DAG uses a BigQueryInsertJobOperator to run a query and then uses BigQueryCheckOperator to verify the results. The DAG is failing intermittently because the query result is not ready when the check operator runs. How should you modify the DAG to ensure the check operator runs only after the query completes successfully?

Accepted Answer

Add a BigQueryTableExistenceSensor before the BigQueryCheckOperator to wait for the table to be created.. The BigQueryInsertJobOperator is synchronous by default, so the DAG should already wait for completion. However, if the check operator fails due to data not being written, adding a sensor operator (e.g., BigQueryTableExistenceSensor) between the two tasks ensures the table exists before checking.

Answer

Change the BigQueryInsertJobOperator to use deferrable mode to make it async.

Answer

Use a PythonOperator to call the BigQuery API directly and wait for the job to finish.

Answer

Increase the timeout of the BigQueryCheckOperator.

Question 4

A data engineer needs to share a large BigQuery table with a different team, but wants to minimize storage costs. The table is 1 TB in size and is updated daily. The other team only needs read access to the data as of a specific point in time (e.g., end of each day). Which BigQuery feature should be used to provide a read-only copy without duplicating the entire table?

Accepted Answer

Table snapshot. BigQuery table clones provide a lightweight, writable copy of a table that initially shares storage with the base table and only incurs costs for changes made to the clone. Snapshots are read-only and also share storage but require the base table to be preserved until snapshot expires. For read-only point-in-time access, a snapshot is more appropriate because it is immutable and cost-effective.

Answer

Table clone

Answer

Authorized views

Answer

Time travel using FOR SYSTEM_TIME AS OF

Question 5

You are designing a Dataflow pipeline that reads from Pub/Sub, performs transformations, and writes to BigQuery. The pipeline must handle schema changes in the incoming data (e.g., new fields appearing). The BigQuery schema should evolve automatically to accept new fields without failing. Which approach should you use?

Accepted Answer

Set the table schema to allow unknown fields and use BigQuery's schema auto-update feature in the pipeline.. Using BigQuery's schema auto-detection combined with specifying the write disposition as WRITE_APPEND and allowing unknown fields can handle schema drift. However, the most robust approach is to use a flexible schema in the pipeline and set BigQuery's schema update options to allow automatic addition of nullable fields.

Answer

Use Dataprep to clean and standardise the data before loading into BigQuery.

Answer

Predefine the BigQuery table schema with all possible fields and use UPDATE to add missing fields.

Answer

Use a JavaScript UDF to parse incoming data and map to a fixed schema, ignoring new fields.

Question 6

You need to set up a BigQuery reservation that provides a baseline of 500 slots for daily workloads and can automatically scale up to 1000 slots during peak times. You want to pay only for the slots used beyond the baseline. Which reservation configuration should you choose?

Accepted Answer

Create a reservation with 500 baseline slots and enable autoscaling up to 1000 slots.. BigQuery reservations support baseline slots with autoscaling, which allows you to set a committed baseline of 500 slots and automatically scale up to 1000 slots during peak demand. You are billed only for the additional slots used beyond the baseline, which matches the requirement of paying only for slots used beyond the baseline.

Answer

Create a reservation with 500 committed slots and another with 500 flex slots.

Answer

Use on-demand pricing with a maximum query cost limit.

Answer

Purchase 1000 committed use slots to ensure consistent capacity.

Question 7

You are configuring Dataplex data quality rules for a BigQuery table. Which three types of rules can be defined using Dataplex's SQL-based rule engine? (Choose three.)

Accepted Answer

Row-level rules (e.g., condition must be true for every row). Dataplex's SQL-based rule engine supports row-level rules that enforce a condition that must be true for every row in a BigQuery table. These rules are defined using a SQL expression that is evaluated per row, and if any row fails the condition, the rule is violated. This allows you to validate data integrity at the most granular level, such as ensuring a column value is always positive.

Answer

Set-level rules (e.g., intersection or difference between two tables)

Answer

Pattern matching rules (e.g., regex on column values)

Question 8

You need to create a reusable Dataflow pipeline for transforming CSV files in Cloud Storage into Avro files in another bucket. The pipeline should be configurable via runtime parameters (e.g., input and output paths). Which approach should you use?

Accepted Answer

Create a Dataflow Flex Template with a Docker image and parameterized metadata.. Dataflow Flex Templates allow you to package your pipeline code and dependencies into a Docker image, and define parameterized metadata (e.g., input and output paths) that are exposed as runtime parameters. This approach provides full customization of the execution environment and supports reusable, configurable pipelines for transforming CSV to Avro across different Cloud Storage buckets.

Answer

Use Cloud Functions triggered by Cloud Storage events.

Answer

Create a Dataflow Classic Template with the pipeline code and parameters.

Answer

Use Cloud Run Jobs to run the transformation as a container.

Question 9

Your Dataflow streaming job is experiencing high system lag. You want to identify the root cause. Which Cloud Monitoring metrics should you examine first? (Choose the best option.)

Accepted Answer

System lag and worker CPU utilization. For streaming Dataflow jobs, system lag measures the maximum time between the event timestamp and when it is processed. High system lag typically indicates that the pipeline cannot keep up with the input rate. Worker CPU utilization is a key metric to check if workers are overloaded. Data freshness is for batch pipelines. Element count alone doesn't show lag. Backlog bytes could be useful but worker CPU is more directly indicative of processing capacity issues.

Answer

Data freshness and worker CPU utilization

Answer

Element count and data freshness

Answer

Backlog bytes and system lag

Question 10

You want to create a cost-efficient snapshot of a large BigQuery table that can be used by other teams for read-only analytics without incurring additional storage costs for the base table data. What should you use?

Accepted Answer

Create a BigQuery table clone of the original table.. A BigQuery table clone creates a read-only, cost-efficient copy of the table that references the underlying storage of the base table, so no additional storage costs are incurred for the base data. Clones are ideal for sharing snapshots for read-only analytics without duplicating storage, and they support time-travel queries within the clone's retention period.

Answer

Create a BigQuery table snapshot of the original table.

Answer

Export the table to Cloud Storage as Avro files and load into a new table.

Answer

Create a view over the original table.

Question 11

You are designing a data pipeline that ingests streaming data from Pub/Sub, processes it with Dataflow, and writes to BigQuery. You need to ensure that schema changes in the incoming data (new fields) are handled without pipeline failure. Which THREE steps should you take? (Choose THREE.)

Accepted Answer

Set the BigQuery table schema to allow automatic addition of new fields by using 'ignore_unknown_values'.. Option A is correct because setting the BigQuery load job or write configuration to ignore_unknown_values allows rows containing new, unrecognized fields to be loaded without failing, so schema drift in the incoming Pub/Sub data does not break the pipeline. Option C is correct because ALTER TABLE ADD COLUMN DDL statements let you explicitly add new nullable columns to the existing BigQuery table schema, which is the standard way to evolve the destination schema for new fields. Option D is correct because configuring the Dataflow pipeline to detect new fields and update the BigQuery table schema (for example via the BigQueryIO write with schema update options or dynamic destinations) keeps the pipeline running and aligns the destination schema with the evolving data. Option B is incorrect because requiring all fields and failing on unknown fields would cause exactly the pipeline failures the scenario wants to avoid. Option E is incorrect because making all columns NULLABLE from the start does not by itself handle new fields that do not yet exist in the table schema; nullability only affects whether existing columns can be empty.

Answer

Configure the BigQuery table to require all fields, causing the pipeline to fail on unknown fields.

Answer

Set all columns in the BigQuery table as NULLABLE from the start.

Question 12

Your Dataflow streaming pipeline is experiencing increasing system lag over time. You have enabled autoscaling and the pipeline is using the default streaming engine. Which metric should you monitor in Cloud Monitoring to determine if the pipeline is falling behind due to slow processing or due to a bottleneck in the output sink?

Accepted Answer

Data freshness. Data freshness measures how far behind the pipeline's output is relative to real time, and it is the metric that reflects whether the sink is keeping up with processed data. When system lag grows but data freshness also degrades, the bottleneck is downstream (the sink); if system lag grows while data freshness stays healthy, the issue is processing. Data freshness is therefore the metric that disambiguates slow processing from a slow output sink.

Answer

Worker CPU utilization

Answer

System lag

Answer

Element count

Question 13

You are running a Dataproc cluster for batch processing. The job is not latency-sensitive and you want to minimize cost. You notice that the cluster is underutilized during the job. Which configuration change would reduce costs most effectively?

Accepted Answer

Use preemptible workers for the worker nodes. Using preemptible workers in Dataproc reduces cost by about 60-80% compared to standard VMs. They are suitable for fault-tolerant batch jobs because they can be terminated at any time.

Answer

Resize the cluster to use larger machines

Answer

Switch to single-node cluster

Answer

Enable autoscaling

Question 14

Your streaming Dataflow pipeline reads from Pub/Sub and writes to BigQuery. You need to update the pipeline to add a new transformation step without losing any messages or causing duplicate processing. Which TWO actions should you take? (Choose 2)

Accepted Answer

Use the Dataflow update command with the same pipeline name and the new template.. Option A is correct because the Dataflow update command (gcloud dataflow jobs update or the Update API) replaces a running streaming job's template in place while preserving the job's state, including Pub/Sub subscription position and BigQuery write state, so messages are not lost and already-processed data is not reprocessed. Option B is correct because taking a snapshot of the streaming pipeline captures the pipeline's state and the unacknowledged messages in the Pub/Sub subscription, providing a rollback point if the update fails and helping avoid message loss or duplicate processing. Option C is wrong because stopping and restarting with a new name creates a new job that loses the original pipeline state and can cause message loss or duplicates. Option D is wrong because draining stops ingestion and processes only the backlog before terminating, which halts the streaming pipeline rather than updating it in place. Option E is wrong because canceling and creating a new pipeline discards the running job's state and Pub/Sub checkpointing, leading to lost or duplicated messages.

Answer

Stop the pipeline, modify it, then restart with a new name.

Answer

Drain the pipeline before making changes.

Answer

Cancel the pipeline and create a new one.

Question 15

You need to monitor the health of a Pub/Sub subscription that feeds into a Dataflow pipeline. Which TWO Cloud Monitoring metrics are most relevant to detect if messages are not being acknowledged promptly? (Choose 2)

Accepted Answer

subscription/oldest_unacked_message_age. Option B, subscription/oldest_unacked_message_age, is correct because it directly measures the age of the oldest unacknowledged message in the subscription backlog; a growing value indicates messages are not being acknowledged promptly, which is exactly the symptom to detect. Option E, subscription/num_undelivered_messages, is correct because it reports the number of messages in the subscription backlog that have not yet been delivered to subscribers, so a persistently high or rising count signals that the Dataflow pipeline is falling behind in processing and acknowledging messages. Option A, subscription/num_outstanding_messages, is not marked correct here because it counts messages delivered to subscribers but not yet acknowledged, which overlaps with backlog detection but is not one of the two selected metrics for this scenario. Option C, topic/send_request_count, measures publish requests to the topic rather than subscription acknowledgment health, so it does not detect unacknowledged messages. Option D, subscription/ack_message_count, counts successful acknowledgments and would only show throughput, not whether messages are lingering unacknowledged.

Answer

subscription/num_outstanding_messages

Answer

topic/send_request_count

Answer

subscription/ack_message_count