PDE Maintaining and Automating Data Workloads Practice Test 3 — 15 Questions

Question 1

A company is migrating their on-premises data warehouse to BigQuery. They have a mix of batch and streaming ingestion. The data team wants to optimize query costs. Which THREE practices should they adopt?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use materialized views for frequently executed aggregations.. Partitioning by date reduces bytes scanned. Using clustered tables improves performance for filter/join queries. Using materialized views can precompute aggregations and reduce scans. Flat-rate pricing is about reservation management, not cost optimization per query. Limiting slots is not a cost optimization; it may cause throttling.

Answer

Switch to flat-rate pricing to cap slot usage.

Answer

Limit the number of concurrent queries by setting a maximum slot capacity.

Question 2

A data engineer is building a Cloud Workflows workflow that orchestrates multiple Cloud Functions and API calls. The workflow should handle transient failures with retries and send a notification to a Pub/Sub topic if the workflow ultimately fails. Which THREE steps should the engineer include in the workflow definition?

Accepted Answer

Use the 'googleapis.pubsub.v1.projects.topics.publish' connector to send a failure notification.. Workflows supports retry policies via 'retry' blocks, conditional steps using 'switch', and Pub/Sub publishing via the 'googleapis.pubsub.v1.projects.topics.publish' connector. The 'try/catch' is not a Workflows construct; instead, use 'step' with 'retry' and 'on_error' for failure handling. 'for' loops are for iteration, not error handling.

Answer

Use a 'for' loop to iterate over retries.

Answer

Use a 'try' / 'catch' block to handle exceptions and route to a failure step.

Question 3

A data engineer needs to orchestrate a complex data pipeline that involves multiple steps including data extraction from Cloud Storage, transformation using Dataflow, and loading into BigQuery. The pipeline has dependencies between tasks and requires monitoring and retries. Which Google Cloud service should be used for orchestration?

Accepted Answer

Cloud Composer. The question asks for a single service to orchestrate a complex data pipeline with multiple steps, task dependencies, retries, and monitoring. Cloud Composer (a managed Apache Airflow service) is specifically designed for such complex workflow orchestration, supporting dependency graphs, automated retries, and built-in monitoring. While other services (Workflows, Cloud Scheduler, Cloud Tasks) can handle simpler orchestration or event-triggered tasks, they lack the full scheduler, retry, and dependency management capabilities required for this use case. Therefore, Cloud Composer is the correct answer.

Answer

Workflows

Answer

Cloud Scheduler

Answer

Cloud Tasks

Question 4

You are running a streaming pipeline with Dataflow that reads from Pub/Sub and writes to BigQuery. You notice that the system lag metric is increasing over time, indicating that messages are taking longer to process. What is the most likely cause and how should you address it?

Accepted Answer

The Dataflow workers are CPU-bound; increase the number of workers or adjust autoscaling settings.. Increasing system lag often means the pipeline is CPU-bound, failing to keep up with the incoming data rate. Updating the pipeline with a higher number of workers (or enabling autoscaling) can resolve this.

Answer

The source Pub/Sub topic has insufficient throughput; increase the number of partitions.

Answer

The BigQuery destination table has too many columns; reduce the number of columns.

Answer

The pipeline uses a batch transform that should be replaced with a streaming transform.

Question 5

A company uses BigQuery flat-rate pricing with 500 slots purchased as a committed use discount. During peak hours, they need additional capacity but do not want to buy more committed slots. They have a secondary project used for ad-hoc queries by analysts. How can they provide burst capacity to the primary project during peak times without increasing committed spend?

Accepted Answer

Create flex slots in the secondary project, create a reservation in the secondary project, and assign the reservation to the primary project.. Flex slots provide temporary capacity without long-term commitment. By creating flex slots in a secondary project and assigning the reservation to the primary project, burst capacity is added during peak times without increasing committed spend. Option B is incorrect because autoscaling can be used with committed use reservations, but it does not provide capacity beyond the purchased slots without incurring additional costs; autoscaling on committed use still uses flex pricing. Option C is incorrect because upgrading to Enterprise Plus does not inherently provide burst capacity without additional slots. Option D is incorrect because purchasing additional committed use slots increases committed spend.

Answer

Enable autoscaling slot management in the primary project's reservation, allowing slots to scale up based on demand.

Answer

Upgrade the primary project's edition to Enterprise Plus to allow bursting.

Answer

Purchase additional committed use slots in the primary project and apply them to the reservation.

Question 6

Your team uses Cloud Composer to run Apache Airflow DAGs. One DAG uses a BigQueryInsertJobOperator to run a query and then uses BigQueryCheckOperator to verify the results. The DAG is failing intermittently because the query result is not ready when the check operator runs. How should you modify the DAG to ensure the check operator runs only after the query completes successfully?

Accepted Answer

Add a BigQueryTableExistenceSensor before the BigQueryCheckOperator to wait for the table to be created.. The BigQueryInsertJobOperator is synchronous by default, so the DAG should already wait for completion. However, if the check operator fails due to data not being written, adding a sensor operator (e.g., BigQueryTableExistenceSensor) between the two tasks ensures the table exists before checking.

Answer

Change the BigQueryInsertJobOperator to use deferrable mode to make it async.

Answer

Use a PythonOperator to call the BigQuery API directly and wait for the job to finish.

Answer

Increase the timeout of the BigQueryCheckOperator.

Question 7

A data engineer needs to share a large BigQuery table with a different team, but wants to minimize storage costs. The table is 1 TB in size and is updated daily. The other team only needs read access to the data as of a specific point in time (e.g., end of each day). Which BigQuery feature should be used to provide a read-only copy without duplicating the entire table?

Accepted Answer

Table snapshot. BigQuery table clones provide a lightweight, writable copy of a table that initially shares storage with the base table and only incurs costs for changes made to the clone. Snapshots are read-only and also share storage but require the base table to be preserved until snapshot expires. For read-only point-in-time access, a snapshot is more appropriate because it is immutable and cost-effective.

Answer

Table clone

Answer

Authorized views

Answer

Time travel using FOR SYSTEM_TIME AS OF

Question 8

You are designing a Dataflow pipeline that reads from Pub/Sub, performs transformations, and writes to BigQuery. The pipeline must handle schema changes in the incoming data (e.g., new fields appearing). The BigQuery schema should evolve automatically to accept new fields without failing. Which approach should you use?

Accepted Answer

Set the table schema to allow unknown fields and use BigQuery's schema auto-update feature in the pipeline.. Using BigQuery's schema auto-detection combined with specifying the write disposition as WRITE_APPEND and allowing unknown fields can handle schema drift. However, the most robust approach is to use a flexible schema in the pipeline and set BigQuery's schema update options to allow automatic addition of nullable fields.

Answer

Use Dataprep to clean and standardise the data before loading into BigQuery.

Answer

Predefine the BigQuery table schema with all possible fields and use UPDATE to add missing fields.

Answer

Use a JavaScript UDF to parse incoming data and map to a fixed schema, ignoring new fields.

Question 9

Which Dataflow feature allows you to package a pipeline into a reusable template that can be deployed with different parameters at runtime?

Accepted Answer

Flex Templates. Dataflow Flex Templates allow you to containerize a pipeline and provide runtime parameters, enabling reusability across different environments or jobs.

Answer

Cloud Dataproc

Answer

Classic Templates

Answer

Dataflow SQL

Question 10

A company runs a Dataflow streaming pipeline that processes financial transactions. They need to apply a new transformation that enriches the data with a lookup from Cloud Bigtable without stopping the pipeline. The pipeline must be updated in a way that minimises data loss and preserves exactly-once semantics. What is the recommended approach?

Accepted Answer

Use the Dataflow update option with the same pipeline name and new version, ensuring the transform is backward compatible.. Dataflow supports updating a streaming pipeline without draining by replacing the pipeline version. Using the --update flag with the same pipeline name and a new version allows the pipeline to be upgraded while preserving the state exactly-once.

Answer

Drain the pipeline first, then start a new pipeline with the updated code.

Answer

Create a new pipeline in parallel and switch the Pub/Sub subscription to the new pipeline.

Answer

Stop the pipeline, update the code, and restart with a new pipeline name.

Question 11

You are monitoring a Dataproc cluster and notice that the cluster utilisation is high, but jobs are running slowly. The cluster uses preemptible workers for cost savings. What is the most likely cause of the performance degradation?

Accepted Answer

The preemptible workers are being preempted frequently, causing task retries and slowdowns.. Preemptible workers can be terminated at any time, causing job restarts and slower execution. The high utilisation indicates many workers are preempted, leading to recomputation.

Answer

The primary workers are using standard disks instead of SSDs.

Answer

The cluster is under-provisioned; increase the number of preemptible workers.

Answer

The cluster is using an older image version; upgrade to the latest.

Question 12

Which BigQuery feature allows you to estimate the cost of a query before running it, by returning the number of bytes that would be processed?

Accepted Answer

--dry_run flag. The --dry_run flag in the BigQuery CLI or the dryRun parameter in the API simulates the query and returns the bytes processed without executing it, allowing cost estimation.

Answer

EXPLAIN statement

Answer

INFORMATION_SCHEMA.JOBS

Answer

Slot estimator

Question 13

Your company stores sensitive customer data in Cloud Storage. You need to inspect the data for personally identifiable information (PII) and de-identify it before sharing with a third party. Which Google Cloud service should you use?

Accepted Answer

Cloud Data Loss Prevention (DLP). Cloud Data Loss Prevention (DLP) is the correct service because it is specifically designed to inspect, classify, and de-identify sensitive data such as PII in Cloud Storage. It provides built-in infoType detectors for over 150 types of PII and supports de-identification techniques like masking, tokenization, and encryption. This directly matches the requirement to inspect and de-identify data before sharing with a third party.

Answer

Security Command Center

Answer

Dataplex

Answer

Cloud KMS

Question 14

You need to set up a BigQuery reservation that provides a baseline of 500 slots for daily workloads and can automatically scale up to 1000 slots during peak times. You want to pay only for the slots used beyond the baseline. Which reservation configuration should you choose?

Accepted Answer

Create a reservation with 500 baseline slots and enable autoscaling up to 1000 slots.. BigQuery reservations support baseline slots with autoscaling, which allows you to set a committed baseline of 500 slots and automatically scale up to 1000 slots during peak demand. You are billed only for the additional slots used beyond the baseline, which matches the requirement of paying only for slots used beyond the baseline.

Answer

Create a reservation with 500 committed slots and another with 500 flex slots.

Answer

Use on-demand pricing with a maximum query cost limit.

Answer

Purchase 1000 committed use slots to ensure consistent capacity.

Question 15

A data engineer needs to monitor a Pub/Sub-based streaming pipeline. Which two Cloud Monitoring metrics should be used to detect a backlog of unprocessed messages? (Choose two.)

Accepted Answer

subscription/oldest_unacked_message_age. The 'subscription/num_undelivered_messages' metric shows the number of messages not yet acknowledged, and 'subscription/oldest_unacked_message_age' indicates how long messages have been waiting. Both help detect backlog.

Answer

topic/byte_cost

Answer

topic/send_request_count

Answer

subscription/ack_message_count