PDE Maintaining and Automating Data Workloads Practice Test 3 — 15 Questions

Question 1

A data engineer needs to orchestrate a complex data pipeline that involves multiple steps including data extraction from Cloud Storage, transformation using Dataflow, and loading into BigQuery. The pipeline has dependencies between tasks and requires monitoring and retries. Which Google Cloud service should be used for orchestration?

Accepted Answer

Cloud Composer. Cloud Composer is a managed Apache Airflow service that provides a robust platform for orchestrating complex workflows with task dependencies, retries, and monitoring.

Answer

Workflows

Answer

Cloud Scheduler

Answer

Cloud Tasks

Question 2

You are running a streaming pipeline with Dataflow that reads from Pub/Sub and writes to BigQuery. You notice that the system lag metric is increasing over time, indicating that messages are taking longer to process. What is the most likely cause and how should you address it?

Accepted Answer

The Dataflow workers are CPU-bound; increase the number of workers or adjust autoscaling settings.. Increasing system lag often means the pipeline is CPU-bound, failing to keep up with the incoming data rate. Updating the pipeline with a higher number of workers (or enabling autoscaling) can resolve this.

Answer

The source Pub/Sub topic has insufficient throughput; increase the number of partitions.

Answer

The BigQuery destination table has too many columns; reduce the number of columns.

Answer

The pipeline uses a batch transform that should be replaced with a streaming transform.

Question 3

A company uses BigQuery flat-rate pricing with 500 slots purchased as a committed use discount. During peak hours, they need additional capacity but do not want to buy more committed slots. They have a secondary project used for ad-hoc queries by analysts. How can they provide burst capacity to the primary project during peak times without increasing committed spend?

Accepted Answer

Create flex slots in the secondary project, create a reservation in the secondary project, and assign the reservation to the primary project.. Flex slots allow temporary capacity in a separate project, and can be assigned to the primary project via reservations. This provides burst capacity without committing to long-term purchases.

Answer

Enable autoscaling slot management in the primary project's reservation, allowing slots to scale up based on demand.

Answer

Upgrade the primary project's edition to Enterprise Plus to allow bursting.

Answer

Purchase additional committed use slots in the primary project and apply them to the reservation.

Question 4

Your team uses Cloud Composer to run Apache Airflow DAGs. One DAG uses a BigQueryInsertJobOperator to run a query and then uses BigQueryCheckOperator to verify the results. The DAG is failing intermittently because the query result is not ready when the check operator runs. How should you modify the DAG to ensure the check operator runs only after the query completes successfully?

Accepted Answer

Add a BigQueryTableExistenceSensor before the BigQueryCheckOperator to wait for the table to be created.. The BigQueryInsertJobOperator is synchronous by default, so the DAG should already wait for completion. However, if the check operator fails due to data not being written, adding a sensor operator (e.g., BigQueryTableExistenceSensor) between the two tasks ensures the table exists before checking.

Answer

Change the BigQueryInsertJobOperator to use deferrable mode to make it async.

Answer

Use a PythonOperator to call the BigQuery API directly and wait for the job to finish.

Answer

Increase the timeout of the BigQueryCheckOperator.

Question 5

A data engineer needs to share a large BigQuery table with a different team, but wants to minimize storage costs. The table is 1 TB in size and is updated daily. The other team only needs read access to the data as of a specific point in time (e.g., end of each day). Which BigQuery feature should be used to provide a read-only copy without duplicating the entire table?

Accepted Answer

Table snapshot. BigQuery table clones provide a lightweight, writable copy of a table that initially shares storage with the base table and only incurs costs for changes made to the clone. Snapshots are read-only and also share storage but require the base table to be preserved until snapshot expires. For read-only point-in-time access, a snapshot is more appropriate because it is immutable and cost-effective.

Answer

Table clone

Answer

Authorized views

Answer

Time travel using FOR SYSTEM_TIME AS OF

Question 6

You are designing a Dataflow pipeline that reads from Pub/Sub, performs transformations, and writes to BigQuery. The pipeline must handle schema changes in the incoming data (e.g., new fields appearing). The BigQuery schema should evolve automatically to accept new fields without failing. Which approach should you use?

Accepted Answer

Set the table schema to allow unknown fields and use BigQuery's schema auto-update feature in the pipeline.. Using BigQuery's schema auto-detection combined with specifying the write disposition as WRITE_APPEND and allowing unknown fields can handle schema drift. However, the most robust approach is to use a flexible schema in the pipeline and set BigQuery's schema update options to allow automatic addition of nullable fields.

Answer

Use Dataprep to clean and standardise the data before loading into BigQuery.

Answer

Predefine the BigQuery table schema with all possible fields and use UPDATE to add missing fields.

Answer

Use a JavaScript UDF to parse incoming data and map to a fixed schema, ignoring new fields.

Question 7

Which Dataflow feature allows you to package a pipeline into a reusable template that can be deployed with different parameters at runtime?

Accepted Answer

Flex Templates. Dataflow Flex Templates allow you to containerize a pipeline and provide runtime parameters, enabling reusability across different environments or jobs.

Answer

Cloud Dataproc

Answer

Classic Templates

Answer

Dataflow SQL

Question 8

A company runs a Dataflow streaming pipeline that processes financial transactions. They need to apply a new transformation that enriches the data with a lookup from Cloud Bigtable without stopping the pipeline. The pipeline must be updated in a way that minimises data loss and preserves exactly-once semantics. What is the recommended approach?

Accepted Answer

Use the Dataflow update option with the same pipeline name and new version, ensuring the transform is backward compatible.. Dataflow supports updating a streaming pipeline without draining by replacing the pipeline version. Using the --update flag with the same pipeline name and a new version allows the pipeline to be upgraded while preserving the state exactly-once.

Answer

Drain the pipeline first, then start a new pipeline with the updated code.

Answer

Create a new pipeline in parallel and switch the Pub/Sub subscription to the new pipeline.

Answer

Stop the pipeline, update the code, and restart with a new pipeline name.

Question 9

You are monitoring a Dataproc cluster and notice that the cluster utilisation is high, but jobs are running slowly. The cluster uses preemptible workers for cost savings. What is the most likely cause of the performance degradation?

Accepted Answer

The preemptible workers are being preempted frequently, causing task retries and slowdowns.. Preemptible workers can be terminated at any time, causing job restarts and slower execution. The high utilisation indicates many workers are preempted, leading to recomputation.

Answer

The primary workers are using standard disks instead of SSDs.

Answer

The cluster is under-provisioned; increase the number of preemptible workers.

Answer

The cluster is using an older image version; upgrade to the latest.

Question 10

Which BigQuery feature allows you to estimate the cost of a query before running it, by returning the number of bytes that would be processed?

Accepted Answer

--dry_run flag. The --dry_run flag in the BigQuery CLI or the dryRun parameter in the API simulates the query and returns the bytes processed without executing it, allowing cost estimation.

Answer

EXPLAIN statement

Answer

INFORMATION_SCHEMA.JOBS

Answer

Slot estimator

Question 11

Your company stores sensitive customer data in Cloud Storage. You need to inspect the data for personally identifiable information (PII) and de-identify it before sharing with a third party. Which Google Cloud service should you use?

Accepted Answer

Cloud Data Loss Prevention (DLP). Cloud Data Loss Prevention (DLP) is the correct service because it is specifically designed to inspect, classify, and de-identify sensitive data such as PII in Cloud Storage. It provides built-in infoType detectors for over 150 types of PII and supports de-identification techniques like masking, tokenization, and encryption. This directly matches the requirement to inspect and de-identify data before sharing with a third party.

Answer

Security Command Center

Answer

Dataplex

Answer

Cloud KMS

Question 12

You need to set up a BigQuery reservation that provides a baseline of 500 slots for daily workloads and can automatically scale up to 1000 slots during peak times. You want to pay only for the slots used beyond the baseline. Which reservation configuration should you choose?

Accepted Answer

Create a reservation with 500 baseline slots and enable autoscaling up to 1000 slots.. Option D is correct because BigQuery reservations support baseline slots with autoscaling, which allows you to set a committed baseline of 500 slots and automatically scale up to 1000 slots during peak demand. You are billed only for the additional slots used beyond the baseline, which matches the requirement of paying only for slots used beyond the baseline.

Answer

Create a reservation with 500 committed slots and another with 500 flex slots.

Answer

Use on-demand pricing with a maximum query cost limit.

Answer

Purchase 1000 committed use slots to ensure consistent capacity.

Question 13

A data engineer needs to monitor a Pub/Sub-based streaming pipeline. Which two Cloud Monitoring metrics should be used to detect a backlog of unprocessed messages? (Choose two.)

Accepted Answer

subscription/oldest_unacked_message_age. The 'subscription/num_undelivered_messages' metric shows the number of messages not yet acknowledged, and 'subscription/oldest_unacked_message_age' indicates how long messages have been waiting. Both help detect backlog.

Answer

topic/byte_cost

Answer

topic/send_request_count

Answer

subscription/ack_message_count

Question 14

You are configuring Dataplex data quality rules for a BigQuery table. Which three types of rules can be defined using Dataplex's SQL-based rule engine? (Choose three.)

Accepted Answer

Row-level rules (e.g., condition must be true for every row). Option A is correct because Dataplex's SQL-based rule engine supports row-level rules that enforce a condition that must be true for every row in a BigQuery table. These rules are defined using a SQL expression that is evaluated per row, and if any row fails the condition, the rule is violated. This allows you to validate data integrity at the most granular level, such as ensuring a column value is always positive.

Answer

Set-level rules (e.g., intersection or difference between two tables)

Answer

Pattern matching rules (e.g., regex on column values)

Question 15

A company runs a Dataflow pipeline that processes a high-volume data stream. They notice that the pipeline's worker CPU utilisation is near 100% and the system lag is increasing. Which three actions can improve performance? (Choose three.)

Accepted Answer

Increase the number of workers.. Option B is correct because increasing the number of workers distributes the processing load across more parallel workers, reducing CPU utilization per worker and allowing the pipeline to keep up with the incoming data stream. This directly addresses both high CPU usage and increasing system lag by scaling out horizontally.

Answer

Increase the worker disk size.

Answer

Use batch processing instead of streaming.