PDE Practice Test 9 — 10 Questions

Question 1

A company processes CSV files that are uploaded to Cloud Storage by external partners. Each file is around 500 MB, and they need to be parsed and loaded into BigQuery. The processing must start as soon as the file arrives. What is the most efficient serverless architecture?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Cloud Storage triggers a Cloud Function that publishes events to Pub/Sub; a Dataflow streaming pipeline reads from Pub/Sub and writes to BigQuery.. It combines Cloud Storage event-driven triggers with Pub/Sub for reliable asynchronous message delivery, and uses Dataflow streaming with autoscaling to handle 500 MB files efficiently. This serverless architecture ensures processing starts immediately upon file arrival, scales to handle large files without manual intervention, and leverages BigQuery's streaming inserts for near-real-time data loading.

Answer

Use Cloud Scheduler to periodically check for new files and process them with Dataflow batch jobs.

Answer

Cloud Storage triggers a Dataproc job that reads the file and loads it into BigQuery.

Answer

Cloud Storage triggers a Cloud Function that directly loads the data into BigQuery using the BigQuery API.

Question 2

Which TWO actions are recommended to improve the reliability of a Cloud Dataflow streaming pipeline that processes event data from Pub/Sub?

Accepted Answer

Enable Dataflow Streaming Engine.. Option B is correct because Dataflow Streaming Engine moves pipeline state and shuffle execution out of the worker VMs and into the Dataflow service, which improves reliability by decoupling state management from worker lifecycle events such as restarts, autoscaling, and updates. Option C is correct because enabling exactly-once processing sinks (for example, BigQuery with row-level deduplication via insertion IDs) prevents duplicate or lost records when retries occur, which is essential for reliable event processing from Pub/Sub. Option A is not recommended because a 10-second acknowledgment deadline is too short for many streaming workloads and would cause premature redelivery and duplicate processing rather than improving reliability. Option D is incorrect because disabling autoscaling does not improve reliability and can actually reduce throughput and resilience during load spikes. Option E is incorrect because micro-batching with a small batch size is not a Dataflow reliability best practice and can increase overhead and latency without addressing state or delivery guarantees.

Answer

Use a pull subscription with a 10-second acknowledgment deadline.

Answer

Disable autoscaling to prevent worker churn.

Answer

Use micro-batch processing with a small batch size.

Question 3

A data engineer needs to design a streaming pipeline that ingests events from multiple sources, enriches them with a lookup table stored in BigQuery (updated every hour), and writes the results to a BigQuery table for real-time dashboards. The pipeline must handle late-arriving data up to 1 hour. Which Dataflow feature should be configured to manage late data?

Accepted Answer

Allowed lateness on the window. Allowed lateness on the window is the Dataflow feature that explicitly defines how long the pipeline will retain window state and accept late-arriving elements after the watermark has passed the end of the window. Setting allowed lateness to 1 hour lets Dataflow keep the window's state for that duration, so events arriving up to 60 minutes late are still processed and emitted with correct results. This directly satisfies the requirement to handle late data up to 1 hour without dropping or misassigning events.

Answer

A custom watermark estimation function

Answer

Using side inputs with a periodic refresh

Answer

A trigger that fires on every late element

Question 4

A retail company ingests point-of-sale events from thousands of stores into Cloud Pub/Sub. They need to process these events in a streaming Dataflow pipeline that enriches each event with store metadata from a slowly changing BigQuery table. The enrichment table is updated only once per day. The pipeline must minimize latency and avoid querying BigQuery for every event. Which approach should they use?

Accepted Answer

Load the BigQuery metadata table into a side input as a key-value map, refreshing it periodically via a scheduled pipeline.. The correct approach is to load the slowly changing metadata into a side input that is refreshed periodically. This allows the streaming pipeline to enrich each event without querying BigQuery per event, keeping latency low. Side inputs are a standard Dataflow pattern for enriching streaming data with reference data that changes infrequently, and they avoid the high cost and latency of per-element external calls.

Answer

Use a CoGroupByKey transform to join the streaming events with a bounded read of the metadata table.

Answer

Configure the pipeline to call the BigQuery Storage Read API for each event to fetch the latest metadata.

Answer

Use BigQueryIO.Read with a query that joins the events to the metadata table, and apply the join in the pipeline.

Question 5

A company runs a critical batch pipeline using Cloud Dataflow. The pipeline processes financial transactions and runs every hour. Recently, some runs have failed due to transient errors (e.g., network timeouts). The engineer wants to automatically retry failed runs without manual intervention. The pipeline is launched from a Cloud Composer DAG using DataflowPythonOperator. What is the BEST way to handle retries?

Accepted Answer

Set the 'retries' parameter in the DAG's default_args to a positive integer.. Cloud Composer (Apache Airflow) natively supports task-level retries via the 'retries' parameter in default_args. When a DataflowPythonOperator fails due to a transient error, Airflow automatically re-executes the task up to the specified number of retries, without requiring custom sensors or external triggers. This is the simplest and most reliable mechanism for handling transient failures in a DAG-driven pipeline.

Answer

Add a DataflowJobStatusSensor in the DAG that waits for job completion and retries if failed.

Answer

Configure the Dataflow pipeline to automatically retry on failure using the --numberOfWorkerHarnessThreads option.

Answer

Use a Cloud Function triggered by Cloud Scheduler to re-launch the pipeline if the Dataflow job fails.

Question 6

An organization uses BigQuery on-demand pricing. To control costs, they want to estimate the bytes processed by a query before running it. Which command or method should they use?

Accepted Answer

Use the bq query --dry_run command. The bq query --dry_run flag validates a query and returns the estimated bytes that would be processed without actually executing it or incurring charges. This is the standard method to estimate on-demand BigQuery costs before running a query, since on-demand pricing is based on bytes processed.

Answer

Use bq ls to list table sizes

Answer

Use BigQuery reservations to get cost estimate

Answer

Use INFORMATION_SCHEMA.JOBS_BY_PROJECT to view past costs

Question 7

A team uses Vertex AI AutoML Tables to train a model. They need to deploy the model for real-time predictions with high availability. Which deployment configuration should they use?

Accepted Answer

Deploy to a Vertex AI Endpoint with multiple replicas and auto-scaling. For real-time predictions with high availability, you need a deployment that can handle traffic spikes and failover. Deploying to a Vertex AI Endpoint with multiple replicas and auto-scaling ensures that the model is served from multiple instances, providing redundancy and the ability to scale up or down based on demand. This configuration meets the high-availability requirement by distributing load and automatically recovering from instance failures.

Answer

Export as a Cloud Function

Answer

Deploy to a Vertex AI Endpoint with 1 replica

Answer

Use a Vertex AI Batch Prediction job

Question 8

A data engineer needs to inspect a BigQuery table for sensitive data such as credit card numbers and email addresses before sharing it with a third party. The engineer also wants to de-identify the data by masking the sensitive columns. Which Google Cloud service should be used?

Accepted Answer

Cloud DLP. Cloud DLP (Data Loss Prevention) is the correct service because it is specifically designed to inspect, classify, and de-identify sensitive data such as credit card numbers and email addresses. It provides built-in infoType detectors for over 150 types of sensitive data and supports masking, tokenization, and other de-identification techniques. The engineer can use Cloud DLP to scan BigQuery tables and then apply transformations to mask the sensitive columns before sharing.

Answer

Dataplex

Answer

BigQuery column-level security

Answer

Data Catalog

Question 9

An organization wants to integrate BigQuery Omni to query data stored in AWS S3. They have set up the necessary connections. What is the primary benefit of using BigQuery Omni over simply copying the data to BigQuery?

Accepted Answer

Ability to use BigQuery ML models on data in S3 without moving data.. BigQuery Omni allows you to query data across clouds without moving it, providing a unified analytics experience. It reduces data egress costs and avoids duplication.

Answer

Automatic encryption of data at rest in S3.

Answer

Lower latency queries due to in-memory caching.

Answer

Support for real-time streaming inserts into S3.

Question 10

A media company streams real-time viewer data from Pub/Sub to BigQuery using a Dataflow pipeline. They need to handle occasional malformed messages without losing valid data. Which pattern should they implement?

Accepted Answer

Implement a dead letter sink to store malformed messages for later analysis. A dead letter sink (e.g., a separate Pub/Sub topic or a BigQuery error table) allows the Dataflow pipeline to route malformed messages out of the main processing path while continuing to process valid data. This pattern ensures no valid data is lost and provides a durable location for later analysis or reprocessing of the malformed records, which is essential for streaming pipelines where data quality issues are intermittent.

Answer

Raise an exception in the pipeline and stop processing

Answer

Use retry logic in the pipeline to reprocess malformed messages indefinitely

Answer

Discard malformed messages and log an error