PDE Ingesting and Processing the Data Practice Test 6 — 15 Questions

Question 1

A streaming pipeline ingests events from Pub/Sub, enriches them via a slow REST API call, and writes the result to BigQuery. The API has a limit of 10 requests per second per client. The pipeline processes 1000 messages per second. Which approach minimizes latency while respecting API limits?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use a global window with a trigger that fires every second, and inside the DoFn limit concurrent API calls to 10.. Using a global window with a trigger every second groups 1000 messages into a batch, and then throttling concurrent API calls to 10 within the DoFn (e.g., using a fixed-size thread pool) respects the API limit while minimizing latency by processing messages in parallel up to the limit. Option B is wrong because fanning out to multiple API instances doesn't help if the limit is per client; the total requests per second across all instances would still exceed the client limit. Option C is wrong because Dataflow Flex Templates are used to run parameterized pipelines, not to solve throttling issues. Option D is wrong because assigning a random key and using a sliding window distributes messages across workers, but without explicit throttling, the API limit could still be exceeded.

Answer

Fan out the stream to multiple REST API instances using Pub/Sub topic splitting.

Answer

Use a Dataflow Flex Template to run multiple pipelines, each processing a subset of messages.

Answer

Assign each message a random key and use a sliding window of 10 seconds; the API call will be distributed across workers.

Question 2

An organization needs to continuously replicate change data from a MySQL database to BigQuery with sub-minute latency. The database is running on-premises. Which Google Cloud service should they use?

Accepted Answer

Cloud Datastream. Datastream is a serverless change data capture (CDC) service that can replicate from MySQL, PostgreSQL, and Oracle to BigQuery or GCS with low latency. Pub/Sub is a messaging service and does not natively connect to MySQL. Dataflow can process streams but requires a CDC connector. BigQuery Data Transfer Service does not support live CDC from on-prem MySQL.

Answer

Cloud Pub/Sub with a custom connector to MySQL

Answer

BigQuery Data Transfer Service for MySQL

Answer

Cloud Dataflow with a JDBC source

Question 3

A company uses BigQuery's Storage Write API in committed mode to stream data. They notice that some writes are failing with 'DEADLINE_EXCEEDED' errors during peak traffic. The pipeline is a Dataflow job using the Beam SDK. What is the MOST likely cause and solution?

Accepted Answer

The default RPC timeout is too low for the write throughput; increase the timeout in the Storage Write API configuration.. Committed mode requires immediate acknowledgment from BigQuery. Under high traffic, the default timeout may be exceeded. The solution is to increase the timeout or switch to buffered mode, which provides higher throughput by batching. The error is not due to schema mismatch or permissions; those would cause different errors. Pub/Sub is not involved in the write path.

Answer

The Dataflow workers lack sufficient memory; increase worker memory.

Answer

The Pub/Sub subscription is not sending acknowledgments; check the subscription.

Answer

The row schema has changed; update the schema before writing.

Question 4

Which Dataflow feature automatically scales the number of workers based on the pipeline's current workload, and also selects the optimal machine type for each worker based on the pipeline's resource requirements?

Accepted Answer

Dataflow Prime. Dataflow Prime is the correct answer because it is the only Dataflow feature that provides both automatic worker scaling (horizontal autoscaling) and intelligent machine type selection (vertical autoscaling). It dynamically adjusts the number of workers based on the pipeline's current workload and selects the optimal machine type (e.g., CPU, memory, or accelerator-optimized) for each worker based on the pipeline's resource requirements, such as CPU utilization, memory pressure, or shuffle throughput.

Answer

Dataflow Shuffle

Answer

Dataflow Streaming Engine

Answer

Dataflow Flex Templates

Question 5

A data pipeline processes JSON files from Cloud Storage, transforms them using Apache Beam, and writes the output to BigQuery. Some records are malformed and cause the pipeline to fail. How should the engineer handle these errors to ensure the pipeline continues processing while preserving the malformed records for analysis?

Accepted Answer

Catch exceptions in a DoFn and write the malformed records to a separate Cloud Storage bucket using a FileIO sink.. It allows the pipeline to continue processing by catching exceptions within a DoFn and writing malformed records to a separate Cloud Storage bucket using a FileIO sink. This preserves the malformed records for later analysis without blocking the main data flow, which is a standard pattern in Apache Beam for handling dead-letter records. The approach ensures fault tolerance while maintaining data integrity for debugging.

Answer

Set the pipeline to retry malformed records indefinitely until they succeed.

Answer

Use a side input to send malformed records to a dead letter queue in Pub/Sub for later reprocessing.

Answer

Log the malformed records to Stackdriver and skip them in the pipeline.

Question 6

A company wants to orchestrate a multi-step data processing workflow that includes calling a Cloud Run service, waiting for its completion, and then running a BigQuery query. The workflow should be serverless and integrate with Cloud Events. Which Google Cloud service should they use?

Accepted Answer

Cloud Workflows. Cloud Workflows is the correct choice because it is a serverless workflow orchestrator that can coordinate multi-step processes involving Cloud Run and BigQuery. It natively supports waiting for asynchronous operations (like Cloud Run job completion) via its 'call' and 'wait' steps, and it can trigger subsequent steps such as BigQuery queries. Additionally, Cloud Workflows can be triggered by Cloud Events, making it fully integrated with the event-driven architecture described.

Answer

Eventarc

Answer

Cloud Composer

Answer

Cloud Dataflow

Question 7

A data analyst needs to transform nested and repeated fields in BigQuery. They have a table with a column of type ARRAY>. Which SQL function should they use to flatten the array into individual rows for analysis?

Accepted Answer

UNNEST. UNNEST is used to flatten arrays into rows. STRUCT is used to group fields. CAST is for type conversion. REPLACE is for string replacement.

Answer

STRUCT

Answer

CAST

Answer

REPLACE

Question 8

A Dataflow pipeline reads from Pub/Sub, applies a keyed stateful ParDo that uses state variables to deduplicate events based on event ID, and writes to BigQuery. During a pipeline update, some events are duplicated in BigQuery. The state is not preserved across updates. Which configuration ensures exactly-once semantics during updates?

Accepted Answer

Take a snapshot of the pipeline before updating, then start the new pipeline from the snapshot.. Draining the pipeline stops it and completes processing in-flight, then the updated pipeline can start fresh. However, because state is lost, duplicates may still occur if the new pipeline processes events that were already committed. To preserve state, use snapshotting: take a snapshot before update and start the new pipeline from the snapshot. BigQuery's Storage Write API with exactly-once semantics can help at the sink but does not prevent duplicate processing if state is lost. As long as the deduplication state is recovered from the snapshot, duplicates are avoided.

Answer

Drain the pipeline and start the updated pipeline; all in-flight data will be processed.

Answer

Cancel the pipeline and restart it; Pub/Sub subscriptions will be rewound.

Answer

Use the Storage Write API's exactly-once delivery mode.

Question 9

A company wants to use dbt (data build tool) to transform data in BigQuery. They have a Cloud Storage bucket containing raw CSV files that are loaded daily into BigQuery via an external table. Which dbt feature should they use to modularize the transformation logic and handle dependencies between models?

Accepted Answer

dbt models with ref(). C is correct because dbt models with the `ref()` function allow you to modularize SQL transformation logic and automatically handle dependencies between models. When you use `ref('model_name')`, dbt builds a dependency graph, ensuring models are executed in the correct order based on their references. This is essential for transforming raw data from an external table into a structured, analytics-ready dataset in BigQuery.

Answer

dbt tests

Answer

dbt snapshots

Answer

dbt seeds

Question 10

A company is building a real-time anomaly detection pipeline using Dataflow. Events are ingested from Pub/Sub, and the pipeline must compute a sliding window average every minute over a 1-hour window. Which TWO configurations are required for this pipeline? (Choose 2)

Accepted Answer

Set the pipeline to use event time for watermarking.. A sliding window of 1-hour length with a 1-minute slide period fits the requirement (every minute, compute over last hour). Fixed window of 1 minute would compute only per-minute, not sliding. Using stateful processing with timers is an alternative but not standard for sliding windows. Dataflow's default watermark is based on event time; processing time would cause incorrect results. The window type and period are the key.

Answer

Use a Fixed window of 1 minute.

Answer

Use stateful processing with a custom timer.

Answer

Set the pipeline to use processing time for watermarking.

Question 11

A retail company wants to trigger a Cloud Run service whenever a new CSV file is uploaded to a specific Cloud Storage bucket. Which THREE components are needed to set up this event-driven architecture? (Choose 3)

Accepted Answer

Cloud Storage bucket with notifications enabled. Cloud Storage buckets must have notifications enabled to publish events to Pub/Sub when objects are created. Without enabling notifications, the bucket cannot emit events that trigger downstream services. This is typically done by configuring the bucket to send notifications to a Pub/Sub topic for each new object upload.

Answer

Cloud Dataflow pipeline

Answer

Cloud Pub/Sub topic

Question 12

A company is migrating on-premises Apache Kafka workloads to Google Cloud. They want to minimize changes to existing producer and consumer applications while leveraging managed services. Which TWO services should they consider? (Choose 2)

Accepted Answer

Dataproc with Apache Kafka. Kafka on Dataproc provides a managed Kafka cluster that is fully compatible, minimizing application changes. Confluent Cloud on Google Cloud can be used but is not a Google-managed service; however, it is a viable partner solution. Pub/Sub is not Kafka API-compatible. Dataflow is not a replacement for Kafka. BigQuery is a data warehouse, not a streaming broker.

Answer

BigQuery

Answer

Cloud Pub/Sub

Answer

Cloud Dataflow

Question 13

A data engineer needs to ingest on-premises Oracle CDC data into BigQuery in near real-time with minimal operational overhead. Which service should they use?

Accepted Answer

Datastream. Datastream is purpose-built for streaming change data capture (CDC) from Oracle and other sources into BigQuery with near-real-time latency and minimal operational overhead. It handles schema propagation, checkpointing, and automatic retries, eliminating the need to manage custom ingestion pipelines.

Answer

Pub/Sub + Dataflow

Answer

Storage Transfer Service

Answer

Transfer Appliance

Question 14

A company is using Pub/Sub to ingest clickstream events and Dataflow to write to BigQuery. They observe that some events are malformed and cause the pipeline to fail. They need a solution that captures malformed events without blocking the pipeline and allows reprocessing later. Which Dataflow pattern should they implement?

Accepted Answer

Write malformed events to a dead letter sink (e.g., another Pub/Sub topic or GCS bucket) and continue processing healthy events. Dead letter sinks (DLQ) are the standard pattern for handling bad records in Dataflow. The pipeline writes malformed records to a separate sink (e.g., Pub/Sub topic or GCS) for later analysis. Side inputs are for enriching data, not error handling. Reshuffle doesn't apply. Output tags (side outputs) can also be used, but explicit dead letter pattern is more standard.

Answer

Use a side input to filter malformed events before the main pipeline

Answer

Use the Reshuffle transform to reattempt failures

Answer

Use logging alerts to notify the team and stop the pipeline on error

Question 15

A data engineer needs to transfer 500 TB of archival data from an on-premises NAS to Cloud Storage. The on-premises network has limited bandwidth (100 Mbps). Which transfer method should they recommend?

Accepted Answer

Transfer Appliance. The Transfer Appliance is a physical device designed for large-scale data transfers (up to petabytes) when network bandwidth is insufficient. With 500 TB of data and only 100 Mbps bandwidth, the theoretical transfer time would be over 500 days, making any online transfer method impractical. The Transfer Appliance bypasses network constraints entirely by shipping the data physically to Google Cloud.

Answer

Storage Transfer Service for on-premises

Answer

gsutil rsync

Answer

Dataflow pipeline reading from NAS