PDE Ingesting and Processing the Data Practice Test 10 — 15 Questions

Question 1

A data pipeline processes JSON files from Cloud Storage, transforms them using Apache Beam, and writes the output to BigQuery. Some records are malformed and cause the pipeline to fail. How should the engineer handle these errors to ensure the pipeline continues processing while preserving the malformed records for analysis?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Catch exceptions in a DoFn and write the malformed records to a separate Cloud Storage bucket using a FileIO sink.. It allows the pipeline to continue processing by catching exceptions within a DoFn and writing malformed records to a separate Cloud Storage bucket using a FileIO sink. This preserves the malformed records for later analysis without blocking the main data flow, which is a standard pattern in Apache Beam for handling dead-letter records. The approach ensures fault tolerance while maintaining data integrity for debugging.

Answer

Set the pipeline to retry malformed records indefinitely until they succeed.

Answer

Use a side input to send malformed records to a dead letter queue in Pub/Sub for later reprocessing.

Answer

Log the malformed records to Stackdriver and skip them in the pipeline.

Question 2

A company wants to orchestrate a multi-step data processing workflow that includes calling a Cloud Run service, waiting for its completion, and then running a BigQuery query. The workflow should be serverless and integrate with Cloud Events. Which Google Cloud service should they use?

Accepted Answer

Cloud Workflows. Cloud Workflows is the correct choice because it is a serverless workflow orchestrator that can coordinate multi-step processes involving Cloud Run and BigQuery. It natively supports waiting for asynchronous operations (like Cloud Run job completion) via its 'call' and 'wait' steps, and it can trigger subsequent steps such as BigQuery queries. Additionally, Cloud Workflows can be triggered by Cloud Events, making it fully integrated with the event-driven architecture described.

Answer

Eventarc

Answer

Cloud Composer

Answer

Cloud Dataflow

Question 3

A data analyst needs to transform nested and repeated fields in BigQuery. They have a table with a column of type ARRAY>. Which SQL function should they use to flatten the array into individual rows for analysis?

Accepted Answer

UNNEST. UNNEST is used to flatten arrays into rows. STRUCT is used to group fields. CAST is for type conversion. REPLACE is for string replacement.

Answer

STRUCT

Answer

CAST

Answer

REPLACE

Question 4

A company wants to use dbt (data build tool) to transform data in BigQuery. They have a Cloud Storage bucket containing raw CSV files that are loaded daily into BigQuery via an external table. Which dbt feature should they use to modularize the transformation logic and handle dependencies between models?

Accepted Answer

dbt models with ref(). C is correct because dbt models with the `ref()` function allow you to modularize SQL transformation logic and automatically handle dependencies between models. When you use `ref('model_name')`, dbt builds a dependency graph, ensuring models are executed in the correct order based on their references. This is essential for transforming raw data from an external table into a structured, analytics-ready dataset in BigQuery.

Answer

dbt tests

Answer

dbt snapshots

Answer

dbt seeds

Question 5

A company is building a real-time anomaly detection pipeline using Dataflow. Events are ingested from Pub/Sub, and the pipeline must compute a sliding window average every minute over a 1-hour window. Which TWO configurations are required for this pipeline? (Choose 2)

Accepted Answer

Set the pipeline to use event time for watermarking.. Option A is correct because a sliding window over event data must be based on event time so that late-arriving events are assigned to the correct window and watermarks track the progress of event time rather than the wall-clock time at which elements are processed. Option B is correct because the requirement is a 1-hour window that is recomputed every minute, which is exactly a SlidingWindow with a duration of 1 hour and a period (slide) of 1 minute. Option C is incorrect because a FixedWindow of 1 minute produces independent non-overlapping 1-minute aggregates, not a 1-hour average refreshed each minute. Option D is incorrect because stateful processing with a custom timer is a lower-level mechanism and is not required when Apache Beam's built-in sliding windows already express the desired semantics. Option E is incorrect because processing-time watermarking ignores the event timestamps and would misassign late or out-of-order Pub/Sub messages, breaking the anomaly detection logic.

Answer

Use a Fixed window of 1 minute.

Answer

Use stateful processing with a custom timer.

Answer

Set the pipeline to use processing time for watermarking.

Question 6

A data engineer needs to ingest on-premises Oracle CDC data into BigQuery in near real-time with minimal operational overhead. Which service should they use?

Accepted Answer

Datastream. Datastream is purpose-built for streaming change data capture (CDC) from Oracle and other sources into BigQuery with near-real-time latency and minimal operational overhead. It handles schema propagation, checkpointing, and automatic retries, eliminating the need to manage custom ingestion pipelines.

Answer

Pub/Sub + Dataflow

Answer

Storage Transfer Service

Answer

Transfer Appliance

Question 7

A company is using Pub/Sub to ingest clickstream events and Dataflow to write to BigQuery. They observe that some events are malformed and cause the pipeline to fail. They need a solution that captures malformed events without blocking the pipeline and allows reprocessing later. Which Dataflow pattern should they implement?

Accepted Answer

Write malformed events to a dead letter sink (e.g., another Pub/Sub topic or GCS bucket) and continue processing healthy events. The dead letter sink pattern is the canonical Dataflow approach for handling malformed or unparseable records: instead of throwing an exception that fails the bundle (and potentially the whole pipeline), the transform routes the bad record to a secondary sink such as a Pub/Sub topic or GCS bucket while healthy events continue downstream. This preserves pipeline throughput and keeps the malformed payloads available for later inspection, correction, and reprocessing. It is the standard implementation of the dead-letter queue pattern in Apache Beam.

Answer

Use a side input to filter malformed events before the main pipeline

Answer

Use the Reshuffle transform to reattempt failures

Answer

Use logging alerts to notify the team and stop the pipeline on error

Question 8

A data engineer needs to transfer 500 TB of archival data from an on-premises NAS to Cloud Storage. The on-premises network has limited bandwidth (100 Mbps). Which transfer method should they recommend?

Accepted Answer

Transfer Appliance. The Transfer Appliance is a physical device designed for large-scale data transfers (up to petabytes) when network bandwidth is insufficient. With 500 TB of data and only 100 Mbps bandwidth, the theoretical transfer time would be over 500 days, making any online transfer method impractical. The Transfer Appliance bypasses network constraints entirely by shipping the data physically to Google Cloud.

Answer

Storage Transfer Service for on-premises

Answer

gsutil rsync

Answer

Dataflow pipeline reading from NAS

Question 9

A data engineer is designing a Dataflow pipeline that reads from Pub/Sub and writes to BigQuery using the Storage Write API in exactly-once mode. The pipeline must handle late-arriving data (up to 1 hour) and maintain correct aggregation results. Which trigger configuration should they use?

Accepted Answer

After watermark trigger with allowed lateness of 1 hour. The After watermark trigger with allowed lateness of 1 hour is the correct choice because it emits results once the watermark passes the end of the window, while still accepting late-arriving events up to 1 hour past the watermark and including them in the correct event-time window. This aligns with the requirement to handle up to 1 hour of late data and maintain correct aggregation results. Note that exactly-once delivery to BigQuery is provided by the Storage Write API sink itself, not by the trigger; the trigger choice affects completeness and correctness of the aggregation, not the exactly-once guarantee.

Answer

Default trigger without allowed lateness

Answer

After count trigger with count of 1000

Answer

Fixed time trigger every 5 minutes

Question 10

A company wants to use dbt to transform data in BigQuery. Their source data is loaded daily into staging tables. They need to run dbt transformations on a schedule and only process tables that have changed. Which dbt feature should they use?

Accepted Answer

dbt incremental models. dbt incremental models are designed to process only new or changed data since the last run, rather than rebuilding the entire table. This is ideal for daily transformations on staging tables where only some data changes. Incremental models use a unique key and a strategy (e.g., merge, delete+insert) to update the target table efficiently.

Answer

dbt snapshots

Answer

dbt seeds

Answer

dbt sources

Question 11

A data engineer needs to create a Dataflow pipeline template that can be reused across multiple environments (dev, staging, prod) with different parameters (e.g., input Pub/Sub topic, output BigQuery table). Which template type should they use?

Accepted Answer

Flex Template. Flex Templates (B) are the correct choice because they package a Dataflow pipeline as a Docker image, allowing environment-specific parameters (e.g., Pub/Sub topic, BigQuery table) to be passed at runtime via the --parameters flag. This enables true reusability across dev, staging, and prod without modifying the template code, unlike Classic Templates which require compile-time parameterization.

Answer

Dataflow Prime

Answer

Classic Template

Answer

Cloud Composer workflow template

Question 12

A company uses Eventarc to trigger a Cloud Run service when new objects appear in a GCS bucket. Recently, the Cloud Run service has been failing with 429 errors (too many requests) during high-velocity uploads. They need to handle the load without losing events. What should they do?

Accepted Answer

Configure the Eventarc trigger to send events to a Pub/Sub topic, and have the Cloud Run service pull from Pub/Sub. Sending events to a Pub/Sub topic decouples event production from consumption. Pub/Sub acts as a buffer that can absorb spikes in event volume, and the Cloud Run service can pull messages at its own pace, preventing 429 errors. This also ensures no events are lost, as Pub/Sub retains unacknowledged messages and retries delivery.

Answer

Use Cloud Functions instead of Cloud Run

Answer

Increase the maximum number of retries on the Eventarc trigger

Answer

Increase the Cloud Run service's request timeout

Question 13

A data engineer needs to query a BigQuery table that contains an array of structs. They want to expand the array into separate rows for each element. Which SQL function should they use?

Accepted Answer

UNNEST. UNNEST is the BigQuery SQL function that flattens an array into a set of rows, one per array element. When applied to an array of structs, it expands each struct into its own row, allowing the struct fields to be selected as columns.

Answer

STRUCT

Answer

ARRAY_AGG

Answer

SPLIT

Question 14

A data engineer is using Spark on Dataproc to process a large dataset. They notice the job is slow due to excessive shuffling. They want to optimize the job by using a more efficient data structure that reduces serialization overhead and provides better memory management. Which Spark API should they use?

Accepted Answer

DataFrames or Datasets. DataFrames and Datasets are built on Spark SQL's Catalyst optimizer and Tungsten execution engine, which provide schema-aware encoding and off-heap memory management. This dramatically reduces serialization overhead compared to RDDs and enables optimizations like predicate pushdown and shuffle partitioning improvements, directly addressing excessive shuffling.

Answer

Spark SQL

Answer

Spark Streaming

Answer

RDDs

Question 15

A company needs to stream real-time user activity data from their application into BigQuery for immediate dashboarding. They want to minimize latency (under 5 seconds) and ensure exactly-once delivery. Which TWO options should they consider? (Choose 2)

Accepted Answer

Use BigQuery Storage Write API in committed mode. Option B is correct because the BigQuery Storage Write API in committed mode provides exactly-once semantics via write streams and offsets, and it supports low-latency streaming ingestion suitable for sub-5-second dashboarding. Option E is correct because Pub/Sub plus Dataflow is the canonical Google Cloud streaming pipeline: Pub/Sub ingests events durably, and Dataflow's BigQueryIO in STREAMING mode with exactly-once processing guarantees deduplication and exactly-once writes to BigQuery. Option A is not ideal because Cloud Functions invoking the BigQuery REST API (tabledata.insertAll) does not provide exactly-once delivery and adds per-event overhead and cold-start latency. Option C is wrong because legacy streaming inserts offer at-least-once semantics and can produce duplicate rows, violating the exactly-once requirement. Option D is not the best fit because running Kafka on Dataproc adds operational complexity, and the Kafka connector typically relies on the Storage Write API or streaming inserts without inherently guaranteeing exactly-once end-to-end delivery in this scenario.

Answer

Use Cloud Functions to receive events and call the BigQuery REST API

Answer

Use BigQuery legacy streaming inserts directly from the application

Answer

Use Apache Kafka on Dataproc and write to BigQuery via the BigQuery Kafka connector