PDE Ingesting and Processing the Data Practice Test 5 — 15 Questions

Question 1

A data engineer needs to ingest daily Salesforce reports into BigQuery without writing custom code. The reports are exported to an Amazon S3 bucket on a schedule. Which service should they use to automate the transfer?

Accepted Answer

BigQuery Data Transfer Service. The BigQuery Data Transfer Service (BQDTS) is the correct choice because it natively supports scheduled, automatic ingestion of Salesforce reports into BigQuery without requiring any custom code. It connects directly to the Salesforce API, extracts the reports, and loads them into BigQuery tables on a user-defined schedule, handling incremental updates and schema detection automatically.

Answer

Cloud Dataproc

Answer

Cloud Composer

Answer

Cloud Storage Transfer Service

Question 2

A company wants to migrate 500 TB of on-premises archival data to Cloud Storage. The data is stored on a SAN and the network link is limited to 1 Gbps. The migration must complete within 10 days. What is the MOST cost-effective approach?

Accepted Answer

Order a Transfer Appliance, copy data locally, and ship it to Google for ingestion.. Option C is correct because the Transfer Appliance is designed for large-scale data migrations where network bandwidth is insufficient. With 500 TB at 1 Gbps, the theoretical transfer time is over 46 days, far exceeding the 10-day window. The appliance allows you to physically ship the data, bypassing network constraints entirely, making it the most cost-effective and timely solution.

Answer

Set up a Cloud VPN and use rsync over the encrypted connection.

Answer

Use BigQuery Data Transfer Service to load the data directly into BigQuery.

Answer

Use Storage Transfer Service to copy data from on-premises to GCS over the existing network.

Question 3

A streaming pipeline ingests events from Pub/Sub, enriches them via a slow REST API call, and writes the result to BigQuery. The API has a limit of 10 requests per second per client. The pipeline processes 1000 messages per second. Which approach minimizes latency while respecting API limits?

Accepted Answer

Use a global window with a trigger that fires every second, and inside the DoFn limit concurrent API calls to 10.. Using a global window of 1 second groups 1000 messages and then throttles API calls to 10 concurrent requests (e.g., via a fixed-size thread pool in a DoFn). This respects the limit while batching work. Beam does not automatically throttle; using a global window on a single key would create a bottleneck. Fanning out to multiple API endpoints does not help if the limit is per client. Dataflow Flex Templates are irrelevant to throttling.

Answer

Fan out the stream to multiple REST API instances using Pub/Sub topic splitting.

Answer

Use a Dataflow Flex Template to run multiple pipelines, each processing a subset of messages.

Answer

Assign each message a random key and use a sliding window of 10 seconds; the API call will be distributed across workers.

Question 4

An organization needs to continuously replicate change data from a MySQL database to BigQuery with sub-minute latency. The database is running on-premises. Which Google Cloud service should they use?

Accepted Answer

Cloud Datastream. Datastream is a serverless change data capture (CDC) service that can replicate from MySQL, PostgreSQL, and Oracle to BigQuery or GCS with low latency. Pub/Sub is a messaging service and does not natively connect to MySQL. Dataflow can process streams but requires a CDC connector. BigQuery Data Transfer Service does not support live CDC from on-prem MySQL.

Answer

Cloud Pub/Sub with a custom connector to MySQL

Answer

BigQuery Data Transfer Service for MySQL

Answer

Cloud Dataflow with a JDBC source

Question 5

A data engineer is building a Dataflow pipeline in Python that reads from BigQuery, transforms data, and writes to Cloud Storage. The pipeline will be deployed in production. Which approach should they use to ensure the pipeline is reusable across environments with different configuration parameters?

Accepted Answer

Use a Dataflow Flex Template. Option C is correct because Dataflow Flex Templates allow you to package a Docker container with your pipeline code and dependencies, enabling parameterization at runtime via the Dataflow UI, CLI, or API. This makes the pipeline reusable across environments (e.g., dev, staging, prod) by passing different configuration parameters (like project IDs, table names, or output paths) without modifying the code. Flex Templates support custom container images and are the recommended approach for production pipelines that need environment-agnostic deployment.

Answer

Create a separate pipeline for each environment with hardcoded values

Answer

Use a Dataflow Classic Template

Answer

Run the pipeline using the DirectRunner for each environment

Question 6

A company uses BigQuery's Storage Write API in committed mode to stream data. They notice that some writes are failing with 'DEADLINE_EXCEEDED' errors during peak traffic. The pipeline is a Dataflow job using the Beam SDK. What is the MOST likely cause and solution?

Accepted Answer

The default RPC timeout is too low for the write throughput; increase the timeout in the Storage Write API configuration.. Committed mode requires immediate acknowledgment from BigQuery. Under high traffic, the default timeout may be exceeded. The solution is to increase the timeout or switch to buffered mode, which provides higher throughput by batching. The error is not due to schema mismatch or permissions; those would cause different errors. Pub/Sub is not involved in the write path.

Answer

The Dataflow workers lack sufficient memory; increase worker memory.

Answer

The Pub/Sub subscription is not sending acknowledgments; check the subscription.

Answer

The row schema has changed; update the schema before writing.

Question 7

Which Dataflow feature automatically scales the number of workers based on the pipeline's current workload, and also selects the optimal machine type for each worker based on the pipeline's resource requirements?

Accepted Answer

Dataflow Prime. Dataflow Prime is the correct answer because it is the only Dataflow feature that provides both automatic worker scaling (horizontal autoscaling) and intelligent machine type selection (vertical autoscaling). It dynamically adjusts the number of workers based on the pipeline's current workload and selects the optimal machine type (e.g., CPU, memory, or accelerator-optimized) for each worker based on the pipeline's resource requirements, such as CPU utilization, memory pressure, or shuffle throughput.

Answer

Dataflow Shuffle

Answer

Dataflow Streaming Engine

Answer

Dataflow Flex Templates

Question 8

A data pipeline processes JSON files from Cloud Storage, transforms them using Apache Beam, and writes the output to BigQuery. Some records are malformed and cause the pipeline to fail. How should the engineer handle these errors to ensure the pipeline continues processing while preserving the malformed records for analysis?

Accepted Answer

Catch exceptions in a DoFn and write the malformed records to a separate Cloud Storage bucket using a FileIO sink.. Option D is correct because it allows the pipeline to continue processing by catching exceptions within a DoFn and writing malformed records to a separate Cloud Storage bucket using a FileIO sink. This preserves the malformed records for later analysis without blocking the main data flow, which is a standard pattern in Apache Beam for handling dead-letter records. The approach ensures fault tolerance while maintaining data integrity for debugging.

Answer

Set the pipeline to retry malformed records indefinitely until they succeed.

Answer

Use a side input to send malformed records to a dead letter queue in Pub/Sub for later reprocessing.

Answer

Log the malformed records to Stackdriver and skip them in the pipeline.

Question 9

A company wants to orchestrate a multi-step data processing workflow that includes calling a Cloud Run service, waiting for its completion, and then running a BigQuery query. The workflow should be serverless and integrate with Cloud Events. Which Google Cloud service should they use?

Accepted Answer

Cloud Workflows. Cloud Workflows is the correct choice because it is a serverless workflow orchestrator that can coordinate multi-step processes involving Cloud Run and BigQuery. It natively supports waiting for asynchronous operations (like Cloud Run job completion) via its 'call' and 'wait' steps, and it can trigger subsequent steps such as BigQuery queries. Additionally, Cloud Workflows can be triggered by Cloud Events, making it fully integrated with the event-driven architecture described.

Answer

Eventarc

Answer

Cloud Composer

Answer

Cloud Dataflow

Question 10

A data analyst needs to transform nested and repeated fields in BigQuery. They have a table with a column of type ARRAY>. Which SQL function should they use to flatten the array into individual rows for analysis?

Accepted Answer

UNNEST. UNNEST is used to flatten arrays into rows. STRUCT is used to group fields. CAST is for type conversion. REPLACE is for string replacement.

Answer

STRUCT

Answer

CAST

Answer

REPLACE

Question 11

A Dataflow pipeline reads from Pub/Sub, applies a keyed stateful ParDo that uses state variables to deduplicate events based on event ID, and writes to BigQuery. During a pipeline update, some events are duplicated in BigQuery. The state is not preserved across updates. Which configuration ensures exactly-once semantics during updates?

Accepted Answer

Take a snapshot of the pipeline before updating, then start the new pipeline from the snapshot.. Draining the pipeline stops it and completes processing in-flight, then the updated pipeline can start fresh. However, because state is lost, duplicates may still occur if the new pipeline processes events that were already committed. To preserve state, use snapshotting: take a snapshot before update and start the new pipeline from the snapshot. BigQuery's Storage Write API with exactly-once semantics can help at the sink but does not prevent duplicate processing if state is lost. As long as the deduplication state is recovered from the snapshot, duplicates are avoided.

Answer

Drain the pipeline and start the updated pipeline; all in-flight data will be processed.

Answer

Cancel the pipeline and restart it; Pub/Sub subscriptions will be rewound.

Answer

Use the Storage Write API's exactly-once delivery mode.

Question 12

A company wants to use dbt (data build tool) to transform data in BigQuery. They have a Cloud Storage bucket containing raw CSV files that are loaded daily into BigQuery via an external table. Which dbt feature should they use to modularize the transformation logic and handle dependencies between models?

Accepted Answer

dbt models with ref(). C is correct because dbt models with the `ref()` function allow you to modularize SQL transformation logic and automatically handle dependencies between models. When you use `ref('model_name')`, dbt builds a dependency graph, ensuring models are executed in the correct order based on their references. This is essential for transforming raw data from an external table into a structured, analytics-ready dataset in BigQuery.

Answer

dbt tests

Answer

dbt snapshots

Answer

dbt seeds

Question 13

A company is building a real-time anomaly detection pipeline using Dataflow. Events are ingested from Pub/Sub, and the pipeline must compute a sliding window average every minute over a 1-hour window. Which TWO configurations are required for this pipeline? (Choose 2)

Accepted Answer

Set the pipeline to use event time for watermarking.. A sliding window of 1-hour length with a 1-minute slide period fits the requirement (every minute, compute over last hour). Fixed window of 1 minute would compute only per-minute, not sliding. Using stateful processing with timers is an alternative but not standard for sliding windows. Dataflow's default watermark is based on event time; processing time would cause incorrect results. The window type and period are the key.

Answer

Use a Fixed window of 1 minute.

Answer

Use stateful processing with a custom timer.

Answer

Set the pipeline to use processing time for watermarking.

Question 14

A retail company wants to trigger a Cloud Run service whenever a new CSV file is uploaded to a specific Cloud Storage bucket. Which THREE components are needed to set up this event-driven architecture? (Choose 3)

Accepted Answer

Cloud Storage bucket with notifications enabled. Option A is correct because Cloud Storage buckets must have notifications enabled to publish events to Pub/Sub when objects are created. Without enabling notifications, the bucket cannot emit events that trigger downstream services. This is typically done by configuring the bucket to send notifications to a Pub/Sub topic for each new object upload.

Answer

Cloud Dataflow pipeline

Answer

Cloud Pub/Sub topic

Question 15

A company is migrating on-premises Apache Kafka workloads to Google Cloud. They want to minimize changes to existing producer and consumer applications while leveraging managed services. Which TWO services should they consider? (Choose 2)

Accepted Answer

Dataproc with Apache Kafka. Kafka on Dataproc provides a managed Kafka cluster that is fully compatible, minimizing application changes. Confluent Cloud on Google Cloud can be used but is not a Google-managed service; however, it is a viable partner solution. Pub/Sub is not Kafka API-compatible. Dataflow is not a replacement for Kafka. BigQuery is a data warehouse, not a streaming broker.

Answer

BigQuery

Answer

Cloud Pub/Sub

Answer

Cloud Dataflow