PDE Ingesting and Processing the Data Practice Test 3 — 15 Questions

Question 1

A data engineer needs to migrate 200 TB of on-premises Oracle data to BigQuery. The network bandwidth is limited to 100 Mbps, and the data must be loaded within 2 weeks. Which Google Cloud service is most appropriate for the initial data transfer?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Transfer Appliance. Transfer Appliance is correct because it is a physical device designed for large-scale offline data transfers when network bandwidth is insufficient. With 200 TB at 100 Mbps, the theoretical transfer time exceeds 190 days, far beyond the 2-week window. Transfer Appliance allows shipping the data directly to Google, bypassing network constraints entirely.

Answer

BigQuery Data Transfer Service for Oracle

Answer

Datastream

Answer

Storage Transfer Service

Question 2

A company uses Pub/Sub to ingest IoT sensor data and wants to process it with a Dataflow pipeline that uses fixed windows of 1 minute to compute average temperature. The pipeline also needs to handle malformed messages by routing them to a dead letter queue. Which TWO configurations should the engineer implement? (Choose TWO.)

Accepted Answer

Enable exactly-once processing for the Pub/Sub subscription used by the pipeline. To handle malformed messages, a dead letter sink (e.g., Pub/Sub topic or BigQuery table) should be used to capture messages that fail processing after retries. The Dataflow pipeline should use the Apache Beam `PubsubIO` source and apply windowing. Enabling exactly-once processing for the Pub/Sub subscription ensures that messages are not lost or duplicated during failures. Option C is correct because exactly-once processing ensures data integrity for streaming pipelines. Option E is correct because dead letter sinks capture failed messages and prevent data loss. Option A is not recommended as it discards data. Option B is irrelevant as deduplication is not needed with exactly-once. Option D is not necessary for this scenario.

Answer

Configure the pipeline to ignore malformed messages using `withCoder()`

Answer

Add a `GroupByKey` transform to deduplicate messages based on a unique ID

Answer

Set up a Cloud Function to reprocess messages from the dead letter topic

Question 3

A data engineer needs to load 10 TB of CSV files from Amazon S3 into Google BigQuery on a daily basis. Which service should they use to automate this transfer?

Accepted Answer

BigQuery Data Transfer Service. Storage Transfer Service can transfer data from Amazon S3 to Google Cloud Storage, but it does not load directly into BigQuery. BigQuery Data Transfer Service can import from Amazon S3 directly into BigQuery tables. Other options are not suitable: Cloud Data Fusion is for ETL pipelines, not simple transfer; Transfer Appliance is for offline petabyte-scale transfers; Dataproc is for Spark/Hadoop jobs.

Answer

Dataproc

Answer

Cloud Data Fusion

Answer

Storage Transfer Service

Question 4

You need to stream real-time user click events from your application into BigQuery for immediate analysis. The events must be available for query within seconds. Which approach is recommended?

Accepted Answer

Use Pub/Sub to Dataflow to BigQuery with the Storage Write API for high-throughput streaming.. Pub/Sub to Dataflow to BigQuery using the Storage Write API provides the highest throughput and reliability with near-real-time latency. Legacy streaming inserts are limited and have higher latency. Direct Pub/Sub to BigQuery subscription is not a native feature. Cloud Functions is not suitable for high-throughput streaming.

Answer

Use Cloud Data Fusion to ingest streaming data from Pub/Sub into BigQuery.

Answer

Use Cloud Functions to receive events from Pub/Sub and insert them into BigQuery using the legacy streaming API.

Answer

Use Pub/Sub with a BigQuery subscription to directly write events into BigQuery.

Question 5

A data engineer needs to transfer 500 TB of on-premises data to Google Cloud Storage. The data is stored on NAS devices and the network bandwidth is limited to 100 Mbps. What is the most cost-effective and timely transfer method?

Accepted Answer

Use Transfer Appliance. At 100 Mbps, transferring 500 TB over the network would take over 500 days. Transfer Appliance is designed for petabyte-scale offline transfer, shipping a physical appliance to your data center. Other options are not feasible due to bandwidth constraints.

Answer

Use Storage Transfer Service over the internet

Answer

Use a VPN connection and rsync

Answer

Use gsutil cp in parallel

Question 6

You are building a Dataflow pipeline in Python that reads messages from Pub/Sub, enriches them with data from a BigQuery table, and writes the results to BigQuery. The enrichment lookup table is large and changes infrequently. Which approach minimizes cost and latency?

Accepted Answer

Use a side input that reads the BigQuery table periodically and caches it.. Using a side input that periodically reads the BigQuery table and caches it avoids querying BigQuery for every incoming message, which would be prohibitively expensive and high-latency. The side input is refreshed at a configurable interval (e.g., every 10 minutes) via a pipeline option, and the cached data is broadcast to all workers, enabling fast, in-memory lookups without per-element I/O. This approach minimizes cost by reducing BigQuery API calls and minimizes latency by avoiding synchronous queries for each message.

Answer

Use a CoGroupByKey transform to join the incoming stream with a stream from BigQuery.

Answer

Use BigQuery IO to query the table for every incoming message.

Answer

Use a stateful DoFn and store the lookup in state per key.

Question 7

You are designing a Dataflow pipeline to process streaming data. The pipeline may encounter malformed records. You need to handle these errors without failing the entire pipeline and store the bad records for later analysis. What is the best practice?

Accepted Answer

Use a dead letter sink to write malformed records to a separate Pub/Sub topic or GCS location.. Dead letter sinks are a common pattern: route erroneous records to a separate output (e.g., Pub/Sub topic or GCS) for later investigation. Writing to BigQuery using Storage Write API with error handling is good, but for malformed records you want to isolate them. Raising exceptions would fail the pipeline. Logging only loses the data.

Answer

Catch the exception and log it, then continue processing.

Answer

Write all records to BigQuery using the Storage Write API and handle errors in the write operation.

Answer

Raise an exception in the DoFn to stop the pipeline for manual intervention.

Question 8

Your company uses Kafka for event streaming. You want to run Kafka on Google Cloud with the ability to auto-scale clusters and use managed infrastructure. Which service should you choose?

Accepted Answer

Dataproc. Dataproc is the correct choice because it is a managed Spark and Hadoop service on Google Cloud that supports running Kafka clusters via initialization actions. It allows auto-scaling of worker nodes and integrates with GCP storage and networking, providing the managed infrastructure required for Kafka event streaming. Note that Confluent Cloud is a third-party managed Kafka service, not a GCP-native service, and Cloud Pub/Sub is a messaging service, not a Kafka replacement. Cloud Dataflow is for data processing pipelines, not for running Kafka itself.

Answer

Cloud Pub/Sub

Answer

Confluent Cloud on GCP

Answer

Cloud Dataflow

Question 9

You need to perform a one-time migration of historical data from an on-premises Teradata data warehouse to BigQuery. The data volume is 50 TB and you have a high-speed network connection (10 Gbps). What is the most efficient way to load the data?

Accepted Answer

Use BigQuery Data Transfer Service for Teradata. BigQuery Data Transfer Service for Teradata is designed for this purpose; it can directly connect to Teradata and transfer data to BigQuery. Exporting to CSV then loading via gsutil is possible but less efficient. Transfer Appliance is for offline transfer but you have high-speed network. Dataproc is not needed.

Answer

Export data from Teradata to CSV files, upload to GCS using gsutil, then load into BigQuery.

Answer

Use Dataproc to run a Spark job that reads from Teradata and writes to BigQuery.

Answer

Use Transfer Appliance to ship the data offline.

Question 10

You have a Dataflow pipeline that processes streaming data with high throughput. You notice that the pipeline is experiencing high latency and the workers are underutilized. Which Dataflow feature can automatically optimize resource allocation?

Accepted Answer

Dataflow Prime. Dataflow Prime (also known as right-fitting) provides vertical autoscaling and resource optimization based on actual usage. Horizontal autoscaling is standard but may not address underutilization. Streaming engine is for scaling the streaming writes but not worker tuning. Flex templates are for deployment, not runtime optimization.

Answer

Flex Templates

Answer

Horizontal autoscaling

Answer

Streaming Engine

Question 11

You are migrating an on-premises PostgreSQL database to Cloud SQL. You need to continuously replicate changes to BigQuery for real-time analytics with minimal latency. Which service should you use?

Accepted Answer

Datastream. Datastream is designed for change data capture (CDC) from databases like PostgreSQL, MySQL, and Oracle to BigQuery or GCS. It provides low-latency replication. Pub/Sub and Dataflow are not directly for CDC. Storage Transfer Service is for file transfers, not database replication.

Answer

Dataflow with JDBC source

Answer

Pub/Sub with a Cloud Function that writes to BigQuery

Answer

Storage Transfer Service

Question 12

You are designing a Dataflow pipeline that needs to exactly-once process events from Pub/Sub and write to BigQuery using the Storage Write API. The pipeline may restart and could reprocess some messages. What setting ensures exactly-once semantics for the output?

Accepted Answer

Use the Storage Write API in committed mode and enable exactly-once semantic in Dataflow. The Storage Write API supports exactly-once semantics when used with the 'committed' mode, which ensures each record is written exactly once. The pipeline also needs to use Pub/Sub with message IDs and deduplication. The other options either do not provide exactly-once or are unreliable.

Answer

Use the legacy streaming inserts with insertId for deduplication

Answer

Use at-least-once delivery on Pub/Sub and idempotent writes to BigQuery

Answer

Use the Storage Write API in buffered mode with deduplication logic

Question 13

You need to process a large volume of event data from Cloud Storage, apply complex transformations using Apache Spark, and then load the results into BigQuery. The data arrives in batches every hour. You want to minimize costs by using preemptible VMs. Which service should you use?

Accepted Answer

Dataproc. Dataproc supports preemptible (now called spot) VMs for cost savings. Dataflow does not support preemptible VMs for workers; it uses standard VMs. Cloud Composer is orchestration only. BigQuery is not for running Spark.

Answer

Cloud Composer

Answer

BigQuery

Answer

Dataflow

Question 14

An organization wants to ingest on-premises Oracle database changes into BigQuery for real-time analytics with minimal latency. The Oracle database is version 19c and has a high transactional volume. Which Google Cloud service should they use?

Accepted Answer

Datastream. Datastream is the correct service because it is purpose-built for real-time, serverless change data capture (CDC) from Oracle databases (including 19c) to BigQuery. It uses LogMiner to read redo logs with minimal overhead, supporting high transactional volumes and sub-second latency without requiring custom code or manual schema management.

Answer

Pub/Sub

Answer

Dataflow

Answer

Storage Transfer Service

Question 15

An organization needs to transfer 50 TB of historical data from an on-premises Hadoop cluster to Google Cloud Storage. The network bandwidth is limited to 100 Mbps. Which transfer method is MOST cost-effective and time-efficient?

Accepted Answer

Transfer Appliance. Transfer Appliance is designed for petabyte-scale offline transfers, shipping physical devices to Google for upload, which is much faster than using limited network bandwidth for 50 TB.

Answer

Storage Transfer Service over the network

Answer

BigQuery Data Transfer Service for Hadoop

Answer

gsutil cp with parallel composite uploads