PDE Ingesting and Processing the Data Practice Test 8 — 15 Questions

Question 1

A Dataflow streaming pipeline is experiencing high latency and frequent OOM errors when processing variable-sized JSON messages from Pub/Sub. The team suspects that the autoscaling is not effective. Which feature should they enable to improve resource utilization?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Dataflow Prime. Dataflow Prime is the correct choice because it provides intelligent resource management that automatically adjusts worker resources (CPU, memory) based on the pipeline's processing demands, which is critical for variable-sized JSON messages. It addresses both high latency and OOM errors by optimizing resource utilization beyond simple autoscaling, including predictive autoscaling and flexible resource scheduling to handle spikes in message size without manual tuning.

Answer

Horizontal autoscaling

Answer

FlexRS

Answer

Streaming Engine

Question 2

A company uses dbt on BigQuery to transform data. They want to run dbt models on a schedule and manage environments (dev, prod). Which GCP service should they use to run dbt jobs?

Accepted Answer

Cloud Composer. Cloud Composer is an Apache Airflow managed service that can schedule dbt runs.

Answer

Dataflow

Answer

Cloud Scheduler

Answer

Cloud Build

Question 3

A company needs to stream data from a MySQL database to BigQuery with a latency under 10 seconds. They also need to handle schema changes automatically. Which TWO services should they combine?

Accepted Answer

BigQuery. Datastream captures CDC and can write to BigQuery directly. Pub/Sub is not needed if Datastream writes directly.

Answer

Dataflow

Answer

Pub/Sub

Answer

Cloud SQL

Question 4

A data team needs to transfer 200 TB of data from Amazon S3 to GCS. The transfer must be incremental, and they need to monitor the transfer progress. Which THREE components should they use?

Accepted Answer

Cloud Monitoring. Storage Transfer Service (STS) can transfer from S3 to GCS with incremental sync. Cloud Monitoring tracks progress. Service account for permissions.

Answer

Dataflow

Answer

Transfer Appliance

Question 5

A company wants to use Eventarc to trigger a Cloud Run service when new objects are created in a GCS bucket. They also need to filter events for a specific bucket and object prefix. Which THREE resources must exist or be created?

Accepted Answer

Cloud Storage bucket. Eventarc trigger, Cloud Run service, and the GCS bucket. The trigger references the bucket and prefix.

Answer

Pub/Sub topic

Answer

Cloud Scheduler job

Question 6

A data engineer needs to load 10 GB of CSV files from Amazon S3 into BigQuery on a daily basis. The files arrive in a specific S3 bucket at 3 AM UTC each day. Which service should be used to automate this transfer?

Accepted Answer

BigQuery Data Transfer Service. BigQuery Data Transfer Service supports scheduled transfers from Amazon S3 directly to BigQuery, making it the appropriate choice for this recurring batch load.

Answer

Cloud Storage Transfer Service

Answer

Dataflow with Pub/Sub

Answer

Transfer Appliance

Question 7

A financial services company receives real-time stock trade data via Pub/Sub. They need to enrich each trade with reference data from a Cloud SQL table and write the results to BigQuery for real-time analytics. The enrichment must handle late-arriving data and ensure exactly-once processing. Which Dataflow streaming pipeline configuration should be used?

Accepted Answer

Build a custom Dataflow pipeline using the Storage Write API with exactly-once semantics and a side input from Cloud SQL. Using the Storage Write API with exactly-once semantics and side inputs to join with reference data provides the required enrichment and exactly-once guarantees.

Answer

Use a Dataflow Flex Template that reads from Pub/Sub, joins in memory, and writes to BigQuery using legacy streaming inserts

Answer

Use Pub/Sub to BigQuery template with streaming inserts and a side input from Cloud SQL

Answer

Deploy a Dataproc Spark Streaming job that reads from Pub/Sub, enriches via JDBC, and writes to BigQuery

Question 8

A company needs to continuously synchronize customer data changes from an on-premises Oracle database to BigQuery for near-real-time analytics. The Oracle database has Change Data Capture (CDC) enabled. Which Google Cloud service should be used to stream these changes with minimal latency and schema evolution support?

Accepted Answer

Use Datastream to stream CDC changes from Oracle to BigQuery. Datastream is designed to stream CDC from Oracle (and MySQL/PostgreSQL) directly to BigQuery or GCS, supporting schema evolution and low-latency replication.

Answer

Deploy a Dataflow pipeline with a JDBC source and Pub/Sub

Answer

Use Cloud SQL with a read replica and enable binary logging

Answer

Use Transfer Appliance to copy Oracle data periodically

Question 9

A data engineer needs to move 500 TB of archival data from an on-premises Hadoop cluster to Cloud Storage. The network bandwidth is limited to 100 Mbps, and the transfer must complete within 30 days. Which method is most cost-effective and reliable?

Accepted Answer

Use Transfer Appliance to ship the data offline. With 100 Mbps, transferring 500 TB over the network would take > 500 days, exceeding the 30-day window. Transfer Appliance is designed for petabyte-scale offline transfers, making it the only feasible option.

Answer

Use Dataproc to copy data in parallel

Answer

Use a VPN and gsutil rsync

Answer

Use Storage Transfer Service over the internet

Question 10

A company is running a Dataflow streaming pipeline that reads from Pub/Sub and writes to BigQuery. They notice that the number of workers is not scaling up to handle increased throughput, causing latency spikes. The pipeline uses a GlobalWindow with default triggering. What is the most likely cause of the under-scaling?

Accepted Answer

The pipeline includes a GroupByKey that creates a hot key, limiting parallelism. Dataflow's autoscaling is based on CPU utilization and throughput. If the pipeline uses a GroupByKey with hot keys, parallelism is limited and workers may not scale effectively.

Answer

The Pub/Sub subscription has a large backlog, but Dataflow automatically scales to handle it

Answer

The pipeline uses the default worker machine type, which is too small

Answer

The pipeline is using legacy streaming inserts instead of the Storage Write API

Question 11

A company wants to move data from an on-premises MySQL database to BigQuery for analytics. They need to capture all changes (inserts, updates, deletes) in near real-time and also perform an initial historical load. Which approach meets these requirements with minimal operational overhead?

Accepted Answer

Use Datastream to backfill historical data and then stream CDC changes to BigQuery. Datastream can perform a backfill of historical data and then stream CDC changes from MySQL to BigQuery in near real-time, providing a single service for both tasks.

Answer

Use a Dataflow pipeline with a JDBC source to read the entire table periodically

Answer

Use a one-time export to CSV and load into BigQuery, then set up a cron job to export incremental changes

Answer

Use Cloud SQL as an intermediary and enable binary logging, then stream to Pub/Sub via a custom connector

Question 12

A data engineer is building a Dataflow pipeline that reads from BigQuery, transforms data using Apache Beam, and writes results to Cloud Storage in Avro format. They need to ensure the pipeline can be easily redeployed with different parameters without modifying code. Which deployment method should they use?

Accepted Answer

Dataflow Flex Templates. Dataflow Flex Templates allow you to package a pipeline as a Docker image and pass runtime parameters, enabling parameterized deployments without code changes.

Answer

Direct deployment using the gcloud command with parameters

Answer

Dataflow Classic Templates

Answer

Deploy as a Cloud Function triggered by Cloud Scheduler

Question 13

A company uses BigQuery to store event data. They need to load data from multiple sources with different schemas and expect frequent schema changes. Which approach provides the most flexibility for schema evolution while minimizing load failures and performance impact?

Accepted Answer

Use the Storage Write API in buffered mode with schema auto-detection. Using the Storage Write API with buffered mode allows schema auto-detection and flexible schema updates without failing loads, and provides better performance than legacy streaming inserts.

Answer

Load data as JSON files in Cloud Storage and use external tables

Answer

Use legacy streaming inserts with schema auto-detect enabled

Answer

Use Dataflow to preprocess and write to BigQuery using Storage Write API in committed mode

Question 14

A company is building a data pipeline that ingests streaming data from Pub/Sub, transforms it with Dataflow, and loads it into BigQuery. They want to handle malformed messages that cannot be parsed. Which TWO actions should they implement for error handling? (Choose 2)

Accepted Answer

Use a dead letter sink to write malformed messages to Cloud Storage or Pub/Sub for later analysis. A dead letter sink (e.g., writing malformed messages to Cloud Storage or a separate Pub/Sub topic) allows the pipeline to continue processing valid data while preserving the problematic records for offline inspection, retries, or debugging. This pattern is a standard best practice in streaming pipelines to avoid data loss and enable recovery without blocking the main data flow.

Answer

Configure the pipeline to drop malformed messages silently

Answer

Use a side input to filter out malformed messages

Answer

Raise an exception in the DoFn to fail the pipeline immediately

Question 15

A data engineer needs to perform a one-time migration of 10 TB of data from on-premises Hadoop HDFS to Cloud Storage. The network link is 1 Gbps. Which TWO services or tools should they consider? (Choose 2)

Accepted Answer

Dataproc with DistCp. For a one-time migration of 10 TB from on-premises HDFS to Cloud Storage over a 1 Gbps link, the most appropriate services are Dataproc with DistCp and Transfer Appliance. Dataproc with DistCp is a proven method for copying data from HDFS to Cloud Storage, leveraging Apache Hadoop's distributed copy tool. Transfer Appliance is ideal for large data volumes when network bandwidth is limited, as it physically ships the data. Cloud Storage Transfer Service does not support HDFS as a source directly, so it is not suitable. gsutil rsync with parallel composite uploads could be used but would be slower and less efficient for this volume.

Answer

Cloud Storage Transfer Service

Answer

BigQuery Data Transfer Service

Answer

gsutil rsync with parallel composite uploads