PDE Ingesting and Processing the Data Practice Test 8 — 15 Questions

Question 1

A data engineer needs to load 10 GB of CSV files from Amazon S3 into BigQuery on a daily basis. The files arrive in a specific S3 bucket at 3 AM UTC each day. Which service should be used to automate this transfer?

Accepted Answer

BigQuery Data Transfer Service. BigQuery Data Transfer Service supports scheduled transfers from Amazon S3 directly to BigQuery, making it the appropriate choice for this recurring batch load.

Answer

Cloud Storage Transfer Service

Answer

Dataflow with Pub/Sub

Answer

Transfer Appliance

Question 2

A financial services company receives real-time stock trade data via Pub/Sub. They need to enrich each trade with reference data from a Cloud SQL table and write the results to BigQuery for real-time analytics. The enrichment must handle late-arriving data and ensure exactly-once processing. Which Dataflow streaming pipeline configuration should be used?

Accepted Answer

Build a custom Dataflow pipeline using the Storage Write API with exactly-once semantics and a side input from Cloud SQL. Using the Storage Write API with exactly-once semantics and side inputs to join with reference data provides the required enrichment and exactly-once guarantees.

Answer

Use a Dataflow Flex Template that reads from Pub/Sub, joins in memory, and writes to BigQuery using legacy streaming inserts

Answer

Use Pub/Sub to BigQuery template with streaming inserts and a side input from Cloud SQL

Answer

Deploy a Dataproc Spark Streaming job that reads from Pub/Sub, enriches via JDBC, and writes to BigQuery

Question 3

A company needs to continuously synchronize customer data changes from an on-premises Oracle database to BigQuery for near-real-time analytics. The Oracle database has Change Data Capture (CDC) enabled. Which Google Cloud service should be used to stream these changes with minimal latency and schema evolution support?

Accepted Answer

Use Datastream to stream CDC changes from Oracle to BigQuery. Datastream is designed to stream CDC from Oracle (and MySQL/PostgreSQL) directly to BigQuery or GCS, supporting schema evolution and low-latency replication.

Answer

Deploy a Dataflow pipeline with a JDBC source and Pub/Sub

Answer

Use Cloud SQL with a read replica and enable binary logging

Answer

Use Transfer Appliance to copy Oracle data periodically

Question 4

A data engineer needs to move 500 TB of archival data from an on-premises Hadoop cluster to Cloud Storage. The network bandwidth is limited to 100 Mbps, and the transfer must complete within 30 days. Which method is most cost-effective and reliable?

Accepted Answer

Use Transfer Appliance to ship the data offline. With 100 Mbps, transferring 500 TB over the network would take > 500 days, exceeding the 30-day window. Transfer Appliance is designed for petabyte-scale offline transfers, making it the only feasible option.

Answer

Use Dataproc to copy data in parallel

Answer

Use a VPN and gsutil rsync

Answer

Use Storage Transfer Service over the internet

Question 5

A company uses Pub/Sub to ingest clickstream data. Each message contains a JSON payload with a nested array of user actions. The data must be written to BigQuery, with each action in the array becoming a separate row. Which BigQuery feature or approach should be used to achieve this transformation?

Accepted Answer

Load the JSON as-is into BigQuery and use UNNEST in a query. BigQuery's UNNEST function is designed to flatten arrays into separate rows, which is exactly what is needed here.

Answer

Use a Dataflow pipeline with a ParDo that explodes the array

Answer

Use a BigQuery scripting loop to iterate over the array

Answer

Preprocess the data with a Dataflow pipeline and write to BigQuery

Question 6

A data engineer needs to orchestrate a series of tasks that include calling external APIs, running BigQuery queries, and sending notifications. The workflow involves conditional branching and parallel steps. Which Google Cloud service should be used?

Accepted Answer

Workflows. Workflows is the correct choice because it is a fully managed orchestration service designed specifically for coordinating multi-step, event-driven workflows that involve conditional branching, parallel execution, and integration with external APIs, BigQuery, and notifications via HTTP calls or service integrations. It provides built-in error handling, retries, and a declarative YAML-based syntax that directly supports the described requirements without needing to manage infrastructure or schedule tasks.

Answer

Cloud Scheduler

Answer

Cloud Composer

Answer

Dataflow

Question 7

A company is running a Dataflow streaming pipeline that reads from Pub/Sub and writes to BigQuery. They notice that the number of workers is not scaling up to handle increased throughput, causing latency spikes. The pipeline uses a GlobalWindow with default triggering. What is the most likely cause of the under-scaling?

Accepted Answer

The pipeline includes a GroupByKey that creates a hot key, limiting parallelism. Dataflow's autoscaling is based on CPU utilization and throughput. If the pipeline uses a GroupByKey with hot keys, parallelism is limited and workers may not scale effectively.

Answer

The Pub/Sub subscription has a large backlog, but Dataflow automatically scales to handle it

Answer

The pipeline uses the default worker machine type, which is too small

Answer

The pipeline is using legacy streaming inserts instead of the Storage Write API

Question 8

A company wants to move data from an on-premises MySQL database to BigQuery for analytics. They need to capture all changes (inserts, updates, deletes) in near real-time and also perform an initial historical load. Which approach meets these requirements with minimal operational overhead?

Accepted Answer

Use Datastream to backfill historical data and then stream CDC changes to BigQuery. Datastream can perform a backfill of historical data and then stream CDC changes from MySQL to BigQuery in near real-time, providing a single service for both tasks.

Answer

Use a Dataflow pipeline with a JDBC source to read the entire table periodically

Answer

Use a one-time export to CSV and load into BigQuery, then set up a cron job to export incremental changes

Answer

Use Cloud SQL as an intermediary and enable binary logging, then stream to Pub/Sub via a custom connector

Question 9

A data engineer is building a Dataflow pipeline that reads from BigQuery, transforms data using Apache Beam, and writes results to Cloud Storage in Avro format. They need to ensure the pipeline can be easily redeployed with different parameters without modifying code. Which deployment method should they use?

Accepted Answer

Dataflow Flex Templates. Dataflow Flex Templates allow you to package a pipeline as a Docker image and pass runtime parameters, enabling parameterized deployments without code changes.

Answer

Direct deployment using the gcloud command with parameters

Answer

Dataflow Classic Templates

Answer

Deploy as a Cloud Function triggered by Cloud Scheduler

Question 10

A company uses BigQuery to store event data. They need to load data from multiple sources with different schemas and expect frequent schema changes. Which approach provides the most flexibility for schema evolution while minimizing load failures and performance impact?

Accepted Answer

Use the Storage Write API in buffered mode with schema auto-detection. Using the Storage Write API with buffered mode allows schema auto-detection and flexible schema updates without failing loads, and provides better performance than legacy streaming inserts.

Answer

Load data as JSON files in Cloud Storage and use external tables

Answer

Use legacy streaming inserts with schema auto-detect enabled

Answer

Use Dataflow to preprocess and write to BigQuery using Storage Write API in committed mode

Question 11

A company is building a data pipeline that ingests streaming data from Pub/Sub, transforms it with Dataflow, and loads it into BigQuery. They want to handle malformed messages that cannot be parsed. Which TWO actions should they implement for error handling? (Choose 2)

Accepted Answer

Use a dead letter sink to write malformed messages to Cloud Storage or Pub/Sub for later analysis. Option C is correct because a dead letter sink (e.g., writing malformed messages to Cloud Storage or a separate Pub/Sub topic) allows the pipeline to continue processing valid data while preserving the problematic records for offline inspection, retries, or debugging. This pattern is a standard best practice in streaming pipelines to avoid data loss and enable recovery without blocking the main data flow.

Answer

Configure the pipeline to drop malformed messages silently

Answer

Use a side input to filter out malformed messages

Answer

Raise an exception in the DoFn to fail the pipeline immediately

Question 12

A data engineer needs to perform a one-time migration of 10 TB of data from on-premises Hadoop HDFS to Cloud Storage. The network link is 1 Gbps. Which TWO services or tools should they consider? (Choose 2)

Accepted Answer

Cloud Storage Transfer Service. Storage Transfer Service can transfer data from HDFS (via an intermediary) but Transfer Appliance is also feasible for large volumes, especially if bandwidth is limited.

Answer

Dataproc with DistCp

Answer

BigQuery Data Transfer Service

Answer

gsutil rsync with parallel composite uploads

Question 13

A company uses Dataflow to process data with Apache Beam in Python. The pipeline reads from Pub/Sub, applies a ParDo that calls an external API for enrichment, and writes to BigQuery. The external API has rate limits and occasionally fails. To improve reliability, which THREE strategies should be implemented? (Choose 3)

Accepted Answer

Batch multiple requests to the external API using a side input. Retry logic with exponential backoff, a dead letter queue for failed records, and batching requests reduce API pressure and handle failures gracefully.

Answer

Switch from Python to Java SDK for better performance

Answer

Increase the number of Dataflow workers to reduce load per worker

Question 14

A data engineer needs to schedule a recurring transfer of data from a partner's Amazon S3 bucket to a Cloud Storage bucket for further processing. Which THREE components or configurations are necessary? (Choose 3)

Accepted Answer

Specification of the source S3 bucket and destination GCS bucket. Scheduling, source and destination locations, and authentication/authorization are essential for a Storage Transfer Service job.

Answer

A VPC network configuration

Answer

A Pub/Sub topic to notify completion

Question 15

A company is migrating a Spark batch job from on-premises to Dataproc. The job uses RDDs for custom transformations and writes output to BigQuery. They want to optimize the job for performance and cost on Dataproc. Which THREE practices should they adopt? (Choose 3)

Accepted Answer

Write output to BigQuery using the BigQuery connector. The BigQuery connector for Spark (com.google.cloud.spark.bigquery) allows direct, efficient writes to BigQuery without intermediate storage. It leverages the BigQuery Storage Write API for high-throughput, exactly-once delivery, which is far more performant than writing via JDBC or saving to GCS and then loading into BigQuery.

Answer

Use PySpark instead of Scala for simplicity

Answer

Increase the number of partitions to match the number of cores exactly