PDE Ingesting and Processing the Data Practice Test 3 — 15 Questions

Question 1

A company is migrating on-premises Apache Kafka workloads to Google Cloud. They want to minimize changes to existing producer and consumer applications while leveraging managed services. Which TWO services should they consider? (Choose 2)

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Dataproc with Apache Kafka. Option C, Dataproc with Apache Kafka, is correct because Dataproc lets you run Apache Kafka clusters on Google Cloud managed infrastructure, so existing producers and consumers that speak the Kafka protocol can connect with minimal or no application changes. Option D, Confluent Cloud on Google Cloud, is correct because it is a fully managed Kafka service available on Google Cloud, preserving the Kafka API and ecosystem so existing producer and consumer applications continue to work without rewriting. Option A, BigQuery, is not a Kafka-compatible streaming broker; it is an analytics data warehouse and would require changing producers/consumers to use its APIs. Option B, Cloud Pub/Sub, is a managed messaging service but uses a different API and semantics than Kafka, so applications would need modification. Option E, Cloud Dataflow, is a managed Apache Beam runner for stream/batch processing, not a Kafka broker replacement, so it does not satisfy the minimal-change requirement for existing Kafka clients.

Answer

BigQuery

Answer

Cloud Pub/Sub

Answer

Cloud Dataflow

Question 2

A data team wants to load millions of small JSON files (each <1 MB) from GCS into BigQuery daily with the lowest cost and fastest performance. They need exactly-once semantics and the ability to detect new files automatically. Which approach is most suitable?

Accepted Answer

Use Dataflow to read files from GCS, combine them, and write to BigQuery using the Storage Write API in exactly-once mode. Dataflow can read the many small GCS files, coalesce them to avoid per-file overhead, and write to BigQuery via the Storage Write API in exactly-once mode, which provides the required exactly-once semantics and high throughput. Dataflow also supports automatic detection of new files (e.g., via a match pattern or Pub/Sub notifications), satisfying the daily incremental load requirement at low cost and fast performance. Option B is unsuitable because the legacy streaming inserts API does not provide exactly-once semantics and is more expensive for high-volume ingestion. Option C does not address file consolidation or exactly-once loading and adds unnecessary copying overhead. Option D uses batch load jobs with a wildcard URI, which cannot guarantee exactly-once semantics and performs poorly with millions of tiny files due to per-file overhead and no automatic new-file detection.

Answer

Use Cloud Functions to trigger on new files and stream each row via the legacy streaming inserts API

Answer

Use Storage Transfer Service to copy files to a staging bucket, then run a scheduled query to load them

Answer

Use BigQuery batch load jobs with a wildcard URI to load all files directly

Question 3

A company runs Apache Kafka on Dataproc for real-time event streaming. They want to archive the Kafka topics to Cloud Storage for long-term retention and later analysis in BigQuery. Which approach is the most cost-effective and operationally simple?

Accepted Answer

Use Kafka Connect with the GCS Sink Connector to write directly to Cloud Storage. Kafka Connect with the GCS Sink Connector is purpose-built for exactly this use case: it directly streams Kafka topics to Cloud Storage in Avro, Parquet, or JSON format without requiring intermediate processing clusters or services. This approach minimizes operational overhead (no Spark or Dataflow jobs to manage) and is cost-effective since it runs as a lightweight connector within the existing Kafka ecosystem, leveraging Dataproc's managed Kafka cluster.

Answer

Use Apache Spark streaming on Dataproc to read from Kafka and write to GCS

Answer

Use Kafka MirrorMaker to replicate topics to a second cluster that writes to GCS

Answer

Use the Pub/Sub connector to publish Kafka messages to Pub/Sub, then a Dataflow job to write to GCS

Question 4

A company wants to migrate their on-premises Teradata data warehouse to BigQuery. They need an automated, one-time transfer of historical data (10 TB) and ongoing incremental daily syncs. Which Google Cloud service should they use?

Accepted Answer

BigQuery Data Transfer Service for Teradata. BigQuery Data Transfer Service (BQ DTS) natively supports Teradata as a source system, providing automated, scheduled transfers of both historical (one-time) and incremental (daily) data directly into BigQuery. It handles schema mapping, parallel loading, and incremental syncs without requiring custom code. This makes it the ideal managed service for the described migration scenario.

Answer

Dataflow custom pipeline

Answer

Storage Transfer Service

Answer

Datastream

Question 5

A data pipeline uses Pub/Sub to ingest events, a Dataflow streaming pipeline to process them, and writes results to BigQuery. The pipeline must handle occasional duplicate events without causing duplicate rows in BigQuery. What is the best approach?

Accepted Answer

Use BigQuery legacy streaming inserts with insertId for deduplication. BigQuery legacy streaming inserts support an insertId field that BigQuery uses for best-effort deduplication within a short window (approximately one minute). When Dataflow writes duplicate events with the same insertId, BigQuery discards the duplicates, preventing duplicate rows. This is the documented mechanism for handling at-least-once delivery from Pub/Sub/Dataflow into BigQuery.

Answer

Use the Storage Write API with the committed stream

Answer

Set a unique constraint on the BigQuery table

Answer

Enable exactly-once processing in Pub/Sub

Question 6

A data engineer needs to create a Dataflow pipeline that reads from Pub/Sub, applies a Python transformation, and writes to BigQuery. The pipeline should be reusable across environments with different parameters. Which deployment method is most appropriate?

Accepted Answer

Flex Template. Flex Templates (Option B) are the most appropriate deployment method because they allow you to package a custom Docker image containing your Python transformation code and dependencies, making the pipeline reusable across environments with different runtime parameters. Unlike Classic Templates, Flex Templates support arbitrary pipeline code and can be parameterized at runtime via the Dataflow UI or API, which is essential for a multi-environment deployment strategy.

Answer

Classic Template

Answer

Direct pipeline submission with gcloud dataflow jobs run

Answer

Cloud Composer to trigger Dataflow jobs

Question 7

A company uses Kafka on Dataproc to ingest streaming data. They want to process the data with Spark Structured Streaming and write results to BigQuery. The team is using Dataproc clusters. Which approach minimizes cost while maintaining performance?

Accepted Answer

Use a Dataproc cluster with standard master nodes and preemptible worker nodes. Using standard (non-preemptible) master nodes with preemptible worker nodes is the cost-optimal configuration that still maintains performance and reliability: masters must be stable to avoid cluster failure, while workers can be preempted and replaced without losing the driver. Preemptible workers cost significantly less than standard VMs, and Spark Structured Streaming can tolerate worker churn as long as the driver (on the master) persists. This balances cost savings with the stability needed for a streaming job.

Answer

Use a Dataproc cluster with all preemptible VMs

Answer

Use a single-node Dataproc cluster

Answer

Use a Dataproc cluster with standard nodes and enable autoscaling

Question 8

Which BigQuery feature allows you to query data directly from Cloud Storage without loading it into BigQuery storage?

Accepted Answer

External tables. BigQuery Omni is for multi-cloud, not external tables. External tables allow querying data in GCS.

Answer

BigQuery Omni

Answer

BigQuery ML

Answer

Federated queries

Question 9

A company wants to build an event-driven application that processes images uploaded to a Cloud Storage bucket. The processing takes up to 10 minutes per image and should be automatically triggered. Which compute option should they use?

Accepted Answer

Cloud Run on Eventarc trigger. Cloud Run with an Eventarc trigger is correct because Cloud Run supports long-running containerized workloads with request timeouts up to 60 minutes, easily covering the 10-minute image processing requirement. Eventarc natively delivers Cloud Storage events (via Cloud Audit Logs) to Cloud Run, enabling automatic, event-driven invocation without managing infrastructure.

Answer

Cloud Functions (2nd gen) with Eventarc trigger

Answer

App Engine

Answer

Cloud Functions (1st gen)

Question 10

A Dataflow streaming pipeline is experiencing high latency and frequent OOM errors when processing variable-sized JSON messages from Pub/Sub. The team suspects that the autoscaling is not effective. Which feature should they enable to improve resource utilization?

Accepted Answer

Dataflow Prime. Dataflow Prime is the correct choice because it provides intelligent resource management that automatically adjusts worker resources (CPU, memory) based on the pipeline's processing demands, which is critical for variable-sized JSON messages. It addresses both high latency and OOM errors by optimizing resource utilization beyond simple autoscaling, including predictive autoscaling and flexible resource scheduling to handle spikes in message size without manual tuning.

Answer

Horizontal autoscaling

Answer

FlexRS

Answer

Streaming Engine

Question 11

A company needs to stream data from a MySQL database to BigQuery with a latency under 10 seconds. They also need to handle schema changes automatically. Which TWO services should they combine?

Accepted Answer

BigQuery. BigQuery (A) is correct because it is the target analytics warehouse where the streamed MySQL data must land, and it natively supports high-throughput streaming inserts and automatic schema evolution for incoming data. Datastream (B) is correct because it is Google Cloud's serverless change data capture (CDC) and replication service that reads the MySQL binary log and streams changes into BigQuery with sub-10-second latency, and it automatically handles schema changes by updating the destination schema. Dataflow (C) is not needed here because Datastream already provides the managed CDC pipeline to BigQuery without custom Apache Beam code. Pub/Sub (D) is a messaging service, not a MySQL CDC replication tool, so it would not by itself read the MySQL binlog or manage schema evolution into BigQuery. Cloud SQL (E) is a managed MySQL/PostgreSQL/SQL Server database, not a streaming or replication service, so it cannot perform the required CDC to BigQuery.

Answer

Dataflow

Answer

Pub/Sub

Answer

Cloud SQL

Question 12

A data team needs to transfer 200 TB of data from Amazon S3 to GCS. The transfer must be incremental, and they need to monitor the transfer progress. Which THREE components should they use?

Accepted Answer

Cloud Monitoring. Storage Transfer Service (E) is the correct GCP service for large-scale, incremental transfers from Amazon S3 to Cloud Storage, as it supports S3 sources, scheduling, and incremental syncs that only move new or changed objects. IAM service account (B) is required because Storage Transfer Service authenticates and runs transfers under a service account, and that account needs the appropriate permissions on the destination bucket (and an AWS access key for the S3 source). Cloud Monitoring (A) is correct because Storage Transfer Service emits transfer metrics and logs that can be tracked in Cloud Monitoring to observe progress and completion. Dataflow (C) is not the right tool here since it is a managed Apache Beam processing service for data pipelines, not a purpose-built S3-to-GCS bulk transfer mechanism. Transfer Appliance (D) is for physically shipping large datasets into GCP when network transfer is impractical, not for an online incremental S3-to-GCS transfer.

Answer

Dataflow

Answer

Transfer Appliance

Question 13

A data engineer needs to load 10 GB of CSV files from Amazon S3 into BigQuery on a daily basis. The files arrive in a specific S3 bucket at 3 AM UTC each day. Which service should be used to automate this transfer?

Accepted Answer

BigQuery Data Transfer Service. BigQuery Data Transfer Service is designed to automate data ingestion from various sources, including Amazon S3, into BigQuery on a scheduled basis. It supports daily transfers and can handle CSV files. This service is the most direct and managed solution for the requirement.

Answer

Cloud Storage Transfer Service

Answer

Dataflow with Pub/Sub

Answer

Transfer Appliance

Question 14

A data engineer needs to move 500 TB of archival data from an on-premises Hadoop cluster to Cloud Storage. The network bandwidth is limited to 100 Mbps, and the transfer must complete within 30 days. Which method is most cost-effective and reliable?

Accepted Answer

Use Transfer Appliance to ship the data offline. Transfer Appliance is a physical device shipped to the customer, loaded with data on-premises, and shipped back to Google, where the data is uploaded to Cloud Storage. At 100 Mbps, transferring 500 TB over the network would take roughly 500+ days (500 TB = 4,000,000 Mb ÷ 100 Mbps ≈ 40,000,000 seconds), far exceeding the 30-day window. Offline transfer sidesteps the bandwidth bottleneck entirely and is the most cost-effective and reliable option for this volume.

Answer

Use Dataproc to copy data in parallel

Answer

Use a VPN and gsutil rsync

Answer

Use Storage Transfer Service over the internet

Question 15

A company uses Pub/Sub to ingest clickstream data. Each message contains a JSON payload with a nested array of user actions. The data must be written to BigQuery, with each action in the array becoming a separate row. Which BigQuery feature or approach should be used to achieve this transformation?

Accepted Answer

Use a Dataflow pipeline with a ParDo that explodes the array. To write each action as a separate row in BigQuery, the nested array must be flattened during the data pipeline, not at query time. A Dataflow pipeline with a ParDo function that iterates over the array and outputs one element per action achieves this transformation before writing to BigQuery. Loading the JSON as-is and using UNNEST in a query only flattens at read time, not write time, so it does not meet the requirement.

Answer

Load the JSON as-is into BigQuery and use UNNEST in a query

Answer

Use a BigQuery scripting loop to iterate over the array

Answer

Preprocess the data with a Dataflow pipeline and write to BigQuery