PDE Ingesting and Processing the Data Practice Test 4 — 15 Questions

Question 1

A company is running a Dataflow streaming pipeline that reads from Pub/Sub and writes to BigQuery. They notice that the number of workers is not scaling up to handle increased throughput, causing latency spikes. The pipeline uses a GlobalWindow with default triggering. What is the most likely cause of the under-scaling?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

The pipeline includes a GroupByKey that creates a hot key, limiting parallelism. In Dataflow streaming pipelines, the maximum parallelism of a GroupByKey (or any keyed aggregation) is bounded by the number of distinct keys. If one key receives a disproportionate share of the data — a 'hot key' — all records for that key are processed by a single worker, so the pipeline cannot scale beyond that bottleneck regardless of how many workers Dataflow provisions. This is the classic cause of under-scaling in streaming pipelines that use GlobalWindow with default triggering, because the default trigger fires on watermark/accumulation and doesn't change the fundamental key-based parallelism limit.

Answer

The Pub/Sub subscription has a large backlog, but Dataflow automatically scales to handle it

Answer

The pipeline uses the default worker machine type, which is too small

Answer

The pipeline is using legacy streaming inserts instead of the Storage Write API

Question 2

A data engineer is building a Dataflow pipeline that reads from BigQuery, transforms data using Apache Beam, and writes results to Cloud Storage in Avro format. They need to ensure the pipeline can be easily redeployed with different parameters without modifying code. Which deployment method should they use?

Accepted Answer

Dataflow Flex Templates. Dataflow Flex Templates package the pipeline as a Docker image and expose runtime parameters via a metadata file, so the same template can be launched repeatedly with different values (input table, output bucket, etc.) without touching the Beam code. This directly satisfies the requirement of redeploying with different parameters without code modification. Classic Templates also accept parameters but require the pipeline to be recompiled and staged to GCS for any change, and they cannot run arbitrary runtime logic.

Answer

Direct deployment using the gcloud command with parameters

Answer

Dataflow Classic Templates

Answer

Deploy as a Cloud Function triggered by Cloud Scheduler

Question 3

A company uses BigQuery to store event data. They need to load data from multiple sources with different schemas and expect frequent schema changes. Which approach provides the most flexibility for schema evolution while minimizing load failures and performance impact?

Accepted Answer

Use the Storage Write API in buffered mode with schema auto-detection. The Storage Write API in buffered mode with schema auto-detection is the best fit because buffered mode batches writes for high throughput, and schema auto-detection lets BigQuery add new columns automatically as source schemas evolve, minimizing load failures. This combination handles frequent schema changes without requiring manual DDL updates or pipeline redeployment, while keeping performance high through batching.

Answer

Load data as JSON files in Cloud Storage and use external tables

Answer

Use legacy streaming inserts with schema auto-detect enabled

Answer

Use Dataflow to preprocess and write to BigQuery using Storage Write API in committed mode

Question 4

A data engineer needs to perform a one-time migration of 10 TB of data from on-premises Hadoop HDFS to Cloud Storage. The network link is 1 Gbps. Which TWO services or tools should they consider? (Choose 2)

Accepted Answer

Dataproc with DistCp. Option A (Dataproc with DistCp) is correct because DistCp is the standard Hadoop tool for large-scale, parallel HDFS-to-HDFS/cloud copies, and running it on a Dataproc cluster lets it write directly to Cloud Storage via the gs:// connector with distributed bandwidth. Option E (Transfer Appliance) is correct because moving 10 TB over a 1 Gbps link would take roughly 24+ hours at best (and often days in practice), so shipping a physical Transfer Appliance is a recommended approach for one-time migrations of this size. Option B (Cloud Storage Transfer Service) is not ideal here because it targets HTTP/HTTPS or cloud-to-cloud sources and is not the primary path for on-premises HDFS data. Option C (BigQuery Data Transfer Service) is for scheduled ingestion into BigQuery, not HDFS-to-GCS migration. Option D (gsutil rsync with parallel composite uploads) is a valid GCS tool but is not designed for HDFS source data and is less suited than DistCp for this Hadoop migration.

Answer

Cloud Storage Transfer Service

Answer

BigQuery Data Transfer Service

Answer

gsutil rsync with parallel composite uploads

Question 5

A company uses Dataflow to process data with Apache Beam in Python. The pipeline reads from Pub/Sub, applies a ParDo that calls an external API for enrichment, and writes to BigQuery. The external API has rate limits and occasionally fails. To improve reliability, which THREE strategies should be implemented? (Choose 3)

Accepted Answer

Batch multiple requests to the external API using a side input. Option C is correct because batching requests to the external API via a side input reduces the number of calls made, which directly mitigates the API's rate limits and improves throughput in a Beam ParDo. Option D is correct because retry logic with exponential backoff handles the API's occasional transient failures gracefully, spacing out retries to avoid overwhelming the service and respecting rate limits. Option E is correct because a dead letter pattern captures records that still fail after retries and writes them to a separate sink, preventing pipeline stalls and preserving failed data for later reprocessing. Option A is not correct because switching from Python to Java does not address API rate limits or failures and is not a reliability strategy. Option B is not correct because adding more workers increases parallelism and thus the aggregate call rate to the external API, which would worsen rate-limit violations rather than improve reliability.

Answer

Switch from Python to Java SDK for better performance

Answer

Increase the number of Dataflow workers to reduce load per worker

Question 6

A company is migrating a Spark batch job from on-premises to Dataproc. The job uses RDDs for custom transformations and writes output to BigQuery. They want to optimize the job for performance and cost on Dataproc. Which THREE practices should they adopt? (Choose 3)

Accepted Answer

Write output to BigQuery using the BigQuery connector. Option A is correct because writing to BigQuery via the BigQuery connector (the spark-bigquery-connector) uses the BigQuery Storage API for parallel, high-throughput reads and streaming inserts, which is far more efficient than exporting to GCS and loading, reducing both runtime and cost. Option B is correct because DataFrames/Datasets leverage Catalyst optimizer and Tungsten execution for query planning and code generation, so replacing RDDs with them where possible improves performance; RDDs should be reserved only for the custom transformations that truly need low-level control. Option E is correct because preemptible (spot) VMs on Dataproc worker nodes cost significantly less than standard VMs, and Spark's fault tolerance (RDD lineage and task re-execution) handles preemption, making them suitable for batch jobs to cut cost. Option C is not inherently a performance or cost optimization—PySpark is convenient but Scala often performs comparably or better, so language choice alone does not optimize the job. Option D is incorrect because blindly setting partitions equal to cores is not optimal; partition count should be tuned to data size and shuffle behavior (e.g., 2-3 tasks per core), and over-partitioning adds scheduling overhead.

Answer

Use PySpark instead of Scala for simplicity

Answer

Increase the number of partitions to match the number of cores exactly

Question 7

A company uses Pub/Sub to ingest IoT sensor data and wants to process it with a Dataflow pipeline that uses fixed windows of 1 minute to compute average temperature. The pipeline also needs to handle malformed messages by routing them to a dead letter queue. Which TWO configurations should the engineer implement? (Choose TWO.)

Accepted Answer

Enable exactly-once processing for the Pub/Sub subscription used by the pipeline. Option C is correct because enabling exactly-once processing on the Pub/Sub subscription ensures Dataflow delivers and processes each IoT message only once, which is essential for accurate 1-minute fixed-window average temperature computations without duplicated readings skewing results. Option E is correct because configuring a dead letter sink, such as another Pub/Sub topic, for messages that exceed the retry limit is the standard mechanism to route malformed or unprocessable messages out of the main pipeline for later inspection or reprocessing. Option A is incorrect because `withCoder()` only specifies how data is encoded/decoded and does not provide any logic to detect or skip malformed messages. Option B is incorrect because `GroupByKey` performs grouping by key, not deduplication by a unique ID, and deduplication is not the stated requirement. Option D is incorrect because a Cloud Function to reprocess messages from the dead letter topic is an optional downstream recovery pattern, not a configuration required to handle malformed messages in the Dataflow pipeline itself.

Answer

Configure the pipeline to ignore malformed messages using `withCoder()`

Answer

Add a `GroupByKey` transform to deduplicate messages based on a unique ID

Answer

Set up a Cloud Function to reprocess messages from the dead letter topic

Question 8

A data engineer needs to schedule a recurring batch load of CSV files from an on-premises SFTP server into BigQuery. The files are generated daily and need to be loaded into a partitioned table by date. Which THREE steps should the engineer take? (Choose THREE.)

Accepted Answer

Use Storage Transfer Service to copy files from the SFTP server to Cloud Storage every day. Option B is correct because Storage Transfer Service natively supports SFTP servers as a source and can run on a daily schedule to move the generated CSV files into a Cloud Storage bucket, which is the required staging step before loading into BigQuery. Option D is correct because once the files are in Cloud Storage, a scheduled BigQuery load job (configured via the Cloud Console or the `bq load` command) can ingest them on a recurring basis into the target table. Option E is correct because to land the data in a partitioned table by date, the load job must specify partitioning, e.g. `--time_partitioning_field=<date_column>` for an existing DATE/TIMESTAMP column or `--range_partitioning` for integer ranges, so each daily load writes into the appropriate partition. Option A is not appropriate because a Cloud Function cannot directly read from an SFTP server without custom networking/credentials handling, and BigQuery loads are better handled by the native load job or Data Transfer Service rather than a bespoke function. Option C is incorrect because BigQuery Data Transfer Service does not support SFTP as a source; its connectors cover services like Cloud Storage, Amazon S3, Google Ads, and various SaaS applications, not generic SFTP servers.

Answer

Create a Cloud Function triggered by Cloud Scheduler to load files directly from SFTP to BigQuery

Answer

Use BigQuery Data Transfer Service with SFTP as a source

Question 9

A company uses Cloud Datastream to replicate data from a MySQL database to BigQuery in near real-time. Which TWO BigQuery features are automatically used by Datastream for optimal performance and consistency? (Choose TWO.)

Accepted Answer

BigQuery Storage Write API. Option D (BigQuery Storage Write API) is correct because Datastream uses this high-throughput, exactly-once ingestion API to write change data into BigQuery with low latency and strong consistency guarantees, rather than relying on the older streaming insert method. Option E (a materialized view that merges the change stream into the final table) is correct because Datastream's BigQuery destination creates a change-data-capture staging table plus a materialized view that continuously merges inserts, updates, and deletes into the final replicated table, keeping it consistent with the source MySQL database. Option A is not used automatically in this flow, since BigQuery Data Transfer Service is a separate scheduled/batch transfer mechanism for sources like SaaS apps and Cloud Storage, not Datastream's real-time replication path. Option B is incorrect because Datastream does not use legacy streaming inserts; it uses the Storage Write API for better performance and exactly-once semantics. Option C is incorrect because Datastream replicates data directly to BigQuery without automatically provisioning a Dataflow pipeline for transformation.

Answer

BigQuery Data Transfer Service

Answer

BigQuery legacy streaming inserts

Answer

A Dataflow pipeline to transform the data

Question 10

A data engineer needs to build a Dataflow pipeline that reads JSON messages from Pub/Sub, transforms them (including filtering, grouping, and enrichment), and writes the results to BigQuery. The pipeline must handle schema evolution in the input messages and minimize data loss. Which THREE settings or features should the engineer use? (Choose THREE.)

Accepted Answer

Use side inputs to enrich the data with reference data from BigQuery. Option A is correct because side inputs are the standard Apache Beam/Dataflow mechanism for enriching a streaming PCollection with slowly-changing reference data pulled from BigQuery, letting the main pipeline join each Pub/Sub message against that lookup table. Option C is correct because routing messages that fail JSON parsing or schema validation to a dead letter queue preserves them for later inspection/replay instead of dropping them, which directly satisfies the minimize-data-loss requirement. Option D is correct because Dataflow autoscaling dynamically adjusts the number of worker instances based on the backlog and CPU utilization of the streaming job, absorbing Pub/Sub volume spikes without manual intervention. Option B is not appropriate because setting allowed lateness to 0 discards late-arriving data rather than minimizing loss, and it does not address schema evolution. Option E is not required here because Streaming Engine reduces worker-side checkpoint and state storage overhead but does not by itself handle schema evolution, enrichment, or data-loss protection.

Answer

Set the `withAllowedLateness` to 0 for windowing to minimize latency

Answer

Enable Streaming Engine to reduce checkpoint size

Question 11

Your team maintains a Dataflow streaming pipeline that reads from Pub/Sub and writes to BigQuery using the Storage Write API with exactly-once semantics. During a regional outage, the pipeline's worker VMs are lost, but the Pub/Sub subscription retains messages for 7 days. After the outage, the pipeline restarts. What happens to messages that were already processed and written to BigQuery before the outage?

Accepted Answer

They are not reprocessed because the pipeline resumes from committed offsets in the Storage Write API, ensuring exactly-once semantics.. The Storage Write API's exactly-once semantics rely on committed offsets that Dataflow persists in its state backend. When the pipeline restarts, it resumes from those offsets rather than reprocessing all messages. This ensures that messages already written before the outage are not written again, preventing duplicates. Pub/Sub's at-least-once delivery and BigQuery's lack of automatic deduplication mean the pipeline must handle exactly-once itself.

Answer

They are reprocessed and cause duplicate rows in BigQuery because the pipeline does not persist state across restarts.

Answer

They are reprocessed, but BigQuery's automatic deduplication removes duplicates based on the message ID.

Answer

They are automatically deduplicated by Pub/Sub before reaching Dataflow, so no duplicates are written.

Question 12

A media analytics company ingests clickstream events from mobile apps. Events arrive as JSON messages published to a Pub/Sub topic at roughly 50,000 messages per second. The team needs to run a Dataflow streaming pipeline that writes the raw events into BigQuery with the lowest possible write cost, while allowing occasional duplicate rows because downstream queries deduplicate by event ID. Which BigQuery write method should the pipeline use?

Accepted Answer

BigQueryIO.Write with the Storage Write API in default stream mode. The Storage Write API in default stream mode offers high-throughput, low-cost ingestion with at-least-once semantics, which matches the requirement to allow duplicates and minimize cost. The legacy streaming inserts are expensive per row, FILE_LOADS with frequent triggers adds latency and job overhead, and direct REST calls do not scale. Thus, default stream mode is the appropriate choice.

Answer

Directly insert rows using the BigQuery REST API from a DoFn

Answer

BigQueryIO.Write with STREAMING_INSERTS

Answer

BigQueryIO.Write with FILE_LOADS and a triggering frequency of 1 minute

Question 13

A logistics company runs an Apache Hadoop cluster on Compute Engine VMs. Analysts need to run existing HiveQL and MapReduce jobs against the same on-premises HDFS data without rewriting them, while gaining the elasticity of a managed service. The security team will not allow raw data to be copied into a new storage system. Which Google Cloud service should you use?

Accepted Answer

Dataproc on Compute Engine with the HDFS connector configured against the existing cluster's NameNode. Dataproc clusters on Compute Engine can be pointed at an external HDFS NameNode using the cluster's Hadoop configuration files, which lets the existing HiveQL and MapReduce jobs execute unchanged against the original data. Because nothing is copied, the security restriction is honored, and the managed service still provides elastic scaling of compute nodes when workloads spike.

Answer

BigQuery external tables over the Hadoop cluster's HDFS directories

Answer

Cloud Data Fusion with the Hadoop source plugin reading HDFS over the network

Answer

Migrate the data to Cloud Storage and run Hive on Dataproc Serverless

Question 14

You are building a Dataflow batch pipeline that reads Avro files from Cloud Storage and writes to BigQuery. The files use a schema that evolves over time with new optional fields. You need the pipeline to handle schema changes without failing and to avoid loading duplicate rows when the job is retried. Which combination of BigQuery features should you use?

Accepted Answer

Use BigQuery Storage Write API with exactly-once semantics and enable schema update options. The Storage Write API provides exactly-once semantics using stream offsets, so retries do not duplicate rows. Enabling schema update options lets BigQuery add new optional fields automatically, so the evolving Avro schema is accommodated without pipeline failures.

Answer

Use the Storage Write API with WRITE_APPEND and disable schema updates

Answer

Use load jobs with write disposition WRITE_APPEND and ALLOW_FIELD_ADDITION

Answer

Use load jobs with WRITE_TRUNCATE and a pre-load deduplication step

Question 15

A logistics company uses a Dataflow streaming pipeline to process GPS pings from delivery vehicles. The pipeline reads from Pub/Sub, performs session windowing with a gap duration of 10 minutes, and writes aggregated trip segments to BigQuery. The team notices that some trip segments are emitted very late, hours after the session should have closed, causing incorrect billing. They want to ensure that late data does not affect already-emitted results and that the pipeline does not wait indefinitely for late data. Which combination of Beam settings should they apply?

Accepted Answer

Set allowed lateness to a finite value (e.g., 1 hour) and use accumulating mode with default trigger. To handle late data without indefinite waiting, set a finite allowed lateness and use accumulating mode. This allows the pipeline to incorporate late data up to the lateness limit and emit updated results, after which late data is discarded. A finite lateness bounds state retention and ensures timely finalization, while accumulating mode ensures that updates are reflected in the output.

Answer

Set allowed lateness to 0 and use discarding mode with early triggers

Answer

Set allowed lateness to a very large value (e.g., 7 days) and use accumulating mode with default trigger

Answer

Use a global window with a repeating trigger and no allowed lateness