PDE Ingesting and Processing the Data Practice Test 2 — 15 Questions

Question 1

Your Dataflow pipeline reads from Pub/Sub, performs transformations, and writes to BigQuery. You notice that the pipeline's autoscaling is not keeping up with sudden spikes in traffic, causing increased lag. The pipeline uses Classic Templates. Which change would most effectively improve autoscaling responsiveness?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Enable Dataflow Streaming Engine on the pipeline.. Enabling Dataflow Streaming Engine reduces the overhead of checkpointing and state management by offloading them to the service side, which allows the pipeline to scale more quickly in response to sudden traffic spikes. This directly addresses the autoscaling lag because Streaming Engine decouples compute from state, enabling faster worker adjustments without the bottleneck of persistent disk-based shuffle.

Answer

Switch to Dataflow Prime with Vertical Autoscaling enabled.

Answer

Increase the initial number of workers to handle the spike.

Answer

Use Flex Templates instead of Classic Templates.

Question 2

You are migrating an existing Kafka cluster to Google Cloud using Dataproc. The cluster handles high-throughput streaming data with strict ordering requirements per partition. Which choice of Dataproc configuration is most appropriate?

Accepted Answer

Use Dataproc with non-preemptible workers and persistent SSD storage for brokers.. Kafka brokers in a Dataproc cluster require persistent, non-preemptible workers to maintain data durability and strict ordering per partition. Preemptible workers can be terminated at any time, causing data loss or rebalancing that violates ordering guarantees. Persistent SSD storage provides the low-latency I/O needed for high-throughput Kafka workloads, while non-preemptible instances ensure broker stability and consistent replication.

Answer

Use Dataflow with Kafka IO instead of Dataproc.

Answer

Use Dataproc with local SSDs for better performance, and enable autoscaling.

Answer

Use Dataproc with preemptible workers to reduce cost, and attach standard persistent disks.

Question 3

You are designing a streaming pipeline that ingests events from Pub/Sub, enriches them with a machine learning model, and writes the results to BigQuery. The ML model is deployed on Cloud Run and has a high latency (500ms per request). You need to minimize the impact of slow ML inference on the overall pipeline throughput. Which approach should you take?

Accepted Answer

Use Dataflow to write events to Pub/Sub, then use a separate Dataflow pipeline that batches calls to Cloud Run.. It uses Dataflow to batch events before sending them to Cloud Run, which amortizes the 500ms per-request latency over multiple events, significantly increasing throughput. By writing events to Pub/Sub and then processing them in a separate Dataflow pipeline with batched calls, you decouple the ingestion from the inference and avoid blocking on each individual request.

Answer

Increase the number of Dataflow workers to compensate for the latency.

Answer

Use Cloud Functions to call Cloud Run and write directly to BigQuery.

Answer

Use Dataflow's ParDo with synchronous calls to Cloud Run for each element.

Question 4

Your team has a Dataflow pipeline that reads from BigQuery, transforms data, and writes to GCS. The pipeline is failing with 'Out of Memory' errors on the worker nodes. The input data is large but fits within the total cluster memory. Which configuration change is most likely to resolve the issue without increasing costs significantly?

Accepted Answer

Use a worker machine type with more memory, such as n2-highmem.. Out of Memory errors on Dataflow workers typically occur when the worker's memory is insufficient for the pipeline's operations, such as large shuffles or stateful processing. Using a worker machine type with more memory (e.g., n2-highmem) directly addresses the memory bottleneck without significantly increasing costs if the total cluster memory is already sufficient but per-worker memory is not. This is the most straightforward fix when the input data fits within total cluster memory but individual workers run out.

Answer

Shard the input into smaller reads using a BigQuery query.

Answer

Increase the disk size per worker.

Answer

Enable Dataflow Prime with vertical autoscaling.

Question 5

A company wants to stream real-time user click events from their web application into BigQuery for immediate analysis. Which combination of services is the most scalable and cost-effective for this use case?

Accepted Answer

Pub/Sub to Dataflow to BigQuery. Pub/Sub is the recommended service for ingesting streaming events, and Dataflow can read from Pub/Sub and write to BigQuery using the Storage Write API for high-throughput, exactly-once semantics.

Answer

Cloud Dataproc Spark Streaming to BigQuery

Answer

App Engine pushing logs to BigQuery

Answer

Cloud Functions writing directly to BigQuery

Question 6

A data engineer needs to load 2 TB of Avro files stored in Cloud Storage into BigQuery on a daily schedule. The schema is static and the data should overwrite the existing table each day. What is the most efficient way to accomplish this?

Accepted Answer

Use bq load with --replace flag in a cron job. The bq load command with the --replace flag is the most efficient method because it directly loads Avro files from Cloud Storage into BigQuery in a single, serverless operation without requiring any intermediate processing. Since the schema is static and the data overwrites the existing table daily, a simple cron job invoking bq load --replace is the simplest, fastest, and most cost-effective solution, avoiding the overhead of additional services like Dataflow or Data Transfer Service.

Answer

Create a BigQuery Data Transfer Service from Cloud Storage

Answer

Create a Dataflow pipeline to read Avro files and stream to BigQuery

Answer

Mount Cloud Storage as a filesystem and use SELECT INTO

Question 7

A company needs to run a Spark ML training job on a Dataproc cluster with high memory per node, but the cluster should automatically scale down when idle to save costs. Which configuration should they use?

Accepted Answer

Use a Dataproc cluster with preemptible secondary workers and cluster autoscaling. It combines preemptible secondary workers for cost-effective high-memory compute with cluster autoscaling, which automatically scales down the cluster when idle. Preemptible VMs are ideal for stateless Spark ML training tasks, and autoscaling ensures the cluster shrinks to save costs during inactivity. This configuration meets the requirement of high memory per node (via primary workers) while minimizing costs through idle scaling.

Answer

Use a single-node cluster with preemptible VMs

Answer

Enable Dataproc's default autoscaling with primary workers as preemptible

Answer

Create a cluster with custom machine types and no autoscaling

Question 8

You are migrating an on-premises Kafka cluster to Google Cloud. The cluster has 50 topics with a total throughput of 200 MB/s. You want to minimize operational overhead. Which approach is the most cost-effective?

Accepted Answer

Use Dataproc to run Kafka on a managed cluster. Managed Kafka services are not available natively on GCP; the recommended approach is to run Kafka on Dataproc, which provides a managed Hadoop/Spark environment with autoscaling.

Answer

Use Pub/Sub with Kafka-compatible client libraries

Answer

Deploy Kafka on Compute Engine instances with a managed instance group

Answer

Use Cloud NAT to route traffic to an on-premises Kafka cluster

Question 9

A data engineer is creating a Dataflow Flex Template for a batch pipeline that reads from BigQuery and writes to Cloud Storage. They need to pass a runtime parameter for the output bucket. How should they define this parameter?

Accepted Answer

Define the parameter in the pipeline's code and use ValueProvider. Dataflow Flex Templates require runtime parameters to be defined as `ValueProvider` objects in the pipeline code. This allows the parameter value to be supplied at job submission time via the `--parameters` flag, enabling the same template to be reused with different output buckets without recompilation.

Answer

Set an environment variable in Cloud Shell

Answer

Use the --parameters flag with the pipeline options

Answer

Hardcode the bucket name in the template

Question 10

A data engineer needs to schedule a nightly transfer of data from an Amazon S3 bucket to Cloud Storage. Which two steps are required to achieve this? (Choose TWO.)

Accepted Answer

Grant appropriate permissions to the transfer service account. Option A is correct because Storage Transfer Service performs transfers using a Google-managed service account, and that service account must be granted the necessary IAM permissions on the destination Cloud Storage bucket (e.g., roles/storage.admin or storage.objectCreator) and, for S3 sources, the corresponding AWS IAM permissions/policy on the source bucket so the transfer can read objects. Option D is correct because the Storage Transfer Service is the managed service designed for this scenario: you create a transfer job specifying the S3 source bucket, the Cloud Storage destination, and a schedule (e.g., a nightly recurrence) to run the transfer automatically. Option B is not required because Storage Transfer Service accesses S3 over the public internet (or via a private network only if you explicitly configure one); VPC peering between AWS and GCP is not a prerequisite. Option C is not required because gsutil rsync in a cron job is a manual/self-managed alternative, not the managed scheduling mechanism the scenario calls for. Option E is not required because a Cloud Function is an event-driven compute service, not the scheduling/transfer mechanism for S3-to-Cloud-Storage jobs.

Answer

Configure a VPC peering between AWS and GCP

Answer

Use gsutil rsync in a cron job

Answer

Set up a Cloud Function to copy files

Question 11

A company is designing a real-time analytics pipeline using Pub/Sub and Dataflow. They need to ensure exactly-once processing and handle late-arriving data. Which two configurations should they implement? (Choose TWO.)

Accepted Answer

Enable exactly-once delivery on Pub/Sub subscription. Option B is correct because enabling exactly-once delivery on the Pub/Sub subscription ensures messages are not redelivered after acknowledgment, which is a prerequisite for end-to-end exactly-once processing in the pipeline. Option D is correct because using a fixed window with allowed lateness and a trigger lets Dataflow emit results for late-arriving data while still producing timely output, directly addressing the late-data requirement. Option A is wrong because a global window with no triggers never fires results, so no analytics output is produced. Option C is wrong because Dataflow's default at-least-once mode permits duplicates and does not satisfy exactly-once processing. Option E is wrong because writing to Cloud Storage and batch loading to BigQuery introduces latency and does not provide exactly-once streaming semantics.

Answer

Set up a global window with no triggers

Answer

Use Dataflow's default at-least-once mode

Answer

Write all data to Cloud Storage and then batch load to BigQuery

Question 12

A data engineer needs to ingest daily Salesforce reports into BigQuery without writing custom code. The reports are exported to an Amazon S3 bucket on a schedule. Which service should they use to automate the transfer?

Accepted Answer

BigQuery Data Transfer Service. The BigQuery Data Transfer Service (BQDTS) is the correct choice because it supports scheduled, automatic ingestion of Salesforce reports exported to Amazon S3 into BigQuery without requiring custom code. It can connect to S3 as a source and load the data into BigQuery tables on a schedule, handling schema detection and incremental updates. This meets the requirement of no-code automation from an S3 bucket.

Answer

Cloud Dataproc

Answer

Cloud Composer

Answer

Cloud Storage Transfer Service

Question 13

A company uses BigQuery's Storage Write API in committed mode to stream data. They notice that some writes are failing with 'DEADLINE_EXCEEDED' errors during peak traffic. The pipeline is a Dataflow job using the Beam SDK. What is the MOST likely cause and solution?

Accepted Answer

The default RPC timeout is too low for the write throughput; increase the timeout in the Storage Write API configuration.. The most likely cause is that the default RPC timeout for the Storage Write API is too low for the write throughput during peak traffic, leading to DEADLINE_EXCEEDED errors. Increasing the timeout in the Storage Write API configuration allows the writes to complete without timing out.

Answer

The Dataflow workers lack sufficient memory; increase worker memory.

Answer

The Pub/Sub subscription is not sending acknowledgments; check the subscription.

Answer

The row schema has changed; update the schema before writing.

Question 14

A Dataflow pipeline reads from Pub/Sub, applies a keyed stateful ParDo that uses state variables to deduplicate events based on event ID, and writes to BigQuery. During a pipeline update, some events are duplicated in BigQuery. The state is not preserved across updates. Which configuration ensures exactly-once semantics during updates?

Accepted Answer

Take a snapshot of the pipeline before updating, then start the new pipeline from the snapshot.. Taking a snapshot of the pipeline before updating preserves the state, including deduplication state, and starting the new pipeline from the snapshot ensures that state is carried over, maintaining exactly-once semantics. Snapshots capture the pipeline's state and allow you to resume processing from a consistent point without reprocessing already-processed events.

Answer

Drain the pipeline and start the updated pipeline; all in-flight data will be processed.

Answer

Cancel the pipeline and restart it; Pub/Sub subscriptions will be rewound.

Answer

Use the Storage Write API's exactly-once delivery mode.

Question 15

A retail company wants to trigger a Cloud Run service whenever a new CSV file is uploaded to a specific Cloud Storage bucket. Which THREE components are needed to set up this event-driven architecture? (Choose 3)

Accepted Answer

Cloud Storage bucket with notifications enabled. Option A is correct because the Cloud Storage bucket must have notifications enabled (via a Pub/Sub notification configuration) so that object-creation events are emitted when a new CSV file is uploaded. Option B is correct because Eventarc is the service that captures those Cloud Storage events and routes them to the target by creating a trigger that delivers events to the Cloud Run service. Option D is correct because the Cloud Run service is the compute target that actually receives the event and executes the application logic in response to the new CSV file. Option C is not needed because Cloud Dataflow is a data-processing service for pipelines, not an event-routing mechanism for triggering Cloud Run. Option E is not needed as a separately configured component because Eventarc automatically provisions and manages the underlying Pub/Sub topic used to transport the Cloud Storage events.

Answer

Cloud Dataflow pipeline

Answer

Cloud Pub/Sub topic