PDE Practice Test 23 — 15 Questions

Question 1

Your Dataflow streaming pipeline writes to BigQuery and occasionally fails with `QuotaExceededError` on streaming inserts during peak hours. You want to reduce insert-driven quota pressure without changing the pipeline's output schema or losing exactly-once semantics. Which change should you make?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Switch the pipeline from the BigQueryIO streaming insert method to the Storage Write API with exactly-once semantics.. Legacy BigQuery streaming inserts draw on a quota pool that is easy to saturate at peak throughput. The Storage Write API is a separate ingestion path with higher and separately metered limits, and its exactly-once mode uses write streams with offsets so retries do not duplicate rows. Switching the sink preserves the output schema while relieving the quota bottleneck.

Answer

Add a `GroupByKey` before the BigQuery sink to batch records into larger write requests.

Answer

Increase the number of Dataflow workers and set `--maxNumWorkers` higher so inserts are spread across more threads.

Answer

Enable `--enableStreamingEngine` and raise the pipeline's `--diskSizeGb` to buffer failed inserts locally.

Question 2

Your team uses Cloud Composer to orchestrate a daily ETL workflow that extracts data from an on-premises database, transforms it in Dataproc, and loads it into BigQuery. The workflow occasionally fails due to transient network issues. You want to automatically retry failed tasks without manual intervention. What should you do?

Accepted Answer

Set the retries parameter and retry_delay on the Airflow tasks in the DAG.. Airflow tasks in Cloud Composer support retries and retry_delay parameters, which allow automatic retries of failed tasks. This is the correct way to handle transient failures. Other options either do not address task-level retries or could cause data duplication.

Answer

Configure the Dataproc cluster to automatically restart on failure.

Answer

Enable automatic retries on the Cloud Composer environment.

Answer

Use Cloud Scheduler to trigger the DAG multiple times until it succeeds.

Question 3

A Dataflow streaming pipeline processes events from Pub/Sub and writes to BigQuery using a dynamically generated table destination based on the event type. The pipeline is experiencing high latency, and the worker CPU utilization is low. Which action is most likely to reduce latency?

Accepted Answer

Enable Dataflow Streaming Engine to improve throughput and reduce latency.. Dataflow Streaming Engine moves state and computation from worker VMs to the backend service, reducing per-worker overhead and enabling better resource utilization. This directly addresses the symptom of high latency with low CPU utilization, which indicates workers are bottlenecked on shuffle or state management rather than compute.

Answer

Increase the batch size parameter in the BigQuery sink to write larger batches.

Answer

Reduce the number of workers to increase CPU utilization per worker.

Answer

Increase the worker disk size to reduce I/O wait time.

Question 4

A financial services firm runs an Apache Airflow workload on Cloud Composer 2 that ingests market data, runs dbt transformations, and loads curated tables into BigQuery. The DAG currently uses a single PythonOperator that runs a long shell command, and the team wants to make failures easier to diagnose and retries more granular. They also want to avoid rerunning already-successful upstream steps. Which change BEST meets these goals?

Accepted Answer

Split the monolithic PythonOperator into a chain of task-level operators such as BigQueryInsertJobOperator and DataprocSubmitJobOperator, with retries configured per task and depends_on_past set appropriately.. Decomposing a monolithic task into discrete operators is the standard Airflow pattern for granular retries and clearer observability. Each task maintains its own state, so a failure in the load step can be retried without rerunning ingestion or transformation. Purpose-built operators emit structured logs and status, and task-level retry configuration gives precise control over failure handling.

Answer

Move the shell command into a Cloud Function and invoke it from the PythonOperator using an HTTP call, keeping the DAG structure unchanged.

Answer

Increase the worker count and worker machine type on the Cloud Composer environment so the single PythonOperator task has more resources and completes faster.

Answer

Set the DAG's max_active_runs to 1 and enable catchup so that failed runs are automatically retried on the next schedule interval.

Question 5

A Dataflow streaming pipeline writes to BigQuery and has run in production for months. The team wants to add a transformation and deploy the change with zero data loss and no interruption to the running pipeline. Which deployment approach should they use?

Accepted Answer

Update the pipeline in place by calling the Dataflow update method with the new job graph and a compatible transform name.. Dataflow's update capability is the supported way to change a running streaming pipeline without stopping it. When the new graph keeps transform names consistent and the changes are compatible, the service swaps the graph while preserving streaming state and maintaining the sink's exactly-once guarantees. Stopping and restarting, running duplicate pipelines, or redeploying templates all either create a processing gap, risk data loss, or cause duplicate output.

Answer

Run the updated pipeline in parallel with the old one, then delete the old pipeline after a fixed time.

Answer

Use a Cloud Scheduler job to redeploy the pipeline template on a cron and let the new job take over.

Answer

Stop the existing pipeline, then start a new pipeline with the updated code and the same job name.

Question 6

A team runs a Dataflow batch pipeline that writes to BigQuery. They want to reduce cost and improve reliability of the write step. The pipeline currently uses BigQueryIO with the default write method. Which two changes should they make to achieve these goals? (Choose two.)

Accepted Answer

Enable the withAutoSharding option on the write transform so the number of shards adapts to the pipeline's throughput.. The Storage Write API lowers ingestion cost and supports exactly-once semantics, addressing both cost and reliability. Enabling withAutoSharding lets the write step adapt shard count to actual throughput, preventing shard bottlenecks and avoiding over-provisioning. Together these changes reduce per-row cost and improve the robustness of the BigQuery write step without altering the pipeline's business logic.

Answer

Increase the number of BigQuery load jobs by setting a very low triggering frequency so each micro-batch is written separately.

Answer

Set the BigQueryIO write disposition to WRITE_TRUNCATE on every micro-batch so the table is always consistent.

Answer

Disable dynamic work rebalancing so that workers process a fixed set of shards for the entire pipeline run.

Question 7

What is the primary purpose of Vertex AI Feature Store?

Accepted Answer

To store and serve features for machine learning models at scale. Vertex AI Feature Store is a managed service for storing, serving, and sharing ML features at scale, providing low-latency online serving and high-throughput batch serving with point-in-time correctness. It centralizes feature definitions so training and serving use consistent feature values, reducing training-serving skew.

Answer

To manage and track ML experiments

Answer

To train machine learning models using AutoML

Answer

To transform raw data into features using SQL

Question 8

A logistics company uses Cloud Pub/Sub to ingest shipment tracking events. They want to archive all events to Cloud Storage for long-term retention and also process them in real time with Dataflow. The events are published to a single topic. Which design should the data engineer use to ensure both archiving and real-time processing without data loss?

Accepted Answer

Create two subscriptions on the topic: one for the Dataflow pipeline and one for a Cloud Function that writes to Cloud Storage.. The correct design is to create two separate subscriptions on the Pub/Sub topic: one for Dataflow and one for a Cloud Function that archives to Cloud Storage. This fan-out pattern ensures that each event is delivered to both consumers independently, preventing data loss and allowing each to process at its own pace. It decouples archiving from real-time processing.

Answer

Configure the Pub/Sub topic to write to Cloud Storage directly using a Pub/Sub to Cloud Storage subscription.

Answer

Use a single subscription for both the Dataflow pipeline and a Cloud Function that writes to Cloud Storage.

Answer

Use a single subscription for the Dataflow pipeline, and have the pipeline write to both Cloud Storage and its real-time processing output.

Question 9

A media company needs to process a large number of small JSON files stored in Cloud Storage. They want to use a serverless, SQL-based approach to transform and aggregate the data without managing infrastructure. Which Google Cloud service should they use?

Accepted Answer

BigQuery. BigQuery is a serverless, SQL-based analytics service that can query data directly from Cloud Storage, including JSON files, using external tables or loading jobs. It eliminates infrastructure management and allows the media company to transform and aggregate data using familiar SQL. This makes it the most suitable choice among the options.

Answer

Cloud Dataproc

Answer

Cloud Dataflow

Answer

Cloud SQL

Question 10

Your organization uses Vertex AI Feature Store to serve features for a real-time fraud detection model. The model is deployed on a Vertex AI endpoint. After a data pipeline update, the model's online predictions became inconsistent. What is the most likely cause?

Accepted Answer

The feature store's online serving values are not synchronized with the batch feature values used during training.. In Vertex AI Feature Store, batch feature values used during model training and online serving values are stored separately. If a data pipeline update changes the batch feature values but the online serving values are not updated or synchronized, the model will receive different feature values at inference time than it was trained on, leading to inconsistent predictions. This is the most common cause of prediction drift after a pipeline change.

Answer

The model's prediction server is running out of memory.

Answer

The model was retrained with a different training dataset.

Answer

The online serving endpoint's model version was accidentally rolled back.

Question 11

A team deployed a model to Vertex AI Endpoint and notices latency spikes during peak hours. What should they first investigate?

Accepted Answer

Check if autoscaling is enabled and configured correctly. Latency spikes during peak hours typically indicate that the serving infrastructure is unable to handle the increased request volume. The first step is to check if autoscaling is enabled and configured correctly on the Vertex AI Endpoint, as this determines whether additional compute nodes are automatically provisioned to match demand. Without proper autoscaling, the endpoint will be overwhelmed, leading to queuing delays and latency spikes.

Answer

Switch to batch prediction

Answer

Reduce number of features

Answer

Increase machine type

Question 12

A company runs a Dataflow pipeline that processes a high-volume data stream. They notice that the pipeline's worker CPU utilisation is near 100% and the system lag is increasing. Which three actions can improve performance? (Choose three.)

Accepted Answer

Increase the number of workers.. Increasing the number of workers distributes the processing load across more parallel workers, reducing CPU utilization per worker and allowing the pipeline to keep up with the incoming data stream. This directly addresses both high CPU usage and increasing system lag by scaling out horizontally.

Answer

Increase the worker disk size.

Answer

Use batch processing instead of streaming.

Question 13

A data engineer is designing a batch ETL pipeline using Cloud Composer and Dataflow. The pipeline must be self-healing and retry on failures. Which Composer feature should they configure?

Accepted Answer

Retry policy on the DAG. Cloud Composer (based on Apache Airflow) allows you to configure a retry policy directly on the DAG or individual tasks. This enables the pipeline to automatically retry failed tasks according to parameters like `retries`, `retry_delay`, and `retry_exponential_backoff`, making the ETL pipeline self-healing without external services.

Answer

Use Cloud Tasks for retries

Answer

Cloud Composer with high availability

Answer

Dataflow retries

Question 14

A company needs to deploy a trained model for real-time predictions with low latency. Which Vertex AI resource should they use?

Accepted Answer

Vertex AI Endpoints. Vertex AI Endpoints are designed for online prediction, providing a managed service that hosts models for real-time inference with low latency. They automatically scale resources and handle traffic routing, making them the correct choice for deploying a trained model that needs to respond to individual prediction requests quickly.

Answer

Cloud TPU

Answer

Vertex AI Batch Prediction

Answer

Cloud Run

Question 15

You need to automate retraining of a model when new training data becomes available every week. The training pipeline runs on Vertex AI Pipelines and is triggered by Cloud Composer. After retraining, you want to evaluate the new model against a golden dataset. If the model's accuracy improves by at least 1%, it should be automatically deployed to the staging endpoint. What is the best way to implement the decision logic?

Accepted Answer

Add a conditional step in the Vertex AI Pipeline to evaluate the model and deploy if the accuracy improvement threshold is met.. Vertex AI Pipelines supports conditional execution natively via the `Condition` component, allowing you to evaluate the new model's accuracy against the golden dataset within the same pipeline and deploy only if the improvement threshold (≥1%) is met. This approach keeps the entire retraining, evaluation, and deployment workflow automated, auditable, and tightly coupled within a single orchestrated pipeline, avoiding external triggers or manual steps.

Answer

Use Cloud Functions to compare metrics and call the endpoint if conditions are met.

Answer

After training, run a batch prediction job on the golden dataset and compare metrics manually.

Answer

Use Vertex AI Experiments to log metrics and set up an alert to manually deploy.