PDE Practice Test 32 — 15 Questions

Question 1

A data engineering team uses Cloud Data Fusion to build ETL pipelines. They have a pipeline that reads from Cloud SQL, transforms data using Wrangler, and writes to BigQuery. The pipeline fails intermittently with a 'connection timeout' error from Cloud SQL. What is the best way to handle this?

Accepted Answer

Configure the Cloud SQL connector in Data Fusion to use retry logic and increase the connection timeout.. Option B is correct because Cloud Data Fusion's Cloud SQL connector can be configured with retry logic and an increased connection timeout to handle transient network issues. This directly addresses the intermittent 'connection timeout' error without requiring architectural changes, as the error is likely due to brief network latency or resource contention, not a persistent connectivity problem.

Answer

Use Cloud NAT to provide a static IP for Data Fusion to whitelist.

Answer

Increase the number of Data Fusion nodes to distribute the load.

Answer

Migrate Cloud SQL to Cloud Spanner to handle higher concurrency.

Question 2

A company uses Cloud Composer to orchestrate Dataproc and BigQuery jobs. They need to implement retry logic for transient failures. Which THREE features can help?

Accepted Answer

DAG retry_delay. Option B is correct because Cloud Composer (Apache Airflow) allows setting `retry_delay` at the DAG level to define the time delay between task retries. This is a native Airflow feature that helps handle transient failures by automatically retrying failed tasks after a specified delay, reducing manual intervention.

Answer

Dataflow pipeline retries

Answer

Cloud Composer high availability

Question 3

A data pipeline reads thousands of JSON files from Cloud Storage, processes them with Cloud Dataflow, and writes to BigQuery. The pipeline sometimes fails because of malformed JSON records. Which three steps should the data engineering team take to improve pipeline reliability? (Choose THREE.)

Accepted Answer

Integrate Cloud Pub/Sub as an intermediary to buffer and allow message retry. Option A is correct because integrating Cloud Pub/Sub as an intermediary decouples the ingestion of JSON files from the Dataflow pipeline. Pub/Sub provides at-least-once delivery and automatic retries for messages that are not acknowledged, which buffers against transient failures and malformed records. This allows the pipeline to pull messages at its own pace and retry processing without losing data.

Answer

Use a try-catch block in the pipeline to retry processing failed records

Answer

Create a Cloud Monitoring alert on pipeline failures

Question 4

Given the query plan, what is the most likely reason this query is efficient despite processing 10 billion rows?

Accepted Answer

The table is partitioned by sale_date.. Option B is correct because partitioning by sale_date enables partition pruning, which allows the query engine to scan only the relevant partitions instead of the entire 10-billion-row table. This drastically reduces the amount of data read and processed, making the query efficient even with a large total row count.

Answer

The query uses a wildcard function.

Answer

The table is materialized.

Answer

The table is clustered by product_id.

Question 5

A company runs a nightly Dataproc batch job to process large log files. The job is idempotent and can tolerate node failures if restarted. Minimizing cost is critical. What is the most cost-effective cluster design?

Accepted Answer

Use preemptible instances for all nodes and enable automatic restart. Preemptible instances cost about 80% less than standard instances, making them the most cost-effective choice for fault-tolerant, idempotent batch jobs. Enabling automatic restart ensures that if a preemptible instance is terminated (which can happen at any time), Dataproc will automatically recreate it, maintaining cluster capacity without manual intervention. This design minimizes cost while preserving the job's ability to complete despite node failures.

Answer

Use standard instances with autoscaling based on YARN memory

Answer

Use all preemptible instances and configure the cluster to delete after the job completes

Answer

Use a single-node cluster with a high-memory machine type

Question 6

What is the root cause of this error and the correct solution?

Accepted Answer

The Dataflow worker service account needs the BigQuery Data Viewer role.. Option D is correct because Dataflow workers execute under a specific service account (compute engine default or custom), and that service account must have the BigQuery Data Viewer role to read data from BigQuery tables. Without this permission, the workers cannot access the source data, causing the job to fail with access errors. The BigQuery User role is insufficient for reading table data, and the BigQuery Admin role is overly permissive and not required for this task.

Answer

The BigQuery table requires authorized view access.

Answer

The user running the job needs the BigQuery Admin role.

Answer

The Dataflow service account needs the BigQuery User role.

Question 7

A team is using BigQuery to analyze petabyte-scale data. They notice that queries are slow and expensive due to full table scans. They have already partitioned by date. What additional optimization should they implement?

Accepted Answer

Cluster by frequently filtered columns. Clustering by frequently filtered columns (option B) organizes data within each partition based on the sort order of those columns. This allows BigQuery to prune blocks during query execution, significantly reducing the amount of data scanned and improving both performance and cost. Since the table is already partitioned by date, clustering adds a secondary ordering that targets the most common filter predicates, avoiding full table scans within each partition.

Answer

Use materialized views

Answer

Convert to native tables

Answer

Use query caching

Question 8

A data engineer is designing a batch ETL pipeline using Cloud Composer and Dataflow. The pipeline must be self-healing and retry on failures. Which Composer feature should they configure?

Accepted Answer

Retry policy on the DAG. Option B is correct because Cloud Composer (based on Apache Airflow) allows you to configure a retry policy directly on the DAG or individual tasks. This enables the pipeline to automatically retry failed tasks according to parameters like `retries`, `retry_delay`, and `retry_exponential_backoff`, making the ETL pipeline self-healing without external services.

Answer

Use Cloud Tasks for retries

Answer

Cloud Composer with high availability

Answer

Dataflow retries

Question 9

A team runs a Dataflow streaming pipeline that reads from Pub/Sub, windows events by processing time, and writes to BigQuery. Some late-arriving events are being dropped. The requirement is to include all events that arrive within 10 minutes of the watermark. Which pipeline configuration should be used?

Accepted Answer

Use fixed windows with .withAllowedLateness(Duration.standardMinutes(10)). Option B is correct because `withAllowedLateness(Duration.standardMinutes(10))` on a fixed window allows late-arriving events to be included up to 10 minutes after the watermark passes the window's end. This directly meets the requirement to retain events arriving within 10 minutes of the watermark, while still using processing-time windows as specified.

Answer

Use sliding windows with no allowed lateness

Answer

Use fixed windows with withAllowedLateness(Duration.standardSeconds(10))

Answer

Switch from processing time to event time and use default triggers

Question 10

A Dataflow pipeline reads from Cloud Pub/Sub and writes to Cloud Storage. The pipeline needs to guarantee exactly-once processing despite worker failures. Which configuration ensures exactly-once semantics?

Accepted Answer

Enable exactly-once delivery on the Pub/Sub subscription and use an idempotent sink. Option D is correct because Pub/Sub subscriptions can be configured with exactly-once delivery (using the `enableExactlyOnceDelivery` flag), which ensures that each message is delivered to the subscriber exactly once. Combining this with an idempotent sink (e.g., Cloud Storage with unique filenames or deduplication logic) guarantees that even if a worker fails and the pipeline retries, the output will not contain duplicates. This is the only option that directly addresses both the source and sink to achieve end-to-end exactly-once semantics.

Answer

Use a side input from a deduplication dataset

Answer

Set the pipeline to use a global window with no early triggers

Answer

Insert a Reshuffle transform after reading

Question 11

A data pipeline using Cloud Pub/Sub and Cloud Dataflow is experiencing duplicate messages. The source system publishes messages at least once. What Dataflow technique ensures exactly-once processing?

Accepted Answer

Use idempotent sinks. Option A is correct because idempotent sinks ensure that even if Cloud Pub/Sub delivers the same message multiple times (due to its at-least-once delivery semantics), the Dataflow pipeline can deduplicate or safely reapply the same data without causing duplicates in the output. This is achieved by designing the sink (e.g., BigQuery with insertId, Cloud Storage with unique filenames) to recognize and ignore repeated writes, effectively providing exactly-once processing semantics downstream.

Answer

Use GlobalWindows

Answer

Set watermark threshold

Answer

Enable streaming engine

Question 12

A company needs to stream data from a fleet of IoT devices to BigQuery for near-real-time analytics. The data volume is unpredictable and can spike during certain events. Which Google Cloud service should be used as the ingestion point to handle variable throughput with minimal operational overhead?

Accepted Answer

Cloud Pub/Sub. Cloud Pub/Sub is the correct choice because it is a fully managed, scalable messaging service designed to decouple data producers from consumers, handling unpredictable and spiky throughput without requiring manual scaling. It can ingest millions of messages per second and buffer them until BigQuery is ready to consume, ensuring near-real-time analytics with minimal operational overhead.

Answer

Cloud Datastore

Answer

Cloud Functions

Answer

Cloud Storage

Question 13

A data engineer is responsible for a batch ETL pipeline that runs daily using Cloud Composer and Dataproc. The pipeline extracts data from Cloud SQL, transforms it with Spark, and loads to BigQuery. Last night, the pipeline failed because the Spark job ran out of memory. The team needs a solution that prevents future failures without manual intervention. Options: A. Use a larger machine type for Dataproc. B. Enable Dataproc autoscaling and configure memory-based scaling. C. Split the Spark job into multiple stages. D. Use Cloud Functions to retry the job.

Accepted Answer

Enable Dataproc autoscaling and configure memory-based scaling. Option A is correct because Dataproc autoscaling with memory-based scaling dynamically adjusts the cluster size based on the memory utilization of running jobs. This prevents out-of-memory failures by automatically adding worker nodes when memory pressure increases, without requiring manual intervention or pre-provisioning oversized clusters. It directly addresses the root cause—insufficient memory during peak processing—while maintaining cost efficiency.

Answer

Use Cloud Functions to retry the job

Answer

Use a larger machine type for Dataproc

Answer

Split the Spark job into multiple stages

Question 14

A data scientist has trained an XGBoost model on Vertex AI and wants to deploy it to an endpoint with automatic scaling based on traffic. What is the recommended deployment approach?

Accepted Answer

Use Vertex AI Endpoints with automatic scaling enabled. Vertex AI Endpoints with automatic scaling enabled is the recommended approach because it directly supports deploying trained models (including XGBoost) as online prediction endpoints with built-in autoscaling based on incoming traffic. This service manages the underlying infrastructure, load balancing, and scaling policies, aligning with the requirement for automatic scaling without additional containerization or serverless overhead.

Answer

Export the model to a container and deploy on Cloud Run

Answer

Use AI Platform Prediction with batch prediction

Answer

Deploy the model as an API on App Engine

Question 15

A company is designing a data processing system that must handle both batch and streaming workloads with unified pipeline code. Which two Google Cloud services are most suitable for implementing a unified batch and streaming pipeline? (Choose TWO.)

Accepted Answer

Apache Beam SDK. Apache Beam SDK (C) provides a unified programming model that allows developers to write a single pipeline that can execute in both batch and streaming modes without code changes. It abstracts the underlying execution engine, making it the correct choice for unified pipeline code.

Answer

Cloud Data Fusion

Answer

BigQuery

Answer

Cloud Dataproc