PDE Practice Test 6 — 10 Questions

Question 1

An e-commerce company runs a daily batch pipeline that processes clickstream data from Cloud Storage using Cloud Dataproc with Spark. The pipeline includes a join between a large fact table and a small dimension table. The dimension table is stored in Cloud Storage as a CSV file. The join is slow due to shuffling. The data engineer considers broadcasting the dimension table. However, the dimension table is updated daily and the pipeline reads the latest version. What is the best approach to implement this optimization?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use DataFrame.join with broadcast hint on the dimension DataFrame. Broadcasting the small dimension table using the broadcast hint (e.g., `broadcast(dimensionDF)`) forces Spark to replicate the dimension data to all executor nodes, eliminating the need for a shuffle during the join. This is ideal when the dimension table is small enough to fit in executor memory, and since the pipeline reads the latest CSV daily, the broadcast will automatically use the updated data without additional code changes.

Answer

Read the fact table and dimension table into separate DataFrames and use standard join

Answer

Read the dimension table as an RDD and collect as a map, then use map-side join

Answer

Increase the spark.sql.autoBroadcastJoinThreshold to a large value

Question 2

A company runs a real-time anomaly detection system on Google Cloud. Streaming data from IoT devices is ingested via Pub/Sub, processed by Dataflow (Apache Beam), and results are written to Bigtable for low-latency serving. Recently, the system has been experiencing increased latency and occasional data loss. The Dataflow pipeline shows high system lag and backlog in Pub/Sub. The Bigtable cluster has 3 nodes and is reporting high CPU utilization (over 90%). The team suspects the issue is with the pipeline configuration. They have already verified that there are no errors in the pipeline code and no network issues. Which action should they take to resolve the issue?

Accepted Answer

Increase the number of Bigtable nodes to handle the write throughput.. The high CPU utilization on Bigtable (over 90%) indicates that the cluster is saturated and cannot keep up with the write throughput from Dataflow. This causes backpressure in the pipeline, leading to increased system lag and backlog in Pub/Sub, and eventually data loss when Pub/Sub messages expire. Increasing the number of Bigtable nodes directly addresses the bottleneck by distributing the write load and reducing CPU pressure, which allows the pipeline to drain the backlog and reduce latency.

Answer

Change the Dataflow worker machine type to n2-standard-8.

Answer

Decrease the batch size in the Dataflow pipeline to reduce latency.

Answer

Increase the number of Dataflow workers to process messages faster.

Question 3

A data engineer needs to ingest on-premises Oracle CDC data into BigQuery in near real-time with minimal operational overhead. Which service should they use?

Accepted Answer

Datastream. Datastream is purpose-built for streaming change data capture (CDC) from Oracle and other sources into BigQuery with near-real-time latency and minimal operational overhead. It handles schema propagation, checkpointing, and automatic retries, eliminating the need to manage custom ingestion pipelines.

Answer

Pub/Sub + Dataflow

Answer

Storage Transfer Service

Answer

Transfer Appliance

Question 4

An organization needs to restrict access to BigQuery and Cloud Storage so that data can only be accessed from within a specific VPC network and cannot be exfiltrated. Which Google Cloud feature should they use?

Accepted Answer

VPC Service Controls. VPC Service Controls (option B) is the correct choice because it creates a security perimeter around Google Cloud services like BigQuery and Cloud Storage, preventing data exfiltration even from within a VPC. It enforces context-aware access based on the VPC network, ensuring data can only be accessed from authorized VPC sources and blocking unauthorized transfers outside the perimeter.

Answer

Private Service Access

Answer

VPC firewall rules

Answer

IAM conditions

Question 5

A company wants to version its ML models and track lineage from training data to deployed model. Which Google Cloud service should they use?

Accepted Answer

Vertex AI ML Metadata. Vertex AI ML Metadata is the Google Cloud service designed to track ML artifacts, executions, and contexts, providing model versioning and lineage from training data through to deployed models. It is the native metadata store for Vertex AI pipelines and experiments.

Answer

Cloud Storage with object versioning

Answer

Data Catalog

Answer

Artifact Registry

Question 6

A data engineer needs to schedule recurring nightly loads from Amazon S3 to Google Cloud Storage. The data is in CSV format and the volume is approximately 500 GB per night. Which Google Cloud service should they use?

Accepted Answer

Storage Transfer Service. The Storage Transfer Service is designed for online data transfers from external cloud providers like Amazon S3 to Google Cloud Storage. It supports scheduling recurring nightly transfers, handles large volumes (500 GB/night), and automatically retries failed transfers, making it the correct choice for this use case.

Answer

Transfer Appliance

Answer

BigQuery Data Transfer Service

Answer

Datastream

Question 7

A company uses Cloud Dataproc for large-scale Spark jobs. They notice that some jobs are failing due to insufficient memory on the worker nodes. They want to improve memory management without over-provisioning. Which three configurations should they apply? (Choose 3)

Accepted Answer

Set spark.executor.memory to a value that fits within the node memory. Setting spark.executor.memory to a value that fits within the node memory ensures that each executor does not exceed the available RAM on a worker node, preventing out-of-memory (OOM) errors. This configuration directly controls the heap size allocated to each executor, and when combined with spark.executor.cores and spark.executor.instances, it allows precise memory budgeting per node. Over-provisioning is avoided by calculating the maximum safe executor memory as (node memory - OS overhead - HDFS cache) / number of executors per node.

Answer

Use local SSDs for temporary storage

Answer

Use preemptible worker nodes for volatile tasks

Question 8

You are designing a row key for Cloud Bigtable to store user activity logs. Each log entry has a timestamp (millisecond precision) and a user ID. There will be millions of writes per second from many users. To avoid hotspotting, which row key design is BEST?

Accepted Answer

hash(userID)#userID#timestamp_millis. Best because it uses a hash of the user ID as the row key prefix, which distributes writes across all Bigtable nodes and avoids hotspotting. Appending the user ID and timestamp ensures uniqueness and supports efficient queries for a specific user's logs. This design prevents the sequential timestamp from creating a single hot node, which is critical for handling millions of writes per second.

Answer

timestamp_millis#hash(userID)

Answer

timestamp_millis#userID

Answer

userID#timestamp_millis

Question 9

A data platform team uses Cloud Composer 2 to orchestrate a DAG that runs a Dataproc Serverless batch job producing a partitioned BigQuery table. The DAG passes the output location to a downstream task that runs a dbt model. The team wants failures in the dbt task to automatically trigger a retry of only that task, and they want the DAG to expose the Dataproc job ID in the Airflow UI for troubleshooting. Which approach BEST satisfies both requirements?

Accepted Answer

Use the DataprocSubmitJobOperator and pass the returned job reference to XCom, then set retries on the dbt task and read the XCom value in its template fields.. DataprocSubmitJobOperator returns a job reference that Airflow pushes to XCom, which makes the job ID available for downstream tasks and visible in the Airflow UI. Configuring retries specifically on the dbt task ensures that a transient failure there triggers only that task to rerun, leaving the already-successful Dataproc submission intact. This combination precisely meets both the observability and granular retry goals.

Answer

Use the DataprocSubmitJobOperator and configure the DAG's default_args with retries, which applies the retry policy to all tasks including the dbt task.

Answer

Use a single PythonOperator that submits the Dataproc job via the API, waits for completion, and then invokes dbt, with retries set on the PythonOperator.

Answer

Use the DataprocSubmitJobOperator with retries set on it, and add a separate downstream PythonOperator that runs dbt without its own retry configuration.

Question 10

A company is using Pub/Sub to ingest clickstream events. They need to ensure that events are delivered to a subscriber at least once, but duplicates can be tolerated. They also need to filter events by type before processing. Which subscription configuration should be used?

Accepted Answer

Pull subscription with a filter on event type attribute. A pull subscription with a filter on the event type attribute satisfies both requirements: Pub/Sub's at-least-once delivery is the default for pull subscriptions, so duplicates are tolerated, and message filtering via subscription filter expressions (on attributes like event type) prevents unwanted events from being delivered. This is the only option that combines at-least-once semantics with server-side filtering.

Answer

Pull subscription with exactly-once delivery enabled

Answer

Push subscription with no filter

Answer

Push subscription with a dead letter topic