PDE Hard Questions — 25 Difficult Questions

Question 1

A company runs a Dataflow streaming pipeline that reads from Cloud Pub/Sub and writes to BigQuery. The pipeline uses a side input that is a large lookup table (10 GB) stored in Cloud Storage. The side input is updated hourly. The pipeline experiences high latency and OOM errors on workers. What is the best approach to resolve this?

Accepted Answer

Use a Cloud Bigtable table as a side input via a RichSDF.. Option A is correct because using a Cloud Bigtable table as a side input via a RichSDF (Rich Splittable DoFn) allows the pipeline to perform point lookups on the large (10 GB) lookup table without loading it entirely into worker memory. This avoids OOM errors and reduces latency by leveraging Bigtable's low-latency, scalable key-value storage, which is ideal for high-throughput streaming pipelines that require frequent, random access to a large, frequently updated dataset.

Answer

Use a side input from a PCollection and broadcast it.

Answer

Increase the number of workers to distribute the side input.

Answer

Increase the worker memory to 16 GB per worker.

Question 2

A data science team uses Vertex AI Pipelines to automate retraining. They want to ensure that only models with performance above a threshold are deployed. Which component should they add to the pipeline?

Accepted Answer

Vertex AI Model Evaluation. Vertex AI Model Evaluation provides built-in evaluation metrics and threshold-based validation that can be used as a pipeline condition to gate model deployment. By adding a Model Evaluation component, the pipeline can compare model performance against a predefined threshold and only proceed to deploy if the metrics (e.g., AUC, precision, recall) meet or exceed the required value.

Answer

Vertex AI Feature Store

Answer

Cloud Build trigger

Answer

Cloud Monitoring alert

Question 3

A data scientist uses Vertex AI Workbench notebooks for model development. They want to share the environment with team members while maintaining version control. Which approach should they use?

Accepted Answer

Use a user-managed notebook instance with multiple users. A user-managed notebook instance with multiple users is the correct approach because Vertex AI Workbench supports collaboration by allowing multiple users to access the same instance via IAM permissions, while the underlying Git integration enables version control. This setup provides a shared, persistent environment where team members can work on the same codebase without duplicating work, and changes can be tracked through Git repositories.

Answer

Use Cloud Shell and clone the repo

Answer

Share the notebook via Cloud Storage

Answer

Store notebooks in Cloud Source Repositories

Question 4

A data engineer is designing a batch ETL pipeline that reads CSV files from Cloud Storage, transforms them using Dataproc, and writes the results to BigQuery. The data volume is expected to grow 10x in the next year. Which design approach best balances cost and performance?

Accepted Answer

Use a Dataproc cluster with preemptible worker nodes and autoscaling enabled.. Option C is correct because preemptible worker nodes significantly reduce cost (up to 80% discount) while autoscaling dynamically adjusts cluster size to match the growing workload, ensuring performance without over-provisioning. This combination handles the 10x data growth efficiently by scaling out during peak loads and scaling in during lulls, using preemptible instances for fault-tolerant tasks like transformation.

Answer

Create a single large persistent Dataproc cluster to handle the peak load.

Answer

Use Cloud Data Fusion to visually design the pipeline and run it on Dataproc.

Answer

Migrate the pipeline to Dataflow with Apache Beam and use flexRS for cost savings.

Question 5

A financial services company uses Cloud Composer to orchestrate a daily workflow that includes a Dataproc job for risk analysis. The workflow sometimes fails because the Dataproc cluster creation times out. The cluster creation typically takes 3 minutes, but occasionally takes over 10 minutes. What is the most effective way to handle this variability?

Accepted Answer

Create a long-running Dataproc cluster that remains idle and reuse it for each workflow.. Option A is correct because creating a long-running Dataproc cluster and reusing it eliminates the variable cluster creation time that causes timeouts. Cloud Composer (Airflow) can manage cluster lifecycle separately from the workflow, ensuring the cluster is always available when the Dataproc job runs. This approach decouples cluster provisioning from job execution, making the workflow resilient to creation delays.

Answer

Implement a retry loop with exponential backoff in the DAG.

Answer

Use preemptible VMs for the cluster to reduce cost and improve creation speed.

Answer

Increase the cluster creation timeout in the Airflow configuration.

Question 6

A company is migrating an on-premises Hadoop cluster to Google Cloud. They need to run existing Spark jobs with minimal modification. Which THREE strategies should they consider? (Choose THREE.)

Accepted Answer

Use Cloud Dataproc with Spark and Hive components.. Option B is correct because Cloud Dataproc is a managed Spark and Hadoop service that supports the same Spark and Hive components used on-premises, allowing existing Spark jobs to run with minimal modification. It provides native integration with Cloud Storage, which can replace HDFS without changing job logic, and the Dataproc Jobs API enables programmatic job submission, preserving existing workflows.

Answer

Migrate to BigQuery for all analytics.

Answer

Rewrite Spark jobs as Dataflow pipelines.

Question 7

A data pipeline ingests sensor data from IoT devices via Cloud Pub/Sub, processes it with Cloud Dataflow, and writes to BigQuery. The pipeline is failing with high latency and data loss. Which troubleshooting step should be taken first?

Accepted Answer

Check Stackdriver logging for error messages.. Option A is correct because Stackdriver (now Cloud Logging) is the first place to investigate when a Dataflow pipeline experiences high latency and data loss. Dataflow automatically logs errors, worker failures, and system messages to Cloud Logging, which can reveal root causes such as insufficient resources, stuck steps, or Pub/Sub subscription issues. Checking logs first avoids premature scaling or configuration changes that may not address the actual problem.

Answer

Disable exactly-once processing in Dataflow.

Answer

Increase the number of Dataflow workers.

Answer

Switch to BigQuery streaming inserts.

Question 8

A company needs to process sensitive healthcare data with strict compliance requirements. They want to use Cloud Dataflow but must ensure data is encrypted end-to-end and audit logs are retained. Which combination of features should they enable?

Accepted Answer

Use Customer-Managed Encryption Keys (CMEK) and VPC Service Controls.. Option A is correct because Customer-Managed Encryption Keys (CMEK) allow the company to control the encryption keys used to protect data at rest in Cloud Dataflow, while VPC Service Controls provide a security perimeter that prevents data exfiltration and ensures end-to-end encryption boundaries. Together, they address the compliance requirement for encryption control and audit logging by restricting data movement within a VPC service perimeter and using customer-managed keys for data encryption.

Answer

Use Data Loss Prevention API to redact sensitive data.

Answer

Enable Cloud Audit Logs and VPC Service Controls.

Answer

Enable default encryption at rest and in transit.

Question 9

A financial services firm processes sensitive transactions using Cloud Dataflow. The pipeline reads from Pub/Sub, performs stateful processing (e.g., fraud detection), and writes to Cloud Spanner. Compliance requires exactly-once processing semantics. Which configuration ensures exactly-once processing?

Accepted Answer

Set Dataflow pipeline to exactly-once mode and design Spanner writes to be idempotent.. Option C is correct because exactly-once processing in a Dataflow pipeline requires the pipeline itself to be set to exactly-once mode (which uses consistent snapshots and transactional sinks) and the output writes to Spanner to be idempotent. This combination ensures that even if a record is reprocessed due to failures, the final state in Spanner remains consistent, satisfying compliance requirements.

Answer

Configure Pub/Sub to use exactly-once delivery mode.

Answer

Use Pub/Sub with at-least-once delivery and Dataflow with at-least-once processing mode.

Answer

Enable Dataflow's streaming engine and use Spanner's built-in retry logic.

Question 10

A healthcare company streams patient monitoring data to Cloud Pub/Sub. A Dataflow pipeline reads the stream, enriches with patient records from BigQuery, and writes to Bigtable for real-time queries. The BigQuery lookup is slow and causes pipeline lag. What is the best approach to improve performance?

Accepted Answer

Use a side input from a BigQuery query with a global window and periodic refresh.. Option D is correct because using a side input from BigQuery with a global window and periodic refresh allows the Dataflow pipeline to cache the patient records in memory across all workers, avoiding per-element slow lookups. This pattern leverages Beam's side input semantics to broadcast a relatively static lookup table, significantly reducing latency compared to synchronous BigQuery queries for each incoming event.

Answer

Increase the number of Dataflow workers and use vertical scaling.

Answer

Use BigQuery's streaming read API in the pipeline.

Answer

Pre-join the data in a batch pipeline and load into Bigtable.

Question 11

A company runs a batch processing job on Dataproc that uses Apache Spark to process 500 GB of data daily. The job completes successfully but takes 4 hours. The team wants to reduce the runtime to under 2 hours without increasing cost. What should they do?

Accepted Answer

Use preemptible VMs for worker nodes and increase the number of workers.. Preemptible VMs cost significantly less than standard VMs (about 60-80% discount). By using preemptible VMs for worker nodes, you can increase the number of workers (and thus parallelism) without increasing cost. This directly reduces runtime by distributing the 500 GB workload across more executors, while the cost savings from preemptible VMs offset the additional nodes.

Answer

Increase the master node's machine type to n2-standard-8.

Answer

Increase the machine type of worker nodes to n2-highmem-8.

Answer

Migrate the job to Dataflow with autoscaling enabled.

Question 12

Which THREE considerations are important when designing a data lake on Google Cloud using Cloud Storage?

Accepted Answer

Choose the appropriate storage class based on access patterns.. Option C is correct because selecting the appropriate storage class (e.g., Standard, Nearline, Coldline, Archive) based on data access patterns directly optimizes cost and performance in Cloud Storage. For a data lake, where data may be accessed frequently initially and rarely later, matching the storage class to the access pattern avoids paying premium rates for infrequently accessed data.

Answer

Use Cloud Storage's eventual consistency model for cost savings.

Answer

Define a schema when writing data to enforce data quality.

Question 13

Which THREE best practices should be followed when designing a Dataflow pipeline for real-time data processing?

Accepted Answer

Set up monitoring alerts for system lag and data freshness.. Option A is correct because monitoring alerts for system lag and data freshness are essential for maintaining operational visibility in real-time Dataflow pipelines. System lag (the time between data ingestion and processing) and data freshness (how current the processed output is) directly impact the pipeline's ability to meet latency SLAs. Without these alerts, issues like worker backpressure or Pub/Sub subscription backlog can go unnoticed, leading to stale or lost data.

Answer

Use static side inputs that are loaded once at pipeline start.

Answer

Use global windows with early triggers for low latency.

Question 14

In the Vertex AI Pipeline component YAML exhibit, the component is designed to evaluate a model and produce metrics. If the threshold_accuracy is set to 0.85, what is the expected behavior of this component?

Accepted Answer

It will output the evaluation metrics, and the pipeline can use them for conditional decisions. In Vertex AI Pipelines, a component's YAML definition specifies inputs, outputs, and implementation. Setting `threshold_accuracy` to 0.85 defines a parameter that the component can use internally, but by itself it does not trigger deployment or cause failure. The component's expected behavior is to output evaluation metrics, and the pipeline can then use those metrics in conditional logic (e.g., via `Condition` or `if/else` tasks) to decide subsequent steps, such as model deployment or retraining.

Answer

It will deploy the model if the accuracy meets the threshold

Answer

It will ignore the threshold_accuracy input if not provided

Answer

It will fail if the model accuracy is below 0.85

Question 15

You have deployed a TensorFlow model on Vertex AI Endpoints with autoscaling. The model receives high traffic during peak hours, but you notice that inference latency increases significantly during cold starts. Which strategy would best minimize cold-start latency without incurring unnecessary cost?

Accepted Answer

Set minNodes to a value that handles baseline traffic, and use traffic splitting to gradually shift traffic to new replicas. Setting minNodes to a value that handles baseline traffic ensures that a minimum number of replicas are always warm, eliminating cold starts for baseline requests. Traffic splitting gradually shifts new traffic to newly created replicas, allowing them to warm up before receiving full load, which minimizes latency spikes without over-provisioning resources.

Answer

Set minNodes to 0 and enable node auto-scaling

Answer

Increase maxNodes to allow more replicas during peak, and rely on Kubernetes Horizontal Pod Autoscaler

Answer

Use Cloud Functions with Cloud Run for the model inference to leverage serverless cold-start mitigation

Question 16

A company runs a Cloud Dataflow streaming pipeline that reads from Cloud Pub/Sub, performs a fixed window of 10 seconds, joins with a slowly-changing dimension table stored in Cloud Bigtable, and writes results to BigQuery. The pipeline has been running for months but recently started exhibiting increasing latency and occasional data loss. The pipeline uses default settings with autoscaling enabled (min 2, max 20 workers). The Bigtable cluster has 3 nodes. The dimensions are updated infrequently. The latency has grown from seconds to minutes. Examining the Dataflow monitoring UI, you see that the 'System Lag' metric is increasing, and some windows are not being emitted. The CPU utilization on Bigtable nodes is below 50%. There are no errors in the logs. Which action is most likely to resolve the issue?

Accepted Answer

Set the pipeline option --maxNumWorkers to a value between 5 and 10.. The increasing system lag and unemitted windows in a streaming pipeline with autoscaling (2–20 workers) and a 3-node Bigtable cluster indicate that the pipeline is bottlenecked by the number of workers, not by Bigtable performance. With default autoscaling, Dataflow may not scale up aggressively enough to handle the sustained load, causing backlog and window expiration. Capping maxNumWorkers to 5–10 ensures sufficient parallelism without over-provisioning, allowing the pipeline to catch up and emit windows reliably.

Answer

Increase the window duration to 30 seconds to reduce the number of windows.

Answer

Redesign the pipeline to use a side input for the dimension table instead of a lookup.

Answer

Increase the number of Bigtable nodes to reduce lookup latency.

Question 17

The exhibit shows a Spark job submitted to Dataproc that fails with an out-of-memory error. Which change should be made to the submission command to resolve the issue?

Accepted Answer

Add --properties spark.executor.memory=8g to the command.. The out-of-memory error indicates that the Spark executors do not have enough memory to process the data. Adding `--properties spark.executor.memory=8g` increases the memory allocated to each executor, directly addressing the root cause. This property overrides the default executor memory (typically 1g or 4g depending on the cluster configuration) and is the standard way to tune executor memory in Spark on Dataproc.

Answer

Use a different Spark example class.

Answer

Increase the number of worker nodes in the cluster.

Answer

Add --driver-memory 8g to the command.

Question 18

A company is building a data lake on Cloud Storage with data from multiple sources. They need to apply schema-on-read and support ad-hoc SQL queries. Which architecture is most suitable?

Accepted Answer

Ingest to Cloud Storage, create BigQuery external tables.. BigQuery external tables allow schema-on-read by defining the schema at query time over data stored in Cloud Storage, enabling ad-hoc SQL queries without loading data into a separate system. This architecture directly supports the requirement for schema-on-read and SQL-based analysis, as BigQuery provides a serverless, scalable SQL engine.

Answer

Ingest to Cloud Spanner, query directly.

Answer

Ingest to Cloud SQL, then export to Cloud Storage for queries.

Answer

Ingest to Cloud Storage, load into Dataproc for queries.

Question 19

A company uses Vertex AI to serve a model that requires GPU for inference. They want to minimize cost while handling variable traffic. Which strategy should they use?

Accepted Answer

Use a Vertex AI Endpoint with GPU and configure auto-scaling to zero when idle. Option B is correct because Vertex AI Endpoints support GPU-accelerated inference with autoscaling, including the ability to scale down to zero replicas when there is no traffic. This minimizes cost by only incurring GPU charges during active inference, while still handling variable traffic through dynamic scaling.

Answer

Deploy the model to Cloud Functions with GPU

Answer

Use Vertex AI Batch Prediction with GPU

Answer

Use a Vertex AI Endpoint with GPU with a fixed number of replicas

Question 20

A data science team is operationalizing a batch prediction job using Vertex AI Batch Prediction. The model uses a custom container that requires a specific GPU for inference. The job processes a large dataset stored in Cloud Storage. The team wants to minimize cost while ensuring the job completes within a 2-hour window. Which configuration should they choose?

Accepted Answer

Use a custom training job with a GPU worker pool and run the inference as a custom job.. Option A is correct because Vertex AI Batch Prediction does not support custom containers with GPU accelerators; it only supports CPUs for batch prediction jobs. To run GPU-accelerated inference on a large dataset, the team must use a custom training job (which supports GPU worker pools) and run inference as a custom job. This approach allows them to leverage GPU hardware for the 2-hour window while minimizing cost by using preemptible VMs or choosing the smallest GPU instance that meets throughput requirements.

Answer

Use a custom machine type with a GPU accelerator in the batch prediction request.

Answer

Use a high-memory machine type (e.g., n1-highmem-32) without GPU to reduce cost.

Answer

Configure a Vertex AI endpoint with GPU and submit batch requests to the endpoint.