PDE Hard Questions — 25 Difficult Questions

Question 1

A manufacturing company wants to detect anomalies in sensor data from thousands of IoT devices in real time. The data is streaming into Pub/Sub. The best solution should use a machine learning model served from AI Platform that scores sensor readings aggregated over 5-minute windows. Which pipeline design meets these requirements?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use Cloud Dataflow with sliding windows to aggregate sensor readings every 5 minutes, then call a trained model hosted on AI Platform Prediction for each window. Cloud Dataflow's sliding windows natively handle the 5-minute aggregation requirement for streaming data, and its ability to call external services via a DoFn allows integration with AI Platform Prediction for real-time model scoring. This design aligns with the need for low-latency, scalable processing of Pub/Sub streams without managing infrastructure.

Answer

Use Cloud Dataproc with Spark Streaming to aggregate data, and use a Spark ML model embedded in the pipeline

Answer

Use BigQuery streaming inserts and run scheduled queries that call the ML model

Answer

Use Cloud Functions triggered by Pub/Sub to process each sensor reading individually

Question 2

A company runs a batch data processing workload using Dataproc clusters that are auto-scaled based on YARN memory utilization. During peak times, jobs take much longer than expected. Analysis shows the cluster is not scaling up despite high YARN memory utilization. What is the most likely cause?

Accepted Answer

The cluster is using primary workers only; auto-scaling only adds secondary workers. Dataproc clusters have two types of workers: primary workers (which run both HDFS and compute) and secondary workers (compute-only). The autoscaler can only add or remove secondary workers; it cannot scale primary workers. If the cluster uses only primary workers, the autoscaler has no secondary workers to add, so it cannot scale up even under high YARN memory utilization. This explains why the cluster remains static during peak times.

Answer

Spark dynamic allocation is disabled, preventing executors from using added workers

Answer

The cluster autoscaler is misconfigured to scale based on CPU, not memory

Answer

The autoscaler is set to scale down secondary workers, not up

Question 3

A model deployed on Vertex AI Endpoint is making predictions with high accuracy but the business team suspects bias against a certain demographic group. You need to analyze the model's predictions for fairness. What is the most effective approach?

Accepted Answer

Use Vertex AI Explainable AI to generate feature attributions for each prediction and analyze whether the demographic feature has disproportionate impact.. Vertex AI Explainable AI provides per-instance feature attributions, which allow you to examine how the model uses each feature—including sensitive demographic attributes—to arrive at a prediction. By analyzing these attributions across demographic groups, you can detect whether the model disproportionately relies on the demographic feature, indicating potential bias. This approach is more granular than aggregate metrics and directly addresses the business team's concern about bias in individual predictions.

Answer

Compute overall fairness metrics by comparing prediction rates across demographic groups.

Answer

Collect more data for the under-represented group and retrain the model.

Answer

Use Vertex AI Model Monitoring to check for training-serving skew on the demographic feature.

Question 4

A company runs large batch prediction jobs on Vertex AI every day. They want to minimize costs while ensuring the jobs complete within a 4-hour window. The model requires significant memory. What is the most cost-effective approach?

Accepted Answer

Use preemptible VMs with a machine type that meets memory requirements. Preemptible VMs (now called Spot VMs) are significantly cheaper than standard VMs (up to 60-80% discount) and are ideal for fault-tolerant batch prediction jobs that can handle interruptions. Since the job has a 4-hour window and the model requires significant memory, using preemptible VMs with a machine type that meets the memory requirements minimizes cost while allowing the job to complete if restarted within the time limit.

Answer

Use Cloud TPUs to accelerate predictions

Answer

Use a smaller machine type (e.g., n1-standard-4) to reduce cost

Answer

Use standard VMs and reduce parallelization

Question 5

A data engineer needs to design a Bigtable row key for a time-series IoT application where each device sends data every second. The query pattern is to retrieve all data for a specific device over a time range. Which row key design minimizes hotspots?

Accepted Answer

hash(device_id)#timestamp (e.g., a3f2#2024-03-15-10:30:00). To avoid hotspots (where all writes hit a single tablet server), the row key should start with a hash of the device ID to distribute writes across the cluster, then append the timestamp for range scans.

Answer

device_id#timestamp (e.g., device123#2024-03-15-10:30:00)

Answer

timestamp#device_id (e.g., 2024-03-15-10:30:00#device123)

Answer

device_type#device_id#timestamp

Question 6

A company is migrating their on-premises Hadoop/Spark workloads to Google Cloud. They need a fully managed service that supports existing Spark jobs with minimal code changes, allows autoscaling, and provides integration with Cloud Storage and BigQuery. The team also wants to avoid managing cluster infrastructure and pay only for what they use. Which TWO services meet these requirements? (Choose two.)

Accepted Answer

Dataproc Serverless (Spark). Dataproc Serverless allows running Spark jobs without managing clusters, with autoscaling and pay-per-use pricing. Dataproc on GKE enables running Spark on Kubernetes with autoscaling and is fully managed. Standard Dataproc requires cluster management and is not serverless. Dataflow is for Beam, not Spark. Cloud Composer is for orchestration, not data processing.

Answer

Standard Dataproc cluster with preemptible workers

Answer

Cloud Composer with Spark

Answer

Dataflow with Spark Runner

Question 7

Your company uses Vertex AI Pipelines to automate the ML lifecycle. The pipeline includes training, evaluation, and deployment steps. You want to ensure that if a pipeline run fails due to a transient error (e.g., resource quota shortage), it automatically retries before marking the run as failed. What is the best way to implement this?

Accepted Answer

In the pipeline component code, implement retry logic using exponential backoff for specific exceptions.. Vertex AI Pipelines does not have built-in retry logic for failed steps. You can wrap each step's logic to catch transient errors and retry, or use a retry mechanism in the container itself. Kubeflow Pipelines' retry policy can be specified. Modifying pipeline code is the most direct way.

Answer

Configure Vertex AI Pipelines to automatically restart failed runs.

Answer

Set a high timeout value for the pipeline so that transient errors resolve before timeout.

Answer

Use Cloud Tasks to schedule pipeline runs and retry upon failure.

Question 8

A company stores sensitive customer data in BigQuery and Cloud Storage. They want to encrypt the data with customer-managed encryption keys (CMEK) and ensure that access to the key material is restricted to only approved networks. Which additional Google Cloud control should they implement to enforce network-based access to the encryption keys?

Accepted Answer

VPC Service Controls. VPC Service Controls (VPC-SC) can create a security perimeter around Cloud KMS and BigQuery/Cloud Storage resources, preventing data exfiltration and restricting access to approved networks. VPC-SC works with CMEK to add an extra layer of network-based access control. Cloud Armor is for HTTP(S) load balancing, IAP is for user identity, and Private Google Access is for on-premises access to public IPs.

Answer

Identity-Aware Proxy (IAP)

Answer

Private Google Access

Answer

Cloud Armor

Question 9

A data pipeline processes sensitive customer data. You need to ensure that only authorised users can query the data in BigQuery, and that the data is encrypted at rest and in transit. Which THREE steps should you take? (Choose three.)

Accepted Answer

Create authorised views in BigQuery to restrict access to sensitive columns. Encryption at rest is enabled by default with CMEK or CSEK. In-transit encryption is default for BigQuery. Authorised views provide fine-grained access control. IAM roles control dataset access.

Answer

Use a Cloud VPN to encrypt data in transit between on-premises and Google Cloud

Answer

Grant the bigquery.dataViewer role at the dataset level to all users

Question 10

A data scientist developed a model using custom training on Vertex AI. They want to automate the entire training-to-deployment process. Which service should they use?

Accepted Answer

Vertex AI Pipelines. Vertex AI Pipelines is the correct choice because it provides a fully managed, serverless orchestration service specifically designed to automate ML workflows, including custom training, hyperparameter tuning, evaluation, and deployment. It integrates natively with Vertex AI services and supports Kubeflow Pipelines SDK or TFX for defining reproducible, end-to-end pipelines, making it the ideal solution for automating the entire training-to-deployment process.

Answer

Cloud Composer

Answer

Cloud Build

Answer

Cloud Functions

Question 11

A company uses BigQuery and wants to reduce query costs by using BI Engine for Looker Studio dashboards. The data is stored in a BigQuery dataset with 5 TB of frequently accessed tables. The dashboards run dozens of concurrent queries. What is the recommended approach to enable BI Engine acceleration?

Accepted Answer

Create a reservation in the Administration panel and assign it to the project.. BI Engine is a reserved capacity service that caches data in memory. You must reserve capacity (amount of memory) for a specific BigQuery region, and then BI Engine automatically accelerates queries from Looker Studio and other BI tools. It does not require you to grant specific IAM roles to users (they just need BigQuery permissions) or to create materialized views. You do not need to enable it per dataset; it works at the project/region level.

Answer

Enable BI Engine by setting the dataset option 'enable_bi_engine=TRUE' in the dataset metadata.

Answer

Grant all Looker Studio users the 'biengine.user' IAM role on the project.

Answer

Create materialized views of the tables and connect Looker Studio to the views.

Question 12

A company has a production machine learning model deployed on Vertex AI Endpoint that predicts customer churn. The model is retrained weekly using a Vertex AI Pipeline that pulls new data from BigQuery. Recently, the model's accuracy has been declining. The data science team suspects data drift but is unsure. They have enabled Vertex AI Model Monitoring but have not set up any alerts. The team wants to diagnose and address the issue quickly. The pipeline runs successfully, and no errors are reported. The model endpoint is serving predictions with average latency of 200ms. What should the team do first?

Accepted Answer

Review Vertex AI Model Monitoring drift reports and set up alerts for significant drift. The team has already enabled Vertex AI Model Monitoring, which automatically tracks feature distributions and prediction statistics over time. The first diagnostic step should be to review the drift reports generated by Model Monitoring to confirm whether data drift is occurring, and then set up alerts so the team is proactively notified of significant drift in the future. This directly addresses the suspected root cause without unnecessary operational changes.

Answer

Immediately trigger a retraining pipeline with more recent data

Answer

Increase the number of replicas to reduce latency

Answer

Examine Cloud Logging for prediction errors

Question 13

A financial services company deploys a fraud detection model on Vertex AI using a custom prediction container that runs a PyTorch model. The model requires GPU acceleration. The deployment succeeds but predictions return an error: 'CUDA error: out of memory'. What should the team do to resolve this issue?

Accepted Answer

Increase the GPU machine type to one with more memory (e.g., from NVIDIA T4 to A100). The CUDA out-of-memory error indicates that the GPU's VRAM is insufficient to load the PyTorch model or process the inference batch. Increasing the GPU machine type to one with more memory, such as from an NVIDIA T4 (16 GB) to an A100 (40 or 80 GB), directly resolves the capacity issue. Vertex AI prediction endpoints allow you to select different accelerator types and sizes, and this change ensures the model fits within GPU memory.

Answer

Change the container to use a CPU-only image to avoid CUDA errors

Answer

Enable Vertex AI Model Monitoring to automatically scale the endpoint

Answer

Add CPU replicas to distribute the inferencing load

Question 14

You are using BigQuery ML to train a matrix factorization model for a recommendation system. The training data consists of user-item interactions. You notice that the model is overfitting. Which of the following hyperparameter changes would most likely reduce overfitting?

Accepted Answer

Increase w_reg (regularization weight) from 0.1 to 0.5. Increasing the L2 regularization weight (w_reg) penalizes large weights and reduces overfitting. Increasing number of factors (num_factors) increases model complexity, worsening overfitting. Decreasing learning rate may help but not as directly as regularization.

Answer

Decrease w_reg (regularization weight) from 0.1 to 0.01

Answer

Increase num_factors from 10 to 20

Answer

Increase num_training_iterations from 10 to 20

Question 15

A financial services company runs a batch Dataflow pipeline daily to process transaction data. The pipeline reads from Cloud Storage, performs complex transformations, and writes to BigQuery. Recently, the pipeline has been failing intermittently with the error: 'Workflow failed. Causes: (9c3f7a2b1d4e): The worker missed 2000 data samples in the last 30 seconds. This can be caused by a variety of factors, including slow work items, network issues, or resource contention.' The team has already increased the number of workers and tried using e2-standard-8 machine types, but the issue persists. The pipeline processes approximately 500 GB of data per run and uses approximately 200 workers. The team suspects that the issue might be related to shuffle operations. What should the team do next to resolve the issue?

Accepted Answer

Increase the persistent disk size per worker to 100 GB.. The error indicates that workers are missing data samples due to slow shuffle operations, often caused by insufficient disk I/O. Increasing the persistent disk size per worker to 100 GB provides more local scratch space for Dataflow's shuffle, reducing disk contention and allowing the shuffle to complete within the 30-second window. This directly addresses the root cause without changing the worker count or machine type.

Answer

Enable Streaming Engine for the pipeline.

Answer

Reduce the number of workers to 100 to decrease shuffle overhead.

Answer

Use Cloud Storage as a shuffle sink.

Question 16

Refer to the exhibit. A team is trying to run a custom prediction container on Vertex AI Endpoint. They get this error when the container starts. What is the most likely cause?

Accepted Answer

The entry point is missing or incorrect. The error occurs when the container starts, which typically happens during the initial health check or readiness probe. Vertex AI Endpoints require a valid entry point (e.g., CMD or ENTRYPOINT in the Dockerfile) to start the prediction server. If the entry point is missing or incorrect, the container fails to launch, resulting in the observed error.

Answer

The container image is too large

Answer

The container is built for a different CPU architecture

Answer

The model file is missing from the container

Question 17

A data engineer is building a Cloud Workflows workflow that orchestrates multiple Cloud Functions and API calls. The workflow should handle transient failures with retries and send a notification to a Pub/Sub topic if the workflow ultimately fails. Which THREE steps should the engineer include in the workflow definition?

Accepted Answer

Use the 'googleapis.pubsub.v1.projects.topics.publish' connector to send a failure notification.. Workflows supports retry policies via 'retry' blocks, conditional steps using 'switch', and Pub/Sub publishing via the 'googleapis.pubsub.v1.projects.topics.publish' connector. The 'try/catch' is not a Workflows construct; instead, use 'step' with 'retry' and 'on_error' for failure handling. 'for' loops are for iteration, not error handling.

Answer

Use a 'for' loop to iterate over retries.

Answer

Use a 'try' / 'catch' block to handle exceptions and route to a failure step.

Question 18

You are a data engineer at a financial services company that uses Vertex AI to train and deploy models for credit risk assessment. The company has strict governance requirements: every model version must be approved by the risk committee before going to production. The approval process can take several days. Currently, the team trains a new model weekly and manually deploys it to a staging endpoint for review, then manually promotes to production after approval. This process is error-prone and slow. You want to automate the pipeline: training should trigger automatically when new data arrives, the model should be automatically deployed to a staging endpoint for review, and after manual approval, it should be promoted to production. Additionally, you need to ensure that if a model in staging performs poorly (e.g., low accuracy), it should not be promoted even if approved. What should you do?

Accepted Answer

Create a Vertex AI Pipeline that: (1) Triggers on new data, (2) Trains model, (3) Evaluates and stores metrics in the model registry, (4) Deploys to staging endpoint as a new model version. Then use a manual approval step (e.g., via Cloud Build approval or external system) to trigger a second pipeline that checks the stored metrics and, if acceptable, deploys to production endpoint.. The best approach uses Vertex AI Pipelines to automatically train and deploy to a staging endpoint. After manual approval, a separate pipeline step checks model performance metrics (which were stored during training/evaluation) and if they meet a threshold, promotes to production. This enforces governance and automation.

Answer

Use Vertex AI Experiments to track model versions, then manually deploy from the Experiments UI.

Answer

Use Cloud Scheduler to run training weekly, then use Cloud Functions to deploy to staging, and after manual approval, use another Cloud Function to check performance and deploy to production.

Answer

Train models on Vertex AI Workbench and use a CI/CD tool like Cloud Build to deploy to staging. Use a Cloud Build approval step to promote to production after manual check.

Question 19

You are designing a data pipeline that ingests streaming data from Pub/Sub, processes it with Dataflow, and writes to BigQuery. You need to ensure that schema changes in the incoming data (new fields) are handled without pipeline failure. Which THREE steps should you take? (Choose THREE.)

Accepted Answer

Set the BigQuery table schema to allow automatic addition of new fields by using 'ignore_unknown_values'.. To handle schema drift, BigQuery allows schema relaxation (adding nullable columns) via DDL. Dataflow can update the destination table schema using BigQuery APIs (e.g., set the schema to include unknown fields). Alternatively, using BigQuery's automatic schema detection with 'ignore_unknown_values' can allow new fields to be added automatically. Setting all fields to NULLABLE in advance is not practical. Setting the table to require all fields causes failures on unknown fields.

Answer

Configure the BigQuery table to require all fields, causing the pipeline to fail on unknown fields.

Answer

Set all columns in the BigQuery table as NULLABLE from the start.

Question 20

A company uses BigQuery flat-rate pricing with 500 slots purchased as a committed use discount. During peak hours, they need additional capacity but do not want to buy more committed slots. They have a secondary project used for ad-hoc queries by analysts. How can they provide burst capacity to the primary project during peak times without increasing committed spend?

Accepted Answer

Create flex slots in the secondary project, create a reservation in the secondary project, and assign the reservation to the primary project.. Flex slots provide temporary capacity without long-term commitment. By creating flex slots in a secondary project and assigning the reservation to the primary project, burst capacity is added during peak times without increasing committed spend. Option B is incorrect because autoscaling can be used with committed use reservations, but it does not provide capacity beyond the purchased slots without incurring additional costs; autoscaling on committed use still uses flex pricing. Option C is incorrect because upgrading to Enterprise Plus does not inherently provide burst capacity without additional slots. Option D is incorrect because purchasing additional committed use slots increases committed spend.

Answer

Enable autoscaling slot management in the primary project's reservation, allowing slots to scale up based on demand.

Answer

Upgrade the primary project's edition to Enterprise Plus to allow bursting.

Answer

Purchase additional committed use slots in the primary project and apply them to the reservation.