PDE Ensuring solution quality — Timed 15 Questions

Question 1

A data pipeline ingests streaming data from Pub/Sub into BigQuery via Dataflow. Recently, the pipeline has been failing with 'deadline exceeded' errors. What is the most likely cause?

Accepted Answer

The Pub/Sub subscription's acknowledgement deadline is too short for the processing time.. Option D is correct because 'deadline exceeded' errors in a Dataflow pipeline reading from Pub/Sub indicate that the subscriber is taking longer to process messages than the acknowledgement deadline allows. When the deadline expires, Pub/Sub redelivers the message, causing duplicate processing and eventual pipeline failure. This is a common issue when processing time exceeds the default 10-second acknowledgement deadline.

Answer

The BigQuery streaming quota is exceeded.

Answer

Dataflow workers are underutilized due to batch size settings.

Answer

Dataflow autoscaling is disabled.

Question 2

A team is designing a data lake on Google Cloud using Cloud Storage and BigQuery. They need to ensure that sensitive data (e.g., PII) is encrypted at rest and have the ability to audit access. Which approach meets these requirements?

Accepted Answer

Use Customer-Managed Encryption Keys (CMEK) and enable Cloud Audit Logs.. Option B is correct because Customer-Managed Encryption Keys (CMEK) allow the team to control and manage the encryption keys used to protect data at rest in Cloud Storage and BigQuery, while enabling Cloud Audit Logs provides the necessary audit trail for access to both the data and the keys. This combination directly satisfies the requirements for encryption at rest and auditability.

Answer

Use Customer-Managed Encryption Keys (CMEK) and enable VPC Service Controls.

Answer

Use Default Encryption and enable Data Loss Prevention (DLP) API.

Answer

Use Customer-Supplied Encryption Keys (CSEK) and enable VPC Service Controls.

Question 3

A company runs a batch processing job on Dataproc that uses Apache Spark to process 500 GB of data daily. The job completes successfully but takes 4 hours. The team wants to reduce the runtime to under 2 hours without increasing cost. What should they do?

Accepted Answer

Use preemptible VMs for worker nodes and increase the number of workers.. Preemptible VMs cost significantly less than standard VMs (about 60-80% discount). By using preemptible VMs for worker nodes, you can increase the number of workers (and thus parallelism) without increasing cost. This directly reduces runtime by distributing the 500 GB workload across more executors, while the cost savings from preemptible VMs offset the additional nodes.

Answer

Increase the master node's machine type to n2-standard-8.

Answer

Increase the machine type of worker nodes to n2-highmem-8.

Answer

Migrate the job to Dataflow with autoscaling enabled.

Question 4

Which TWO actions are recommended to improve the reliability of a Cloud Dataflow streaming pipeline that processes event data from Pub/Sub?

Accepted Answer

Enable Dataflow Streaming Engine.. Option B is correct because enabling Dataflow Streaming Engine moves state and computation from worker VMs to the backend service, reducing the impact of worker scaling and preemption. This improves reliability by providing consistent performance and fault tolerance for streaming pipelines, especially those with high throughput or stateful processing.

Answer

Use a pull subscription with a 10-second acknowledgment deadline.

Answer

Disable autoscaling to prevent worker churn.

Answer

Use micro-batch processing with a small batch size.

Question 5

A data analyst runs a complex SQL query in BigQuery that joins multiple large tables and receives the above error. Which action is most likely to resolve the issue?

Accepted Answer

Increase the number of slots allocated to the project.. The error indicates that the query exceeded the available slot resources in the BigQuery project. Increasing the number of slots allocated to the project (option D) directly addresses this by providing more compute capacity for parallel query execution, which is the correct action to resolve resource exhaustion in BigQuery's serverless architecture.

Answer

Use a larger number of workers in the query execution.

Answer

Use smaller tables by sampling data.

Answer

Add clustering on join columns.

Question 6

A company runs a real-time anomaly detection system on Google Cloud. Streaming data from IoT devices is ingested via Pub/Sub, processed by Dataflow (Apache Beam), and results are written to Bigtable for low-latency serving. Recently, the system has been experiencing increased latency and occasional data loss. The Dataflow pipeline shows high system lag and backlog in Pub/Sub. The Bigtable cluster has 3 nodes and is reporting high CPU utilization (over 90%). The team suspects the issue is with the pipeline configuration. They have already verified that there are no errors in the pipeline code and no network issues. Which action should they take to resolve the issue?

Accepted Answer

Increase the number of Bigtable nodes to handle the write throughput.. The high CPU utilization on Bigtable (over 90%) indicates that the cluster is saturated and cannot keep up with the write throughput from Dataflow. This causes backpressure in the pipeline, leading to increased system lag and backlog in Pub/Sub, and eventually data loss when Pub/Sub messages expire. Increasing the number of Bigtable nodes directly addresses the bottleneck by distributing the write load and reducing CPU pressure, which allows the pipeline to drain the backlog and reduce latency.

Answer

Change the Dataflow worker machine type to n2-standard-8.

Answer

Decrease the batch size in the Dataflow pipeline to reduce latency.

Answer

Increase the number of Dataflow workers to process messages faster.

Question 7

Drag and drop the steps to create a Cloud Composer environment for Apache Airflow into the correct order.

Question 8

Drag and drop the steps to migrate an on-premises MySQL database to Cloud SQL using Database Migration Service into the correct order.

Question 9

Match each machine learning term to its description.

Question 10

Match each data encryption concept to its description.

Question 11

A data pipeline processes streaming data with Dataflow. The team notices occasional data duplication in BigQuery. What is the best approach to ensure exactly-once processing?

Accepted Answer

Configure Dataflow with exactly-once sinks using file staging and deduplication.. Option B is correct because Dataflow's exactly-once sink mechanism, which uses file staging and deduplication, ensures no duplicates. Option A (at-least-once delivery) can cause duplicates unless dedup is applied, but that's not automatic. Option C adds unnecessary complexity. Option D is incorrect because BigQuery does not natively support idempotent writes.

Answer

Use Pub/Sub with at-least-once delivery and deduplicate in BigQuery using a unique identifier.

Answer

Use Cloud Functions to deduplicate messages before they enter the pipeline.

Answer

Enable idempotent writes in BigQuery.

Question 12

A company is deploying a large-scale streaming application on Google Kubernetes Engine. They need to ensure the application can handle sudden traffic spikes without dropping data. Which architectural pattern is most appropriate?

Accepted Answer

Use a Pub/Sub topic as a buffer and autoscale consumer pods based on Pub/Sub subscription backlog.. Option A is correct because using a Pub/Sub buffer decouples producers from consumers, allowing autoscaling of consumers to handle spikes. Option B is wasteful and not dynamically scalable. Option C uses Cloud SQL, which is not designed for high-throughput buffering. Option D only addresses retries, not overall throughput capacity.

Answer

Implement custom retry logic with exponential backoff in the application.

Answer

Use Cloud SQL as a temporary buffer and process from there.

Answer

Pre-provision 3x the expected peak capacity to handle spikes.

Question 13

A data science team uses AI Platform Training with hyperparameter tuning. They observe that some trials fail due to transient errors. To improve solution quality and reduce costs, what should they do?

Accepted Answer

Set the maxFailedTrials parameter to a high value (e.g., 10).. Option D is correct because setting maxFailedTrials to a high value allows more trials to complete despite transient failures, improving the chance of finding a good model without wasting resources on re-running failed trials. Option A increases parallelism but still pays for failed trials. Option B (early stopping) prunes unpromising trials, but does not address transient errors. Option C increases cost per trial without solving the failure issue.

Answer

Enable early stopping using a Bayesian optimization algorithm.

Answer

Use larger machine types for each trial.

Answer

Increase the number of parallel trials.

Question 14

A company runs batch jobs on Dataproc. They need to ensure that if a job fails, it automatically retries with exponential backoff. What is the recommended approach?

Accepted Answer

Use Dataproc Workflow Templates with the maxAttempts parameter set to 3.. Option C is correct because Dataproc Workflow Templates support configuring maxAttempts and retry policy in the template, enabling automatic retries with exponential backoff. Option A (Composer) is overkill for simple retry. Option B (cron job) would need custom logic. Option D (Cloud Functions) also requires custom implementation.

Answer

Schedule a cron job to check job status and restart manually.

Answer

Use a Cloud Function triggered by Stackdriver alerts to restart the job.

Answer

Create a Cloud Composer DAG that monitors job status and retries on failure.

Question 15

A team developed a microservice that writes logs to stdout. They want to centralize logs for analysis. Which GCP service should they use to automatically collect and store logs?

Accepted Answer

Use the Cloud Logging client library (google-cloud-logging) for the microservice's language.. Option D is correct because Cloud Logging with the client library automatically captures stdout logs and sends them to Cloud Logging. Option A (Cloud Logging agent) is for VMs, not containers. Option B (Cloud Storage) is for object storage. Option C (Pub/Sub) is for messaging, not log collection.

Answer

Install the Cloud Logging agent on the VM running the microservice.

Answer

Publish logs to a Pub/Sub topic and later store them.

Answer

Write logs directly to Cloud Storage.