PDE Designing data processing systems — Timed 10 Questions

Question 1

A company is migrating on-premises Apache Spark jobs to Google Cloud Dataproc. They want to reduce operational overhead and minimize costs. Which architecture is most appropriate?

Accepted Answer

Use Dataproc clusters with auto-scaling and preemptible VMs.. Option D is correct because Dataproc clusters with auto-scaling and preemptible VMs directly address the need to reduce operational overhead and minimize costs for on-premises Spark migrations. Auto-scaling dynamically adjusts cluster size based on workload, while preemptible VMs (which cost 60-80% less than standard VMs) handle fault-tolerant tasks, making this the most cost-effective and operationally efficient architecture for Spark on Dataproc.

Answer

Use Cloud Dataproc Serverless for all Spark jobs.

Answer

Migrate jobs to Cloud Dataflow.

Answer

Run Spark on Compute Engine instances with startup scripts.

Question 2

A data pipeline ingests sensor data from IoT devices via Cloud Pub/Sub, processes it with Cloud Dataflow, and writes to BigQuery. The pipeline is failing with high latency and data loss. Which troubleshooting step should be taken first?

Accepted Answer

Check Stackdriver logging for error messages.. Option A is correct because Stackdriver (now Cloud Logging) is the first place to investigate when a Dataflow pipeline experiences high latency and data loss. Dataflow automatically logs errors, worker failures, and system messages to Cloud Logging, which can reveal root causes such as insufficient resources, stuck steps, or Pub/Sub subscription issues. Checking logs first avoids premature scaling or configuration changes that may not address the actual problem.

Answer

Disable exactly-once processing in Dataflow.

Answer

Increase the number of Dataflow workers.

Answer

Switch to BigQuery streaming inserts.

Question 3

A company needs to process real-time clickstream data and store it in a data warehouse for SQL-based analytics. The data volume is moderate. Which combination of Google Cloud services is most cost-effective?

Accepted Answer

Cloud Pub/Sub, Cloud Dataflow, BigQuery. Option C is correct because Cloud Pub/Sub ingests real-time clickstream data, Cloud Dataflow processes it with low latency, and BigQuery provides a serverless, SQL-based data warehouse that is cost-effective for moderate data volumes due to its pay-per-query pricing and automatic scaling. This combination avoids the overhead of managing clusters (Dataproc) or expensive storage (Cloud Spanner) while directly supporting SQL analytics.

Answer

Cloud Pub/Sub, Cloud Dataproc, Cloud Storage

Answer

Cloud Pub/Sub, Cloud Dataflow, Cloud Spanner

Answer

Cloud Pub/Sub, Cloud Dataflow, Cloud Storage

Question 4

A financial company processes transactions in real-time and requires exactly-once processing semantics. They also need to reprocess historical data for backtesting. Which Google Cloud service should they use?

Accepted Answer

Cloud Dataflow. Cloud Dataflow (D) is correct because it provides exactly-once processing semantics via its distributed snapshot mechanism (based on the MillWheel paper) and supports both real-time streaming and batch processing for historical backtesting under a unified programming model. This allows the company to reprocess historical data using the same pipeline code, ensuring consistency across real-time and batch modes.

Answer

Cloud Pub/Sub

Answer

Cloud Functions

Answer

Cloud Dataproc

Question 5

A company is building a data lake on Cloud Storage with data from multiple sources. They need to apply schema-on-read and support ad-hoc SQL queries. Which architecture is most suitable?

Accepted Answer

Ingest to Cloud Storage, create BigQuery external tables.. BigQuery external tables allow schema-on-read by defining the schema at query time over data stored in Cloud Storage, enabling ad-hoc SQL queries without loading data into a separate system. This architecture directly supports the requirement for schema-on-read and SQL-based analysis, as BigQuery provides a serverless, scalable SQL engine.

Answer

Ingest to Cloud Spanner, query directly.

Answer

Ingest to Cloud SQL, then export to Cloud Storage for queries.

Answer

Ingest to Cloud Storage, load into Dataproc for queries.

Question 6

A company wants to stream data from Cloud Pub/Sub into BigQuery with minimal latency. They have a small team and limited operational resources. Which approach is best?

Accepted Answer

Use a Dataflow pipeline with a BigQuery subscription.. Option D is correct because a Dataflow pipeline with a BigQuery subscription provides a fully managed, serverless streaming solution that directly ingests messages from Pub/Sub and writes them to BigQuery with minimal latency. Dataflow handles autoscaling, checkpointing, and exactly-once semantics, which aligns with the team's limited operational resources. The BigQuery subscription (via the Pub/Sub to BigQuery template) eliminates the need for custom code or cluster management, ensuring low-latency streaming without operational overhead.

Answer

Write a custom application on Compute Engine that polls Pub/Sub and writes to BigQuery.

Answer

Create a Dataproc cluster running a Spark Streaming job.

Answer

Create a Cloud Function that writes to BigQuery.

Question 7

A company has a batch ETL job that runs daily using Cloud Dataflow. The job reads from Cloud Storage, transforms data, and writes to BigQuery. Recently, the job started failing with 'Resources have been exhausted' errors. What is the most likely cause?

Accepted Answer

The project has reached its Dataflow API quota.. The 'Resources have been exhausted' error in Cloud Dataflow typically indicates that the project has reached its Dataflow API quota, such as the maximum number of concurrent jobs or API requests per minute. This is a common issue when multiple jobs run simultaneously or when the quota is set low by default. The error is distinct from resource exhaustion in the underlying compute or storage layers.

Answer

The Cloud Storage bucket has been deleted.

Answer

The input data volume has increased significantly.

Answer

The BigQuery output table schema has changed.

Question 8

A company needs to process sensitive healthcare data with strict compliance requirements. They want to use Cloud Dataflow but must ensure data is encrypted end-to-end and audit logs are retained. Which combination of features should they enable?

Accepted Answer

Use Customer-Managed Encryption Keys (CMEK) and VPC Service Controls.. Option A is correct because Customer-Managed Encryption Keys (CMEK) allow the company to control the encryption keys used to protect data at rest in Cloud Dataflow, while VPC Service Controls provide a security perimeter that prevents data exfiltration and ensures end-to-end encryption boundaries. Together, they address the compliance requirement for encryption control and audit logging by restricting data movement within a VPC service perimeter and using customer-managed keys for data encryption.

Answer

Use Data Loss Prevention API to redact sensitive data.

Answer

Enable Cloud Audit Logs and VPC Service Controls.

Answer

Enable default encryption at rest and in transit.

Question 9

A company is running a Cloud Dataflow streaming pipeline that aggregates events in 1-minute windows. They notice that the watermark is lagging significantly behind real-time. What is the most likely cause?

Accepted Answer

A hot key is causing data skew.. A hot key causes data skew, which means a disproportionate amount of data is assigned to a single key. In Cloud Dataflow, this leads to a single worker processing the bulk of the events, creating a processing bottleneck. The watermark, which tracks the progress of event-time processing, cannot advance until all data for a given window is processed, so the skewed key delays watermark progression significantly behind real-time.

Answer

The window duration is too short.

Answer

The pipeline was recently updated.

Answer

The allowed lateness is set too high.

Question 10

A data engineer is designing a batch processing system using Cloud Dataproc. Which TWO practices improve performance and reduce costs? (Choose TWO.)

Accepted Answer

Set autoscaling policies based on YARN memory.. Option B is correct because autoscaling policies based on YARN memory allow the cluster to dynamically add or remove worker nodes in response to actual resource demand from running jobs. This prevents over-provisioning (reducing costs) and ensures sufficient resources for job completion (improving performance), as Cloud Dataproc directly monitors YARN memory metrics to trigger scaling actions.

Answer

Always use persistent disks for all nodes.

Answer

Store intermediate data in HDFS.

Answer

Use the largest machine types for master nodes.