PDE Designing data processing systems — 20 Questions

Question 1

A company is migrating on-premises Apache Spark jobs to Google Cloud Dataproc. They want to reduce operational overhead and minimize costs. Which architecture is most appropriate?

Accepted Answer

Use Dataproc clusters with auto-scaling and preemptible VMs.. Option D is correct because Dataproc clusters with auto-scaling and preemptible VMs directly address the need to reduce operational overhead and minimize costs for on-premises Spark migrations. Auto-scaling dynamically adjusts cluster size based on workload, while preemptible VMs (which cost 60-80% less than standard VMs) handle fault-tolerant tasks, making this the most cost-effective and operationally efficient architecture for Spark on Dataproc.

Answer

Use Cloud Dataproc Serverless for all Spark jobs.

Answer

Migrate jobs to Cloud Dataflow.

Answer

Run Spark on Compute Engine instances with startup scripts.

Question 2

A data pipeline ingests sensor data from IoT devices via Cloud Pub/Sub, processes it with Cloud Dataflow, and writes to BigQuery. The pipeline is failing with high latency and data loss. Which troubleshooting step should be taken first?

Accepted Answer

Check Stackdriver logging for error messages.. Option A is correct because Stackdriver (now Cloud Logging) is the first place to investigate when a Dataflow pipeline experiences high latency and data loss. Dataflow automatically logs errors, worker failures, and system messages to Cloud Logging, which can reveal root causes such as insufficient resources, stuck steps, or Pub/Sub subscription issues. Checking logs first avoids premature scaling or configuration changes that may not address the actual problem.

Answer

Disable exactly-once processing in Dataflow.

Answer

Increase the number of Dataflow workers.

Answer

Switch to BigQuery streaming inserts.

Question 3

A company needs to process real-time clickstream data and store it in a data warehouse for SQL-based analytics. The data volume is moderate. Which combination of Google Cloud services is most cost-effective?

Accepted Answer

Cloud Pub/Sub, Cloud Dataflow, BigQuery. Option C is correct because Cloud Pub/Sub ingests real-time clickstream data, Cloud Dataflow processes it with low latency, and BigQuery provides a serverless, SQL-based data warehouse that is cost-effective for moderate data volumes due to its pay-per-query pricing and automatic scaling. This combination avoids the overhead of managing clusters (Dataproc) or expensive storage (Cloud Spanner) while directly supporting SQL analytics.

Answer

Cloud Pub/Sub, Cloud Dataproc, Cloud Storage

Answer

Cloud Pub/Sub, Cloud Dataflow, Cloud Spanner

Answer

Cloud Pub/Sub, Cloud Dataflow, Cloud Storage

Question 4

A financial company processes transactions in real-time and requires exactly-once processing semantics. They also need to reprocess historical data for backtesting. Which Google Cloud service should they use?

Accepted Answer

Cloud Dataflow. Cloud Dataflow (D) is correct because it provides exactly-once processing semantics via its distributed snapshot mechanism (based on the MillWheel paper) and supports both real-time streaming and batch processing for historical backtesting under a unified programming model. This allows the company to reprocess historical data using the same pipeline code, ensuring consistency across real-time and batch modes.

Answer

Cloud Pub/Sub

Answer

Cloud Functions

Answer

Cloud Dataproc

Question 5

A company is building a data lake on Cloud Storage with data from multiple sources. They need to apply schema-on-read and support ad-hoc SQL queries. Which architecture is most suitable?

Accepted Answer

Ingest to Cloud Storage, create BigQuery external tables.. BigQuery external tables allow schema-on-read by defining the schema at query time over data stored in Cloud Storage, enabling ad-hoc SQL queries without loading data into a separate system. This architecture directly supports the requirement for schema-on-read and SQL-based analysis, as BigQuery provides a serverless, scalable SQL engine.

Answer

Ingest to Cloud Spanner, query directly.

Answer

Ingest to Cloud SQL, then export to Cloud Storage for queries.

Answer

Ingest to Cloud Storage, load into Dataproc for queries.

Question 6

A company wants to stream data from Cloud Pub/Sub into BigQuery with minimal latency. They have a small team and limited operational resources. Which approach is best?

Accepted Answer

Use a Dataflow pipeline with a BigQuery subscription.. Option D is correct because a Dataflow pipeline with a BigQuery subscription provides a fully managed, serverless streaming solution that directly ingests messages from Pub/Sub and writes them to BigQuery with minimal latency. Dataflow handles autoscaling, checkpointing, and exactly-once semantics, which aligns with the team's limited operational resources. The BigQuery subscription (via the Pub/Sub to BigQuery template) eliminates the need for custom code or cluster management, ensuring low-latency streaming without operational overhead.

Answer

Write a custom application on Compute Engine that polls Pub/Sub and writes to BigQuery.

Answer

Create a Dataproc cluster running a Spark Streaming job.

Answer

Create a Cloud Function that writes to BigQuery.

Question 7

A company has a batch ETL job that runs daily using Cloud Dataflow. The job reads from Cloud Storage, transforms data, and writes to BigQuery. Recently, the job started failing with 'Resources have been exhausted' errors. What is the most likely cause?

Accepted Answer

The project has reached its Dataflow API quota.. The 'Resources have been exhausted' error in Cloud Dataflow typically indicates that the project has reached its Dataflow API quota, such as the maximum number of concurrent jobs or API requests per minute. This is a common issue when multiple jobs run simultaneously or when the quota is set low by default. The error is distinct from resource exhaustion in the underlying compute or storage layers.

Answer

The Cloud Storage bucket has been deleted.

Answer

The input data volume has increased significantly.

Answer

The BigQuery output table schema has changed.

Question 8

A company needs to process sensitive healthcare data with strict compliance requirements. They want to use Cloud Dataflow but must ensure data is encrypted end-to-end and audit logs are retained. Which combination of features should they enable?

Accepted Answer

Use Customer-Managed Encryption Keys (CMEK) and VPC Service Controls.. Option A is correct because Customer-Managed Encryption Keys (CMEK) allow the company to control the encryption keys used to protect data at rest in Cloud Dataflow, while VPC Service Controls provide a security perimeter that prevents data exfiltration and ensures end-to-end encryption boundaries. Together, they address the compliance requirement for encryption control and audit logging by restricting data movement within a VPC service perimeter and using customer-managed keys for data encryption.

Answer

Use Data Loss Prevention API to redact sensitive data.

Answer

Enable Cloud Audit Logs and VPC Service Controls.

Answer

Enable default encryption at rest and in transit.

Question 9

A company is running a Cloud Dataflow streaming pipeline that aggregates events in 1-minute windows. They notice that the watermark is lagging significantly behind real-time. What is the most likely cause?

Accepted Answer

A hot key is causing data skew.. A hot key causes data skew, which means a disproportionate amount of data is assigned to a single key. In Cloud Dataflow, this leads to a single worker processing the bulk of the events, creating a processing bottleneck. The watermark, which tracks the progress of event-time processing, cannot advance until all data for a given window is processed, so the skewed key delays watermark progression significantly behind real-time.

Answer

The window duration is too short.

Answer

The pipeline was recently updated.

Answer

The allowed lateness is set too high.

Question 10

A data engineer is designing a batch processing system using Cloud Dataproc. Which TWO practices improve performance and reduce costs? (Choose TWO.)

Accepted Answer

Set autoscaling policies based on YARN memory.. Option B is correct because autoscaling policies based on YARN memory allow the cluster to dynamically add or remove worker nodes in response to actual resource demand from running jobs. This prevents over-provisioning (reducing costs) and ensures sufficient resources for job completion (improving performance), as Cloud Dataproc directly monitors YARN memory metrics to trigger scaling actions.

Answer

Always use persistent disks for all nodes.

Answer

Store intermediate data in HDFS.

Answer

Use the largest machine types for master nodes.

Question 11

A company is migrating an on-premises Hadoop cluster to Google Cloud. They need to run existing Spark jobs with minimal modification. Which THREE strategies should they consider? (Choose THREE.)

Accepted Answer

Use Cloud Dataproc with Spark and Hive components.. Option B is correct because Cloud Dataproc is a managed Spark and Hadoop service that supports the same Spark and Hive components used on-premises, allowing existing Spark jobs to run with minimal modification. It provides native integration with Cloud Storage, which can replace HDFS without changing job logic, and the Dataproc Jobs API enables programmatic job submission, preserving existing workflows.

Answer

Migrate to BigQuery for all analytics.

Answer

Rewrite Spark jobs as Dataflow pipelines.

Question 12

A data pipeline uses Cloud Pub/Sub to ingest events, then a Cloud Dataflow job writes to BigQuery. The Dataflow job is failing with 'deadline exceeded' errors. Which TWO actions can resolve this? (Choose TWO.)

Accepted Answer

Increase the number of Dataflow workers.. Increasing the number of Dataflow workers (Option A) is correct because 'deadline exceeded' errors typically indicate that the pipeline is falling behind on processing due to insufficient parallelism. By adding more workers, the workload is distributed across more virtual machines, reducing the per-worker load and allowing the pipeline to keep up with the incoming Pub/Sub stream, thereby avoiding timeouts when writing to BigQuery.

Answer

Switch to BigQuery Storage Write API.

Answer

Decrease the batch size for writes to BigQuery.

Answer

Change the windowing from fixed to global.

Question 13

The exhibit shows a Spark job submitted to Dataproc that fails with an out-of-memory error. Which change should be made to the submission command to resolve the issue?

Accepted Answer

Add --properties spark.executor.memory=8g to the command.. The out-of-memory error indicates that the Spark executors do not have enough memory to process the data. Adding `--properties spark.executor.memory=8g` increases the memory allocated to each executor, directly addressing the root cause. This property overrides the default executor memory (typically 1g or 4g depending on the cluster configuration) and is the standard way to tune executor memory in Spark on Dataproc.

Answer

Use a different Spark example class.

Answer

Increase the number of worker nodes in the cluster.

Answer

Add --driver-memory 8g to the command.

Question 14

The exhibit shows a Cloud Logging query result. A data engineer sees this log for a streaming Dataflow job. What is the most likely cause?

Accepted Answer

The job has reached the maximum number of workers allowed by the project quota.. The log shows that the Dataflow job is not scaling up despite pending work. This typically occurs when the job has reached the maximum number of workers allowed by the project quota. Dataflow uses the Compute Engine default worker quota, and if the job attempts to exceed that limit, it will stop scaling and log messages indicating that it cannot add more workers.

Answer

The job is experiencing network latency.

Answer

The job is using too much memory per worker.

Answer

The job has insufficient permissions to scale.

Question 15

The exhibit shows an IAM policy for a BigQuery dataset. A Dataflow job is failing with 'Access Denied: Table ... User does not have bigquery.tables.get permission'. Which additional role should be granted to the service account?

Accepted Answer

roles/bigquery.dataEditor. The error indicates the service account lacks the `bigquery.tables.get` permission, which is required to read table metadata. `roles/bigquery.dataEditor` includes this permission along with `bigquery.tables.get`, `bigquery.tables.update`, and `bigquery.tables.export`, making it the minimal role that resolves the access denied error for a Dataflow job reading from a BigQuery table.

Answer

roles/bigquery.admin

Answer

roles/bigquery.user

Answer

roles/bigquery.jobUser

Question 16

A company runs a Cloud Dataflow streaming pipeline that reads from Cloud Pub/Sub, performs a fixed window of 10 seconds, joins with a slowly-changing dimension table stored in Cloud Bigtable, and writes results to BigQuery. The pipeline has been running for months but recently started exhibiting increasing latency and occasional data loss. The pipeline uses default settings with autoscaling enabled (min 2, max 20 workers). The Bigtable cluster has 3 nodes. The dimensions are updated infrequently. The latency has grown from seconds to minutes. Examining the Dataflow monitoring UI, you see that the 'System Lag' metric is increasing, and some windows are not being emitted. The CPU utilization on Bigtable nodes is below 50%. There are no errors in the logs. Which action is most likely to resolve the issue?

Accepted Answer

Set the pipeline option --maxNumWorkers to a value between 5 and 10.. The increasing system lag and unemitted windows in a streaming pipeline with autoscaling (2–20 workers) and a 3-node Bigtable cluster indicate that the pipeline is bottlenecked by the number of workers, not by Bigtable performance. With default autoscaling, Dataflow may not scale up aggressively enough to handle the sustained load, causing backlog and window expiration. Capping maxNumWorkers to 5–10 ensures sufficient parallelism without over-provisioning, allowing the pipeline to catch up and emit windows reliably.

Answer

Increase the window duration to 30 seconds to reduce the number of windows.

Answer

Redesign the pipeline to use a side input for the dimension table instead of a lookup.

Answer

Increase the number of Bigtable nodes to reduce lookup latency.

Question 17

A company uses Cloud Dataproc to run nightly Spark ETL jobs that process about 500 GB of data each night. The jobs currently take 4 hours to complete. The company wants to reduce the runtime to under 2 hours to meet a new SLA. The cluster is configured with 10 worker nodes (n1-standard-4) and 1 master node (n1-standard-4). The jobs are CPU-bound and use only default settings. The cluster is deleted after each job and recreated. The data is stored in Cloud Storage. The company is open to increasing cost but wants the most cost-effective solution to meet the SLA. Which approach should they take?

Accepted Answer

Increase the number of worker nodes to 20 and use preemptible VMs for half of them.. Option C is correct because adding more worker nodes (from 10 to 20) directly increases parallelism for CPU-bound Spark jobs, and using preemptible VMs for half of them reduces cost while still meeting the SLA. Since the job is CPU-bound and uses default settings, scaling horizontally with a mix of standard and preemptible VMs is the most cost-effective way to halve runtime, as Spark can efficiently distribute the workload across more cores.

Answer

Use a regional Cloud Storage bucket to improve read throughput.

Answer

Replace worker nodes with n1-highmem-16 instances to increase memory.

Answer

Change machine type to n2-standard-8 for all nodes.

Question 18

A company runs a batch ETL pipeline on Cloud Dataproc. During peak hours, the job takes longer than expected. The pipeline reads from Cloud Storage, transforms data, and writes to BigQuery. What is the most cost-effective way to improve performance without redesigning the pipeline?

Accepted Answer

Add a secondary worker group using preemptible VMs and increase the number of workers.. Adding a secondary worker group with preemptible VMs is the most cost-effective way to improve performance because it allows you to scale out the cluster horizontally with compute instances that are significantly cheaper (up to 80% discount) than regular VMs. This directly addresses the bottleneck of processing capacity during peak hours without requiring any pipeline redesign, as Cloud Dataproc can automatically distribute work across additional workers.

Answer

Enable local SSDs on all worker nodes.

Answer

Increase the master node's machine type to n1-highmem-32.

Answer

Use Cloud Composer to schedule the job with a higher priority.

Question 19

A retail company processes real-time clickstream data using Cloud Pub/Sub and Dataflow. The pipeline aggregates events by user session and writes to Bigtable for low-latency queries. However, users report that session data is sometimes missing or duplicated. What is the most likely cause?

Accepted Answer

Pub/Sub provides at-least-once delivery, and Dataflow does not deduplicate by default.. D is correct because Pub/Sub offers at-least-once delivery, meaning the same message may be delivered multiple times. Dataflow does not automatically deduplicate messages unless explicitly configured (e.g., using idempotent sinks or custom deduplication logic). Without deduplication, the same session event can be processed more than once, leading to duplicate session data in Bigtable.

Answer

Session windowing is configured with too short a gap duration.

Answer

Bigtable schema design causes row key collisions.

Answer

Dataflow's default behavior discards late-arriving data.

Question 20

A financial services firm processes sensitive transactions using Cloud Dataflow. The pipeline reads from Pub/Sub, performs stateful processing (e.g., fraud detection), and writes to Cloud Spanner. Compliance requires exactly-once processing semantics. Which configuration ensures exactly-once processing?

Accepted Answer

Set Dataflow pipeline to exactly-once mode and design Spanner writes to be idempotent.. Option C is correct because exactly-once processing in a Dataflow pipeline requires the pipeline itself to be set to exactly-once mode (which uses consistent snapshots and transactional sinks) and the output writes to Spanner to be idempotent. This combination ensures that even if a record is reprocessed due to failures, the final state in Spanner remains consistent, satisfying compliance requirements.

Answer

Configure Pub/Sub to use exactly-once delivery mode.

Answer

Use Pub/Sub with at-least-once delivery and Dataflow with at-least-once processing mode.

Answer

Enable Dataflow's streaming engine and use Spanner's built-in retry logic.