PDE Building and operationalizing data processing systems — Timed 20 Questions

Question 1

A company is migrating its on-premises Apache Spark jobs to Dataproc. The jobs read from and write to Cloud Storage. After migration, the jobs are slower than expected. The Dataproc cluster uses standard worker machines with local SSDs. What is the most likely cause of the performance degradation?

Accepted Answer

The jobs use the Cloud Storage connector instead of HDFS, causing network latency.. D is correct because the performance degradation is most likely due to network latency when using the Cloud Storage connector instead of HDFS. Cloud Storage is an object store accessed over the network, while HDFS leverages local SSDs for data locality and faster I/O. In Dataproc, jobs that read/write to Cloud Storage incur higher latency compared to using HDFS on local SSDs, especially for shuffle-heavy Spark workloads.

Answer

The Spark shuffle service is not enabled on the cluster.

Answer

The local SSDs are not mounted or are misconfigured.

Answer

The Cloud Storage connector is not using the gRPC protocol.

Question 2

A data pipeline ingests real-time events from Cloud Pub/Sub into BigQuery using Dataflow. The pipeline uses a sliding window of 5 minutes with a 1-minute period to aggregate event counts. Recently, the pipeline started failing with 'The worker failed to provide a heartbeat.' The Dataflow logs show high CPU usage on the workers. What is the best course of action to resolve the issue?

Accepted Answer

Increase the number of workers and enable autoscaling to distribute the load.. The 'worker failed to provide a heartbeat' error combined with high CPU usage indicates that workers are overloaded and cannot process data fast enough to maintain their heartbeat to the Dataflow service. Increasing the number of workers and enabling autoscaling distributes the computational load across more machines, reducing per-worker CPU pressure and allowing heartbeats to be sent on time. This directly addresses the root cause of resource exhaustion.

Answer

Reduce the number of workers to minimize coordination overhead.

Answer

Use a global window with a trigger to reduce state size.

Answer

Change the windowing to a fixed 5-minute window to reduce computations.

Question 3

A company wants to process large CSV files stored in Cloud Storage and load them into BigQuery. The files are generated daily and each file is about 10 GB. The data is not time-sensitive and can be processed within a 24-hour window. Which service is most cost-effective for this use case?

Accepted Answer

Dataproc Serverless with PySpark. Dataproc Serverless with PySpark is the most cost-effective choice because it eliminates cluster management overhead and automatically scales resources based on workload, charging only for the processing time used. For 10 GB CSV files processed daily within a 24-hour window, the serverless model avoids the fixed costs of a persistent cluster, making it ideal for batch, non-time-sensitive jobs. PySpark's native support for CSV parsing and BigQuery integration via the Spark BigQuery connector ensures efficient data loading without additional services.

Answer

Dataflow with batch mode

Answer

Cloud Data Fusion

Answer

BigQuery Data Transfer Service

Question 4

A financial services company uses Cloud Composer to orchestrate a daily workflow that includes a Dataproc job for risk analysis. The workflow sometimes fails because the Dataproc cluster creation times out. The cluster creation typically takes 3 minutes, but occasionally takes over 10 minutes. What is the most effective way to handle this variability?

Accepted Answer

Create a long-running Dataproc cluster that remains idle and reuse it for each workflow.. Option A is correct because creating a long-running Dataproc cluster and reusing it eliminates the variable cluster creation time that causes timeouts. Cloud Composer (Airflow) can manage cluster lifecycle separately from the workflow, ensuring the cluster is always available when the Dataproc job runs. This approach decouples cluster provisioning from job execution, making the workflow resilient to creation delays.

Answer

Implement a retry loop with exponential backoff in the DAG.

Answer

Use preemptible VMs for the cluster to reduce cost and improve creation speed.

Answer

Increase the cluster creation timeout in the Airflow configuration.

Question 5

A company is using Dataflow to stream data from Cloud Pub/Sub to BigQuery. The pipeline includes a custom ParDo transformation that enriches the data with external API calls. The pipeline is experiencing high latency and occasional failures due to API timeouts. What strategy should be employed to improve reliability and performance?

Accepted Answer

Use a DoFn with stateful processing and batch API calls using asynchronous HTTP client.. Option C is correct because using a DoFn with stateful processing and an asynchronous HTTP client allows the pipeline to batch API calls and handle timeouts without blocking the main processing thread. This reduces latency by enabling concurrent requests and improves reliability through retry logic and state management, which is essential for external API enrichment in Dataflow.

Answer

Remove the enrichment step and store raw data in BigQuery.

Answer

Use a global window to accumulate all data before enrichment.

Answer

Increase the number of workers to parallelize API calls.

Question 6

A data engineer needs to process a large dataset (500 TB) stored in Cloud Storage using Dataproc. The processing job requires reading the entire dataset and writing results back to Cloud Storage. The job is expected to run for 6 hours. Which configuration minimizes cost?

Accepted Answer

Use a cluster with a mix of standard and preemptible VMs.. Option C is correct because preemptible VMs cost about 80% less than standard VMs, and mixing them with standard VMs provides fault tolerance for the job's 6-hour duration. Since the job reads and writes to Cloud Storage (not local HDFS), local SSDs are unnecessary, and a single-node cluster would lack the parallelism needed to process 500 TB efficiently within 6 hours. Using a mix of standard (for critical master/worker nodes) and preemptible VMs (for worker nodes) minimizes cost while ensuring job completion.

Answer

Use a single-node cluster with standard VMs.

Answer

Use a cluster with local SSDs for faster I/O.

Answer

Use a cluster with n1-highmem-32 instances and 1000 cores.

Question 7

A company runs a Dataflow streaming pipeline that reads from Cloud Pub/Sub and writes to BigQuery. The pipeline uses a side input that is a large lookup table (10 GB) stored in Cloud Storage. The side input is updated hourly. The pipeline experiences high latency and OOM errors on workers. What is the best approach to resolve this?

Accepted Answer

Use a Cloud Bigtable table as a side input via a RichSDF.. Option A is correct because using a Cloud Bigtable table as a side input via a RichSDF (Rich Splittable DoFn) allows the pipeline to perform point lookups on the large (10 GB) lookup table without loading it entirely into worker memory. This avoids OOM errors and reduces latency by leveraging Bigtable's low-latency, scalable key-value storage, which is ideal for high-throughput streaming pipelines that require frequent, random access to a large, frequently updated dataset.

Answer

Use a side input from a PCollection and broadcast it.

Answer

Increase the number of workers to distribute the side input.

Answer

Increase the worker memory to 16 GB per worker.

Question 8

A company uses Cloud Composer to orchestrate data pipelines. One DAG fails intermittently with the error: 'Task received SIGTERM signal.' The task runs a long-running Dataproc job. What is the most likely cause?

Accepted Answer

The Airflow task timed out due to the default execution timeout.. The default Airflow task execution timeout is 28 days in Cloud Composer, but individual tasks can have a shorter `execution_timeout` set in the DAG definition. When a long-running Dataproc job exceeds this timeout, Airflow sends a SIGTERM signal to the task to kill it, resulting in the observed error. This is the most likely cause because the error message directly indicates a forced termination by the Airflow scheduler, not an infrastructure or code failure.

Answer

The Dataproc cluster was preempted by Google Cloud.

Answer

The Dataproc job failed due to an error in the code.

Answer

The Cloud Composer environment ran out of disk space.

Question 9

Which TWO factors should be considered when choosing between Cloud Dataflow and Dataproc for a batch processing pipeline?

Accepted Answer

Dataflow is built for data processing patterns, while Dataproc is better for general-purpose compute.. Option B is correct because Dataflow is purpose-built for data processing patterns like batch and stream processing with unified programming models (Apache Beam), while Dataproc is optimized for general-purpose compute workloads such as running custom Spark, Hadoop, or ML jobs. Option D is correct because Dataflow provides automatic horizontal autoscaling based on pipeline throughput, whereas Dataproc requires manual cluster sizing or configuration of autoscaling policies, which are not as granular or reactive as Dataflow's.

Answer

Dataproc allows custom Docker containers, while Dataflow does not.

Answer

Dataproc supports Python, while Dataflow only supports Java.

Answer

Dataflow supports Java and Python, while Dataproc only supports Java.

Question 10

Which THREE best practices should be followed when designing a Dataflow pipeline for real-time data processing?

Accepted Answer

Set up monitoring alerts for system lag and data freshness.. Option A is correct because monitoring alerts for system lag and data freshness are essential for maintaining operational visibility in real-time Dataflow pipelines. System lag (the time between data ingestion and processing) and data freshness (how current the processed output is) directly impact the pipeline's ability to meet latency SLAs. Without these alerts, issues like worker backpressure or Pub/Sub subscription backlog can go unnoticed, leading to stale or lost data.

Answer

Use static side inputs that are loaded once at pipeline start.

Answer

Use global windows with early triggers for low latency.

Question 11

Which TWO actions can reduce the cost of running a Dataproc cluster for a nightly batch job?

Accepted Answer

Use preemptible VMs for worker nodes.. Preemptible VMs (Option C) are significantly cheaper than standard VMs because Compute Engine can terminate them at any time, making them ideal for fault-tolerant, stateless batch jobs like nightly data processing on Dataproc. Deleting the cluster after the job completes (Option E) eliminates ongoing compute costs for idle resources, which is a best practice for ephemeral workloads.

Answer

Increase the number of worker nodes for faster processing.

Answer

Use high-memory machine types for master node.

Answer

Attach local SSDs to all nodes.

Question 12

Refer to the exhibit. A Dataflow pipeline writes to BigQuery table employee_records. The pipeline was working yesterday but fails today. What is the most likely cause?

Accepted Answer

The pipeline code was changed to send an integer for the last_name field.. Option B is correct because if the pipeline code was changed to send an integer for the last_name field, BigQuery will reject the write due to a schema mismatch. BigQuery enforces strict type checking at ingestion time; an integer value cannot be written into a STRING column unless the schema explicitly allows coercion. Since the pipeline was working yesterday, the most likely change is in the data type being sent, not the schema itself.

Answer

The pipeline dropped the last_name field entirely.

Answer

The BigQuery table quota was exceeded.

Answer

The BigQuery table schema was changed from STRING to INTEGER for last_name.

Question 13

Refer to the exhibit. A Dataflow streaming pipeline subscribes to this Pub/Sub subscription. The pipeline occasionally takes more than 10 seconds to process a message. Which behavior will occur?

Accepted Answer

The message will be redelivered after 10 seconds if not acknowledged.. Option C is correct because Pub/Sub delivery requires an acknowledgment within the configurable `ackDeadlineSeconds` (default 10 seconds). If the pipeline takes longer than the ack deadline to process a message, Pub/Sub considers the message unacknowledged and redelivers it. This is the standard behavior for at-least-once delivery in Google Cloud Pub/Sub.

Answer

The message will be sent to the dead letter topic immediately.

Answer

The message will be retried with exponential backoff as per retry policy.

Answer

The message will be dropped after 10 seconds due to expiration policy.

Question 14

A company runs a critical real-time data pipeline using Dataflow that ingests events from Cloud Pub/Sub, performs aggregations using sliding windows, and writes results to BigQuery. The pipeline is deployed in us-central1. The pipeline's latency has increased recently, and the Dataflow monitoring shows that the 'system lag' metric is consistently above 5 minutes. The pipeline is using Streaming Engine and has 10 workers with 4 vCPUs each. The pipeline processes approximately 100,000 events per second. The team has verified that the source Pub/Sub topic has sufficient publish throughput and the BigQuery table has no quota issues. The pipeline logs show that some workers are experiencing GC overhead limit exceeded errors. The pipeline code uses stateful processing with a custom keyed state for deduplication. What is the most likely cause of the increased latency?

Accepted Answer

The stateful processing is causing large state sizes that lead to GC overhead; use a more efficient state backend or increase worker memory.. The GC overhead limit exceeded errors indicate that workers are spending too much time garbage collecting, which is a classic symptom of excessive heap memory usage. Stateful processing with custom keyed state for deduplication can cause large per-key state sizes, especially with sliding windows that maintain overlapping state for each key. This forces the JVM to constantly garbage collect, increasing system lag beyond 5 minutes. Using a more efficient state backend (e.g., reducing state size or using Dataflow's built-in deduplication) or increasing worker memory directly addresses the root cause.

Answer

The number of workers is insufficient; increasing to 20 workers will reduce latency.

Answer

The sliding window duration is too long; reducing it to 1 minute will improve performance.

Answer

The deduplication logic is causing a bottleneck; removing it will reduce latency.

Question 15

A company uses Cloud Composer to orchestrate a daily ETL pipeline that includes multiple Dataproc jobs. The pipeline processes sensitive financial data. The security team requires that all data in transit be encrypted, and all Cloud Storage buckets used by the pipeline should have uniform bucket-level access enabled and VPC Service Controls. The pipeline currently uses a single Cloud Composer environment in us-east1. The Dataproc clusters are created using the standard image and use custom service accounts with minimal permissions. The pipeline runs successfully during testing, but in production, the Dataproc jobs fail with 'Access Denied' errors when trying to write to a Cloud Storage bucket. The bucket has uniform bucket-level access enabled and is inside a VPC Service Controls perimeter. The Dataproc service account has the Storage Object Admin role at the project level. What is the most likely cause of the access denied error?

Accepted Answer

The Dataproc cluster is not in the VPC Service Controls perimeter.. The Dataproc cluster is created outside the VPC Service Controls perimeter, so even though the service account has the Storage Object Admin role at the project level, requests from the cluster are blocked by the perimeter's ingress/egress rules. VPC Service Controls enforce a security boundary that prevents resources outside the perimeter from accessing protected services like Cloud Storage, regardless of IAM permissions. The 'Access Denied' error in production, despite successful testing, strongly indicates a perimeter configuration mismatch.

Answer

The service account does not have the Storage Object Admin role on the bucket.

Answer

Data in transit encryption is not enabled for the Cloud Storage bucket.

Answer

Uniform bucket-level access prevents writes from service accounts.

Question 16

A company is building a real-time streaming pipeline using Pub/Sub and Dataflow to process clickstream data. The pipeline writes aggregated metrics to BigQuery every 10 seconds using a fixed window. During peak traffic, some windows produce duplicate rows in BigQuery. What is the most likely cause?

Accepted Answer

Dataflow is retrying BigQuery streaming inserts after a timeout, and the retries succeed even though the original insert succeeded.. Option A is correct because Dataflow uses at-least-once semantics for streaming inserts into BigQuery. When a streaming insert times out, Dataflow retries the insert, and if the original insert actually succeeded but the acknowledgment was lost, the retry produces a duplicate row. This is a known behavior of BigQuery streaming inserts with retry logic.

Answer

The pipeline uses default triggers instead of after-watermark triggers.

Answer

The fixed window duration is too short, causing overlapping windows.

Answer

The pipeline is using too many Dataflow workers, causing load balancing issues.

Question 17

A data engineer is designing a batch ETL pipeline that reads CSV files from Cloud Storage, transforms them using Dataproc, and writes the results to BigQuery. The data volume is expected to grow 10x in the next year. Which design approach best balances cost and performance?

Accepted Answer

Use a Dataproc cluster with preemptible worker nodes and autoscaling enabled.. Option C is correct because preemptible worker nodes significantly reduce cost (up to 80% discount) while autoscaling dynamically adjusts cluster size to match the growing workload, ensuring performance without over-provisioning. This combination handles the 10x data growth efficiently by scaling out during peak loads and scaling in during lulls, using preemptible instances for fault-tolerant tasks like transformation.

Answer

Create a single large persistent Dataproc cluster to handle the peak load.

Answer

Use Cloud Data Fusion to visually design the pipeline and run it on Dataproc.

Answer

Migrate the pipeline to Dataflow with Apache Beam and use flexRS for cost savings.

Question 18

A data team uses Cloud Composer to orchestrate Airflow DAGs. They need to ensure that a downstream task runs only if at least two out of three upstream sensor tasks succeed. Which TWO configurations should they combine?

Accepted Answer

Set trigger_rule to 'none_failed_or_skipped' and use a condition.. Option A is correct because the 'none_failed_or_skipped' trigger rule triggers the downstream task when all upstream tasks have succeeded or been skipped. Combined with a condition (e.g., using a PythonOperator or BranchPythonOperator) that checks whether at least two of the three sensor tasks succeeded, this ensures the downstream task runs only when the required threshold is met. This approach leverages Airflow's built-in trigger rules and conditional logic to implement a quorum-based dependency.

Answer

Set trigger_rule to 'one_success'.

Answer

Set trigger_rule to 'all_done'.

Answer

Set trigger_rule to 'none_failed'.

Question 19

A Dataflow pipeline reads log files from Cloud Storage, parses them into LogEvent objects, and writes to BigQuery. The pipeline fails with the above errors. What is the most likely cause?

Accepted Answer

The LogEvent class does not have a no-argument constructor.. Apache Beam's SDK requires that custom types used as PCollection elements (like LogEvent) have a no-argument constructor so that the framework can deserialize objects during distributed processing, especially when using the Dataflow runner. Without it, the pipeline fails at runtime with a serialization error because Beam's default coder (e.g., SerializableCoder) cannot reconstruct the object.

Answer

The pipeline is missing required import statements for LogEvent.

Answer

The BigQuery table schema does not match the LogEvent fields.

Answer

The log files are not in the expected format, causing parsing failures.

Question 20

A financial services company runs a batch Dataflow pipeline daily to process transaction data. The pipeline reads from Cloud Storage, performs complex transformations, and writes to BigQuery. Recently, the pipeline has been failing intermittently with the error: 'Workflow failed. Causes: (9c3f7a2b1d4e): The worker missed 2000 data samples in the last 30 seconds. This can be caused by a variety of factors, including slow work items, network issues, or resource contention.' The team has already increased the number of workers and tried using e2-standard-8 machine types, but the issue persists. The pipeline processes approximately 500 GB of data per run and uses approximately 200 workers. The team suspects that the issue might be related to shuffle operations. What should the team do next to resolve the issue?

Accepted Answer

Increase the persistent disk size per worker to 100 GB.. The error indicates that workers are missing data samples due to slow shuffle operations, often caused by insufficient disk I/O. Increasing the persistent disk size per worker to 100 GB provides more local scratch space for Dataflow's shuffle, reducing disk contention and allowing the shuffle to complete within the 30-second window. This directly addresses the root cause without changing the worker count or machine type.

Answer

Enable Streaming Engine for the pipeline.

Answer

Reduce the number of workers to 100 to decrease shuffle overhead.

Answer

Use Cloud Storage as a shuffle sink.