PDE Practice Test 30 — 15 Questions

Question 1

You are designing a streaming pipeline using Cloud Dataflow with exactly-once semantics. The source is Pub/Sub and the sink is Cloud Bigtable. The pipeline must handle late data up to 10 minutes. You need to minimize cost while maintaining correctness. Which configuration should you use?

Accepted Answer

Global window with allowed lateness 10 minutes and trigger=afterWatermark with early firings. Option C is correct because a global window with an after-watermark trigger and early firings is the most cost-effective way to handle unbounded data from Pub/Sub with exactly-once semantics, while allowing up to 10 minutes of lateness. Fixed or sliding windows would create many small window states, increasing Bigtable write costs and shuffle overhead. The global window minimizes state and processing, and the trigger ensures results are emitted promptly without accumulating panes.

Answer

Fixed windows of 1 minute with allowed lateness 10 minutes and accumulating fired panes

Answer

Sliding windows of 1 minute with allowed lateness 10 minutes and accumulating fired panes

Answer

Session windows of 5 minutes with gap duration 1 minute and discarding fired panes

Question 2

Your company wants to analyze real-time user clickstream data from a website. The data arrives as JSON messages via an HTTP endpoint. The pipeline should be able to handle spikes in traffic, provide low-latency insights, and store the raw data in a data lake for historical analysis. Which Google Cloud service should you use to ingest and process the streaming data?

Accepted Answer

Cloud Pub/Sub combined with Dataflow. Cloud Pub/Sub is the correct ingestion service because it provides a highly scalable, fully managed message queue that can handle traffic spikes by decoupling producers from consumers. Dataflow (Apache Beam) then processes the streaming data with low latency, supports exactly-once semantics, and can write raw data to a data lake like Cloud Storage for historical analysis. This combination meets all requirements: spike handling, low-latency insights, and raw data storage.

Answer

Cloud Dataproc

Answer

Cloud Functions

Answer

Cloud IoT Core

Question 3

Your team uses Cloud Dataproc to run a Spark ML training job. The job is failing with an error: 'Container killed by YARN for exceeding memory limits.' What should you do to fix this?

Accepted Answer

Increase the spark.executor.memory property. The error 'Container killed by YARN for exceeding memory limits' indicates that the Spark executor process is using more memory than the YARN container allows. Increasing `spark.executor.memory` allocates a larger YARN container for each executor, providing the necessary headroom for the Spark application's memory demands, including overhead for off-heap memory and JVM internals.

Answer

Use preemptible VMs for faster execution

Answer

Increase the number of worker nodes

Answer

Enable the external shuffle service

Question 4

Which THREE features of Cloud Pub/Sub guarantee at-least-once delivery and enable exactly-once processing downstream? (Choose three.)

Accepted Answer

Subscriber-retry policy with exponential backoff.. Option A is correct because a subscriber-retry policy with exponential backoff ensures that messages that fail to be processed are retried with increasing delays, preventing transient failures from causing message loss. This mechanism, combined with Pub/Sub's persistent storage, guarantees that each message is delivered at least once, as the subscriber will keep retrying until it acknowledges the message.

Answer

Message ordering by message key.

Answer

Cloud Dataproc integration for message replay.

Question 5

A company is designing a data lake on Google Cloud. The data lake will store raw, curated, and analytics-ready data. Security requirements include: data must be encrypted at rest and in transit, access must be controlled based on data sensitivity (public, internal, confidential), and all access to sensitive data must be audited. The company also wants to minimize data transfer costs for frequently accessed curated datasets. Which combination of services and configurations best meets these requirements?

Accepted Answer

Use Cloud Storage with CMEK, bucket-level IAM, and object ACLs. Use Cloud Data Loss Prevention API to classify data. Enable Cloud Audit Logs. Use Cloud CDN to cache curated datasets.. Option D is correct because it combines CMEK for encryption at rest (with Cloud HSM for key management), bucket-level IAM and object ACLs for granular access control based on data sensitivity, Cloud Audit Logs for auditing access to sensitive data, and Cloud CDN to cache curated datasets, reducing data transfer costs for frequently accessed data. This configuration meets all security requirements (encryption at rest and in transit, access control, auditing) while optimizing cost for frequent access.

Answer

Use Cloud Storage with default encryption, bucket policies, and Cloud Audit Logs. For frequent access, use Cloud CDN.

Answer

Use Cloud Storage with CMEK, and use Cloud HSM for key storage. Use Cloud Audit Logs. Avoid caching to ensure security.

Answer

Use Cloud Storage with SSE-C, bucket policies, and Cloud Audit Logs. Use Cloud Load Balancing for caching.

Question 6

A company has a Cloud Functions function that triggers on new files in Cloud Storage and writes a message to Pub/Sub for downstream processing. Recently, the function has been timing out after 60 seconds. The downstream processing is critical. What is the best solution?

Accepted Answer

Increase function timeout to 540 seconds and delegate heavy processing to Cloud Dataflow. Option D is correct because Cloud Functions has a maximum timeout of 540 seconds (9 minutes) for HTTP-triggered functions, and by increasing the timeout you allow the function to complete its work. Delegating heavy processing to Cloud Dataflow offloads the computationally intensive tasks, preventing future timeouts and ensuring scalable, reliable downstream processing for critical workloads.

Answer

Replace Cloud Functions with a Cloud Run job that has longer timeout

Answer

Increase the function memory to 2 GB to speed up execution

Answer

Reduce the function timeout to 30 seconds to force faster execution

Question 7

A company uses Cloud Composer (Airflow) to orchestrate a daily batch job that runs a custom Python script on a Compute Engine instance. The process is slow because the instance takes 2 minutes to boot. How can you reduce the total runtime?

Accepted Answer

Create a custom image with the script and dependencies pre-installed. Option C is correct because creating a custom image with the script and dependencies pre-installed eliminates the need to install packages or configure the environment at boot time. In Cloud Composer, when a Compute Engine instance is provisioned via a BashOperator or SSHOperator, the boot process includes OS initialization and package installation. A custom image bypasses these steps, reducing boot time from minutes to seconds, directly addressing the 2-minute boot delay.

Answer

Switch to Dataproc Serverless to avoid VM boot time

Answer

Use a larger machine type for faster provisioning

Answer

Use a GPU-accelerated instance to speed up the script

Question 8

You are designing a streaming data pipeline that must guarantee exactly-once processing semantics for financial transactions. The pipeline reads from Cloud Pub/Sub and writes to Cloud Bigtable. Each transaction has a unique transaction ID. Which features do you need to implement to ensure exactly-once semantics end-to-end?

Accepted Answer

Use Dataflow with exactly-once processing, and ensure the Bigtable sink uses idempotent mutations based on the transaction ID.. Option B is correct because Dataflow's exactly-once processing guarantees that each record is processed precisely once, and idempotent Bigtable mutations (keyed by transaction ID) ensure that even if a mutation is retried, the result is the same. This combination provides end-to-end exactly-once semantics: Dataflow handles source-side deduplication and checkpointing, while Bigtable's idempotent writes prevent duplicates at the sink.

Answer

Use Cloud Pub/Sub with synchronous pull and manually commit offsets after successfully writing to Bigtable.

Answer

Use Dataflow with at-least-once processing and implement deduplication in a windowed transform using the transaction ID.

Answer

Use Cloud Pub/Sub with exactly-once delivery enabled, and write to Bigtable using single-row transactions.

Question 9

Your team runs a weekly batch ETL pipeline using Cloud Dataproc. The pipeline reads raw data from Cloud Storage, transforms it with Apache Spark, and writes results to BigQuery. Recently, the pipeline has been failing with the error 'Out of Memory' during the shuffle phase. The cluster uses standard worker nodes (n1-standard-4). What is the most effective way to resolve this without increasing total cost?

Accepted Answer

Increase the number of Spark partitions by setting spark.sql.shuffle.partitions to a higher value.. The 'Out of Memory' error during the shuffle phase indicates that individual executor tasks are processing too much data per partition. Increasing `spark.sql.shuffle.partitions` reduces the amount of data each task handles, lowering memory pressure per executor without adding more nodes or upgrading hardware. This directly addresses the shuffle memory bottleneck while keeping the total cluster cost unchanged.

Answer

Increase the number of worker nodes by adding more n1-standard-4 instances.

Answer

Enable dynamic allocation and use preemptible VMs for some workers.

Answer

Switch worker nodes to n1-highmem-4 instances to provide more memory.

Question 10

A data pipeline uses Cloud Composer to orchestrate Dataflow and BigQuery jobs. The pipeline fails intermittently with dependency errors. Which design change can improve reliability?

Accepted Answer

Use retries with exponential backoff. Cloud Composer (Apache Airflow) tasks can fail due to transient issues like API rate limits or resource contention. Implementing retries with exponential backoff allows the DAG to automatically re-attempt failed tasks with increasing delays, reducing the impact of intermittent failures without manual intervention. This is a standard Airflow pattern for improving reliability in orchestrated pipelines.

Answer

Switch to Cloud Functions for orchestration

Answer

Increase worker count in Dataflow

Answer

Use a simpler DAG with fewer dependencies

Question 11

A company needs to process large files (100GB each) from Cloud Storage using Dataproc. They want to minimize job execution time. Which configuration is most appropriate?

Accepted Answer

Use a cluster with preemptible worker nodes and high-CPU machine types. Option B is correct because preemptible worker nodes are significantly cheaper than standard nodes, allowing you to scale out the cluster with many more workers for the same cost, which directly reduces job execution time for embarrassingly parallel data processing tasks. High-CPU machine types are ideal for compute-intensive Dataproc jobs like data transformation or machine learning, as they provide more vCPUs per core for parallel processing. This combination maximizes parallelism and minimizes wall-clock time for large-scale batch jobs.

Answer

Use a single-node cluster

Answer

Use HDFS for input data to avoid network latency

Answer

Use a cluster with many standard worker nodes

Question 12

A company building a real-time analytics pipeline with Pub/Sub and Dataflow. Which THREE best practices should they follow?

Accepted Answer

Use event time processing with watermarks and allowed lateness. Option A is correct because in streaming pipelines, event time processing with watermarks and allowed lateness is essential for handling out-of-order data. Watermarks track the progress of event time, and allowed lateness specifies how long to wait for late-arriving data before considering it as late, ensuring accurate windowed aggregations.

Answer

Use exactly-once processing for all transforms

Answer

Use event time processing only for batch pipelines

Question 13

A gaming company uses Pub/Sub to ingest player events and Dataflow for real-time analytics. They notice that the Pub/Sub subscription backlog is growing despite the Dataflow pipeline running continuously. The pipeline has a 1-hour window for aggregations. What is the most effective way to reduce the backlog?

Accepted Answer

Increase the Dataflow pipeline's worker count via autoscaling.. Increasing the Dataflow pipeline's worker count via autoscaling directly addresses the backlog by adding more parallel processing capacity to consume messages from the Pub/Sub subscription faster. Since the pipeline is continuously running but the backlog grows, the bottleneck is processing throughput, not pipeline availability. Autoscaling allows Dataflow to dynamically allocate more workers based on the backlog size, matching consumption rate to the incoming message rate.

Answer

Use a push subscription instead of pull.

Answer

Decrease the window duration to 10 minutes.

Answer

Enable Pub/Sub topic retention.

Question 14

A data engineering team needs to build a data integration pipeline that involves connecting to multiple sources, performing data transformations with visual editing, and then running custom machine learning algorithms. The team has both data analysts and data scientists. Which approach is most suitable?

Accepted Answer

Use Cloud Data Fusion for the initial ingestion and transformations, then export the data to Cloud Dataproc for the ML algorithms. Option D is correct because it leverages Cloud Data Fusion's visual, no-code interface for data ingestion and transformation, which is ideal for data analysts, and then exports the prepared data to Cloud Dataproc, which provides native support for custom machine learning algorithms using Spark or Hadoop, meeting the data scientists' needs. This separation of concerns optimizes the pipeline for both user groups and avoids forcing all tasks into a single tool that may not excel at both visual ETL and custom ML.

Answer

Use Cloud Composer to orchestrate both Data Fusion and Dataproc

Answer

Use only Cloud Dataproc for all steps

Answer

Use only Cloud Data Fusion for all steps

Question 15

A media company uses Cloud Data Loss Prevention (DLP) API to inspect and de-identify sensitive data before loading into BigQuery. They want to reduce costs by sampling the data during inspection. Which configuration should they use?

Accepted Answer

Set the sample method to 'RANDOM' with a percentage.. Option B is correct because the Cloud DLP API supports a 'sample_method' of 'RANDOM' with a 'sampling_percentage' to inspect only a random subset of rows. This directly reduces the volume of data scanned, lowering costs while still providing statistically representative coverage for sensitive data discovery.

Answer

Use the 'ROWS' limit in the inspection job.

Answer

Use a hybrid inspection with a BigQuery sample table.

Answer

Use the 'BYTES_LIMIT' parameter.