PDE Ingesting and Processing the Data Practice Test 3 — 15 Questions

Question 1

You are building a streaming pipeline to ingest real-time clickstream data from a website into BigQuery for immediate analysis. The data must be available in BigQuery within seconds and you need to handle late-arriving data (e.g., browser offline events) that may arrive hours later. Which approach should you use?

Accepted Answer

Use Pub/Sub with Dataflow, writing to BigQuery using the Storage Write API in committed mode.. Option A is correct because Pub/Sub provides a scalable, durable ingestion layer for real-time clickstream data, and Dataflow can handle late-arriving data via its built-in watermark and trigger mechanisms. The Storage Write API in committed mode ensures exactly-once semantics and low-latency writes to BigQuery, meeting the sub-second availability requirement while preserving data consistency for delayed events.

Answer

Use Cloud Logging to capture logs and export to BigQuery via a sink.

Answer

Use Pub/Sub with Cloud Functions, writing each event directly via BigQuery legacy streaming inserts.

Answer

Use Datastream to stream clickstream data from Cloud SQL to BigQuery.

Question 2

A company wants to transfer 500 TB of data from an on-premises Hadoop cluster to Google Cloud Storage (GCS) for processing with Dataproc. The on-premises network has a 1 Gbps dedicated link to Google Cloud. The data must be transferred as quickly as possible, minimizing network usage. Which transfer method should they use?

Accepted Answer

Use Transfer Appliance to physically ship the data.. Transfer Appliance is the correct method because the dataset is 500 TB and the network link is only 1 Gbps. At 1 Gbps, the theoretical maximum transfer time is over 46 days, and real-world throughput (due to overhead, congestion, and Hadoop data characteristics) would be even longer. Transfer Appliance physically ships the data, bypassing the network bottleneck entirely and minimizing network usage, which is the stated requirement.

Answer

Use Storage Transfer Service over the 1 Gbps link.

Answer

Use gsutil cp in parallel with multiple threads.

Answer

Use BigQuery Data Transfer Service for Hadoop.

Question 3

Your team is processing a large dataset with Apache Beam on Dataflow. The pipeline sometimes fails due to transient errors when writing to a BigQuery sink. You need to ensure that failed records are not lost and can be reprocessed later without blocking the pipeline. What is the best approach?

Accepted Answer

Use a side output (e.g., via TupleTag) to write failed records to a dead letter sink (e.g., GCS or Pub/Sub) and continue processing the main output.. Using a dead letter pattern with a side output to write failed records to a GCS bucket (or Pub/Sub) allows the pipeline to continue processing healthy records while failed records are stored for later analysis and reprocessing.

Answer

Configure the pipeline to use at-least-once semantics and rely on Dataflow to retry the entire bundle.

Answer

Increase the number of workers to reduce the chance of transient errors.

Answer

Use a try-catch block in the DoFn and log the error; continue processing other elements.

Question 4

You are designing a near-real-time CDC pipeline to replicate changes from an on-premises PostgreSQL database to BigQuery for analytics. The source database has high transaction volume and you must ensure minimal impact on the source. Which Google Cloud service should you use to ingest the change data?

Accepted Answer

Datastream to stream changes to GCS, then load into BigQuery.. Datastream is purpose-built for CDC from MySQL, PostgreSQL, and Oracle to BigQuery or GCS. It reads the database logs (e.g., WAL) to capture changes with low latency and minimal impact on the source.

Answer

Pub/Sub with a custom connector that polls the database every minute.

Answer

Use BigQuery Data Transfer Service for PostgreSQL.

Answer

Use Dataflow with a JDBC IO connector to read from PostgreSQL.

Question 5

You are loading 10 GB of daily CSV files from a GCS bucket into a BigQuery table. The files contain some malformed rows that you want to skip. Which BigQuery load configuration should you use?

Accepted Answer

Use the 'max_bad_records' option set to a value like 10.. BigQuery allows setting max_bad_records in load jobs; records exceeding this threshold cause the job to fail. Setting max_bad_records to a value greater than 0 allows the load to succeed while skipping malformed rows.

Answer

Use the 'skip_leading_rows' option.

Answer

Use the 'ignore_unknown_values' option.

Answer

Use the 'allow_jagged_rows' option.

Question 6

Your Dataflow pipeline reads from Pub/Sub, performs transformations, and writes to BigQuery. You notice that the pipeline's autoscaling is not keeping up with sudden spikes in traffic, causing increased lag. The pipeline uses Classic Templates. Which change would most effectively improve autoscaling responsiveness?

Accepted Answer

Enable Dataflow Streaming Engine on the pipeline.. Enabling Dataflow Streaming Engine reduces the overhead of checkpointing and state management by offloading them to the service side, which allows the pipeline to scale more quickly in response to sudden traffic spikes. This directly addresses the autoscaling lag because Streaming Engine decouples compute from state, enabling faster worker adjustments without the bottleneck of persistent disk-based shuffle.

Answer

Switch to Dataflow Prime with Vertical Autoscaling enabled.

Answer

Increase the initial number of workers to handle the spike.

Answer

Use Flex Templates instead of Classic Templates.

Question 7

You are migrating an existing Kafka cluster to Google Cloud using Dataproc. The cluster handles high-throughput streaming data with strict ordering requirements per partition. Which choice of Dataproc configuration is most appropriate?

Accepted Answer

Use Dataproc with non-preemptible workers and persistent SSD storage for brokers.. Option D is correct because Kafka brokers in a Dataproc cluster require persistent, non-preemptible workers to maintain data durability and strict ordering per partition. Preemptible workers can be terminated at any time, causing data loss or rebalancing that violates ordering guarantees. Persistent SSD storage provides the low-latency I/O needed for high-throughput Kafka workloads, while non-preemptible instances ensure broker stability and consistent replication.

Answer

Use Dataflow with Kafka IO instead of Dataproc.

Answer

Use Dataproc with local SSDs for better performance, and enable autoscaling.

Answer

Use Dataproc with preemptible workers to reduce cost, and attach standard persistent disks.

Question 8

Your team uses dbt to transform data in BigQuery. You need to schedule dbt runs to refresh materialized tables and views every hour. The transformations include both full refreshes and incremental models. What is the most efficient way to orchestrate these dbt runs on Google Cloud?

Accepted Answer

Use Cloud Composer (Airflow) to schedule and run dbt commands.. Cloud Composer (Airflow) is the recommended orchestration tool for complex workflows like dbt runs, supporting dependencies, retries, and scheduling. Cloud Scheduler alone cannot run dbt directly; it can trigger a Cloud Function to run dbt, but that is less maintainable. Cloud Build is CI/CD, not scheduling. Using a cron job on Compute Engine is possible but not managed.

Answer

Use Cloud Build with a trigger to run dbt every hour.

Answer

Use Cloud Scheduler to trigger a Cloud Function that runs dbt.

Answer

Set up a cron job on a Compute Engine instance to run dbt.

Question 9

You need to ingest Google Ads performance data into BigQuery on a daily basis for reporting. Which service should you use?

Accepted Answer

BigQuery Data Transfer Service for Google Ads. The BigQuery Data Transfer Service for Google Ads is the correct choice because it provides a fully managed, scheduled connector that automatically ingests Google Ads performance data into BigQuery on a daily basis without requiring any custom code. It handles authentication, schema mapping, and incremental loads, making it the simplest and most reliable solution for this specific use case.

Answer

Cloud Scheduler to call Google Ads API and load to BigQuery

Answer

Pub/Sub with a Google Ads subscriber

Answer

Storage Transfer Service for Google Ads

Question 10

You are designing a streaming pipeline that ingests events from Pub/Sub, enriches them with a machine learning model, and writes the results to BigQuery. The ML model is deployed on Cloud Run and has a high latency (500ms per request). You need to minimize the impact of slow ML inference on the overall pipeline throughput. Which approach should you take?

Accepted Answer

Use Dataflow to write events to Pub/Sub, then use a separate Dataflow pipeline that batches calls to Cloud Run.. Option A is correct because it uses Dataflow to batch events before sending them to Cloud Run, which amortizes the 500ms per-request latency over multiple events, significantly increasing throughput. By writing events to Pub/Sub and then processing them in a separate Dataflow pipeline with batched calls, you decouple the ingestion from the inference and avoid blocking on each individual request.

Answer

Increase the number of Dataflow workers to compensate for the latency.

Answer

Use Cloud Functions to call Cloud Run and write directly to BigQuery.

Answer

Use Dataflow's ParDo with synchronous calls to Cloud Run for each element.

Question 11

Your team has a Dataflow pipeline that reads from BigQuery, transforms data, and writes to GCS. The pipeline is failing with 'Out of Memory' errors on the worker nodes. The input data is large but fits within the total cluster memory. Which configuration change is most likely to resolve the issue without increasing costs significantly?

Accepted Answer

Use a worker machine type with more memory, such as n2-highmem.. The default Dataflow worker machine type may have insufficient memory per core for the pipeline's operations. Using a high-memory machine type (e.g., n2-highmem) increases memory per worker without necessarily increasing the number of workers, thus controlling costs.

Answer

Shard the input into smaller reads using a BigQuery query.

Answer

Increase the disk size per worker.

Answer

Enable Dataflow Prime with vertical autoscaling.

Question 12

You need to react to changes in a GCS bucket (e.g., new object creation) and trigger a Cloud Run service to process the new file. Which Google Cloud service should you use to route the event?

Accepted Answer

Eventarc. Eventarc is the correct choice because it is purpose-built to route events from Google Cloud sources (like Cloud Storage) to Cloud Run. It directly supports Cloud Storage audit logs and Pub/Sub event triggers, allowing you to react to object creation events without custom middleware. Eventarc handles the event routing, filtering, and delivery to your Cloud Run service automatically.

Answer

Pub/Sub directly with a Cloud Run subscription

Answer

Cloud Tasks

Answer

Cloud Scheduler

Question 13

You need to ingest streaming data from a custom application into BigQuery with exactly-once semantics and low latency. The data volume is up to 10 MB/s. Which TWO services should you combine?

Accepted Answer

Pub/Sub. Pub/Sub provides reliable, low-latency message ingestion, and Dataflow can read from Pub/Sub and write to BigQuery using the Storage Write API, which supports exactly-once semantics. The Storage Write API with committed mode ensures exactly-once delivery.

Answer

Cloud Functions

Answer

BigQuery legacy streaming inserts

Answer

Datastream

Question 14

Your company has a Dataproc cluster that runs Spark jobs. You need to choose between RDDs, DataFrames, and Datasets for a new job that performs complex aggregations on structured data. Which TWO statements are correct regarding performance and ease of use?

Accepted Answer

DataFrames store data in a columnar format, allowing better compression.. DataFrames are optimized with Catalyst optimizer and Tungsten execution, providing better performance than RDDs for structured data. Datasets combine type safety with optimized execution, but for most analytics workloads, DataFrames are sufficient and simpler.

Answer

DataFrames and Datasets are both available in PySpark.

Answer

RDDs are easier to use than DataFrames for complex aggregations.

Answer

Datasets provide compile-time type safety and are always faster than DataFrames.

Question 15

You are building a BigQuery table that contains nested and repeated fields (e.g., order with line items). You need to write a query that counts the number of line items per order. Which TWO SQL functions/techniques can you use?

Accepted Answer

UNNEST with COUNT. Option B is correct because UNNEST flattens the repeated line items array into individual rows, allowing COUNT to aggregate the number of line items per order. Option D is correct because ARRAY_LENGTH directly returns the number of elements in the repeated field array, which corresponds to the line item count.

Answer

Window function ROW_NUMBER

Answer

STRUCT with aggregation

Answer

SELECT * EXCEPT