PDE Ingesting and Processing the Data Practice Test 4 — 15 Questions

Question 1

A company wants to trigger a Cloud Run service whenever a new file is uploaded to a specific Cloud Storage bucket. Which event-driven solution should they use?

Accepted Answer

Eventarc with Cloud Storage trigger and Cloud Run destination. Eventarc is the recommended service for routing events from Cloud Storage to Cloud Run because it provides a fully managed, event-driven architecture with built-in filtering and retry logic. When a new file is uploaded, Cloud Storage emits a notification that Eventarc captures and delivers directly to the Cloud Run service as an HTTP request, enabling serverless processing without polling or additional infrastructure.

Answer

Cloud Scheduler to periodically poll the bucket

Answer

Cloud Functions triggered by Cloud Storage

Answer

Pub/Sub with a push subscription to Cloud Run

Question 2

A data engineer is using Apache Spark on Dataproc to process a large dataset. They need to perform complex aggregation and transformation with high performance. The dataset has a known schema and they want to take advantage of Catalyst optimizer. Which Spark API should they use?

Accepted Answer

DataFrames. DataFrames provide high-level API with Catalyst optimizer for performance, making them ideal for complex aggregations and transformations on structured data.

Answer

Spark SQL only

Answer

Datasets

Answer

RDDs

Question 3

A company uses Workflows to orchestrate a series of Google Cloud services for data processing. They need to call an external HTTP API as part of the workflow and handle potential failures with retries. Which Workflows feature should they use?

Accepted Answer

Retry policy on the step. Workflows provides a built-in retry policy that can be configured on individual steps to automatically retry an HTTP call upon transient failures (e.g., 5xx server errors or network timeouts). This allows the workflow to handle external API failures without custom code, using exponential backoff and a maximum retry count.

Answer

Subworkflows

Answer

Parallel steps

Answer

Conditional steps

Question 4

A data engineering team needs to ingest streaming data from an existing Kafka cluster (on-premises) into Google Cloud for real-time analytics. They want to minimize changes to the existing Kafka setup and avoid long-term operational overhead. Which TWO approaches should they consider?

Accepted Answer

Deploy a Kafka Connect cluster on Google Cloud with the Pub/Sub sink connector. Using Kafka Connect with the Pub/Sub connector or setting up a Dataproc cluster running Kafka with mirroring are two ways to bridge on-premises Kafka to GCP with minimal changes.

Answer

Use Storage Transfer Service to copy Kafka logs from on-premises to GCS

Answer

Use Datastream to capture changes from Kafka

Answer

Replace the on-premises Kafka cluster with Google Cloud Pub/Sub

Question 5

A large enterprise is migrating its data warehouse from Teradata to BigQuery. They need to transfer historical data (100 TB) and set up ongoing daily incremental loads. They also need to transform the data using dbt. Which THREE Google Cloud services should they use?

Accepted Answer

BigQuery Data Transfer Service for Teradata. BigQuery Data Transfer Service for Teradata handles both historical and incremental loads, dbt runs on BigQuery for transformations, and Cloud Composer can orchestrate the dbt runs on a schedule.

Answer

Datastream

Answer

Transfer Appliance

Question 6

You are building a streaming pipeline to ingest real-time clickstream data from a website into BigQuery for immediate analysis. The data must be available in BigQuery within seconds and you need to handle late-arriving data (e.g., browser offline events) that may arrive hours later. Which approach should you use?

Accepted Answer

Use Pub/Sub with Dataflow, writing to BigQuery using the Storage Write API in committed mode.. Pub/Sub provides a scalable, durable ingestion layer for real-time clickstream data, and Dataflow can handle late-arriving data via its built-in watermark and trigger mechanisms. The Storage Write API in committed mode ensures exactly-once semantics and low-latency writes to BigQuery, meeting the sub-second availability requirement while preserving data consistency for delayed events.

Answer

Use Cloud Logging to capture logs and export to BigQuery via a sink.

Answer

Use Pub/Sub with Cloud Functions, writing each event directly via BigQuery legacy streaming inserts.

Answer

Use Datastream to stream clickstream data from Cloud SQL to BigQuery.

Question 7

A company wants to transfer 500 TB of data from an on-premises Hadoop cluster to Google Cloud Storage (GCS) for processing with Dataproc. The on-premises network has a 1 Gbps dedicated link to Google Cloud. The data must be transferred as quickly as possible, minimizing network usage. Which transfer method should they use?

Accepted Answer

Use Transfer Appliance to physically ship the data.. Transfer Appliance is the correct method because the dataset is 500 TB and the network link is only 1 Gbps. At 1 Gbps, the theoretical maximum transfer time is over 46 days, and real-world throughput (due to overhead, congestion, and Hadoop data characteristics) would be even longer. Transfer Appliance physically ships the data, bypassing the network bottleneck entirely and minimizing network usage, which is the stated requirement.

Answer

Use Storage Transfer Service over the 1 Gbps link.

Answer

Use gsutil cp in parallel with multiple threads.

Answer

Use BigQuery Data Transfer Service for Hadoop.

Question 8

Your team is processing a large dataset with Apache Beam on Dataflow. The pipeline sometimes fails due to transient errors when writing to a BigQuery sink. You need to ensure that failed records are not lost and can be reprocessed later without blocking the pipeline. What is the best approach?

Accepted Answer

Use a side output (e.g., via TupleTag) to write failed records to a dead letter sink (e.g., GCS or Pub/Sub) and continue processing the main output.. Using a dead letter pattern with a side output to write failed records to a GCS bucket (or Pub/Sub) allows the pipeline to continue processing healthy records while failed records are stored for later analysis and reprocessing.

Answer

Configure the pipeline to use at-least-once semantics and rely on Dataflow to retry the entire bundle.

Answer

Increase the number of workers to reduce the chance of transient errors.

Answer

Use a try-catch block in the DoFn and log the error; continue processing other elements.

Question 9

You are designing a near-real-time CDC pipeline to replicate changes from an on-premises PostgreSQL database to BigQuery for analytics. The source database has high transaction volume and you must ensure minimal impact on the source. Which Google Cloud service should you use to ingest the change data?

Accepted Answer

Datastream to stream changes to GCS, then load into BigQuery.. Datastream is purpose-built for CDC from MySQL, PostgreSQL, and Oracle to BigQuery or GCS. It reads the database logs (e.g., WAL) to capture changes with low latency and minimal impact on the source.

Answer

Pub/Sub with a custom connector that polls the database every minute.

Answer

Use BigQuery Data Transfer Service for PostgreSQL.

Answer

Use Dataflow with a JDBC IO connector to read from PostgreSQL.

Question 10

You are loading 10 GB of daily CSV files from a GCS bucket into a BigQuery table. The files contain some malformed rows that you want to skip. Which BigQuery load configuration should you use?

Accepted Answer

Use the 'max_bad_records' option set to a value like 10.. BigQuery allows setting max_bad_records in load jobs; records exceeding this threshold cause the job to fail. Setting max_bad_records to a value greater than 0 allows the load to succeed while skipping malformed rows.

Answer

Use the 'skip_leading_rows' option.

Answer

Use the 'ignore_unknown_values' option.

Answer

Use the 'allow_jagged_rows' option.

Question 11

Your Dataflow pipeline reads from Pub/Sub, performs transformations, and writes to BigQuery. You notice that the pipeline's autoscaling is not keeping up with sudden spikes in traffic, causing increased lag. The pipeline uses Classic Templates. Which change would most effectively improve autoscaling responsiveness?

Accepted Answer

Enable Dataflow Streaming Engine on the pipeline.. Enabling Dataflow Streaming Engine reduces the overhead of checkpointing and state management by offloading them to the service side, which allows the pipeline to scale more quickly in response to sudden traffic spikes. This directly addresses the autoscaling lag because Streaming Engine decouples compute from state, enabling faster worker adjustments without the bottleneck of persistent disk-based shuffle.

Answer

Switch to Dataflow Prime with Vertical Autoscaling enabled.

Answer

Increase the initial number of workers to handle the spike.

Answer

Use Flex Templates instead of Classic Templates.

Question 12

You are migrating an existing Kafka cluster to Google Cloud using Dataproc. The cluster handles high-throughput streaming data with strict ordering requirements per partition. Which choice of Dataproc configuration is most appropriate?

Accepted Answer

Use Dataproc with non-preemptible workers and persistent SSD storage for brokers.. Kafka brokers in a Dataproc cluster require persistent, non-preemptible workers to maintain data durability and strict ordering per partition. Preemptible workers can be terminated at any time, causing data loss or rebalancing that violates ordering guarantees. Persistent SSD storage provides the low-latency I/O needed for high-throughput Kafka workloads, while non-preemptible instances ensure broker stability and consistent replication.

Answer

Use Dataflow with Kafka IO instead of Dataproc.

Answer

Use Dataproc with local SSDs for better performance, and enable autoscaling.

Answer

Use Dataproc with preemptible workers to reduce cost, and attach standard persistent disks.

Question 13

Your team uses dbt to transform data in BigQuery. You need to schedule dbt runs to refresh materialized tables and views every hour. The transformations include both full refreshes and incremental models. What is the most efficient way to orchestrate these dbt runs on Google Cloud?

Accepted Answer

Use Cloud Composer (Airflow) to schedule and run dbt commands.. Cloud Composer (Airflow) is the recommended orchestration tool for complex workflows like dbt runs, supporting dependencies, retries, and scheduling. Cloud Scheduler alone cannot run dbt directly; it can trigger a Cloud Function to run dbt, but that is less maintainable. Cloud Build is CI/CD, not scheduling. Using a cron job on Compute Engine is possible but not managed.

Answer

Use Cloud Build with a trigger to run dbt every hour.

Answer

Use Cloud Scheduler to trigger a Cloud Function that runs dbt.

Answer

Set up a cron job on a Compute Engine instance to run dbt.

Question 14

You need to ingest Google Ads performance data into BigQuery on a daily basis for reporting. Which service should you use?

Accepted Answer

BigQuery Data Transfer Service for Google Ads. The BigQuery Data Transfer Service for Google Ads is the correct choice because it provides a fully managed, scheduled connector that automatically ingests Google Ads performance data into BigQuery on a daily basis without requiring any custom code. It handles authentication, schema mapping, and incremental loads, making it the simplest and most reliable solution for this specific use case.

Answer

Cloud Scheduler to call Google Ads API and load to BigQuery

Answer

Pub/Sub with a Google Ads subscriber

Answer

Storage Transfer Service for Google Ads

Question 15

You are designing a streaming pipeline that ingests events from Pub/Sub, enriches them with a machine learning model, and writes the results to BigQuery. The ML model is deployed on Cloud Run and has a high latency (500ms per request). You need to minimize the impact of slow ML inference on the overall pipeline throughput. Which approach should you take?

Accepted Answer

Use Dataflow to write events to Pub/Sub, then use a separate Dataflow pipeline that batches calls to Cloud Run.. It uses Dataflow to batch events before sending them to Cloud Run, which amortizes the 500ms per-request latency over multiple events, significantly increasing throughput. By writing events to Pub/Sub and then processing them in a separate Dataflow pipeline with batched calls, you decouple the ingestion from the inference and avoid blocking on each individual request.

Answer

Increase the number of Dataflow workers to compensate for the latency.

Answer

Use Cloud Functions to call Cloud Run and write directly to BigQuery.

Answer

Use Dataflow's ParDo with synchronous calls to Cloud Run for each element.