PDE Ingesting and Processing the Data Practice Test 8 — 15 Questions

Question 1

You are building a Dataflow pipeline that reads from Pub/Sub, applies transformations, and writes to BigQuery. The pipeline must handle late-arriving data and ensure that the windowing and triggering are correct. Which THREE configurations should you consider? (Choose 3)

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use the BigQuery Storage Write API with committed mode to ensure exactly-once writes.. Option C is correct because the BigQuery Storage Write API in committed mode (exactly-once semantics) uses write streams with offsets and client-side deduplication, which is the recommended way to guarantee exactly-once writes from a Dataflow streaming pipeline into BigQuery. Option D is correct because setting an allowed lateness duration (via withAllowedLateness) on the windowing strategy tells Dataflow how long to retain window state and keep accepting late-arriving elements after the watermark passes the window end, which is exactly the requirement for handling late data. Option E is correct because configuring a triggering frequency (for example, AfterProcessingTime.pastFirstElementInPane().plusDelayOf(...) or a repeated trigger) controls how often intermediate or final results are emitted from each window, which directly governs the windowing/triggering behavior described in the scenario. Option A does not belong because Dataflow Streaming Engine is a runner feature that offloads state and windowing execution to the service for scalability and cost benefits; it does not itself provide exactly-once processing, which comes from the pipeline's sources, sinks, and deduplication logic. Option B does not belong because side inputs are used to enrich streaming records with additional (often static or slowly changing) data, which is unrelated to handling late-arriving data or to windowing and triggering correctness.

Answer

Enable Dataflow Streaming Engine for exactly-once processing.

Answer

Use side inputs to enrich streaming data with static data.

Question 2

A data engineer needs to schedule recurring nightly loads from Amazon S3 to Google Cloud Storage. The data is in CSV format and the volume is approximately 500 GB per night. Which Google Cloud service should they use?

Accepted Answer

Storage Transfer Service. The Storage Transfer Service is designed for online data transfers from external cloud providers like Amazon S3 to Google Cloud Storage. It supports scheduling recurring nightly transfers, handles large volumes (500 GB/night), and automatically retries failed transfers, making it the correct choice for this use case.

Answer

Transfer Appliance

Answer

BigQuery Data Transfer Service

Answer

Datastream

Question 3

A company needs to load data from a MySQL database into BigQuery daily. The data volume is 10 GB per day and the schema changes occasionally. They want to minimize costs and operational overhead. What is the MOST appropriate approach?

Accepted Answer

Export MySQL data to CSV, upload to GCS, and use BigQuery load jobs. For a daily batch load of 10 GB with occasional schema changes, the most cost-effective and low-overhead approach is to export the MySQL data to CSV or Parquet, upload it to Cloud Storage, and use BigQuery load jobs. BigQuery load jobs are free (you only pay for storage and queries), and they can handle schema changes via schema autodetect or by updating the table schema. Datastream is designed for continuous change data capture with low latency, which is unnecessary and likely more expensive for a once-daily batch load. BigQuery Data Transfer Service does not support MySQL as a source, and Cloud SQL federated queries are for querying external data, not for loading it into BigQuery.

Answer

Use Datastream to stream changes from MySQL to BigQuery

Answer

Use BigQuery Data Transfer Service for MySQL

Answer

Use Cloud SQL federated query from BigQuery

Question 4

A media company streams real-time viewer data from Pub/Sub to BigQuery using a Dataflow pipeline. They need to handle occasional malformed messages without losing valid data. Which pattern should they implement?

Accepted Answer

Implement a dead letter sink to store malformed messages for later analysis. A dead letter sink (e.g., a separate Pub/Sub topic or a BigQuery error table) allows the Dataflow pipeline to route malformed messages out of the main processing path while continuing to process valid data. This pattern ensures no valid data is lost and provides a durable location for later analysis or reprocessing of the malformed records, which is essential for streaming pipelines where data quality issues are intermittent.

Answer

Raise an exception in the pipeline and stop processing

Answer

Use retry logic in the pipeline to reprocess malformed messages indefinitely

Answer

Discard malformed messages and log an error

Question 5

A team uses dbt on BigQuery to transform data in their data warehouse. They have a large table with nested and repeated fields (arrays and structs). The transformation needs to normalize this data into a star schema. Which dbt feature and BigQuery SQL feature should they use together?

Accepted Answer

dbt models with BigQuery UNNEST and CROSS JOIN. To normalize nested and repeated fields (arrays and structs) into a star schema, you need to flatten the arrays into separate rows. BigQuery's UNNEST operator, when used with CROSS JOIN, expands each array element into its own row, effectively denormalizing the nested structure. dbt models (SQL SELECT statements) are the correct dbt feature to define these transformations as version-controlled, reusable SQL files. Together, they allow you to write a dbt model that uses CROSS JOIN UNNEST to produce dimension and fact tables from a single nested table.

Answer

dbt hooks with BigQuery STRUCT access

Answer

dbt snapshots with BigQuery JSON functions

Answer

dbt seeds with BigQuery ARRAY_AGG

Question 6

A company wants to trigger a Cloud Run service whenever a new file is uploaded to a specific Cloud Storage bucket. Which event-driven solution should they use?

Accepted Answer

Eventarc with Cloud Storage trigger and Cloud Run destination. Eventarc is the recommended service for routing events from Cloud Storage to Cloud Run because it provides a fully managed, event-driven architecture with built-in filtering and retry logic. When a new file is uploaded, Cloud Storage emits a notification that Eventarc captures and delivers directly to the Cloud Run service as an HTTP request, enabling serverless processing without polling or additional infrastructure.

Answer

Cloud Scheduler to periodically poll the bucket

Answer

Cloud Functions triggered by Cloud Storage

Answer

Pub/Sub with a push subscription to Cloud Run

Question 7

A data engineer is using Apache Spark on Dataproc to process a large dataset. They need to perform complex aggregation and transformation with high performance. The dataset has a known schema and they want to take advantage of Catalyst optimizer. Which Spark API should they use?

Accepted Answer

DataFrames. DataFrames are the correct choice because they combine a declarative, structured API with the Catalyst optimizer, which builds and optimizes a logical plan before execution. With a known schema, DataFrames let Spark apply rule-based and cost-based optimizations such as predicate pushdown, column pruning, and join reordering. This delivers high performance for complex aggregations and transformations without sacrificing the structured API benefits.

Answer

Spark SQL only

Answer

Datasets

Answer

RDDs

Question 8

A company uses Workflows to orchestrate a series of Google Cloud services for data processing. They need to call an external HTTP API as part of the workflow and handle potential failures with retries. Which Workflows feature should they use?

Accepted Answer

Retry policy on the step. Workflows provides a built-in retry policy that can be configured on individual steps to automatically retry an HTTP call upon transient failures (e.g., 5xx server errors or network timeouts). This allows the workflow to handle external API failures without custom code, using exponential backoff and a maximum retry count.

Answer

Subworkflows

Answer

Parallel steps

Answer

Conditional steps

Question 9

A data engineer needs to load data from CSV files in Cloud Storage into BigQuery. The CSV files have a header row and some columns contain nested JSON strings. Which TWO methods can they use to load this data into BigQuery?

Accepted Answer

Use the Storage Write API to write rows from a custom application. Option D is correct because a BigQuery load job natively supports the CSV format, including a header row via the --skip_leading_rows parameter, and can ingest Cloud Storage files directly into a native BigQuery table. Option B is correct because the Storage Write API lets a custom application parse the CSV and nested JSON strings itself and then stream the resulting rows into BigQuery, giving full control over transformation during load. Option A is wrong because Datastream is a change data capture (CDC) and replication service for databases such as MySQL, PostgreSQL, Oracle, and SQL Server, not a CSV file loader. Option C is wrong because an external table (federated query) only queries data in place in Cloud Storage; it does not load the data into BigQuery. Option E is wrong because gsutil only copies objects between Cloud Storage locations and cannot write data into BigQuery tables.

Answer

Use Datastream to load CSV files

Answer

Create a federated query using an external table

Answer

Use gsutil to copy files into BigQuery

Question 10

Your team is processing a large dataset with Apache Beam on Dataflow. The pipeline sometimes fails due to transient errors when writing to a BigQuery sink. You need to ensure that failed records are not lost and can be reprocessed later without blocking the pipeline. What is the best approach?

Accepted Answer

Use a side output (e.g., via TupleTag) to write failed records to a dead letter sink (e.g., GCS or Pub/Sub) and continue processing the main output.. Using a dead letter pattern with a side output to write failed records to a GCS bucket (or Pub/Sub) allows the pipeline to continue processing healthy records while failed records are stored for later analysis and reprocessing.

Answer

Configure the pipeline to use at-least-once semantics and rely on Dataflow to retry the entire bundle.

Answer

Increase the number of workers to reduce the chance of transient errors.

Answer

Use a try-catch block in the DoFn and log the error; continue processing other elements.

Question 11

You are loading 10 GB of daily CSV files from a GCS bucket into a BigQuery table. The files contain some malformed rows that you want to skip. Which BigQuery load configuration should you use?

Accepted Answer

Use the 'max_bad_records' option set to a value like 10.. The 'max_bad_records' option allows you to specify the maximum number of bad records that BigQuery can ignore during a load job. Setting it to a value like 10 means that up to 10 malformed rows will be skipped; if more than 10 bad records are encountered, the job fails. This is the correct configuration to skip a limited number of malformed rows while ensuring data quality.

Answer

Use the 'skip_leading_rows' option.

Answer

Use the 'ignore_unknown_values' option.

Answer

Use the 'allow_jagged_rows' option.

Question 12

Your team uses dbt to transform data in BigQuery. You need to schedule dbt runs to refresh materialized tables and views every hour. The transformations include both full refreshes and incremental models. What is the most efficient way to orchestrate these dbt runs on Google Cloud?

Accepted Answer

Use Cloud Composer (Airflow) to schedule and run dbt commands.. Cloud Composer (Airflow) is the recommended orchestration tool for complex workflows like dbt runs, supporting dependencies, retries, and scheduling. Cloud Scheduler alone cannot run dbt directly; it can trigger a Cloud Function to run dbt, but that is less maintainable. Cloud Build is CI/CD, not scheduling. Using a cron job on Compute Engine is possible but not managed.

Answer

Use Cloud Build with a trigger to run dbt every hour.

Answer

Use Cloud Scheduler to trigger a Cloud Function that runs dbt.

Answer

Set up a cron job on a Compute Engine instance to run dbt.

Question 13

You need to ingest Google Ads performance data into BigQuery on a daily basis for reporting. Which service should you use?

Accepted Answer

BigQuery Data Transfer Service for Google Ads. The BigQuery Data Transfer Service for Google Ads is the correct choice because it provides a fully managed, scheduled connector that automatically ingests Google Ads performance data into BigQuery on a daily basis without requiring any custom code. It handles authentication, schema mapping, and incremental loads, making it the simplest and most reliable solution for this specific use case.

Answer

Cloud Scheduler to call Google Ads API and load to BigQuery

Answer

Pub/Sub with a Google Ads subscriber

Answer

Storage Transfer Service for Google Ads

Question 14

You need to react to changes in a GCS bucket (e.g., new object creation) and trigger a Cloud Run service to process the new file. Which Google Cloud service should you use to route the event?

Accepted Answer

Eventarc. Eventarc is the correct choice because it is purpose-built to route events from Google Cloud sources (like Cloud Storage) to Cloud Run. It directly supports Cloud Storage audit logs and Pub/Sub event triggers, allowing you to react to object creation events without custom middleware. Eventarc handles the event routing, filtering, and delivery to your Cloud Run service automatically.

Answer

Pub/Sub directly with a Cloud Run subscription

Answer

Cloud Tasks

Answer

Cloud Scheduler

Question 15

You need to ingest streaming data from a custom application into BigQuery with exactly-once semantics and low latency. The data volume is up to 10 MB/s. Which TWO services should you combine?

Accepted Answer

Pub/Sub. Option A (Pub/Sub) is correct because it provides a highly scalable, low-latency messaging service that decouples the custom application from downstream processing, ingesting up to 10 MB/s and beyond while buffering the stream reliably. Option D (Dataflow with Storage Write API) is correct because Dataflow can read from Pub/Sub and write to BigQuery using the Storage Write API, which supports exactly-once semantics via its exactly-once delivery mode and provides low-latency, high-throughput ingestion. Together, Pub/Sub plus Dataflow with the Storage Write API form the recommended architecture for exactly-once streaming ingestion into BigQuery. Option B (Cloud Functions) is not appropriate here because it is event-driven and not designed for sustained high-throughput streaming pipelines with exactly-once guarantees. Option C (BigQuery legacy streaming inserts) does not provide exactly-once semantics and is being deprecated in favor of the Storage Write API. Option E (Datastream) is a change data capture (CDC) and replication service for databases, not a general-purpose streaming ingestion path for custom application data.

Answer

Cloud Functions

Answer

BigQuery legacy streaming inserts

Answer

Datastream