PDE Ingesting and Processing the Data Practice Test 2 — 15 Questions

Question 1

A company needs to run a Spark ML training job on a Dataproc cluster with high memory per node, but the cluster should automatically scale down when idle to save costs. Which configuration should they use?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use a Dataproc cluster with preemptible secondary workers and cluster autoscaling. It combines preemptible secondary workers for cost-effective high-memory compute with cluster autoscaling, which automatically scales down the cluster when idle. Preemptible VMs are ideal for stateless Spark ML training tasks, and autoscaling ensures the cluster shrinks to save costs during inactivity. This configuration meets the requirement of high memory per node (via primary workers) while minimizing costs through idle scaling.

Answer

Use a single-node cluster with preemptible VMs

Answer

Enable Dataproc's default autoscaling with primary workers as preemptible

Answer

Create a cluster with custom machine types and no autoscaling

Question 2

Which BigQuery feature allows you to write data with exactly-once semantics, high throughput, and the ability to buffer data before making it available for queries?

Accepted Answer

Storage Write API with buffered mode. The Storage Write API with buffered mode (option D) is correct because it provides exactly-once semantics for data ingestion, high throughput via gRPC streaming, and the ability to buffer data in memory before making it available for queries. This mode allows you to commit rows in a stream, ensuring no duplicates, while the buffering stage gives you control over when data becomes visible in BigQuery.

Answer

BigQuery load jobs

Answer

BigQuery Data Transfer Service

Answer

Legacy streaming inserts

Question 3

A company wants to transform data using dbt (data build tool) on BigQuery. They have a CI/CD pipeline and need to version-control their transformations. Which setup is recommended?

Accepted Answer

Deploy dbt models in a Cloud Build pipeline that runs dbt run. Dbt is designed for version-controlled, SQL-based transformations, and integrating it with Cloud Build allows you to run `dbt run` as part of a CI/CD pipeline. This setup ensures that every change to dbt models is automatically tested and deployed, which aligns with the requirement for version control and automated deployment on BigQuery.

Answer

Create Dataflow pipelines for each transformation

Answer

Use Cloud Composer to orchestrate dbt jobs

Answer

Run dbt directly on BigQuery using scripting

Question 4

Which three of the following are valid BigQuery data loading methods? (Choose THREE.)

Accepted Answer

Data Transfer Service from Amazon S3. The BigQuery Data Transfer Service supports automated ingestion from Amazon S3, allowing you to schedule and manage recurring transfers of data stored in S3 buckets directly into BigQuery. This is a fully managed service that handles the extraction, transformation, and loading (ETL) process without requiring custom code or infrastructure.

Answer

Using Cloud SQL to write to BigQuery

Answer

Direct file upload via Cloud Dataproc

Question 5

A data engineer is building a Dataflow pipeline in Python that reads from BigQuery, transforms data, and writes to Cloud Storage. The pipeline will be deployed in production. Which approach should they use to ensure the pipeline is reusable across environments with different configuration parameters?

Accepted Answer

Use a Dataflow Flex Template. Dataflow Flex Templates allow you to package a Docker container with your pipeline code and dependencies, enabling parameterization at runtime via the Dataflow UI, CLI, or API. This makes the pipeline reusable across environments (e.g., dev, staging, prod) by passing different configuration parameters (like project IDs, table names, or output paths) without modifying the code. Flex Templates support custom container images and are the recommended approach for production pipelines that need environment-agnostic deployment.

Answer

Create a separate pipeline for each environment with hardcoded values

Answer

Use a Dataflow Classic Template

Answer

Run the pipeline using the DirectRunner for each environment

Question 6

A data team wants to load millions of small JSON files (each <1 MB) from GCS into BigQuery daily with the lowest cost and fastest performance. They need exactly-once semantics and the ability to detect new files automatically. Which approach is most suitable?

Accepted Answer

Use Dataflow to read files from GCS, combine them, and write to BigQuery using the Storage Write API in exactly-once mode. BigQuery batch load jobs with wildcard URIs can load many small files, but each load job creates a slot reservation and has overhead, making it slower and potentially more expensive for millions of files. Dataflow can read the files from GCS, combine them into larger batches, and write to BigQuery using the Storage Write API. The Storage Write API supports both streaming and batch ingestion with exactly-once semantics, and when used from Dataflow in a batch pipeline, it provides high-throughput, low-latency writing with automatic deduplication. Additionally, Dataflow can detect new files automatically via file triggers or a streaming source like Pub/Sub notifications. This combination minimizes cost and maximizes performance.

Answer

Use Cloud Functions to trigger on new files and stream each row via the legacy streaming inserts API

Answer

Use Storage Transfer Service to copy files to a staging bucket, then run a scheduled query to load them

Answer

Use BigQuery batch load jobs with a wildcard URI to load all files directly

Question 7

A data engineer is designing a Dataflow pipeline that reads from Pub/Sub and writes to BigQuery using the Storage Write API in exactly-once mode. The pipeline must handle late-arriving data (up to 1 hour) and maintain correct aggregation results. Which trigger configuration should they use?

Accepted Answer

After watermark trigger with allowed lateness of 1 hour. The After watermark trigger with allowed lateness of 1 hour ensures that the pipeline waits for the watermark to pass before emitting a pane, while allowing late-arriving data up to 1 hour to be included in the correct aggregation window. This is required for exactly-once processing with the Storage Write API, as it prevents duplicate or dropped records from late data.

Answer

Default trigger without allowed lateness

Answer

After count trigger with count of 1000

Answer

Fixed time trigger every 5 minutes

Question 8

A company uses Eventarc to trigger a Cloud Run service when new objects appear in a GCS bucket. Recently, the Cloud Run service has been failing with 429 errors (too many requests) during high-velocity uploads. They need to handle the load without losing events. What should they do?

Accepted Answer

Configure the Eventarc trigger to send events to a Pub/Sub topic, and have the Cloud Run service pull from Pub/Sub. Sending events to a Pub/Sub topic decouples event production from consumption. Pub/Sub acts as a buffer that can absorb spikes in event volume, and the Cloud Run service can pull messages at its own pace, preventing 429 errors. This also ensures no events are lost, as Pub/Sub retains unacknowledged messages and retries delivery.

Answer

Use Cloud Functions instead of Cloud Run

Answer

Increase the maximum number of retries on the Eventarc trigger

Answer

Increase the Cloud Run service's request timeout

Question 9

A company needs to stream real-time user activity data from their application into BigQuery for immediate dashboarding. They want to minimize latency (under 5 seconds) and ensure exactly-once delivery. Which TWO options should they consider? (Choose 2)

Accepted Answer

Use BigQuery Storage Write API in committed mode. The BigQuery Storage Write API in committed mode provides exactly-once delivery semantics and low-latency streaming (typically under 5 seconds) directly into BigQuery. It is designed for real-time ingestion with strong consistency guarantees, making it ideal for immediate dashboarding.

Answer

Use Cloud Functions to receive events and call the BigQuery REST API

Answer

Use BigQuery legacy streaming inserts directly from the application

Answer

Use Apache Kafka on Dataproc and write to BigQuery via the BigQuery Kafka connector

Question 10

A data engineer is designing a batch processing pipeline that runs daily. The pipeline reads CSV files from GCS, transforms them using Python, and writes the results to BigQuery. They need to parameterize the pipeline for different environments and run it on a schedule. Which THREE components should they use? (Choose 3)

Accepted Answer

Cloud Composer. Cloud Composer (A) is correct because it provides managed orchestration, scheduling, and parameterization via Airflow DAGs. Dataflow Flex Templates (C) are correct because they enable parameterized Dataflow pipelines that can be triggered by Composer. Dataproc (B) is also correct because it can run Python-based batch transforms on a cluster, and Composer can schedule and parameterize Dataproc jobs. Thus, all three together cover scheduling, processing, and parameterization.

Answer

Storage Transfer Service

Answer

Cloud Functions

Question 11

A data engineer needs to load a 10 GB CSV file from GCS into BigQuery. The file contains some malformed rows that should be skipped. Which approach is most efficient?

Accepted Answer

Use the bq command-line tool with the --max_bad_records flag. The `bq` command-line tool's `--max_bad_records` flag allows BigQuery's native CSV loader to skip malformed rows up to a specified limit during a load job. This is the most efficient approach for a one-time batch load of a 10 GB file, as it avoids the overhead of spinning up separate processing clusters (Dataproc, Dataflow) or streaming each row individually, leveraging BigQuery's optimized ingestion pipeline.

Answer

Use Dataproc to run a Spark job that cleans the data and writes to BigQuery

Answer

Use the Storage Write API to stream each row, skipping bad ones in code

Answer

Use a Dataflow pipeline to read CSV, filter bad rows, and write to BigQuery

Question 12

A team needs to orchestrate a multi-step workflow that involves calling external APIs, running BigQuery queries, and conditionally executing Cloud Functions. Which Google Cloud service is best suited for this?

Accepted Answer

Workflows. Workflows is a serverless orchestration service that allows you to define multi-step workflows as a sequence of steps, including HTTP calls to external APIs, BigQuery queries, and conditional logic to invoke Cloud Functions. It integrates natively with other Google Cloud services via the Workflows API and supports error handling, retries, and parallel steps, making it ideal for this use case.

Answer

Dataflow

Answer

Cloud Composer

Answer

Cloud Scheduler

Question 13

A company uses Pub/Sub to ingest clickstream data. Each message contains a JSON payload with a nested array of user actions. The data must be written to BigQuery, with each action in the array becoming a separate row. Which BigQuery feature or approach should be used to achieve this transformation?

Accepted Answer

Use a Dataflow pipeline with a ParDo that explodes the array. To write each action as a separate row in BigQuery, the nested array must be flattened during the data pipeline, not at query time. A Dataflow pipeline with a ParDo function that iterates over the array and outputs one element per action achieves this transformation before writing to BigQuery. Loading the JSON as-is and using UNNEST in a query only flattens at read time, not write time, so it does not meet the requirement.

Answer

Load the JSON as-is into BigQuery and use UNNEST in a query

Answer

Use a BigQuery scripting loop to iterate over the array

Answer

Preprocess the data with a Dataflow pipeline and write to BigQuery

Question 14

A data engineer needs to orchestrate a series of tasks that include calling external APIs, running BigQuery queries, and sending notifications. The workflow involves conditional branching and parallel steps. Which Google Cloud service should be used?

Accepted Answer

Workflows. Workflows is the correct choice because it is a fully managed orchestration service designed specifically for coordinating multi-step, event-driven workflows that involve conditional branching, parallel execution, and integration with external APIs, BigQuery, and notifications via HTTP calls or service integrations. It provides built-in error handling, retries, and a declarative YAML-based syntax that directly supports the described requirements without needing to manage infrastructure or schedule tasks.

Answer

Cloud Scheduler

Answer

Cloud Composer

Answer

Dataflow

Question 15

A company is migrating a Spark batch job from on-premises to Dataproc. The job uses RDDs for custom transformations and writes output to BigQuery. They want to optimize the job for performance and cost on Dataproc. Which THREE practices should they adopt? (Choose 3)

Accepted Answer

Write output to BigQuery using the BigQuery connector. The BigQuery connector for Spark (com.google.cloud.spark.bigquery) allows direct, efficient writes to BigQuery without intermediate storage. It leverages the BigQuery Storage Write API for high-throughput, exactly-once delivery, which is far more performant than writing via JDBC or saving to GCS and then loading into BigQuery.

Answer

Use PySpark instead of Scala for simplicity

Answer

Increase the number of partitions to match the number of cores exactly