PDE Ingesting and Processing the Data Practice Test 11 — 15 Questions

Question 1

A data engineer is designing a batch processing pipeline that runs daily. The pipeline reads CSV files from GCS, transforms them using Python, and writes the results to BigQuery. They need to parameterize the pipeline for different environments and run it on a schedule. Which THREE components should they use? (Choose 3)

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Cloud Composer. Cloud Composer (A) is correct because it is a managed Apache Airflow service that provides DAG-based scheduling and parameterization, making it ideal for orchestrating a daily batch pipeline with environment-specific variables. Dataproc (B) is correct because it offers managed Spark/Hadoop clusters that can run Python-based transformations over the CSV data read from GCS at batch scale. Dataflow Flex Template (C) is correct because it packages a Dataflow pipeline (including Python transforms) into a reusable, parameterized template that can be invoked with different runtime parameters per environment. Storage Transfer Service (D) is not appropriate here because it only moves data between storage systems and performs no transformation or scheduling logic. Cloud Functions (E) is unsuitable because it is an event-driven, short-lived serverless function, not designed for orchestrating daily batch pipelines or large-scale data transformation.

Answer

Storage Transfer Service

Answer

Cloud Functions

Question 2

A company uses Workflows to orchestrate a multi-step data pipeline. One step calls an HTTP endpoint that may take up to 10 minutes, but the default Workflows timeout is too short. They also need to handle transient errors with retries. Which TWO configurations should they apply? (Choose 2)

Accepted Answer

Set a step timeout of 600 seconds for the HTTP call step. Option A is correct because Workflows steps support a per-step timeout, and setting it to 600 seconds (10 minutes) ensures the HTTP call step is allowed to run for the full duration instead of being cut off by the shorter default step timeout. Option E is correct because transient errors (such as 5xx responses or connection resets) should be handled by attaching a retry policy to the step with conditions that match those transient failures, so the step is retried automatically. Option B is not appropriate because Workflows does not use dead letter queues for failed steps; failures are surfaced through execution errors and logs. Option C is wrong because relying on the default retry policy does not let them target transient errors with appropriate conditions, and the default may not retry at all. Option D is wrong because setting the workflow execution timeout to 600 seconds would cap the entire workflow at 10 minutes, which is not the same as giving the HTTP step enough time and could still cause premature termination.

Answer

Configure a dead letter queue for failed steps

Answer

Use the default retry policy on the step

Answer

Set the workflow execution timeout to 600 seconds

Question 3

A company wants to ingest data from an on-premises Oracle database into BigQuery in near real-time with minimal latency. The database has a high volume of inserts and updates. Which service should they use?

Accepted Answer

Datastream. Datastream is correct because it is a serverless change data capture (CDC) service that replicates data from Oracle to BigQuery in near real-time with minimal latency. It reads the Oracle redo logs (via LogMiner or binary log reader) to capture inserts, updates, and deletes and streams them to BigQuery.

Answer

BigQuery Data Transfer Service

Answer

Pub/Sub

Answer

Storage Transfer Service

Question 4

A data engineer needs to load a 10 GB CSV file from GCS into BigQuery. The file contains some malformed rows that should be skipped. Which approach is most efficient?

Accepted Answer

Use the bq command-line tool with the --max_bad_records flag. The `bq` command-line tool's `--max_bad_records` flag allows BigQuery's native CSV loader to skip malformed rows up to a specified limit during a load job. This is the most efficient approach for a one-time batch load of a 10 GB file, as it avoids the overhead of spinning up separate processing clusters (Dataproc, Dataflow) or streaming each row individually, leveraging BigQuery's optimized ingestion pipeline.

Answer

Use Dataproc to run a Spark job that cleans the data and writes to BigQuery

Answer

Use the Storage Write API to stream each row, skipping bad ones in code

Answer

Use a Dataflow pipeline to read CSV, filter bad rows, and write to BigQuery

Question 5

A team wants to transfer data from an on-premises Hadoop cluster to Cloud Storage for processing. The cluster is located in a remote area with limited bandwidth. They need to transfer 500 TB of data. Which service should they use?

Accepted Answer

Transfer Appliance. Transfer Appliance is designed for petabyte-scale offline transfers when bandwidth is limited.

Answer

BigQuery Data Transfer Service

Answer

Storage Transfer Service

Answer

Dataproc with gsutil

Question 6

A company uses Google Ads and wants to automatically load their advertising data into BigQuery daily. They also need to transform the data with SQL and schedule a recurring query. Which combination of services meets these requirements with minimal operational overhead?

Accepted Answer

BigQuery Data Transfer Service for Google Ads and scheduled queries. BigQuery Data Transfer Service has a built-in Google Ads connector that automatically loads advertising data into BigQuery on a schedule, and BigQuery scheduled queries let you transform that data with SQL on a recurring basis — both fully managed with no infrastructure. This combination meets the daily load and SQL transformation requirements with minimal operational overhead.

Answer

Cloud Functions triggered by Cloud Scheduler to call Google Ads API and load into BigQuery

Answer

Cloud Composer to extract Google Ads API and Dataflow to transform

Answer

Storage Transfer Service to move CSV files to GCS, then load into BigQuery

Question 7

A team needs to orchestrate a multi-step workflow that involves calling external APIs, running BigQuery queries, and conditionally executing Cloud Functions. Which Google Cloud service is best suited for this?

Accepted Answer

Workflows. Workflows is a serverless orchestration service that allows you to define multi-step workflows as a sequence of steps, including HTTP calls to external APIs, BigQuery queries, and conditional logic to invoke Cloud Functions. It integrates natively with other Google Cloud services via the Workflows API and supports error handling, retries, and parallel steps, making it ideal for this use case.

Answer

Dataflow

Answer

Cloud Composer

Answer

Cloud Scheduler

Question 8

A company uses dbt on BigQuery to transform data. They want to run dbt models on a schedule and manage environments (dev, prod). Which GCP service should they use to run dbt jobs?

Accepted Answer

Cloud Composer. Cloud Composer is an Apache Airflow managed service that can schedule dbt runs.

Answer

Dataflow

Answer

Cloud Scheduler

Answer

Cloud Build

Question 9

A company wants to use Eventarc to trigger a Cloud Run service when new objects are created in a GCS bucket. They also need to filter events for a specific bucket and object prefix. Which THREE resources must exist or be created?

Accepted Answer

Cloud Storage bucket. Option A (Cloud Storage bucket) is correct because Eventarc's Cloud Storage events originate from an actual GCS bucket, and the trigger's bucket filter must reference that existing bucket where objects are created. Option D (Eventarc trigger) is correct because the trigger is the resource that connects the event source to the destination, defines the event type (e.g., google.cloud.storage.object.v1.finalized), and applies the bucket and object-prefix attribute filters. Option E (Cloud Run service) is correct because it is the event destination that Eventarc invokes when a matching object-creation event fires. Option B (Pub/Sub topic) is not required because Eventarc manages its own internal Pub/Sub transport for Cloud Storage events; you do not create or specify a topic. Option C (Cloud Scheduler job) is not required because scheduling is unrelated to event-driven object-creation triggers.

Answer

Pub/Sub topic

Answer

Cloud Scheduler job

Question 10

A financial services company receives real-time stock trade data via Pub/Sub. They need to enrich each trade with reference data from a Cloud SQL table and write the results to BigQuery for real-time analytics. The enrichment must handle late-arriving data and ensure exactly-once processing. Which Dataflow streaming pipeline configuration should be used?

Accepted Answer

Build a custom Dataflow pipeline using the Storage Write API with exactly-once semantics and a side input from Cloud SQL. The BigQuery Storage Write API with exactly-once semantics is the only option that satisfies both the exactly-once requirement and the need for enrichment with Cloud SQL reference data. A custom Dataflow pipeline can use the Storage Write API's exactly-once mode while applying a side input (or CoGroupByKey) to join the streaming trades with the Cloud SQL reference table. This combination handles late-arriving data via allowed lateness and windowing while guaranteeing no duplicate writes to BigQuery.

Answer

Use a Dataflow Flex Template that reads from Pub/Sub, joins in memory, and writes to BigQuery using legacy streaming inserts

Answer

Use Pub/Sub to BigQuery template with streaming inserts and a side input from Cloud SQL

Answer

Deploy a Dataproc Spark Streaming job that reads from Pub/Sub, enriches via JDBC, and writes to BigQuery

Question 11

A company needs to continuously synchronize customer data changes from an on-premises Oracle database to BigQuery for near-real-time analytics. The Oracle database has Change Data Capture (CDC) enabled. Which Google Cloud service should be used to stream these changes with minimal latency and schema evolution support?

Accepted Answer

Use Datastream to stream CDC changes from Oracle to BigQuery. Datastream is Google Cloud's fully managed, serverless CDC and replication service that natively supports Oracle sources (via LogMiner or XStream) and BigQuery destinations. It streams changes with low latency, handles schema evolution automatically, and requires no pipeline code to maintain. This matches the requirement for minimal latency and schema evolution support.

Answer

Deploy a Dataflow pipeline with a JDBC source and Pub/Sub

Answer

Use Cloud SQL with a read replica and enable binary logging

Answer

Use Transfer Appliance to copy Oracle data periodically

Question 12

A data engineer needs to orchestrate a series of tasks that include calling external APIs, running BigQuery queries, and sending notifications. The workflow involves conditional branching and parallel steps. Which Google Cloud service should be used?

Accepted Answer

Workflows. Workflows is the correct choice because it is a fully managed orchestration service designed specifically for coordinating multi-step, event-driven workflows that involve conditional branching, parallel execution, and integration with external APIs, BigQuery, and notifications via HTTP calls or service integrations. It provides built-in error handling, retries, and a declarative YAML-based syntax that directly supports the described requirements without needing to manage infrastructure or schedule tasks.

Answer

Cloud Scheduler

Answer

Cloud Composer

Answer

Dataflow

Question 13

A company wants to move data from an on-premises MySQL database to BigQuery for analytics. They need to capture all changes (inserts, updates, deletes) in near real-time and also perform an initial historical load. Which approach meets these requirements with minimal operational overhead?

Accepted Answer

Use Datastream to backfill historical data and then stream CDC changes to BigQuery. Datastream is correct because it provides serverless change data capture (CDC) from MySQL to BigQuery, supporting both historical backfill and continuous replication of inserts, updates, and deletes with minimal operational overhead. It reads the MySQL binary log (binlog) to stream changes in near real-time and can write directly to BigQuery or Cloud Storage.

Answer

Use a Dataflow pipeline with a JDBC source to read the entire table periodically

Answer

Use a one-time export to CSV and load into BigQuery, then set up a cron job to export incremental changes

Answer

Use Cloud SQL as an intermediary and enable binary logging, then stream to Pub/Sub via a custom connector

Question 14

A company is building a data pipeline that ingests streaming data from Pub/Sub, transforms it with Dataflow, and loads it into BigQuery. They want to handle malformed messages that cannot be parsed. Which TWO actions should they implement for error handling? (Choose 2)

Accepted Answer

Use a dead letter sink to write malformed messages to Cloud Storage or Pub/Sub for later analysis. Option C is correct because a dead letter sink (often implemented in Dataflow via a tagged output or a separate Pub/Sub topic/Cloud Storage path) captures unparseable records so they can be inspected and reprocessed without losing data or halting the pipeline. Option E is correct because logging the parsing error and continuing lets the pipeline keep processing valid messages, which is essential for a resilient streaming pipeline where one bad record should not stop the flow. Option A is not appropriate because silently dropping malformed messages loses data and provides no visibility for debugging or remediation. Option B is not the right pattern because a side input is used to supply supplementary data to a DoFn, not to filter or quarantine malformed records. Option D is wrong because raising an exception in the DoFn would fail the pipeline immediately, causing unnecessary downtime and blocking valid messages from being processed.

Answer

Configure the pipeline to drop malformed messages silently

Answer

Use a side input to filter out malformed messages

Answer

Raise an exception in the DoFn to fail the pipeline immediately

Question 15

A data engineer needs to schedule a recurring transfer of data from a partner's Amazon S3 bucket to a Cloud Storage bucket for further processing. Which THREE components or configurations are necessary? (Choose 3)

Accepted Answer

Specification of the source S3 bucket and destination GCS bucket. Storage Transfer Service is the Google Cloud service designed to move data from external sources like Amazon S3 into Cloud Storage, so the transfer job must be defined as a scheduled transfer job (option C), which controls the recurring execution. Every transfer job requires the source and destination locations to be identified, so specifying the source S3 bucket and destination GCS bucket (option B) is mandatory for the job to know what to copy and where to place it. Because the source is an external AWS S3 bucket, Storage Transfer Service must authenticate to AWS, which requires AWS credentials such as an access key ID and secret access key (option E). A VPC network configuration (option A) is not required because Storage Transfer Service is a fully managed service that does not need customer VPC networking for S3-to-GCS transfers. A Pub/Sub topic (option D) is optional, used only if you want notifications about transfer completion, not a necessary component for the transfer itself.

Answer

A VPC network configuration

Answer

A Pub/Sub topic to notify completion