PDE Ingesting and Processing the Data Practice Test 7 — 15 Questions

Question 1

A company wants to ingest data from an on-premises Oracle database into BigQuery in near real-time with minimal latency. The database has a high volume of inserts and updates. Which service should they use?

Accepted Answer

Datastream. Datastream is designed for CDC from Oracle and other sources to BigQuery or GCS in near real-time.

Answer

BigQuery Data Transfer Service

Answer

Pub/Sub

Answer

Storage Transfer Service

Question 2

A data engineer needs to load a 10 GB CSV file from GCS into BigQuery. The file contains some malformed rows that should be skipped. Which approach is most efficient?

Accepted Answer

Use the bq command-line tool with the --max_bad_records flag. Option D is correct because the `bq` command-line tool's `--max_bad_records` flag allows BigQuery's native CSV loader to skip malformed rows up to a specified limit during a load job. This is the most efficient approach for a one-time batch load of a 10 GB file, as it avoids the overhead of spinning up separate processing clusters (Dataproc, Dataflow) or streaming each row individually, leveraging BigQuery's optimized ingestion pipeline.

Answer

Use Dataproc to run a Spark job that cleans the data and writes to BigQuery

Answer

Use the Storage Write API to stream each row, skipping bad ones in code

Answer

Use a Dataflow pipeline to read CSV, filter bad rows, and write to BigQuery

Question 3

A team wants to transfer data from an on-premises Hadoop cluster to Cloud Storage for processing. The cluster is located in a remote area with limited bandwidth. They need to transfer 500 TB of data. Which service should they use?

Accepted Answer

Transfer Appliance. Transfer Appliance is designed for petabyte-scale offline transfers when bandwidth is limited.

Answer

BigQuery Data Transfer Service

Answer

Storage Transfer Service

Answer

Dataproc with gsutil

Question 4

A data pipeline uses Pub/Sub to ingest events, a Dataflow streaming pipeline to process them, and writes results to BigQuery. The pipeline must handle occasional duplicate events without causing duplicate rows in BigQuery. What is the best approach?

Accepted Answer

Use BigQuery legacy streaming inserts with insertId for deduplication. BigQuery does not enforce primary keys; deduplication must be handled in the pipeline using idempotent writes or a dedup step.

Answer

Use the Storage Write API with the committed stream

Answer

Set a unique constraint on the BigQuery table

Answer

Enable exactly-once processing in Pub/Sub

Question 5

A company uses Google Ads and wants to automatically load their advertising data into BigQuery daily. They also need to transform the data with SQL and schedule a recurring query. Which combination of services meets these requirements with minimal operational overhead?

Accepted Answer

BigQuery Data Transfer Service for Google Ads and scheduled queries. BigQuery Data Transfer Service can automatically load Google Ads data; scheduled queries handle transformation.

Answer

Cloud Functions triggered by Cloud Scheduler to call Google Ads API and load into BigQuery

Answer

Cloud Composer to extract Google Ads API and Dataflow to transform

Answer

Storage Transfer Service to move CSV files to GCS, then load into BigQuery

Question 6

A data engineer needs to create a Dataflow pipeline that reads from Pub/Sub, applies a Python transformation, and writes to BigQuery. The pipeline should be reusable across environments with different parameters. Which deployment method is most appropriate?

Accepted Answer

Flex Template. Flex Templates (Option B) are the most appropriate deployment method because they allow you to package a custom Docker image containing your Python transformation code and dependencies, making the pipeline reusable across environments with different runtime parameters. Unlike Classic Templates, Flex Templates support arbitrary pipeline code and can be parameterized at runtime via the Dataflow UI or API, which is essential for a multi-environment deployment strategy.

Answer

Classic Template

Answer

Direct pipeline submission with gcloud dataflow jobs run

Answer

Cloud Composer to trigger Dataflow jobs

Question 7

A company uses Kafka on Dataproc to ingest streaming data. They want to process the data with Spark Structured Streaming and write results to BigQuery. The team is using Dataproc clusters. Which approach minimizes cost while maintaining performance?

Accepted Answer

Use a Dataproc cluster with standard master nodes and preemptible worker nodes. Preemptible VMs are cost-effective for worker nodes; master nodes should be standard for reliability.

Answer

Use a Dataproc cluster with all preemptible VMs

Answer

Use a single-node Dataproc cluster

Answer

Use a Dataproc cluster with standard nodes and enable autoscaling

Question 8

Which BigQuery feature allows you to query data directly from Cloud Storage without loading it into BigQuery storage?

Accepted Answer

External tables. BigQuery Omni is for multi-cloud, not external tables. External tables allow querying data in GCS.

Answer

BigQuery Omni

Answer

BigQuery ML

Answer

Federated queries

Question 9

A company wants to build an event-driven application that processes images uploaded to a Cloud Storage bucket. The processing takes up to 10 minutes per image and should be automatically triggered. Which compute option should they use?

Accepted Answer

Cloud Run on Eventarc trigger. Cloud Functions have a 9-minute timeout; Cloud Run can handle up to 60 minutes and is triggered by Eventarc for GCS events.

Answer

Cloud Functions (2nd gen) with Eventarc trigger

Answer

App Engine

Answer

Cloud Functions (1st gen)

Question 10

A Dataflow streaming pipeline is experiencing high latency and frequent OOM errors when processing variable-sized JSON messages from Pub/Sub. The team suspects that the autoscaling is not effective. Which feature should they enable to improve resource utilization?

Accepted Answer

Dataflow Prime. Dataflow Prime is the correct choice because it provides intelligent resource management that automatically adjusts worker resources (CPU, memory) based on the pipeline's processing demands, which is critical for variable-sized JSON messages. It addresses both high latency and OOM errors by optimizing resource utilization beyond simple autoscaling, including predictive autoscaling and flexible resource scheduling to handle spikes in message size without manual tuning.

Answer

Horizontal autoscaling

Answer

FlexRS

Answer

Streaming Engine

Question 11

A team needs to orchestrate a multi-step workflow that involves calling external APIs, running BigQuery queries, and conditionally executing Cloud Functions. Which Google Cloud service is best suited for this?

Accepted Answer

Workflows. Workflows is a serverless orchestration service that allows you to define multi-step workflows as a sequence of steps, including HTTP calls to external APIs, BigQuery queries, and conditional logic to invoke Cloud Functions. It integrates natively with other Google Cloud services via the Workflows API and supports error handling, retries, and parallel steps, making it ideal for this use case.

Answer

Dataflow

Answer

Cloud Composer

Answer

Cloud Scheduler

Question 12

A company uses dbt on BigQuery to transform data. They want to run dbt models on a schedule and manage environments (dev, prod). Which GCP service should they use to run dbt jobs?

Accepted Answer

Cloud Composer. Cloud Composer is an Apache Airflow managed service that can schedule dbt runs.

Answer

Dataflow

Answer

Cloud Scheduler

Answer

Cloud Build

Question 13

A company needs to stream data from a MySQL database to BigQuery with a latency under 10 seconds. They also need to handle schema changes automatically. Which TWO services should they combine?

Accepted Answer

BigQuery. Datastream captures CDC and can write to BigQuery directly. Pub/Sub is not needed if Datastream writes directly.

Answer

Dataflow

Answer

Pub/Sub

Answer

Cloud SQL

Question 14

A data team needs to transfer 200 TB of data from Amazon S3 to GCS. The transfer must be incremental, and they need to monitor the transfer progress. Which THREE components should they use?

Accepted Answer

Cloud Monitoring. Storage Transfer Service (STS) can transfer from S3 to GCS with incremental sync. Cloud Monitoring tracks progress. Service account for permissions.

Answer

Dataflow

Answer

Transfer Appliance

Question 15

A company wants to use Eventarc to trigger a Cloud Run service when new objects are created in a GCS bucket. They also need to filter events for a specific bucket and object prefix. Which THREE resources must exist or be created?

Accepted Answer

Cloud Storage bucket. Eventarc trigger, Cloud Run service, and the GCS bucket. The trigger references the bucket and prefix.

Answer

Pub/Sub topic

Answer

Cloud Scheduler job