PDE Ingesting and Processing the Data Practice Test 7 — 15 Questions

Question 1

A company runs Apache Kafka on Dataproc for real-time event streaming. They want to archive the Kafka topics to Cloud Storage for long-term retention and later analysis in BigQuery. Which approach is the most cost-effective and operationally simple?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use Kafka Connect with the GCS Sink Connector to write directly to Cloud Storage. Kafka Connect with the GCS Sink Connector is purpose-built for exactly this use case: it directly streams Kafka topics to Cloud Storage in Avro, Parquet, or JSON format without requiring intermediate processing clusters or services. This approach minimizes operational overhead (no Spark or Dataflow jobs to manage) and is cost-effective since it runs as a lightweight connector within the existing Kafka ecosystem, leveraging Dataproc's managed Kafka cluster.

Answer

Use Apache Spark streaming on Dataproc to read from Kafka and write to GCS

Answer

Use Kafka MirrorMaker to replicate topics to a second cluster that writes to GCS

Answer

Use the Pub/Sub connector to publish Kafka messages to Pub/Sub, then a Dataflow job to write to GCS

Question 2

A company wants to use dbt to transform data in BigQuery. Their source data is loaded daily into staging tables. They need to run dbt transformations on a schedule and only process tables that have changed. Which dbt feature should they use?

Accepted Answer

dbt incremental models. dbt incremental models allow processing only new or changed records based on a configured timestamp or unique key. dbt snapshots capture historical changes. dbt seeds load CSV files. dbt sources are for configuration, not incremental processing.

Answer

dbt snapshots

Answer

dbt seeds

Answer

dbt sources

Question 3

A data engineer needs to create a Dataflow pipeline template that can be reused across multiple environments (dev, staging, prod) with different parameters (e.g., input Pub/Sub topic, output BigQuery table). Which template type should they use?

Accepted Answer

Flex Template. Flex Templates (B) are the correct choice because they package a Dataflow pipeline as a Docker image, allowing environment-specific parameters (e.g., Pub/Sub topic, BigQuery table) to be passed at runtime via the --parameters flag. This enables true reusability across dev, staging, and prod without modifying the template code, unlike Classic Templates which require compile-time parameterization.

Answer

Dataflow Prime

Answer

Classic Template

Answer

Cloud Composer workflow template

Question 4

A data engineer needs to query a BigQuery table that contains an array of structs. They want to expand the array into separate rows for each element. Which SQL function should they use?

Accepted Answer

UNNEST. UNNEST is used to flatten arrays into a set of rows. CROSS JOIN UNNEST is standard. STRUCT is for creating structs, ARRAY_AGG is for aggregation, and SPLIT is for strings. The question asks to expand an array, which is exactly UNNEST.

Answer

STRUCT

Answer

ARRAY_AGG

Answer

SPLIT

Question 5

A company wants to migrate their on-premises Teradata data warehouse to BigQuery. They need an automated, one-time transfer of historical data (10 TB) and ongoing incremental daily syncs. Which Google Cloud service should they use?

Accepted Answer

BigQuery Data Transfer Service for Teradata. BigQuery Data Transfer Service supports Teradata as a source for both one-time and scheduled transfers. Storage Transfer Service is for file-based transfers. Datastream is for CDC, not Teradata. Dataflow could be custom-built but Data Transfer Service is purpose-built for this scenario.

Answer

Dataflow custom pipeline

Answer

Storage Transfer Service

Answer

Datastream

Question 6

A data engineer is using Spark on Dataproc to process a large dataset. They notice the job is slow due to excessive shuffling. They want to optimize the job by using a more efficient data structure that reduces serialization overhead and provides better memory management. Which Spark API should they use?

Accepted Answer

DataFrames or Datasets. Spark DataFrames/Datasets use Tungsten execution engine, which provides optimized serialization and memory management. RDDs lack these optimizations. Spark SQL is a module, not an API. Spark Streaming is for streaming.

Answer

Spark SQL

Answer

Spark Streaming

Answer

RDDs

Question 7

A company uses Workflows to orchestrate a multi-step data pipeline. One step calls an HTTP endpoint that may take up to 10 minutes, but the default Workflows timeout is too short. They also need to handle transient errors with retries. Which TWO configurations should they apply? (Choose 2)

Accepted Answer

Set a step timeout of 600 seconds for the HTTP call step. To extend the timeout, set a step timeout of 600 seconds (10 minutes). To handle transient errors, use a retry policy with appropriate conditions. Setting the entire workflow timeout to 10 minutes is not necessary if individual step timeouts are set. The default retry policy does not cover all transient errors. Adding a dead letter queue is for event-driven patterns, not Workflows.

Answer

Configure a dead letter queue for failed steps

Answer

Use the default retry policy on the step

Answer

Set the workflow execution timeout to 600 seconds

Question 8

A company wants to ingest data from an on-premises Oracle database into BigQuery in near real-time with minimal latency. The database has a high volume of inserts and updates. Which service should they use?

Accepted Answer

Datastream. Datastream is designed for CDC from Oracle and other sources to BigQuery or GCS in near real-time.

Answer

BigQuery Data Transfer Service

Answer

Pub/Sub

Answer

Storage Transfer Service

Question 9

A team wants to transfer data from an on-premises Hadoop cluster to Cloud Storage for processing. The cluster is located in a remote area with limited bandwidth. They need to transfer 500 TB of data. Which service should they use?

Accepted Answer

Transfer Appliance. Transfer Appliance is designed for petabyte-scale offline transfers when bandwidth is limited.

Answer

BigQuery Data Transfer Service

Answer

Storage Transfer Service

Answer

Dataproc with gsutil

Question 10

A data pipeline uses Pub/Sub to ingest events, a Dataflow streaming pipeline to process them, and writes results to BigQuery. The pipeline must handle occasional duplicate events without causing duplicate rows in BigQuery. What is the best approach?

Accepted Answer

Use BigQuery legacy streaming inserts with insertId for deduplication. BigQuery does not enforce primary keys; deduplication must be handled in the pipeline using idempotent writes or a dedup step.

Answer

Use the Storage Write API with the committed stream

Answer

Set a unique constraint on the BigQuery table

Answer

Enable exactly-once processing in Pub/Sub

Question 11

A company uses Google Ads and wants to automatically load their advertising data into BigQuery daily. They also need to transform the data with SQL and schedule a recurring query. Which combination of services meets these requirements with minimal operational overhead?

Accepted Answer

BigQuery Data Transfer Service for Google Ads and scheduled queries. BigQuery Data Transfer Service can automatically load Google Ads data; scheduled queries handle transformation.

Answer

Cloud Functions triggered by Cloud Scheduler to call Google Ads API and load into BigQuery

Answer

Cloud Composer to extract Google Ads API and Dataflow to transform

Answer

Storage Transfer Service to move CSV files to GCS, then load into BigQuery

Question 12

A data engineer needs to create a Dataflow pipeline that reads from Pub/Sub, applies a Python transformation, and writes to BigQuery. The pipeline should be reusable across environments with different parameters. Which deployment method is most appropriate?

Accepted Answer

Flex Template. Flex Templates (Option B) are the most appropriate deployment method because they allow you to package a custom Docker image containing your Python transformation code and dependencies, making the pipeline reusable across environments with different runtime parameters. Unlike Classic Templates, Flex Templates support arbitrary pipeline code and can be parameterized at runtime via the Dataflow UI or API, which is essential for a multi-environment deployment strategy.

Answer

Classic Template

Answer

Direct pipeline submission with gcloud dataflow jobs run

Answer

Cloud Composer to trigger Dataflow jobs

Question 13

A company uses Kafka on Dataproc to ingest streaming data. They want to process the data with Spark Structured Streaming and write results to BigQuery. The team is using Dataproc clusters. Which approach minimizes cost while maintaining performance?

Accepted Answer

Use a Dataproc cluster with standard master nodes and preemptible worker nodes. Preemptible VMs are cost-effective for worker nodes; master nodes should be standard for reliability.

Answer

Use a Dataproc cluster with all preemptible VMs

Answer

Use a single-node Dataproc cluster

Answer

Use a Dataproc cluster with standard nodes and enable autoscaling

Question 14

Which BigQuery feature allows you to query data directly from Cloud Storage without loading it into BigQuery storage?

Accepted Answer

External tables. BigQuery Omni is for multi-cloud, not external tables. External tables allow querying data in GCS.

Answer

BigQuery Omni

Answer

BigQuery ML

Answer

Federated queries

Question 15

A company wants to build an event-driven application that processes images uploaded to a Cloud Storage bucket. The processing takes up to 10 minutes per image and should be automatically triggered. Which compute option should they use?

Accepted Answer

Cloud Run on Eventarc trigger. Cloud Functions have a 9-minute timeout; Cloud Run can handle up to 60 minutes and is triggered by Eventarc for GCS events.

Answer

Cloud Functions (2nd gen) with Eventarc trigger

Answer

App Engine

Answer

Cloud Functions (1st gen)