PDE Practice Test 14 — 15 Questions

Question 1

You are designing a Dataflow pipeline that needs to exactly-once process events from Pub/Sub and write to BigQuery using the Storage Write API. The pipeline may restart and could reprocess some messages. What setting ensures exactly-once semantics for the output?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use the Storage Write API in committed mode and enable exactly-once semantic in Dataflow. The Storage Write API supports exactly-once semantics when used with the 'committed' mode, which ensures each record is written exactly once. The pipeline also needs to use Pub/Sub with message IDs and deduplication. The other options either do not provide exactly-once or are unreliable.

Answer

Use the legacy streaming inserts with insertId for deduplication

Answer

Use at-least-once delivery on Pub/Sub and idempotent writes to BigQuery

Answer

Use the Storage Write API in buffered mode with deduplication logic

Question 2

A company wants to use BigQuery ML to train a time-series forecasting model on historical sales data. The data is recorded daily for 3 years. They need to evaluate model accuracy using time-series aware cross-validation. Which two options should they configure in the CREATE MODEL statement? (Choose TWO)

Accepted Answer

Specify a time_series_timestamp_col and time_series_data_col. For ARIMA+ models, you can set 'horizon' (forecast length) and 'data_frequency' (auto-detect or set). Cross-validation is not built-in for ARIMA; instead, you evaluate on held-out periods.

Answer

Set the data_frequency parameter to 'daily'

Answer

Use the 'num_trials' parameter for hyperparameter tuning

Answer

Set the 'split_method' to 'time_series'

Question 3

A data engineer needs to process data in a Dataflow pipeline that reads from a Pub/Sub topic. The pipeline must group events into 5-minute windows and compute the average value per key. Which Beam transform should they use after windowing?

Accepted Answer

Combine.perKey. Combine.perKey with an averaging function computes per-key average. GroupByKey groups by key but requires manual combination. ParDo is for element-wise processing. CoGroupByKey joins multiple PCollections.

Answer

ParDo

Answer

GroupByKey

Answer

CoGroupByKey

Question 4

Which Google Cloud service provides a visual interface for building ETL pipelines using a drag-and-drop design and includes pre-built transforms from a marketplace?

Accepted Answer

Cloud Data Fusion. Cloud Data Fusion offers a visual, code-free ETL tool with a rich set of plugins from the Hub. Dataprep is for data wrangling, not full ETL.

Answer

Dataproc

Answer

Dataprep

Answer

BigQuery

Question 5

You need to process large-scale log files (hundreds of terabytes) using Apache Spark on Google Cloud. The job runs nightly and you want to minimise costs. Which Dataproc cluster configuration is MOST cost-effective?

Accepted Answer

Standard cluster with preemptible secondary workers. Preemptible VMs are significantly cheaper than standard VMs and suitable for fault-tolerant batch jobs like nightly Spark processing. Standard mode is fine but using preemptible workers reduces cost.

Answer

Single-node cluster

Answer

Standard cluster with preemptible workers for the primary worker nodes

Answer

Standard cluster with all standard (non-preemptible) workers

Question 6

You need to allow a data analyst to run queries on a BigQuery dataset but prevent them from modifying the data or deleting the dataset. Which IAM role should you grant?

Accepted Answer

roles/bigquery.dataViewer. BigQuery Data Viewer grants read-only access to datasets and tables. roles/bigquery.dataViewer allows running queries and viewing metadata but not modifying or deleting data.

Answer

roles/bigquery.dataOwner

Answer

roles/bigquery.jobUser

Answer

roles/bigquery.dataEditor

Question 7

A company is building a real-time anomaly detection pipeline using Dataflow. Events are ingested from Pub/Sub, and the pipeline must compute a sliding window average every minute over a 1-hour window. Which TWO configurations are required for this pipeline? (Choose 2)

Accepted Answer

Set the pipeline to use event time for watermarking.. A sliding window of 1-hour length with a 1-minute slide period fits the requirement (every minute, compute over last hour). Fixed window of 1 minute would compute only per-minute, not sliding. Using stateful processing with timers is an alternative but not standard for sliding windows. Dataflow's default watermark is based on event time; processing time would cause incorrect results. The window type and period are the key.

Answer

Use a Fixed window of 1 minute.

Answer

Use stateful processing with a custom timer.

Answer

Set the pipeline to use processing time for watermarking.

Question 8

Your data engineering team needs to process a continuous stream of clickstream events from a website and update a real-time dashboard showing user activity over the last hour. The pipeline should have minimal operational overhead and support exactly-once processing semantics. Which Google Cloud service should you use?

Accepted Answer

Cloud Dataflow with Apache Beam. Dataflow with Apache Beam is the only service among the options that natively supports exactly-once processing and can handle both streaming and batch pipelines with minimal operational overhead. Dataflow is fully managed, handles autoscaling, and provides exactly-once guarantees for streaming data.

Answer

Cloud Dataproc with Apache Spark Streaming

Answer

Cloud Data Fusion with batch pipelines

Answer

Cloud Pub/Sub Lite with push subscriptions

Question 9

A company wants to use AutoML Tables to build a classification model on a dataset with 100 features and 500,000 rows. They need to deploy the model for online predictions with low latency (<100 ms). Which deployment option should they choose?

Accepted Answer

Deploy the model to an endpoint in Vertex AI using the AutoML endpoint service. AutoML Tables supports online prediction endpoints that are deployed on a dedicated cluster, providing low latency for real-time predictions.

Answer

Export the model as a TF SavedModel and deploy on Cloud Run

Answer

Deploy the model on AI Platform Prediction

Answer

Use batch prediction in Vertex AI

Question 10

You are migrating on-premises Hadoop jobs to Google Cloud. The existing jobs use Spark for ETL and Hive for querying. You want to minimize changes to the existing code and maintain the ability to use Hive queries with the same metastore across multiple clusters. Which service combination should you use?

Accepted Answer

Cloud Dataproc with Cloud Storage and Dataproc Metastore. Dataproc is the managed Hadoop/Spark service that allows you to run existing Spark and Hive code without modification. Dataproc Metastore provides a fully managed Hive metastore that can be shared across multiple Dataproc clusters. Cloud Storage is used instead of HDFS for storing data, but the metastore is the key component.

Answer

Cloud Dataflow with Beam SQL

Answer

Cloud Dataproc with Dataproc on GKE

Answer

Cloud BigQuery with external tables on Cloud Storage

Question 11

A data engineer needs to query a BigQuery table that contains an array of structs. They want to expand the array into separate rows for each element. Which SQL function should they use?

Accepted Answer

UNNEST. UNNEST is used to flatten arrays into a set of rows. CROSS JOIN UNNEST is standard. STRUCT is for creating structs, ARRAY_AGG is for aggregation, and SPLIT is for strings. The question asks to expand an array, which is exactly UNNEST.

Answer

STRUCT

Answer

ARRAY_AGG

Answer

SPLIT

Question 12

A company stores sensitive customer data in BigQuery. They need to implement column-level security to restrict access to personally identifiable information (PII) columns. Which two BigQuery features can they use together? (Choose TWO)

Accepted Answer

Authorized views. Column-level security can be achieved via policy tags (with Data Catalog) and then authorized views that filter columns based on user roles. Policy tags enforce access control at the column level.

Answer

BigQuery row-level security

Answer

BigQuery foreign key constraints

Answer

BigQuery data masking

Question 13

A Dataflow pipeline reads from Pub/Sub, applies a keyed stateful ParDo that uses state variables to deduplicate events based on event ID, and writes to BigQuery. During a pipeline update, some events are duplicated in BigQuery. The state is not preserved across updates. Which configuration ensures exactly-once semantics during updates?

Accepted Answer

Take a snapshot of the pipeline before updating, then start the new pipeline from the snapshot.. Draining the pipeline stops it and completes processing in-flight, then the updated pipeline can start fresh. However, because state is lost, duplicates may still occur if the new pipeline processes events that were already committed. To preserve state, use snapshotting: take a snapshot before update and start the new pipeline from the snapshot. BigQuery's Storage Write API with exactly-once semantics can help at the sink but does not prevent duplicate processing if state is lost. As long as the deduplication state is recovered from the snapshot, duplicates are avoided.

Answer

Drain the pipeline and start the updated pipeline; all in-flight data will be processed.

Answer

Cancel the pipeline and restart it; Pub/Sub subscriptions will be rewound.

Answer

Use the Storage Write API's exactly-once delivery mode.

Question 14

A retail company uses Dataflow to process real-time clickstream data. They need to enrich each event with customer profile data from Cloud Bigtable and session metadata from Cloud Spanner. Which two Dataflow features should they use?

Accepted Answer

ParDo. Side inputs allow reading from Bigtable and Spanner in a non-blocking way. ParDo is for per-element processing where enrichment occurs. GroupByKey and Windowing are not needed for this enrichment step.

Answer

Windowing

Answer

CoGroupByKey

Answer

GroupByKey

Question 15

A company wants to use Cloud Data Fusion to build ETL pipelines. They need to connect to a legacy on-premises database using JDBC and also want to use prebuilt transforms from the Hub. Which two features should they use?

Accepted Answer

CDAP JDBC plugin and the Hub. Cloud Data Fusion uses CDAP plugins for JDBC connections and the Hub provides prebuilt transforms. Plugins are the mechanism; Hub is where they are sourced. Wrangler is for data preparation, not sink. Dataproc is not needed as Data Fusion runs on its own infrastructure.

Answer

Cloud SQL JDBC driver and Cloud Functions

Answer

Dataproc Metastore and Cloud Storage sink

Answer

Wrangler and Dataproc