PDE Practice Test 20 — 15 Questions

Question 1

A company is using Cloud Storage to store raw logs. They want to use Cloud Data Fusion to transform and load the data into BigQuery on a daily schedule. The transformations are complex and involve joining multiple datasets. What is the most efficient way to run these pipelines?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use Cloud Data Fusion to design the pipeline and schedule it to run on a Dataproc cluster. Cloud Data Fusion supports scheduling pipelines and runs them on Dataproc. This is the standard way to run batch pipelines.

Answer

Use Cloud Composer to orchestrate Dataproc jobs that run the transformations

Answer

Use Cloud Functions to trigger a Dataflow job that does the transformations

Answer

Use Cloud Dataprep to design the transformation and export to BigQuery

Question 2

You are designing a Dataflow pipeline that reads from Pub/Sub and writes to BigQuery. The pipeline must handle late-arriving data (up to 1 hour) and group events into 10-minute windows. Which configuration is correct?

Accepted Answer

Use fixed windows of 10 minutes with allowed lateness of 1 hour and a trigger that fires after watermark plus early firings. To handle late data, you need to set the allowed lateness to 1 hour. The trigger with AfterWatermark with early firings ensures results are emitted on time and updated when late data arrives.

Answer

Use global windows with a trigger that fires every 10 minutes

Answer

Use sliding windows of 10 minutes with a 5-minute period and allowed lateness of 1 hour

Answer

Use fixed windows of 10 minutes with allowed lateness of 0 seconds

Question 3

You need to create a BigQuery table that stores customer transaction data. The table will be queried frequently by a customer_id column to retrieve recent transactions (last 30 days). Which table design optimizes query performance and cost?

Accepted Answer

Partition by transaction_date and cluster by customer_id. Partitioning by transaction_date allows queries to scan only relevant partitions. Clustering by customer_id sorts data within each partition by customer_id, further reducing the amount of data scanned for queries filtering on customer_id. This combination is best for time-range queries with frequent customer_id filters.

Answer

Partition by customer_id and cluster by transaction_date

Answer

Partition by ingestion_time and cluster by customer_id

Answer

Cluster by transaction_date and customer_id without partitioning

Question 4

A company uses BigQuery with partitioned tables by ingestion time. They notice that queries scanning recent partitions are fast but queries scanning older partitions are slow. What is the most likely cause?

Accepted Answer

The older partitions lack clustering metadata because clustering was enabled after data was ingested. Clustering can improve query performance by sorting data within partitions. If older partitions were created before clustering was enabled, they are not clustered, leading to slower scans. Re-clustering only applies to new data unless you manually rewrite older partitions.

Answer

Older partitions are stored on slower storage tiers

Answer

The table has too many partitions, causing high metadata overhead

Answer

Queries are using different SQL syntax for older partitions

Question 5

A data engineer needs to create a BigQuery table that is optimized for queries that filter on a 'customer_id' column and sort by 'transaction_date'. The table will be used for interactive analysis. Which combination of table features should be used?

Accepted Answer

Partition by transaction_date and cluster by customer_id. Clustering sorts data based on one or more columns, improving query performance for filters and sorts on those columns. Partitioning by date/timestamp can further improve performance for time-range queries. For this scenario, clustering on customer_id and partitioning by transaction_date is optimal.

Answer

Partition by customer_id and cluster by transaction_date

Answer

Cluster by both customer_id and transaction_date

Answer

Use a materialized view with customer_id and transaction_date

Question 6

You need to transform and clean messy CSV data using a visual interface without writing code. The transformation should be scheduled to run weekly. Which Google Cloud service should you use?

Accepted Answer

Cloud Dataprep. Cloud Dataprep (Trifacta) provides a visual interface for data wrangling, allowing users to create recipes and schedule jobs.

Answer

Cloud Dataflow

Answer

Dataproc

Answer

Cloud Data Fusion

Question 7

An organization runs periodic Apache Spark jobs on Dataproc to process data from Cloud Storage. They want to reduce costs by using preemptible instances for worker nodes. What is a key consideration when using preemptible instances in Dataproc?

Accepted Answer

Jobs must be designed to handle node preemption, and overall job runtime may increase. Preemptible VMs can be terminated at any time, so Spark jobs must be fault-tolerant. Dataproc handles this by automatically rescheduling failed tasks, but the job may take longer.

Answer

Preemptible instances cannot be used with the standard cluster mode

Answer

Preemptible instances are only available in certain regions

Answer

Jobs will automatically restart from the last checkpoint without any performance impact

Question 8

You need to analyse streaming data from thousands of IoT devices, each sending temperature readings every second. You want to calculate the average temperature per device over the last 5 minutes, updating every minute. Which windowing strategy should you use in Dataflow?

Accepted Answer

Sliding windows of length 5 minutes with a period of 1 minute. Sliding windows of length 5 minutes with a period of 1 minute give the desired overlapping windows: every minute, you get the average over the last 5 minutes.

Answer

Global windows with a trigger firing every minute

Answer

Fixed windows of 5 minutes

Answer

Session windows with a gap duration of 1 minute

Question 9

Your company uses Cloud Data Fusion to build ETL pipelines. You have a pipeline that reads from Cloud Storage, transforms data using a custom Wrangler recipe, and writes to BigQuery. The pipeline is failing with an error indicating that the Wrangler directive is invalid. You have verified the recipe works in the Cloud Data Fusion Studio. What is the most likely cause of the failure?

Accepted Answer

The Wrangler plugin is not deployed in the Cloud Data Fusion instance. When a pipeline that works in the studio fails at runtime, common issues include differences in environment (e.g., runtime arguments, service account permissions, or plugin versions). But the most likely cause is that the pipeline configuration does not include the necessary plugins or the runtime environment is missing the required artifacts. In Cloud Data Fusion, the studio uses a local or preview environment, while the pipeline runs on a separate Cloud Data Fusion instance with its own set of plugins. If the Wrangler plugin is not deployed to the runtime environment, the directive will fail.

Answer

The pipeline is using a different version of the Wrangler plugin

Answer

The Cloud Storage bucket is in a different region than the Data Fusion instance

Answer

The service account used in the pipeline does not have permissions to write to BigQuery

Question 10

A data engineer is designing a real-time fraud detection system using Dataflow. The system must detect patterns across events from multiple users within a sliding window of 10 minutes. Events arrive on Pub/Sub topics per user. Which approach should they use to join the streams?

Accepted Answer

Use CoGroupByKey on the two streams using a common key like user_id. CoGroupByKey joins multiple PCollections by key. Using user_id as common key, both streams can be joined. Side inputs and Union are not for joining. Flatten merges PCollections of same type.

Answer

Use a side input to read one stream as a map and enrich the other stream

Answer

Use Flatten to merge the streams and then Partition

Answer

Use Union to combine both streams into one and then apply GroupByKey

Question 11

A company needs to process data from a legacy system that outputs CSV files daily. They want to visually build transformations without writing code. Which Google Cloud service should they use?

Accepted Answer

Dataprep. Dataprep is a visual data wrangling tool for exploring and cleaning data.

Answer

Dataproc

Answer

Dataflow

Answer

Cloud Data Fusion

Question 12

Your team has a Dataflow pipeline that reads from BigQuery, transforms data, and writes to GCS. The pipeline is failing with 'Out of Memory' errors on the worker nodes. The input data is large but fits within the total cluster memory. Which configuration change is most likely to resolve the issue without increasing costs significantly?

Accepted Answer

Use a worker machine type with more memory, such as n2-highmem.. The default Dataflow worker machine type may have insufficient memory per core for the pipeline's operations. Using a high-memory machine type (e.g., n2-highmem) increases memory per worker without necessarily increasing the number of workers, thus controlling costs.

Answer

Shard the input into smaller reads using a BigQuery query.

Answer

Increase the disk size per worker.

Answer

Enable Dataflow Prime with vertical autoscaling.

Question 13

An organization needs to continuously replicate change data from a MySQL database to BigQuery with sub-minute latency. The database is running on-premises. Which Google Cloud service should they use?

Accepted Answer

Cloud Datastream. Datastream is a serverless change data capture (CDC) service that can replicate from MySQL, PostgreSQL, and Oracle to BigQuery or GCS with low latency. Pub/Sub is a messaging service and does not natively connect to MySQL. Dataflow can process streams but requires a CDC connector. BigQuery Data Transfer Service does not support live CDC from on-prem MySQL.

Answer

Cloud Pub/Sub with a custom connector to MySQL

Answer

BigQuery Data Transfer Service for MySQL

Answer

Cloud Dataflow with a JDBC source

Question 14

A data engineer needs to schedule a recurring transfer of data from a partner's Amazon S3 bucket to a Cloud Storage bucket for further processing. Which THREE components or configurations are necessary? (Choose 3)

Accepted Answer

Specification of the source S3 bucket and destination GCS bucket. Scheduling, source and destination locations, and authentication/authorization are essential for a Storage Transfer Service job.

Answer

A VPC network configuration

Answer

A Pub/Sub topic to notify completion

Question 15

You are migrating an on-premises PostgreSQL database to Cloud SQL. You need to continuously replicate changes to BigQuery for real-time analytics with minimal latency. Which service should you use?

Accepted Answer

Datastream. Datastream is designed for change data capture (CDC) from databases like PostgreSQL, MySQL, and Oracle to BigQuery or GCS. It provides low-latency replication. Pub/Sub and Dataflow are not directly for CDC. Storage Transfer Service is for file transfers, not database replication.

Answer

Dataflow with JDBC source

Answer

Pub/Sub with a Cloud Function that writes to BigQuery

Answer

Storage Transfer Service