PDE Practice Test 19 — 15 Questions

Question 1

Your team runs a Dataflow streaming pipeline that reads from Pub/Sub and writes to BigQuery. During a deployment, the pipeline is stopped and later restarted with the same job name but without draining. After the restart, downstream reports show duplicate rows for events that were processed just before the stop. You need the pipeline to resume without reprocessing already-published messages. What should you do?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Drain the pipeline before stopping it, then start the replacement pipeline from the saved state.. Stopping a streaming job without draining leaves in-flight Pub/Sub messages unacknowledged, so they are redelivered when a replacement job starts, producing duplicates. Draining first lets the pipeline finish processing and commit those acknowledgements and BigQuery writes, so the replacement resumes from a consistent state and avoids reprocessing the same events.

Answer

Increase the number of workers with --maxNumWorkers before restarting the pipeline.

Answer

Enable --enableStreamingEngine on the restarted pipeline to preserve message state.

Answer

Restart the pipeline using the --update option so the transform graph is replaced in place.

Question 2

A healthcare analytics group must build a pipeline that ingests HL7 messages from an on-premises interface engine, must retain raw messages for seven years for compliance, and must expose de-identified aggregates to analysts. The security team requires that protected health information never be written to a dataset analysts can query, and that all data be encrypted with keys the organization manages and can revoke. Which two design choices satisfy these requirements? (Choose two.)

Accepted Answer

De-identify the messages in Dataflow, write only the de-identified aggregates into a separate BigQuery dataset secured with CMEK and policy tags, and grant analysts access only to that dataset.. The requirements demand a hard separation between raw PHI and anything analysts can query, plus organization-managed revocable encryption and seven-year retention. Landing raw messages in Cloud Storage with CMEK and a retention lock, accessible only to the ingestion service account, satisfies retention and key control. De-identifying in Dataflow and publishing only aggregates to a separate CMEK-protected BigQuery dataset keeps PHI out of analyst-facing storage entirely, so both boundaries hold independently.

Answer

Load the raw messages directly into a BigQuery table that analysts can query, and rely on column-level security to mask the PHI columns.

Answer

Store the raw messages in BigQuery and use authorized views so analysts query a view that filters out PHI columns instead of the base table.

Answer

Encrypt the raw messages with a customer-supplied key in the on-premises interface engine and upload only the ciphertext to Cloud Storage, decrypting in Dataflow for de-identification and discarding the key afterward.

Question 3

A logistics company collects GPS pings from delivery vehicles into Pub/Sub and needs to compute the distance traveled per vehicle per hour. The data volume is high and bursty, and the company wants a managed service that automatically scales the number of workers based on load while allowing custom windowing and stateful processing. Which service should they use?

Accepted Answer

Dataflow with a streaming pipeline using windowing and stateful DoFn. Dataflow provides managed autoscaling and native support for windowing and stateful processing, which are required to compute per-vehicle hourly distance from a bursty stream. The other options either lack stateful windowing, do not scale automatically, or are not designed for continuous stream processing.

Answer

Cloud Functions triggered by Pub/Sub messages

Answer

Dataproc with a Spark Streaming job on a fixed-size cluster

Answer

BigQuery with a scheduled query that reads from a Pub/Sub subscription

Question 4

A company uses Dataproc Serverless for Spark batch jobs. They notice that some jobs are failing due to out-of-memory (OOM) errors. Which configuration parameter should they adjust to allocate more memory per executor?

Accepted Answer

Set spark.executor.memory to a higher value, e.g., 8g. OOM errors in Spark executors are resolved by increasing executor memory, which is controlled by spark.executor.memory. Setting it to a higher value such as 8g allocates more heap to each executor, allowing it to process larger partitions without running out of memory. This is the direct and correct configuration for the reported symptom.

Answer

Use a custom image with more memory

Answer

Set spark.driver.memory to a higher value

Answer

Increase the number of workers by setting --num-workers

Question 5

A financial firm ingests trade confirmations from a partner's SFTP server into Cloud Storage, then loads them into BigQuery. The partner drops a variable number of files at unpredictable times, and the firm must detect each new file within seconds and trigger a downstream Cloud Run job. The files must not be processed twice if the trigger fires more than once. Which design should you implement?

Accepted Answer

Configure an Eventarc trigger on the bucket's object.finalize events to invoke Cloud Run, and have the function record the object generation in Firestore before processing.. Eventarc on object.finalize gives sub-minute, event-driven detection without polling, and the object generation number serves as a natural idempotency key because it uniquely identifies a version of the object. Writing that key to Firestore before doing the work makes duplicate deliveries harmless. Polling designs miss the latency target, and designs without a deduplication record allow the same file to be processed again on redelivery.

Answer

Schedule a Cloud Scheduler job every minute that lists the bucket and invokes Cloud Run for any object newer than the last run.

Answer

Use a Pub/Sub topic populated by Cloud Storage notifications and subscribe with a pull subscription that Cloud Run polls on a schedule.

Answer

Attach a Cloud Functions (1st gen) trigger to the bucket and let it call Cloud Run synchronously for each new object.

Question 6

A data science team uses Vertex AI Pipelines to automate retraining. They want to ensure that only models with performance above a threshold are deployed. Which component should they add to the pipeline?

Accepted Answer

Vertex AI Model Evaluation. Vertex AI Model Evaluation provides built-in evaluation metrics and threshold-based validation that can be used as a pipeline condition to gate model deployment. By adding a Model Evaluation component, the pipeline can compare model performance against a predefined threshold and only proceed to deploy if the metrics (e.g., AUC, precision, recall) meet or exceed the required value.

Answer

Vertex AI Feature Store

Answer

Cloud Build trigger

Answer

Cloud Monitoring alert

Question 7

A data platform team uses Cloud Data Fusion to move data from an on-premises relational database into BigQuery. They need the pipeline to run on a fixed schedule, capture only rows changed since the last successful run, and avoid re-reading the entire source table each night. The source table has an updated_at column that is reliably populated. Which two approaches should they use? (Choose two.)

Accepted Answer

Configure the pipeline to run on a schedule using the Data Fusion scheduler or a Cloud Composer trigger. Scheduling the pipeline and filtering the source query on a persisted updated_at watermark together deliver a recurring job that reads only changed rows. The watermark must be written after a successful load so failures do not advance it prematurely, and the query-based Database source plugin is what makes the filter possible without custom code.

Answer

Use a Replicator pipeline with the default full-table snapshot mode for each run

Answer

Enable the Cloud Data Fusion lineage feature to track which rows were previously loaded

Answer

Use the BigQuery sink plugin with write disposition set to truncate before each nightly load

Question 8

A media analytics team needs to copy 200 TB of historical log files from an on-premises NFS server into Cloud Storage, then transform them with a Dataflow batch job. The NFS server is reachable only from the corporate network, and the team wants to minimize transfer time and cost. Which approach should they use?

Accepted Answer

Use Storage Transfer Service with an agent pool deployed in the corporate network to transfer directly from the NFS server to Cloud Storage.. Storage Transfer Service with an agent pool is purpose-built for moving large datasets from on-premises sources like NFS to Cloud Storage. Agents run inside the corporate network, so no inbound firewall changes are needed, and the service handles parallelism, retries, and integrity checks. Manual rsync, Transfer Appliance, and single-machine VPN copies are slower or inappropriate at this scale.

Answer

Create a Cloud VPN tunnel and use gcloud storage cp from an on-premises machine to upload the files.

Answer

Mount the NFS share on a Compute Engine VM and run gsutil -m rsync to copy the data to Cloud Storage.

Answer

Use Transfer Appliance to ship the data to Google, then load it into Cloud Storage.

Question 9

A data engineer manages a Cloud SQL for MySQL instance that stores order records. Compliance requires that the data be recoverable to any point in time within the last 30 days, and the team wants the smallest possible recovery window. They also want to avoid managing their own backup rotation scripts. Which configuration should the engineer implement?

Accepted Answer

Enable point-in-time recovery (PITR) with a transaction log retention of 30 days alongside automated backups.. Point-in-time recovery in Cloud SQL combines the latest automated backup with continuously archived binary logs, enabling restoration to a chosen timestamp. Extending transaction log retention to 30 days matches the compliance window while keeping the process fully managed. Daily backups alone or nightly dumps cannot achieve arbitrary-timestamp recovery, and read replicas address availability rather than historical restore.

Answer

Configure a read replica in a second zone and promote it if the primary fails.

Answer

Enable automated backups with a 30-day retention period and rely on the daily backup only.

Answer

Export nightly mysqldump files to a Cloud Storage bucket with a 30-day lifecycle rule.

Question 10

A company processes IoT sensor data in near real-time. They ingest data via Cloud Pub/Sub, then a Dataflow streaming pipeline writes to Bigtable for low-latency queries. Recently, they observed increased Pub/Sub message backlog during traffic spikes. What is the most effective scaling strategy?

Accepted Answer

Increase Dataflow worker count and adjust autoscaling configuration. The increased Pub/Sub backlog during traffic spikes indicates that the Dataflow pipeline is unable to consume messages as fast as they are being published. Increasing the Dataflow worker count and adjusting autoscaling configuration allows the pipeline to scale horizontally, processing more messages per second and reducing the backlog. Pub/Sub itself is designed to handle high throughput, so the bottleneck is the consumer (Dataflow), not the ingestion layer.

Answer

Increase Pub/Sub subscription throughput by increasing the number of partitions

Answer

Use a Cloud Scheduler to throttle Pub/Sub publishing

Answer

Add a Cloud Function to pre-process messages before they are consumed by Dataflow

Question 11

You are designing a Dataflow pipeline that joins two unbounded PCollections from different sources. Which transform should you use?

Accepted Answer

CoGroupByKey. CoGroupByKey performs a key-based join of multiple PCollections. It can handle unbounded streams with appropriate windowing.

Answer

ParDo

Answer

Flatten

Answer

GroupByKey

Question 12

A data engineer manages a BigQuery dataset holding a 40 TB partitioned table of point-of-sale transactions. Analysts frequently run queries that filter on a region column and a sale_date column together, but each query scans the full table because the region predicate is not reducing bytes billed. The engineer wants to reduce bytes scanned without changing the analytical queries or the write pipeline. What should the engineer do?

Accepted Answer

Create a clustered table on the region column, because clustering co-locates rows with similar values inside each partition so block pruning can skip irrelevant data.. Clustering is the right lever because it physically sorts data within each partition by the region column, enabling block pruning when region predicates are present. Partitioning already handles sale_date, so adding clustering on the other frequently filtered column reduces scanned bytes while leaving the queries and ingestion pipeline untouched.

Answer

Define a materialized view that aggregates the table by region and sale_date and let the optimizer rewrite all analyst queries to use it.

Answer

Enable the require_partition_filter option on the table so that every query must include a sale_date predicate.

Answer

Convert the table to an external table over Cloud Storage so that only the matching Parquet row groups are read at query time.

Question 13

A media company stores 900 TB of video master files in a Cloud Storage bucket in the US multi-region. Legal requires that the objects be retained for seven years and cannot be deleted or overwritten by any user, including project owners, during that period. The company also wants to minimize storage cost for objects that are rarely accessed after the first 90 days. The data engineer must implement a compliant configuration. (Choose two.)

Accepted Answer

Add an Object Lifecycle Management rule that transitions objects to Coldline storage 90 days after creation.. A locked retention policy is the only Cloud Storage mechanism that prevents even project owners from deleting or overwriting objects, providing the required WORM guarantee for seven years. Pairing it with a lifecycle transition to Coldline after 90 days meets the cost objective because Coldline targets infrequently accessed data without conflicting with retention.

Answer

Set the bucket's default storage class to Standard and rely on the multi-region location for durability.

Answer

Enable Uniform bucket-level access and grant the storage.objectViewer role to analysts.

Answer

Add a lifecycle rule that deletes objects 90 days after creation to control cost.

Question 14

A logistics company ingests GPS pings from delivery vans into Pub/Sub, and a Dataflow streaming pipeline writes them to BigQuery. Latency requirements are lenient (about 5 minutes), but the finance team needs the pipeline's cost to be predictable and low, and the data volume fluctuates by a factor of ten between day and night. The team wants to minimize per-element cost without losing data. Which configuration should the data engineer choose?

Accepted Answer

Streaming Engine with autoscaling and a bounded maximum worker count, plus windowing with a 5-minute trigger so BigQuery writes are batched.. The scenario combines wildly variable throughput with a lenient latency target and a hard cost ceiling. Autoscaling with a bounded maximum worker count is the standard way to match capacity to demand while capping worst-case spend, and a five-minute windowing trigger amortizes BigQuery write overhead across many elements. Together these choices honor the latency allowance, keep costs predictable, and avoid the data-loss risk of an undersized fixed pool.

Answer

Batch mode with a 5-minute micro-batch schedule that reads Pub/Sub snapshots and writes to BigQuery.

Answer

Streaming Engine with autoscaling, a low --maxNumWorkers, and a streaming trigger that emits results at least every 5 minutes.

Answer

Streaming Engine with a fixed worker count sized for peak daytime load and --maxNumWorkers capped at that value.

Question 15

Your team is migrating a batch ETL job from an on-premises Hadoop cluster to Dataproc. The job reads CSV files from Cloud Storage, joins them with a slowly changing dimension table in BigQuery, and writes aggregated results back to BigQuery. The on-premises job used Hive on Tez and took six hours. You need to reduce runtime on Dataproc while minimizing cost. Which approach should you take?

Accepted Answer

Run the job as a Dataproc Serverless for Spark batch, reading the CSV files from Cloud Storage and using the BigQuery connector to read the dimension table, with autoscaling enabled.. Dataproc Serverless for Spark runs batch workloads on ephemeral, autoscaling infrastructure that is released when the batch finishes, so cost tracks actual execution rather than idle cluster time. Reading CSV from Cloud Storage and using the BigQuery connector for the dimension table lets Spark optimize the join, often broadcasting the small dimension. This reduces runtime compared with the legacy Tez engine and avoids paying for a cluster that sits idle between runs.

Answer

Create a Dataproc cluster with a fixed number of standard workers, install Hive, and run the same Hive on Tez script to preserve the existing logic.

Answer

Create a long-running Dataproc cluster with many preemptible workers and run the job with Spark SQL, caching the BigQuery dimension table as a Spark DataFrame.

Answer

Export the CSV files to a temporary BigQuery table, perform the join and aggregation entirely in BigQuery SQL, and schedule the query with Cloud Scheduler.