PDE Practice Test 49 — 15 Questions

Question 1

A Dataflow batch pipeline reads CSV files from Cloud Storage, joins them with a slowly changing dimension stored in BigQuery, and writes the enriched output to BigQuery. The dimension table is large and the join is causing excessive shuffle and worker memory pressure. The team wants to reduce shuffle while keeping the join logic in the pipeline. Which approach should they use?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Load the dimension side into a side input and use it in a ParDo to enrich each record.. Using the dimension as a side input lets each worker enrich records locally, avoiding the shuffle and hot-key concentration that CoGroupByKey or a grouped join would introduce. It preserves the in-pipeline join logic and relieves memory pressure. Provisioning more workers or moving the join to BigQuery does not reduce the shuffle as requested.

Answer

Increase the number of workers and raise the disk size per worker to absorb the shuffle.

Answer

Use a CoGroupByKey on the two collections and process the grouped results.

Answer

Write the CSV data to BigQuery first, then run a SQL join in BigQuery and export the result.

Question 2

A media company stores final video masters in a Cloud Storage bucket. Regulatory rules require that each object be unalterable for seven years, and the company must be able to prove retention compliance to auditors. Objects are written once and never edited. The data engineer needs the strongest native Cloud Storage control that prevents deletion or overwrite for the required period. What should the engineer configure?

Accepted Answer

A bucket retention policy with a seven-year retention period and a locked retention policy.. A locked bucket retention policy is the native Cloud Storage feature that enforces immutability for a fixed period and cannot be weakened once locked. It directly satisfies the write-once, provable-retention requirement, whereas versioning, signed URLs, and IAM policies only govern recovery or access. For regulatory retention of final masters, the locked retention policy is the correct control.

Answer

Uniform bucket-level access with IAM roles limited to a single compliance group.

Answer

Object Versioning on the bucket plus a lifecycle rule to delete noncurrent versions.

Answer

A signed URL with a seven-year expiration that is shared only with the compliance team.

Question 3

A healthcare company needs to ingest HL7 messages from an on-premises system into Google Cloud. The messages arrive continuously and must be processed in near-real-time, with transformations applied before loading into BigQuery. The company wants to use a fully managed service for message ingestion and a serverless data processing service. They also need to ensure that the pipeline can handle bursts of traffic and that the data is encrypted at rest. Which TWO Google Cloud services should they use? (Choose two.)

Accepted Answer

Cloud Dataflow. Pub/Sub provides scalable, managed ingestion of HL7 messages with encryption at rest, and Dataflow offers serverless stream processing with autoscaling and exactly-once semantics. Together, they form a fully managed pipeline that can handle bursts and transform data before loading into BigQuery. Other services are either not serverless or not designed for real-time processing.

Answer

Cloud Data Fusion

Answer

Cloud Dataproc

Answer

Cloud Composer

Question 4

You are designing a Dataflow pipeline that reads from Pub/Sub and writes to BigQuery. The pipeline must handle late-arriving data and emit correct results. You need to ensure that the pipeline's windowing and triggering strategy produces accurate aggregations. Which combination of windowing and triggering should you use?

Accepted Answer

Fixed windows with a trigger that fires after watermark passes and allowed lateness greater than 0.. To handle late-arriving data and produce accurate aggregations, you need a windowing strategy that allows late elements to be incorporated. Fixed windows with a trigger that fires after the watermark and a positive allowed lateness achieve this: the trigger emits results when the watermark passes, and any late data within the allowed lateness is added to the window, causing the trigger to fire again and update the results. This ensures completeness and correctness.

Answer

Session windows with default trigger and no allowed lateness.

Answer

Sliding windows with repeated trigger and allowed lateness set to a positive value.

Answer

Fixed windows with default trigger and allowed lateness of 0.

Question 5

A data engineer is designing a BigQuery table for a fraud detection system. The table will store 500 TB of transaction records and will be queried constantly by filtering on a transaction timestamp column and a customer_id column. The engineer needs to minimize the amount of data scanned by these queries. What should the engineer do?

Accepted Answer

Partition the table by the transaction timestamp column and cluster by the customer_id column.. Partitioning by transaction timestamp and clustering by customer_id is the recommended approach for large BigQuery tables that are frequently filtered on a time column and a high-cardinality key. Partition pruning limits scanned partitions, and clustering sorts data within partitions so that filters on customer_id read fewer blocks, lowering both cost and latency.

Answer

Create a materialized view that pre-aggregates transactions by customer_id and timestamp.

Answer

Use the BigQuery Storage Write API to stream data into the table with a fixed schema.

Answer

Set the table's expiration time to 30 days to limit the amount of historical data stored.

Question 6

A financial services company must retain trade records for seven years in Cloud Storage. Regulators require that no object can be deleted or overwritten before its retention period expires, even by project owners, and that the policy cannot be removed. The company also needs to prove compliance during audits. Which combination of controls should they implement?

Accepted Answer

Set a bucket retention policy and lock it, then upload objects with a retention period covering seven years.. A locked bucket retention policy enforces WORM semantics: objects cannot be deleted or overwritten until their retention period elapses, and the lock prevents removal or reduction of the policy. Uploading records with a seven-year retention period satisfies the regulatory timeline and creates auditable proof. Other controls either can be changed, do not block deletion, or address confidentiality rather than immutability.

Answer

Enable Object Versioning and apply an Organization Policy that denies storage.objects.delete for all principals.

Answer

Configure a lifecycle rule that transitions objects to Archive storage after 30 days and deletes them after seven years.

Answer

Use Customer-Managed Encryption Keys in Cloud KMS and revoke key access after each upload.

Question 7

A data engineer is loading a 2 TB CSV dataset from Cloud Storage into a partitioned BigQuery table. The load job fails with an error indicating too many errors during parsing. The source files have inconsistent column counts and some rows contain embedded newlines. The engineer wants to load the data reliably with minimal preprocessing. What should the engineer do?

Accepted Answer

Convert the CSV files to newline-delimited JSON or Parquet, then load the converted files into BigQuery.. CSV parsing is delimiter-based and breaks when rows contain embedded newlines or inconsistent column counts. Converting to newline-delimited JSON or Parquet removes delimiter ambiguity and lets BigQuery load the data reliably, addressing the root cause rather than suppressing errors or increasing throughput.

Answer

Load the CSV files into a single external table and query it with the ignoreUnknownValues option.

Answer

Set the maxBadRecords option to a high value and rerun the load job with the same CSV files.

Answer

Increase the load job's maximum parallelism by splitting the CSV files into smaller objects in Cloud Storage.

Question 8

A financial services company stores transaction logs in Cloud Storage. Regulatory requirements mandate that each object be retained for exactly seven years and that no user, including project owners, can delete or modify the objects during that period. The company wants to enforce this with minimal administrative overhead. What should they do?

Accepted Answer

Create a bucket with a retention policy of seven years and lock the policy.. A locked retention policy on a Cloud Storage bucket enforces a minimum retention period and prevents deletion or modification of objects, even by project owners. Locking the policy makes it permanent, satisfying the seven-year WORM requirement. Lifecycle rules, IAM restrictions, or a nonexistent 'bucket policy' cannot provide the same immutable guarantee, making the locked retention policy the correct and minimal-overhead solution.

Answer

Enable Uniform Bucket-Level Access and grant only the Storage Object Viewer role to all users.

Answer

Configure a Cloud Storage bucket lock using a bucket policy with a seven-year retention period.

Answer

Use Object Lifecycle Management to transition objects to Archive storage after 30 days and delete after seven years.

Question 9

You need to ingest data from a Cloud Storage bucket into BigQuery. The data is in Avro format and you want to minimize the time to insight. Which method should you use?

Accepted Answer

Use the BigQuery Load job with the Avro source format and enable autodetect schema.. The BigQuery Load job natively supports Avro and can automatically detect the schema from the Avro file's embedded schema. This makes it the fastest and simplest way to ingest Avro data from Cloud Storage, minimizing time to insight. Other methods like Dataflow, Streaming API, or Cloud Data Fusion add unnecessary complexity, latency, or cost for a straightforward batch load.

Answer

Use Cloud Data Fusion to read the Avro files and load them into BigQuery.

Answer

Use the BigQuery Streaming API to stream the Avro records from Cloud Storage.

Answer

Use a Dataflow pipeline to read the Avro files and write to BigQuery.

Question 10

Your team runs a Cloud Composer 2 environment (composer-2.1.0-airflow-2.6.3) that executes a daily BigQuery ETL workflow. The workflow must not run on weekends. You want to implement this with minimal code and without modifying the DAG's task logic. What should you do?

Accepted Answer

Set the DAG's schedule_interval to '0 2 * * 1-5' and set catchup=False.. The correct approach is to use a cron expression that runs only Monday through Friday, such as '0 2 * * 1-5', combined with catchup=False to avoid backfilling. This leverages Airflow's native scheduling, requires no changes to task logic, and keeps the DAG simple. Other methods either add external dependencies or still trigger the DAG on weekends.

Answer

Add a BranchPythonOperator that checks the day of the week and skips downstream tasks on weekends.

Answer

Set the DAG's schedule_interval to '@daily' and add a ShortCircuitOperator that stops execution on weekends.

Answer

Create a Cloud Scheduler job that triggers the Composer DAG only on weekdays via the Airflow REST API.

Question 11

You manage a Cloud Dataflow streaming pipeline that reads from Pub/Sub and writes to BigQuery. The pipeline uses the BigQueryIO write transform with STREAMING_INSERTS. You need to ensure exactly-once processing semantics for the BigQuery writes. What should you do?

Accepted Answer

Use BigQueryIO.Write with withMethod(STORAGE_WRITE_API) and set withNumStorageWriteApiStreams to a value greater than zero.. To achieve exactly-once processing for BigQuery writes in a Dataflow streaming pipeline, use the BigQuery Storage Write API with withMethod(STORAGE_WRITE_API) and configure at least one stream. This method provides transactional writes and deduplication, ensuring each record is written exactly once. Other methods like STREAMING_INSERTS with insertId offer only best-effort deduplication and cannot guarantee exactly-once semantics.

Answer

Add a BigQuery insertId to each record and rely on BigQuery's built-in deduplication for streaming inserts.

Answer

Switch to BigQueryIO.Write with withMethod(FILE_LOADS) and set withTriggeringFrequency to a low value.

Answer

Enable exactly-once by setting the pipeline option --experiments=use_runner_v2 and using BigQueryIO.Write with withMethod(STREAMING_INSERTS).

Question 12

A data engineer schedules a Cloud Composer 2 environment to run a DAG that triggers a Dataflow batch job every night. The DAG sometimes fails because the Dataflow job takes longer than the default task timeout. The engineer wants the DAG to wait for the Dataflow job to finish rather than timing out. Which change should the engineer make?

Accepted Answer

Set the Dataflow operator's execution_timeout and deferrable parameters so the task waits for job completion.. Cloud Composer 2 Dataflow operators can block until the job reaches a terminal state, and the deferrable variant frees the worker while waiting. Setting execution_timeout to a value longer than the expected job duration prevents premature task failure. Schedule interval, Cloud Function wrapping, and retry policies do not make a task wait for the underlying Dataflow job.

Answer

Add a retry policy to the DAG so failed tasks are retried until the Dataflow job finishes.

Answer

Increase the DAG's schedule interval so runs start less frequently.

Answer

Move the Dataflow launch into a Cloud Function and call it from the DAG.

Question 13

You are designing a Dataflow pipeline that reads from Pub/Sub and writes to BigQuery. The pipeline uses the Storage Write API with exactly-once semantics. You need to ensure that the pipeline can handle a sudden spike in traffic without losing data or causing duplicates. Which configuration should you adjust to improve throughput while maintaining exactly-once?

Accepted Answer

Increase the number of Storage Write API streams.. The Storage Write API uses a configurable number of streams to write data to BigQuery. Each stream can handle a certain throughput, and increasing the number of streams allows more parallel writes, improving overall throughput. This adjustment maintains exactly-once semantics because each stream is managed with its own offset and deduplication. Autoscaling and worker count help with processing but do not directly increase the write throughput limit imposed by the number of streams.

Answer

Increase the maximum number of workers.

Answer

Enable autoscaling for the Dataflow job.

Answer

Switch to STREAMING_INSERTS method.

Question 14

You are building a Dataflow pipeline that reads Avro-formatted files from Cloud Storage. The files use a schema that is updated frequently. You want to minimize pipeline restarts due to schema changes. Which approach should you take?

Accepted Answer

Read the Avro files using a schema inferred at runtime from the Avro file metadata.. Avro files include their schema in the file header, so a Dataflow pipeline can read that schema at runtime and adapt to changes automatically. This avoids the need to hardcode or manually update schemas, reducing pipeline restarts when the source schema evolves.

Answer

Convert all Avro files to CSV before ingestion and use a fixed CSV schema.

Answer

Use a BigQuery load job with schema autodetect instead of Dataflow.

Answer

Use a fixed Avro schema in your pipeline code and manually update it each time the source schema changes.

Question 15

You have a Dataflow batch pipeline that processes data from Cloud Storage and writes to BigQuery. The pipeline uses a custom DoFn that sometimes throws exceptions due to malformed input records. You want to ensure that the pipeline continues processing valid records while logging the malformed ones for later analysis, without failing the entire job. Which Dataflow feature should you use?

Accepted Answer

Use a side output to capture and log the malformed records.. The correct approach is to use a side output to capture malformed records. Side outputs allow you to emit elements that fail processing to a separate PCollection, which can then be written to a dead-letter sink for analysis. This enables the pipeline to continue processing valid records without failing. Other options either do not exist, drop data, or are not applicable to batch pipelines. Side outputs are a core Dataflow feature for error handling.

Answer

Configure the pipeline to use a dead-letter queue in Pub/Sub.

Answer

Set the pipeline's failure mode to 'continue' in the pipeline options.

Answer

Use a ParDo with a try-catch block and log the exception, then drop the record.