PDE Ingesting and Processing the Data Practice Test 6 — 15 Questions

Question 1

A media analytics firm runs a Dataflow batch pipeline that reads roughly 40 TB of compressed log files from Cloud Storage each night and writes aggregated results to BigQuery. The pipeline currently uses TextIO with a glob pattern, and job completion now routinely exceeds the nightly window. The files are gzip-compressed and each is about 200 MB, and the team cannot change the file format. Which change should the data engineer make to improve throughput with the least rework?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use TextIO with the withCompressionType option plus a splittable compression codec, or decompress the files before the pipeline reads them.. The limiting factor is that gzip members cannot be split, so each 200 MB file is read by exactly one worker regardless of how many workers the job has. Making the input splittable — either by changing the compression codec or by pre-decompressing the files — lets Beam distribute byte-range reads across many workers. Options that add workers or shuffles do not increase per-file read parallelism and therefore cannot shorten the nightly run within the stated constraint of keeping the file format.

Answer

Set --numWorkers to a much higher value so Beam can assign more workers to the source.

Answer

Add a GroupByKey followed by a Reshuffle before the aggregation to force more parallelism.

Answer

Replace TextIO with AvroIO and ask upstream systems to emit Avro container files.

Question 2

A data platform team needs to ingest Avro files that arrive in a Cloud Storage bucket into BigQuery on a recurring schedule without writing application code. They want the load to handle schema changes in the Avro files automatically and to be observable through Cloud Logging. (Choose two.)

Accepted Answer

Use a BigQuery Data Transfer Service Cloud Storage transfer configured with Avro as the source format.. The BigQuery Data Transfer Service Cloud Storage transfer is the managed, code-free way to schedule Avro ingestion into BigQuery, and it handles schema evolution while exposing run history and logs for monitoring. Datastream targets CDC relational sources, scheduled queries cannot read Cloud Storage files, and a Cloud Run job re-introduces custom code, so the managed transfer is the right pairing.

Answer

Create a scheduled query that calls the BigQuery Storage Write API to append the Avro records.

Answer

Deploy a Cloud Run job that runs bq load on a Cloud Scheduler trigger for each file.

Answer

Configure a Datastream stream from Cloud Storage to BigQuery with Avro as the format.

Question 3

A financial institution runs a Dataflow pipeline that reads from Pub/Sub and writes to BigQuery using the Storage Write API in exactly-once mode. The pipeline processes stock trades and must guarantee that each trade is written exactly once. The team observes duplicate trades in BigQuery during a pipeline update. What is the most likely cause?

Accepted Answer

The exactly-once mode in Storage Write API requires the use of a deduplication key, which was not configured. Exactly-once writes with the Storage Write API require a deduplication mechanism, typically a primary key or a unique token per row. Without configuring this, the API cannot identify and discard duplicate writes, leading to duplicates. During pipeline updates, at-least-once processing can cause retries, and without deduplication, these retries result in duplicate rows.

Answer

The pipeline update caused a change in the BigQuery table schema, leading to duplicates

Answer

Pub/Sub message IDs are not unique, causing the pipeline to treat distinct messages as duplicates

Answer

The Storage Write API does not support exactly-once semantics with Pub/Sub as a source

Question 4

A gaming company writes player event records to Cloud Bigtable and needs a downstream analytics copy in BigQuery that stays within a few minutes of the source. The Bigtable schema uses a row key of player ID plus reverse timestamp, and events are immutable once written. The team wants a managed, low-maintenance path that does not require running their own change-data-capture agents on Bigtable. Which approach should they choose?

Accepted Answer

Use the Bigtable to BigQuery template in the Dataflow job templates to export table rows into BigQuery on a schedule.. A managed Dataflow template that scans Bigtable and writes to BigQuery avoids custom code and operational overhead. Because the event records are immutable, periodic full or incremental scans produce a consistent analytics copy, and scheduling the template at short intervals keeps freshness within minutes. Change streams and HBase export both require custom pipeline or tooling ownership.

Answer

Enable Bigtable change streams and consume them with a custom Dataflow pipeline that writes to BigQuery.

Answer

Export Bigtable to Cloud Storage using the HBase export utility, then load the files into BigQuery with a scheduled load job.

Answer

Create a federated external table in BigQuery that points at the Bigtable instance and query it directly.

Question 5

You are designing a Dataflow pipeline that reads Avro files from Cloud Storage, validates each record against a schema, and writes valid records to BigQuery using the Storage Write API. Invalid records must be routed to a dead-letter bucket for later analysis. You need the pipeline to be idempotent on retries and to minimize duplicate writes to BigQuery. Which two actions should you take? (Choose two.)

Accepted Answer

Use a side output from the validation DoFn to send invalid records to Cloud Storage, and rely on the runner's default retry policy for the BigQuery sink.. Idempotent ingestion into BigQuery requires coordination between the runner's delivery guarantee and the sink's commit semantics. Exactly-once processing in Dataflow, combined with the Storage Write API's exactly-once streams and a deterministic record identifier derived from the source file, ensures retries do not create duplicate rows. Routing invalid records through a side output to a dead-letter bucket keeps the main path clean and satisfies the validation requirement without introducing competing consumers.

Answer

Configure the pipeline to use at-least-once processing and add a BigQuery MERGE statement after loading to deduplicate rows by a surrogate key.

Answer

Set the pipeline's windowing to a global window with a triggering strategy that emits on every element, so failures are visible immediately.

Answer

Write invalid records to the dead-letter bucket using a separate pipeline that reads from the same Pub/Sub subscription used by the main pipeline.

Question 6

A startup stores application logs as JSON files in Cloud Storage and wants to run ad hoc SQL queries on them with BigQuery, but the logs contain nested fields that the analysts want to query as structured columns. Which approach best meets this need?

Accepted Answer

Create a BigQuery external table over the JSON files and let BigQuery infer the nested schema.. An external table over the JSON files with schema autodetect lets BigQuery infer nested RECORD fields so analysts can query them as structured columns with standard SQL. The other approaches either lose the nested structure, cannot run SQL, or rely on unsupported mechanisms. This keeps the logs in place and delivers the ad hoc query experience the startup wants.

Answer

Load the JSON files into a single STRING column in a native table and parse with JSON functions at query time.

Answer

Use the Cloud Storage JSON API to query the files and return results to BigQuery.

Answer

Create a BigQuery table function that reads the bucket at query time with a UDF.

Question 7

You are using Cloud Data Fusion to build a batch pipeline that reads data from a Cloud Storage bucket and writes to BigQuery. You need to transform the data by applying a series of steps such as parsing CSV, filtering rows, and aggregating. Which Cloud Data Fusion component should you use to define these transformations visually?

Accepted Answer

Wrangler. Wrangler is the visual data transformation tool in Cloud Data Fusion. It allows users to connect to data sources, explore data, and apply transformations through a graphical interface. For batch pipelines that require parsing CSV, filtering, and aggregating, Wrangler provides the necessary capabilities without writing code. Pipeline Studio is for orchestrating the pipeline, but the actual transformation steps are defined in Wrangler.

Answer

CDAP

Answer

Pipeline Studio

Answer

Data Fusion UI

Question 8

Your team runs a Cloud Dataflow streaming pipeline that reads messages from a Pub/Sub subscription, applies a stateful deduplication transform keyed by message ID with a 10-minute allowed lateness, and writes results to BigQuery. During a regional outage, the pipeline is stopped and restarted after 45 minutes using the same subscription. After the restart, you notice duplicate records in BigQuery for messages that were already processed before the outage. The Pub/Sub subscription has a message retention duration of 7 days and the pipeline uses exactly-once processing mode with the BigQueryIO Storage Write API. What is the most likely cause of the duplicates?

Accepted Answer

The stateful deduplication transform lost its state because Dataflow state is not durable across pipeline restarts; therefore, previously seen message IDs are no longer recognized.. The stateful deduplication transform maintains a window of seen message IDs with a 10-minute allowed lateness. When the pipeline is stopped for 45 minutes, state older than the allowed lateness is discarded. Upon restart, messages that were processed before the outage but are still within the Pub/Sub retention period are redelivered and are no longer in state, so they are reprocessed and written to BigQuery, causing duplicates.

Answer

The BigQueryIO Storage Write API does not support exactly-once semantics when the pipeline is restarted; it only provides at-least-once guarantees.

Answer

The Dataflow pipeline's deduplication transform uses a session window instead of a fixed window, causing messages to be grouped incorrectly after the restart.

Answer

The Pub/Sub subscription was not configured with exactly-once delivery, so messages are delivered at least once, and the pipeline's deduplication logic failed to filter them.

Question 9

A financial firm uses a Dataflow streaming pipeline to process trades from Pub/Sub and write enriched records to BigQuery. The pipeline uses a 10-minute fixed window and default trigger settings. Traders complain that results for a given window sometimes appear in BigQuery up to 20 minutes after the window closes, even when no data is late. Which action should you take to make results available sooner after the window closes?

Accepted Answer

Investigate and reduce watermark lag by ensuring Pub/Sub message timestamps are set correctly and increasing the number of workers to process the backlog faster.. In Dataflow, the watermark determines when a window is considered complete. If the watermark lags because of unprocessed backlog or incorrect timestamps, the default trigger fires late even when no data is late. Fixing timestamp assignment and adding workers to drain the backlog advances the watermark, so windows close and emit on time.

Answer

Switch the windowing strategy from fixed windows to session windows with a gap duration of 10 minutes.

Answer

Use a processing-time trigger with a repeated firing interval of one minute instead of relying on the watermark.

Answer

Add an early trigger or set the allowed lateness to a smaller value and use a trigger that fires when the watermark passes the end of the window.

Question 10

An IoT platform publishes device telemetry to Pub/Sub at a sustained rate of hundreds of thousands of messages per second across many devices. A Dataflow streaming pipeline reads the topic and writes to BigQuery. The team observes that Pub/Sub is throttling the subscriber and the pipeline lags behind. They need to increase read throughput while keeping per-device ordering for stateful processing. What should they do?

Accepted Answer

Use a single subscription with ordering enabled and increase the number of Dataflow workers.. Ordering in Pub/Sub is scoped to a subscription and enforced through ordering keys, so a single subscription is required to keep per-device sequences intact. Raising the Dataflow worker count increases concurrent reads and processing, which lifts throughput past the throttling limit while preserving the ordering guarantee for stateful operators.

Answer

Increase the number of subscriptions on the topic and have the pipeline read from all of them.

Answer

Enable message deduplication on the subscription and raise the acknowledgement deadline.

Answer

Switch the pipeline to read from the topic using a BigQuery subscription instead of a Pub/Sub subscription.

Question 11

You are designing a Dataflow pipeline that reads from Pub/Sub and writes to BigQuery. The pipeline must guarantee exactly-once processing and handle duplicate messages that may be delivered by Pub/Sub. Which two approaches should you implement to achieve exactly-once semantics? (Choose two.)

Accepted Answer

Apply a deduplication transform using a stateful DoFn keyed by a unique message identifier.. To achieve exactly-once semantics, both source-side deduplication and sink-side transactional writes are necessary. Using the BigQuery Storage Write API with exactly-once semantics ensures that writes are idempotent and transactional, preventing duplicates at the sink. A stateful DoFn keyed by a unique message identifier deduplicates messages that may be redelivered by Pub/Sub. Together, these provide end-to-end exactly-once processing.

Answer

Configure the pipeline to use at-least-once processing and rely on BigQuery's ability to automatically remove duplicate rows.

Answer

Set the Pub/Sub subscription to exactly-once delivery and use the Dataflow runner's default processing mode.

Answer

Use Pub/Sub message deduplication by setting the message ID to a unique value.

Question 12

A financial services firm ingests trade confirmations as JSON files into a Cloud Storage bucket and must load them into BigQuery with schema validation and partitioning by trade date. Files arrive continuously and some contain malformed JSON that must be quarantined rather than failing the whole load. They want to use Dataflow batch jobs orchestrated by Cloud Composer and need reliable handling of the malformed records. Which two approaches should they use? (Choose two.)

Accepted Answer

Use a Dataflow pipeline that reads from Cloud Storage, parses JSON with a DoFn that routes parse failures to a dead-letter Cloud Storage prefix via a tagged output.. Malformed JSON needs an explicit routing step so a single bad file does not abort the batch load, and tagged outputs in a Dataflow DoFn provide that dead-letter pattern. The validated records then land in a table partitioned by trade date, which supports pruning and partition-level operations. Skipping bad records, truncating the table, or relying on autodetection all fail to quarantine bad data or preserve prior loads.

Answer

Enable BigQuery's automatic schema detection on every load job so that malformed JSON is coerced into string columns.

Answer

Use Cloud Composer's BigQueryInsertJobOperator to submit a load job and set writeDisposition to WRITE_TRUNCATE on each run.

Answer

Configure the BigQuery load job to skip the first 1,000 bad records and continue with the rest of the file.

Question 13

A financial company needs to ingest a continuous stream of stock trades from an on-premises Kafka cluster into BigQuery for real-time analytics. The company wants to minimize latency and operational overhead. Which Google Cloud service should they use to ingest the data?

Accepted Answer

Dataflow. Dataflow offers native integration with Kafka through the Apache Beam KafkaIO connector, allowing direct ingestion from on-premises Kafka to BigQuery. It is a managed service that handles scaling and fault tolerance, minimizing operational overhead. Dataflow supports exactly-once processing and can achieve low latency, making it the best choice for real-time streaming analytics from Kafka.

Answer

Pub/Sub

Answer

BigQuery Data Transfer Service

Answer

Cloud Data Fusion

Question 14

A retail company ingests point-of-sale events into Pub/Sub and processes them with a Dataflow streaming pipeline that writes to BigQuery. They need the pipeline to handle spikes in event volume without losing data and to keep costs predictable. Which two actions should they take? (Choose two.)

Accepted Answer

Set the Dataflow pipeline to use --maxNumWorkers to cap the number of workers and enable autoscaling with THROUGHPUT_BASED.. To absorb spikes without loss, autoscaling with a bounded maximum worker count lets the pipeline scale up and back down while keeping cost predictable. A dead-letter topic preserves messages that fail processing, so they are not lost and do not block the subscription. Retention changes, global windows, and Streaming Engine do not address both the spike and cost requirements.

Answer

Switch the pipeline to use a global window with a triggering frequency of one hour to reduce write operations to BigQuery.

Answer

Increase the Pub/Sub message retention duration to the maximum of 31 days to prevent message loss during spikes.

Answer

Enable Streaming Engine and set --enable_streaming_engine=true to offload state management from workers.

Question 15

You are designing a Dataflow batch pipeline that reads from BigQuery using BigQueryIO.read(), processes data with a GroupByKey, and writes to Cloud Storage. The pipeline runs daily and processes 10 TB of data. You notice that the pipeline is slow and often fails with 'Shuffle operation failed' errors. You have verified that the BigQuery table is partitioned by date and the query selects only the previous day's data. What should you do to improve performance and reliability?

Accepted Answer

Replace the GroupByKey with Combine.perKey() to perform partial aggregation before the shuffle, reducing the amount of data shuffled.. The shuffle failure is likely due to data skew in the GroupByKey operation, where a few keys have a disproportionately large number of values, overwhelming a single worker. Combine.perKey() performs partial aggregation on each worker before shuffling, reducing the data volume and distributing the load more evenly. This addresses the root cause and improves both performance and reliability.

Answer

Use BigQueryIO.read() with the withMethod(Method.DIRECT_READ) option and ensure the query uses a partition filter to leverage partition pruning.

Answer

Switch to using BigQueryIO.readTableRows() with a query that filters on the partition column, and enable the BigQuery Storage Read API.

Answer

Increase the number of workers and use a larger machine type to provide more memory for the shuffle operation.