PDE Practice Test 4 — 25 Questions

Question 1

A company wants to transfer 500 TB of data from an on-premises Hadoop cluster to Google Cloud Storage (GCS) for processing with Dataproc. The on-premises network has a 1 Gbps dedicated link to Google Cloud. The data must be transferred as quickly as possible, minimizing network usage. Which transfer method should they use?

Accepted Answer

Use Transfer Appliance to physically ship the data.. Transfer Appliance is the correct method because the dataset is 500 TB and the network link is only 1 Gbps. At 1 Gbps, the theoretical maximum transfer time is over 46 days, and real-world throughput (due to overhead, congestion, and Hadoop data characteristics) would be even longer. Transfer Appliance physically ships the data, bypassing the network bottleneck entirely and minimizing network usage, which is the stated requirement.

Answer

Use Storage Transfer Service over the 1 Gbps link.

Answer

Use gsutil cp in parallel with multiple threads.

Answer

Use BigQuery Data Transfer Service for Hadoop.

Question 2

A data engineer needs to schedule a nightly transfer of data from an Amazon S3 bucket to Cloud Storage. Which two steps are required to achieve this? (Choose TWO.)

Accepted Answer

Grant appropriate permissions to the transfer service account. Option A is correct because the Storage Transfer Service uses a Google-managed service account to access the source S3 bucket. You must grant this service account the appropriate IAM permissions (e.g., `s3:GetObject` and `s3:ListBucket`) on the S3 bucket via an AWS IAM policy. Without these permissions, the transfer job cannot read the data from S3.

Answer

Configure a VPC peering between AWS and GCP

Answer

Use gsutil rsync in a cron job

Answer

Set up a Cloud Function to copy files

Question 3

Your Dataflow pipeline reads from Pub/Sub, performs transformations, and writes to BigQuery. You notice that the pipeline's autoscaling is not keeping up with sudden spikes in traffic, causing increased lag. The pipeline uses Classic Templates. Which change would most effectively improve autoscaling responsiveness?

Accepted Answer

Enable Dataflow Streaming Engine on the pipeline.. Enabling Dataflow Streaming Engine reduces the overhead of checkpointing and state management by offloading them to the service side, which allows the pipeline to scale more quickly in response to sudden traffic spikes. This directly addresses the autoscaling lag because Streaming Engine decouples compute from state, enabling faster worker adjustments without the bottleneck of persistent disk-based shuffle.

Answer

Switch to Dataflow Prime with Vertical Autoscaling enabled.

Answer

Increase the initial number of workers to handle the spike.

Answer

Use Flex Templates instead of Classic Templates.

Question 4

You are designing a streaming pipeline that ingests events from Pub/Sub, enriches them with a machine learning model, and writes the results to BigQuery. The ML model is deployed on Cloud Run and has a high latency (500ms per request). You need to minimize the impact of slow ML inference on the overall pipeline throughput. Which approach should you take?

Accepted Answer

Use Dataflow to write events to Pub/Sub, then use a separate Dataflow pipeline that batches calls to Cloud Run.. Option A is correct because it uses Dataflow to batch events before sending them to Cloud Run, which amortizes the 500ms per-request latency over multiple events, significantly increasing throughput. By writing events to Pub/Sub and then processing them in a separate Dataflow pipeline with batched calls, you decouple the ingestion from the inference and avoid blocking on each individual request.

Answer

Increase the number of Dataflow workers to compensate for the latency.

Answer

Use Cloud Functions to call Cloud Run and write directly to BigQuery.

Answer

Use Dataflow's ParDo with synchronous calls to Cloud Run for each element.

Question 5

You are building a BigQuery table that contains nested and repeated fields (e.g., order with line items). You need to write a query that counts the number of line items per order. Which TWO SQL functions/techniques can you use?

Accepted Answer

UNNEST with COUNT. Option B is correct because UNNEST flattens the repeated line items array into individual rows, allowing COUNT to aggregate the number of line items per order. Option D is correct because ARRAY_LENGTH directly returns the number of elements in the repeated field array, which corresponds to the line item count.

Answer

Window function ROW_NUMBER

Answer

STRUCT with aggregation

Answer

SELECT * EXCEPT

Question 6

Which Dataflow feature automatically scales the number of workers based on the pipeline's current workload, and also selects the optimal machine type for each worker based on the pipeline's resource requirements?

Accepted Answer

Dataflow Prime. Dataflow Prime is the correct answer because it is the only Dataflow feature that provides both automatic worker scaling (horizontal autoscaling) and intelligent machine type selection (vertical autoscaling). It dynamically adjusts the number of workers based on the pipeline's current workload and selects the optimal machine type (e.g., CPU, memory, or accelerator-optimized) for each worker based on the pipeline's resource requirements, such as CPU utilization, memory pressure, or shuffle throughput.

Answer

Dataflow Shuffle

Answer

Dataflow Streaming Engine

Answer

Dataflow Flex Templates

Question 7

A data engineer needs to create a Dataflow pipeline that reads from Pub/Sub, applies a Python transformation, and writes to BigQuery. The pipeline should be reusable across environments with different parameters. Which deployment method is most appropriate?

Accepted Answer

Flex Template. Flex Templates (Option B) are the most appropriate deployment method because they allow you to package a custom Docker image containing your Python transformation code and dependencies, making the pipeline reusable across environments with different runtime parameters. Unlike Classic Templates, Flex Templates support arbitrary pipeline code and can be parameterized at runtime via the Dataflow UI or API, which is essential for a multi-environment deployment strategy.

Answer

Classic Template

Answer

Direct pipeline submission with gcloud dataflow jobs run

Answer

Cloud Composer to trigger Dataflow jobs

Question 8

A data engineer needs to transfer 500 TB of archival data from an on-premises NAS to Cloud Storage. The on-premises network has limited bandwidth (100 Mbps). Which transfer method should they recommend?

Accepted Answer

Transfer Appliance. Option C is correct because the Transfer Appliance is a physical device designed for large-scale data transfers (up to petabytes) when network bandwidth is insufficient. With 500 TB of data and only 100 Mbps bandwidth, the theoretical transfer time would be over 500 days, making any online transfer method impractical. The Transfer Appliance bypasses network constraints entirely by shipping the data physically to Google Cloud.

Answer

Storage Transfer Service for on-premises

Answer

gsutil rsync

Answer

Dataflow pipeline reading from NAS

Question 9

A data engineer needs to load 2 TB of Avro files stored in Cloud Storage into BigQuery on a daily schedule. The schema is static and the data should overwrite the existing table each day. What is the most efficient way to accomplish this?

Accepted Answer

Use bq load with --replace flag in a cron job. The bq load command with the --replace flag is the most efficient method because it directly loads Avro files from Cloud Storage into BigQuery in a single, serverless operation without requiring any intermediate processing. Since the schema is static and the data overwrites the existing table daily, a simple cron job invoking bq load --replace is the simplest, fastest, and most cost-effective solution, avoiding the overhead of additional services like Dataflow or Data Transfer Service.

Answer

Create a BigQuery Data Transfer Service from Cloud Storage

Answer

Create a Dataflow pipeline to read Avro files and stream to BigQuery

Answer

Mount Cloud Storage as a filesystem and use SELECT INTO

Question 10

Drag and drop the steps to deploy a Cloud Dataflow pipeline from a template into the correct order.

Question 11

Drag and drop the steps to migrate an on-premises MySQL database to Cloud SQL using Database Migration Service into the correct order.

Question 12

A data engineer is creating a Dataflow Flex Template for a batch pipeline that reads from BigQuery and writes to Cloud Storage. They need to pass a runtime parameter for the output bucket. How should they define this parameter?

Accepted Answer

Define the parameter in the pipeline's code and use ValueProvider. Option D is correct because Dataflow Flex Templates require runtime parameters to be defined as `ValueProvider` objects in the pipeline code. This allows the parameter value to be supplied at job submission time via the `--parameters` flag, enabling the same template to be reused with different output buckets without recompilation.

Answer

Set an environment variable in Cloud Shell

Answer

Use the --parameters flag with the pipeline options

Answer

Hardcode the bucket name in the template

Question 13

Drag and drop the steps to set up Cloud IAP (Identity-Aware Proxy) for an App Engine app into the correct order.

Question 14

Drag and drop the steps to set up a Pub/Sub topic with a push subscription to an HTTPS endpoint into the correct order.

Question 15

Which BigQuery feature allows you to write data with exactly-once semantics, high throughput, and the ability to buffer data before making it available for queries?

Accepted Answer

Storage Write API with buffered mode. The Storage Write API with buffered mode (option D) is correct because it provides exactly-once semantics for data ingestion, high throughput via gRPC streaming, and the ability to buffer data in memory before making it available for queries. This mode allows you to commit rows in a stream, ensuring no duplicates, while the buffering stage gives you control over when data becomes visible in BigQuery.

Answer

BigQuery load jobs

Answer

BigQuery Data Transfer Service

Answer

Legacy streaming inserts

Question 16

A company wants to transform data using dbt (data build tool) on BigQuery. They have a CI/CD pipeline and need to version-control their transformations. Which setup is recommended?

Accepted Answer

Deploy dbt models in a Cloud Build pipeline that runs dbt run. Option B is correct because dbt is designed for version-controlled, SQL-based transformations, and integrating it with Cloud Build allows you to run `dbt run` as part of a CI/CD pipeline. This setup ensures that every change to dbt models is automatically tested and deployed, which aligns with the requirement for version control and automated deployment on BigQuery.

Answer

Create Dataflow pipelines for each transformation

Answer

Use Cloud Composer to orchestrate dbt jobs

Answer

Run dbt directly on BigQuery using scripting

Question 17

Which three of the following are valid BigQuery data loading methods? (Choose THREE.)

Accepted Answer

Data Transfer Service from Amazon S3. The BigQuery Data Transfer Service supports automated ingestion from Amazon S3, allowing you to schedule and manage recurring transfers of data stored in S3 buckets directly into BigQuery. This is a fully managed service that handles the extraction, transformation, and loading (ETL) process without requiring custom code or infrastructure.

Answer

Using Cloud SQL to write to BigQuery

Answer

Direct file upload via Cloud Dataproc

Question 18

A data engineer needs to ingest daily Salesforce reports into BigQuery without writing custom code. The reports are exported to an Amazon S3 bucket on a schedule. Which service should they use to automate the transfer?

Accepted Answer

BigQuery Data Transfer Service. The BigQuery Data Transfer Service (BQDTS) is the correct choice because it natively supports scheduled, automatic ingestion of Salesforce reports into BigQuery without requiring any custom code. It connects directly to the Salesforce API, extracts the reports, and loads them into BigQuery tables on a user-defined schedule, handling incremental updates and schema detection automatically.

Answer

Cloud Dataproc

Answer

Cloud Composer

Answer

Cloud Storage Transfer Service

Question 19

A data pipeline processes JSON files from Cloud Storage, transforms them using Apache Beam, and writes the output to BigQuery. Some records are malformed and cause the pipeline to fail. How should the engineer handle these errors to ensure the pipeline continues processing while preserving the malformed records for analysis?

Accepted Answer

Catch exceptions in a DoFn and write the malformed records to a separate Cloud Storage bucket using a FileIO sink.. Option D is correct because it allows the pipeline to continue processing by catching exceptions within a DoFn and writing malformed records to a separate Cloud Storage bucket using a FileIO sink. This preserves the malformed records for later analysis without blocking the main data flow, which is a standard pattern in Apache Beam for handling dead-letter records. The approach ensures fault tolerance while maintaining data integrity for debugging.

Answer

Set the pipeline to retry malformed records indefinitely until they succeed.

Answer

Use a side input to send malformed records to a dead letter queue in Pub/Sub for later reprocessing.

Answer

Log the malformed records to Stackdriver and skip them in the pipeline.

Question 20

A company runs Apache Kafka on Dataproc for real-time event streaming. They want to archive the Kafka topics to Cloud Storage for long-term retention and later analysis in BigQuery. Which approach is the most cost-effective and operationally simple?

Accepted Answer

Use Kafka Connect with the GCS Sink Connector to write directly to Cloud Storage. Option D is correct because Kafka Connect with the GCS Sink Connector is purpose-built for exactly this use case: it directly streams Kafka topics to Cloud Storage in Avro, Parquet, or JSON format without requiring intermediate processing clusters or services. This approach minimizes operational overhead (no Spark or Dataflow jobs to manage) and is cost-effective since it runs as a lightweight connector within the existing Kafka ecosystem, leveraging Dataproc's managed Kafka cluster.

Answer

Use Apache Spark streaming on Dataproc to read from Kafka and write to GCS

Answer

Use Kafka MirrorMaker to replicate topics to a second cluster that writes to GCS

Answer

Use the Pub/Sub connector to publish Kafka messages to Pub/Sub, then a Dataflow job to write to GCS