PDE Ingesting and Processing the Data Practice Test 4 — 15 Questions

Question 1

A data engineer needs to transfer 5 PB of historical data from an on-premises Hadoop cluster to Cloud Storage. The network bandwidth is limited to 1 Gbps, and the transfer must complete within 30 days. Which transfer method should they use?

Accepted Answer

Transfer Appliance. Option D is correct because the Transfer Appliance is a physical device designed for large-scale data transfers when network bandwidth is insufficient. With 5 PB of data and a 1 Gbps link, the theoretical maximum transfer time is over 500 days (5 PB × 8 bits/byte / 1 Gbps / 86400 seconds/day), far exceeding the 30-day window. The Transfer Appliance bypasses network constraints by shipping data physically to Google Cloud.

Answer

gsutil rsync over the internet

Answer

BigQuery Data Transfer Service

Answer

Storage Transfer Service for on-premises

Question 2

A company wants to stream real-time user click events from their web application into BigQuery for immediate analysis. Which combination of services is the most scalable and cost-effective for this use case?

Accepted Answer

Pub/Sub to Dataflow to BigQuery. Pub/Sub is the recommended service for ingesting streaming events, and Dataflow can read from Pub/Sub and write to BigQuery using the Storage Write API for high-throughput, exactly-once semantics.

Answer

Cloud Dataproc Spark Streaming to BigQuery

Answer

App Engine pushing logs to BigQuery

Answer

Cloud Functions writing directly to BigQuery

Question 3

You are designing a Dataflow pipeline that reads from Pub/Sub and writes to BigQuery. Some incoming messages are malformed and fail to parse. How should you handle these messages to ensure the pipeline continues processing without data loss?

Accepted Answer

Write malformed messages to a dead-letter sink (e.g., Pub/Sub topic or GCS) and continue processing. The recommended pattern is to use a dead-letter queue (e.g., a separate Pub/Sub topic or a GCS bucket) to store failed messages after a retry threshold is reached. This preserves messages for later analysis without blocking the main pipeline.

Answer

Configure Pub/Sub to retry indefinitely until the message is processed

Answer

Use a try-catch block in the pipeline and ignore malformed messages

Answer

Set the pipeline to fail and alert the team via Cloud Monitoring

Question 4

A data engineer needs to load 2 TB of Avro files stored in Cloud Storage into BigQuery on a daily schedule. The schema is static and the data should overwrite the existing table each day. What is the most efficient way to accomplish this?

Accepted Answer

Use bq load with --replace flag in a cron job. The bq load command with the --replace flag is the most efficient method because it directly loads Avro files from Cloud Storage into BigQuery in a single, serverless operation without requiring any intermediate processing. Since the schema is static and the data overwrites the existing table daily, a simple cron job invoking bq load --replace is the simplest, fastest, and most cost-effective solution, avoiding the overhead of additional services like Dataflow or Data Transfer Service.

Answer

Create a BigQuery Data Transfer Service from Cloud Storage

Answer

Create a Dataflow pipeline to read Avro files and stream to BigQuery

Answer

Mount Cloud Storage as a filesystem and use SELECT INTO

Question 5

Which Google Cloud service is designed to replicate data from MySQL, PostgreSQL, and Oracle databases to BigQuery or Cloud Storage in near real-time?

Accepted Answer

Datastream. Datastream is a serverless CDC service that ingests change data from relational databases into GCS or BigQuery.

Answer

Cloud Data Fusion

Answer

Dataflow

Answer

Pub/Sub

Question 6

A company needs to run a Spark ML training job on a Dataproc cluster with high memory per node, but the cluster should automatically scale down when idle to save costs. Which configuration should they use?

Accepted Answer

Use a Dataproc cluster with preemptible secondary workers and cluster autoscaling. Option D is correct because it combines preemptible secondary workers for cost-effective high-memory compute with cluster autoscaling, which automatically scales down the cluster when idle. Preemptible VMs are ideal for stateless Spark ML training tasks, and autoscaling ensures the cluster shrinks to save costs during inactivity. This configuration meets the requirement of high memory per node (via primary workers) while minimizing costs through idle scaling.

Answer

Use a single-node cluster with preemptible VMs

Answer

Enable Dataproc's default autoscaling with primary workers as preemptible

Answer

Create a cluster with custom machine types and no autoscaling

Question 7

You are migrating an on-premises Kafka cluster to Google Cloud. The cluster has 50 topics with a total throughput of 200 MB/s. You want to minimize operational overhead. Which approach is the most cost-effective?

Accepted Answer

Use Dataproc to run Kafka on a managed cluster. Managed Kafka services are not available natively on GCP; the recommended approach is to run Kafka on Dataproc, which provides a managed Hadoop/Spark environment with autoscaling.

Answer

Use Pub/Sub with Kafka-compatible client libraries

Answer

Deploy Kafka on Compute Engine instances with a managed instance group

Answer

Use Cloud NAT to route traffic to an on-premises Kafka cluster

Question 8

A data engineer is creating a Dataflow Flex Template for a batch pipeline that reads from BigQuery and writes to Cloud Storage. They need to pass a runtime parameter for the output bucket. How should they define this parameter?

Accepted Answer

Define the parameter in the pipeline's code and use ValueProvider. Option D is correct because Dataflow Flex Templates require runtime parameters to be defined as `ValueProvider` objects in the pipeline code. This allows the parameter value to be supplied at job submission time via the `--parameters` flag, enabling the same template to be reused with different output buckets without recompilation.

Answer

Set an environment variable in Cloud Shell

Answer

Use the --parameters flag with the pipeline options

Answer

Hardcode the bucket name in the template

Question 9

Which BigQuery feature allows you to write data with exactly-once semantics, high throughput, and the ability to buffer data before making it available for queries?

Accepted Answer

Storage Write API with buffered mode. The Storage Write API with buffered mode (option D) is correct because it provides exactly-once semantics for data ingestion, high throughput via gRPC streaming, and the ability to buffer data in memory before making it available for queries. This mode allows you to commit rows in a stream, ensuring no duplicates, while the buffering stage gives you control over when data becomes visible in BigQuery.

Answer

BigQuery load jobs

Answer

BigQuery Data Transfer Service

Answer

Legacy streaming inserts

Question 10

A company wants to transform data using dbt (data build tool) on BigQuery. They have a CI/CD pipeline and need to version-control their transformations. Which setup is recommended?

Accepted Answer

Deploy dbt models in a Cloud Build pipeline that runs dbt run. Option B is correct because dbt is designed for version-controlled, SQL-based transformations, and integrating it with Cloud Build allows you to run `dbt run` as part of a CI/CD pipeline. This setup ensures that every change to dbt models is automatically tested and deployed, which aligns with the requirement for version control and automated deployment on BigQuery.

Answer

Create Dataflow pipelines for each transformation

Answer

Use Cloud Composer to orchestrate dbt jobs

Answer

Run dbt directly on BigQuery using scripting

Question 11

You are using Dataproc to run a Spark job that reads data from Cloud Storage, performs aggregations, and writes results back to Cloud Storage. The job is failing with out-of-memory errors on the shuffle. Which optimization should you apply?

Accepted Answer

Increase spark.sql.shuffle.partitions. For shuffle-heavy operations, increasing the number of partitions reduces the size of each partition, reducing memory pressure. Alternatively, using DataFrames with optimized serialization (e.g., Kryo) helps.

Answer

Use RDDs instead of DataFrames

Answer

Increase spark.executor.memory

Answer

Decrease the number of executors

Question 12

An organization needs to trigger a Cloud Run service whenever a new file is uploaded to a specific Cloud Storage bucket. Which service should they use to set up this event-driven architecture?

Accepted Answer

Eventarc with a trigger for Cloud Storage events. Eventarc can capture Cloud Storage events (e.g., OBJECT_FINALIZE) and route them to Cloud Run, Cloud Functions, or Workflows. It supports CloudEvents standard.

Answer

Pub/Sub notifications on the bucket with a push subscription to Cloud Run

Answer

Cloud Scheduler calling Cloud Run on a schedule

Answer

Cloud Functions with a GCS trigger

Question 13

A data engineer needs to schedule a nightly transfer of data from an Amazon S3 bucket to Cloud Storage. Which two steps are required to achieve this? (Choose TWO.)

Accepted Answer

Grant appropriate permissions to the transfer service account. Option A is correct because the Storage Transfer Service uses a Google-managed service account to access the source S3 bucket. You must grant this service account the appropriate IAM permissions (e.g., `s3:GetObject` and `s3:ListBucket`) on the S3 bucket via an AWS IAM policy. Without these permissions, the transfer job cannot read the data from S3.

Answer

Configure a VPC peering between AWS and GCP

Answer

Use gsutil rsync in a cron job

Answer

Set up a Cloud Function to copy files

Question 14

Which three of the following are valid BigQuery data loading methods? (Choose THREE.)

Accepted Answer

Data Transfer Service from Amazon S3. The BigQuery Data Transfer Service supports automated ingestion from Amazon S3, allowing you to schedule and manage recurring transfers of data stored in S3 buckets directly into BigQuery. This is a fully managed service that handles the extraction, transformation, and loading (ETL) process without requiring custom code or infrastructure.

Answer

Using Cloud SQL to write to BigQuery

Answer

Direct file upload via Cloud Dataproc

Question 15

A company is designing a real-time analytics pipeline using Pub/Sub and Dataflow. They need to ensure exactly-once processing and handle late-arriving data. Which two configurations should they implement? (Choose TWO.)

Accepted Answer

Enable exactly-once delivery on Pub/Sub subscription. To achieve exactly-once semantics in Dataflow, enable exactly-once mode for the Pub/Sub source. To handle late data, use a sliding window with allowed lateness and a trigger to emit early results.

Answer

Set up a global window with no triggers

Answer

Use Dataflow's default at-least-once mode

Answer

Write all data to Cloud Storage and then batch load to BigQuery