PDE Ingesting and Processing the Data Practice Test 5 — 15 Questions

Question 1

Your team has a Dataflow pipeline that reads from BigQuery, transforms data, and writes to GCS. The pipeline is failing with 'Out of Memory' errors on the worker nodes. The input data is large but fits within the total cluster memory. Which configuration change is most likely to resolve the issue without increasing costs significantly?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use a worker machine type with more memory, such as n2-highmem.. The default Dataflow worker machine type may have insufficient memory per core for the pipeline's operations. Using a high-memory machine type (e.g., n2-highmem) increases memory per worker without necessarily increasing the number of workers, thus controlling costs.

Answer

Shard the input into smaller reads using a BigQuery query.

Answer

Increase the disk size per worker.

Answer

Enable Dataflow Prime with vertical autoscaling.

Question 2

You need to react to changes in a GCS bucket (e.g., new object creation) and trigger a Cloud Run service to process the new file. Which Google Cloud service should you use to route the event?

Accepted Answer

Eventarc. Eventarc is the correct choice because it is purpose-built to route events from Google Cloud sources (like Cloud Storage) to Cloud Run. It directly supports Cloud Storage audit logs and Pub/Sub event triggers, allowing you to react to object creation events without custom middleware. Eventarc handles the event routing, filtering, and delivery to your Cloud Run service automatically.

Answer

Pub/Sub directly with a Cloud Run subscription

Answer

Cloud Tasks

Answer

Cloud Scheduler

Question 3

You need to ingest streaming data from a custom application into BigQuery with exactly-once semantics and low latency. The data volume is up to 10 MB/s. Which TWO services should you combine?

Accepted Answer

Pub/Sub. Pub/Sub provides reliable, low-latency message ingestion, and Dataflow can read from Pub/Sub and write to BigQuery using the Storage Write API, which supports exactly-once semantics. The Storage Write API with committed mode ensures exactly-once delivery.

Answer

Cloud Functions

Answer

BigQuery legacy streaming inserts

Answer

Datastream

Question 4

Your company has a Dataproc cluster that runs Spark jobs. You need to choose between RDDs, DataFrames, and Datasets for a new job that performs complex aggregations on structured data. Which TWO statements are correct regarding performance and ease of use?

Accepted Answer

DataFrames store data in a columnar format, allowing better compression.. DataFrames are optimized with Catalyst optimizer and Tungsten execution, providing better performance than RDDs for structured data. Datasets combine type safety with optimized execution, but for most analytics workloads, DataFrames are sufficient and simpler.

Answer

DataFrames and Datasets are both available in PySpark.

Answer

RDDs are easier to use than DataFrames for complex aggregations.

Answer

Datasets provide compile-time type safety and are always faster than DataFrames.

Question 5

You are building a BigQuery table that contains nested and repeated fields (e.g., order with line items). You need to write a query that counts the number of line items per order. Which TWO SQL functions/techniques can you use?

Accepted Answer

UNNEST with COUNT. UNNEST flattens the repeated line items array into individual rows, allowing COUNT to aggregate the number of line items per order. Option D is correct because ARRAY_LENGTH directly returns the number of elements in the repeated field array, which corresponds to the line item count.

Answer

Window function ROW_NUMBER

Answer

STRUCT with aggregation

Answer

SELECT * EXCEPT

Question 6

You are designing a Dataflow pipeline that reads from Pub/Sub and writes to BigQuery. Some incoming messages are malformed and fail to parse. How should you handle these messages to ensure the pipeline continues processing without data loss?

Accepted Answer

Write malformed messages to a dead-letter sink (e.g., Pub/Sub topic or GCS) and continue processing. The recommended pattern is to use a dead-letter queue (e.g., a separate Pub/Sub topic or a GCS bucket) to store failed messages after a retry threshold is reached. This preserves messages for later analysis without blocking the main pipeline.

Answer

Configure Pub/Sub to retry indefinitely until the message is processed

Answer

Use a try-catch block in the pipeline and ignore malformed messages

Answer

Set the pipeline to fail and alert the team via Cloud Monitoring

Question 7

Which Google Cloud service is designed to replicate data from MySQL, PostgreSQL, and Oracle databases to BigQuery or Cloud Storage in near real-time?

Accepted Answer

Datastream. Datastream is a serverless CDC service that ingests change data from relational databases into GCS or BigQuery.

Answer

Cloud Data Fusion

Answer

Dataflow

Answer

Pub/Sub

Question 8

You are migrating an on-premises Kafka cluster to Google Cloud. The cluster has 50 topics with a total throughput of 200 MB/s. You want to minimize operational overhead. Which approach is the most cost-effective?

Accepted Answer

Use Dataproc to run Kafka on a managed cluster. Managed Kafka services are not available natively on GCP; the recommended approach is to run Kafka on Dataproc, which provides a managed Hadoop/Spark environment with autoscaling.

Answer

Use Pub/Sub with Kafka-compatible client libraries

Answer

Deploy Kafka on Compute Engine instances with a managed instance group

Answer

Use Cloud NAT to route traffic to an on-premises Kafka cluster

Question 9

A data engineer is creating a Dataflow Flex Template for a batch pipeline that reads from BigQuery and writes to Cloud Storage. They need to pass a runtime parameter for the output bucket. How should they define this parameter?

Accepted Answer

Define the parameter in the pipeline's code and use ValueProvider. Dataflow Flex Templates require runtime parameters to be defined as `ValueProvider` objects in the pipeline code. This allows the parameter value to be supplied at job submission time via the `--parameters` flag, enabling the same template to be reused with different output buckets without recompilation.

Answer

Set an environment variable in Cloud Shell

Answer

Use the --parameters flag with the pipeline options

Answer

Hardcode the bucket name in the template

Question 10

You are using Dataproc to run a Spark job that reads data from Cloud Storage, performs aggregations, and writes results back to Cloud Storage. The job is failing with out-of-memory errors on the shuffle. Which optimization should you apply?

Accepted Answer

Increase spark.sql.shuffle.partitions. For shuffle-heavy operations, increasing the number of partitions reduces the size of each partition, reducing memory pressure. Alternatively, using DataFrames with optimized serialization (e.g., Kryo) helps.

Answer

Use RDDs instead of DataFrames

Answer

Increase spark.executor.memory

Answer

Decrease the number of executors

Question 11

An organization needs to trigger a Cloud Run service whenever a new file is uploaded to a specific Cloud Storage bucket. Which service should they use to set up this event-driven architecture?

Accepted Answer

Eventarc with a trigger for Cloud Storage events. Eventarc can capture Cloud Storage events (e.g., OBJECT_FINALIZE) and route them to Cloud Run, Cloud Functions, or Workflows. It supports CloudEvents standard.

Answer

Pub/Sub notifications on the bucket with a push subscription to Cloud Run

Answer

Cloud Scheduler calling Cloud Run on a schedule

Answer

Cloud Functions with a GCS trigger

Question 12

A data engineer needs to schedule a nightly transfer of data from an Amazon S3 bucket to Cloud Storage. Which two steps are required to achieve this? (Choose TWO.)

Accepted Answer

Grant appropriate permissions to the transfer service account. The Storage Transfer Service uses a Google-managed service account to access the source S3 bucket. You must grant this service account the appropriate IAM permissions (e.g., `s3:GetObject` and `s3:ListBucket`) on the S3 bucket via an AWS IAM policy. Without these permissions, the transfer job cannot read the data from S3.

Answer

Configure a VPC peering between AWS and GCP

Answer

Use gsutil rsync in a cron job

Answer

Set up a Cloud Function to copy files

Question 13

A company is designing a real-time analytics pipeline using Pub/Sub and Dataflow. They need to ensure exactly-once processing and handle late-arriving data. Which two configurations should they implement? (Choose TWO.)

Accepted Answer

Enable exactly-once delivery on Pub/Sub subscription. To achieve exactly-once semantics, enable exactly-once delivery on the Pub/Sub subscription (option B). To handle late-arriving data, use a fixed window with allowed lateness and a trigger (option D). This configuration ensures that late data is still processed within the allowed lateness period, while the trigger allows early results to be emitted before the window closes.

Answer

Set up a global window with no triggers

Answer

Use Dataflow's default at-least-once mode

Answer

Write all data to Cloud Storage and then batch load to BigQuery

Question 14

A data engineer needs to ingest daily Salesforce reports into BigQuery without writing custom code. The reports are exported to an Amazon S3 bucket on a schedule. Which service should they use to automate the transfer?

Accepted Answer

BigQuery Data Transfer Service. The BigQuery Data Transfer Service (BQDTS) is the correct choice because it supports scheduled, automatic ingestion of Salesforce reports exported to Amazon S3 into BigQuery without requiring custom code. It can connect to S3 as a source and load the data into BigQuery tables on a schedule, handling schema detection and incremental updates. This meets the requirement of no-code automation from an S3 bucket.

Answer

Cloud Dataproc

Answer

Cloud Composer

Answer

Cloud Storage Transfer Service

Question 15

A company wants to migrate 500 TB of on-premises archival data to Cloud Storage. The data is stored on a SAN and the network link is limited to 1 Gbps. The migration must complete within 10 days. What is the MOST cost-effective approach?

Accepted Answer

Order a Transfer Appliance, copy data locally, and ship it to Google for ingestion.. The Transfer Appliance is designed for large-scale data migrations where network bandwidth is insufficient. With 500 TB at 1 Gbps, the theoretical transfer time is over 46 days, far exceeding the 10-day window. The appliance allows you to physically ship the data, bypassing network constraints entirely, making it the most cost-effective and timely solution.

Answer

Set up a Cloud VPN and use rsync over the encrypted connection.

Answer

Use BigQuery Data Transfer Service to load the data directly into BigQuery.

Answer

Use Storage Transfer Service to copy data from on-premises to GCS over the existing network.