PDE Ingesting and Processing the Data — 10 Questions

Question 1

A data engineer needs to load 10 TB of CSV files from Amazon S3 into Google BigQuery on a daily basis. Which service should they use to automate this transfer?

Accepted Answer

BigQuery Data Transfer Service. BigQuery Data Transfer Service is designed to automate data ingestion from external sources, including Amazon S3, into BigQuery on a scheduled basis. It supports CSV and other formats, handles large volumes (10 TB daily), and provides built-in scheduling, monitoring, and retry logic. This makes it the most efficient and managed solution for recurring S3-to-BigQuery loads.

Answer

Dataproc

Answer

Cloud Data Fusion

Answer

Storage Transfer Service

Question 2

Your company uses Kafka for event streaming. You want to run Kafka on Google Cloud with the ability to auto-scale clusters and use managed infrastructure. Which service should you choose?

Accepted Answer

Dataproc. Dataproc is the correct choice because it is a managed Spark and Hadoop service on Google Cloud that supports running Kafka clusters via initialization actions. It allows auto-scaling of worker nodes and integrates with GCP storage and networking, providing the managed infrastructure required for Kafka event streaming. Note that Confluent Cloud is a third-party managed Kafka service, not a GCP-native service, and Cloud Pub/Sub is a messaging service, not a Kafka replacement. Cloud Dataflow is for data processing pipelines, not for running Kafka itself.

Answer

Cloud Pub/Sub

Answer

Confluent Cloud on GCP

Answer

Cloud Dataflow

Question 3

You need to perform a one-time migration of historical data from an on-premises Teradata data warehouse to BigQuery. The data volume is 50 TB and you have a high-speed network connection (10 Gbps). What is the most efficient way to load the data?

Accepted Answer

Use BigQuery Data Transfer Service for Teradata. BigQuery Data Transfer Service for Teradata is designed for this purpose; it can directly connect to Teradata and transfer data to BigQuery. Exporting to CSV then loading via gsutil is possible but less efficient. Transfer Appliance is for offline transfer but you have high-speed network. Dataproc is not needed.

Answer

Export data from Teradata to CSV files, upload to GCS using gsutil, then load into BigQuery.

Answer

Use Dataproc to run a Spark job that reads from Teradata and writes to BigQuery.

Answer

Use Transfer Appliance to ship the data offline.

Question 4

You have a Dataflow pipeline that processes streaming data with high throughput. You notice that the pipeline is experiencing high latency and the workers are underutilized. Which Dataflow feature can automatically optimize resource allocation?

Accepted Answer

Dataflow Prime. Dataflow Prime is Google Cloud's next-generation Dataflow runner that automatically optimizes resource allocation, including right-sizing worker resources and using features like 'Vertical Autoscaling' to adjust memory per worker. It addresses exactly the scenario described — high latency with underutilized workers — by dynamically tuning resources rather than relying on static worker configurations.

Answer

Flex Templates

Answer

Horizontal autoscaling

Answer

Streaming Engine

Question 5

Your organization uses dbt (data build tool) for transformations on BigQuery. You need to run dbt models on a schedule and manage versions. Which Google Cloud service can execute dbt jobs in a serverless manner?

Accepted Answer

Cloud Composer. Cloud Composer is the correct choice because it is a fully managed, serverless (managed) Apache Airflow service on Google Cloud that can schedule and orchestrate dbt jobs, including version management through Airflow's DAGs and connections to BigQuery. It natively supports running dbt commands via BashOperator or KubernetesPodOperator, making it suitable for scheduled dbt transformations on BigQuery. Dataflow is a managed Apache Beam service for stream and batch data processing, not for orchestrating dbt jobs. Cloud Build is a CI/CD service for building and deploying artifacts, not a scheduler for recurring dbt runs. Cloud Run runs containerized applications serverlessly but lacks built-in scheduling and orchestration for dbt workflows.

Answer

Dataflow

Answer

Cloud Build

Answer

Cloud Run

Question 6

An organization wants to ingest on-premises Oracle database changes into BigQuery for real-time analytics with minimal latency. The Oracle database is version 19c and has a high transactional volume. Which Google Cloud service should they use?

Accepted Answer

Datastream. Datastream is the correct service because it is purpose-built for real-time, serverless change data capture (CDC) from Oracle databases (including 19c) to BigQuery. It uses LogMiner to read redo logs with minimal overhead, supporting high transactional volumes and sub-second latency without requiring custom code or manual schema management.

Answer

Pub/Sub

Answer

Dataflow

Answer

Storage Transfer Service

Question 7

A company runs a Dataflow pipeline that reads from Pub/Sub, transforms data, and writes to BigQuery. The pipeline uses classic templates and is deployed in batch mode. They notice that the pipeline does not scale well under high load, causing a backlog in Pub/Sub. Which improvement would BEST address the scaling issue?

Accepted Answer

Switch to Dataflow Prime to enable vertical autoscaling and right-fitting. The correct option is B: switching to Dataflow Prime enables vertical autoscaling (right-fitting), which dynamically adjusts worker memory and CPU per step to handle skewed or heavy loads better than horizontal-only scaling, directly addressing the poor scaling under high load. Since the pipeline is already in batch mode reading from Pub/Sub, the backlog is caused by insufficient per-worker resources or inefficient scaling, which Prime's right-fitting resolves. Option A is wrong because the pipeline is already in batch mode, so changing to batch is a no-op or contradictory. Option C is wrong because using a larger machine type for all workers is a static, manual change that over-provisions and does not adapt to load. Option D is wrong because manually increasing workers is a manual, horizontal-only fix that does not address per-worker resource bottlenecks and is not automatic or optimal.

Answer

Change the pipeline to use batch mode instead of streaming

Answer

Use a larger machine type for all workers

Answer

Increase the number of workers manually in the pipeline configuration

Question 8

An organization needs to transfer 50 TB of historical data from an on-premises Hadoop cluster to Google Cloud Storage. The network bandwidth is limited to 100 Mbps. Which transfer method is MOST cost-effective and time-efficient?

Accepted Answer

Transfer Appliance. Transfer Appliance is designed for petabyte-scale offline transfers, shipping physical devices to Google for upload, which is much faster than using limited network bandwidth for 50 TB.

Answer

Storage Transfer Service over the network

Answer

BigQuery Data Transfer Service for Hadoop

Answer

gsutil cp with parallel composite uploads

Question 9

A data engineer is designing a Dataflow pipeline in Python that reads from Pub/Sub, applies complex transformations using external libraries, and writes to BigQuery. The pipeline must be deployed as a reusable, version-controlled template that can be easily updated without re-uploading the pipeline code each time. Which approach should they use?

Accepted Answer

Use a Flex Template with a custom Docker image that contains the pipeline code and dependencies. A Flex Template allows you to package both the pipeline code and its dependencies (including external libraries) into a custom Docker image. This image is stored in Artifact Registry and can be version-controlled, enabling updates by simply rebuilding and pushing a new image tag without re-uploading the pipeline code each time. This meets the requirement for a reusable, version-controlled template that can be easily updated.

Answer

Use a Classic Template and store the pipeline code in a Cloud Storage bucket

Answer

Use Cloud Composer to trigger the pipeline each time with updated parameters

Answer

Use Dataflow Prime and deploy the pipeline directly from the Apache Beam SDK

Question 10

A financial services company needs to ingest real-time trade data from multiple sources into BigQuery for immediate fraud detection. The data volume is high (1 million messages per second) and each message must be available for queries within seconds. They are considering the Storage Write API. Which stream mode should they choose to balance data availability and cost?

Accepted Answer

Buffered mode. Buffered mode (option C) is correct because it provides the best balance between data availability and cost for high-volume, real-time fraud detection. In buffered mode, data is written to BigQuery's managed storage within seconds, making it available for queries almost immediately, while the cost is lower than committed mode because buffered mode does not require an additional commit step. This mode is ideal for streaming use cases where latency is critical but cost efficiency is also a priority.

Answer

Legacy streaming inserts

Answer

Pending mode

Answer

Committed mode