75 of 94 questions · Page 1/2 · Pde Ingestion Processing topic · Answers revealed
A company needs to stream data from a MySQL database to BigQuery with a latency under 10 seconds. They also need to handle schema changes automatically. Which TWO services should they combine?
Target for the streamed data.
Why this answer
Datastream captures CDC and can write to BigQuery directly. Pub/Sub is not needed if Datastream writes directly.
You are building a Dataflow pipeline in Python that reads messages from Pub/Sub, enriches them with data from a BigQuery table, and writes the results to BigQuery. The enrichment lookup table is large and changes infrequently. Which approach minimizes cost and latency?
Side inputs are ideal for distributing a static lookup table to all workers. The data can be refreshed on a schedule.
Why this answer
Using a side input that periodically reads the BigQuery table and caches it avoids querying BigQuery for every incoming message, which would be prohibitively expensive and high-latency. The side input is refreshed at a configurable interval (e.g., every 10 minutes) via a pipeline option, and the cached data is broadcast to all workers, enabling fast, in-memory lookups without per-element I/O. This approach minimizes cost by reducing BigQuery API calls and minimizes latency by avoiding synchronous queries for each message.
Exam trap
Google often tests the misconception that querying BigQuery per message is acceptable in streaming pipelines, but the trap here is that candidates overlook the cost and latency implications of per-element I/O, especially with BigQuery's pricing model and query latency.
How to eliminate wrong answers
Option A is wrong because CoGroupByKey requires both inputs to be bounded or both unbounded streams; here, the BigQuery table is a bounded dataset, and Pub/Sub is unbounded, so CoGroupByKey would not work without windowing and would introduce unnecessary complexity and latency. Option B is wrong because querying BigQuery for every incoming message would cause extremely high API costs (BigQuery charges per byte processed) and high latency (each query takes hundreds of milliseconds to seconds), making it impractical for a streaming pipeline. Option D is wrong because storing the lookup in state per key would require partitioning the lookup table across keys, which is inefficient for a large, infrequently changing table; state is per-key and not shared across keys, so each worker would need to load and maintain its own copy, leading to memory waste and complex state management.
You need to perform a one-time migration of historical data from an on-premises Teradata data warehouse to BigQuery. The data volume is 50 TB and you have a high-speed network connection (10 Gbps). What is the most efficient way to load the data?
This service automates the transfer from Teradata to BigQuery, handling schema and data types.
Why this answer
BigQuery Data Transfer Service for Teradata is designed for this purpose; it can directly connect to Teradata and transfer data to BigQuery. Exporting to CSV then loading via gsutil is possible but less efficient. Transfer Appliance is for offline transfer but you have high-speed network.
Dataproc is not needed.
A company uses Workflows to orchestrate a series of Google Cloud services for data processing. They need to call an external HTTP API as part of the workflow and handle potential failures with retries. Which Workflows feature should they use?
Retry policy allows specifying retry conditions and limits for a step.
Why this answer
Workflows provides a built-in retry policy that can be configured on individual steps to automatically retry an HTTP call upon transient failures (e.g., 5xx server errors or network timeouts). This allows the workflow to handle external API failures without custom code, using exponential backoff and a maximum retry count.
Exam trap
Google Cloud Workflows often tests the distinction between workflow orchestration features (retry, subworkflows, parallel, conditional) and candidates mistakenly choose parallel steps or subworkflows thinking they inherently provide fault tolerance, but only a retry policy directly addresses automatic retries on failure.
How to eliminate wrong answers
Option B is wrong because subworkflows are used to encapsulate reusable sequences of steps, not to handle retries on a single HTTP call. Option C is wrong because parallel steps execute multiple branches concurrently, which does not provide retry logic for a single failing step. Option D is wrong because conditional steps (e.g., switch/if-else) control the flow based on conditions but do not automatically retry a failed HTTP request.
Your team has a Dataflow pipeline that reads from BigQuery, transforms data, and writes to GCS. The pipeline is failing with 'Out of Memory' errors on the worker nodes. The input data is large but fits within the total cluster memory. Which configuration change is most likely to resolve the issue without increasing costs significantly?
High-memory machines provide more memory per core, addressing OOM.
Why this answer
The default Dataflow worker machine type may have insufficient memory per core for the pipeline's operations. Using a high-memory machine type (e.g., n2-highmem) increases memory per worker without necessarily increasing the number of workers, thus controlling costs.
A company uses Cloud Datastream to replicate data from a MySQL database to BigQuery in near real-time. Which TWO BigQuery features are automatically used by Datastream for optimal performance and consistency? (Choose TWO.)
Datastream uses the Storage Write API for streaming replication.
Why this answer
Datastream uses the BigQuery Storage Write API (option D) to stream change data capture (CDC) events into BigQuery with exactly-once semantics and high throughput. It also automatically creates a materialized view (option E) that merges the change stream into the final table, ensuring consistent, near real-time replication without manual merge logic.
Exam trap
Google often tests the misconception that Datastream requires an intermediate Dataflow pipeline or legacy streaming inserts, when in fact it natively leverages the Storage Write API and materialized views for optimal performance and consistency.
Which Dataflow feature automatically scales the number of workers based on the pipeline's current workload, and also selects the optimal machine type for each worker based on the pipeline's resource requirements?
Dataflow Prime offers vertical autoscaling and right-fitting of worker machine types.
Why this answer
Dataflow Prime is the correct answer because it is the only Dataflow feature that provides both automatic worker scaling (horizontal autoscaling) and intelligent machine type selection (vertical autoscaling). It dynamically adjusts the number of workers based on the pipeline's current workload and selects the optimal machine type (e.g., CPU, memory, or accelerator-optimized) for each worker based on the pipeline's resource requirements, such as CPU utilization, memory pressure, or shuffle throughput.
Exam trap
Google often tests the distinction between horizontal autoscaling (adding/removing workers) and vertical autoscaling (changing machine type), and the trap here is that candidates assume Dataflow Shuffle or Streaming Engine handle scaling, when in fact they only optimize specific pipeline phases (shuffle or state management) without affecting worker count or machine type.
How to eliminate wrong answers
Option A is wrong because Dataflow Shuffle is a service that separates the shuffle operation from worker VMs, improving scalability and reliability, but it does not handle worker scaling or machine type selection. Option C is wrong because Dataflow Streaming Engine moves state storage and computation away from worker VMs for streaming pipelines, reducing resource overhead, but it does not automatically scale workers or select machine types. Option D is wrong because Dataflow Flex Templates allow you to package and reuse pipeline code with custom container images, but they do not provide any autoscaling or machine type optimization; scaling is handled separately by the Dataflow service.
Your team is processing a large dataset with Apache Beam on Dataflow. The pipeline sometimes fails due to transient errors when writing to a BigQuery sink. You need to ensure that failed records are not lost and can be reprocessed later without blocking the pipeline. What is the best approach?
This pattern isolates bad records, allows the pipeline to continue, and stores the failed records for later reprocessing.
Why this answer
Using a dead letter pattern with a side output to write failed records to a GCS bucket (or Pub/Sub) allows the pipeline to continue processing healthy records while failed records are stored for later analysis and reprocessing.
An organization needs to continuously replicate change data from a MySQL database to BigQuery with sub-minute latency. The database is running on-premises. Which Google Cloud service should they use?
Datastream is purpose-built for CDC from common databases to BigQuery or GCS.
Why this answer
Datastream is a serverless change data capture (CDC) service that can replicate from MySQL, PostgreSQL, and Oracle to BigQuery or GCS with low latency. Pub/Sub is a messaging service and does not natively connect to MySQL. Dataflow can process streams but requires a CDC connector.
BigQuery Data Transfer Service does not support live CDC from on-prem MySQL.
An organization wants to ingest on-premises Oracle database changes into BigQuery for real-time analytics with minimal latency. The Oracle database is version 19c and has a high transactional volume. Which Google Cloud service should they use?
Datastream supports real-time CDC from Oracle to BigQuery and GCS.
Why this answer
Datastream is the correct service because it is purpose-built for real-time, serverless change data capture (CDC) from Oracle databases (including 19c) to BigQuery. It uses LogMiner to read redo logs with minimal overhead, supporting high transactional volumes and sub-second latency without requiring custom code or manual schema management.
Exam trap
Candidates often incorrectly select Pub/Sub or Dataflow as generic streaming solutions, but Datastream is the only Google Cloud service that natively supports Oracle CDC without additional infrastructure.
How to eliminate wrong answers
Option B (Pub/Sub) is wrong because it is a messaging service for asynchronous event ingestion, not a CDC tool; it cannot directly read Oracle redo logs or handle schema evolution from a source database. Option C (Dataflow) is wrong because, while it can process streaming data, it requires a separate CDC connector (e.g., Debezium) and manual pipeline setup, adding complexity and latency compared to Datastream's managed Oracle-to-BigQuery integration. Option D (Storage Transfer Service) is wrong because it is designed for bulk file transfers (e.g., from on-premises NAS or S3) and cannot capture real-time database changes or connect to Oracle redo logs.
A data engineer needs to schedule a recurring transfer of data from a partner's Amazon S3 bucket to a Cloud Storage bucket for further processing. Which THREE components or configurations are necessary? (Choose 3)
Source and destination are required.
Why this answer
Scheduling, source and destination locations, and authentication/authorization are essential for a Storage Transfer Service job.
You are loading 10 GB of daily CSV files from a GCS bucket into a BigQuery table. The files contain some malformed rows that you want to skip. Which BigQuery load configuration should you use?
max_bad_records specifies the number of allowed bad records; if the number of bad records exceeds this, the load fails.
Why this answer
BigQuery allows setting max_bad_records in load jobs; records exceeding this threshold cause the job to fail. Setting max_bad_records to a value greater than 0 allows the load to succeed while skipping malformed rows.
A data pipeline processes JSON files from Cloud Storage, transforms them using Apache Beam, and writes the output to BigQuery. Some records are malformed and cause the pipeline to fail. How should the engineer handle these errors to ensure the pipeline continues processing while preserving the malformed records for analysis?
This follows the dead letter pattern: malformed records are written to a separate sink, allowing the pipeline to continue and enabling later analysis.
Why this answer
It allows the pipeline to continue processing by catching exceptions within a DoFn and writing malformed records to a separate Cloud Storage bucket using a FileIO sink. This preserves the malformed records for later analysis without blocking the main data flow, which is a standard pattern in Apache Beam for handling dead-letter records. The approach ensures fault tolerance while maintaining data integrity for debugging.
Exam trap
A common misconception in Google Cloud data pipelines is that error handling should involve retrying indefinitely or simply logging errors to Cloud Logging, but the correct approach is to isolate and persist malformed records to a durable sink like Cloud Storage for later analysis.
How to eliminate wrong answers
Option A is wrong because retrying malformed records indefinitely would cause the pipeline to hang or exhaust resources, as malformed records will never succeed due to inherent data issues. Option B is wrong because using a side input to send malformed records to a Pub/Sub dead letter queue is not a direct pattern in Apache Beam; side inputs are for broadcasting data to all elements, not for error handling, and Pub/Sub would require additional setup and does not inherently preserve the records for analysis without a separate sink. Option C is wrong because logging malformed records to Stackdriver and skipping them loses the data permanently, as logs are not designed for structured storage or reprocessing of the original records.
You are migrating an on-premises Kafka cluster to Google Cloud. The cluster has 50 topics with a total throughput of 200 MB/s. You want to minimize operational overhead. Which approach is the most cost-effective?
Why this answer
Managed Kafka services are not available natively on GCP; the recommended approach is to run Kafka on Dataproc, which provides a managed Hadoop/Spark environment with autoscaling.
A data engineer needs to schedule a nightly transfer of data from an Amazon S3 bucket to Cloud Storage. Which two steps are required to achieve this? (Choose TWO.)
Why this answer
The Storage Transfer Service uses a Google-managed service account to access the source S3 bucket. You must grant this service account the appropriate IAM permissions (e.g., `s3:GetObject` and `s3:ListBucket`) on the S3 bucket via an AWS IAM policy. Without these permissions, the transfer job cannot read the data from S3.
Exam trap
Google often tests the misconception that you can use a simple command-line tool like `gsutil rsync` in a cron job for production-scale scheduled transfers, but the correct approach is to use the managed Storage Transfer Service which handles permissions, scheduling, and reliability natively.
You are building a BigQuery table that contains nested and repeated fields (e.g., order with line items). You need to write a query that counts the number of line items per order. Which TWO SQL functions/techniques can you use?
UNNEST expands the array, then COUNT(*) gives the number of line items per order.
Why this answer
UNNEST flattens the repeated line items array into individual rows, allowing COUNT to aggregate the number of line items per order. Option D is correct because ARRAY_LENGTH directly returns the number of elements in the repeated field array, which corresponds to the line item count.
Exam trap
Google often tests the distinction between functions that operate on arrays directly (like ARRAY_LENGTH) versus those that require row-level expansion (like UNNEST), and candidates may mistakenly choose window functions or STRUCT-based aggregation that do not directly count array elements.
A data engineer needs to transfer 500 TB of archival data from an on-premises NAS to Cloud Storage. The on-premises network has limited bandwidth (100 Mbps). Which transfer method should they recommend?
Transfer Appliance is the best choice for large offline data transfer when network bandwidth is limited.
Why this answer
The Transfer Appliance is a physical device designed for large-scale data transfers (up to petabytes) when network bandwidth is insufficient. With 500 TB of data and only 100 Mbps bandwidth, the theoretical transfer time would be over 500 days, making any online transfer method impractical. The Transfer Appliance bypasses network constraints entirely by shipping the data physically to Google Cloud.
Exam trap
The exam often tests the misconception that any cloud-native tool (like Storage Transfer Service or gsutil) can handle large data volumes regardless of bandwidth, ignoring the physical reality of network transfer times for archival-scale data.
How to eliminate wrong answers
Option A is wrong because Storage Transfer Service for on-premises requires network connectivity and is designed for smaller, incremental transfers, not for 500 TB over a 100 Mbps link. Option B is wrong because gsutil rsync is a command-line tool that relies on network bandwidth and would take an impractical amount of time (over 500 days) to transfer 500 TB at 100 Mbps. Option D is wrong because a Dataflow pipeline reading from NAS would still need to stream data over the limited 100 Mbps network, resulting in the same bandwidth bottleneck and excessive transfer time.
You are migrating an on-premises PostgreSQL database to Cloud SQL. You need to continuously replicate changes to BigQuery for real-time analytics with minimal latency. Which service should you use?
Datastream is the managed CDC service that can stream changes from PostgreSQL to BigQuery with minimal latency.
Why this answer
Datastream is designed for change data capture (CDC) from databases like PostgreSQL, MySQL, and Oracle to BigQuery or GCS. It provides low-latency replication. Pub/Sub and Dataflow are not directly for CDC.
Storage Transfer Service is for file transfers, not database replication.
A data engineer needs to load 10 TB of CSV files from Amazon S3 into Google BigQuery on a daily basis. Which service should they use to automate this transfer?
BigQuery Data Transfer Service supports scheduled transfers from Amazon S3 directly into BigQuery.
Why this answer
Storage Transfer Service can transfer data from Amazon S3 to Google Cloud Storage, but it does not load directly into BigQuery. BigQuery Data Transfer Service can import from Amazon S3 directly into BigQuery tables. Other options are not suitable: Cloud Data Fusion is for ETL pipelines, not simple transfer; Transfer Appliance is for offline petabyte-scale transfers; Dataproc is for Spark/Hadoop jobs.
A company is designing a real-time analytics pipeline using Pub/Sub and Dataflow. They need to ensure exactly-once processing and handle late-arriving data. Which two configurations should they implement? (Choose TWO.)
Why this answer
To achieve exactly-once semantics, enable exactly-once delivery on the Pub/Sub subscription (option B). To handle late-arriving data, use a fixed window with allowed lateness and a trigger (option D). This configuration ensures that late data is still processed within the allowed lateness period, while the trigger allows early results to be emitted before the window closes.
A company wants to use dbt (data build tool) to transform data in BigQuery. They have a Cloud Storage bucket containing raw CSV files that are loaded daily into BigQuery via an external table. Which dbt feature should they use to modularize the transformation logic and handle dependencies between models?
Models define transformations and dependencies; ref() handles lineage and ordering.
Why this answer
C is correct because dbt models with the `ref()` function allow you to modularize SQL transformation logic and automatically handle dependencies between models. When you use `ref('model_name')`, dbt builds a dependency graph, ensuring models are executed in the correct order based on their references. This is essential for transforming raw data from an external table into a structured, analytics-ready dataset in BigQuery.
Exam trap
Candidates often confuse the purpose of dbt components: models with `ref()` manage transformation logic and dependencies, while tests handle data quality, snapshots track historical changes, and seeds load static data.
How to eliminate wrong answers
Option A is wrong because dbt tests are used for validating data quality (e.g., uniqueness, not null) and do not handle transformation logic or dependency management. Option B is wrong because dbt snapshots are designed to capture historical changes in slowly changing dimensions (Type 2 SCDs), not to modularize transformation logic or manage model dependencies. Option D is wrong because dbt seeds are used to load static CSV files directly into the warehouse as tables, not to transform data or manage dependencies between models.
A company wants to build an event-driven application that processes images uploaded to a Cloud Storage bucket. The processing takes up to 10 minutes per image and should be automatically triggered. Which compute option should they use?
Cloud Run can handle long-running requests (up to 60 minutes) and is triggered by Eventarc for GCS events.
Why this answer
Cloud Functions have a 9-minute timeout; Cloud Run can handle up to 60 minutes and is triggered by Eventarc for GCS events.
A data pipeline uses Pub/Sub to ingest events, a Dataflow streaming pipeline to process them, and writes results to BigQuery. The pipeline must handle occasional duplicate events without causing duplicate rows in BigQuery. What is the best approach?
Legacy streaming inserts use insertId to deduplicate within a short window.
Why this answer
BigQuery does not enforce primary keys; deduplication must be handled in the pipeline using idempotent writes or a dedup step.
A company is migrating on-premises Apache Kafka workloads to Google Cloud. They want to minimize changes to existing producer and consumer applications while leveraging managed services. Which TWO services should they consider? (Choose 2)
Managed Kafka cluster on Dataproc; compatible with existing applications.
Why this answer
Kafka on Dataproc provides a managed Kafka cluster that is fully compatible, minimizing application changes. Confluent Cloud on Google Cloud can be used but is not a Google-managed service; however, it is a viable partner solution. Pub/Sub is not Kafka API-compatible.
Dataflow is not a replacement for Kafka. BigQuery is a data warehouse, not a streaming broker.
A company wants to move data from an on-premises MySQL database to BigQuery for analytics. They need to capture all changes (inserts, updates, deletes) in near real-time and also perform an initial historical load. Which approach meets these requirements with minimal operational overhead?
Datastream handles both backfill and CDC seamlessly.
Why this answer
Datastream can perform a backfill of historical data and then stream CDC changes from MySQL to BigQuery in near real-time, providing a single service for both tasks.
A company uses Dataflow to process data with Apache Beam in Python. The pipeline reads from Pub/Sub, applies a ParDo that calls an external API for enrichment, and writes to BigQuery. The external API has rate limits and occasionally fails. To improve reliability, which THREE strategies should be implemented? (Choose 3)
Batching reduces the number of API calls, helping with rate limits.
Why this answer
Retry logic with exponential backoff, a dead letter queue for failed records, and batching requests reduce API pressure and handle failures gracefully.
You are designing a Dataflow pipeline that needs to exactly-once process events from Pub/Sub and write to BigQuery using the Storage Write API. The pipeline may restart and could reprocess some messages. What setting ensures exactly-once semantics for the output?
Committed mode guarantees exactly-once writes, and Dataflow can coordinate with Pub/Sub to avoid duplicates.
Why this answer
The Storage Write API supports exactly-once semantics when used with the 'committed' mode, which ensures each record is written exactly once. The pipeline also needs to use Pub/Sub with message IDs and deduplication. The other options either do not provide exactly-once or are unreliable.
A team wants to transfer data from an on-premises Hadoop cluster to Cloud Storage for processing. The cluster is located in a remote area with limited bandwidth. They need to transfer 500 TB of data. Which service should they use?
Offline physical appliance for large data transfers; ideal for remote areas with low bandwidth.
Why this answer
Transfer Appliance is designed for petabyte-scale offline transfers when bandwidth is limited.
A company wants to ingest data from an on-premises Oracle database into BigQuery in near real-time with minimal latency. The database has a high volume of inserts and updates. Which service should they use?
Datastream streams change data from Oracle, MySQL, PostgreSQL to BigQuery or GCS in near real-time.
Why this answer
Datastream is designed for CDC from Oracle and other sources to BigQuery or GCS in near real-time.
A data team needs to transfer 200 TB of data from Amazon S3 to GCS. The transfer must be incremental, and they need to monitor the transfer progress. Which THREE components should they use?
Provides dashboards and alerts for transfer progress.
Why this answer
Storage Transfer Service (STS) can transfer from S3 to GCS with incremental sync. Cloud Monitoring tracks progress. Service account for permissions.
Which BigQuery feature allows you to query data directly from Cloud Storage without loading it into BigQuery storage?
External tables in BigQuery reference data in GCS and can be queried directly.
Why this answer
BigQuery Omni is for multi-cloud, not external tables. External tables allow querying data in GCS.
A company wants to migrate their on-premises Teradata data warehouse to BigQuery. They need an automated, one-time transfer of historical data (10 TB) and ongoing incremental daily syncs. Which Google Cloud service should they use?
This service is designed to schedule and automate transfers from Teradata to BigQuery, both initial and incremental.
Why this answer
BigQuery Data Transfer Service supports Teradata as a source for both one-time and scheduled transfers. Storage Transfer Service is for file-based transfers. Datastream is for CDC, not Teradata.
Dataflow could be custom-built but Data Transfer Service is purpose-built for this scenario.
Your company uses Kafka for event streaming. You want to run Kafka on Google Cloud with the ability to auto-scale clusters and use managed infrastructure. Which service should you choose?
Dataproc supports running Kafka as an optional component on managed clusters, giving you control and scalability.
Why this answer
Dataproc is the correct choice because it is a managed Spark and Hadoop service on Google Cloud that supports running Kafka clusters via initialization actions. It allows auto-scaling of worker nodes and integrates with GCP storage and networking, providing the managed infrastructure required for Kafka event streaming. Note that Confluent Cloud is a third-party managed Kafka service, not a GCP-native service, and Cloud Pub/Sub is a messaging service, not a Kafka replacement.
Cloud Dataflow is for data processing pipelines, not for running Kafka itself.
Exam trap
The trap is that candidates may confuse third-party managed Kafka services (like Confluent Cloud) with GCP-native managed infrastructure, or assume Cloud Pub/Sub is equivalent to Kafka for event streaming, when Dataproc is the correct GCP-native service for running Kafka itself with auto-scaling and managed resources.
How to eliminate wrong answers
Option A is wrong because Cloud Pub/Sub is a fully managed messaging service, not a Kafka-compatible platform; it does not run Kafka clusters or support auto-scaling of Kafka-specific infrastructure. Option B is wrong because Confluent Cloud on GCP is a third-party managed Kafka service, not a native GCP service, and while it offers auto-scaling, the question asks for a service you choose to run Kafka on Google Cloud with managed infrastructure, implying a GCP-native solution; Confluent Cloud is a separate platform, not a GCP service. Option C is wrong because Cloud Dataflow is a stream and batch processing service based on Apache Beam, not a Kafka cluster management service; it can consume from Kafka but does not host or auto-scale Kafka clusters.
Your team uses dbt to transform data in BigQuery. You need to schedule dbt runs to refresh materialized tables and views every hour. The transformations include both full refreshes and incremental models. What is the most efficient way to orchestrate these dbt runs on Google Cloud?
Cloud Composer is managed Airflow, ideal for scheduling and orchestrating dbt runs with dependencies.
Why this answer
Cloud Composer (Airflow) is the recommended orchestration tool for complex workflows like dbt runs, supporting dependencies, retries, and scheduling. Cloud Scheduler alone cannot run dbt directly; it can trigger a Cloud Function to run dbt, but that is less maintainable. Cloud Build is CI/CD, not scheduling.
Using a cron job on Compute Engine is possible but not managed.
You are designing a near-real-time CDC pipeline to replicate changes from an on-premises PostgreSQL database to BigQuery for analytics. The source database has high transaction volume and you must ensure minimal impact on the source. Which Google Cloud service should you use to ingest the change data?
Datastream captures changes from the database logs and streams them to GCS or BigQuery directly, with low impact.
Why this answer
Datastream is purpose-built for CDC from MySQL, PostgreSQL, and Oracle to BigQuery or GCS. It reads the database logs (e.g., WAL) to capture changes with low latency and minimal impact on the source.
A data engineer needs to move 500 TB of archival data from an on-premises Hadoop cluster to Cloud Storage. The network bandwidth is limited to 100 Mbps, and the transfer must complete within 30 days. Which method is most cost-effective and reliable?
Transfer Appliance allows offline shipping of large data volumes, bypassing bandwidth limits.
Why this answer
With 100 Mbps, transferring 500 TB over the network would take > 500 days, exceeding the 30-day window. Transfer Appliance is designed for petabyte-scale offline transfers, making it the only feasible option.
An organization needs to transfer 50 TB of historical data from an on-premises Hadoop cluster to Google Cloud Storage. The network bandwidth is limited to 100 Mbps. Which transfer method is MOST cost-effective and time-efficient?
Transfer Appliance allows shipping data physically, bypassing network bandwidth limitations for large datasets.
Why this answer
Transfer Appliance is designed for petabyte-scale offline transfers, shipping physical devices to Google for upload, which is much faster than using limited network bandwidth for 50 TB.
A data analyst needs to transform nested and repeated fields in BigQuery. They have a table with a column of type ARRAY<STRUCT<...>>. Which SQL function should they use to flatten the array into individual rows for analysis?
UNNEST converts array elements into rows, allowing analysis of nested data.
Why this answer
UNNEST is used to flatten arrays into rows. STRUCT is used to group fields. CAST is for type conversion.
REPLACE is for string replacement.
You need to process a large volume of event data from Cloud Storage, apply complex transformations using Apache Spark, and then load the results into BigQuery. The data arrives in batches every hour. You want to minimize costs by using preemptible VMs. Which service should you use?
Dataproc clusters can use preemptible VMs for cost-efficient batch processing with Spark.
Why this answer
Dataproc supports preemptible (now called spot) VMs for cost savings. Dataflow does not support preemptible VMs for workers; it uses standard VMs. Cloud Composer is orchestration only.
BigQuery is not for running Spark.
A company uses dbt on BigQuery to transform data. They want to run dbt models on a schedule and manage environments (dev, prod). Which GCP service should they use to run dbt jobs?
Managed Airflow with DAGs, scheduling, and environment separation.
Why this answer
Cloud Composer is an Apache Airflow managed service that can schedule dbt runs.
A company wants to transfer 500 TB of data from an on-premises Hadoop cluster to Google Cloud Storage (GCS) for processing with Dataproc. The on-premises network has a 1 Gbps dedicated link to Google Cloud. The data must be transferred as quickly as possible, minimizing network usage. Which transfer method should they use?
Transfer Appliance can handle 500 TB in a single appliance, transferring the data offline within days.
Why this answer
Transfer Appliance is the correct method because the dataset is 500 TB and the network link is only 1 Gbps. At 1 Gbps, the theoretical maximum transfer time is over 46 days, and real-world throughput (due to overhead, congestion, and Hadoop data characteristics) would be even longer. Transfer Appliance physically ships the data, bypassing the network bottleneck entirely and minimizing network usage, which is the stated requirement.
Exam trap
The trap here is that candidates assume parallel transfers (gsutil cp) or managed services (Storage Transfer Service) can overcome bandwidth limitations, but they ignore the fundamental physics of a 1 Gbps link and the sheer size of 500 TB.
How to eliminate wrong answers
Option A is wrong because Storage Transfer Service still uses the 1 Gbps network link, which would take weeks to transfer 500 TB, failing the 'as quickly as possible' and 'minimizing network usage' requirements. Option B is wrong because gsutil cp with parallel threads still operates over the same 1 Gbps link and cannot exceed its bandwidth; it also does not minimize network usage. Option D is wrong because BigQuery Data Transfer Service for Hadoop is designed for scheduled, incremental loads from Hadoop to BigQuery, not for bulk initial transfer to GCS, and it still uses the network link.
A data engineer is creating a Dataflow Flex Template for a batch pipeline that reads from BigQuery and writes to Cloud Storage. They need to pass a runtime parameter for the output bucket. How should they define this parameter?
Why this answer
Dataflow Flex Templates require runtime parameters to be defined as `ValueProvider` objects in the pipeline code. This allows the parameter value to be supplied at job submission time via the `--parameters` flag, enabling the same template to be reused with different output buckets without recompilation.
Exam trap
Google often tests the distinction between defining a parameter (using `ValueProvider` in code) and supplying its value (using `--parameters` at submission), leading candidates to mistakenly choose Option B as the complete solution.
How to eliminate wrong answers
Option A is wrong because environment variables in Cloud Shell are not accessible to the Dataflow service at runtime; they are only available in the shell session and cannot be passed into a Flex Template job. Option B is wrong because the `--parameters` flag is used to supply values to `ValueProvider` parameters at job submission, but it does not define the parameter itself — the parameter must first be declared as a `ValueProvider` in the pipeline code. Option C is wrong because hardcoding the bucket name defeats the purpose of using a Flex Template, which is designed to be parameterized and reusable across different environments and runs.
You are using Dataproc to run a Spark job that reads data from Cloud Storage, performs aggregations, and writes results back to Cloud Storage. The job is failing with out-of-memory errors on the shuffle. Which optimization should you apply?
Why this answer
For shuffle-heavy operations, increasing the number of partitions reduces the size of each partition, reducing memory pressure. Alternatively, using DataFrames with optimized serialization (e.g., Kryo) helps.
You have a Dataflow pipeline that processes streaming data with high throughput. You notice that the pipeline is experiencing high latency and the workers are underutilized. Which Dataflow feature can automatically optimize resource allocation?
Dataflow Prime offers vertical autoscaling and right-fitting to optimize worker resources.
Why this answer
Dataflow Prime (also known as right-fitting) provides vertical autoscaling and resource optimization based on actual usage. Horizontal autoscaling is standard but may not address underutilization. Streaming engine is for scaling the streaming writes but not worker tuning.
Flex templates are for deployment, not runtime optimization.
A company wants to trigger a Cloud Run service whenever a new file is uploaded to a specific Cloud Storage bucket. Which event-driven solution should they use?
Eventarc natively supports Cloud Storage events and routes to Cloud Run.
Why this answer
Eventarc is the recommended service for routing events from Cloud Storage to Cloud Run because it provides a fully managed, event-driven architecture with built-in filtering and retry logic. When a new file is uploaded, Cloud Storage emits a notification that Eventarc captures and delivers directly to the Cloud Run service as an HTTP request, enabling serverless processing without polling or additional infrastructure.
Exam trap
The trap here is that candidates confuse Cloud Functions (option C) as the only serverless compute option for Cloud Storage events, overlooking that Eventarc is the modern, preferred service for routing events to Cloud Run, and that Pub/Sub (option D) requires manual setup not shown in the question.
How to eliminate wrong answers
Option B is wrong because Cloud Scheduler is a cron job service for scheduled, not event-driven, tasks; periodically polling a bucket introduces latency and inefficiency, and it cannot react instantly to uploads. Option C is wrong because Cloud Functions triggered by Cloud Storage is a valid event-driven approach, but the question specifically asks for a Cloud Run destination, and Cloud Functions cannot directly invoke Cloud Run without additional integration. Option D is wrong because Pub/Sub with a push subscription to Cloud Run requires manually configuring Cloud Storage to publish to Pub/Sub, which adds complexity and is not the native, recommended pattern for Cloud Storage events; Eventarc abstracts this by directly managing the event flow from Cloud Storage to Cloud Run.
A company uses BigQuery's Storage Write API in committed mode to stream data. They notice that some writes are failing with 'DEADLINE_EXCEEDED' errors during peak traffic. The pipeline is a Dataflow job using the Beam SDK. What is the MOST likely cause and solution?
High traffic can cause RPCs to exceed the default timeout; increasing the timeout allows more time for acknowledgment.
Why this answer
Committed mode requires immediate acknowledgment from BigQuery. Under high traffic, the default timeout may be exceeded. The solution is to increase the timeout or switch to buffered mode, which provides higher throughput by batching.
The error is not due to schema mismatch or permissions; those would cause different errors. Pub/Sub is not involved in the write path.
A company wants to use Eventarc to trigger a Cloud Run service when new objects are created in a GCS bucket. They also need to filter events for a specific bucket and object prefix. Which THREE resources must exist or be created?
The source of events.
Why this answer
Eventarc trigger, Cloud Run service, and the GCS bucket. The trigger references the bucket and prefix.
Which Google Cloud service is designed to replicate data from MySQL, PostgreSQL, and Oracle databases to BigQuery or Cloud Storage in near real-time?
Why this answer
Datastream is a serverless CDC service that ingests change data from relational databases into GCS or BigQuery.
A company wants to orchestrate a multi-step data processing workflow that includes calling a Cloud Run service, waiting for its completion, and then running a BigQuery query. The workflow should be serverless and integrate with Cloud Events. Which Google Cloud service should they use?
Workflows is a serverless orchestration service that can call Cloud Run, BigQuery, and other APIs, and can be triggered by Eventarc.
Why this answer
Cloud Workflows is the correct choice because it is a serverless workflow orchestrator that can coordinate multi-step processes involving Cloud Run and BigQuery. It natively supports waiting for asynchronous operations (like Cloud Run job completion) via its 'call' and 'wait' steps, and it can trigger subsequent steps such as BigQuery queries. Additionally, Cloud Workflows can be triggered by Cloud Events, making it fully integrated with the event-driven architecture described.
Exam trap
The trap here is that candidates confuse Google Cloud's Eventarc (event routing) with workflow orchestration, assuming that routing events alone can handle sequencing and waiting, when in fact Eventarc lacks the state management and step coordination required for multi-step workflows.
How to eliminate wrong answers
Option A is wrong because Eventarc is a service for routing events from various sources to targets (like Cloud Run, Cloud Functions), but it does not provide workflow orchestration capabilities such as waiting for completion or sequencing steps. Option C is wrong because Cloud Composer is a managed Apache Airflow service that is not serverless (it requires provisioning and managing a cluster of workers) and is overkill for a simple multi-step workflow; it is designed for complex, scheduled pipelines, not lightweight event-driven orchestration. Option D is wrong because Cloud Dataflow is a stream and batch data processing service (based on Apache Beam) that focuses on transforming data pipelines, not on orchestrating heterogeneous services like Cloud Run and BigQuery queries; it lacks native workflow sequencing and event-driven triggers.
A data engineering team needs to ingest streaming data from an existing Kafka cluster (on-premises) into Google Cloud for real-time analytics. They want to minimize changes to the existing Kafka setup and avoid long-term operational overhead. Which TWO approaches should they consider?
This allows streaming from on-prem Kafka to Pub/Sub without modifying the existing Kafka setup.
Why this answer
Using Kafka Connect with the Pub/Sub connector or setting up a Dataproc cluster running Kafka with mirroring are two ways to bridge on-premises Kafka to GCP with minimal changes.
A company is building a data pipeline that ingests streaming data from Pub/Sub, transforms it with Dataflow, and loads it into BigQuery. They want to handle malformed messages that cannot be parsed. Which TWO actions should they implement for error handling? (Choose 2)
This allows reprocessing without blocking the main pipeline.
Why this answer
A dead letter sink (e.g., writing malformed messages to Cloud Storage or a separate Pub/Sub topic) allows the pipeline to continue processing valid data while preserving the problematic records for offline inspection, retries, or debugging. This pattern is a standard best practice in streaming pipelines to avoid data loss and enable recovery without blocking the main data flow.
Exam trap
Google Cloud often tests the misconception that raising an exception (Option D) is acceptable for error handling in streaming pipelines, but the correct approach is to isolate failures using a dead letter sink (Option C) while logging errors (Option E) to maintain pipeline continuity.
A retail company wants to trigger a Cloud Run service whenever a new CSV file is uploaded to a specific Cloud Storage bucket. Which THREE components are needed to set up this event-driven architecture? (Choose 3)
The source of events; bucket must be configured to send notifications.
Why this answer
Cloud Storage buckets must have notifications enabled to publish events to Pub/Sub when objects are created. Without enabling notifications, the bucket cannot emit events that trigger downstream services. This is typically done by configuring the bucket to send notifications to a Pub/Sub topic for each new object upload.
Exam trap
Google often tests the misconception that you must manually create a Pub/Sub topic as a separate component, when in fact Eventarc manages it automatically, making the Pub/Sub topic an implicit part of the Eventarc trigger rather than a distinct required component.
You need to ingest streaming data from a custom application into BigQuery with exactly-once semantics and low latency. The data volume is up to 10 MB/s. Which TWO services should you combine?
Pub/Sub is the recommended message ingestion service for streaming data.
Why this answer
Pub/Sub provides reliable, low-latency message ingestion, and Dataflow can read from Pub/Sub and write to BigQuery using the Storage Write API, which supports exactly-once semantics. The Storage Write API with committed mode ensures exactly-once delivery.
A data engineer needs to load 10 GB of CSV files from Amazon S3 into BigQuery on a daily basis. The files arrive in a specific S3 bucket at 3 AM UTC each day. Which service should be used to automate this transfer?
BigQuery Data Transfer Service can schedule and automate data loads from Amazon S3 directly into BigQuery.
Why this answer
BigQuery Data Transfer Service supports scheduled transfers from Amazon S3 directly to BigQuery, making it the appropriate choice for this recurring batch load.
You are building a streaming pipeline to ingest real-time clickstream data from a website into BigQuery for immediate analysis. The data must be available in BigQuery within seconds and you need to handle late-arriving data (e.g., browser offline events) that may arrive hours later. Which approach should you use?
This provides low-latency streaming, late data handling via Dataflow's triggers, and efficient writes.
Why this answer
Pub/Sub provides a scalable, durable ingestion layer for real-time clickstream data, and Dataflow can handle late-arriving data via its built-in watermark and trigger mechanisms. The Storage Write API in committed mode ensures exactly-once semantics and low-latency writes to BigQuery, meeting the sub-second availability requirement while preserving data consistency for delayed events.
Exam trap
The trap here is that candidates assume legacy streaming inserts (Option C) are sufficient for real-time needs, but they overlook the 1-hour buffer delay and lack of late-data handling, which are explicitly tested in the PDE exam's focus on streaming pipelines with out-of-order events.
How to eliminate wrong answers
Option B is wrong because Cloud Logging is designed for log ingestion and analysis, not for high-throughput real-time clickstream pipelines; exporting logs via a sink introduces latency (typically minutes) and cannot guarantee sub-second BigQuery availability. Option C is wrong because BigQuery legacy streaming inserts have a 1-hour buffer before data is available for queries, do not support exactly-once semantics, and Cloud Functions lack the stateful processing capabilities (e.g., windowing, triggers) needed to handle late-arriving data correctly. Option D is wrong because Datastream is built for continuous replication from databases like Cloud SQL to BigQuery, not for ingesting raw clickstream events from a website; it requires an intermediary database, which adds unnecessary complexity and latency.
A data engineer needs to ingest daily Salesforce reports into BigQuery without writing custom code. The reports are exported to an Amazon S3 bucket on a schedule. Which service should they use to automate the transfer?
Supports Amazon S3 as a source for scheduled transfers directly into BigQuery.
Why this answer
The BigQuery Data Transfer Service (BQDTS) is the correct choice because it supports scheduled, automatic ingestion of Salesforce reports exported to Amazon S3 into BigQuery without requiring custom code. It can connect to S3 as a source and load the data into BigQuery tables on a schedule, handling schema detection and incremental updates. This meets the requirement of no-code automation from an S3 bucket.
Exam trap
The trap here is that candidates often confuse Cloud Storage Transfer Service (which only moves files between storage buckets) with BigQuery Data Transfer Service (which directly ingests from SaaS applications like Salesforce into BigQuery), leading them to pick option D when the requirement is for a no-code, direct-to-BigQuery solution.
How to eliminate wrong answers
Option A is wrong because Cloud Dataproc is a managed Spark/Hadoop service for running big data processing jobs, not a no-code data ingestion tool; it would require writing custom code to extract from Salesforce and load into BigQuery. Option C is wrong because Cloud Composer is a managed Apache Airflow service for orchestrating workflows; while it could be used to build a custom pipeline, it requires writing DAGs and code, which contradicts the 'without writing custom code' requirement. Option D is wrong because Cloud Storage Transfer Service is designed for moving data between cloud storage systems (e.g., S3 to GCS) and does not directly ingest data into BigQuery or connect to Salesforce APIs.
You need to ingest Google Ads performance data into BigQuery on a daily basis for reporting. Which service should you use?
This service is specifically designed to import data from Google Ads into BigQuery on a scheduled basis.
Why this answer
The BigQuery Data Transfer Service for Google Ads is the correct choice because it provides a fully managed, scheduled connector that automatically ingests Google Ads performance data into BigQuery on a daily basis without requiring any custom code. It handles authentication, schema mapping, and incremental loads, making it the simplest and most reliable solution for this specific use case.
Exam trap
Google often tests the distinction between fully managed services (like BigQuery Data Transfer Service) and generic infrastructure components (like Cloud Scheduler or Pub/Sub) that require custom development, leading candidates to overcomplicate the solution by choosing a more flexible but less appropriate option.
How to eliminate wrong answers
Option B is wrong because Cloud Scheduler is a cron job service that can trigger HTTP requests, but it does not natively integrate with the Google Ads API or handle the complex authentication, pagination, and schema mapping required to load data into BigQuery; you would still need to build and maintain a custom application. Option C is wrong because Pub/Sub is a messaging service for asynchronous event streaming, not a batch ingestion tool; while you could theoretically publish Google Ads data to Pub/Sub, there is no native Google Ads subscriber, and you would need to build a custom subscriber to write to BigQuery, which is far more complex than using the dedicated transfer service. Option D is wrong because Storage Transfer Service is designed for moving data from on-premises or cloud storage (like S3 or HTTP endpoints) into Google Cloud Storage, not for directly ingesting data from Google Ads into BigQuery.
You need to react to changes in a GCS bucket (e.g., new object creation) and trigger a Cloud Run service to process the new file. Which Google Cloud service should you use to route the event?
Eventarc handles events from GCS and other sources, routing them to Cloud Run.
Why this answer
Eventarc is the correct choice because it is purpose-built to route events from Google Cloud sources (like Cloud Storage) to Cloud Run. It directly supports Cloud Storage audit logs and Pub/Sub event triggers, allowing you to react to object creation events without custom middleware. Eventarc handles the event routing, filtering, and delivery to your Cloud Run service automatically.
Exam trap
Google often tests the misconception that Pub/Sub is the direct answer for any event routing, but the trap here is that Eventarc is the managed service that simplifies the integration between GCS and Cloud Run, making it the correct choice over raw Pub/Sub.
How to eliminate wrong answers
Option A is wrong because Pub/Sub directly with a Cloud Run subscription requires you to manually configure a Pub/Sub topic and subscription, and Cloud Run can only pull messages via a push subscription; Eventarc abstracts this complexity and provides native integration with Cloud Storage events. Option B is wrong because Cloud Tasks is a task queue for asynchronous execution of HTTP requests, not designed for event-driven routing from GCS; it would require you to manually publish tasks in response to events, adding unnecessary overhead. Option D is wrong because Cloud Scheduler is a cron job scheduler for periodic tasks, not an event router; it cannot react to real-time object creation events in a GCS bucket.
You are migrating an existing Kafka cluster to Google Cloud using Dataproc. The cluster handles high-throughput streaming data with strict ordering requirements per partition. Which choice of Dataproc configuration is most appropriate?
Non-preemptible workers provide stability for Kafka brokers, and SSDs offer low latency for high-throughput streaming.
Why this answer
Kafka brokers in a Dataproc cluster require persistent, non-preemptible workers to maintain data durability and strict ordering per partition. Preemptible workers can be terminated at any time, causing data loss or rebalancing that violates ordering guarantees. Persistent SSD storage provides the low-latency I/O needed for high-throughput Kafka workloads, while non-preemptible instances ensure broker stability and consistent replication.
Exam trap
Google Cloud often tests the misconception that preemptible VMs or local SSDs are acceptable for stateful, ordered workloads like Kafka, when in fact they violate durability and ordering guarantees due to ephemeral storage and abrupt termination.
How to eliminate wrong answers
Option A is wrong because Dataflow with Kafka IO is a serverless stream processing service, not a Kafka cluster migration target; the question asks about migrating an existing Kafka cluster to Dataproc, not replacing it with a different processing paradigm. Option B is wrong because local SSDs are ephemeral and lose data on instance termination, which is incompatible with Kafka's durability and ordering requirements; autoscaling can cause partition rebalancing that disrupts strict ordering. Option C is wrong because preemptible workers can be terminated at any time, leading to data loss and partition leader re-elections that break ordering guarantees; standard persistent disks have higher latency than SSDs, degrading Kafka's throughput.
A company wants to use dbt to transform data in BigQuery. Their source data is loaded daily into staging tables. They need to run dbt transformations on a schedule and only process tables that have changed. Which dbt feature should they use?
Incremental models only process new/changed records, reducing cost and runtime.
Why this answer
dbt incremental models allow processing only new or changed records based on a configured timestamp or unique key. dbt snapshots capture historical changes. dbt seeds load CSV files. dbt sources are for configuration, not incremental processing.
A company wants to stream real-time clickstream data from a website into BigQuery for near-real-time analytics. They expect peaks of 10,000 events per second. Which combination of services is most suitable for ingestion?
This is the modern recommended architecture: Pub/Sub for ingestion, Dataflow for processing, Storage Write API for high-throughput streaming ingestion into BigQuery.
Why this answer
Pub/Sub is designed for high-throughput event ingestion, Dataflow provides real-time stream processing, and the BigQuery Storage Write API offers exactly-once semantics and high throughput. Therefore, option C is the most suitable. Option A uses Cloud Functions, which are not designed for high-throughput streaming (10,000 events per second would likely exceed typical limits).
Option B sends data directly from the web to Dataflow without a buffer, which could lead to data loss during spikes; Pub/Sub provides a durable buffer. Option D uses legacy streaming inserts, which are deprecated and have lower throughput and no exactly-once guarantees.
A company runs Apache Kafka on Dataproc for real-time event streaming. They want to archive the Kafka topics to Cloud Storage for long-term retention and later analysis in BigQuery. Which approach is the most cost-effective and operationally simple?
Kafka Connect GCS Sink Connector is purpose-built, simple to configure, and runs on the same Dataproc cluster.
Why this answer
Kafka Connect with the GCS Sink Connector is purpose-built for exactly this use case: it directly streams Kafka topics to Cloud Storage in Avro, Parquet, or JSON format without requiring intermediate processing clusters or services. This approach minimizes operational overhead (no Spark or Dataflow jobs to manage) and is cost-effective since it runs as a lightweight connector within the existing Kafka ecosystem, leveraging Dataproc's managed Kafka cluster.
Exam trap
A common mistake in Google exams is to think that streaming data to Cloud Storage requires a full streaming pipeline (Spark, Dataflow) or an intermediary service like Pub/Sub, when in fact Kafka Connect provides a native, lightweight, and cost-effective sink directly to GCS.
How to eliminate wrong answers
Option A is wrong because using Apache Spark streaming on Dataproc to read from Kafka and write to GCS introduces unnecessary compute overhead, latency, and operational complexity (managing Spark jobs, checkpointing, and resource scaling) compared to a direct connector. Option B is wrong because Kafka MirrorMaker is designed for cross-cluster replication, not for writing to GCS; it would require an additional sink to write to GCS, adding complexity and cost without any benefit. Option C is wrong because routing Kafka messages through Pub/Sub adds latency, extra cost (Pub/Sub egress and Dataflow processing), and operational complexity (managing a Pub/Sub topic, subscription, and Dataflow pipeline) when a direct connector to GCS exists.
Your Dataflow pipeline reads from Pub/Sub, performs transformations, and writes to BigQuery. You notice that the pipeline's autoscaling is not keeping up with sudden spikes in traffic, causing increased lag. The pipeline uses Classic Templates. Which change would most effectively improve autoscaling responsiveness?
Streaming Engine improves autoscaling by decoupling compute from state, allowing workers to scale more quickly.
Why this answer
Enabling Dataflow Streaming Engine reduces the overhead of checkpointing and state management by offloading them to the service side, which allows the pipeline to scale more quickly in response to sudden traffic spikes. This directly addresses the autoscaling lag because Streaming Engine decouples compute from state, enabling faster worker adjustments without the bottleneck of persistent disk-based shuffle.
Exam trap
A common misconception in Google Professional Data Engineer exams is that Flex Templates improve runtime performance or autoscaling, when in fact they only affect deployment flexibility, not the underlying execution engine's scaling behavior.
How to eliminate wrong answers
Option B is wrong because Dataflow Prime with Vertical Autoscaling adjusts the CPU/memory of existing workers, not the number of workers, so it does not improve horizontal autoscaling responsiveness to sudden traffic spikes. Option C is wrong because increasing the initial number of workers only sets a starting point; it does not improve the pipeline's ability to scale up dynamically during a spike, and it may waste resources during low traffic. Option D is wrong because Flex Templates only affect how the pipeline is deployed and parameterized, not the runtime autoscaling behavior; Classic Templates and Flex Templates share the same autoscaling mechanisms.
A company needs to continuously synchronize customer data changes from an on-premises Oracle database to BigQuery for near-real-time analytics. The Oracle database has Change Data Capture (CDC) enabled. Which Google Cloud service should be used to stream these changes with minimal latency and schema evolution support?
Datastream directly supports Oracle CDC and streams to BigQuery with schema evolution.
Why this answer
Datastream is designed to stream CDC from Oracle (and MySQL/PostgreSQL) directly to BigQuery or GCS, supporting schema evolution and low-latency replication.
A data engineer needs to ingest on-premises Oracle CDC data into BigQuery in near real-time with minimal operational overhead. Which service should they use?
Datastream is purpose-built for serverless CDC from databases to Google Cloud destinations like BigQuery and GCS.
Why this answer
Datastream is purpose-built for streaming change data capture (CDC) from Oracle and other sources into BigQuery with near-real-time latency and minimal operational overhead. It handles schema propagation, checkpointing, and automatic retries, eliminating the need to manage custom ingestion pipelines.
Exam trap
Google often tests the distinction between batch migration tools (Storage Transfer Service, Transfer Appliance) and streaming CDC services (Datastream), leading candidates to choose a batch option when the question explicitly requires near-real-time ingestion.
How to eliminate wrong answers
Option A is wrong because Pub/Sub + Dataflow requires building and maintaining a custom pipeline to handle Oracle CDC, including log mining and transformation logic, which increases operational overhead compared to a managed service. Option B is wrong because Storage Transfer Service is designed for bulk batch transfers of files from cloud or on-premises storage to Google Cloud, not for streaming CDC from a live database. Option C is wrong because Transfer Appliance is a physical device for offline, high-volume data migration, which cannot provide near-real-time streaming and introduces significant latency.
An organization needs to trigger a Cloud Run service whenever a new file is uploaded to a specific Cloud Storage bucket. Which service should they use to set up this event-driven architecture?
Why this answer
Eventarc can capture Cloud Storage events (e.g., OBJECT_FINALIZE) and route them to Cloud Run, Cloud Functions, or Workflows. It supports CloudEvents standard.
A company is building a real-time anomaly detection pipeline using Dataflow. Events are ingested from Pub/Sub, and the pipeline must compute a sliding window average every minute over a 1-hour window. Which TWO configurations are required for this pipeline? (Choose 2)
Event time ensures windows based on actual event occurrence time, necessary for correct sliding window semantics.
Why this answer
A sliding window of 1-hour length with a 1-minute slide period fits the requirement (every minute, compute over last hour). Fixed window of 1 minute would compute only per-minute, not sliding. Using stateful processing with timers is an alternative but not standard for sliding windows.
Dataflow's default watermark is based on event time; processing time would cause incorrect results. The window type and period are the key.
A data engineer needs to build a Dataflow pipeline that reads JSON messages from Pub/Sub, transforms them (including filtering, grouping, and enrichment), and writes the results to BigQuery. The pipeline must handle schema evolution in the input messages and minimize data loss. Which THREE settings or features should the engineer use? (Choose THREE.)
Correct. Side inputs enable enrichment of streaming data with reference data from BigQuery, supporting schema evolution by allowing dynamic lookup.
Why this answer
Side inputs allow the pipeline to enrich streaming data with reference data from BigQuery, which is a common requirement for handling schema evolution and enrichment. Option C is correct: a dead letter queue captures messages that fail to parse or validate, preventing data loss and enabling reprocessing of failed messages. Option D is correct: autoscaling adjusts the number of workers dynamically to handle spikes in message volume, ensuring no data is lost due to backpressure.
Option B is incorrect: setting `withAllowedLateness` to 0 does not help with schema evolution or minimize data loss; it simply drops late data, which could cause data loss. Option E is incorrect: Streaming Engine improves checkpoint performance but is not directly related to schema evolution or data loss minimization.
A company uses Google Ads and wants to automatically load their advertising data into BigQuery daily. They also need to transform the data with SQL and schedule a recurring query. Which combination of services meets these requirements with minimal operational overhead?
Direct integration with scheduled queries for transformation.
Why this answer
BigQuery Data Transfer Service can automatically load Google Ads data; scheduled queries handle transformation.
A financial services company receives real-time stock trade data via Pub/Sub. They need to enrich each trade with reference data from a Cloud SQL table and write the results to BigQuery for real-time analytics. The enrichment must handle late-arriving data and ensure exactly-once processing. Which Dataflow streaming pipeline configuration should be used?
Storage Write API with exactly-once ensures no duplicates, and side input allows enrichment from Cloud SQL.
Why this answer
Using the Storage Write API with exactly-once semantics and side inputs to join with reference data provides the required enrichment and exactly-once guarantees.
A data engineer needs to schedule a recurring batch load of CSV files from an on-premises SFTP server into BigQuery. The files are generated daily and need to be loaded into a partitioned table by date. Which THREE steps should the engineer take? (Choose THREE.)
Storage Transfer Service supports scheduled transfers from SFTP to Cloud Storage.
Why this answer
The Storage Transfer Service is designed to move data from on-premises sources (including SFTP servers) into Cloud Storage on a scheduled basis. This is the recommended first step for ingesting files from an external SFTP server into Google Cloud, as it handles the network transfer, retries, and scheduling natively without requiring custom code.
Exam trap
A common mistake is assuming that BigQuery Data Transfer Service can directly ingest from SFTP, but it only supports a limited set of SaaS and cloud sources, not on-premises SFTP servers.
You are designing a Dataflow pipeline to process streaming data. The pipeline may encounter malformed records. You need to handle these errors without failing the entire pipeline and store the bad records for later analysis. What is the best practice?
This is the recommended pattern: isolate bad records for later reprocessing while allowing the pipeline to continue.
Why this answer
Dead letter sinks are a common pattern: route erroneous records to a separate output (e.g., Pub/Sub topic or GCS) for later investigation. Writing to BigQuery using Storage Write API with error handling is good, but for malformed records you want to isolate them. Raising exceptions would fail the pipeline.
Logging only loses the data.
A large enterprise is migrating its data warehouse from Teradata to BigQuery. They need to transfer historical data (100 TB) and set up ongoing daily incremental loads. They also need to transform the data using dbt. Which THREE Google Cloud services should they use?
Supports both backfill and incremental transfers from Teradata to BigQuery.
Why this answer
BigQuery Data Transfer Service for Teradata handles both historical and incremental loads, dbt runs on BigQuery for transformations, and Cloud Composer can orchestrate the dbt runs on a schedule.
A company wants to migrate 500 TB of on-premises archival data to Cloud Storage. The data is stored on a SAN and the network link is limited to 1 Gbps. The migration must complete within 10 days. What is the MOST cost-effective approach?
Transfer Appliance is designed for large offline transfers when network speed is a constraint.
Why this answer
The Transfer Appliance is designed for large-scale data migrations where network bandwidth is insufficient. With 500 TB at 1 Gbps, the theoretical transfer time is over 46 days, far exceeding the 10-day window. The appliance allows you to physically ship the data, bypassing network constraints entirely, making it the most cost-effective and timely solution.
Exam trap
The trap here is that candidates underestimate the time required for network transfer at 1 Gbps and overestimate the practicality of compression or incremental sync, failing to recognize that physical shipping is the only viable option for multi-petabyte data within a tight deadline.
How to eliminate wrong answers
Option A is wrong because rsync over a 1 Gbps Cloud VPN would take approximately 46 days for 500 TB (assuming full utilization, which is unrealistic due to overhead and encryption), far exceeding the 10-day deadline. Option B is wrong because BigQuery Data Transfer Service is for loading data from SaaS applications (e.g., Google Ads, Amazon S3) or other cloud sources into BigQuery, not for ingesting on-premises archival data into Cloud Storage. Option D is wrong because Storage Transfer Service relies on the existing 1 Gbps network link, which would require over 46 days for 500 TB, violating the 10-day requirement.
A data engineer is using Spark on Dataproc to process a large dataset. They notice the job is slow due to excessive shuffling. They want to optimize the job by using a more efficient data structure that reduces serialization overhead and provides better memory management. Which Spark API should they use?
DataFrames and Datasets use the Catalyst optimizer and Tungsten execution engine, improving performance and memory efficiency.
Why this answer
Spark DataFrames/Datasets use Tungsten execution engine, which provides optimized serialization and memory management. RDDs lack these optimizations. Spark SQL is a module, not an API.
Spark Streaming is for streaming.
Ready to test yourself?
Try a timed practice session using only Pde Ingestion Processing questions.