Courseiva

CCNA Ingesting and Processing the Data Questions

75 of 107 questions · Page 1/2 · Ingesting and Processing the Data · Answers revealed

Practice these questionsDomain overviewAll questions
1
MCQmedium

A data engineer needs to transfer 5 PB of historical data from an on-premises Hadoop cluster to Cloud Storage. The network bandwidth is limited to 1 Gbps, and the transfer must complete within 30 days. Which transfer method should they use?

A.gsutil rsync over the internet
B.BigQuery Data Transfer Service
C.Storage Transfer Service for on-premises
D.Transfer Appliance
AnswerD

Transfer Appliance is a physical, shippable device for offline bulk migration. At 5 PB over a 1 Gbps link, online transfer would take far longer than 30 days, so shipping appliances satisfies the bandwidth and deadline constraints.

Why this answer

The Transfer Appliance is a physical device designed for large-scale data transfers when network bandwidth is insufficient. With 5 PB of data and a 1 Gbps link, the theoretical maximum transfer time is over 500 days (5 PB × 8 bits/byte / 1 Gbps / 86400 seconds/day), far exceeding the 30-day window. The Transfer Appliance bypasses network constraints by shipping data physically to Google Cloud.

Option A (gsutil rsync over the internet) is incorrect because it relies on the same 1 Gbps bandwidth, making it impossible to transfer 5 PB within 30 days. Option B (BigQuery Data Transfer Service) is designed for transferring data into BigQuery from cloud sources, not for large on-premises data transfers. Option C (Storage Transfer Service for on-premises) is intended for smaller or incremental transfers over a network, not for a 5 PB initial load within 30 days.

Exam trap

The trap here is that candidates may overestimate network transfer speeds or assume that cloud-native services like Storage Transfer Service can handle any volume, ignoring the fundamental bandwidth math that makes physical shipping the only viable option for 5 PB within 30 days.

How to eliminate wrong answers

Option A is wrong because gsutil rsync over the internet at 1 Gbps would take approximately 500 days to transfer 5 PB, which exceeds the 30-day deadline; it also lacks reliability for such massive transfers over a public network. Option B is wrong because BigQuery Data Transfer Service is designed for scheduled imports from SaaS applications (e.g., Google Ads, Amazon S3) and does not support direct on-premises Hadoop transfers. Option C is wrong because Storage Transfer Service for on-premises requires a network connection (typically via a staging bucket or partner interconnect) and still relies on the same 1 Gbps bandwidth, making it impossible to meet the 30-day requirement.

Practice this question →
2
MCQmedium

You are building a Dataflow pipeline in Python that reads messages from Pub/Sub, enriches them with data from a BigQuery table, and writes the results to BigQuery. The enrichment lookup table is large and changes infrequently. Which approach minimizes cost and latency?

A.Use a CoGroupByKey transform to join the incoming stream with a stream from BigQuery.
B.Use BigQuery IO to query the table for every incoming message.
C.Use a side input that reads the BigQuery table periodically and caches it.
D.Use a stateful DoFn and store the lookup in state per key.
AnswerC

A side input reads the slowly changing BigQuery enrichment table periodically and caches it in memory across workers, avoiding a per-element BigQuery lookup. This cuts both query cost and per-message latency, satisfying the large, infrequently changing lookup constraint.

Why this answer

Using a side input that periodically reads the BigQuery table and caches it avoids querying BigQuery for every incoming message, which would be prohibitively expensive and high-latency. The side input is refreshed at a configurable interval (e.g., every 10 minutes) via a pipeline option, and the cached data is broadcast to all workers, enabling fast, in-memory lookups without per-element I/O. This approach minimizes cost by reducing BigQuery API calls and minimizes latency by avoiding synchronous queries for each message.

Exam trap

Google often tests the misconception that querying BigQuery per message is acceptable in streaming pipelines, but the trap here is that candidates overlook the cost and latency implications of per-element I/O, especially with BigQuery's pricing model and query latency.

How to eliminate wrong answers

Option A is wrong because CoGroupByKey requires both inputs to be bounded or both unbounded streams; here, the BigQuery table is a bounded dataset, and Pub/Sub is unbounded, so CoGroupByKey would not work without windowing and would introduce unnecessary complexity and latency. Option B is wrong because querying BigQuery for every incoming message would cause extremely high API costs (BigQuery charges per byte processed) and high latency (each query takes hundreds of milliseconds to seconds), making it impractical for a streaming pipeline. Option D is wrong because storing the lookup in state per key would require partitioning the lookup table across keys, which is inefficient for a large, infrequently changing table; state is per-key and not shared across keys, so each worker would need to load and maintain its own copy, leading to memory waste and complex state management.

Practice this question →
3
MCQmedium

A company uses Workflows to orchestrate a series of Google Cloud services for data processing. They need to call an external HTTP API as part of the workflow and handle potential failures with retries. Which Workflows feature should they use?

A.Retry policy on the step
B.Subworkflows
C.Parallel steps
D.Conditional steps
AnswerA

Attaching a retry policy to the HTTP call step lets Workflows automatically re-invoke the external API on transient failures, using configurable backoff and attempt limits. This directly satisfies the stem's requirement to handle potential failures with retries, without custom error-handling logic or additional orchestration components.

Why this answer

Workflows provides a built-in retry policy that can be configured on individual steps to automatically retry an HTTP call upon transient failures (e.g., 5xx server errors or network timeouts). This allows the workflow to handle external API failures without custom code, using exponential backoff and a maximum retry count.

Exam trap

Google Cloud Workflows often tests the distinction between workflow orchestration features (retry, subworkflows, parallel, conditional) and candidates mistakenly choose parallel steps or subworkflows thinking they inherently provide fault tolerance, but only a retry policy directly addresses automatic retries on failure.

How to eliminate wrong answers

Option B is wrong because subworkflows are used to encapsulate reusable sequences of steps, not to handle retries on a single HTTP call. Option C is wrong because parallel steps execute multiple branches concurrently, which does not provide retry logic for a single failing step. Option D is wrong because conditional steps (e.g., switch/if-else) control the flow based on conditions but do not automatically retry a failed HTTP request.

Practice this question →
4
MCQeasy

A media company wants to analyze clickstream data stored in Cloud Storage as JSON files. They need to run SQL queries directly on these files without loading them into BigQuery. Which BigQuery feature should they use?

A.BigQuery Data Transfer Service
B.External tables
C.BigQuery Omni
D.BigQuery BI Engine
AnswerB

External tables in BigQuery allow querying data directly from Cloud Storage in formats like JSON, CSV, Avro, Parquet, and ORC. They provide a schema and can be queried with SQL without loading data into BigQuery storage. This meets the requirement of querying JSON files in GCS.

Why this answer

External tables enable BigQuery to query data stored in Cloud Storage directly, supporting JSON and other formats. This avoids data duplication and loading costs, making it ideal for ad-hoc analysis of raw files. Other features either load data into BigQuery or serve different purposes.

Exam trap

The trap here is confusing external tables with data loading services like BigQuery Data Transfer Service, which move data rather than query in place.

Practice this question →
5
MCQhard

A team uses dbt on BigQuery to transform data in their data warehouse. They have a large table with nested and repeated fields (arrays and structs). The transformation needs to normalize this data into a star schema. Which dbt feature and BigQuery SQL feature should they use together?

A.dbt hooks with BigQuery STRUCT access
B.dbt models with BigQuery UNNEST and CROSS JOIN
C.dbt snapshots with BigQuery JSON functions
D.dbt seeds with BigQuery ARRAY_AGG
AnswerB

UNNEST flattens BigQuery arrays and structs into individual rows, while CROSS JOIN multiplies each parent row across its unnested elements. Together inside dbt models they convert nested, repeated source data into the normalised fact and dimension tables of a star schema.

Why this answer

To normalize nested and repeated fields (arrays and structs) into a star schema, you need to flatten the arrays into separate rows. BigQuery's UNNEST operator, when used with CROSS JOIN, expands each array element into its own row, effectively denormalizing the nested structure. dbt models (SQL SELECT statements) are the correct dbt feature to define these transformations as version-controlled, reusable SQL files. Together, they allow you to write a dbt model that uses CROSS JOIN UNNEST to produce dimension and fact tables from a single nested table.

Exam trap

A common pitfall in this question is confusing the purpose of dbt features: hooks (automation), snapshots (SCD type 2), and seeds (CSV loading) are not designed for flattening nested structures. The correct approach is to use dbt models with BigQuery's UNNEST and CROSS JOIN to expand arrays into rows.

How to eliminate wrong answers

Option A is wrong because dbt hooks are SQL or shell commands executed at specific points in the dbt run (e.g., before/after model builds) and are not designed for transforming nested data into a star schema; BigQuery STRUCT access alone cannot flatten arrays. Option C is wrong because dbt snapshots are used for slowly changing dimension (SCD) tracking over time, not for normalizing nested data; BigQuery JSON functions are for parsing JSON strings, not for unnesting native arrays and structs. Option D is wrong because dbt seeds are CSV files loaded into the warehouse as static lookup tables, not for transforming existing data; ARRAY_AGG is an aggregation function that creates arrays, the opposite of the flattening needed here.

Practice this question →
6
MCQhard

You are building a Dataflow pipeline that reads from Cloud Storage and writes to BigQuery. The pipeline must handle files that are compressed with gzip and contain JSON data. You need to ensure that the pipeline can process these files efficiently and write to BigQuery with minimal errors. Which approach should you take?

A.Use TextIO to read the gzip files, parse the JSON using a DoFn, and write to BigQuery using BigQueryIO with the STORAGE_WRITE_API method.
B.Use TextIO to read the gzip files, but disable compression detection and manually decompress each file in a DoFn before parsing JSON.
C.Use AvroIO to read the gzip files, convert the Avro records to JSON, and write to BigQuery using BigQueryIO with the FILE_LOADS method.
D.Use BigQueryIO to read the gzip files directly, parse the JSON, and write to BigQuery using the default write method.
AnswerA

TextIO can read gzip-compressed files transparently, and a DoFn can parse the JSON into a TableRow. Writing with BigQueryIO using the STORAGE_WRITE_API method provides high-performance, exactly-once writes. This combination efficiently processes compressed JSON files and handles errors through dead-letter patterns if configured, meeting the requirements.

Why this answer

TextIO natively supports reading gzip-compressed files, simplifying ingestion. Parsing JSON in a DoFn allows transformation into BigQuery-compatible rows. Using BigQueryIO with the STORAGE_WRITE_API method ensures high-throughput, exactly-once writes, which is critical for efficient and reliable loading.

This combination addresses the compressed format and minimizes errors through robust write semantics.

Exam trap

The trap here is assuming that BigQueryIO can read files or that manual decompression is necessary, when TextIO already handles gzip seamlessly.

Practice this question →
7
MCQeasy

Which Dataflow feature automatically scales the number of workers based on the pipeline's current workload, and also selects the optimal machine type for each worker based on the pipeline's resource requirements?

A.Dataflow Shuffle
B.Dataflow Prime
C.Dataflow Streaming Engine
D.Dataflow Flex Templates
AnswerB

Dataflow Prime provides vertical autoscaling, dynamically resizing worker machine types to match each stage's resource demands, alongside horizontal worker-count scaling. This satisfies the stem's dual requirement: automatic worker scaling plus optimal machine-type selection based on the pipeline's resource requirements.

Why this answer

Dataflow Prime is the correct answer because it is the only Dataflow feature that provides both automatic worker scaling (horizontal autoscaling) and intelligent machine type selection (vertical autoscaling). It dynamically adjusts the number of workers based on the pipeline's current workload and selects the optimal machine type (e.g., CPU, memory, or accelerator-optimized) for each worker based on the pipeline's resource requirements, such as CPU utilization, memory pressure, or shuffle throughput.

Exam trap

Google often tests the distinction between horizontal autoscaling (adding/removing workers) and vertical autoscaling (changing machine type), and the trap here is that candidates assume Dataflow Shuffle or Streaming Engine handle scaling, when in fact they only optimize specific pipeline phases (shuffle or state management) without affecting worker count or machine type.

How to eliminate wrong answers

Option A is wrong because Dataflow Shuffle is a service that separates the shuffle operation from worker VMs, improving scalability and reliability, but it does not handle worker scaling or machine type selection. Option C is wrong because Dataflow Streaming Engine moves state storage and computation away from worker VMs for streaming pipelines, reducing resource overhead, but it does not automatically scale workers or select machine types. Option D is wrong because Dataflow Flex Templates allow you to package and reuse pipeline code with custom container images, but they do not provide any autoscaling or machine type optimization; scaling is handled separately by the Dataflow service.

Practice this question →
8
MCQhard

Your team is processing a large dataset with Apache Beam on Dataflow. The pipeline sometimes fails due to transient errors when writing to a BigQuery sink. You need to ensure that failed records are not lost and can be reprocessed later without blocking the pipeline. What is the best approach?

A.Configure the pipeline to use at-least-once semantics and rely on Dataflow to retry the entire bundle.
B.Increase the number of workers to reduce the chance of transient errors.
C.Use a try-catch block in the DoFn and log the error; continue processing other elements.
D.Use a side output (e.g., via TupleTag) to write failed records to a dead letter sink (e.g., GCS or Pub/Sub) and continue processing the main output.
AnswerD

A TupleTag side output diverts records that fail the BigQuery write into a dead letter sink while the main pipeline continues. This satisfies both constraints: failed records are retained for later reprocessing, and the pipeline is not blocked by transient sink errors.

Why this answer

Using a dead letter pattern with a side output to write failed records to a GCS bucket (or Pub/Sub) allows the pipeline to continue processing healthy records while failed records are stored for later analysis and reprocessing.

Practice this question →
9
Multi-Selecteasy

A data engineer needs to load data from CSV files in Cloud Storage into BigQuery. The CSV files have a header row and some columns contain nested JSON strings. Which TWO methods can they use to load this data into BigQuery?

Select 2 answers
A.Use Datastream to load CSV files
B.Use the Storage Write API to write rows from a custom application
C.Create a federated query using an external table
D.Create a BigQuery load job with the CSV format
E.Use gsutil to copy files into BigQuery
AnswersB, D

The Storage Write API can be used to stream data from CSV after parsing.

Why this answer

Option D is correct because a BigQuery load job natively supports the CSV format, including a header row via the --skip_leading_rows parameter, and can ingest Cloud Storage files directly into a native BigQuery table. Option B is correct because the Storage Write API lets a custom application parse the CSV and nested JSON strings itself and then stream the resulting rows into BigQuery, giving full control over transformation during load. Option A is wrong because Datastream is a change data capture (CDC) and replication service for databases such as MySQL, PostgreSQL, Oracle, and SQL Server, not a CSV file loader.

Option C is wrong because an external table (federated query) only queries data in place in Cloud Storage; it does not load the data into BigQuery. Option E is wrong because gsutil only copies objects between Cloud Storage locations and cannot write data into BigQuery tables.

Exam trap

Google often tests the distinction between loading data into BigQuery (permanent storage) versus querying external data sources (federated queries), causing candidates to mistakenly choose Option C as a valid loading method.

Practice this question →
10
MCQmedium

Your organization runs a batch Dataflow pipeline that reads from BigQuery, transforms records, and writes Parquet files to Cloud Storage partitioned by event date. The pipeline currently writes all files into a single directory and downstream Hive-style queries scan the entire dataset. You need to restructure the output so that queries scan only the relevant date partitions, while keeping the pipeline idempotent on re-runs. What should you do?

A.Use AvroIO with a custom filename policy that includes the event date, and enable the pipeline's --diskSizeGb option to improve write throughput.
B.Use FileIO.writeDynamic with a destination function that maps each record to a path like gs://bucket/events/event_date=YYYY-MM-DD/, and write to a temporary location before atomically moving files into place.
C.Write all files to a single directory and create a BigQuery external table over the bucket, then rely on BigQuery's automatic partition pruning from the file names.
D.Add a GroupByKey transform keyed by event date before writing, then use TextIO to write one file per date into a flat directory with the date embedded in the file name.
AnswerB

writeDynamic lets each record choose its output directory based on the event date, producing Hive-style partition paths that downstream engines prune. Writing to a temporary location and then moving files into the final partition directory ensures that a failed re-run does not leave partial data visible to queries, which preserves idempotency. This combination directly satisfies both the partition-pruning and idempotency requirements.

Why this answer

Hive-style partition pruning depends on directory structure, not file naming. FileIO.writeDynamic with a destination function that emits paths containing event_date=YYYY-MM-DD creates the directories that query engines use to skip irrelevant data. Writing first to a temporary location and then moving completed files into the final partition directory prevents partially written partitions from being visible if the pipeline fails and is re-run, preserving idempotency.

Exam trap

The trap here is assuming that embedding the date in a file name or file extension is enough for partition pruning, when engines actually prune on directory paths such as event_date=YYYY-MM-DD.

Practice this question →
11
MCQeasy

A data engineer needs to ingest data from a Cloud Storage bucket into BigQuery. The data is in CSV format and is updated daily with new files. The engineer wants to minimize manual intervention and ensure that new files are automatically loaded into BigQuery. Which Google Cloud service should be used to orchestrate this?

A.Cloud Functions
B.Cloud Scheduler
C.Cloud Data Fusion
D.Cloud Composer
AnswerA

Cloud Functions can be triggered by object finalize events in Cloud Storage, allowing automatic execution of code when a new file is uploaded. This enables a serverless, event-driven approach to load new CSV files into BigQuery with minimal intervention. It is simple, cost-effective, and directly addresses the need for automatic ingestion upon file arrival.

Why this answer

Cloud Functions provides an event-driven, serverless way to respond to new files in Cloud Storage. By configuring a Cloud Function to trigger on object finalize events, the function can execute a BigQuery load job for each new CSV file. This approach requires no polling, minimal code, and automatically scales.

Other services like Cloud Scheduler or Cloud Composer could work but involve more overhead and are less direct for this specific requirement.

Exam trap

The trap here is overlooking event-driven options and defaulting to scheduled or orchestrated solutions, which add unnecessary complexity for simple file arrival triggers.

Practice this question →
12
MCQmedium

An organization needs to continuously replicate change data from a MySQL database to BigQuery with sub-minute latency. The database is running on-premises. Which Google Cloud service should they use?

A.Cloud Pub/Sub with a custom connector to MySQL
B.BigQuery Data Transfer Service for MySQL
C.Cloud Dataflow with a JDBC source
D.Cloud Datastream
AnswerD

Datastream provides serverless change data capture, reading MySQL binlogs and streaming inserts, updates and deletes into BigQuery with sub-minute latency. It supports on-premises sources through private connectivity, satisfying the continuous replication and low-latency constraints without managing replication infrastructure.

Why this answer

Cloud Datastream is a serverless change data capture (CDC) and replication service that supports continuous replication from MySQL (including on-premises) to BigQuery with sub-second latency. It handles schema conversion, and integrates with BigQuery for real-time analytics. This is the purpose-built service for the requirement.

Exam trap

PDE often tests the misconception that Dataflow or Pub/Sub are the default for CDC, when Datastream is the managed service specifically designed for low-latency database replication to BigQuery.

How to eliminate wrong answers

Option A is wrong because Cloud Pub/Sub with a custom connector requires building and managing a custom CDC solution, which is complex and may not achieve sub-minute latency reliably. Option B is wrong because BigQuery Data Transfer Service for MySQL is for batch transfers, not continuous CDC with sub-minute latency. Option C is wrong because Cloud Dataflow with a JDBC source can be used for CDC but requires custom development and is not as streamlined as Datastream; it may also not guarantee sub-minute latency without significant tuning.

Practice this question →
13
MCQmedium

A data engineer is building a Dataflow pipeline in Python that reads from BigQuery, transforms data, and writes to Cloud Storage. The pipeline will be deployed in production. Which approach should they use to ensure the pipeline is reusable across environments with different configuration parameters?

A.Create a separate pipeline for each environment with hardcoded values
B.Use a Dataflow Classic Template
C.Use a Dataflow Flex Template
D.Run the pipeline using the DirectRunner for each environment
AnswerC

Flex Templates package the pipeline as a Docker image with a metadata file, so runtime parameters such as input, output and project values are supplied at job launch. This satisfies the reusability requirement across environments without editing code, unlike classic templates with fixed dependencies.

Why this answer

Dataflow Flex Templates allow you to package a Docker container with your pipeline code and dependencies, enabling parameterization at runtime via the Dataflow UI, CLI, or API. This makes the pipeline reusable across environments (e.g., dev, staging, prod) by passing different configuration parameters (like project IDs, table names, or output paths) without modifying the code. Flex Templates support custom container images and are the recommended approach for production pipelines that need environment-agnostic deployment.

Exam trap

Google often tests the distinction between Classic Templates and Flex Templates, where candidates mistakenly choose Classic Templates because they are simpler, but Flex Templates are required for custom environments and parameterized production reuse.

How to eliminate wrong answers

Option A is wrong because creating separate pipelines with hardcoded values violates the principle of reusability and introduces maintenance overhead; any change requires updating multiple pipeline copies, increasing the risk of configuration drift. Option B is wrong because Classic Templates are limited to the Apache Beam SDK's built-in I/O transforms and do not support custom container images or complex dependencies, making them less flexible for production pipelines that may require custom code or third-party libraries. Option D is wrong because the DirectRunner is intended for local testing and development only; it runs the pipeline in a single JVM process and cannot handle the scalability, distributed execution, or environment-specific configuration needed for production deployment.

Practice this question →
14
Multi-Selecteasy

A data engineer needs to schedule a recurring transfer of data from a partner's Amazon S3 bucket to a Cloud Storage bucket for further processing. Which THREE components or configurations are necessary? (Choose 3)

Select 3 answers
A.A VPC network configuration
B.Specification of the source S3 bucket and destination GCS bucket
C.A scheduled transfer job in Storage Transfer Service
D.A Pub/Sub topic to notify completion
E.Authentication credentials for AWS (e.g., access key and secret)
AnswersB, C, E

Storage Transfer Service requires the source and destination locations to be defined so it knows which S3 bucket to read from and which Cloud Storage bucket to write into. Specifying both endpoints is the fundamental configuration the transfer job acts upon.

Why this answer

Storage Transfer Service is the Google Cloud service designed to move data from external sources like Amazon S3 into Cloud Storage, so the transfer job must be defined as a scheduled transfer job (option C), which controls the recurring execution. Every transfer job requires the source and destination locations to be identified, so specifying the source S3 bucket and destination GCS bucket (option B) is mandatory for the job to know what to copy and where to place it. Because the source is an external AWS S3 bucket, Storage Transfer Service must authenticate to AWS, which requires AWS credentials such as an access key ID and secret access key (option E).

A VPC network configuration (option A) is not required because Storage Transfer Service is a fully managed service that does not need customer VPC networking for S3-to-GCS transfers. A Pub/Sub topic (option D) is optional, used only if you want notifications about transfer completion, not a necessary component for the transfer itself.

Exam trap

PDE often tests the misconception that additional components like VPC or Pub/Sub are required for Storage Transfer Service, when only the source/destination, job, and credentials are necessary.

Practice this question →
15
MCQeasy

You are loading 10 GB of daily CSV files from a GCS bucket into a BigQuery table. The files contain some malformed rows that you want to skip. Which BigQuery load configuration should you use?

A.Use the 'skip_leading_rows' option.
B.Use the 'ignore_unknown_values' option.
C.Use the 'max_bad_records' option set to a value like 10.
D.Use the 'allow_jagged_rows' option.
AnswerC

Setting max_bad_records to 10 lets the load job tolerate up to ten malformed rows per file, skipping them while importing the valid data. Without it, any single bad row aborts the entire 10 GB load.

Why this answer

The 'max_bad_records' option allows you to specify the maximum number of bad records that BigQuery can ignore during a load job. Setting it to a value like 10 means that up to 10 malformed rows will be skipped; if more than 10 bad records are encountered, the job fails. This is the correct configuration to skip a limited number of malformed rows while ensuring data quality.

Exam trap

PDE often tests the confusion between options that handle different types of CSV parsing issues, such as skipping headers versus skipping malformed rows, and candidates may incorrectly choose 'ignore_unknown_values' or 'allow_jagged_rows' for general malformed data.

How to eliminate wrong answers

Option A is wrong because 'skip_leading_rows' is used to skip header rows in CSV files, not malformed data rows. Option B is wrong because 'ignore_unknown_values' ignores extra columns in the data that do not match the schema, but it does not skip rows with malformed data (e.g., type mismatches). Option D is wrong because 'allow_jagged_rows' allows rows with missing trailing columns, but it does not handle other types of malformed data like invalid values.

Practice this question →
16
MCQmedium

A data pipeline processes JSON files from Cloud Storage, transforms them using Apache Beam, and writes the output to BigQuery. Some records are malformed and cause the pipeline to fail. How should the engineer handle these errors to ensure the pipeline continues processing while preserving the malformed records for analysis?

A.Set the pipeline to retry malformed records indefinitely until they succeed.
B.Use a side input to send malformed records to a dead letter queue in Pub/Sub for later reprocessing.
C.Log the malformed records to Stackdriver and skip them in the pipeline.
D.Catch exceptions in a DoFn and write the malformed records to a separate Cloud Storage bucket using a FileIO sink.
AnswerD

Wrapping parsing in a DoFn try/except lets the pipeline emit valid records downstream while routing malformed ones to a separate Cloud Storage bucket via FileIO. This satisfies both constraints: the pipeline continues processing, and the bad records are preserved for later analysis.

Why this answer

It allows the pipeline to continue processing by catching exceptions within a DoFn and writing malformed records to a separate Cloud Storage bucket using a FileIO sink. This preserves the malformed records for later analysis without blocking the main data flow, which is a standard pattern in Apache Beam for handling dead-letter records. The approach ensures fault tolerance while maintaining data integrity for debugging.

Exam trap

A common misconception in Google Cloud data pipelines is that error handling should involve retrying indefinitely or simply logging errors to Cloud Logging, but the correct approach is to isolate and persist malformed records to a durable sink like Cloud Storage for later analysis.

How to eliminate wrong answers

Option A is wrong because retrying malformed records indefinitely would cause the pipeline to hang or exhaust resources, as malformed records will never succeed due to inherent data issues. Option B is wrong because using a side input to send malformed records to a Pub/Sub dead letter queue is not a direct pattern in Apache Beam; side inputs are for broadcasting data to all elements, not for error handling, and Pub/Sub would require additional setup and does not inherently preserve the records for analysis without a separate sink. Option C is wrong because logging malformed records to Stackdriver and skipping them loses the data permanently, as logs are not designed for structured storage or reprocessing of the original records.

Practice this question →
17
Multi-Selectmedium

You are building a BigQuery table that contains nested and repeated fields (e.g., order with line items). You need to write a query that counts the number of line items per order. Which TWO SQL functions/techniques can you use?

Select 2 answers
A.Window function ROW_NUMBER
B.UNNEST with COUNT
C.STRUCT with aggregation
D.ARRAY_LENGTH
E.SELECT * EXCEPT
AnswersB, D

UNNEST flattens the repeated line_items array into individual rows, letting COUNT aggregate them per order. This directly satisfies the stem's requirement to count line items within nested, repeated fields, since COUNT alone cannot traverse array elements without first unnesting them into a queryable row set.

Why this answer

UNNEST flattens the repeated line items array into individual rows, allowing COUNT to aggregate the number of line items per order. Option D is correct because ARRAY_LENGTH directly returns the number of elements in the repeated field array, which corresponds to the line item count.

Exam trap

Google often tests the distinction between functions that operate on arrays directly (like ARRAY_LENGTH) versus those that require row-level expansion (like UNNEST), and candidates may mistakenly choose window functions or STRUCT-based aggregation that do not directly count array elements.

Practice this question →
18
MCQmedium

A data engineer needs to migrate 200 TB of on-premises Oracle data to BigQuery. The network bandwidth is limited to 100 Mbps, and the data must be loaded within 2 weeks. Which Google Cloud service is most appropriate for the initial data transfer?

A.Transfer Appliance
B.BigQuery Data Transfer Service for Oracle
C.Datastream
D.Storage Transfer Service
AnswerA

At 100 Mbps, 200 TB needs roughly 185 days of continuous transfer, far beyond the two-week deadline. Transfer Appliance ships data physically, bypassing the bandwidth constraint entirely, then uploads to Cloud Storage for loading into BigQuery.

Why this answer

Transfer Appliance is correct because it is a physical device designed for large-scale offline data transfers when network bandwidth is insufficient. With 200 TB at 100 Mbps, the theoretical transfer time exceeds 190 days, far beyond the 2-week window. Transfer Appliance allows shipping the data directly to Google, bypassing network constraints entirely.

Exam trap

The trap here is that candidates may assume online services like Storage Transfer Service or BigQuery Data Transfer Service can handle large volumes if given enough time, ignoring the hard bandwidth calculation that proves 200 TB at 100 Mbps is impossible within 2 weeks.

How to eliminate wrong answers

Option B is wrong because BigQuery Data Transfer Service for Oracle is a scheduled, incremental transfer service that relies on network connectivity and cannot handle the initial bulk load of 200 TB within the bandwidth limit. Option C is wrong because Datastream is a real-time change data capture (CDC) service for streaming changes, not designed for initial bulk transfers of large datasets. Option D is wrong because Storage Transfer Service is an online transfer tool that moves data over the network, which would be bottlenecked by the 100 Mbps link and cannot complete 200 TB within 2 weeks.

Practice this question →
19
MCQeasy

A data engineer needs to transfer 500 TB of archival data from an on-premises NAS to Cloud Storage. The on-premises network has limited bandwidth (100 Mbps). Which transfer method should they recommend?

A.Storage Transfer Service for on-premises
B.gsutil rsync
C.Transfer Appliance
D.Dataflow pipeline reading from NAS
AnswerC

At 100 Mbps, transferring 500 TB over the network would take months, far exceeding any practical window. Transfer Appliance ships physical storage to Google, sidestepping the bandwidth constraint entirely, whereas online methods remain bottlenecked by the same limited link.

Why this answer

The Transfer Appliance is a physical device designed for large-scale data transfers (up to petabytes) when network bandwidth is insufficient. With 500 TB of data and only 100 Mbps bandwidth, the theoretical transfer time would be over 500 days, making any online transfer method impractical. The Transfer Appliance bypasses network constraints entirely by shipping the data physically to Google Cloud.

Exam trap

The exam often tests the misconception that any cloud-native tool (like Storage Transfer Service or gsutil) can handle large data volumes regardless of bandwidth, ignoring the physical reality of network transfer times for archival-scale data.

How to eliminate wrong answers

Option A is wrong because Storage Transfer Service for on-premises requires network connectivity and is designed for smaller, incremental transfers, not for 500 TB over a 100 Mbps link. Option B is wrong because gsutil rsync is a command-line tool that relies on network bandwidth and would take an impractical amount of time (over 500 days) to transfer 500 TB at 100 Mbps. Option D is wrong because a Dataflow pipeline reading from NAS would still need to stream data over the limited 100 Mbps network, resulting in the same bandwidth bottleneck and excessive transfer time.

Practice this question →
20
Multi-Selectmedium

You need to create a Dataflow pipeline that reads from Pub/Sub and writes to BigQuery. The pipeline must handle malformed messages by writing them to a dead-letter table in BigQuery. Which two Apache Beam transforms or patterns should you use to achieve this? (Choose two.)

Select 2 answers
A.Use a ParDo transform with a side output for failed messages.
B.Write the failed messages to a separate BigQuery table using the side output from the ParDo.
C.Write all messages to a BigQuery table and use a view to filter out malformed records.
D.Use a ParDo transform that throws an exception for malformed messages, and configure a dead-letter queue in Dataflow.
E.Use a Filter transform to separate valid and invalid messages.
AnswersA, B

A ParDo transform can process each message and, upon detecting a malformed record, output it to a side output (using MultiOutputReceiver or withOutputTags). The main output continues with valid records. This is a standard pattern for dead-letter handling in Apache Beam, allowing separate processing of errors.

Why this answer

To handle malformed messages, you need to detect them and route them separately. A ParDo transform with a side output allows you to process each message and emit malformed ones to a secondary output. Then, you can write that side output to a BigQuery dead-letter table.

The combination of a ParDo with side output and a separate BigQueryIO.Write for that output achieves the required dead-letter pattern.

Exam trap

The trap here is thinking that a Filter transform alone can route failed messages to a different sink, but it only splits the PCollection and still requires additional transforms to write the failed messages.

Practice this question →
21
MCQhard

A Dataflow batch pipeline reads CSV files from Cloud Storage, joins them with a slowly changing dimension stored in BigQuery, and writes the enriched output to BigQuery. The dimension table is large and the join is causing excessive shuffle and worker memory pressure. The team wants to reduce shuffle while keeping the join logic in the pipeline. Which approach should they use?

A.Increase the number of workers and raise the disk size per worker to absorb the shuffle.
B.Use a CoGroupByKey on the two collections and process the grouped results.
C.Write the CSV data to BigQuery first, then run a SQL join in BigQuery and export the result.
D.Load the dimension side into a side input and use it in a ParDo to enrich each record.
AnswerD

A side input broadcasts the dimension data to every worker so the join happens locally without a shuffle of the main dataset. For a large but manageable dimension table, this eliminates the shuffle that causes memory pressure and speeds up enrichment. It keeps the join logic in the pipeline as required.

Why this answer

Using the dimension as a side input lets each worker enrich records locally, avoiding the shuffle and hot-key concentration that CoGroupByKey or a grouped join would introduce. It preserves the in-pipeline join logic and relieves memory pressure. Provisioning more workers or moving the join to BigQuery does not reduce the shuffle as requested.

Exam trap

The trap here is reaching for a grouped join such as CoGroupByKey by default, when a side input avoids the shuffle entirely for a broadcastable dimension.

Practice this question →
22
MCQhard

You are migrating an on-premises PostgreSQL database to Cloud SQL. You need to continuously replicate changes to BigQuery for real-time analytics with minimal latency. Which service should you use?

A.Dataflow with JDBC source
B.Pub/Sub with a Cloud Function that writes to BigQuery
C.Storage Transfer Service
D.Datastream
AnswerD

Datastream provides serverless change data capture, streaming PostgreSQL changes continuously into BigQuery with minimal latency. Unlike batch extracts or scheduled jobs, it replicates ongoing changes in near real time, meeting the continuous replication and low-latency analytics constraint.

Why this answer

Datastream is Google Cloud's serverless change data capture (CDC) service that continuously replicates changes from sources like PostgreSQL, MySQL, Oracle, and SQL Server into BigQuery or Cloud Storage with low latency. It reads the source's replication log and streams changes without impacting production workloads significantly. This directly matches the requirement for continuous, near-real-time replication into BigQuery.

Exam trap

PDE often tests CDC versus batch ingestion; candidates pick Dataflow because it is the general-purpose pipeline tool, but Dataflow alone does not provide native database CDC.

How to eliminate wrong answers

Option A is wrong because Dataflow with a JDBC source performs batch or scheduled reads, not continuous CDC, so it cannot deliver minimal-latency change replication. Option B is wrong because Pub/Sub with a Cloud Function requires custom application logic to capture and publish changes, and it does not natively read PostgreSQL replication logs. Option C is wrong because Storage Transfer Service moves files between object stores and has no database CDC capability.

Practice this question →
23
MCQhard

You are designing a Dataflow pipeline that reads from Pub/Sub and writes to BigQuery. The pipeline must handle late-arriving data and emit correct results. You need to ensure that the pipeline's windowing and triggering strategy produces accurate aggregations. Which combination of windowing and triggering should you use?

A.Session windows with default trigger and no allowed lateness.
B.Fixed windows with a trigger that fires after watermark passes and allowed lateness greater than 0.
C.Sliding windows with repeated trigger and allowed lateness set to a positive value.
D.Fixed windows with default trigger and allowed lateness of 0.
AnswerB

Fixed windows with a trigger that fires after the watermark passes the window end, combined with allowed lateness greater than 0, ensures that late data within the allowed lateness period is included in the window's aggregation. This setup produces accurate results for both on-time and late-arriving data. The trigger can also be configured to fire early or on repeated updates, but the key is to allow lateness so that late elements are not dropped.

Why this answer

To handle late-arriving data and produce accurate aggregations, you need a windowing strategy that allows late elements to be incorporated. Fixed windows with a trigger that fires after the watermark and a positive allowed lateness achieve this: the trigger emits results when the watermark passes, and any late data within the allowed lateness is added to the window, causing the trigger to fire again and update the results. This ensures completeness and correctness.

Exam trap

The trap here is assuming that the default trigger with zero allowed lateness is sufficient for late data, when in fact late elements are discarded after the watermark passes unless allowed lateness is set.

Practice this question →
24
MCQmedium

A company wants to use dbt (data build tool) to transform data in BigQuery. They have a Cloud Storage bucket containing raw CSV files that are loaded daily into BigQuery via an external table. Which dbt feature should they use to modularize the transformation logic and handle dependencies between models?

A.dbt tests
B.dbt snapshots
C.dbt models with ref()
D.dbt seeds
AnswerC

The ref() function creates a directed acyclic graph between dbt models, letting each model reference upstream ones by name rather than hard-coded table paths. This satisfies the modularisation and dependency-handling requirement, since dbt resolves build order automatically and materialises each model against the BigQuery external table.

Why this answer

C is correct because dbt models with the `ref()` function allow you to modularize SQL transformation logic and automatically handle dependencies between models. When you use `ref('model_name')`, dbt builds a dependency graph, ensuring models are executed in the correct order based on their references. This is essential for transforming raw data from an external table into a structured, analytics-ready dataset in BigQuery.

Exam trap

Candidates often confuse the purpose of dbt components: models with `ref()` manage transformation logic and dependencies, while tests handle data quality, snapshots track historical changes, and seeds load static data.

How to eliminate wrong answers

Option A is wrong because dbt tests are used for validating data quality (e.g., uniqueness, not null) and do not handle transformation logic or dependency management. Option B is wrong because dbt snapshots are designed to capture historical changes in slowly changing dimensions (Type 2 SCDs), not to modularize transformation logic or manage model dependencies. Option D is wrong because dbt seeds are used to load static CSV files directly into the warehouse as tables, not to transform data or manage dependencies between models.

Practice this question →
25
Multi-Selectmedium

A healthcare company needs to ingest HL7 messages from an on-premises system into Google Cloud. The messages arrive continuously and must be processed in near-real-time, with transformations applied before loading into BigQuery. The company wants to use a fully managed service for message ingestion and a serverless data processing service. They also need to ensure that the pipeline can handle bursts of traffic and that the data is encrypted at rest. Which TWO Google Cloud services should they use? (Choose two.)

Select 2 answers
A.Cloud Data Fusion
B.Cloud Dataflow
C.Cloud Dataproc
D.Cloud Pub/Sub
E.Cloud Composer
AnswersB, D

Dataflow is a fully managed, serverless service for stream and batch processing. It can read from Pub/Sub, apply transformations to HL7 messages, and write to BigQuery. It autoscales to handle traffic bursts and integrates with Cloud KMS for encryption. Dataflow provides exactly-once processing when needed and is ideal for near-real-time transformations.

Why this answer

Pub/Sub provides scalable, managed ingestion of HL7 messages with encryption at rest, and Dataflow offers serverless stream processing with autoscaling and exactly-once semantics. Together, they form a fully managed pipeline that can handle bursts and transform data before loading into BigQuery. Other services are either not serverless or not designed for real-time processing.

Exam trap

The trap here is selecting Dataproc or Data Fusion because they are data processing services, but they are not fully serverless and may require cluster management, which contradicts the requirement.

Practice this question →
26
MCQmedium

A company wants to move data from an on-premises MySQL database to BigQuery for analytics. They need to capture all changes (inserts, updates, deletes) in near real-time and also perform an initial historical load. Which approach meets these requirements with minimal operational overhead?

A.Use a Dataflow pipeline with a JDBC source to read the entire table periodically
B.Use Datastream to backfill historical data and then stream CDC changes to BigQuery
C.Use a one-time export to CSV and load into BigQuery, then set up a cron job to export incremental changes
D.Use Cloud SQL as an intermediary and enable binary logging, then stream to Pub/Sub via a custom connector
AnswerB

Datastream provides serverless change data capture from MySQL, streaming inserts, updates and deletes into BigQuery in near real-time, while its backfill capability performs the initial historical load. This satisfies both requirements without managing replication infrastructure, minimising operational overhead.

Why this answer

Datastream is correct because it provides serverless change data capture (CDC) from MySQL to BigQuery, supporting both historical backfill and continuous replication of inserts, updates, and deletes with minimal operational overhead. It reads the MySQL binary log (binlog) to stream changes in near real-time and can write directly to BigQuery or Cloud Storage.

Exam trap

PDE often tests the difference between batch and streaming ingestion; candidates incorrectly choose Dataflow or custom scripts for CDC, not realizing Datastream is the fully managed, serverless CDC service designed for minimal operational overhead.

How to eliminate wrong answers

Option A is wrong because a Dataflow JDBC pipeline reading entire tables periodically is batch-oriented, not near real-time, and does not capture deletes or updates efficiently; it also requires managing the pipeline and handling schema changes manually. Option C is wrong because one-time CSV export plus cron-based incremental exports is not real-time, cannot capture deletes, and requires custom scripting and scheduling, increasing operational overhead. Option D is wrong because using Cloud SQL as an intermediary and custom Pub/Sub connectors adds unnecessary complexity and operational burden; Datastream natively supports on-premises MySQL without needing Cloud SQL.

Practice this question →
27
MCQeasy

You are building a Dataflow pipeline that reads from Pub/Sub and writes to BigQuery. You need to ensure that no duplicates are written to BigQuery even if the pipeline retries messages. Which feature should you use?

A.Use a GroupByKey transform to deduplicate messages before writing to BigQuery.
B.Write to BigQuery using streaming inserts and then run a periodic deduplication job.
C.Use Pub/Sub's exactly-once delivery and rely on BigQuery's default streaming insert deduplication.
D.Enable exactly-once processing in Dataflow and use BigQuery's Storage Write API with exactly-once semantics.
AnswerD

Dataflow's exactly-once processing combined with the BigQuery Storage Write API's exactly-once delivery ensures that each message is written exactly once, even on retries. This is the most robust way to avoid duplicates in the target table.

Why this answer

The combination of Dataflow's exactly-once processing and the BigQuery Storage Write API's exactly-once semantics guarantees that each message is written once, even if the pipeline retries. This is the recommended approach for avoiding duplicates in streaming pipelines.

Exam trap

The trap here is relying on Pub/Sub's exactly-once delivery alone, which does not extend to the BigQuery sink and does not prevent duplicate writes.

Practice this question →
28
Multi-Selectmedium

Which three of the following are valid BigQuery data loading methods? (Choose THREE.)

Select 3 answers
A.Data Transfer Service from Amazon S3
B.Using Cloud SQL to write to BigQuery
C.Direct file upload via Cloud Dataproc
D.Batch load from Cloud Storage
E.Streaming inserts using the legacy streaming API
AnswersA, D, E

BigQuery Data Transfer Service natively supports scheduled, recurring transfers from Amazon S3 into BigQuery, satisfying the stem's requirement for a valid loading method. It handles authentication, incremental refreshes and backfills automatically, unlike one-off manual loads, making it a legitimate ingestion path alongside batch loads and streaming inserts.

Why this answer

Option A is correct because BigQuery Data Transfer Service natively supports scheduled, managed transfers from Amazon S3 into BigQuery, making it a valid loading method. Option D is correct because batch loading from Cloud Storage is one of the primary and most common ways to load data into BigQuery, using load jobs that read from GCS URIs. Option E is correct because streaming inserts via the legacy streaming API (tabledata.insertAll) are a supported, documented method for loading data into BigQuery in real time.

Option B is not a valid loading method because Cloud SQL is a managed relational database service and does not write directly into BigQuery; data would need to be exported and then loaded via a supported path. Option C is not valid because Cloud Dataproc is a managed Spark/Hadoop service, and there is no direct file upload mechanism from Dataproc into BigQuery; Dataproc would typically write to Cloud Storage, which is then loaded into BigQuery.

Exam trap

Google often tests the distinction between data processing services (like Dataproc) and actual data loading methods, leading candidates to confuse a processing step with a direct ingestion path.

Practice this question →
29
MCQhard

A company uses Eventarc to trigger a Cloud Run service when new objects appear in a GCS bucket. Recently, the Cloud Run service has been failing with 429 errors (too many requests) during high-velocity uploads. They need to handle the load without losing events. What should they do?

A.Use Cloud Functions instead of Cloud Run
B.Increase the maximum number of retries on the Eventarc trigger
C.Increase the Cloud Run service's request timeout
D.Configure the Eventarc trigger to send events to a Pub/Sub topic, and have the Cloud Run service pull from Pub/Sub
AnswerD

This decouples the event source from the consumer, allowing the Cloud Run service to process at its own pace and reducing 429 errors.

Why this answer

Sending events to a Pub/Sub topic decouples event production from consumption. Pub/Sub acts as a buffer that can absorb spikes in event volume, and the Cloud Run service can pull messages at its own pace, preventing 429 errors. This also ensures no events are lost, as Pub/Sub retains unacknowledged messages and retries delivery.

Exam trap

Google often tests the concept of decoupling event sources from consumers using a message queue or buffer like Pub/Sub, and the trap here is that candidates may think increasing retries or switching to Cloud Functions solves the load issue, when the real need is to absorb bursts via Pub/Sub.

How to eliminate wrong answers

Option A is wrong because Cloud Functions has similar concurrency limits and would also suffer from 429 errors under high load; it does not provide buffering. Option B is wrong because increasing retries on the Eventarc trigger only re-delivers failed events but does not address the root cause of the Cloud Run service being overwhelmed by too many concurrent requests. Option C is wrong because increasing the request timeout does not reduce the number of concurrent requests; it only allows longer processing time per request, which does not prevent the service from being overloaded.

Practice this question →
30
MCQeasy

You need to ingest data from a Cloud Storage bucket into BigQuery. The data is in Avro format and you want to minimize the time to insight. Which method should you use?

A.Use Cloud Data Fusion to read the Avro files and load them into BigQuery.
B.Use the BigQuery Streaming API to stream the Avro records from Cloud Storage.
C.Use a Dataflow pipeline to read the Avro files and write to BigQuery.
D.Use the BigQuery Load job with the Avro source format and enable autodetect schema.
AnswerD

Loading Avro files directly into BigQuery using a load job is the fastest and simplest method for batch ingestion. Avro is a supported format, and autodetect can infer the schema from the Avro file's embedded schema, eliminating manual schema definition. This approach minimizes time to insight because it avoids intermediate processing steps. It is ideal for one-time or scheduled batch loads from Cloud Storage.

Why this answer

The BigQuery Load job natively supports Avro and can automatically detect the schema from the Avro file's embedded schema. This makes it the fastest and simplest way to ingest Avro data from Cloud Storage, minimizing time to insight. Other methods like Dataflow, Streaming API, or Cloud Data Fusion add unnecessary complexity, latency, or cost for a straightforward batch load.

Exam trap

The trap here is overengineering the solution by choosing a pipeline or streaming service when a simple load job is sufficient and more efficient for batch Avro ingestion.

Practice this question →
31
MCQhard

You are designing a Dataflow pipeline that needs to exactly-once process events from Pub/Sub and write to BigQuery using the Storage Write API. The pipeline may restart and could reprocess some messages. What setting ensures exactly-once semantics for the output?

A.Use the legacy streaming inserts with insertId for deduplication
B.Use at-least-once delivery on Pub/Sub and idempotent writes to BigQuery
C.Use the Storage Write API in buffered mode with deduplication logic
D.Use the Storage Write API in committed mode and enable exactly-once semantic in Dataflow
AnswerD

Committed mode on the Storage Write API makes each write atomic and idempotent via stream offsets, and enabling exactly-once in Dataflow deduplicates retried bundles. Together they prevent duplicate rows when the pipeline restarts and reprocesses Pub/Sub messages.

Why this answer

The BigQuery Storage Write API in committed mode, combined with Dataflow's exactly-once processing, provides exactly-once semantics for writes to BigQuery. Committed mode uses a stream-based protocol where records are written and committed atomically, and Dataflow's exactly-once mode ensures that on pipeline restart, records are not duplicated. This is the recommended configuration for exactly-once streaming into BigQuery.

Exam trap

The trap is choosing legacy streaming inserts with insertId, which only provides best-effort deduplication, instead of the Storage Write API committed mode with Dataflow exactly-once, which is the actual mechanism for exactly-once semantics.

How to eliminate wrong answers

Option A is wrong because legacy streaming inserts with insertId provide best-effort deduplication, not true exactly-once semantics, and are deprecated for new pipelines. Option B is wrong because at-least-once delivery plus idempotent writes does not guarantee exactly-once output; duplicates can still occur if idempotency is imperfect. Option C is wrong because buffered mode does not provide the same exactly-once guarantees as committed mode and requires manual deduplication logic, which is not a built-in exactly-once setting.

Practice this question →
32
MCQmedium

Your team runs a Dataflow streaming pipeline that reads from Pub/Sub and writes to BigQuery. During a deployment, the pipeline is stopped and later restarted with the same job name but without draining. After the restart, downstream reports show duplicate rows for events that were processed just before the stop. You need the pipeline to resume without reprocessing already-published messages. What should you do?

A.Increase the number of workers with --maxNumWorkers before restarting the pipeline.
B.Drain the pipeline before stopping it, then start the replacement pipeline from the saved state.
C.Enable --enableStreamingEngine on the restarted pipeline to preserve message state.
D.Restart the pipeline using the --update option so the transform graph is replaced in place.
AnswerB

Draining a streaming Dataflow job stops ingestion of new data, finishes processing the in-flight elements, and commits the Pub/Sub acknowledgements and BigQuery writes. When the replacement job starts, it resumes from the committed state rather than replaying messages that were already acknowledged, which prevents the duplicate rows observed in the reports.

Why this answer

Stopping a streaming job without draining leaves in-flight Pub/Sub messages unacknowledged, so they are redelivered when a replacement job starts, producing duplicates. Draining first lets the pipeline finish processing and commit those acknowledgements and BigQuery writes, so the replacement resumes from a consistent state and avoids reprocessing the same events.

Exam trap

The trap here is assuming that restarting with the same job name or adding the update flag preserves exactly-once progress, when only draining commits in-flight Pub/Sub acknowledgements.

Practice this question →
33
Multi-Selectmedium

Which TWO statements are true about BigQuery Data Transfer Service? (Choose 2)

Select 2 answers
A.It supports data transformation during transfer.
B.It can transfer data from Cloud SQL to BigQuery directly.
C.It is only available in the US and EU regions.
D.It supports scheduled transfers from Amazon S3 and Redshift.
E.It can transfer data from Google Ads, YouTube, and Google Ad Manager into BigQuery.
AnswersD, E

BigQuery Data Transfer Service provides connectors that pull data from Amazon S3 and Amazon Redshift on scheduled, managed transfers into BigQuery. This cross-cloud ingestion capability satisfies the requirement, distinguishing it from services limited to Google-native sources.

Why this answer

Option D is correct because BigQuery Data Transfer Service provides built-in connectors for Amazon S3 and Amazon Redshift, enabling recurring, scheduled (and on-demand) transfers of external data into BigQuery without custom code. Option E is correct because the service also offers first-party Google connectors, including Google Ads, YouTube (Channel/Content Owner reports), and Google Ad Manager, which load those marketing datasets into BigQuery on a schedule. Option A is not correct because the Data Transfer Service is designed for ingestion/movement, not in-flight transformation; transformations are typically done after loading using BigQuery SQL, views, or scheduled queries.

Option B is not correct because Cloud SQL is not a native Data Transfer Service source; moving Cloud SQL data into BigQuery generally requires tools such as Datastream, the Cloud SQL federated query/EXTERNAL_QUERY, or export/load pipelines. Option C is not correct because BigQuery Data Transfer Service is available in many more locations than just US and EU, and its regional availability follows BigQuery dataset locations rather than being limited to those two regions.

Exam trap

A common misconception is that BigQuery Data Transfer Service can perform ETL transformations during transfer, but it is strictly an EL (Extract and Load) service without built-in transformation capabilities.

Practice this question →
34
MCQmedium

A team wants to transfer data from an on-premises Hadoop cluster to Cloud Storage for processing. The cluster is located in a remote area with limited bandwidth. They need to transfer 500 TB of data. Which service should they use?

A.Transfer Appliance
B.BigQuery Data Transfer Service
C.Storage Transfer Service
D.Dataproc with gsutil
AnswerA

Transfer Appliance ships a physical storage device to the remote site, letting the team copy 500 TB locally and return it to Google for upload. This bypasses the limited bandwidth that makes network-based transfer of that volume impractical.

Why this answer

Transfer Appliance is designed for petabyte-scale offline transfers when bandwidth is limited.

Practice this question →
35
MCQmedium

A company wants to ingest data from an on-premises Oracle database into BigQuery in near real-time with minimal latency. The database has a high volume of inserts and updates. Which service should they use?

A.Datastream
B.BigQuery Data Transfer Service
C.Pub/Sub
D.Storage Transfer Service
AnswerA

Datastream provides serverless change data capture (CDC) by reading Oracle redo logs, streaming inserts and updates continuously into BigQuery with sub-minute latency. This directly satisfies the near real-time, high-volume insert-and-update constraint, unlike batch extraction tools that cannot capture ongoing changes.

Why this answer

Datastream is correct because it is a serverless change data capture (CDC) service that replicates data from Oracle to BigQuery in near real-time with minimal latency. It reads the Oracle redo logs (via LogMiner or binary log reader) to capture inserts, updates, and deletes and streams them to BigQuery.

Exam trap

PDE often tests the difference between batch transfer services and CDC services; candidates incorrectly choose BigQuery Data Transfer Service or Pub/Sub for real-time Oracle replication, not realizing Datastream is the managed CDC solution.

How to eliminate wrong answers

Option B is wrong because BigQuery Data Transfer Service is designed for scheduled, batch data transfers from various sources (e.g., SaaS apps, Cloud Storage) and does not support real-time CDC from Oracle. Option C is wrong because Pub/Sub is a messaging service, not a CDC tool; it cannot read Oracle redo logs or replicate database changes without custom code. Option D is wrong because Storage Transfer Service is for moving files between object stores (e.g., S3 to GCS), not for database replication.

Practice this question →
36
MCQeasy

A data engineer needs to orchestrate a series of tasks that include calling external APIs, running BigQuery queries, and sending notifications. The workflow involves conditional branching and parallel steps. Which Google Cloud service should be used?

A.Workflows
B.Cloud Scheduler
C.Cloud Composer
D.Dataflow
AnswerA

Workflows is a fully managed orchestration service whose YAML-based syntax natively supports conditional branching, parallel branches and connector calls to external APIs, BigQuery and Pub/Sub. This directly satisfies the stem's requirement for branching and parallel steps without managing compute infrastructure.

Why this answer

Workflows is the correct choice because it is a fully managed orchestration service designed specifically for coordinating multi-step, event-driven workflows that involve conditional branching, parallel execution, and integration with external APIs, BigQuery, and notifications via HTTP calls or service integrations. It provides built-in error handling, retries, and a declarative YAML-based syntax that directly supports the described requirements without needing to manage infrastructure or schedule tasks.

Exam trap

The trap here is that candidates often confuse Cloud Scheduler as an orchestrator because it can trigger workflows, but it lacks the conditional branching and parallel execution capabilities required for this scenario, while Cloud Composer is mistakenly chosen due to its familiarity with Airflow, despite being heavier than necessary for a simple orchestration task.

How to eliminate wrong answers

Option B is wrong because Cloud Scheduler is a cron-based job scheduler that triggers tasks on a fixed schedule, not an orchestrator for complex workflows with conditional branching and parallel steps. Option C is wrong because Cloud Composer is a managed Apache Airflow service that can orchestrate workflows, but it is overkill for this use case, requires managing DAGs and infrastructure overhead, and is not the simplest or most cost-effective choice for a lightweight, event-driven workflow. Option D is wrong because Dataflow is a stream and batch data processing service based on Apache Beam, designed for transforming and analyzing data pipelines, not for orchestrating tasks like API calls, BigQuery queries, and notifications with conditional logic.

Practice this question →
37
MCQhard

You are designing a Dataflow pipeline that reads from Pub/Sub and writes to BigQuery. The pipeline uses the Storage Write API with exactly-once semantics. You need to ensure that the pipeline can handle a sudden spike in traffic without losing data or causing duplicates. Which configuration should you adjust to improve throughput while maintaining exactly-once?

A.Increase the number of Storage Write API streams.
B.Increase the maximum number of workers.
C.Enable autoscaling for the Dataflow job.
D.Switch to STREAMING_INSERTS method.
AnswerA

The Storage Write API allows multiple streams to be used concurrently. Increasing the number of streams (via withNumStorageWriteApiStreams) can improve throughput by parallelizing writes. This is a key tuning parameter for high-volume pipelines while preserving exactly-once semantics, as each stream maintains its own offset and deduplication.

Why this answer

The Storage Write API uses a configurable number of streams to write data to BigQuery. Each stream can handle a certain throughput, and increasing the number of streams allows more parallel writes, improving overall throughput. This adjustment maintains exactly-once semantics because each stream is managed with its own offset and deduplication.

Autoscaling and worker count help with processing but do not directly increase the write throughput limit imposed by the number of streams.

Exam trap

The trap here is assuming that scaling the Dataflow workers alone will increase BigQuery write throughput, but the Storage Write API has its own stream-based throughput limit that must be tuned separately.

Practice this question →
38
MCQmedium

You are building a Dataflow pipeline that reads Avro-formatted files from Cloud Storage. The files use a schema that is updated frequently. You want to minimize pipeline restarts due to schema changes. Which approach should you take?

A.Convert all Avro files to CSV before ingestion and use a fixed CSV schema.
B.Use a BigQuery load job with schema autodetect instead of Dataflow.
C.Read the Avro files using a schema inferred at runtime from the Avro file metadata.
D.Use a fixed Avro schema in your pipeline code and manually update it each time the source schema changes.
AnswerC

Avro files embed their schema in the file header, so a Dataflow pipeline can dynamically read that schema and adapt to changes without code changes or restarts. This is the standard approach for handling evolving Avro schemas in a streaming or batch pipeline.

Why this answer

Avro files include their schema in the file header, so a Dataflow pipeline can read that schema at runtime and adapt to changes automatically. This avoids the need to hardcode or manually update schemas, reducing pipeline restarts when the source schema evolves.

Exam trap

The trap here is assuming that Avro schemas must be defined statically in the pipeline code, when in fact Avro's self-describing format allows dynamic schema resolution.

Practice this question →
39
MCQeasy

Which BigQuery feature allows you to write data with exactly-once semantics, high throughput, and the ability to buffer data before making it available for queries?

A.BigQuery load jobs
B.BigQuery Data Transfer Service
C.Legacy streaming inserts
D.Storage Write API with buffered mode
AnswerD

Storage Write API's buffered mode holds records in a stream until you commit them, delivering exactly-once semantics through stream-level offsets and deduplication. This satisfies the stem's requirement to buffer data before query visibility, while sustaining high throughput — unlike the legacy streaming API, which inserts rows immediately.

Why this answer

The Storage Write API with buffered mode (option D) is correct because it provides exactly-once semantics for data ingestion, high throughput via gRPC streaming, and the ability to buffer data in memory before making it available for queries. This mode allows you to commit rows in a stream, ensuring no duplicates, while the buffering stage gives you control over when data becomes visible in BigQuery.

Exam trap

A common trap is assuming that legacy streaming inserts (option C) provide exactly-once semantics, but they actually offer at-least-once delivery. The Storage Write API with buffered mode is the only correct choice for exactly-once, high-throughput, buffered writes.

How to eliminate wrong answers

Option A is wrong because BigQuery load jobs offer at-least-once semantics (duplicates possible on retry) and do not support buffering before query availability; they write data directly to tables. Option B is wrong because BigQuery Data Transfer Service is a scheduled, managed service for importing data from external sources (e.g., Google Ads, Amazon S3) and does not provide high-throughput streaming or buffered write semantics. Option C is wrong because legacy streaming inserts (tabledata.insertAll) provide at-least-once semantics (duplicates can occur) and lack the buffered mode that defers query visibility; they also have lower throughput and no exactly-once guarantee.

Practice this question →
40
MCQmedium

A logistics company uploads nightly shipment manifests as newline-delimited JSON files into a Cloud Storage bucket. Analysts need to query these files with standard SQL immediately after upload, but the schema evolves frequently and the company wants to avoid managing a load job. Which approach should they use?

A.Create a BigQuery external table over the Cloud Storage bucket using the BigLake connection with JSON format and schema autodetect.
B.Mount the bucket in Dataproc and run a Hive external table over the JSON files.
C.Use the Storage Transfer Service to copy the JSON into a BigQuery dataset, then query the dataset.
D.Load each file into a native BigQuery table with a scheduled query that runs bq load every night.
AnswerA

BigQuery external tables over Cloud Storage let you query files in place with standard SQL, and BigLake connections add governance plus support for schema autodetect on newline-delimited JSON. Because the data stays in Cloud Storage, no load job or pipeline maintenance is required, and new files matching the URI prefix are picked up automatically, which fits a frequently evolving manifest schema.

Why this answer

External tables let BigQuery read files directly from Cloud Storage, so analysts can run standard SQL over the manifests without a load job, and BigLake-backed external tables support schema autodetection for newline-delimited JSON. Because the schema changes often, autodetect combined with in-place querying avoids brittle load pipelines while still exposing the data through the BigQuery SQL surface.

Exam trap

The trap here is assuming that querying files in Cloud Storage requires moving them into BigQuery storage first, when external tables can query them in place.

Practice this question →
41
MCQmedium

Your team uses dbt to transform data in BigQuery. You need to schedule dbt runs to refresh materialized tables and views every hour. The transformations include both full refreshes and incremental models. What is the most efficient way to orchestrate these dbt runs on Google Cloud?

A.Use Cloud Composer (Airflow) to schedule and run dbt commands.
B.Use Cloud Build with a trigger to run dbt every hour.
C.Use Cloud Scheduler to trigger a Cloud Function that runs dbt.
D.Set up a cron job on a Compute Engine instance to run dbt.
AnswerA

Cloud Composer runs Airflow DAGs that invoke dbt commands, giving dependency management, retries and hourly scheduling across both full-refresh and incremental models. Airflow's operators orchestrate the dbt runs and surface failures, which raw cron or Cloud Scheduler cannot manage as cleanly.

Why this answer

Cloud Composer (Airflow) is the recommended orchestration tool for complex workflows like dbt runs, supporting dependencies, retries, and scheduling. Cloud Scheduler alone cannot run dbt directly; it can trigger a Cloud Function to run dbt, but that is less maintainable. Cloud Build is CI/CD, not scheduling.

Using a cron job on Compute Engine is possible but not managed.

Practice this question →
42
MCQeasy

A logistics company stores delivery manifests as newline-delimited JSON files in a Cloud Storage bucket. Analysts want to run SQL queries over these files immediately without importing them into a BigQuery dataset or paying for duplicate storage. Which BigQuery capability should they use?

A.Use the BigQuery Storage Write API to stream the files into a dataset.
B.Mount the bucket as a BigQuery dataset using the Cloud Storage connector.
C.Create an external table over the Cloud Storage bucket using the JSON type.
D.Load the JSON files with a scheduled query that runs every hour.
AnswerC

BigQuery external tables can point directly at newline-delimited JSON objects in Cloud Storage, letting analysts run SQL without loading the data into managed storage. This satisfies the requirement to query immediately and avoid duplicate storage costs, since the data stays in the bucket and is read at query time.

Why this answer

External tables let BigQuery read newline-delimited JSON directly from Cloud Storage at query time, so analysts get immediate SQL access without loading data into managed storage. The files remain in the bucket, avoiding duplicate storage charges while still supporting standard SQL over the manifests.

Exam trap

The trap here is reaching for a loading or streaming mechanism when the requirement explicitly forbids duplicating storage and demands immediate access.

Practice this question →
43
MCQmedium

A company needs to load data from a MySQL database into BigQuery daily. The data volume is 10 GB per day and the schema changes occasionally. They want to minimize costs and operational overhead. What is the MOST appropriate approach?

A.Use Datastream to stream changes from MySQL to BigQuery
B.Use BigQuery Data Transfer Service for MySQL
C.Export MySQL data to CSV, upload to GCS, and use BigQuery load jobs
D.Use Cloud SQL federated query from BigQuery
AnswerC

This approach requires manual handling of schema changes and daily exports, which is more operational overhead.

Why this answer

For a daily batch load of 10 GB with occasional schema changes, the most cost-effective and low-overhead approach is to export the MySQL data to CSV or Parquet, upload it to Cloud Storage, and use BigQuery load jobs. BigQuery load jobs are free (you only pay for storage and queries), and they can handle schema changes via schema autodetect or by updating the table schema. Datastream is designed for continuous change data capture with low latency, which is unnecessary and likely more expensive for a once-daily batch load.

BigQuery Data Transfer Service does not support MySQL as a source, and Cloud SQL federated queries are for querying external data, not for loading it into BigQuery.

Exam trap

A common misconception is that Datastream is always the best choice for moving data from MySQL to BigQuery. However, Datastream is intended for continuous replication and is not cost-optimal for periodic batch loads. For a daily batch load, the classic extract-load pattern (export to GCS, then BigQuery load job) is more appropriate and cost-effective.

How to eliminate wrong answers

Option B is wrong because BigQuery Data Transfer Service for MySQL is not a supported service; BigQuery Data Transfer Service supports sources like Google Ads, Amazon S3, and Teradata, but not direct MySQL connections. Option C is wrong because exporting MySQL to CSV, uploading to GCS, and using load jobs incurs higher operational overhead (manual scripting, schema management) and does not handle schema changes gracefully, requiring manual intervention for each change. Option D is wrong because Cloud SQL federated queries from BigQuery are designed for ad-hoc querying of live Cloud SQL data, not for daily bulk ingestion, and they do not persist data in BigQuery, leading to repeated query costs and no historical retention.

Practice this question →
44
MCQhard

A logistics company streams vehicle telemetry into a Pub/Sub topic. Each message contains a vehicle ID and a timestamp, and the Dataflow pipeline computes per-vehicle distance using a stateful DoFn with a ValueState timer set to fire after 5 minutes of event-time inactivity. During a regional network outage, some vehicles stop sending data for 20 minutes and then resume with correctly ordered timestamps. After the outage, operators notice that some late-arriving records are being dropped before the stateful computation. Which pipeline setting should be adjusted to retain those records for processing?

A.Switch the pipeline's windowing from event-time to processing-time windows.
B.Set an allowed lateness on the windowing strategy that exceeds the 20-minute outage gap.
C.Increase the Pub/Sub subscription acknowledgement deadline so messages remain outstanding longer.
D.Change the stateful DoFn timer from event-time to processing-time so it fires only when data resumes.
AnswerB

Allowed lateness extends the window's lifetime beyond the watermark so records arriving after the watermark passes the window end are still processed instead of being dropped. A value greater than 20 minutes covers the outage gap. The stateful DoFn timers continue to fire, but late elements are routed into the still-open window and contribute to distance calculations.

Why this answer

Late data handling is governed by allowed lateness on the window, not by Pub/Sub delivery settings or the timer domain. When the watermark advances past a window end after a 20-minute gap, records with earlier timestamps are dropped unless the window is kept alive. Setting allowed lateness beyond the outage gap lets the stateful computation include the resumed telemetry in the correct event-time window.

Exam trap

The trap here is confusing message-level delivery guarantees in Pub/Sub with event-time window semantics in Beam, where late data is discarded based on the watermark rather than on acknowledgement timing.

Practice this question →
45
MCQeasy

A data analyst needs to transform nested and repeated fields in BigQuery. They have a table with a column of type ARRAY<STRUCT<...>>. Which SQL function should they use to flatten the array into individual rows for analysis?

A.STRUCT
B.CAST
C.UNNEST
D.REPLACE
AnswerC

UNNEST expands an ARRAY into a set of rows, and combined with a CROSS JOIN or comma join it flattens ARRAY<STRUCT<...>> columns so each struct becomes an individual row. This directly satisfies the flattening requirement for analysis.

Why this answer

UNNEST is used to flatten arrays into rows. STRUCT is used to group fields. CAST is for type conversion.

REPLACE is for string replacement.

Practice this question →
46
MCQhard

You need to process a large volume of event data from Cloud Storage, apply complex transformations using Apache Spark, and then load the results into BigQuery. The data arrives in batches every hour. You want to minimize costs by using preemptible VMs. Which service should you use?

A.Cloud Composer
B.BigQuery
C.Dataproc
D.Dataflow
AnswerC

Dataproc runs Apache Spark natively and supports preemptible VMs for worker nodes, cutting compute costs substantially. It handles hourly batch ingestion from Cloud Storage, applies the complex Spark transformations, and writes results into BigQuery, matching the cost-minimisation constraint.

Why this answer

Dataproc is a managed Apache Spark and Hadoop service that supports preemptible VMs, making it the ideal choice for batch processing with complex transformations using Spark while minimizing costs. It integrates natively with Cloud Storage for input and BigQuery for output, and preemptible VMs can reduce compute costs by up to 80%. The hourly batch pattern aligns with Dataproc's job-based execution model.

Exam trap

PDE often tests the confusion between Dataflow and Dataproc, where candidates pick Dataflow for Spark workloads, but Dataflow uses Apache Beam, not Spark, and does not support preemptible VMs in the same cost-optimized way.

How to eliminate wrong answers

Option A is wrong because Cloud Composer is a workflow orchestration service (managed Airflow) used to schedule and monitor pipelines, not to run Spark transformations itself. Option B is wrong because BigQuery is a serverless data warehouse for analytics, not a Spark processing engine; it cannot run custom Spark code. Option D is wrong because Dataflow is based on Apache Beam, not Apache Spark, and while it supports batch processing, it does not natively run Spark jobs and does not use preemptible VMs in the same way (it uses worker VMs but the programming model is Beam, not Spark).

Practice this question →
47
MCQmedium

A company uses dbt on BigQuery to transform data. They want to run dbt models on a schedule and manage environments (dev, prod). Which GCP service should they use to run dbt jobs?

A.Dataflow
B.Cloud Composer
C.Cloud Scheduler
D.Cloud Build
AnswerB

Cloud Composer provides managed Apache Airflow scheduling, letting dbt jobs run on cron-like schedules with separate dev and prod environments configured as distinct DAG variables or connections. This satisfies both the scheduling and environment-management requirements.

Why this answer

Cloud Composer is an Apache Airflow managed service that can schedule dbt runs.

Practice this question →
48
MCQhard

Your team is migrating a batch ETL job from an on-premises Hadoop cluster to Dataproc. The job reads CSV files from Cloud Storage, joins them with a slowly changing dimension table in BigQuery, and writes aggregated results back to BigQuery. The on-premises job used Hive on Tez and took six hours. You need to reduce runtime on Dataproc while minimizing cost. Which approach should you take?

A.Run the job as a Dataproc Serverless for Spark batch, reading the CSV files from Cloud Storage and using the BigQuery connector to read the dimension table, with autoscaling enabled.
B.Create a Dataproc cluster with a fixed number of standard workers, install Hive, and run the same Hive on Tez script to preserve the existing logic.
C.Create a long-running Dataproc cluster with many preemptible workers and run the job with Spark SQL, caching the BigQuery dimension table as a Spark DataFrame.
D.Export the CSV files to a temporary BigQuery table, perform the join and aggregation entirely in BigQuery SQL, and schedule the query with Cloud Scheduler.
AnswerA

Dataproc Serverless for Spark provisions resources on demand, scales with the workload, and shuts down when the batch completes, so you pay only for the execution time. The BigQuery connector reads the dimension table efficiently, and Spark's optimizer can broadcast the dimension if it is small. This combination reduces runtime versus the legacy Tez job and minimizes cost by avoiding an idle cluster.

Why this answer

Dataproc Serverless for Spark runs batch workloads on ephemeral, autoscaling infrastructure that is released when the batch finishes, so cost tracks actual execution rather than idle cluster time. Reading CSV from Cloud Storage and using the BigQuery connector for the dimension table lets Spark optimize the join, often broadcasting the small dimension. This reduces runtime compared with the legacy Tez engine and avoids paying for a cluster that sits idle between runs.

Exam trap

The trap here is defaulting to a persistent cluster with preemptible workers for cost savings, when preemption can extend runtime and an always-on cluster bills for idle time.

Practice this question →
49
MCQeasy

A media company stores 400 TB of compressed JSON clickstream logs in a Cloud Storage bucket. Analysts need to run ad hoc SQL over this data several times a day, and the team wants to avoid the cost and delay of loading it into BigQuery. Which BigQuery capability should you use?

A.Load the JSON files into a BigQuery native table partitioned by ingestion time.
B.Create an external table over the Cloud Storage files using the BigLake connection, and query it directly.
C.Use the Storage Transfer Service to copy the bucket into a second bucket in the same region as the dataset.
D.Create a BigQuery dataset and use the bq command-line tool to stream the JSON records with the insertAll API.
AnswerB

BigQuery external tables backed by a BigLake connection let you run standard SQL over data that remains in Cloud Storage, so the clickstream logs are queried in place without an ingestion job. This satisfies the ad hoc querying need and avoids duplication of the 400 TB, while the connection provides fine-grained access control and metadata caching.

Why this answer

External tables over Cloud Storage, accessed through a BigLake connection, keep the data in its original bucket while exposing it to BigQuery SQL. This removes both the load latency and the duplicate storage cost that the team is trying to avoid. Native loading, bucket-to-bucket copying, and streaming inserts all either duplicate the data or provide no query surface at all.

Exam trap

The trap here is treating the BigLake connection as a data movement mechanism, when it is actually a governance and metadata layer that lets BigQuery read the Cloud Storage objects in place without copying them.

Practice this question →
50
MCQmedium

You are building a Dataflow pipeline that reads messages from Pub/Sub, performs windowed aggregations, and writes results to BigQuery. The pipeline must handle late-arriving data, but after a certain point, you want to stop accepting late data to allow the pipeline to emit final results. Which Apache Beam concept should you use to define when to stop accepting late data?

A.Side input
B.Allowed lateness
C.Watermark
D.Trigger
AnswerB

Allowed lateness specifies how long after the watermark passes the end of a window the pipeline will still accept and process late data. After this period, late data is dropped. This is exactly what is needed to bound the waiting period for late-arriving events, enabling final results to be emitted. It is a key parameter in windowing strategies.

Why this answer

Allowed lateness defines the maximum time after the watermark that late data will still be processed. Once this period elapses, the pipeline drops late data and can finalize results. This directly addresses the requirement to stop accepting late data after a certain point, while still accommodating some delay.

Other concepts like watermarks and triggers manage timing but do not enforce a cutoff for late data.

Exam trap

The trap here is confusing the watermark (which estimates completeness) with allowed lateness (which explicitly sets a deadline for late data).

Practice this question →
51
MCQhard

You are using Dataproc to run a Spark job that reads data from Cloud Storage, performs aggregations, and writes results back to Cloud Storage. The job is failing with out-of-memory errors on the shuffle. Which optimization should you apply?

A.Increase spark.sql.shuffle.partitions
B.Use RDDs instead of DataFrames
C.Increase spark.executor.memory
D.Decrease the number of executors
AnswerA

Raising `spark.sql.shuffle.partitions` splits shuffle data into more, smaller partitions, so each executor task holds less in memory during aggregation. This directly relieves the out-of-memory condition on the shuffle stage, satisfying the stem's constraint without changing cluster size or storage layout.

Why this answer

Out-of-memory errors during a shuffle in Spark are usually caused by too few shuffle partitions, which makes each partition (and its in-memory sort/aggregation buffer) too large. Increasing spark.sql.shuffle.partitions splits the shuffle into more, smaller partitions, reducing per-task memory pressure and allowing the job to complete.

Exam trap

PDE often tests the reflex to 'add more memory' for any OOM — candidates pick C, but shuffle OOM is a partitioning/skew problem, and the correct first lever is spark.sql.shuffle.partitions, not executor memory.

How to eliminate wrong answers

Option B is wrong because RDDs are lower-level than DataFrames and lack Catalyst/Tungsten optimizations — switching to RDDs would generally increase memory usage and slow the job, not fix shuffle OOM. Option C is wrong because increasing spark.executor.memory can help only if the executor itself is the bottleneck; shuffle OOM is typically caused by skewed or oversized partitions, and simply adding memory often just delays the failure or causes GC pressure. Option D is wrong because decreasing executors reduces total cluster parallelism and memory available for the shuffle, making OOM more likely, not less.

Practice this question →
52
MCQmedium

A retail analytics team loads point-of-sale transactions into BigQuery every 15 minutes using a load job that appends records. A nightly deduplication job currently removes duplicate transaction IDs, but the team wants duplicates eliminated at ingestion time without adding a separate job. The source system can emit a stable transaction ID for each record. What should you do?

A.Change the load job to write to a staging table, then run a MERGE statement to upsert into the final table.
B.Add a unique constraint to the destination table's transaction ID column so duplicate inserts are rejected.
C.Partition the table by transaction date and cluster it by transaction ID, then rely on the storage engine to drop duplicates.
D.Use a MERGE statement in the load pipeline that matches on transaction ID, or write through the Storage Write API with the transaction ID as the primary key.
AnswerD

Both a MERGE keyed on transaction ID and the Storage Write API with a primary key collapse duplicates as part of the write itself. The Storage Write API in particular applies the primary key at commit time, so a re-delivered transaction ID is discarded without any separate cleanup job, satisfying the ingestion-time deduplication requirement.

Why this answer

Deduplication at write time requires the write path itself to enforce identity: either a MERGE keyed on transaction ID or the Storage Write API with a primary key. Physical layout choices such as partitioning and clustering improve query performance but never remove rows, and BigQuery does not enforce unique constraints. A staging-plus-MERGE design works but reintroduces the extra job the team is trying to eliminate.

Exam trap

The trap here is assuming that a unique constraint or a clustering key on transaction ID prevents duplicate rows from being stored, when BigQuery enforces neither for deduplication purposes.

Practice this question →
53
MCQeasy

A startup uploads JSON log files to a Cloud Storage bucket throughout the day and wants BigQuery to query them within minutes with no ETL code and no data duplication. The files use newline-delimited JSON and the schema is stable. The team wants the lowest-effort option that still lets queries see newly arrived files automatically. Which approach should they use?

A.Schedule a nightly bq load job to append the JSON files into a native BigQuery table
B.Build a Dataflow streaming job that parses the JSON and writes rows to BigQuery
C.Use the Storage Write API to stream each uploaded file's contents directly into a BigQuery table
D.Create a BigQuery external table over the Cloud Storage bucket using the newline-delimited JSON format
AnswerD

An external table over Cloud Storage lets BigQuery query the JSON files in place without loading or duplicating data, and newly added files that match the URI pattern are visible to subsequent queries automatically. No ETL code or pipeline is required, so this is the lowest-effort option that still reflects new arrivals within minutes.

Why this answer

A BigQuery external table reads newline-delimited JSON directly from Cloud Storage, so no data is copied and no pipeline code is needed. Because the table definition is a URI pattern over the bucket, files added later are picked up by subsequent queries, which satisfies the fast-visibility and low-effort requirements.

Exam trap

The trap here is assuming data must be loaded into BigQuery storage before it can be queried, when external tables query Cloud Storage in place.

Practice this question →
54
MCQeasy

A company wants to trigger a Cloud Run service whenever a new file is uploaded to a specific Cloud Storage bucket. Which event-driven solution should they use?

A.Eventarc with Cloud Storage trigger and Cloud Run destination
B.Cloud Scheduler to periodically poll the bucket
C.Cloud Functions triggered by Cloud Storage
D.Pub/Sub with a push subscription to Cloud Run
AnswerA

Eventarc receives Cloud Storage object-finalise events and routes them to Cloud Run, providing native event-driven invocation without polling. This satisfies the requirement to trigger the service whenever a new file lands in the specified bucket.

Why this answer

Eventarc is the recommended service for routing events from Cloud Storage to Cloud Run because it provides a fully managed, event-driven architecture with built-in filtering and retry logic. When a new file is uploaded, Cloud Storage emits a notification that Eventarc captures and delivers directly to the Cloud Run service as an HTTP request, enabling serverless processing without polling or additional infrastructure.

Exam trap

The trap here is that candidates confuse Cloud Functions (option C) as the only serverless compute option for Cloud Storage events, overlooking that Eventarc is the modern, preferred service for routing events to Cloud Run, and that Pub/Sub (option D) requires manual setup not shown in the question.

How to eliminate wrong answers

Option B is wrong because Cloud Scheduler is a cron job service for scheduled, not event-driven, tasks; periodically polling a bucket introduces latency and inefficiency, and it cannot react instantly to uploads. Option C is wrong because Cloud Functions triggered by Cloud Storage is a valid event-driven approach, but the question specifically asks for a Cloud Run destination, and Cloud Functions cannot directly invoke Cloud Run without additional integration. Option D is wrong because Pub/Sub with a push subscription to Cloud Run requires manually configuring Cloud Storage to publish to Pub/Sub, which adds complexity and is not the native, recommended pattern for Cloud Storage events; Eventarc abstracts this by directly managing the event flow from Cloud Storage to Cloud Run.

Practice this question →
55
MCQmedium

A media analytics team needs to copy 200 TB of historical log files from an on-premises NFS server into Cloud Storage, then transform them with a Dataflow batch job. The NFS server is reachable only from the corporate network, and the team wants to minimize transfer time and cost. Which approach should they use?

A.Create a Cloud VPN tunnel and use gcloud storage cp from an on-premises machine to upload the files.
B.Mount the NFS share on a Compute Engine VM and run gsutil -m rsync to copy the data to Cloud Storage.
C.Use Storage Transfer Service with an agent pool deployed in the corporate network to transfer directly from the NFS server to Cloud Storage.
D.Use Transfer Appliance to ship the data to Google, then load it into Cloud Storage.
AnswerC

Storage Transfer Service supports on-premises sources through agent pools, which run in the corporate network and pull data from NFS to Cloud Storage. This avoids staging through a VPN and is designed for large-scale, reliable transfers with scheduling and integrity checks, making it the right fit for 200 TB from an NFS source.

Why this answer

Storage Transfer Service with an agent pool is purpose-built for moving large datasets from on-premises sources like NFS to Cloud Storage. Agents run inside the corporate network, so no inbound firewall changes are needed, and the service handles parallelism, retries, and integrity checks. Manual rsync, Transfer Appliance, and single-machine VPN copies are slower or inappropriate at this scale.

Exam trap

The trap here is assuming a VPN plus command-line copy is sufficient for large on-premises transfers, when agent-based Storage Transfer Service is the scalable, managed option.

Practice this question →
56
MCQmedium

A company wants to transform data using dbt (data build tool) on BigQuery. They have a CI/CD pipeline and need to version-control their transformations. Which setup is recommended?

A.Create Dataflow pipelines for each transformation
B.Deploy dbt models in a Cloud Build pipeline that runs dbt run
C.Use Cloud Composer to orchestrate dbt jobs
D.Run dbt directly on BigQuery using scripting
AnswerB

Running dbt models inside a Cloud Build pipeline executes the transformations on BigQuery while the dbt project files remain in source control, giving versioned, repeatable deployments. This satisfies the CI/CD and version-control constraints without manual execution.

Why this answer

Dbt is designed for version-controlled, SQL-based transformations, and integrating it with Cloud Build allows you to run `dbt run` as part of a CI/CD pipeline. This setup ensures that every change to dbt models is automatically tested and deployed, which aligns with the requirement for version control and automated deployment on BigQuery.

Exam trap

This question tests the distinction between orchestration (Cloud Composer) and CI/CD (Cloud Build), so candidates mistakenly choose Cloud Composer because they think scheduling equals version control, but the question explicitly requires version control and CI/CD, not just scheduling.

How to eliminate wrong answers

Option A is wrong because Dataflow pipelines are intended for stream or batch data processing using Apache Beam, not for version-controlled SQL transformations; they add unnecessary complexity and cost for simple transformation logic. Option C is wrong because Cloud Composer (Apache Airflow) is an orchestration tool for scheduling and monitoring workflows, not a CI/CD pipeline for version-controlled dbt models; while it can run dbt, it is not the recommended setup for version control and automated deployment. Option D is wrong because running dbt directly on BigQuery using scripting bypasses version control, CI/CD integration, and proper environment management, leading to manual, error-prone processes.

Practice this question →
57
MCQeasy

A data engineer needs to load a 10 GB CSV file from GCS into BigQuery. The file contains some malformed rows that should be skipped. Which approach is most efficient?

A.Use Dataproc to run a Spark job that cleans the data and writes to BigQuery
B.Use the Storage Write API to stream each row, skipping bad ones in code
C.Use a Dataflow pipeline to read CSV, filter bad rows, and write to BigQuery
D.Use the bq command-line tool with the --max_bad_records flag
AnswerD

bq load with --max_bad_records skips malformed rows efficiently.

Why this answer

The `bq` command-line tool's `--max_bad_records` flag allows BigQuery's native CSV loader to skip malformed rows up to a specified limit during a load job. This is the most efficient approach for a one-time batch load of a 10 GB file, as it avoids the overhead of spinning up separate processing clusters (Dataproc, Dataflow) or streaming each row individually, leveraging BigQuery's optimized ingestion pipeline.

Exam trap

Google often tests the misconception that complex ETL pipelines (Spark, Dataflow) are always required for data cleaning, when in fact BigQuery's native load options like `--max_bad_records` can handle common malformed row scenarios directly and more efficiently.

How to eliminate wrong answers

Option A is wrong because using Dataproc to run a Spark job introduces unnecessary complexity and cost; for a simple load with malformed row skipping, a native BigQuery load job is far more efficient without needing a separate cluster. Option B is wrong because the Storage Write API is designed for real-time streaming, not batch loading a 10 GB file; streaming each row would be slower, more expensive, and less reliable than a single batch load with `--max_bad_records`. Option C is wrong because a Dataflow pipeline adds unnecessary processing overhead and cost; while it can filter bad rows, BigQuery's native load job with `--max_bad_records` achieves the same result more directly without requiring a separate data processing service.

Practice this question →
58
Multi-Selectmedium

A company wants to use Eventarc to trigger a Cloud Run service when new objects are created in a GCS bucket. They also need to filter events for a specific bucket and object prefix. Which THREE resources must exist or be created?

Select 3 answers
A.Cloud Storage bucket
B.Pub/Sub topic
C.Cloud Scheduler job
D.Eventarc trigger
E.Cloud Run service
AnswersA, D, E

The Cloud Storage bucket is the event source whose object-creation notifications Eventarc consumes. It must exist before any trigger can target it, satisfying the scenario's requirement to react to new objects, and its name forms part of the trigger's filtering criteria.

Why this answer

Option A (Cloud Storage bucket) is correct because Eventarc's Cloud Storage events originate from an actual GCS bucket, and the trigger's bucket filter must reference that existing bucket where objects are created. Option D (Eventarc trigger) is correct because the trigger is the resource that connects the event source to the destination, defines the event type (e.g., google.cloud.storage.object.v1.finalized), and applies the bucket and object-prefix attribute filters. Option E (Cloud Run service) is correct because it is the event destination that Eventarc invokes when a matching object-creation event fires.

Option B (Pub/Sub topic) is not required because Eventarc manages its own internal Pub/Sub transport for Cloud Storage events; you do not create or specify a topic. Option C (Cloud Scheduler job) is not required because scheduling is unrelated to event-driven object-creation triggers.

Exam trap

PDE often tests the misconception that you must manually create a Pub/Sub topic for Eventarc — Eventarc handles it automatically, so it's not a required resource.

Practice this question →
59
MCQeasy

Which Google Cloud service is designed to replicate data from MySQL, PostgreSQL, and Oracle databases to BigQuery or Cloud Storage in near real-time?

A.Cloud Data Fusion
B.Datastream
C.Dataflow
D.Pub/Sub
AnswerB

Datastream is a serverless change data capture service that streams MySQL, PostgreSQL and Oracle changes into BigQuery or Cloud Storage with low latency. Its native source connectors and near real-time replication match the requirement, unlike batch-oriented transfer or replication tools.

Why this answer

Datastream is a serverless CDC service that ingests change data from relational databases into GCS or BigQuery.

Practice this question →
60
MCQeasy

Your company is migrating an on-premises Hadoop cluster to Google Cloud. You need to transform large datasets using Spark SQL. Which Google Cloud service should you use?

A.Dataflow
B.Dataproc
C.BigQuery
D.Cloud Dataprep
AnswerB

Dataproc is a managed Spark and Hadoop service, so existing Spark SQL jobs run with minimal refactoring. It satisfies the migration constraint by providing managed cluster provisioning and autoscaling while preserving the Spark SQL transformation logic used on-premises.

Why this answer

Dataproc is the managed Spark and Hadoop service on Google Cloud, purpose-built for running existing Spark SQL workloads with minimal changes. It allows you to spin up a cluster, run your Spark SQL transformations on large datasets stored in Cloud Storage or BigQuery, and then tear it down, making it the direct equivalent of an on-premises Hadoop cluster in the cloud.

Exam trap

Google Cloud certification exams often test the distinction between managed Spark (Dataproc) and serverless SQL (BigQuery) or Beam-based processing (Dataflow), trapping candidates who see 'SQL' and immediately think of BigQuery without recognizing the Spark SQL execution context.

How to eliminate wrong answers

Option A is wrong because Dataflow is a unified stream and batch processing service based on Apache Beam, not Spark SQL; migrating Spark SQL code to Dataflow would require rewriting the entire pipeline in Beam. Option C is wrong because BigQuery is a serverless data warehouse for SQL analytics, not a platform for running Spark SQL transformations; it does not support Spark execution engines. Option D is wrong because Cloud Dataprep is a visual data preparation tool for cleaning and structuring data, not a service for running Spark SQL jobs; it cannot execute custom Spark code.

Practice this question →
61
MCQmedium

A company wants to orchestrate a multi-step data processing workflow that includes calling a Cloud Run service, waiting for its completion, and then running a BigQuery query. The workflow should be serverless and integrate with Cloud Events. Which Google Cloud service should they use?

A.Eventarc
B.Cloud Workflows
C.Cloud Composer
D.Cloud Dataflow
AnswerB

Cloud Workflows orchestrates the multi-step sequence, invoking the Cloud Run service, awaiting its completion, then running the BigQuery query, all serverless. It natively consumes Cloud Events via Eventarc triggers, satisfying the stem's event-integration constraint without managing infrastructure.

Why this answer

Cloud Workflows is the correct choice because it is a serverless workflow orchestrator that can coordinate multi-step processes involving Cloud Run and BigQuery. It natively supports waiting for asynchronous operations (like Cloud Run job completion) via its 'call' and 'wait' steps, and it can trigger subsequent steps such as BigQuery queries. Additionally, Cloud Workflows can be triggered by Cloud Events, making it fully integrated with the event-driven architecture described.

Exam trap

The trap here is that candidates confuse Google Cloud's Eventarc (event routing) with workflow orchestration, assuming that routing events alone can handle sequencing and waiting, when in fact Eventarc lacks the state management and step coordination required for multi-step workflows.

How to eliminate wrong answers

Option A is wrong because Eventarc is a service for routing events from various sources to targets (like Cloud Run, Cloud Functions), but it does not provide workflow orchestration capabilities such as waiting for completion or sequencing steps. Option C is wrong because Cloud Composer is a managed Apache Airflow service that is not serverless (it requires provisioning and managing a cluster of workers) and is overkill for a simple multi-step workflow; it is designed for complex, scheduled pipelines, not lightweight event-driven orchestration. Option D is wrong because Cloud Dataflow is a stream and batch data processing service (based on Apache Beam) that focuses on transforming data pipelines, not on orchestrating heterogeneous services like Cloud Run and BigQuery queries; it lacks native workflow sequencing and event-driven triggers.

Practice this question →
62
MCQhard

A data engineer is designing a Dataflow pipeline that reads from Pub/Sub and writes to BigQuery using the Storage Write API in exactly-once mode. The pipeline must handle late-arriving data (up to 1 hour) and maintain correct aggregation results. Which trigger configuration should they use?

A.Default trigger without allowed lateness
B.After count trigger with count of 1000
C.Fixed time trigger every 5 minutes
D.After watermark trigger with allowed lateness of 1 hour
AnswerD

This configuration ensures late data within 1 hour is included in aggregations.

Why this answer

The After watermark trigger with allowed lateness of 1 hour is the correct choice because it emits results once the watermark passes the end of the window, while still accepting late-arriving events up to 1 hour past the watermark and including them in the correct event-time window. This aligns with the requirement to handle up to 1 hour of late data and maintain correct aggregation results. Note that exactly-once delivery to BigQuery is provided by the Storage Write API sink itself, not by the trigger; the trigger choice affects completeness and correctness of the aggregation, not the exactly-once guarantee.

Exam trap

Google often tests the misconception that any trigger with a time-based interval (like fixed 5-minute triggers) can handle late data, but only watermark-based triggers with allowed lateness correctly align with event-time windows and late-arriving records.

How to eliminate wrong answers

Option A is wrong because the default trigger fires only when the watermark passes, with no allowed lateness, so any data arriving more than a few seconds late would be dropped, violating the requirement to handle up to 1 hour of late data. Option B is wrong because an After count trigger of 1000 fires after every 1000 elements regardless of window boundaries or watermark, which does not respect the 1-hour lateness requirement and can cause incomplete or overlapping aggregations. Option C is wrong because a Fixed time trigger every 5 minutes fires periodically without regard to watermark or lateness, leading to premature emissions and incorrect aggregation results when late data arrives after the trigger fires.

Practice this question →
63
Multi-Selectmedium

A company is building a data pipeline that ingests streaming data from Pub/Sub, transforms it with Dataflow, and loads it into BigQuery. They want to handle malformed messages that cannot be parsed. Which TWO actions should they implement for error handling? (Choose 2)

Select 2 answers
A.Configure the pipeline to drop malformed messages silently
B.Use a side input to filter out malformed messages
C.Use a dead letter sink to write malformed messages to Cloud Storage or Pub/Sub for later analysis
D.Raise an exception in the DoFn to fail the pipeline immediately
E.Log the error and continue processing the next message
AnswersC, E

A dead letter sink captures unparseable messages instead of failing the pipeline, satisfying the requirement to handle malformed records. Dataflow's dead letter pattern writes them to Cloud Storage or Pub/Sub, preserving them for later analysis while the main pipeline continues.

Why this answer

Option C is correct because a dead letter sink (often implemented in Dataflow via a tagged output or a separate Pub/Sub topic/Cloud Storage path) captures unparseable records so they can be inspected and reprocessed without losing data or halting the pipeline. Option E is correct because logging the parsing error and continuing lets the pipeline keep processing valid messages, which is essential for a resilient streaming pipeline where one bad record should not stop the flow. Option A is not appropriate because silently dropping malformed messages loses data and provides no visibility for debugging or remediation.

Option B is not the right pattern because a side input is used to supply supplementary data to a DoFn, not to filter or quarantine malformed records. Option D is wrong because raising an exception in the DoFn would fail the pipeline immediately, causing unnecessary downtime and blocking valid messages from being processed.

Exam trap

Google Cloud often tests the misconception that raising an exception (Option D) is acceptable for error handling in streaming pipelines, but the correct approach is to isolate failures using a dead letter sink (Option C) while logging errors (Option E) to maintain pipeline continuity.

Practice this question →
64
MCQeasy

You need to copy a 3 TB dataset from an on-premises Hadoop Distributed File System (HDFS) cluster to Cloud Storage over a 1 Gbps dedicated interconnect. The data is a one-time historical backfill and must be transferred with integrity verification. You want to minimize operational overhead and avoid writing custom transfer code. Which service should you use?

A.Cloud Data Fusion with an HDFS source plugin
B.gcloud storage cp with parallel composite uploads
C.DistCp job writing directly to a Cloud Storage bucket
D.Storage Transfer Service with an HDFS source
AnswerD

Storage Transfer Service natively supports on-premises HDFS as a source and Cloud Storage as a destination. It performs parallel transfers, integrity checks, and retries without custom code, which fits the one-time historical backfill over a dedicated interconnect. It also handles the large file set efficiently and provides transfer logs and monitoring out of the box.

Why this answer

Storage Transfer Service is purpose-built for moving large datasets from on-premises sources including HDFS into Cloud Storage. It handles parallelism, retries, integrity verification, and monitoring without custom code, which matches the requirement to minimize operational overhead for a one-time backfill over a dedicated interconnect.

Exam trap

The trap here is assuming that any tool that can read HDFS is equally suited for bulk transfer, when transfer-specific managed services provide integrity and retry behavior that pipeline tools do not.

Practice this question →
65
Multi-Selectmedium

You need to ingest streaming data from a custom application into BigQuery with exactly-once semantics and low latency. The data volume is up to 10 MB/s. Which TWO services should you combine?

Select 2 answers
A.Pub/Sub
B.Cloud Functions
C.BigQuery legacy streaming inserts
D.Dataflow with Storage Write API
E.Datastream
AnswersA, D

Pub/Sub provides the messaging layer that satisfies the exactly-once delivery constraint: its subscription-level exactly-once delivery feature deduplicates redelivered messages using acknowledgement IDs, so each record reaches BigQuery once. It also sustains the 10 MB/s throughput with low latency, unlike batch-oriented alternatives.

Why this answer

Option A (Pub/Sub) is correct because it provides a highly scalable, low-latency messaging service that decouples the custom application from downstream processing, ingesting up to 10 MB/s and beyond while buffering the stream reliably. Option D (Dataflow with Storage Write API) is correct because Dataflow can read from Pub/Sub and write to BigQuery using the Storage Write API, which supports exactly-once semantics via its exactly-once delivery mode and provides low-latency, high-throughput ingestion. Together, Pub/Sub plus Dataflow with the Storage Write API form the recommended architecture for exactly-once streaming ingestion into BigQuery.

Option B (Cloud Functions) is not appropriate here because it is event-driven and not designed for sustained high-throughput streaming pipelines with exactly-once guarantees. Option C (BigQuery legacy streaming inserts) does not provide exactly-once semantics and is being deprecated in favor of the Storage Write API. Option E (Datastream) is a change data capture (CDC) and replication service for databases, not a general-purpose streaming ingestion path for custom application data.

Exam trap

PDE often tests the misconception that BigQuery legacy streaming inserts provide exactly-once semantics, when in fact they only offer at-least-once delivery and can duplicate data; candidates must recognize that the Storage Write API is required for exactly-once.

Practice this question →
66
MCQeasy

A team needs to orchestrate a multi-step workflow that involves calling external APIs, running BigQuery queries, and conditionally executing Cloud Functions. Which Google Cloud service is best suited for this?

A.Dataflow
B.Workflows
C.Cloud Composer
D.Cloud Scheduler
AnswerB

Workflows orchestrates multi-step processes using YAML or JSON definitions, sequencing HTTP calls to external APIs, BigQuery jobs, and conditional Cloud Functions invocations. It directly satisfies the stem's requirement for conditional branching across heterogeneous services, unlike single-purpose tools such as Cloud Scheduler or Pub/Sub, which cannot express stateful multi-step logic.

Why this answer

Workflows is a serverless orchestration service that allows you to define multi-step workflows as a sequence of steps, including HTTP calls to external APIs, BigQuery queries, and conditional logic to invoke Cloud Functions. It integrates natively with other Google Cloud services via the Workflows API and supports error handling, retries, and parallel steps, making it ideal for this use case.

Exam trap

Candidates often confuse orchestration services (Workflows) with data processing services (Dataflow) or scheduling services (Cloud Scheduler), leading them to choose Dataflow because they mistake data processing for workflow orchestration.

How to eliminate wrong answers

Option A is wrong because Dataflow is a stream and batch data processing service based on Apache Beam, not an orchestration tool for coordinating API calls, BigQuery queries, and Cloud Functions. Option C is wrong because Cloud Composer is a managed Apache Airflow service that is designed for complex, scheduled workflows with dependencies, but it is heavier, requires more setup, and is overkill for a simple multi-step orchestration that Workflows handles more efficiently. Option D is wrong because Cloud Scheduler is a cron job service for triggering tasks on a schedule, but it cannot orchestrate conditional logic, API calls, or BigQuery queries within a single workflow.

Practice this question →
67
MCQhard

A financial firm ingests trade confirmations from a partner's SFTP server into Cloud Storage, then loads them into BigQuery. The partner drops a variable number of files at unpredictable times, and the firm must detect each new file within seconds and trigger a downstream Cloud Run job. The files must not be processed twice if the trigger fires more than once. Which design should you implement?

A.Schedule a Cloud Scheduler job every minute that lists the bucket and invokes Cloud Run for any object newer than the last run.
B.Use a Pub/Sub topic populated by Cloud Storage notifications and subscribe with a pull subscription that Cloud Run polls on a schedule.
C.Attach a Cloud Functions (1st gen) trigger to the bucket and let it call Cloud Run synchronously for each new object.
D.Configure an Eventarc trigger on the bucket's object.finalize events to invoke Cloud Run, and have the function record the object generation in Firestore before processing.
AnswerD

Eventarc delivers Cloud Storage object.finalize events within seconds of upload, satisfying the latency requirement. Because at-least-once delivery can produce duplicate events, recording the object's generation number as an idempotency key before doing work ensures a repeated trigger for the same object version is skipped, preventing double processing.

Why this answer

Eventarc on object.finalize gives sub-minute, event-driven detection without polling, and the object generation number serves as a natural idempotency key because it uniquely identifies a version of the object. Writing that key to Firestore before doing the work makes duplicate deliveries harmless. Polling designs miss the latency target, and designs without a deduplication record allow the same file to be processed again on redelivery.

Exam trap

The trap here is assuming that a Cloud Storage event fires exactly once per object, when delivery is at-least-once and the same object can generate repeated triggers that must be reconciled by a durable idempotency key.

Practice this question →
68
MCQeasy

You need to ingest Google Ads performance data into BigQuery on a daily basis for reporting. Which service should you use?

A.BigQuery Data Transfer Service for Google Ads
B.Cloud Scheduler to call Google Ads API and load to BigQuery
C.Pub/Sub with a Google Ads subscriber
D.Storage Transfer Service for Google Ads
AnswerA

BigQuery Data Transfer Service provides a managed, scheduled connector that pulls Google Ads performance data directly into BigQuery daily without custom extraction code. This satisfies the stem's daily ingestion requirement, handling authentication, incremental transfers and backfill automatically.

Why this answer

The BigQuery Data Transfer Service for Google Ads is the correct choice because it provides a fully managed, scheduled connector that automatically ingests Google Ads performance data into BigQuery on a daily basis without requiring any custom code. It handles authentication, schema mapping, and incremental loads, making it the simplest and most reliable solution for this specific use case.

Exam trap

Google often tests the distinction between fully managed services (like BigQuery Data Transfer Service) and generic infrastructure components (like Cloud Scheduler or Pub/Sub) that require custom development, leading candidates to overcomplicate the solution by choosing a more flexible but less appropriate option.

How to eliminate wrong answers

Option B is wrong because Cloud Scheduler is a cron job service that can trigger HTTP requests, but it does not natively integrate with the Google Ads API or handle the complex authentication, pagination, and schema mapping required to load data into BigQuery; you would still need to build and maintain a custom application. Option C is wrong because Pub/Sub is a messaging service for asynchronous event streaming, not a batch ingestion tool; while you could theoretically publish Google Ads data to Pub/Sub, there is no native Google Ads subscriber, and you would need to build a custom subscriber to write to BigQuery, which is far more complex than using the dedicated transfer service. Option D is wrong because Storage Transfer Service is designed for moving data from on-premises or cloud storage (like S3 or HTTP endpoints) into Google Cloud Storage, not for directly ingesting data from Google Ads into BigQuery.

Practice this question →
69
MCQeasy

You need to react to changes in a GCS bucket (e.g., new object creation) and trigger a Cloud Run service to process the new file. Which Google Cloud service should you use to route the event?

A.Pub/Sub directly with a Cloud Run subscription
B.Cloud Tasks
C.Eventarc
D.Cloud Scheduler
AnswerC

Eventarc routes Cloud Storage object-creation events to Cloud Run, satisfying the requirement to trigger the service on new files. It delivers CloudEvents from Google Cloud sources directly to serverless endpoints, unlike Pub/Sub alone, which requires custom wiring, or Cloud Scheduler, which polls rather than reacting to bucket changes.

Why this answer

Eventarc is the correct choice because it is purpose-built to route events from Google Cloud sources (like Cloud Storage) to Cloud Run. It directly supports Cloud Storage audit logs and Pub/Sub event triggers, allowing you to react to object creation events without custom middleware. Eventarc handles the event routing, filtering, and delivery to your Cloud Run service automatically.

Exam trap

Google often tests the misconception that Pub/Sub is the direct answer for any event routing, but the trap here is that Eventarc is the managed service that simplifies the integration between GCS and Cloud Run, making it the correct choice over raw Pub/Sub.

How to eliminate wrong answers

Option A is wrong because Pub/Sub directly with a Cloud Run subscription requires you to manually configure a Pub/Sub topic and subscription, and Cloud Run can only pull messages via a push subscription; Eventarc abstracts this complexity and provides native integration with Cloud Storage events. Option B is wrong because Cloud Tasks is a task queue for asynchronous execution of HTTP requests, not designed for event-driven routing from GCS; it would require you to manually publish tasks in response to events, adding unnecessary overhead. Option D is wrong because Cloud Scheduler is a cron job scheduler for periodic tasks, not an event router; it cannot react to real-time object creation events in a GCS bucket.

Practice this question →
70
MCQmedium

A company wants to use dbt to transform data in BigQuery. Their source data is loaded daily into staging tables. They need to run dbt transformations on a schedule and only process tables that have changed. Which dbt feature should they use?

A.dbt snapshots
B.dbt incremental models
C.dbt seeds
D.dbt sources
AnswerB

dbt incremental models process only new or changed rows since the last run, using a configured unique key and filter. This satisfies the requirement to run on a schedule while avoiding full reprocessing of unchanged staging tables.

Why this answer

dbt incremental models are designed to process only new or changed data since the last run, rather than rebuilding the entire table. This is ideal for daily transformations on staging tables where only some data changes. Incremental models use a unique key and a strategy (e.g., merge, delete+insert) to update the target table efficiently.

Exam trap

The trap is confusing incremental models with snapshots. Candidates might think snapshots process only changed data, but snapshots are for tracking history, not for efficient incremental updates.

How to eliminate wrong answers

Option A is wrong because dbt snapshots are used to track historical changes (SCD Type 2) by capturing changes over time, not for incremental processing of source data. Option C is wrong because dbt seeds are CSV files loaded into the data warehouse, typically for static reference data, not for transforming large datasets. Option D is wrong because dbt sources are configurations that define and document source tables, but they do not provide incremental processing logic.

Practice this question →
71
MCQeasy

A company wants to stream real-time clickstream data from a website into BigQuery for near-real-time analytics. They expect peaks of 10,000 events per second. Which combination of services is most suitable for ingestion?

A.Cloud Storage → Cloud Functions → BigQuery
B.Direct Web → Dataflow → BigQuery
C.Pub/Sub → Dataflow → BigQuery (Storage Write API)
D.Pub/Sub → Dataflow → BigQuery (legacy streaming inserts)
AnswerC

Pub/Sub decouples ingestion from processing, absorbing 10,000 events per second spikes without loss, while Dataflow provides exactly-once streaming transformation. The Storage Write API commits rows directly into BigQuery, satisfying the near-real-time analytics requirement without staging files in Cloud Storage.

Why this answer

For high-throughput, real-time streaming into BigQuery, the combination of Pub/Sub for ingestion, Dataflow for stream processing, and BigQuery's Storage Write API for writing is the most suitable. Pub/Sub handles the 10,000 events per second peaks reliably, Dataflow provides scalable stream processing, and the Storage Write API offers exactly-once semantics and better performance than legacy streaming inserts.

Exam trap

The trap is assuming that any streaming combination works equally well, but the exam expects knowledge that the Storage Write API is preferred over legacy streaming inserts for performance and exactly-once semantics, and that Pub/Sub is essential for handling peak loads.

How to eliminate wrong answers

Option A is wrong because Cloud Storage → Cloud Functions → BigQuery is a batch-oriented or event-driven approach that does not handle high-throughput real-time streaming well; Cloud Functions have concurrency limits and are not designed for sustained 10k EPS. Option B is wrong because Direct Web → Dataflow → BigQuery bypasses Pub/Sub, losing the buffering and decoupling benefits needed for peak loads; Dataflow alone may struggle with ingestion spikes without a message queue. Option D is wrong because Pub/Sub → Dataflow → BigQuery (legacy streaming inserts) uses the older streaming API, which is more expensive, has lower throughput limits, and does not provide exactly-once semantics, making it less suitable than the Storage Write API.

Practice this question →
72
MCQhard

A company needs to continuously synchronize customer data changes from an on-premises Oracle database to BigQuery for near-real-time analytics. The Oracle database has Change Data Capture (CDC) enabled. Which Google Cloud service should be used to stream these changes with minimal latency and schema evolution support?

A.Deploy a Dataflow pipeline with a JDBC source and Pub/Sub
B.Use Cloud SQL with a read replica and enable binary logging
C.Use Transfer Appliance to copy Oracle data periodically
D.Use Datastream to stream CDC changes from Oracle to BigQuery
AnswerD

Datastream provides native Oracle CDC replication, reading redo logs to stream changes into BigQuery with minimal latency. Its built-in schema evolution handling automatically propagates source DDL changes, satisfying the stem's requirement for continuous near-real-time synchronisation without custom extraction code.

Why this answer

Datastream is Google Cloud's fully managed, serverless CDC and replication service that natively supports Oracle sources (via LogMiner or XStream) and BigQuery destinations. It streams changes with low latency, handles schema evolution automatically, and requires no pipeline code to maintain. This matches the requirement for minimal latency and schema evolution support.

Exam trap

PDE often tests whether candidates recognize that Datastream is the purpose-built managed CDC service, while Dataflow+JDBC is a custom polling approach that candidates mistakenly equate with CDC.

How to eliminate wrong answers

Option A is wrong because a custom Dataflow JDBC pipeline polls the source rather than reading CDC logs, introducing latency and requiring custom code to handle schema changes — it does not leverage Oracle's CDC capability. Option B is wrong because Cloud SQL is a managed MySQL/PostgreSQL/SQL Server service; it cannot act as a read replica of an on-premises Oracle database, and binary logging is a MySQL concept, not Oracle. Option C is wrong because Transfer Appliance is a physical appliance for bulk offline data transfer, which is the opposite of near-real-time streaming and provides no CDC or schema evolution.

Practice this question →
73
MCQeasy

A data engineer needs to ingest on-premises Oracle CDC data into BigQuery in near real-time with minimal operational overhead. Which service should they use?

A.Pub/Sub + Dataflow
B.Storage Transfer Service
C.Transfer Appliance
D.Datastream
AnswerD

Datastream provides serverless change data capture from Oracle into BigQuery with minimal operational overhead, replicating changes in near real-time. It satisfies both constraints directly, unlike batch extraction tools or custom pipelines that require managing infrastructure.

Why this answer

Datastream is purpose-built for streaming change data capture (CDC) from Oracle and other sources into BigQuery with near-real-time latency and minimal operational overhead. It handles schema propagation, checkpointing, and automatic retries, eliminating the need to manage custom ingestion pipelines.

Exam trap

Google often tests the distinction between batch migration tools (Storage Transfer Service, Transfer Appliance) and streaming CDC services (Datastream), leading candidates to choose a batch option when the question explicitly requires near-real-time ingestion.

How to eliminate wrong answers

Option A is wrong because Pub/Sub + Dataflow requires building and maintaining a custom pipeline to handle Oracle CDC, including log mining and transformation logic, which increases operational overhead compared to a managed service. Option B is wrong because Storage Transfer Service is designed for bulk batch transfers of files from cloud or on-premises storage to Google Cloud, not for streaming CDC from a live database. Option C is wrong because Transfer Appliance is a physical device for offline, high-volume data migration, which cannot provide near-real-time streaming and introduces significant latency.

Practice this question →
74
MCQmedium

An organization needs to trigger a Cloud Run service whenever a new file is uploaded to a specific Cloud Storage bucket. Which service should they use to set up this event-driven architecture?

A.Eventarc with a trigger for Cloud Storage events
B.Pub/Sub notifications on the bucket with a push subscription to Cloud Run
C.Cloud Scheduler calling Cloud Run on a schedule
D.Cloud Functions with a GCS trigger
AnswerA

Eventarc natively routes Cloud Storage object-finalise events to Cloud Run, satisfying the requirement to trigger on each new upload. It delivers events through the CloudEvents standard with built-in filtering by bucket and event type, so no polling or manual Pub/Sub plumbing is needed.

Why this answer

Eventarc is the recommended Google Cloud service for routing events from Cloud Storage (and 90+ other sources) to Cloud Run, Cloud Functions, or GKE. It provides a native Cloud Storage trigger type, handles authentication, retries, and dead-lettering, and integrates directly with Cloud Run's event delivery model. This is the canonical event-driven pattern for GCS-to-Cloud-Run.

Exam trap

PDE often tests the distinction between Eventarc (managed event routing) and raw Pub/Sub notifications — candidates pick Pub/Sub because it 'works,' missing that Eventarc is the recommended, lower-overhead abstraction for Cloud Run triggers.

How to eliminate wrong answers

Option B is wrong because while Pub/Sub notifications on a bucket with a push subscription to Cloud Run can technically work, it requires manual Pub/Sub topic creation, IAM wiring, and does not provide the first-class Cloud Storage event schema or filtering that Eventarc offers — it is more operational overhead and not the recommended pattern. Option C is wrong because Cloud Scheduler is time-based, not event-based; it cannot react to a file upload. Option D is wrong because Cloud Functions with a GCS trigger targets Cloud Functions (1st/2nd gen), not Cloud Run — the question specifically asks about triggering a Cloud Run service.

Practice this question →
75
Multi-Selectmedium

A company is building a real-time anomaly detection pipeline using Dataflow. Events are ingested from Pub/Sub, and the pipeline must compute a sliding window average every minute over a 1-hour window. Which TWO configurations are required for this pipeline? (Choose 2)

Select 2 answers
A.Set the pipeline to use event time for watermarking.
B.Use a Sliding window of 1 hour with a 1-minute slide.
C.Use a Fixed window of 1 minute.
D.Use stateful processing with a custom timer.
E.Set the pipeline to use processing time for watermarking.
AnswersA, B

Event-time watermarking derives progress from the timestamp embedded in each Pub/Sub message rather than arrival time, so the one-hour sliding window aggregates correctly despite network delays and out-of-order events. Without it, window results would be non-deterministic and inaccurate.

Why this answer

Option A is correct because a sliding window over event data must be based on event time so that late-arriving events are assigned to the correct window and watermarks track the progress of event time rather than the wall-clock time at which elements are processed. Option B is correct because the requirement is a 1-hour window that is recomputed every minute, which is exactly a SlidingWindow with a duration of 1 hour and a period (slide) of 1 minute. Option C is incorrect because a FixedWindow of 1 minute produces independent non-overlapping 1-minute aggregates, not a 1-hour average refreshed each minute.

Option D is incorrect because stateful processing with a custom timer is a lower-level mechanism and is not required when Apache Beam's built-in sliding windows already express the desired semantics. Option E is incorrect because processing-time watermarking ignores the event timestamps and would misassign late or out-of-order Pub/Sub messages, breaking the anomaly detection logic.

Exam trap

The trap is assuming a FixedWindow can produce a rolling average — candidates who conflate 'window size' with 'update frequency' pick FixedWindow of 1 minute and miss that sliding windows require both a duration and a period.

Practice this question →
Page 1 of 2 · 107 questions totalNext →

Ready to test yourself?

Try a timed practice session using only Ingesting and Processing the Data questions.

Start 20-question session