Scenario practice questions

Your company ingests millions of events per second into a Pub/Sub topic. The downstream consumer must process events with minimal latency and high throughput. However, the consumer occasionally falls behind during traffic spikes, and you need to ensure no data loss while minimizing costs. Which subscription type and configuration should you choose?

Trap 1: Push subscription with a load balancer

Push subscriptions have a fixed rate limit and cannot scale to handle millions of events per second effectively.

Trap 2: Push subscription with endpoint on Cloud Run

Push subscriptions are not designed for high throughput (millions of events/sec) and can cause backlogs.

Trap 3: Pull subscription with exactly-once delivery disabled

Disabling exactly-once delivery may lead to duplicates; but even with exactly-once, pull is still the correct choice for throughput.

A
Push subscription with a load balancer
Why wrong: Push subscriptions have a fixed rate limit and cannot scale to handle millions of events per second effectively.
B
Pull subscription with flow control settings
Pull subscriptions allow the subscriber to pull messages at its own pace, and flow control helps prevent overwhelming the consumer. This combination handles high throughput efficiently.
C
Push subscription with endpoint on Cloud Run
Why wrong: Push subscriptions are not designed for high throughput (millions of events/sec) and can cause backlogs.
D
Pull subscription with exactly-once delivery disabled
Why wrong: Disabling exactly-once delivery may lead to duplicates; but even with exactly-once, pull is still the correct choice for throughput.

Question 2mediummultiple choice

What is the most likely cause of data duplication after this command?

Network Topology

Trap 1: The Pub/Sub source is not exactly-once.

Pub/Sub exactly-once would prevent duplicates from source, but duplication is likely from sink side.

Trap 2: The pipeline uses at-least-once semantics.

At-least-once can cause duplicates but the pipeline was working before, so the change likely caused it.

Trap 3: The snapshot was taken before scaling.

Snapshots are used for recovery, not for causing duplicates.

A
The Pub/Sub source is not exactly-once.
Why wrong: Pub/Sub exactly-once would prevent duplicates from source, but duplication is likely from sink side.
B
The pipeline uses at-least-once semantics.
Why wrong: At-least-once can cause duplicates but the pipeline was working before, so the change likely caused it.
C
The snapshot was taken before scaling.
Why wrong: Snapshots are used for recovery, not for causing duplicates.
D
The BigQuery sink is not idempotent.
If the sink is not idempotent, duplicate data can be written when workers are re-added or when job state is replayed.

Question 3mediummultiple choice

A company wants to design a data pipeline for real-time fraud detection. The system must process streaming financial transactions, enrich them with user profiles from a lookup table, and flag suspicious activities within seconds. Which architecture pattern would be MOST suitable?

Trap 1: Pub/Sub combined with Cloud Functions for stateless processing

Cloud Functions are stateless and not designed for stream processing with stateful operations like enrichment and windowing.

Trap 2: Batch processing with hourly micro-batches using Dataflow

Hourly batches would be too slow for real-time fraud detection.

Trap 3: Lambda architecture with a batch layer for historical analysis and…

Lambda adds unnecessary complexity. The need is purely real-time fraud detection, not batch historical analysis.

A
Pub/Sub combined with Cloud Functions for stateless processing
Why wrong: Cloud Functions are stateless and not designed for stream processing with stateful operations like enrichment and windowing.
B
Kappa architecture using a single stream processing framework like Apache Beam
Kappa processes everything as a stream, suitable for real-time fraud detection with enrichment from a side input.
C
Batch processing with hourly micro-batches using Dataflow
Why wrong: Hourly batches would be too slow for real-time fraud detection.
D
Lambda architecture with a batch layer for historical analysis and a speed layer for real-time processing
Why wrong: Lambda adds unnecessary complexity. The need is purely real-time fraud detection, not batch historical analysis.

Question 4easymultiple choice

A data engineer needs to create a BigQuery table that is optimized for queries that filter on a 'customer_id' column and sort by 'transaction_date'. The table will be used for interactive analysis. Which combination of table features should be used?

Trap 1: Partition by customer_id and cluster by transaction_date

Partitioning is typically used on date/timestamp columns, not on high-cardinality string columns like customer_id; clustering is better for that.

Trap 2: Cluster by both customer_id and transaction_date

Clustering alone does not provide the partition pruning benefit of partitioning on date; for large tables, partitioning is recommended.

Trap 3: Use a materialized view with customer_id and transaction_date

Materialized views store pre-computed results but are not for optimizing raw table queries; they are for aggregate queries.

A
Partition by customer_id and cluster by transaction_date
Why wrong: Partitioning is typically used on date/timestamp columns, not on high-cardinality string columns like customer_id; clustering is better for that.
B
Cluster by both customer_id and transaction_date
Why wrong: Clustering alone does not provide the partition pruning benefit of partitioning on date; for large tables, partitioning is recommended.
C
Use a materialized view with customer_id and transaction_date
Why wrong: Materialized views store pre-computed results but are not for optimizing raw table queries; they are for aggregate queries.
D
Partition by transaction_date and cluster by customer_id
Partitioning by date allows BigQuery to prune partitions for queries with date ranges. Clustering by customer_id sorts data within partitions, speeding up customer-level queries.

Question 5mediummultiple choice

A company wants to migrate their on-premises Teradata data warehouse to BigQuery. They need an automated, one-time transfer of historical data (10 TB) and ongoing incremental daily syncs. Which Google Cloud service should they use?

Trap 1: Dataflow custom pipeline

While possible, it requires custom development and maintenance, whereas Data Transfer Service is managed.

Trap 2: Storage Transfer Service

Storage Transfer Service transfers files between cloud storage systems, not from Teradata.

Trap 3: Datastream

Datastream supports CDC from MySQL, PostgreSQL, and Oracle, not Teradata.

A
BigQuery Data Transfer Service for Teradata
This service is designed to schedule and automate transfers from Teradata to BigQuery, both initial and incremental.
B
Dataflow custom pipeline
Why wrong: While possible, it requires custom development and maintenance, whereas Data Transfer Service is managed.
C
Storage Transfer Service
Why wrong: Storage Transfer Service transfers files between cloud storage systems, not from Teradata.
D
Datastream
Why wrong: Datastream supports CDC from MySQL, PostgreSQL, and Oracle, not Teradata.

Question 6mediummultiple choice

An organization uses Cloud Storage to store backup files. They want to automatically delete files older than 90 days, and after deletion, move remaining files to Nearline storage if not accessed for 30 days. Which Cloud Storage feature should they configure?

Trap 1: Object Versioning

Versioning keeps multiple versions of objects, but does not automate deletion or storage class transitions.

Trap 2: Retention Policies

Retention policies prevent deletion, not schedule deletion or storage class changes.

Trap 3: Bucket Lock

Bucket Lock is for WORM compliance, not lifecycle management.

A
Object Versioning
Why wrong: Versioning keeps multiple versions of objects, but does not automate deletion or storage class transitions.
B
Retention Policies
Why wrong: Retention policies prevent deletion, not schedule deletion or storage class changes.
C
Bucket Lock
Why wrong: Bucket Lock is for WORM compliance, not lifecycle management.
D
Object Lifecycle Management
Lifecycle rules can delete objects after a specified age and change storage class based on last access time (using Condition with LastAccessTime).

Question 7easymultiple choice

A company is designing a streaming data pipeline to process real-time clickstream events. They need to aggregate events by session window with a 5-minute gap and enable exactly-once processing semantics. Which Google Cloud service should they use?

Trap 1: Cloud Pub/Sub with Cloud Functions

Cloud Functions are stateless and do not natively support session windowing.

Trap 2: Cloud Dataproc with Spark Streaming

Spark Streaming provides at-least-once semantics by default; exactly-once requires additional configuration.

Trap 3: Cloud Bigtable with Dataflow templates

Bigtable is a NoSQL database, not designed for stream windowing; templates are limited in flexibility.

A
Cloud Pub/Sub with Cloud Functions
Why wrong: Cloud Functions are stateless and do not natively support session windowing.
B
Cloud Dataflow with Apache Beam
Dataflow with Beam natively supports session windows and exactly-once processing via its processing guarantees.
C
Cloud Dataproc with Spark Streaming
Why wrong: Spark Streaming provides at-least-once semantics by default; exactly-once requires additional configuration.
D
Cloud Bigtable with Dataflow templates
Why wrong: Bigtable is a NoSQL database, not designed for stream windowing; templates are limited in flexibility.

Question 8easymultiple choice

You need to schedule a Dataproc Spark job to run at 2 AM every day, and upon completion, trigger a BigQuery load job. Which Cloud Composer operator should you use to run the Spark job?

Trap 1: DataflowPythonOperator

This operator runs Dataflow pipelines, not Dataproc.

Trap 2: BigQueryOperator

This operator runs BigQuery jobs, not Dataproc Spark.

Trap 3: DataprocClusterCreateOperator

This operator creates a cluster, not submit a job.

A
DataflowPythonOperator
Why wrong: This operator runs Dataflow pipelines, not Dataproc.
B
BigQueryOperator
Why wrong: This operator runs BigQuery jobs, not Dataproc Spark.
C
DataprocClusterCreateOperator
Why wrong: This operator creates a cluster, not submit a job.
D
DataprocSubmitJobOperator
This operator submits a job (e.g., Spark, PySpark) to an existing Dataproc cluster.

Question 9mediummultiple choice

You have deployed a classification model on Vertex AI Endpoints. The model's training data had a balanced class distribution, but over time, the production data has shifted such that one class appears 90% of the time. The model's overall accuracy remains high, but the recall for the minority class has dropped significantly. What is the best approach to detect and address this issue?

Trap 1: Retrain the model daily on the entire historical dataset

Retraining on old data can dilute recent shift; better to use sliding window or weighted approach.

Trap 2: Increase the number of replicas on the endpoint to reduce latency

Scaling replicas improves throughput/latency but does not affect model accuracy or drift.

Trap 3: Adjust the decision threshold to improve minority class recall

Adjusting threshold helps recall but does not address the underlying data drift; performance may degrade further.

A
Retrain the model daily on the entire historical dataset
Why wrong: Retraining on old data can dilute recent shift; better to use sliding window or weighted approach.
B
Set up Vertex AI Model Monitoring to detect skew and drift, and retrain using a sliding window of recent data
Model Monitoring detects skew/drift; retraining on recent data adapts to new distribution.
C
Increase the number of replicas on the endpoint to reduce latency
Why wrong: Scaling replicas improves throughput/latency but does not affect model accuracy or drift.
D
Adjust the decision threshold to improve minority class recall
Why wrong: Adjusting threshold helps recall but does not address the underlying data drift; performance may degrade further.