Hard Difficulty Questions

Use Cloud Dataproc with Spark Streaming to aggregate data, and use a Spark ML model embedded in the pipeline

Why it fails: Spark ML embedded in Dataproc bypasses the AI Platform-served model the requirements mandate, and Spark Streaming micro-batches do not natively align to 5-minute windows with endpoint calls. Dataproc fits migrating existing Spark workloads, not this design.

Use BigQuery streaming inserts and run scheduled queries that call the ML model

Why it fails: Scheduled BigQuery queries run at intervals, not continuously over 5-minute windows, and cannot invoke an AI Platform endpoint per window in real time. BigQuery suits batch analytics; streaming windowed scoring with a served model needs Dataflow.

Use Cloud Functions triggered by Pub/Sub to process each sensor reading individually

Why it fails: Cloud Functions processes each reading individually, so no 5-minute windowed aggregation occurs and the AI Platform model never receives the grouped input it requires. It is tempting because Cloud Functions suits lightweight per-message event handling, such as filtering or enriching single Pub/Sub events, but windowed stream aggregation needs Dataflow.

Use Data Loss Prevention API to redact sensitive data.

Why it fails: Not encryption.

Enable Cloud Audit Logs and VPC Service Controls.

Why it fails: Missing encryption control.

Enable default encryption at rest and in transit.

Why it fails: No customer-managed keys.

Use Cloud TPUs to accelerate predictions

Why it fails: Cloud TPUs target training and high-throughput tensor workloads, and Vertex AI batch prediction on TPUs does not address the model's significant memory requirement or guarantee the four-hour window at lower cost. It is tempting because TPUs accelerate large-scale matrix computation, and would be correct for cost-efficient distributed training of compatible models.

Use a smaller machine type (e.g., n1-standard-4) to reduce cost

Why it fails: An n1-standard-4 provides only 15 GB of memory, insufficient for a model requiring significant memory, so the job would fail or spill rather than complete within four hours. It is tempting because shrinking machine size is the obvious cost lever, and would be correct for lightweight models with modest memory footprints.

Use standard VMs and reduce parallelization

Why it fails: Standard VMs with reduced parallelisation lengthen runtime, risking breach of the four-hour window while offering no memory advantage over Vertex AI's managed batch prediction options. It is tempting because fewer parallel workers lowers compute spend, and would be correct when jobs are not time-constrained and cost dominates.

Use materialized views

Why it fails: Materialised views precompute results but do not reduce bytes scanned when the query filters columns outside the materialised set, so full scans persist. They are tempting because they accelerate repeated aggregations, and would be correct when identical expensive aggregate queries recur rather than varying column selections across a partitioned table.

Convert to native tables

Why it fails: Native tables change storage format, not scan volume; clustering is the missing mechanism that prunes blocks within each date partition. Converting tables is tempting when legacy external tables lack partitioning or clustering support, but here partitioning already exists, so the full-scan cost persists.

Use query caching

Why it fails: Query caching only returns results for byte-identical repeated queries within the cache window; novel analytical queries still scan full tables. It is tempting because it is free and automatic, and would be correct when the same query is re-run frequently, not when varied queries drive the cost.

Configure Pub/Sub to retry indefinitely until the message is processed

Why it fails: Infinite retries cause the malformed message to be redelivered forever, blocking the subscription's progress and never resolving the parse failure. It is tempting because retries handle transient errors such as timeouts, which is the right approach when failures are temporary rather than caused by permanently invalid message content.

Use a try-catch block in the pipeline and ignore malformed messages

Why it fails: Catching and discarding malformed messages silently drops them, violating the no-data-loss requirement; they must be routed to a dead-letter destination for later inspection. It is tempting because try-catch keeps the pipeline running, which is the correct pattern when malformed records are genuinely disposable and no audit trail is required.

Set the pipeline to fail and alert the team via Cloud Monitoring

Why it fails: Failing the pipeline halts all processing and leaves the malformed message unhandled, so throughput stops and no dead-letter record is produced. It is tempting because failing fast with Cloud Monitoring alerts surfaces problems immediately, which suits pipelines where any malformed input signals an upstream defect that must be fixed before continuing.

It will deploy the model if the accuracy meets the threshold

Why it fails: The component only evaluates and emits metrics; model deployment is a separate pipeline step. It is tempting because thresholds commonly gate promotion, but this YAML's outputs are metrics, not deployment actions, so meeting the threshold does not itself deploy the model.

It will ignore the threshold_accuracy input if not provided

Why it fails: The component compares the produced metric against threshold_accuracy and emits a boolean outcome; an unset input would use its default, not be ignored. It is tempting because optional inputs can be omitted, but the YAML defines a default value, so the threshold still governs evaluation rather than being skipped.

It will fail if the model accuracy is below 0.85

Why it fails: The component outputs a boolean metric indicating whether accuracy meets the threshold; it does not raise a pipeline failure. It is tempting because threshold breaches often trigger alerts, but this component's contract is to emit the comparison result, leaving failure handling to downstream conditional logic.

Enable autoscaling slot management in the primary project's reservation, allowing slots to scale up based on demand.

Why it fails: Incorrect. Autoscaling can be used with committed use reservations to scale slots up to a maximum, but it still requires base committed slots. It does not provide purely temporary capacity without commitment, unlike flex slots.

Upgrade the primary project's edition to Enterprise Plus to allow bursting.

Why it fails: Incorrect. Upgrading to Enterprise Plus changes pricing and features but does not directly provide burst capacity without additional spend.

Purchase additional committed use slots in the primary project and apply them to the reservation.

Why it fails: Incorrect. Purchasing additional committed use slots increases committed spend, which is what the company wants to avoid.

Enable preemptible VMs for cost savings.

Why it fails: Preemptible VMs are reclaimed within 24 hours, so executors vanish mid-job and tasks must rerun, undermining the reliability the scenario demands. They suit fault-tolerant, checkpointed or short-lived batch work where cost outweighs guaranteed completion, not jobs needing dependable output.

Enable idle timeout to automatically delete clusters.

Why it fails: Idle timeout merely deletes a cluster after inactivity; it neither prevents task failure nor validates data, so reliability and quality remain unaddressed. It is intended to curb spend on forgotten ephemeral clusters, and would be the right pick for a cost-control question rather than this one.

Use custom machine types for better performance.

Why it fails: Custom machine types tune vCPU-to-memory ratios for a workload's resource profile, which affects performance and cost but not task retry behaviour or data validation. They belong in a sizing or tuning scenario, not one asking how to guarantee job reliability and data quality.

Cloud Composer

Why it fails: Cloud Composer is a workflow orchestration service. It can schedule and manage batch jobs, but it is not designed for real-time stream processing or handling late-arriving data. It does not provide a unified programming model for batch and streaming. Therefore, it does not meet the core processing requirements of this scenario.

Cloud Dataproc

Why it fails: Cloud Dataproc is a managed Hadoop and Spark service. While it can process batch and micro-batch data, it does not provide a unified codebase for both streaming and batch with the same API as Apache Beam. It also requires cluster management, which adds operational overhead. Dataproc is not the best fit for a unified real-time and batch processing architecture.

BigQuery

Why it fails: BigQuery is a serverless data warehouse for analytics. It can store and analyze the unified dataset, but it is not a data processing service for ingesting and transforming streaming and batch data. While BigQuery can be a sink for the processed data, it does not provide the processing capabilities required to handle late-arriving data and unify codebases.

Create a materialized view that pre-aggregates the data by day.

Why it fails: A materialised view pre-aggregates results but does not prune partitions; the query still reads the full fact table unless a date filter is applied. It is tempting for repeated aggregate queries, where it would be correct, but here the requirement is restricting scanned partitions to the last seven days.

Cluster the fact table on the join key used in the query.

Why it fails: Clustering on the join key speeds up join processing but does not reduce bytes scanned, since clustering prunes only on the clustered columns and the query still reads every date partition. It is tempting for join-heavy queries, where it would be correct, but partition pruning on the date column is what limits scanned bytes.

Change the table to use time-unit column partitioning with a 1-day partition interval.

Why it fails: The fact table is already partitioned by date, so redefining partitioning as a time-unit column with a 1-day interval changes nothing about pruning. It is tempting because day-level partitioning is the right pattern for date-filtered scans, but the existing setup already satisfies it; the query needs a date predicate.

Disable preemptible workers for the job.

Why it fails: Disabling preemptible workers removes the cost saving the cluster was built around, abandoning the scenario's premise instead of tolerating preemption. It is tempting because it eliminates failures outright, and it would be correct for jobs with strict completion deadlines where spot capacity cannot be risked.

Use persistent disks for preemptible workers.

Why it fails: Persistent disks preserve data across preemption, but the Spark executors themselves still terminate, so in-flight tasks are lost and must be recomputed. It is tempting because persistence sounds like resilience, and it would be correct for stateful workloads needing data to survive instance replacement.

Enable automatic restart of the Spark driver on failure.

Why it fails: Restarting the driver does not help when executor nodes are preempted; the driver is typically on a standard node and survives. It is tempting because driver restart is a known resilience pattern, and it would be correct when the driver itself fails on a non-preemptible master.

Insufficient compute capacity (nodes) allocated to the instance.

Why it fails: While compute can be an issue, 50% latency increase typically indicates structural contention rather than capacity.

Incorrect indexing of secondary indexes.

Why it fails: Indexing affects read performance, not write commit latency.

Network bandwidth constraints between regions.

Why it fails: Network latency may contribute but is less likely to cause such a sharp increase during peak hours without key contention.

Set minNodes to 0 and enable node auto-scaling

Why it fails: This would increase cold starts, as nodes can scale down to zero.

Increase maxNodes to allow more replicas during peak, and rely on Kubernetes Horizontal Pod Autoscaler

Why it fails: Kubernetes HPA is not directly used in Vertex AI Endpoints; increasing maxNodes alone does not prevent cold starts.

Use Cloud Functions with Cloud Run for the model inference to leverage serverless cold-start mitigation

Why it fails: Cloud Functions are not designed for large model serving with GPU needs.

Cloud SQL (MySQL) with cross-region read replicas

Why it fails: Cloud SQL does not provide strong consistency across regions; replicas are eventually consistent.

BigQuery multi-region

Why it fails: BigQuery is for analytics, not transactional OLTP.

Cloud Bigtable multi-cluster routing

Why it fails: Bigtable offers eventual consistency across clusters, not strong consistency.

Configure a dead letter queue for failed steps

Why it fails: A dead letter queue captures failed executions for later inspection; it does not extend the step timeout or retry transient errors. DLQs are correct when you must preserve failed payloads for replay or audit after retries are exhausted, not as the mechanism that keeps a 10-minute call alive.

Use the default retry policy on the step

Why it fails: The default retry policy only covers a fixed set of error codes with limited attempts, so it cannot be tuned to the endpoint's transient failures. Default retries suit simple, predictable steps; here the step needs a custom retry predicate with backoff matched to the HTTP endpoint's behaviour.

Set the workflow execution timeout to 600 seconds

Why it fails: The workflow execution timeout bounds the whole pipeline, not the individual HTTP step, so a 600-second value still leaves the step's own default timeout truncating the 10-minute call. Workflow-level timeouts suit capping total pipeline duration, not extending a single long-running step.

Metric: topic/send_message_operation_count; filter: topic_id

Why it fails: topic/send_message_operation_count measures publish operations on the topic, not delivery latency at the subscription, so it cannot detect messages older than one hour. It is tempting because it monitors Pub/Sub activity, and it is the correct metric for alerting on publish failures or send operation rates.

Metric: subscription/ack_message_count; filter: subscription_id

Why it fails: ack_message_count counts acknowledgements over a period, revealing throughput rather than how long the oldest unacknowledged message has waited. It is tempting because low acknowledgement rates suggest stuck messages, and it is the correct metric for alerting on acknowledgement rate or consumer throughput degradation.

Metric: subscription/num_undelivered_messages; filter: subscription_id

Why it fails: num_undelivered_messages counts backlog size, not message age, so it cannot detect messages older than one hour. It is tempting because a large backlog often correlates with staleness, and it is the correct metric for alerting on queue depth or volume thresholds rather than age.

Dataplex Universal Catalog

Why it fails: Universal Catalog provides metadata discovery, search and governance across assets, not rule execution against table contents. It tempts because it surfaces schema and tag information about the email column, but validating not-null and regex constraints requires Dataplex Data Quality, which scans the data itself.

Dataplex Lake

Why it fails: A Dataplex lake is a logical container organising zones and assets for governance; it performs no data validation. It tempts because the BigQuery table would be registered as an asset within a lake, but enforcing not-null and regex rules needs Data Quality tasks attached to that asset.

Dataplex Data Lineage

Why it fails: Data Lineage tracks column-level provenance and transformation flow across Dataplex assets; it does not evaluate not-null or regex conditions. It tempts because lineage also profiles metadata about columns, but enforcement of quality rules belongs to Dataplex Data Quality tasks, which run checks and publish results.

FIRST_VALUE(event_time) OVER (PARTITION BY user_id ORDER BY event_time)

Why it fails: FIRST_VALUE returns the partition's earliest event_time for every row, so each difference is measured against the user's first event, not the immediately preceding one. It is tempting for baselines, but LAG gives the prior row's timestamp.

LEAD(event_time) OVER (PARTITION BY user_id ORDER BY event_time)

Why it fails: LEAD returns the following row's event_time, giving the gap to the next event rather than the previous one. It is tempting because it is the mirror of the required function, but the stem asks for the difference from the preceding event.

ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY event_time)

Why it fails: ROW_NUMBER assigns a sequential position per partition; it returns an integer rank, not a timestamp, so no interval can be computed from it. It is tempting for ordering events, but LAG supplies the previous row's value needed for the subtraction.

Create a reservation with 500 committed slots and another with 500 flex slots.

Why it fails: Splitting into a 500-slot commitment plus a separate 500-slot flex reservation fixes capacity at 1000 and bills both reservations, so it does not scale automatically from a 500 baseline. It suits workloads needing two distinct, independently managed slot pools.

Use on-demand pricing with a maximum query cost limit.

Why it fails: On-demand pricing bills per query with no slot reservation, so no 500-slot baseline exists and scaling to 1000 is not guaranteed. It is the right model for unpredictable, infrequent workloads where capacity commitments would be wasted.

Purchase 1000 committed use slots to ensure consistent capacity.

Why it fails: Purchasing 1000 committed use slots bills for the full 1000 continuously, ignoring the requirement to pay only for slots beyond a 500-slot baseline. Committed use discounts suit steady, predictable consumption where you want lower per-slot pricing across the whole commitment; here autoscaling from a 500-slot baseline is needed instead.

The trigger is firing too early, causing the window to close prematurely

Why it fails: The trigger fires early and at watermark, but that does not cause late data to appear in a different window. Late data is still assigned to its original window if allowed lateness is set.

The watermark is incorrectly estimated, allowing late data to be included

Why it fails: Watermark estimation can be imperfect, but the event's timestamp is after the window end, so it is late by definition. Allowed lateness is the mechanism that lets it be included, not watermark error.

The window duration is actually 2 minutes due to a misconfiguration

Why it fails: If the window were 2 minutes, the event would fall into [10:00, 10:02) or similar; but the output pane is labeled as [10:00,10:01), indicating the window is 1 minute fixed.