Hard Difficulty Questions

Configure the component that produces the Metrics artifact to also output the metric as a pipeline parameter.

Why it fails: Emitting the metric as a component output parameter is precisely the mechanism that makes the value available to dsl.If, so this option is not incorrect. It is the standard pattern: the component reads the metric from its Metrics artefact and returns it as a typed output parameter.

Use the importer component to convert the artifact into a parameter.

Why it fails: An importer component brings external artefacts into a pipeline; it does not extract a metadata value into a scalar pipeline parameter. It is tempting because importers do surface artefacts, but converting metadata to a parameter requires reading that value and emitting it as an output parameter.

Use the artifact directly in the dsl.If condition, as artifacts are comparable.

Why it fails: dsl.If conditions evaluate pipeline parameters, not artefact objects; artefacts are not comparable in the condition expression. Referencing an artefact directly fails because the condition compiler expects a scalar value, which is why the metric must first be surfaced as a parameter.

Use full fine-tuning of all model parameters

Why it fails: Full fine-tuning is computationally expensive and not cost-minimizing.

Use TPU v4 pods for training

Why it fails: TPUs are powerful but typically more expensive than GPU spot instances for fine-tuning; not a cost-minimization strategy.

Use a scheduled pipeline that always retrains

Why it fails: Always retraining ignores the condition that retraining occur only when the new week's data significantly changes the distribution, wasting compute and potentially degrading the model. It would be correct where drift is continuous or unknown, but here the stem explicitly requires a significance gate.

Use Cloud Monitoring alerts on data drift to trigger retraining

Why it fails: Cloud Monitoring alerts fire on operational metrics and thresholds, not on statistical comparison of consecutive weeks' feature distributions, so they cannot express the significance condition. They would be correct for notifying on latency or error-rate breaches, not for gating AutoML retraining on distribution shift.

Use Cloud Functions on schedule to compare distributions

Why it fails: Cloud Functions can compute a distribution comparison, but the stem requires weekly scheduling plus a significance test gating retraining; a standalone function supplies neither the pipeline trigger nor Vertex AI's managed drift detection. It would suit lightweight event-driven tasks, not orchestrating AutoML retraining.

Cloud Scheduler for the schedule, and Cloud Pub/Sub with Push subscription to Vertex AI for event-driven.

Why it fails: Cloud Pub/Sub cannot directly trigger Vertex AI pipelines; it requires a subscriber like Cloud Functions.

Cloud Functions for both schedule and event-driven, using cron trigger.

Why it fails: Cloud Functions cannot use a cron trigger to respond to new data arriving in a Cloud Storage bucket; cron triggers are time-based, not event-based. While Cloud Functions are suitable for both event-driven processing via Cloud Storage events and scheduled execution when paired with Cloud Scheduler, the option incorrectly suggests a single cron trigger mechanism for both requirements. Cloud Functions are often a correct choice for lightweight, event-driven tasks that need to react to various Google Cloud events or execute on a schedule, acting as the compute layer that then orchestrates other services.

Vertex AI Pipelines built-in scheduler for schedule, and Cloud Pub/Sub for event-driven.

Why it fails: Vertex AI Pipelines does not have a built-in scheduler; it relies on Cloud Scheduler. Also Pub/Sub alone cannot trigger pipelines.

Rebuild the entire index hourly as a batch job and redeploy it.

Why it fails: Rebuilding the whole index hourly discards the existing serving index and forces a full redeploy, adding latency and downtime that streaming updates avoid. It is tempting because batch rebuilds are correct when the embedding model or index schema changes and incremental updates cannot apply.

Create a new index each hour and use traffic splitting to gradually shift traffic.

Why it fails: Vertex AI Vector Search supports streaming updates to an existing index, so hourly rebuilds and traffic splitting add operational overhead without improving freshness. It is tempting because traffic splitting safely validates a new index, which is right when changing index configuration such as distance measure or shard size.

Use a brute-force index instead of ANN to ensure accuracy after updates.

Why it fails: Brute-force search computes exact distances against every vector, so query latency scales linearly with index size and rises sharply as hourly embeddings accumulate. It suits small, static datasets where recall matters more than speed. Vertex AI Vector Search's ANN indexes already support streaming updates, giving freshness without that penalty.

Brute-force index

Why it fails: Brute-force compares the query against every vector, giving exact results but latency and cost scale linearly with 10 million embeddings, missing the 10ms target. It is correct for small datasets or ground-truth benchmarking; Vertex AI Vector Search's approximate nearest-neighbour index is required here.

Hash-based index

Why it fails: Hash-based indexes bucket vectors by hash, which supports exact-match lookup but not nearest-neighbour ranking, so similarity search quality collapses. They are appropriate for duplicate detection or exact retrieval; approximate nearest-neighbour indexing is needed for low-latency similarity over 10 million embeddings.

Tree-based index

Why it fails: Tree-based indexes partition space hierarchically and degrade at high dimensionality, where traversal prunes poorly and latency rises beyond the 10ms target. They suit lower-dimensional or smaller datasets; Vertex AI Vector Search's scalable approximate nearest-neighbour index is required for 10 million high-dimensional embeddings.

Compress the model file and decompress on startup.

Why it fails: Decompression adds CPU work at container start, so total cold-start time grows rather than shrinks; the 5 GB file still transfers from the registry. Compression suits reducing image size for storage or bandwidth-limited pulls, not latency-sensitive serving where the model must be resident before the first prediction.

Use a machine type with local SSD to speed up model loading.

Why it fails: Local SSD only accelerates reads after the image and model are already on the host; the dominant cost is pulling the 5 GB artifact from the registry over the network. Local SSD is the right lever for disk-bound training or inference I/O, not for registry download latency.

Switch to batch prediction to avoid online cold start.

Why it fails: Batch prediction removes online serving entirely, so it cannot reduce cold start latency for an online endpoint; it changes the workload type instead. It is tempting when latency is unacceptable, but the requirement is faster online startup, which needs image and model loading optimisations.

A manual QA step where data scientists approve each deployment.

Why it fails: Manual approval gates deployments on human availability, so weekly retraining cannot run unattended; drift and performance monitoring need automated triggers instead. It is tempting because human sign-off suits regulated, low-frequency releases where auditability outweighs cadence, not scheduled weekly pipelines.

A manual review of new data before it is used for training.

Why it fails: Manual review adds human latency and does not scale to weekly retraining, and it provides no drift or performance monitoring. It is tempting where regulatory approval of training data is mandatory, but automated pipelines with Vertex AI Pipelines and Model Monitoring satisfy this scenario.

Deploy the preprocessing logic as a Cloud Function and invoke it before calling the prediction endpoint

Why it fails: A Cloud Function adds a separate network hop and service call before prediction, which is exactly the latency and extra-call overhead the team wants to eliminate. Cloud Functions suit decoupled, event-driven preprocessing outside the model. Embedding the tokenisation and normalisation layers inside the PyTorch module keeps inference self-contained.

Wrap the preprocessing logic in a Flask application and deploy it as a separate microservice in front of the prediction endpoint

Why it fails: Deploying preprocessing as a separate Flask microservice introduces network round-trip latency between the microservice and the prediction endpoint, directly contradicting the requirement to reduce prediction latency. This approach is tempting because it cleanly separates concerns and would be correct if the preprocessing logic needed independent scaling or was shared across multiple services, but here the goal is to embed logic within the model artifact to avoid any inter-service calls.

Use TensorFlow Transform to convert preprocessing into a SavedModel and call it from the PyTorch model

Why it fails: TensorFlow Transform emits a SavedModel for TensorFlow serving signatures, so a PyTorch model cannot invoke it inline during forward passes. It suits TF-based pipelines needing consistent train/serve feature engineering. Embedding tokenisation and normalisation directly in the PyTorch module avoids the extra service call the stem requires.

Document AI with a custom extractor processor and use Cloud Functions for human review

Why it fails: Document AI custom extractors handle scanned PDFs, but Cloud Functions provides no human-in-the-loop review queue or console; reviewers cannot adjudicate high-risk applications. It suits lightweight event-driven automation, whereas the human review step requires Document AI's Human-in-the-Loop configuration.

Cloud Vision API to detect text and tables, then send to Cloud Dataflow for processing

Why it fails: Cloud Vision returns raw OCR text without loan-application field or table structure, and Dataflow only moves that text onward. It suits generic OCR pipelines feeding downstream analytics, not schema-aware extraction with a built-in human review step for high-risk applications.

Vertex AI AutoML Vision to train a custom model for document parsing

Why it fails: AutoML Vision classifies images and detects objects; it does not return structured key-value fields or table rows from scanned PDFs. It suits image labelling or custom classification tasks, not document field extraction, and provides no native human-review workflow for high-risk applications.

Use Vertex AI Experiments to compare models offline, then deploy the winner

Why it fails: Vertex AI Experiments compares offline runs; it cannot split live traffic across two deployed models on one endpoint. It is tempting because offline evaluation suits choosing a winner before deployment, but the scenario requires 10% of production requests routed to the challenger, which only traffic-split configuration on the endpoint delivers.

Deploy the challenger model to a separate endpoint and use a load balancer to split traffic

Why it fails: Deploying the challenger to a separate endpoint with an external load balancer introduces an additional network hop and fails to leverage Vertex AI’s native traffic-splitting mechanism, which directly assigns a percentage of requests to each model version within a single endpoint. This approach is tempting because it mirrors a standard microservices pattern for distributing load across independent services, and would be correct if the goal were to balance traffic between entirely separate applications rather than between model versions within the same managed serving infrastructure.

Update the champion model with a new version and use model version aliases

Why it fails: Model version aliases point a single deployed version to a named reference; they do not split live request traffic between two versions on one endpoint. Aliases suit promoting a validated version or rolling back. The 10% challenger split requires configuring traffic-split percentages across deployed model versions on the endpoint.

Use Cloud Composer to orchestrate preprocessing

Why it fails: Cloud Composer orchestrates and schedules preprocessing tasks, but each DAG defines its own steps, so teams can still implement transformations differently. Composer suits workflow scheduling and dependency management. Enforcing identical preprocessing across teams needs a reusable, versioned component rather than orchestration alone.

Write shared Python packages in Artifact Registry

Why it fails: Shared Python packages in Artifact Registry distribute code, but nothing forces teams to call them, so preprocessing can still diverge. They suit versioned dependency management across projects. Enforcing consistency requires a pipeline component that every team's workflow must execute, not an optional import.

Use Cloud Dataflow templates

Why it fails: Dataflow templates run a fixed pipeline per job; they do not package reusable preprocessing logic that teams import into their own training code. Templates suit repeatable batch or streaming jobs, not enforcing shared transformations across independent model builds. A shared library is what teams actually call from their pipelines.

Use Cloud Composer (Airflow) with a daily DAG to run SQL queries on BigQuery and export results

Why it fails: Daily batch DAGs cannot compute 7-day and 30-day rolling windows over a continuous event stream with the freshness online serving demands, and Composer's scheduler overhead adds cost without streaming semantics. It suits scheduled ETL orchestration, not continuous feature computation; Dataflow with sliding windows would be the correct choice.

Schedule a query in BigQuery using scheduled queries and export results to Feature Store

Why it fails: Scheduled queries recompute the entire window on each run, so cost and latency scale with table size rather than with newly arrived events. It is tempting because BigQuery scheduled queries suit periodic batch refreshes of static datasets, where a nightly full recomputation is acceptable and streaming freshness is not required.

Use Cloud Functions triggered by Pub/Sub to compute features on the fly

Why it fails: Cloud Functions triggered by Pub/Sub recompute each event independently, so they cannot maintain the cross-event state a 7-day average or 30-day sum requires; each invocation sees only its own payload. This pattern suits stateless per-event transforms such as enrichment or validation, where no windowing across events is needed.

Data size of input

Why it fails: Input data size describes the volume fed to the job, not whether it completes without errors; a job can fail on any size. It is tempting because sizing informs resource planning and cost estimation, and would be a valid capacity metric, but it cannot signal job failure or completion.

Prediction requests per second

Why it fails: Batch prediction processes a bounded dataset in one job, so requests per second is not a meaningful throughput signal and does not indicate errors. It is tempting because it is the standard health metric for online endpoints, where it would correctly reveal serving load, but batch jobs report progress differently.

Model endpoint latency

Why it fails: A batch prediction job writes outputs to a destination rather than serving through an endpoint, so endpoint latency is never emitted and cannot reveal job errors. It is tempting because latency is the canonical metric for online inference, where it would correctly flag slow responses, but no endpoint exists here.

Use Cloud Monitoring custom metrics and log-based alerts manually.

Why it fails: Custom metrics and log-based alerts must be built and tuned by the team, and they detect only what is explicitly instrumented, not statistical distribution shift. This suits bespoke operational telemetry; feature drift needs the managed skew and drift detection that compares distributions to a baseline automatically.

Use Vertex AI Experiments to compare distributions.

Why it fails: Experiments compare runs and metrics for model evaluation, not continuous feature distributions, so no automatic alert triggers on drift. It suits tracking training iterations and hyperparameters; monitoring distribution deviation against a baseline requires Vertex AI Feature Store's drift detection configuration.

Export features to BigQuery and set up scheduled queries with alerts.

Why it fails: Scheduled BigQuery queries add latency and require custom comparison logic, so alerts fire after the drift window rather than automatically. Export suits ad-hoc analysis or reporting; Vertex AI Feature Store's native drift monitoring computes distribution deviation against the baseline and raises alerts directly.

Compute engine CPU and memory logs for each pipeline step.

Why it fails: Pipeline-step CPU and memory metrics describe the underlying Compute Engine VMs, not the ML workflow itself. Vertex AI Pipelines exposes task-level execution metadata, artefacts, and lineage; the dashboard needs those pipeline-run signals. Host metrics would be the right choice when diagnosing infrastructure saturation on self-managed training VMs.

Online prediction latency and request count from the deployed model endpoint.

Why it fails: The pipeline covers training, evaluation, and batch prediction, so endpoint request metrics sit outside its scope — no deployed online endpoint is involved. Online latency and request counts belong on a dashboard for a real-time prediction service, where traffic volume and tail latency drive autoscaling and SLO alerting.

Store model artifacts in Cloud Storage with versioned paths and deploy directly from Storage without using the Model Registry.

Why it fails: Bypassing the registry loses version tracking and alias management.

Create two separate models in the registry: one for staging and one for production, and copy artifacts between them after approval.

Why it fails: Creating two separate models in the registry breaks the lineage tracking and versioning continuity that Vertex AI Model Registry provides, as the staging and production models would be independent entities with no automated promotion path; manual artifact copying introduces risk of configuration drift and bypasses the registry’s built-in alias system (e.g., `staging`, `production`). This approach is tempting because it mirrors a common pattern for separating environments in traditional software deployments, where distinct registries or repositories isolate staging from production, and it would be correct if the team needed to maintain entirely independent model lifecycles with no shared version history.

Upload each model version with the same model ID and use labels to differentiate staging vs production.

Why it fails: Labels are not designed for deployment promotion; aliases are the intended mechanism.

The container port should be 8080 instead of 8501.

Why it fails: Port 8501 is TensorFlow Serving's default, and Vertex AI accepts custom container ports, so changing it would not fix the upload error. Container ports are tempting because misconfigured serving ports cause prediction failures, and specifying 8501 is correct when deploying a TensorFlow Serving image that listens there.

The service account does not have permission to access the bucket.

Why it fails: A bucket permission error would surface as an access-denied or storage failure, not the error produced by uploading a SavedModel directory lacking the required serving signature. IAM roles are tempting because upload failures often stem from permissions, and granting storage access is correct when the service account genuinely cannot read the artefact.

The container image is for TensorFlow 2.11 but the model was saved with an older version.

Why it fails: The error stems from the SavedModel lacking a serving signature, not from a version mismatch between container image and model. Version alignment is tempting because TensorFlow 2.11 models can fail under mismatched runtimes, and pinning the matching serving image is correct when incompatibility genuinely causes load failures.

Use a batch processing system for both training and serving to ensure identical feature calculations.

Why it fails: Batch processing cannot serve live Pub/Sub transactions, so it breaks the real-time requirement entirely. It is tempting because a single batch pipeline guarantees identical feature logic for training and scoring, which suits offline scoring or periodic retraining jobs rather than streaming fraud detection.

Implement separate feature engineering pipelines for training and serving, but document them carefully.

Why it fails: Separate pipelines let the two implementations diverge over time, so training and serving features drift despite documentation. A shared transformation, such as a Vertex AI Feature Store or a common Dataflow pipeline, guarantees identical logic. Separate pipelines suit teams whose training and serving stacks genuinely cannot share code.

Ensure that both training and serving read from the same Cloud Storage location.

Why it fails: Sharing a Cloud Storage location does not align feature computation; Dataflow applies transforms at serving, so skew persists. It is tempting because a common data source feels like consistency, and it would be correct for storing shared training artefacts or model files, not for unifying feature logic.

The model is stored in a regional bucket and the Vertex AI endpoint is in a different region.

Why it fails: Cross-region bucket and endpoint placement causes latency or access-permission failures, not a MemoryError during model loading. It is tempting because region mismatch is a common Vertex AI deployment pitfall, but that scenario surfaces as storage access errors, not RAM exhaustion at container startup.

The machine type does not support TensorFlow models larger than 1 GB.

Why it fails: Vertex AI imposes no fixed 1 GB TensorFlow size ceiling; n1-standard-4 provides 15 GB RAM, so a 2 GB artifact fits. The claim is tempting because machine types do bound deployable model size, but the actual limit derives from available memory, not an arbitrary 1 GB cap.

The model file is corrupted or missing dependencies, causing a crash.

Why it fails: A MemoryError at startup indicates the container exhausted its allocated RAM while loading the model, not a corrupt artifact or missing dependency, which would raise import or deserialisation errors instead. Corruption is tempting because it also prevents startup, but the reported symptom is specifically memory exhaustion during model loading.