Hard Difficulty Questions

Share notebooks via Colab Enterprise with real-time editing

Why wrong: Sharing notebooks does not enforce version control of data and dependencies.

Allow each team to use their own preferred environment

Why wrong: Different environments lead to unreproducible results.

Always use random seeds for all random operations

Why wrong: Random seeds are not always applicable and can be inconsistent.

Run hyperparameter tuning jobs with a large search space

Why wrong: Larger searches require more trials, increasing cost.

Manually scale up resources during peak times and scale down during off-peak

Why wrong: Manual scaling is not efficient; Vertex AI handles scaling based on job requirements.

Use a larger machine type for training to complete faster

Why wrong: Larger machines cost more; faster completion may not offset cost.

Migrate the model to Cloud TPU using TensorFlow to benefit from higher throughput.

Why wrong: Migration may require significant code changes and TPUs may not be cost-effective for this model.

Use a smaller GPU, such as NVIDIA T4, and increase the number of replicas to maintain throughput.

Why wrong: Smaller GPU may increase latency and cost per request due to higher replica count.

Reduce the number of min replicas to 0 and scale from 0 on each request.

Why wrong: Scaling from zero adds cold start latency and does not improve GPU utilization.

Increase the complexity of the model by switching from a feedforward neural network to a gradient boosted tree ensemble, and then deploy without retraining.

Why wrong: Changing model architecture does not fix the data drift; the model still needs retraining on new data.

Route all predictions to human reviewers until the model can be re-evaluated, and then manually correct the outputs.

Why wrong: Routing predictions to human reviewers introduces manual latency and cost, failing the requirement for minimal downtime and low cost; the model's accuracy degradation stems from covariate shift and concept drift, which human correction does not address at the data or retraining level. This option is tempting because human-in-the-loop validation is a standard fallback for high-stakes predictions where immediate automated accuracy cannot be guaranteed, such as in fraud detection or medical diagnosis, where a temporary manual override protects against catastrophic errors while a model is rebuilt.

Adjust the prediction threshold for the 'product_price' feature to account for the price shift, and monitor for another month.

Why wrong: Threshold adjustment only works for binary classification, not regression, and does not correct overall prediction accuracy.

Use a larger machine type to reduce the number of workers.

Why wrong: Use a larger machine type to reduce the number of workers — this increases cost per worker and may not reduce overall cost; it could even increase cost if the larger machine is underutilized.

Use TPUs instead of GPUs to accelerate processing.

Why wrong: Use TPUs instead of GPUs to accelerate processing — TPUs are generally more expensive and only beneficial for specific models (e.g., large neural networks); not a general cost-saving measure.

Enable Cloud Storage audit logs and require all model files to be stored in a bucket

Why wrong: This does not capture model-specific actions like deployment or evaluation.

Use Cloud Logging to collect logs from all services and search for model names

Why wrong: This is cumbersome and does not provide structured metadata.

Ask team members to maintain a shared spreadsheet of changes

Why wrong: Manual logging is unreliable and not auditable.

Enable auto-scaling with a higher minimum number of replicas.

Why wrong: Helps with overall load but not per-request latency spikes.

Use a larger machine type with more CPUs.

Why wrong: May not address the root cause of spikes; could be compute-bound.

Use a GPU-based machine.

Why wrong: May improve throughput but not necessarily tail latency spikes.

Compress the model file and decompress on startup.

Why wrong: Decompression adds overhead and does not reduce total time.

Use a machine type with local SSD to speed up model loading.

Why wrong: Local SSD is ephemeral and may not persist across restarts; also not the most effective.

Switch to batch prediction to avoid online cold start.

Why wrong: Batch prediction doesn't reduce online cold start latency.

Use full fine-tuning of all model parameters

Why wrong: Full fine-tuning is computationally expensive and not cost-minimizing.

Use TPU v4 pods for training

Why wrong: TPUs are powerful but typically more expensive than GPU spot instances for fine-tuning; not a cost-minimization strategy.

Use Cloud Scheduler to run a job every hour that checks for new files in Cloud Storage and starts the pipeline if new files exist.

Why wrong: This is not event-driven and introduces latency; also requires polling logic.

Configure a Cloud Build trigger that listens to the Pub/Sub topic and executes a build step that submits the pipeline run.

Why wrong: Cloud Build triggers listening to Pub/Sub alone cannot satisfy the requirement for new training data arriving in a Cloud Storage bucket. The scenario demands a dual condition: both the Pub/Sub notification and the data arrival must be present to trigger the Vertex AI pipeline. While Cloud Build is excellent for initiating CI/CD processes, such as code deployment or running tests, directly from Pub/Sub messages, it does not natively combine Pub/Sub and Cloud Storage event conditions into a single trigger for this specific orchestration.

Create a Dataflow streaming pipeline that reads from Pub/Sub and triggers the Vertex AI pipeline via a custom sink.

Why wrong: Dataflow is overkill for just triggering a pipeline; adds complexity and cost.

Increase the target CPU utilization to 90% to allow more requests per instance.

Why wrong: Higher utilization may cause overload, worsening errors.

Enable canary traffic splitting to reduce load on the main endpoint.

Why wrong: Canary deployment doesn't directly handle spikes; it's for rollout.

Reduce the model batch size from 32 to 1 to lower memory per request.

Why wrong: Smaller batch size increases CPU overhead per request and may not resolve 502s if memory is insufficient.

Model A is being overloaded because autoscaling is based on aggregate traffic.

Why wrong: Autoscaling usually monitors per-model metrics.

The traffic split is misconfigured, causing requests to be routed incorrectly.

Why wrong: Traffic split works correctly; latency increase due to contention.

Model B's logging is generating too much output, slowing down the predictor.

Why wrong: Logging overhead is minimal and not model-specific in this case.

The request JSON is malformed due to a missing comma between instances.

Why wrong: The JSON is valid; the error is about shape mismatch, not parsing.

The endpoint path is incorrect; it should include the model version.

Why wrong: The endpoint path is correct; the error is a prediction failure, not a routing error.

The request is sending 3 separate instances but the model expects only 1.

Why wrong: The request sends one instance with 3 features, not 3 instances. The error says shape (1,3) vs expected (None,2).

Import the component from Google Cloud Build

Why wrong: Cloud Build is for building containers, not for ML pipeline components.

Use the 'aiplatform' Python SDK to define the component

Why wrong: The aiplatform SDK is for interacting with Vertex AI services, not for reusing prebuilt components.

Copy the component code into the pipeline definition

Why wrong: This duplicates code and makes updates harder.

Deploy multiple models and use an ensemble to average predictions

Why wrong: Ensemble does not directly address drift.

Manually review model predictions daily

Why wrong: Manual review is not scalable or automatic.

Monitor prediction errors and flag when confidence is low

Why wrong: This monitors prediction quality, not data drift.

Reduce the min_replica_count to 0 to allow scaling down aggressively and add more replicas during spikes.

Why wrong: min=0 increases cold start latency, not reducing overall latency.

Increase the machine type to n1-highmem-16 with more vCPUs.

Why wrong: More CPU may help but the deep learning models will still be slow on CPU.

Switch the model to Vertex AI batch prediction and run predictions every hour.

Why wrong: Batch prediction does not meet real-time 200ms SLA.

Reduce the size of the file being downloaded.

Why wrong: Reducing file size may help but is not a reliability practice; also may not be feasible.

Increase the machine type for the component.

Why wrong: Machine type affects compute resources, not network timeout; not directly related.

Store batch prediction results in BigQuery.

Why wrong: Not required; you can store in GCS. The question asks for steps, but you could, but it's not among the top three.

Use Cloud Functions to convert images to TFRecord.

Why wrong: Not scalable for 100 TB.

The custom container is not compatible with Vertex AI

Why wrong: Vertex AI supports custom containers.

The model is cached and needs cache invalidation

Why wrong: Caching may affect but not the primary cause.

The new model version was not deployed to the same endpoint

Why wrong: If deployed to same endpoint, traffic splitting controls routing.

Use Vertex AI Experiments to track the traffic split percentages.

Why wrong: Experiments track training runs, not endpoint traffic splits.

Enable Cloud Memorystore to cache identical requests for both models.

Why wrong: Caching is irrelevant for A/B testing traffic splitting.

Use Cloud Monitoring to track custom metrics like conversion rate per model version.

Why wrong: Monitoring is necessary but not a step in traffic splitting; it's a separate activity.