AI0-001 AI Implementation and Operations Practice Test 2 — 15 Questions

Question 1

A team deploys a real-time fraud detection model on a streaming platform. The model must produce predictions within 100 milliseconds per event. Initial latency is 150 ms. Which optimization is most likely to meet the latency requirement?

Accepted Answer

Apply model quantization to reduce precision from FP32 to INT8.. Model quantization reduces the numerical precision of the model's weights and activations from FP32 to INT8, which decreases memory footprint and speeds up inference. This optimization directly addresses the 150 ms latency by enabling faster arithmetic operations on modern hardware, often cutting inference time by 2-4x, which can bring latency below the 100 ms requirement.

Answer

Increase the batch size to process more events simultaneously.

Answer

Add more feature engineering steps to improve model accuracy.

Answer

Migrate from a decision tree ensemble to a deep neural network.

Question 2

A DevOps team is deploying a machine learning model using a CI/CD pipeline. They want to ensure the model is reproducible and traceable. Which TWO practices should they implement?

Accepted Answer

Version the training dataset and code using Git and DVC.. Option A and D are correct because versioning data and code, and using containers ensure reproducibility. Option B is wrong because manual deployment is error-prone and not traceable. Option C is wrong as it does not track the model. Option E is wrong because manual tagging is inconsistent.

Answer

Manually deploy the model to production after approval.

Answer

Store only the final model artifact in a shared drive.

Answer

Use a spreadsheet to record model version numbers.

Question 3

An AI operations team is monitoring a deployed image classification model. They notice a gradual increase in prediction confidence but a drop in accuracy. Which THREE actions should they take to diagnose the issue?

Accepted Answer

Analyze the model's calibration curve to see if confidence scores align with actual accuracy.. Option A is correct because a calibration curve (reliability diagram) directly compares predicted confidence scores against actual accuracy. In this scenario, increasing confidence with dropping accuracy indicates miscalibration—the model is becoming overconfident. Analyzing the calibration curve reveals whether the confidence scores systematically deviate from true probabilities, which is the core diagnostic step for this specific symptom.

Answer

Increase the size of the training dataset by collecting more unlabeled data.

Answer

Retrain the model immediately with the most recent data.

Question 4

Based on the exhibit, what is the most likely cause of the accuracy drop?

Accepted Answer

Data drift in the 'income' feature has caused the model to become less accurate.. The exhibit shows a sudden and sustained drop in model accuracy coinciding with a shift in the distribution of the 'income' feature. This is a classic symptom of data drift, where the statistical properties of the input feature change over time, causing the model's learned patterns to no longer match the production data. Option B correctly identifies this as the most likely cause because the model was trained on a prior income distribution and is now encountering values outside that range.

Answer

A required feature is missing from the production data pipeline.

Answer

The model was overfitted to the training data.

Answer

The model's confidence threshold needs to be adjusted.

Question 5

You are an AI engineer at a financial services firm. The company has deployed a gradient boosting model to predict loan default risk. The model takes features such as credit score, debt-to-income ratio, loan amount, and employment length. In production, the model processes about 10,000 predictions per day with an average latency of 50ms. Recently, the accuracy has dropped from 92% to 85%. You also notice that the average credit score of applicants has increased significantly because the marketing team launched a campaign targeting prime borrowers. The model was originally trained on data from the past three years, which included a mix of prime and subprime borrowers. You need to restore model performance while minimizing downtime and retraining cost. Which action should you take first?

Accepted Answer

Retrain the model using the last three months of production data with labels.. The drop in accuracy is due to data drift—the production data now has a different distribution (higher credit scores) than the training data. Retraining on the most recent three months of production data with labels directly addresses this shift by adapting the model to the new population, and it minimizes downtime because it uses existing infrastructure and avoids complex architectural changes.

Answer

Add a regularization term to penalize high credit scores.

Answer

Deploy an ensemble of the original model and a neural network.

Answer

Reject all predictions where the confidence score is below 0.9.

Question 6

A company deployed a machine learning model on a cloud inference service. Users report high latency during peak hours. The model is deployed on a single instance. Which action should the team take to reduce latency without significant architectural changes?

Accepted Answer

Enable autoscaling for the inference instances. Enabling autoscaling allows the inference service to automatically add instances during high demand, distributing the load and reducing latency. Increasing the model size would worsen latency. Switching to a batch inference pipeline would increase latency for real-time requests. Adding an API gateway does not address compute capacity.

Answer

Increase the model size to improve accuracy

Answer

Switch to a batch inference pipeline

Answer

Add an API gateway to route requests

Question 7

A data science team uses Git for version control of model code and DVC for data versioning. They want to implement a model registry to track trained models, their hyperparameters, and performance metrics. Which tool is specifically designed for this purpose and integrates with the existing workflow?

Accepted Answer

MLflow Model Registry. MLflow Model Registry is specifically designed for managing model versions, tracking metadata, and integrating with Git and DVC. Apache Airflow is for workflow orchestration, not model registry. Kubernetes is for container orchestration. Docker is for containerization.

Answer

Apache Airflow

Answer

Docker

Answer

Kubernetes

Question 8

An ML team monitors a production model using a dashboard that shows daily performance metrics. Over the past month, the model's accuracy has dropped from 92% to 87%, while the data distribution of input features has remained stable according to statistical tests. Which type of model drift is most likely occurring?

Accepted Answer

Concept drift. Concept drift occurs when the relationship between input features and the target variable changes, even if the input data distribution remains stable. In this scenario, the model's accuracy declines from 92% to 87% while input feature distributions are unchanged, indicating that the underlying mapping from features to labels has shifted—a classic sign of concept drift.

Answer

Data drift (covariate shift)

Answer

Model decay

Answer

Overfitting

Question 9

A company must deploy a new model version with zero downtime. The current model is served via a REST API on a Kubernetes cluster. Which deployment strategy should the team use to gradually shift traffic to the new version while monitoring for errors?

Accepted Answer

Canary deployment. A canary deployment gradually shifts a small percentage of traffic to the new model version while the majority continues to hit the stable version. This allows the team to monitor for errors and roll back quickly if issues arise, achieving zero downtime. It is the ideal strategy for validating a new model in production with minimal risk.

Answer

Blue-green deployment

Answer

Recreate deployment

Answer

Rolling update

Question 10

An operations team monitors a classification model in production. The confusion matrix for the model shows the following values: TP=1500, FN=500, FP=600, TN=2400. Which metric should the team calculate to assess the model's ability to avoid false positives?

Accepted Answer

Precision. Precision (TP / (TP + FP)) directly measures the proportion of positive identifications that were actually correct, making it the ideal metric to assess the model's ability to avoid false positives. With TP=1500 and FP=600, precision is 1500/(1500+600)=0.714, indicating that 71.4% of predicted positives are true positives, while 28.6% are false positives. The question explicitly asks about avoiding false positives, which is the inverse of precision's focus on the correctness of positive predictions.

Answer

Accuracy

Answer

F1-score

Answer

Recall

Question 11

An ML engineering team has a retraining pipeline that triggers automatically when model accuracy drops below a threshold. Recently, the model's accuracy has been fluctuating, causing frequent retraining and high compute costs. The team suspects the data distribution is changing slowly. Which approach should the team implement to reduce unnecessary retraining while maintaining model performance?

Accepted Answer

Implement a statistical drift detection method on input features. Option B is correct because implementing a statistical drift detection method (e.g., using KL divergence, PSI, or ADWIN) on input features allows the team to identify when the data distribution has genuinely changed, rather than reacting to random accuracy fluctuations. This reduces unnecessary retraining by triggering the pipeline only when statistically significant drift is detected, maintaining model performance without the high compute costs of frequent retraining.

Answer

Use a simpler model to reduce variability

Answer

Increase the frequency of model retraining

Answer

Reduce the batch size for inference

Question 12

A company deploys an AI model via a REST API that handles sensitive customer data. To secure the endpoint, the security team requires that only authenticated and authorized applications can invoke the API. Which mechanism should be implemented?

Accepted Answer

API key or bearer token in the HTTP header. Option A is correct because API keys or bearer tokens (e.g., OAuth 2.0 access tokens) are the standard mechanism for authenticating and authorizing client applications when invoking a REST API. These tokens are passed in the HTTP Authorization header, allowing the server to verify the client's identity and permissions before processing requests containing sensitive customer data.

Answer

TLS encryption for the connection

Answer

Input sanitization to prevent injection

Answer

IP whitelisting

Question 13

A data scientist submits a model training job to a cloud ML platform. The job fails with an error: "Out of memory: Killed process." The training code is proven to work on the developer's local machine with 16GB RAM. The cloud instance has 32GB RAM. What is the most likely cause?

Accepted Answer

The training code is loading the entire dataset into memory without batch processing. Option B is correct because the error 'Out of memory: Killed process' occurs when the operating system's OOM killer terminates a process that exceeds available RAM. Even though the cloud instance has 32GB RAM, if the training code loads the entire dataset into memory without batch processing (e.g., using pandas.read_csv() without chunking), it can consume far more memory than the dataset size due to Python object overhead, temporary copies, and intermediate tensors. The local machine with 16GB RAM may have worked due to a smaller dataset or different memory pressure, but the cloud instance's 32GB is insufficient if the code is not memory-efficient.

Answer

The cloud instance does not have enough memory

Answer

The training model is too large for the GPU memory

Answer

The cloud instance ran out of disk space

Question 14

A company serves a large language model (LLM) on a Kubernetes cluster. The inference latency is acceptable but the cost is high due to GPU usage. The model is 7 billion parameters and requires 16GB GPU memory. The team wants to reduce cost without increasing latency. Which strategy should they implement?

Accepted Answer

Use model quantization to reduce precision. Model quantization reduces the precision of the model's weights (e.g., from FP32 to INT8), which decreases the GPU memory footprint from 16GB to approximately 4GB for a 7B parameter model. This directly lowers GPU cost per inference while maintaining acceptable latency, as the model can run on fewer or cheaper GPUs without increasing inference time.

Answer

Increase the batch size for inference

Answer

Add more GPU nodes to distribute the load

Answer

Switch to CPU-based inference

Question 15

A team monitors a production model for bias. They measure the selection rate for two demographic groups and find a significant difference. Which TWO actions should the team take to mitigate bias? (Choose two.)

Accepted Answer

Retrain the model with a balanced training dataset. Retraining with a balanced training dataset (Option C) directly addresses the root cause of bias by ensuring the model learns from equal representation of both demographic groups, which reduces skewed selection rates. This is a standard data-level mitigation technique in AI fairness, as it prevents the model from overfitting to majority patterns.

Answer

Increase the complexity of the model to capture more patterns

Answer

Add more training data from both groups

Answer

Remove the protected attribute from the model input