AI0-001 AI Implementation and Operations — Timed 50 Questions

Question 1

A company deploys a computer vision model for quality inspection on a manufacturing line. After deployment, the model's accuracy drops from 95% to 80% over two weeks. Which action is most likely to address this issue?

50 minute time limit — choose an answer to begin.

Accepted Answer

Retrain the model using recently collected production data.. The accuracy drop over two weeks indicates data drift or concept drift, where the production data distribution changes over time. Retraining the model with recently collected production data realigns it with the current data distribution, directly addressing the drift. Option B (increasing confidence threshold) may reduce false positives but does not fix the underlying drift and could lower recall. Option C (decreasing learning rate) is irrelevant for inference; it only affects training and cannot be applied post-deployment to fix drift. Option D (deploying an ensemble) adds computational overhead and does not resolve drift; it might even mask the issue without correcting it.

Answer

Increase the confidence threshold for predictions.

Answer

Decrease the learning rate of the training algorithm.

Answer

Deploy an additional ensemble of models for redundancy.

Question 2

A data scientist fine-tunes a large language model for a legal document summarization task. After fine-tuning, the model performs well on test data but produces summaries that include hallucinated legal clauses. Which mitigation strategy is most effective?

Accepted Answer

Implement retrieval-augmented generation (RAG) to provide factual context.. Implement retrieval-augmented generation (RAG) to provide factual context. RAG reduces hallucinations by allowing the model to retrieve relevant, factual information from an external knowledge base during generation, grounding its output in verified data. Option A (different tokenizer) does not address the core issue of factual accuracy. Option B (decrease temperature) affects randomness but does not prevent the model from fabricating content. Option D (max token limit) truncates output but does not stop the model from including false information within that limit.

Answer

Use a different tokenizer during fine-tuning.

Answer

Decrease the temperature parameter to 0.1 during inference.

Answer

Set a maximum token limit of 50 for each summary.

Question 3

A developer sees the above error during inference on a deployed image classification model. What is the most likely cause?

Accepted Answer

The input images are not being resized to the required dimensions. The error during inference is typically caused by a mismatch between the input tensor shape expected by the model and the shape of the provided image. Image classification models are trained on images of a fixed size. If input images are not resized to the required dimensions, the model's input layer expects a different shape, resulting in a shape mismatch error. Option B correctly identifies this as the most likely cause.

Answer

The model version is incompatible with the serving framework

Answer

The inference server does not support batch processing

Answer

The model is overfitting to a specific image size

Question 4

A data science team uses a CI/CD pipeline for ML models. They need to ensure that each model version is traceable back to the exact training data and hyperparameters. Which practice should be implemented?

Accepted Answer

Use a model registry with metadata tracking (e.g., MLflow). A model registry, such as MLflow, serves as a centralized repository that tracks model versions along with metadata like training data snapshots and hyperparameters, ensuring full traceability. Git LFS (Option B) only handles large files, not metadata. Storing artifacts in blob storage with timestamped filenames (Option C) lacks structured tracking and query capabilities. A shared spreadsheet (Option D) is error-prone and not integrated into the CI/CD pipeline.

Answer

Use Git LFS for model files

Answer

Store model artifacts in blob storage with timestamped filenames

Answer

Record hyperparameters in a shared spreadsheet

Question 5

A CI/CD pipeline for a computer vision model uses canary deployment. After deploying a new version to 5% of traffic, the pipeline automatically rolls back due to a spike in error rate. The new model's inference time is 20% higher than the previous version. The operations team finds that the error is caused by timeout in the inference service. Which action should be taken to prevent future rollbacks?

Accepted Answer

Optimize the model using TensorRT or ONNX Runtime before deployment. The root cause of the timeout is the 20% higher inference time of the new model. Optimizing the model using TensorRT or ONNX Runtime reduces inference latency directly, addressing the performance bottleneck that causes timeouts. This prevents the spike in error rate and subsequent rollback without masking the underlying issue.

Answer

Increase the timeout threshold for inference requests

Answer

Implement a fallback to the previous model when timeout occurs

Answer

Reduce the canary percentage to 1% to minimize impact

Question 6

A healthcare AI system that diagnoses medical images must provide explanations for its predictions to comply with regulatory requirements. Which technique should the team implement?

Accepted Answer

Apply model interpretability methods such as SHAP or LIME.. SHAP (SHapley Additive exPlanations) and LIME (Local Interpretable Model-agnostic Explanations) are established model interpretability techniques that provide per-prediction explanations, which are essential for regulatory compliance in healthcare AI. These methods generate feature attribution scores or local surrogate models to explain why a specific diagnosis was made, meeting transparency requirements without sacrificing model performance.

Answer

Reduce the model's accuracy to make it simpler.

Answer

Only deploy rule-based systems.

Answer

Use a more complex deep learning model.

Question 7

Based on the exhibit, which action is permitted by this policy?

Accepted Answer

Deploy a new model to an endpoint.. The policy grants permissions to create a new model, create an endpoint configuration, and create an endpoint. It does not allow updating, deleting, or invoking an endpoint. Therefore, deploying a new model (which requires creating a new endpoint) is permitted.

Answer

Update an existing endpoint.

Answer

Delete an endpoint.

Answer

Invoke an endpoint for inference.

Question 8

Which TWO of the following are best practices for monitoring AI models in production?

Accepted Answer

Set up alerts for prediction latency and error rates.. Monitoring prediction latency and error rates is a core operational practice for AI models in production. High latency can degrade user experience and indicate resource bottlenecks, while error rates (e.g., 4xx/5xx HTTP status codes or model-specific failures) directly reflect service health. These metrics are typically collected via tools like Prometheus or cloud monitoring services and should trigger alerts to enable rapid incident response.

Answer

Monitor model accuracy only at deployment time.

Answer

Regularly retrain without checking performance.

Answer

Freeze the model version once deployed to avoid changes.

Question 9

Refer to the exhibit. The monitoring dashboard for a deployed churn prediction model shows a drift detected flag. However, the error rate and latency are within acceptable ranges. What is the most appropriate immediate action?

Accepted Answer

Investigate the type and severity of drift before deciding. When drift is detected but performance metrics like error rate and latency are still acceptable, it is important to investigate the type and severity of drift before taking any action. Drift may be benign or may indicate a shift that will eventually degrade performance. Option A is wrong because automatic retraining could be risky if the drift is temporary or benign. Option B is wrong because rolling back immediately discards potential improvements and could be unnecessary. Option C is wrong because ignoring drift may lead to future degradation.

Answer

Trigger automatic retraining using the latest data

Answer

Roll back to the previous model version immediately

Answer

Ignore the drift since performance metrics are stable

Question 10

Which TWO techniques are most effective for ensuring model explainability in a production loan approval AI system subject to regulatory review? (Select TWO.)

Accepted Answer

Use SHAP values to understand feature contributions. The correct answers are B (SHAP values) and D (LIME). SHAP values provide both global and local feature importance, satisfying explainability for regulatory review. LIME generates local explanations for individual predictions, which is crucial for understanding specific loan decisions. Option A is incorrect because replacing the model with a decision tree may not be feasible or maintain performance. Option C is incorrect because attention weights are not reliable or standardized for explainability. Option E is incorrect because permutation importance offers global importance but lacks local explanations.

Answer

Replace the model with a decision tree for transparency

Answer

Rely on the model's internal attention weights (if transformer-based)

Answer

Calculate global feature importance using permutation importance

Question 11

A deployed NLP sentiment analysis model experiences a sharp decline in accuracy on customer reviews. The team has verified the input data format and pipeline are correct. Which THREE actions should be taken to diagnose and remediate? (Choose 3.)

Accepted Answer

Analyze recent user input for distribution shifts compared to training data.. A sharp decline in accuracy often stems from data drift, where the distribution of incoming customer reviews differs from the training data. Analyzing recent user input for distribution shifts (e.g., using statistical tests like Kolmogorov-Smirnov or population stability index) directly identifies whether the model is encountering unseen patterns. This is a standard first step in diagnosing model degradation in production NLP systems.

Answer

Immediately retrain the model with all available data.

Answer

Increase the size of the training dataset by adding synthetic data.

Question 12

The exhibit shows the output of a drift monitoring command for a fraud detection model. The team has an automated pipeline that triggers retraining when the overall average drift score exceeds 0.10. Based on the exhibit, what should the operations team do next?

Accepted Answer

Manually analyze the drift in 'amount' and 'location' and investigate potential causes.. The exhibit shows that the overall average drift score is below 0.10, so the automated retraining pipeline should not trigger. However, individual features like 'amount' and 'location' show elevated drift values that warrant manual investigation to understand root causes before any retraining decision. The team should analyze these specific features to determine if the drift is due to genuine data distribution changes or data quality issues.

Answer

Force retraining on all features to ensure the model adapts to the new data distribution.

Answer

No action is needed because the model is performing within acceptable drift limits.

Answer

Initiate the automated retraining pipeline since the average drift exceeds 0.05.

Question 13

A company deployed a chatbot using a pre-trained language model. Users report that the chatbot provides incorrect answers to domain-specific questions. Which approach should the AI team prioritize to improve accuracy without retraining the entire model?

Accepted Answer

Fine-tune the model on a curated dataset of domain-specific conversations.. Fine-tuning on a curated domain-specific dataset is the most efficient way to improve accuracy for specialized queries without retraining the entire model. It adjusts the model's weights using a smaller, targeted dataset, preserving general language understanding while adapting to domain terminology and context.

Answer

Increase the temperature parameter to reduce randomness.

Answer

Collect more general training data and retrain the model from scratch.

Answer

Roll back to a previous version of the model that was more accurate.

Question 14

An AI system misclassifies rare but critical events. The team considers using synthetic data. Which consideration is MOST important for ensuring the synthetic data improves performance on real rare events?

Accepted Answer

The synthetic data should accurately represent the distribution and features of real rare events.. Synthetic data must faithfully replicate the distribution and feature space of real rare events to enable the model to learn meaningful decision boundaries. If the synthetic data does not capture the true underlying patterns—such as specific sensor readings or transaction anomalies—the model will fail to generalize to actual rare events, defeating the purpose of augmentation.

Answer

The synthetic data should include a wide variety of events, even if not realistic.

Answer

The synthetic data should be generated using an unsupervised generative model.

Answer

The synthetic data should be as large as possible to cover all possibilities.

Question 15

A data scientist trains a regression model and notices the training loss is low but validation loss is high. Which technique should be applied FIRST to address this issue?

Accepted Answer

Apply L1 or L2 regularization to the model.. The scenario describes overfitting, where the model memorizes the training data but fails to generalize to unseen data. Applying L1 or L2 regularization (Option D) is the correct first step because it adds a penalty to the loss function for large weights, discouraging complexity and reducing overfitting without requiring additional data or architectural changes.

Answer

Increase the learning rate.

Answer

Add more layers to the neural network.

Answer

Increase the size of the training dataset.

Question 16

A company deploys an AI model for loan approval. The model shows bias against a protected group. The team decides to use adversarial debiasing. What is the PRIMARY advantage of this approach?

Accepted Answer

It reduces bias while preserving predictive performance by learning representations that are invariant to sensitive attributes.. Adversarial debiasing is an in-processing technique that trains a primary model to predict the target (e.g., loan approval) while simultaneously training an adversary to predict the sensitive attribute from the model's learned representations. The primary model is penalized when the adversary succeeds, forcing it to learn representations that are invariant to the sensitive attribute. This reduces bias while preserving predictive performance because the model retains the ability to learn task-relevant patterns that are not correlated with the protected attribute.

Answer

It guarantees the model's predictions are private.

Answer

It is simpler to implement than pre-processing techniques.

Answer

It ensures equal approval rates across all groups.

Question 17

An AIOps platform monitors server metrics and triggers alerts. The team notices too many false positives. Which adjustment should be made to the anomaly detection model?

Accepted Answer

Raise the anomaly score threshold for triggering alerts.. Raising the anomaly score threshold (Option D) directly reduces false positives by requiring a higher deviation from normal behavior before an alert is triggered. In AIOps platforms, the anomaly score is a numeric value (e.g., 0–100) that quantifies how unusual a metric is; a higher threshold means only more extreme deviations generate alerts, filtering out minor fluctuations that were incorrectly flagged.

Answer

Use a more complex model to better fit the data.

Answer

Shorten the observation window to detect anomalies faster.

Answer

Increase the training data to include more normal patterns.

Question 18

A team deploys a machine learning model as a REST API. They want to monitor model drift. Which metric is MOST appropriate for detecting drift in the input data distribution?

Accepted Answer

Population stability index (PSI) comparing training and recent data.. Population stability index (PSI) is the most appropriate metric for detecting drift in input data distribution because it directly measures the shift between the training data distribution and the recent production data distribution. PSI is calculated by binning both distributions and computing the sum of (proportion in bin of recent data minus proportion in bin of training data) times the natural log of their ratio, making it sensitive to changes in feature distributions without requiring ground truth labels.

Answer

Model accuracy on a recent holdout set.

Answer

F1 score on the training data.

Answer

Root mean squared error (RMSE) on test data.

Question 19

A company uses an AI system to recommend products. The recommendation accuracy is high, but users complain about lack of diversity. Which strategy should the team adopt to improve diversity without significantly sacrificing accuracy?

Accepted Answer

Modify the loss function to include a term that penalizes overly similar recommendations.. Modifying the loss function to include a diversity penalty directly addresses the lack of recommendation diversity at the algorithmic level. By adding a regularization term that penalizes overly similar recommendations, the model learns to balance accuracy with variety, ensuring that the output set remains diverse without a significant drop in relevance. This approach is a standard technique in recommendation systems, often implemented via Determinantal Point Processes (DPPs) or diversity-aware loss functions.

Answer

Randomly replace some recommendations with popular items.

Answer

Use only popularity-based recommendations.

Answer

Increase the number of recommendations and use collaborative filtering.

Question 20

A machine learning engineer is deploying a model to production. Which TWO practices are essential for ensuring reproducibility of model predictions?

Accepted Answer

Version-control the model artifact (e.g., using MLflow or DVC).. Version-controlling the model artifact (D) is essential because it allows you to reproduce the exact model binary that generated a prediction, ensuring that any changes to the model code, hyperparameters, or training data do not silently alter outputs. Tools like MLflow or DVC store the model along with its metadata, enabling rollback and auditability in production.

Answer

Increase the number of training epochs to ensure convergence.

Answer

Use the same GPU hardware for both training and inference.

Answer

Use parallel data loading to speed up inference.