AI0-001 AI Implementation and Operations Practice Test 3 — 15 Questions

Question 1

Which TWO techniques are most effective for ensuring model explainability in a production loan approval AI system subject to regulatory review? (Select TWO.)

Accepted Answer

Use SHAP values to understand feature contributions. Option B (SHAP values) is correct because SHAP provides theoretically grounded, additive feature-attribution values based on Shapley values from cooperative game theory, giving consistent and locally accurate explanations of how each input feature contributed to an individual loan decision — exactly what regulators need to justify adverse actions. Option D (LIME) is correct because it generates local, model-agnostic surrogate explanations for each individual prediction, which is essential in loan approvals where each applicant's decision must be independently explainable for compliance and adverse-action notices. Together, SHAP and LIME cover both rigorous global/local attribution and per-decision interpretability required under regulations like ECOA/Reg B and GDPR Article 22. Option A is not ideal because replacing a high-performing model with a single decision tree sacrifices predictive accuracy and may still be too complex to interpret at scale, and it is not one of the marked answers. Option C is not reliable because attention weights are not faithful explanations of model reasoning and can be misleading, so they fail regulatory scrutiny. Option E is not selected because permutation importance only gives global, aggregate feature rankings and cannot explain individual loan decisions, which is the core regulatory requirement.

Answer

Replace the model with a decision tree for transparency

Answer

Rely on the model's internal attention weights (if transformer-based)

Answer

Calculate global feature importance using permutation importance

Question 2

A large financial services company deploys multiple AI models on a shared Kubernetes cluster with GPU nodes. The models serve real-time fraud detection and credit scoring. Recently, the operations team observed frequent out-of-memory (OOM) errors during peak hours, causing inference failures. The monitoring dashboards show GPU memory utilization averaging 90% during peak times, and pods are being evicted. The team has allocated 8GB per pod and the total cluster GPU memory is 32GB. The models require at least 4GB each, but the fraud detection model occasionally spikes to 7GB. Which course of action best resolves the OOM errors while maintaining high availability?

Accepted Answer

Set resource limits and requests per model based on observed usage, and implement pod priority classes. It uses Kubernetes resource management features—setting precise resource requests and limits based on observed GPU memory usage—combined with pod priority classes to ensure critical fraud detection pods are scheduled and retained during contention. This prevents OOM errors by capping memory per pod while allowing the spike-prone fraud model to be prioritized over less critical workloads, maintaining high availability without overprovisioning.

Answer

Reduce the batch size and model complexity for all models to lower memory footprint

Answer

Provision larger GPU nodes with 48GB memory each

Answer

Increase the memory request for all pods to 8GB to ensure they have enough

Question 3

A company deploys a machine learning model for fraud detection. After one month, the false positive rate has increased significantly. The model is retrained weekly on all historical data. What is the MOST effective immediate action?

Accepted Answer

Retrain the model on only the most recent 30 days of data.. The false positive rate increase suggests the model is reacting to a shift in the underlying data distribution (concept drift). Retraining on only the most recent 30 days of data (option D) is the most effective immediate action because it focuses the model on the current fraud patterns, discarding stale historical data that may no longer be representative. This approach directly addresses the drift by adapting the model to the latest behavior.

Answer

Replace the model with a simpler logistic regression model.

Answer

Continue retraining weekly on all historical data.

Answer

Adjust the classification threshold to reduce false positives.

Question 4

A retail company uses a time-series model to forecast daily sales for inventory management. The model is a seasonal ARIMA trained on three years of daily data. It performed well during initial validation but after deployment, forecasts became inaccurate during holiday seasons, often underestimating demand by up to 40%. The data science team examined the features and found that the training data did not include any holiday indicators. They also discovered that the model's residuals show strong autocorrelation during holiday weeks. The company needs to improve the forecast for the upcoming holiday season. They have access to historical sales data with holiday dates and are considering several approaches. Which approach will BEST address the issue?

Accepted Answer

Engineer holiday-related features (e.g., holiday flag, days before/after) and retrain the ARIMA model with these features as exogenous variables.. The core issue is that the ARIMA model lacks holiday-related information, causing systematic underestimation during holiday periods. By engineering holiday features (e.g., binary flags, days-before/after indicators) and including them as exogenous variables in a SARIMAX model, the model can directly learn the demand spikes associated with holidays. This directly addresses the residual autocorrelation during holiday weeks and improves forecast accuracy without changing the underlying time-series framework.

Answer

Decrease the learning rate and increase the number of training epochs.

Answer

Create an ensemble of the ARIMA model and a seasonal naive model to average forecasts.

Answer

Replace the ARIMA model with a deep learning LSTM model trained on the same data.

Question 5

An insurance company runs a claims-triage model in production. Over four months, the input feature distribution for 'claim_amount' has drifted upward while the model's AUC remains stable at 0.86. The operations team wants to know whether to retrain. Which monitoring signal provides the MOST direct evidence that retraining is warranted?

Accepted Answer

A rising population stability index (PSI) on claim_amount combined with a measurable drop in business KPIs such as claims-handling time.. Retraining decisions should combine a drift signal on inputs with evidence that model outputs no longer serve business objectives. PSI quantifies how far the claim_amount distribution moved, and the decline in claims-handling time shows real-world harm. Input drift alone, latency changes, static holdout AUC, or missing-value shifts do not demonstrate that the model's decisions have degraded, so they cannot independently justify retraining.

Answer

A statistically significant increase in the model's inference latency across the same period.

Answer

A stable AUC of 0.86 measured against the historical holdout test set.

Answer

An increase in the number of features whose missing-value rate changed by more than two percent.

Question 6

An AI system misclassifies rare but critical events. The team considers using synthetic data. Which consideration is MOST important for ensuring the synthetic data improves performance on real rare events?

Accepted Answer

The synthetic data should accurately represent the distribution and features of real rare events.. Synthetic data must faithfully replicate the distribution and feature space of real rare events to enable the model to learn meaningful decision boundaries. If the synthetic data does not capture the true underlying patterns—such as specific sensor readings or transaction anomalies—the model will fail to generalize to actual rare events, defeating the purpose of augmentation.

Answer

The synthetic data should include a wide variety of events, even if not realistic.

Answer

The synthetic data should be generated using an unsupervised generative model.

Answer

The synthetic data should be as large as possible to cover all possibilities.

Question 7

An AIOps platform monitors server metrics and triggers alerts. The team notices too many false positives. Which adjustment should be made to the anomaly detection model?

Accepted Answer

Raise the anomaly score threshold for triggering alerts.. Raising the anomaly score threshold (Option D) directly reduces false positives by requiring a higher deviation from normal behavior before an alert is triggered. In AIOps platforms, the anomaly score is a numeric value (e.g., 0–100) that quantifies how unusual a metric is; a higher threshold means only more extreme deviations generate alerts, filtering out minor fluctuations that were incorrectly flagged.

Answer

Use a more complex model to better fit the data.

Answer

Shorten the observation window to detect anomalies faster.

Answer

Increase the training data to include more normal patterns.

Question 8

A team deploys a machine learning model as a REST API. They want to monitor model drift. Which metric is MOST appropriate for detecting drift in the input data distribution?

Accepted Answer

Population stability index (PSI) comparing training and recent data.. Population stability index (PSI) is the most appropriate metric for detecting drift in input data distribution because it directly measures the shift between the training data distribution and the recent production data distribution. PSI is calculated by binning both distributions and computing the sum of (proportion in bin of recent data minus proportion in bin of training data) times the natural log of their ratio, making it sensitive to changes in feature distributions without requiring ground truth labels.

Answer

Model accuracy on a recent holdout set.

Answer

F1 score on the training data.

Answer

Root mean squared error (RMSE) on test data.

Question 9

A machine learning engineer is deploying a model to production. Which TWO practices are essential for ensuring reproducibility of model predictions?

Accepted Answer

Version-control the model artifact (e.g., using MLflow or DVC).. Option D is correct because version-controlling the model artifact with tools like MLflow or DVC ensures that the exact trained weights, hyperparameters, and code revision used in production can be retrieved and audited, which is fundamental to reproducing identical predictions. Option E is correct because fixing random seeds across all libraries (e.g., NumPy, TensorFlow, and Python's random module) eliminates nondeterminism in weight initialization, data shuffling, and dropout, so the same inputs yield the same outputs. Option A is not essential for reproducibility since more epochs change the model rather than guarantee deterministic, repeatable predictions. Option B is unnecessary because reproducibility depends on deterministic software and versioned artifacts, not on identical GPU hardware, and inference can run on different accelerators. Option C is also irrelevant because parallel data loading affects throughput and latency, not the determinism or reproducibility of the model's predictions.

Answer

Increase the number of training epochs to ensure convergence.

Answer

Use the same GPU hardware for both training and inference.

Answer

Use parallel data loading to speed up inference.

Question 10

An organization is implementing an AI governance framework. Which THREE components are essential for compliance with ethical AI standards?

Accepted Answer

Data privacy protection measures (e.g., differential privacy).. Option A (data privacy protection measures such as differential privacy) is essential because ethical AI standards require safeguarding personal data, and techniques like differential privacy provide formal, quantifiable guarantees that individual records cannot be inferred from model outputs, supporting regulations like GDPR. Option D (model explainability and interpretability mechanisms) is essential because stakeholders must understand how decisions are reached; methods such as SHAP, LIME, or inherently interpretable models enable accountability and contestability required by ethical frameworks. Option E (regular bias auditing of models) is essential because systematic auditing detects disparate impact across protected groups, using metrics like demographic parity or equalized odds, ensuring fairness is continuously monitored rather than assumed. Option B is not required because ethical compliance concerns how models are governed and used, not whether they are open-sourced; proprietary models can be fully ethical. Option C is incorrect because maximizing accuracy for revenue is a business objective, not an ethical compliance requirement, and can even conflict with fairness and privacy goals.

Answer

Open-source licensing of all models.

Answer

Maximizing model accuracy to increase revenue.

Question 11

A team trained a ResNet-50 model with the configuration shown. The high training accuracy and lower validation accuracy suggest overfitting. Which change to the training configuration is MOST likely to reduce overfitting?

Accepted Answer

Add dropout layers after convolutional layers.. Adding dropout layers after convolutional layers is a regularization technique that randomly drops a fraction of neurons during training, which forces the network to learn more robust features and reduces overfitting. This directly addresses the symptom of high training accuracy with lower validation accuracy by preventing the model from relying too heavily on specific neurons.

Answer

Reduce number of epochs to 5.

Answer

Increase batch size to 64.

Answer

Increase learning rate to 0.01.

Question 12

A company deploys a computer vision model for quality inspection on a manufacturing line. After deployment, the model's accuracy drops from 95% to 80% over two weeks. Which action is most likely to address this issue?

Accepted Answer

Retrain the model using recently collected production data.. The accuracy drop over two weeks indicates data drift or concept drift, where the production data distribution changes over time. Retraining the model with recently collected production data realigns it with the current data distribution, directly addressing the drift. Option B (increasing confidence threshold) may reduce false positives but does not fix the underlying drift and could lower recall. Option C (decreasing learning rate) is irrelevant for inference; it only affects training and cannot be applied post-deployment to fix drift. Option D (deploying an ensemble) adds computational overhead and does not resolve drift; it might even mask the issue without correcting it.

Answer

Increase the confidence threshold for predictions.

Answer

Decrease the learning rate of the training algorithm.

Answer

Deploy an additional ensemble of models for redundancy.

Question 13

An organization is implementing an AI-powered chatbot for customer service. The chatbot must comply with GDPR and handle data subject access requests (DSARs). Which design approach best ensures compliance?

Accepted Answer

Implement an audit trail that logs interactions with a unique user identifier, and provide a mechanism to delete logs upon user request.. GDPR requires that personal data be stored only as long as necessary and that data subjects have the right to erasure. By logging interactions with a unique user identifier and providing a deletion mechanism, the chatbot can fulfill DSARs while maintaining an audit trail for compliance monitoring. This approach balances operational needs with regulatory obligations.

Answer

Minimize data collection by not logging any user interactions.

Answer

Anonymize all user data before logging interactions.

Answer

Encrypt all chat logs and store them indefinitely for audit purposes.

Question 14

A data scientist fine-tunes a large language model for a legal document summarization task. After fine-tuning, the model performs well on test data but produces summaries that include hallucinated legal clauses. Which mitigation strategy is most effective?

Accepted Answer

Implement retrieval-augmented generation (RAG) to provide factual context.. Implement retrieval-augmented generation (RAG) to provide factual context. RAG reduces hallucinations by allowing the model to retrieve relevant, factual information from an external knowledge base during generation, grounding its output in verified data. Option A (different tokenizer) does not address the core issue of factual accuracy. Option B (decrease temperature) affects randomness but does not prevent the model from fabricating content. Option D (max token limit) truncates output but does not stop the model from including false information within that limit.

Answer

Use a different tokenizer during fine-tuning.

Answer

Decrease the temperature parameter to 0.1 during inference.

Answer

Set a maximum token limit of 50 for each summary.

Question 15

A team deploys a real-time fraud detection model on a streaming platform. The model must produce predictions within 100 milliseconds per event. Initial latency is 150 ms. Which optimization is most likely to meet the latency requirement?

Accepted Answer

Apply model quantization to reduce precision from FP32 to INT8.. Model quantization reduces the numerical precision of the model's weights and activations from FP32 to INT8, which decreases memory footprint and speeds up inference. This optimization directly addresses the 150 ms latency by enabling faster arithmetic operations on modern hardware, often cutting inference time by 2-4x, which can bring latency below the 100 ms requirement.

Answer

Increase the batch size to process more events simultaneously.

Answer

Add more feature engineering steps to improve model accuracy.

Answer

Migrate from a decision tree ensemble to a deep neural network.