AI0-001 AI Implementation and Operations Practice Test 7 — 15 Questions

Question 1

An organization deploys an AI model on edge devices for real-time image classification. Which metric is most important to monitor for ensuring the device's operational health?

Accepted Answer

Inference memory consumption. For edge devices with limited resources, inference memory consumption is the most critical operational health metric because exceeding available memory can cause the model to crash or the device to become unresponsive. Unlike accuracy or confidence, memory usage directly reflects whether the device can sustain real-time inference without resource exhaustion.

Answer

Model calibration error

Answer

Average prediction confidence

Answer

Model accuracy on local test data

Question 2

Which TWO are best practices for deploying AI models in a containerized production environment? (Select TWO.)

Accepted Answer

Use an orchestration platform like Kubernetes for scaling and health management. Option C is correct because Kubernetes (or a similar orchestrator) provides horizontal pod autoscaling, liveness/readiness probes, and rolling updates, which are essential for reliably scaling and health-managing AI inference services in production. Option D is correct because packaging the model together with its runtime dependencies (libraries, CUDA/cuDNN versions, Python packages) into a single immutable container image guarantees reproducible, portable deployments across environments. Option A is not a best practice because pulling the 'latest' tag yields non-deterministic, unreproducible builds and can silently introduce breaking changes; images should be pinned to immutable version or digest tags. Option B is not recommended because baking large model artifacts into the image bloats it, slows pulls, and forces a full image rebuild for every model update; models are better mounted from object storage or a model registry. Option E is not generally applicable since most AI/ML containers are Python-based rather than JVM-based, and heap tuning is workload-specific rather than a universal deployment best practice.

Answer

Always pull the latest image tag for automatic updates

Answer

Store model artifacts inside the container image for portability

Answer

Configure JVM heap arguments inside the container if using Java

Question 3

Which THREE are common pitfalls when operationalizing AI models? (Select THREE.)

Accepted Answer

Training-serving skew due to differences in data preprocessing. Option A is correct because training-serving skew occurs when the preprocessing, feature engineering, or transformations applied during training differ from those applied at inference time, causing the model to receive inputs that do not match its learned distribution and degrading predictions. Option C is correct because deployed models degrade over time due to data drift, concept drift, and changing user behavior, so without monitoring for performance drift (e.g., tracking accuracy, latency, and input distributions) failures go undetected. Option D is correct because operationalizing AI requires serving infrastructure that can handle production traffic, scaling, and latency requirements; ignoring scalability leads to outages or unacceptable response times under load. Option B is not a pitfall but often a deliberate, sound engineering choice, since simpler models are easier to debug, maintain, and explain. Option E is not a pitfall either; automating retraining is a recommended MLOps practice that helps keep models current, provided it is paired with validation and monitoring.

Answer

Using simpler models that are easier to debug

Answer

Automating the model retraining process

Question 4

An e-commerce company uses a machine learning model to recommend products to users. The model is retrained weekly and deployed to production. For the past three weeks, the model's click-through rate (CTR) has been stable except on Mondays, when it drops by 15%. Analysis reveals that the training data is extracted on Sundays and includes only weekday behavior. On Mondays, user behavior shifts due to weekend browsing patterns not captured in the training data. The team wants to maintain a weekly retraining cadence but fix the Monday performance drop. Which solution best addresses the Monday CTR drop without changing the retraining frequency?

Accepted Answer

Modify the data pipeline to include the full week (including the past weekend) in each retraining. It directly addresses the root cause: the training data excludes weekend behavior, causing the model to be blind to Monday patterns. By modifying the data pipeline to include the full week (including the past weekend) in each retraining, the model learns from weekend browsing patterns and can generalize to Monday user behavior without changing the weekly retraining cadence. This ensures the training distribution matches the inference distribution on Mondays, stabilizing CTR.

Answer

Deploy a separate model specifically for Monday predictions

Answer

Serve the previous week's model on Mondays to use older but stable patterns

Answer

Change to daily retraining to include weekend data more promptly

Question 5

A startup has developed a natural language processing model for sentiment analysis. Their CI/CD pipeline includes a step that runs unit tests on the model's output format and a validation step that checks accuracy on a static test dataset. Recently, the pipeline often fails during the validation step, but the failures are inconsistent—sometimes the same model version passes, sometimes fails. The team suspects the test dataset is small and randomly sampled. They need a reliable validation process to deploy models with confidence. Which approach should the team implement?

Accepted Answer

Fix the test dataset to be larger and more representative, and use a statistical test to compare against baseline. The core issue is that the static test dataset is too small and randomly sampled, leading to inconsistent validation results. By fixing the dataset to be larger and more representative, and using a statistical test (e.g., a paired t-test or McNemar's test) to compare the model's accuracy against a baseline, the team can reliably determine if performance changes are statistically significant, eliminating the randomness that causes pipeline failures to be inconsistent.

Answer

Replace the static test set with k-fold cross-validation in each pipeline run

Answer

Increase the accuracy threshold to 95% so only very good models pass

Answer

Remove the validation step and rely on unit tests only

Question 6

A data scientist is monitoring a deployed image classification model. Which TWO actions are best practices for detecting model drift? (Choose 2.)

Accepted Answer

Use a holdout test set to periodically evaluate model accuracy.. Option C is correct because periodically scoring the model against a fixed holdout test set gives a direct, repeatable measurement of accuracy degradation, which is the core signal of model drift. Option E is correct because tracking the distribution of input features over time (e.g., via histograms or statistical tests like KL divergence or PSI) detects data drift, which typically precedes and causes model drift. Option A is not a detection practice but a remediation action, and retraining blindly on a schedule does not identify drift. Option B is a modeling change that may affect generalization but does nothing to detect drift in a deployed model. Option D is weaker than C and E because average prediction confidence can shift for reasons unrelated to drift and is not a reliable standalone drift indicator.

Answer

Schedule automatic weekly retraining of the model.

Answer

Increase the model's complexity to improve generalization.

Answer

Monitor the average prediction confidence of the model.

Question 7

A deployed NLP sentiment analysis model experiences a sharp decline in accuracy on customer reviews. The team has verified the input data format and pipeline are correct. Which THREE actions should be taken to diagnose and remediate? (Choose 3.)

Accepted Answer

Analyze recent user input for distribution shifts compared to training data.. Option A is correct because when a deployed NLP model's accuracy drops while the input format and pipeline are verified as correct, the most likely cause is data drift — the statistical distribution of recent user input has shifted away from the training distribution, so comparing recent inputs against training data (e.g., via feature/token distributions, embeddings, or KS tests) is the proper first diagnostic step. Option D is correct because reverting to a previously well-performing model version is a valid remediation and diagnostic tactic: it restores service quality immediately and, if the older version still performs well on the new data, it confirms the regression was introduced by the newer model rather than by the data itself. Option E is correct because concept drift — a change in the relationship between inputs and the target label (e.g., sentiment words taking on new meaning) — is a leading cause of accuracy decay in production NLP models, so a structured root cause analysis targeting concept drift (and distinguishing it from data drift) is essential to select the right fix. Option B is not appropriate as stated because blindly retraining with all available data, including potentially mislabeled or drifted recent data, can propagate the problem and does not diagnose the cause. Option C is not appropriate because adding synthetic data addresses data scarcity, not the verified accuracy decline, and synthetic data can introduce its own distributional biases without first identifying the root cause.

Answer

Immediately retrain the model with all available data.

Answer

Increase the size of the training dataset by adding synthetic data.

Question 8

A large e-commerce company has deployed a real-time product recommendation system using a neural collaborative filtering model. The model was trained on six months of user click and purchase data. For the first three months after deployment, the click-through rate (CTR) improved by 15%. However, starting in the fourth month, CTR began decreasing steadily despite no changes to the system infrastructure or data pipeline. The product manager suspects model decay but the engineering team insists the model is static and should not degrade. The data science lead suggests investigating further. They have access to production logs, A/B testing framework, and historical model versions. What is the BEST course of action to diagnose and address the issue?

Accepted Answer

Set up an A/B test comparing the current model against the original baseline model using recent traffic.. Setting up an A/B test comparing the current model against the original baseline model using recent traffic directly isolates whether the model's predictive performance has degraded due to concept drift (changes in user behavior over time). Since the model is static but the data distribution has shifted, the A/B test provides empirical evidence of decay by measuring CTR differences under identical conditions, which is the standard diagnostic step before any retraining or feature engineering.

Answer

Re-deploy the model with additional features such as time of day and user device.

Answer

Increase the frequency of batch inference from hourly to every 10 minutes to improve responsiveness.

Answer

Retrain the model on only the most recent 30 days of data and replace the current model.

Question 9

A financial institution uses a machine learning model to approve personal loans. The model was trained on historical data that includes applicant age, income, credit score, and loan amount. Compliance officers have received customer complaints suggesting the model may be discriminating against applicants over 60 years old. Initial analysis shows that the approval rate for applicants over 60 is 20 percentage points lower than for younger applicants with similar credit profiles. The data science team has been asked to investigate and remediate any bias. They have access to the training data, model coefficients, and can retrain or modify the model. What is the FIRST step the team should take?

Accepted Answer

Conduct a fairness audit using appropriate metrics such as disparate impact ratio on the current model.. The first step in addressing potential bias is to conduct a fairness audit using established metrics like the disparate impact ratio (e.g., the 80% rule from the US Equal Employment Opportunity Commission). This quantifies whether the model's approval rate for applicants over 60 is less than 80% of the rate for the younger group, providing a legally and technically sound baseline before any remediation. Without this measurement, any subsequent changes (like resampling or removing features) could be misguided or ineffective.

Answer

Replace the model with a third-party vendor model that claims to be bias-free.

Answer

Re-sample the training data to have equal numbers of applicants over and under 60.

Answer

Remove the age feature from the training data and retrain the model.

Question 10

A small business launched a customer support chatbot powered by a pre-trained language model. The chatbot was fine-tuned on a dataset of past support tickets. For the first week, it performed well, accurately answering 85% of queries. After a routine software update that included a new version of the underlying language model library, the chatbot's accuracy dropped to 60% and it began giving nonsensical responses to some questions. The update did not change any code or configuration specific to the chatbot. The business has a backup of the previous environment. What is the MOST appropriate immediate action?

Accepted Answer

Roll back the software update to the previous version of the language model library.. The most appropriate immediate action is to roll back the software update to the previous version of the language model library (Option D). The accuracy drop and nonsensical responses are directly caused by the library update, which likely changed internal model behavior (e.g., tokenization, attention mechanisms, or default hyperparameters) without any code or configuration changes. Restoring the previous environment immediately resolves the issue and allows the business to investigate the library changes in a controlled manner.

Answer

Retrain the chatbot on the original dataset using the new library version.

Answer

Add more intents to the chatbot's configuration to cover the errors.

Answer

Increase the model's temperature parameter to 1.5 to encourage more varied responses.

Question 11

A healthcare AI startup has developed a model to detect diabetic retinopathy from retinal images. The model achieved 96% sensitivity and 94% specificity on a validation set from the same distribution as the training data. After deployment in a rural clinic, the model's sensitivity drops to 80%. The data team analyzes the clinical images from the clinic and finds that the images have lower resolution and different lighting conditions compared to the training dataset. The team has the ability to collect more data from the clinic and retrain the model. What is the BEST course of action?

Accepted Answer

Collect additional retinal images from the rural clinic, label them, and retrain the model including the new data.. The performance drop is caused by a domain shift (lower resolution, different lighting) between the training and deployment data. The most direct and effective solution is to collect labeled images from the target domain (rural clinic) and retrain the model, which aligns with the principle of domain adaptation through data augmentation. This approach addresses the root cause by exposing the model to the actual distribution it will encounter in production.

Answer

Reduce the model's complexity by removing several convolutional layers to improve generalization.

Answer

Apply transfer learning using a model pre-trained on a different medical imaging dataset.

Answer

Implement adversarial validation to identify which images are out-of-distribution and filter them out.

Question 12

A financial services firm has deployed an AI model for real-time credit scoring. The operations team needs to ensure the model remains reliable and compliant over time. Which TWO actions should the team prioritize? (Choose two.)

Accepted Answer

Implement automated monitoring for data drift and model performance metrics.. Automated monitoring for data drift and model performance metrics is essential for maintaining reliability and compliance in a real-time credit scoring system. Data drift detection (e.g., using population stability index or KL divergence) alerts the team when input distributions shift, which could degrade model accuracy and lead to non-compliant decisions. Continuous monitoring of metrics like AUC, precision, and recall ensures the model stays within regulatory thresholds without manual intervention.

Answer

Deploy a model versioning system with automated rollback capabilities.

Answer

Schedule monthly manual retraining of the model using historical data.

Answer

Generate weekly compliance reports for regulatory review.

Question 13

The exhibit shows the output of a drift monitoring command for a fraud detection model. The team has an automated pipeline that triggers retraining when the overall average drift score exceeds 0.10. Based on the exhibit, what should the operations team do next?

Accepted Answer

Manually analyze the drift in 'amount' and 'location' and investigate potential causes.. The exhibit shows that the overall average drift score is below 0.10, so the automated retraining pipeline should not trigger. However, individual features like 'amount' and 'location' show elevated drift values that warrant manual investigation to understand root causes before any retraining decision. The team should analyze these specific features to determine if the drift is due to genuine data distribution changes or data quality issues.

Answer

Force retraining on all features to ensure the model adapts to the new data distribution.

Answer

No action is needed because the model is performing within acceptable drift limits.

Answer

Initiate the automated retraining pipeline since the average drift exceeds 0.05.

Question 14

A hospital's radiology department uses an AI model to detect lung nodules in CT scans. The model was trained on data from a specific brand of scanners and patient demographics common in Europe. Recently, the hospital acquired new scanners from a different manufacturer and started serving a more diverse patient population. Over the past month, the model's false-positive rate has increased by 15% and false-negative rate by 8%. The radiologists are losing confidence and are considering abandoning the AI tool altogether. The IT team has verified that the model inference is running correctly and the hardware is performing as expected. The data science team suspects the problem is related to the change in input data distribution. The hospital's AI operations policy requires that any model update must be validated on at least 500 recent cases before deployment. What is the BEST course of action for the AI operations team?

Accepted Answer

Collect 500 recent CT scans from the new scanners, retrain the model on a combined old and new dataset, and validate before deployment.. The model's performance degradation is likely due to data drift: the input data distribution has changed because of new scanners and a more diverse patient population. The best course is to collect recent data representative of the new distribution, retrain the model on a combined dataset (old and new), and validate on at least 500 recent cases as per policy. This addresses the root cause and ensures the model generalizes to the new data.

Answer

Roll back to the previous model version and restrict use of the AI tool to only European patients.

Answer

Retrain the model using the original training data but with increased regularization to avoid overfitting.

Answer

Adjust the model's decision threshold to reduce false positives and then monitor for two weeks.

Question 15

A financial services firm runs a real-time credit-scoring model on an Amazon SageMaker endpoint. The model must not degrade: the team needs automatic detection of distributional drift in the incoming feature data and an alert when drift exceeds a threshold, without retraining the model. Which SageMaker capability should they configure?

Accepted Answer

SageMaker Model Monitor with a data quality baseline and drift detection schedule. Model Monitor is designed to continuously evaluate endpoint input data against a statistical baseline and raise CloudWatch alarms when drift exceeds configured thresholds. It inspects live inference requests without modifying the model, satisfying the need for automatic distributional drift detection and alerting. The other services either retrain, tune, or explain the model rather than monitor production feature drift.

Answer

SageMaker Pipelines with a conditional step that retrains the model

Answer

SageMaker Automatic Model Tuning with a hyperparameter search job

Answer

SageMaker Clarify with a bias configuration on the endpoint