AI0-001 AI Implementation and Operations Practice Test 4 — 15 Questions

Question 1

A DevOps team is deploying a machine learning model using a CI/CD pipeline. They want to ensure the model is reproducible and traceable. Which TWO practices should they implement?

Accepted Answer

Version the training dataset and code using Git and DVC.. Option A is correct because versioning both the training dataset and the code with Git and DVC (Data Version Control) captures the exact data and source revisions used, which is essential for reproducing and tracing a model. Option E is correct because packaging the model in a Docker container with a fixed (pinned) base image locks down the OS libraries, dependencies, and runtime environment, ensuring the model behaves identically across environments and builds. Option B is not appropriate because manual deployment after approval is error-prone and not automated or traceable, undermining CI/CD reproducibility. Option C is wrong because storing only the final model artifact in a shared drive loses the training data, code, and environment context needed for reproducibility. Option D is wrong because a spreadsheet is a manual, non-versioned record that cannot reliably or automatically trace model versions.

Answer

Manually deploy the model to production after approval.

Answer

Store only the final model artifact in a shared drive.

Answer

Use a spreadsheet to record model version numbers.

Question 2

An AI operations team is monitoring a deployed image classification model. They notice a gradual increase in prediction confidence but a drop in accuracy. Which THREE actions should they take to diagnose the issue?

Accepted Answer

Analyze the model's calibration curve to see if confidence scores align with actual accuracy.. Option A is correct because a calibration curve (reliability diagram) directly compares predicted confidence against observed accuracy, revealing whether the model has become overconfident — exactly the symptom of rising confidence with falling accuracy. Option C is correct because comparing training versus production input feature distributions (e.g., via drift metrics like PSI or KL divergence) detects data drift or covariate shift that can degrade accuracy while inflating confidence. Option D is correct because evaluating on a held-out test set collected at deployment time establishes a stable baseline, distinguishing genuine model degradation from changes in the production data distribution. Option B is not appropriate because collecting unlabeled data does not diagnose the cause and cannot be used for supervised evaluation without labels. Option E is not appropriate because immediate retraining without root-cause analysis risks masking the problem and may reinforce the drift or labeling issues causing the confidence-accuracy mismatch.

Answer

Increase the size of the training dataset by collecting more unlabeled data.

Answer

Retrain the model immediately with the most recent data.

Question 3

A company deployed a machine learning model on a cloud inference service. Users report high latency during peak hours. The model is deployed on a single instance. Which action should the team take to reduce latency without significant architectural changes?

Accepted Answer

Enable autoscaling for the inference instances. Enabling autoscaling for the inference instances directly addresses the root cause: a single instance cannot handle peak-hour traffic, causing queuing and high latency. Autoscaling horizontally adds more instances to distribute the load, reducing per-request response time without re-architecting the system. This is a standard cloud-native pattern for stateless inference services and requires minimal configuration change.

Answer

Increase the model size to improve accuracy

Answer

Switch to a batch inference pipeline

Answer

Add an API gateway to route requests

Question 4

A data science team uses Git for version control of model code and DVC for data versioning. They want to implement a model registry to track trained models, their hyperparameters, and performance metrics. Which tool is specifically designed for this purpose and integrates with the existing workflow?

Accepted Answer

MLflow Model Registry. MLflow Model Registry is specifically designed for managing model versions, tracking metadata, and integrating with Git and DVC. Apache Airflow is for workflow orchestration, not model registry. Kubernetes is for container orchestration. Docker is for containerization.

Answer

Apache Airflow

Answer

Docker

Answer

Kubernetes

Question 5

An ML team monitors a production model using a dashboard that shows daily performance metrics. Over the past month, the model's accuracy has dropped from 92% to 87%, while the data distribution of input features has remained stable according to statistical tests. Which type of model drift is most likely occurring?

Accepted Answer

Concept drift. Concept drift occurs when the relationship between input features and the target variable changes, even if the input data distribution remains stable. In this scenario, the model's accuracy declines from 92% to 87% while input feature distributions are unchanged, indicating that the underlying mapping from features to labels has shifted—a classic sign of concept drift.

Answer

Data drift (covariate shift)

Answer

Model decay

Answer

Overfitting

Question 6

A company must deploy a new model version with zero downtime. The current model is served via a REST API on a Kubernetes cluster. Which deployment strategy should the team use to gradually shift traffic to the new version while monitoring for errors?

Accepted Answer

Canary deployment. A canary deployment gradually shifts a small percentage of traffic to the new model version while the majority continues to hit the stable version. This allows the team to monitor for errors and roll back quickly if issues arise, achieving zero downtime. It is the ideal strategy for validating a new model in production with minimal risk.

Answer

Blue-green deployment

Answer

Recreate deployment

Answer

Rolling update

Question 7

An ML engineering team has a retraining pipeline that triggers automatically when model accuracy drops below a threshold. Recently, the model's accuracy has been fluctuating, causing frequent retraining and high compute costs. The team suspects the data distribution is changing slowly. Which approach should the team implement to reduce unnecessary retraining while maintaining model performance?

Accepted Answer

Implement a statistical drift detection method on input features. Implementing a statistical drift detection method (e.g., using KL divergence, PSI, or ADWIN) on input features allows the team to identify when the data distribution has genuinely changed, rather than reacting to random accuracy fluctuations. This reduces unnecessary retraining by triggering the pipeline only when statistically significant drift is detected, maintaining model performance without the high compute costs of frequent retraining.

Answer

Use a simpler model to reduce variability

Answer

Increase the frequency of model retraining

Answer

Reduce the batch size for inference

Question 8

A company deploys an AI model via a REST API that handles sensitive customer data. To secure the endpoint, the security team requires that only authenticated and authorized applications can invoke the API. Which mechanism should be implemented?

Accepted Answer

API key or bearer token in the HTTP header. API keys or bearer tokens (e.g., OAuth 2.0 access tokens) are the standard mechanism for authenticating and authorizing client applications when invoking a REST API. These tokens are passed in the HTTP Authorization header, allowing the server to verify the client's identity and permissions before processing requests containing sensitive customer data.

Answer

TLS encryption for the connection

Answer

Input sanitization to prevent injection

Answer

IP whitelisting

Question 9

An ML operations team needs to monitor a deployed model's performance. Which TWO metrics are most useful for detecting concept drift in a regression model? (Choose two.)

Accepted Answer

Distribution of residuals between predictions and actuals. Option B is correct because the distribution of residuals between predictions and actuals directly reveals concept drift: if the relationship between inputs and target changes, the residual distribution will shift (e.g., become biased or wider) even when input features look unchanged. Option E is correct because tracking MAE over a sliding time window is a standard regression performance monitor; a sustained increase in MAE relative to a baseline indicates that the model's learned mapping no longer matches the current data-generating process, which is the practical signature of concept drift. Option A is not the best choice here because input feature distribution shifts indicate data drift (covariate shift), not concept drift, and inputs can drift without the input-target relationship changing. Option C is wrong because classification accuracy applies to classification models, not regression. Option D is wrong because inference latency is an operational/system metric and says nothing about changes in the input-target relationship.

Answer

Distribution of input features

Answer

Classification accuracy

Answer

Model inference latency

Question 10

A model serving pod is failing with OOMKilled. What is the most likely cause?

Accepted Answer

The model requires more memory than the 2Gi limit. An OOMKilled error in Kubernetes indicates that a container exceeded its memory limit and was terminated by the Out Of Memory (OOM) killer. The most common cause is that the model's inference or training workload requires more memory than the configured resource limit (e.g., 2Gi), forcing the kernel to kill the process. This is a direct result of the container's memory request/limit mismatch with the actual consumption.

Answer

The container image is corrupted

Answer

The model version is outdated

Answer

The Kubernetes cluster has run out of disk space

Question 11

A data scientist is deploying a machine learning model to production. The model was trained on an imbalanced dataset. Which technique should be used during deployment to mitigate bias without retraining the model?

Accepted Answer

Apply post-processing calibration to adjust decision thresholds. Post-processing calibration adjusts the decision threshold of the model to account for the class imbalance present in the training data. This technique modifies the output probabilities or classification boundary without requiring access to the original training data or retraining the model, making it suitable for deployment scenarios where the model is already fixed.

Answer

Use an ensemble of models trained on balanced subsets

Answer

Rebalance the dataset using SMOTE before inference

Answer

Remove sensitive features from the input data

Question 12

A company uses a large language model (LLM) to generate customer support responses. They notice the model sometimes produces harmful outputs. Which implementation strategy best reduces this risk while maintaining performance?

Accepted Answer

Fine-tune the model using reinforcement learning from human feedback. Reinforcement learning from human feedback (RLHF) directly trains the model to align its outputs with human preferences for safety and helpfulness, reducing harmful outputs while preserving performance. Unlike superficial filters or prompts, RLHF adjusts the model's internal behavior through reward modeling and policy optimization, making it the most effective strategy for sustained safety improvements.

Answer

Implement a keyword-based output filter

Answer

Use a smaller, less capable model

Answer

Add system prompts instructing the model to be safe

Question 13

During an AI model deployment, the operations team notices that inference requests are taking longer than expected. Which component is most likely causing the bottleneck?

Accepted Answer

The machine learning model's size and architecture. The machine learning model's size and architecture directly determine the computational complexity of inference. Larger models with more parameters or deeper architectures require more matrix multiplications and memory bandwidth, which increases latency per request. This is the most common bottleneck in AI deployment because the model itself is the core computation unit, and its inference time scales with its complexity.

Answer

Input data preprocessing pipeline

Answer

API gateway rate limiting

Answer

Database connection pool size

Question 14

A financial institution deploys an AI credit scoring model. After six months, the model's performance drops significantly. Analysis shows that the relationship between features and labels has changed. Which term describes this phenomenon?

Accepted Answer

Concept drift. Concept drift occurs when the statistical relationship between input features and the target label changes over time, which is exactly what happened when the credit scoring model's performance dropped due to a shift in the feature-label relationship. This is distinct from data drift, which only involves changes in the input data distribution without affecting the label mapping.

Answer

Model decay

Answer

Overfitting

Answer

Data drift

Question 15

An organization is deploying an AI model on edge devices with limited computational resources. Which model optimization technique is most appropriate?

Accepted Answer

Apply model quantization. Model quantization reduces the precision of the model's weights and activations (e.g., from 32-bit floating point to 8-bit integer), which significantly decreases memory footprint and computational requirements. This makes it ideal for deployment on edge devices with limited resources, as it enables faster inference with minimal accuracy loss.

Answer

Perform additional feature engineering

Answer

Use an ensemble of models

Answer

Increase the training dataset size