AI0-001 AI Implementation and Operations Practice Test 4 — 15 Questions

Question 1

An AI system experiences degraded accuracy over time due to changes in user behavior. Which monitoring metric should be prioritized to detect this issue earliest?

Accepted Answer

Data drift detection on input features. Data drift detection monitors changes in input distribution, which often precedes accuracy drop. Option A is wrong because accuracy is a lagging indicator. Option C is wrong because latency doesn't reflect data shift. Option D is wrong because AUC is also lagging.

Answer

API response latency

Answer

Area under the ROC curve (AUC)

Answer

Model accuracy on a holdout validation set

Question 2

A company uses a large language model (LLM) to generate customer support responses. They notice the model sometimes produces harmful outputs. Which implementation strategy best reduces this risk while maintaining performance?

Accepted Answer

Fine-tune the model using reinforcement learning from human feedback. Reinforcement learning from human feedback (RLHF) directly trains the model to align its outputs with human preferences for safety and helpfulness, reducing harmful outputs while preserving performance. Unlike superficial filters or prompts, RLHF adjusts the model's internal behavior through reward modeling and policy optimization, making it the most effective strategy for sustained safety improvements.

Answer

Implement a keyword-based output filter

Answer

Use a smaller, less capable model

Answer

Add system prompts instructing the model to be safe

Question 3

During an AI model deployment, the operations team notices that inference requests are taking longer than expected. Which component is most likely causing the bottleneck?

Accepted Answer

The machine learning model's size and architecture. The machine learning model's size and architecture directly determine the computational complexity of inference. Larger models with more parameters or deeper architectures require more matrix multiplications and memory bandwidth, which increases latency per request. This is the most common bottleneck in AI deployment because the model itself is the core computation unit, and its inference time scales with its complexity.

Answer

Input data preprocessing pipeline

Answer

API gateway rate limiting

Answer

Database connection pool size

Question 4

A data engineering team is designing a pipeline to train a model on streaming data. The data arrives in a time-series format. Which approach should they use to ensure the model reflects current trends without catastrophic forgetting?

Accepted Answer

Implement incremental learning with periodic validation. Incremental learning (also called online learning) allows the model to update its parameters continuously as new streaming data arrives, without requiring access to historical data. By coupling this with periodic validation on a held-out set, the team can detect concept drift and ensure the model adapts to current trends while avoiding catastrophic forgetting, which occurs when new updates overwrite previously learned patterns.

Answer

Use a sliding window of the most recent data for training

Answer

Deploy an ensemble of models trained on different time periods

Answer

Retrain the entire model from scratch every week

Question 5

A financial institution deploys an AI credit scoring model. After six months, the model's performance drops significantly. Analysis shows that the relationship between features and labels has changed. Which term describes this phenomenon?

Accepted Answer

Concept drift. Concept drift occurs when the statistical relationship between input features and the target label changes over time, which is exactly what happened when the credit scoring model's performance dropped due to a shift in the feature-label relationship. This is distinct from data drift, which only involves changes in the input data distribution without affecting the label mapping.

Answer

Model decay

Answer

Overfitting

Answer

Data drift

Question 6

An organization is deploying an AI model on edge devices with limited computational resources. Which model optimization technique is most appropriate?

Accepted Answer

Apply model quantization. Model quantization reduces the precision of the model's weights and activations (e.g., from 32-bit floating point to 8-bit integer), which significantly decreases memory footprint and computational requirements. This makes it ideal for deployment on edge devices with limited resources, as it enables faster inference with minimal accuracy loss.

Answer

Perform additional feature engineering

Answer

Use an ensemble of models

Answer

Increase the training dataset size

Question 7

A company uses an AI model to predict equipment failures. The model outputs a probability of failure. To minimize false alarms, the operations team wants a high precision. Which deployment strategy should they implement?

Accepted Answer

Increase the decision threshold for positive classification. To minimize false alarms and achieve high precision, the operations team should increase the decision threshold for positive classification. A higher threshold means the model only predicts a failure when it is very confident, reducing the number of false positives (false alarms) at the cost of potentially missing some true failures (lower recall). This directly controls the precision-recall trade-off without changing the underlying model.

Answer

Retrain the model on more recent data

Answer

Decrease the decision threshold

Answer

Use an ensemble of models with voting

Question 8

A team is implementing an ML pipeline using a feature store. Which benefit does a feature store primarily provide in an AI operations context?

Accepted Answer

Consistency of feature computation between training and inference. A feature store ensures that feature engineering logic is stored, versioned, and reused consistently across both training and inference pipelines. This eliminates training-serving skew, a common cause of model degradation in production, by guaranteeing that the same transformations are applied to data regardless of when or where it is computed.

Answer

Automated scaling of inference endpoints

Answer

Real-time monitoring of model performance

Answer

Automatic model versioning and rollback

Question 9

Which TWO actions should be taken to ensure an AI model complies with GDPR requirements when processing personal data?

Accepted Answer

Limit data collection to only what is necessary for the model. GDPR's data minimization principle (Article 5(1)(c)) requires that personal data collected be adequate, relevant, and limited to what is necessary for the purpose for which it is processed. In AI model training, this means collecting only the features essential for the model's objective, reducing the risk of processing excessive or irrelevant personal data.

Answer

Provide a full explanation of model predictions

Answer

Store all user data for a minimum of 10 years

Answer

Anonymize all personal data before use

Question 10

Which THREE factors are most critical to consider when designing a continuous integration/continuous deployment (CI/CD) pipeline for machine learning?

Accepted Answer

Data quality and schema validation. Data quality and schema validation (A) are critical because ML pipelines are highly sensitive to data drift, missing values, and schema mismatches that can silently degrade model performance. Without automated validation at the CI stage, bad data can pass through and corrupt model training or inference, leading to unreliable outputs in production.

Answer

A/B testing framework for comparing models

Answer

Automated unit testing of application code

Question 11

An AI system is being implemented in a healthcare setting. Which TWO ethical considerations should be prioritized?

Accepted Answer

Ensuring the model does not exhibit racial or gender bias. AI systems in healthcare must avoid racial or gender bias to ensure equitable treatment outcomes. Biased models can lead to misdiagnosis or unequal care, violating ethical standards and regulatory requirements like HIPAA. Prioritizing fairness aligns with responsible AI implementation in clinical settings.

Answer

Maximizing cost reduction for the hospital

Answer

Replacing human judgment entirely with AI

Answer

Using open-source models to reduce licensing costs

Question 12

A company deploys a deep learning model for real-time image classification. After deployment, they notice high inference latency exceeding the 100ms SLA. Which action would most likely reduce latency without significantly impacting accuracy?

Accepted Answer

Apply model quantization. Model quantization reduces the precision of the model's weights and activations (e.g., from 32-bit floating point to 8-bit integer), which significantly decreases memory bandwidth and computational requirements during inference. This directly lowers latency without fundamentally altering the model's learned representations, so accuracy degradation is typically minimal (often <1-2%).

Answer

Add more training data to improve model robustness

Answer

Replace the model with a simpler logistic regression model

Answer

Increase batch size for inference

Question 13

An AI system for fraud detection shows a gradual decline in precision over several weeks, though recall remains stable. Which type of model drift is most likely occurring?

Accepted Answer

Concept drift. Concept drift occurs when the statistical relationship between input features and the target variable changes over time, causing the model's decision boundary to become less accurate. In this scenario, precision is declining while recall remains stable, indicating that the model is producing more false positives even though it still catches the same proportion of true positives. This is a classic sign of concept drift, where the underlying definition of fraud has shifted, not the data distribution itself.

Answer

Data drift

Answer

Covariate shift

Answer

Label drift

Question 14

A batch inference pipeline fails intermittently with out-of-memory errors when processing large datasets. The pipeline uses pandas DataFrames and feeds a pre-trained model. Which change would most effectively reduce memory consumption?

Accepted Answer

Split the data into smaller chunks and process sequentially. Splitting a large dataset into smaller chunks and processing them sequentially directly addresses the root cause of the out-of-memory error: the entire dataset is loaded into memory at once via pandas DataFrames. By processing data in batches, each chunk fits within the available RAM, preventing memory exhaustion while still allowing the pipeline to complete the full inference workload.

Answer

Increase the instance size of the compute node

Answer

Use a database instead of CSV files

Answer

Convert the model to use half-precision

Question 15

A company wants to roll out a new recommendation model to production. They decide to run an A/B test where 10% of users see the new model and 90% see the old model. After one week, the new model shows a 5% improvement in click-through rate. What is the next best action?

Accepted Answer

Increase the testing percentage gradually while monitoring performance metrics and guardrails. A 5% improvement observed over only one week with a 10% traffic split is insufficient to confirm statistical significance or rule out novelty effects, data drift, or seasonal bias. The recommended best practice in AI deployment is to gradually increase the testing percentage (e.g., 10% → 25% → 50% → 100%) while continuously monitoring performance metrics and guardrails (e.g., click-through rate, conversion rate, latency, and error rates) to ensure the new model generalizes safely across the full user population.

Answer

Immediately roll out the new model to 100% of users

Answer

Revert to the old model because the improvement is minimal

Answer

Run the test for another month to ensure statistical significance