AAIR AI Lifecycle Risk Management — 20 Questions

Question 1

During the AI lifecycle, when should a 'Data Quality' assessment be performed to minimize long-term risk?

Accepted Answer

During the data ingestion and preprocessing stage.. Data quality must be validated prior to training to ensure model reliability.

Answer

After model deployment in production.

Answer

During the final model sign-off.

Answer

During the decommission phase.

Question 2

Which AI lifecycle stage is most susceptible to the risk of 'data leakage' where training data inadvertently contains information from the future/target?

Accepted Answer

Data preparation and training phase.. Data leakage typically occurs during the data preparation and feature engineering stages of the AI lifecycle.

Answer

Model monitoring phase.

Answer

Model decommission phase.

Answer

Deployment phase.

Question 3

A practitioner is setting up a model monitoring service in AWS SageMaker Model Monitor. They observe that the ground truth data is significantly delayed. What action ensures risk identification remains effective?

Accepted Answer

Configure Data Quality Monitoring on input features only.. When ground truth is delayed, practitioners must rely on feature drift detection rather than accuracy metrics.

Answer

Disable monitoring until ground truth is available.

Answer

Manually force a model redeployment.

Answer

Reduce the sampling rate of the inference logs.

Question 4

You are utilizing Azure Machine Learning to manage a deployment. You detect a sudden drop in model performance due to 'concept drift'. Which specific configuration in the Azure ML Model Monitoring dashboard should be adjusted to better detect this?

Accepted Answer

Adjust the feature drift sensitivity threshold for categorical variables.. Concept drift occurs when the relationship between input variables and target variables changes, requiring adjustments to data drift detection parameters.

Answer

Increase the retraining frequency of the pipeline.

Answer

Disable the data lineage capture to improve latency.

Answer

Switch the model to a higher-capacity compute instance.

Question 5

You are managing a model that utilizes 'Online Learning'. What is the most critical risk requiring constant lifecycle vigilance?

Accepted Answer

Adversarial data poisoning and instability.. Online learning models are constantly updating, making them highly susceptible to 'poisoning' attacks.

Answer

Hardware failure in the training cluster.

Answer

Data drift from batch updates.

Answer

Memory exhaustion due to high throughput.

Question 6

A financial institution uses an AI model to approve loans. After deployment, they notice the model is rejecting loan applications from a specific region at a rate 30% higher than historical human benchmarks. What is the most appropriate next step in the risk lifecycle?

Accepted Answer

Conduct a bias investigation by comparing input distribution shifts between training data and current inference requests.. Investigation of bias requires a root cause analysis of input distributions and model weights compared to the baseline.

Answer

Ignore the shift as it may be a temporary market trend.

Answer

Immediately shut down the system.

Answer

Train a secondary model to override the primary one.

Question 7

In the context of the Google Cloud Vertex AI Model Registry, what is the best approach to mitigate the risk associated with a model update that performs poorly on edge cases?

Accepted Answer

Implement traffic splitting with a shadow deployment.. A/B testing (or canary deployments) allows for traffic splitting to validate performance before full rollout.

Answer

Increase the training epoch count for the new version.

Answer

Directly replace the active production model version.

Answer

Update the training data to remove edge case records.

Question 8

During the design phase of a machine learning model, a practitioner identifies that the training dataset has significant underrepresentation of a minority demographic. What is the most effective proactive risk mitigation strategy?

Accepted Answer

Apply synthetic data generation techniques to balance the training distribution.. Data augmentation or oversampling techniques are standard practices to address bias before training begins.

Answer

Deploy the model with a human-in-the-loop oversight mechanism.

Answer

Monitor prediction confidence intervals during the post-deployment phase.

Answer

Increase the regularization parameters in the model architecture.

Question 9

Why is 'Explainability' considered a key risk management control in the AI lifecycle?

Accepted Answer

It enables identification of potential bias, errors, or unexpected behavior in model decision-making.. Explainability allows stakeholders to audit decisions and ensure they comply with regulatory requirements (like GDPR).

Answer

It guarantees the model will never fail.

Answer

It increases the inference speed of the model.

Answer

It allows the model to train on less data.

Question 10

A practitioner is reviewing the 'Model Card' for an AI system. What is the primary purpose of this artifact in the AI lifecycle risk management process?

Accepted Answer

To provide transparency regarding the model's limitations, intended use cases, and performance metrics.. Model cards provide transparency and documentation about the intended use, limitations, and performance characteristics.

Answer

To automate the retraining pipeline for the model.

Answer

To serve as the primary authentication token for API access.

Answer

To define the deployment strategy in Kubernetes.

Question 11

You are performing a 'Model Drift' analysis. Which metric is most indicative of performance degradation without access to real-time ground truth?

Accepted Answer

Kullback-Leibler (KL) Divergence between training and inference feature distributions.. Statistical distribution shifts (like KL Divergence) in input data serve as a proxy for performance degradation.

Answer

The number of concurrent API requests.

Answer

The memory consumption of the container.

Answer

Model accuracy on historical data.

Question 12

A developer is using 'Shadow Deployments' for a new AI model. What is the main risk being addressed?

Accepted Answer

Operational risk of deploying a poorly performing model to production.. Shadow deployments allow real-world data validation without exposing end-users to potential model failures.

Answer

Model documentation compliance.

Answer

Training data leakage.

Answer

Hardware infrastructure costs.

Question 13

During the 'Monitoring' phase, which activity is most critical for identifying model 'feedback loops'?

Accepted Answer

Tracking the distribution of prediction outputs over time for systemic shifts.. Feedback loops occur when model outputs influence the future training data, which must be detected by comparing output distributions over time.

Answer

Validating the hardware security keys.

Answer

Checking for system uptime.

Answer

Analyzing the training pipeline latency.

Question 14

Which of the following is an example of 'Data Quality Risk' in the AI lifecycle?

Accepted Answer

High rates of mislabeled training data.. Data quality risk includes issues like missing values, noisy data, or incorrect labeling that degrade model performance.

Answer

Setting a high learning rate.

Answer

Using Python instead of R.

Answer

Using a GPU that is two generations old.

Question 15

A company is implementing 'Human-in-the-Loop' (HITL) for its AI decision system. What is the primary risk of relying on HITL?

Accepted Answer

The 'automation bias' or 'rubber stamping' of decisions by human reviewers.. Human fatigue and cognitive bias can lead to poor oversight, potentially creating a false sense of security.

Answer

It requires expensive API calls.

Answer

It causes the model to become slower.

Answer

It forces the model to use a specific programming language.

Question 16

When managing AI lifecycle risk for a model in a regulated industry, which artifact serves as the most important audit trail for the model's provenance?

Accepted Answer

The end-to-end model lineage and version control logs.. Model lineage/provenance trackers record the exact data versions and training parameters used for a specific model version.

Answer

The user access logs for the dashboard.

Answer

The model's current latency metrics.

Answer

The cloud provider's billing statement.

Question 17

What is the primary risk objective of 'Model Versioning' in the AI lifecycle?

Accepted Answer

To ensure auditability, reproducibility, and the ability to roll back to a known stable state.. Version control is essential for reproducibility and rollback, minimizing the risk of unrecoverable failures.

Answer

To prevent unauthorized users from viewing the code.

Answer

To reduce the amount of disk space used.

Answer

To increase the model's speed in production.

Question 18

You are assessing risk for an AI model that uses 'Transfer Learning'. What is the most critical risk to manage regarding the pre-trained base model?

Accepted Answer

The base model may have been trained on data that contains inherent, unidentifiable biases.. Transfer learning risks include the inheritance of latent biases from the pre-trained weights.

Answer

The base model cannot be used with custom datasets.

Answer

The base model will always run out of memory.

Answer

The base model requires a proprietary operating system.

Question 19

A team notices that an AI model's performance decreases when the input data contains abbreviations that were not present in the training set. This is a risk associated with which lifecycle stage?

Accepted Answer

Data preprocessing and training design.. This is a Data Quality/Data Representation risk, where the model's understanding of data is limited by the training corpus.

Answer

Infrastructure provisioning.

Answer

Monitoring phase.

Answer

Deployment phase.

Question 20

When a 'Data Drift' alert is triggered in a production environment, what is the first step in the risk management lifecycle?

Accepted Answer

Perform a root cause analysis to verify if the shift is statistical noise or a genuine change in input patterns.. The first step is always to validate the alert to ensure it's not a false positive before taking drastic action.

Answer

Immediately retrain the model.

Answer

Delete all current production logs.

Answer

Roll back to the previous model version.