AAIR AI Lifecycle Risk Management — Timed 10 Questions

Question 1

You are utilizing Azure Machine Learning to manage a deployment. You detect a sudden drop in model performance due to 'concept drift'. Which specific configuration in the Azure ML Model Monitoring dashboard should be adjusted to better detect this?

10 minute time limit — choose an answer to begin.

Accepted Answer

Adjust the feature drift sensitivity threshold for categorical variables.. Concept drift detection in Azure Machine Learning model monitoring is configured through drift thresholds on features and predictions. For categorical variables, the relevant setting is the feature drift sensitivity threshold, which controls how much distribution shift triggers an alert. Adjusting this threshold for categorical features allows the monitor to detect concept drift manifesting as changes in categorical input distributions.

Answer

Increase the retraining frequency of the pipeline.

Answer

Disable the data lineage capture to improve latency.

Answer

Switch the model to a higher-capacity compute instance.

Question 2

Which TWO activities are part of the 'Risk Identification' step in the AI lifecycle?

Accepted Answer

Documenting the limitations and failure modes of the model.. Option A is correct because Risk Identification in the AI lifecycle requires documenting the model's limitations and failure modes, which surfaces potential risks such as performance degradation, unsafe outputs, or misuse scenarios before mitigation planning begins. Option E is correct because performing bias and fairness testing on training and test sets is a core risk identification activity that detects disparate impact, representation gaps, and label bias that could cause harm or non-compliance. The remaining options do not belong: B concerns project budgeting, C is a corporate policy task unrelated to AI risk analysis, and D is a user-interface design decision, none of which identify AI-specific risks.

Answer

Calculating the total budget for the project.

Answer

Updating the company's social media policy.

Answer

Choosing the color scheme for the model's web dashboard.

Question 3

During the AI lifecycle, when should a 'Data Quality' assessment be performed to minimize long-term risk?

Accepted Answer

During the data ingestion and preprocessing stage.. Data quality assessment must happen during data ingestion and preprocessing, because that is when data is collected, cleaned, validated, and transformed before it reaches model training. Catching quality issues here prevents garbage-in-garbage-out propagation and is far cheaper than remediating after deployment. This early-stage control minimizes long-term risk across the entire AI lifecycle.

Answer

After model deployment in production.

Answer

During the final model sign-off.

Answer

During the decommission phase.

Question 4

Which AI lifecycle stage is most susceptible to the risk of 'data leakage' where training data inadvertently contains information from the future/target?

Accepted Answer

Data preparation and training phase.. Data leakage occurs when information that should not be available at inference time (e.g., the target label, future data, or holdout data) is inadvertently included in the training set. This happens during data preparation and training, where feature engineering, joins, and train/test splits are performed. If a feature is computed using the target or future timestamps, the model learns a shortcut that inflates training metrics but fails in production.

Answer

Model monitoring phase.

Answer

Model decommission phase.

Answer

Deployment phase.

Question 5

A practitioner is setting up a model monitoring service in AWS SageMaker Model Monitor. They observe that the ground truth data is significantly delayed. What action ensures risk identification remains effective?

Accepted Answer

Configure Data Quality Monitoring on input features only.. When ground truth labels are delayed, SageMaker Model Monitor cannot compute model quality metrics (accuracy, precision, etc.) because those require labels. The effective fallback is to enable Data Quality Monitoring on input features, which detects drift, missing values, and distribution shifts without needing labels, preserving risk identification in the interim.

Answer

Disable monitoring until ground truth is available.

Answer

Manually force a model redeployment.

Answer

Reduce the sampling rate of the inference logs.

Question 6

You are managing a model that utilizes 'Online Learning'. What is the most critical risk requiring constant lifecycle vigilance?

Accepted Answer

Adversarial data poisoning and instability.. Online learning continuously updates model parameters from streaming data, which means the model's training distribution is directly influenced by whatever data arrives in production. This creates a critical, ongoing risk of adversarial data poisoning — attackers can inject malicious samples into the live stream to corrupt the model — as well as instability from feedback loops and non-stationary data. Constant lifecycle vigilance is required because the model can degrade or be subverted between reviews.

Answer

Hardware failure in the training cluster.

Answer

Data drift from batch updates.

Answer

Memory exhaustion due to high throughput.

Question 7

A financial institution uses an AI model to approve loans. After deployment, they notice the model is rejecting loan applications from a specific region at a rate 30% higher than historical human benchmarks. What is the most appropriate next step in the risk lifecycle?

Accepted Answer

Conduct a bias investigation by comparing input distribution shifts between training data and current inference requests.. A 30% higher rejection rate for a specific region versus historical human benchmarks is a classic bias/drift signal, so the correct risk-lifecycle step is to investigate by comparing the input distribution at inference time against the training distribution. This determines whether the disparity stems from data drift, a biased feature, or a genuine change in the population, and it produces evidence before any remediation decision. Ignoring, shutting down, or building an override model are premature or inappropriate responses.

Answer

Ignore the shift as it may be a temporary market trend.

Answer

Immediately shut down the system.

Answer

Train a secondary model to override the primary one.

Question 8

In the context of the Google Cloud Vertex AI Model Registry, what is the best approach to mitigate the risk associated with a model update that performs poorly on edge cases?

Accepted Answer

Implement traffic splitting with a shadow deployment.. Traffic splitting with a shadow deployment lets the new model version receive live traffic (or a copy of it) without affecting production responses, so its behaviour on edge cases can be observed before any real user impact. This directly mitigates the risk of a poorly performing update by gating promotion on measured performance. It is the standard safe-rollout pattern in Vertex AI Model Registry.

Answer

Increase the training epoch count for the new version.

Answer

Directly replace the active production model version.

Answer

Update the training data to remove edge case records.

Question 9

During the design phase of a machine learning model, a practitioner identifies that the training dataset has significant underrepresentation of a minority demographic. What is the most effective proactive risk mitigation strategy?

Accepted Answer

Apply synthetic data generation techniques to balance the training distribution.. When a training dataset underrepresents a demographic, the most effective proactive mitigation at the design phase is to correct the data distribution itself — synthetic data generation (e.g., SMOTE, CTGAN, or targeted augmentation) can increase representation of the minority group so the model learns its patterns rather than defaulting to the majority. Addressing the root cause in the data is preferable to downstream patches because it improves model behavior for the affected group across all future inferences.

Answer

Deploy the model with a human-in-the-loop oversight mechanism.

Answer

Monitor prediction confidence intervals during the post-deployment phase.

Answer

Increase the regularization parameters in the model architecture.

Question 10

Why is 'Explainability' considered a key risk management control in the AI lifecycle?

Accepted Answer

It enables identification of potential bias, errors, or unexpected behavior in model decision-making.. Explainability provides visibility into how a model arrives at its outputs, which allows teams to detect bias, errors, and unexpected behavior before they cause harm. In risk management terms, it is a detective and preventive control that supports incident investigation, regulatory compliance, and stakeholder trust. Without it, failures can go undiagnosed and accountability becomes impossible.

Answer

It guarantees the model will never fail.

Answer

It increases the inference speed of the model.

Answer

It allows the model to train on less data.