AAIR AI Lifecycle Risk Management — 20 Questions

Question 1

You are utilizing Azure Machine Learning to manage a deployment. You detect a sudden drop in model performance due to 'concept drift'. Which specific configuration in the Azure ML Model Monitoring dashboard should be adjusted to better detect this?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Adjust the feature drift sensitivity threshold for categorical variables.. Concept drift detection in Azure Machine Learning model monitoring is configured through drift thresholds on features and predictions. For categorical variables, the relevant setting is the feature drift sensitivity threshold, which controls how much distribution shift triggers an alert. Adjusting this threshold for categorical features allows the monitor to detect concept drift manifesting as changes in categorical input distributions.

Answer

Increase the retraining frequency of the pipeline.

Answer

Disable the data lineage capture to improve latency.

Answer

Switch the model to a higher-capacity compute instance.

Question 2

Which TWO activities are part of the 'Risk Identification' step in the AI lifecycle?

Accepted Answer

Documenting the limitations and failure modes of the model.. Option A is correct because Risk Identification in the AI lifecycle requires documenting the model's limitations and failure modes, which surfaces potential risks such as performance degradation, unsafe outputs, or misuse scenarios before mitigation planning begins. Option E is correct because performing bias and fairness testing on training and test sets is a core risk identification activity that detects disparate impact, representation gaps, and label bias that could cause harm or non-compliance. The remaining options do not belong: B concerns project budgeting, C is a corporate policy task unrelated to AI risk analysis, and D is a user-interface design decision, none of which identify AI-specific risks.

Answer

Calculating the total budget for the project.

Answer

Updating the company's social media policy.

Answer

Choosing the color scheme for the model's web dashboard.

Question 3

During the AI lifecycle, when should a 'Data Quality' assessment be performed to minimize long-term risk?

Accepted Answer

During the data ingestion and preprocessing stage.. Data quality assessment must happen during data ingestion and preprocessing, because that is when data is collected, cleaned, validated, and transformed before it reaches model training. Catching quality issues here prevents garbage-in-garbage-out propagation and is far cheaper than remediating after deployment. This early-stage control minimizes long-term risk across the entire AI lifecycle.

Answer

After model deployment in production.

Answer

During the final model sign-off.

Answer

During the decommission phase.

Question 4

Which AI lifecycle stage is most susceptible to the risk of 'data leakage' where training data inadvertently contains information from the future/target?

Accepted Answer

Data preparation and training phase.. Data leakage occurs when information that should not be available at inference time (e.g., the target label, future data, or holdout data) is inadvertently included in the training set. This happens during data preparation and training, where feature engineering, joins, and train/test splits are performed. If a feature is computed using the target or future timestamps, the model learns a shortcut that inflates training metrics but fails in production.

Answer

Model monitoring phase.

Answer

Model decommission phase.

Answer

Deployment phase.

Question 5

A practitioner is setting up a model monitoring service in AWS SageMaker Model Monitor. They observe that the ground truth data is significantly delayed. What action ensures risk identification remains effective?

Accepted Answer

Configure Data Quality Monitoring on input features only.. When ground truth labels are delayed, SageMaker Model Monitor cannot compute model quality metrics (accuracy, precision, etc.) because those require labels. The effective fallback is to enable Data Quality Monitoring on input features, which detects drift, missing values, and distribution shifts without needing labels, preserving risk identification in the interim.

Answer

Disable monitoring until ground truth is available.

Answer

Manually force a model redeployment.

Answer

Reduce the sampling rate of the inference logs.

Question 6

You are managing a model that utilizes 'Online Learning'. What is the most critical risk requiring constant lifecycle vigilance?

Accepted Answer

Adversarial data poisoning and instability.. Online learning continuously updates model parameters from streaming data, which means the model's training distribution is directly influenced by whatever data arrives in production. This creates a critical, ongoing risk of adversarial data poisoning — attackers can inject malicious samples into the live stream to corrupt the model — as well as instability from feedback loops and non-stationary data. Constant lifecycle vigilance is required because the model can degrade or be subverted between reviews.

Answer

Hardware failure in the training cluster.

Answer

Data drift from batch updates.

Answer

Memory exhaustion due to high throughput.

Question 7

A financial institution uses an AI model to approve loans. After deployment, they notice the model is rejecting loan applications from a specific region at a rate 30% higher than historical human benchmarks. What is the most appropriate next step in the risk lifecycle?

Accepted Answer

Conduct a bias investigation by comparing input distribution shifts between training data and current inference requests.. A 30% higher rejection rate for a specific region versus historical human benchmarks is a classic bias/drift signal, so the correct risk-lifecycle step is to investigate by comparing the input distribution at inference time against the training distribution. This determines whether the disparity stems from data drift, a biased feature, or a genuine change in the population, and it produces evidence before any remediation decision. Ignoring, shutting down, or building an override model are premature or inappropriate responses.

Answer

Ignore the shift as it may be a temporary market trend.

Answer

Immediately shut down the system.

Answer

Train a secondary model to override the primary one.

Question 8

In the context of the Google Cloud Vertex AI Model Registry, what is the best approach to mitigate the risk associated with a model update that performs poorly on edge cases?

Accepted Answer

Implement traffic splitting with a shadow deployment.. Traffic splitting with a shadow deployment lets the new model version receive live traffic (or a copy of it) without affecting production responses, so its behaviour on edge cases can be observed before any real user impact. This directly mitigates the risk of a poorly performing update by gating promotion on measured performance. It is the standard safe-rollout pattern in Vertex AI Model Registry.

Answer

Increase the training epoch count for the new version.

Answer

Directly replace the active production model version.

Answer

Update the training data to remove edge case records.

Question 9

During the design phase of a machine learning model, a practitioner identifies that the training dataset has significant underrepresentation of a minority demographic. What is the most effective proactive risk mitigation strategy?

Accepted Answer

Apply synthetic data generation techniques to balance the training distribution.. When a training dataset underrepresents a demographic, the most effective proactive mitigation at the design phase is to correct the data distribution itself — synthetic data generation (e.g., SMOTE, CTGAN, or targeted augmentation) can increase representation of the minority group so the model learns its patterns rather than defaulting to the majority. Addressing the root cause in the data is preferable to downstream patches because it improves model behavior for the affected group across all future inferences.

Answer

Deploy the model with a human-in-the-loop oversight mechanism.

Answer

Monitor prediction confidence intervals during the post-deployment phase.

Answer

Increase the regularization parameters in the model architecture.

Question 10

Why is 'Explainability' considered a key risk management control in the AI lifecycle?

Accepted Answer

It enables identification of potential bias, errors, or unexpected behavior in model decision-making.. Explainability provides visibility into how a model arrives at its outputs, which allows teams to detect bias, errors, and unexpected behavior before they cause harm. In risk management terms, it is a detective and preventive control that supports incident investigation, regulatory compliance, and stakeholder trust. Without it, failures can go undiagnosed and accountability becomes impossible.

Answer

It guarantees the model will never fail.

Answer

It increases the inference speed of the model.

Answer

It allows the model to train on less data.

Question 11

A practitioner is reviewing the 'Model Card' for an AI system. What is the primary purpose of this artifact in the AI lifecycle risk management process?

Accepted Answer

To provide transparency regarding the model's limitations, intended use cases, and performance metrics.. The Model Card is a standardized transparency artifact (popularized by Mitchell et al., 2019) that documents a model's intended use, limitations, performance metrics across subgroups, training data provenance, and ethical considerations. In AI lifecycle risk management, it enables downstream consumers, auditors, and governance teams to assess whether a model is appropriate for a given context and to identify residual risks. It is a documentation and disclosure mechanism, not an operational or security component.

Answer

To automate the retraining pipeline for the model.

Answer

To serve as the primary authentication token for API access.

Answer

To define the deployment strategy in Kubernetes.

Question 12

You are performing a 'Model Drift' analysis. Which metric is most indicative of performance degradation without access to real-time ground truth?

Accepted Answer

Kullback-Leibler (KL) Divergence between training and inference feature distributions.. Statistical distribution shifts (like KL Divergence) in input data serve as a proxy for performance degradation.

Answer

The number of concurrent API requests.

Answer

The memory consumption of the container.

Answer

Model accuracy on historical data.

Question 13

A developer is using 'Shadow Deployments' for a new AI model. What is the main risk being addressed?

Accepted Answer

Operational risk of deploying a poorly performing model to production.. Shadow deployments allow real-world data validation without exposing end-users to potential model failures.

Answer

Model documentation compliance.

Answer

Training data leakage.

Answer

Hardware infrastructure costs.

Question 14

During the 'Monitoring' phase, which activity is most critical for identifying model 'feedback loops'?

Accepted Answer

Tracking the distribution of prediction outputs over time for systemic shifts.. Feedback loops occur when a model's own predictions influence future input data, causing the model's output distribution to drift in a self-reinforcing way. Tracking the distribution of prediction outputs over time is the most direct way to detect these systemic shifts, because a feedback loop manifests as a gradual, self-amplifying change in what the model predicts. This is a monitoring-phase activity, distinct from training or infrastructure checks.

Answer

Validating the hardware security keys.

Answer

Checking for system uptime.

Answer

Analyzing the training pipeline latency.

Question 15

Which of the following is an example of 'Data Quality Risk' in the AI lifecycle?

Accepted Answer

High rates of mislabeled training data.. High rates of mislabeled training data are a classic Data Quality Risk because labels directly determine what the model learns; systematic mislabeling introduces bias and degrades accuracy. Data quality risks encompass issues with accuracy, completeness, consistency, and provenance of training data. Mislabeling is a data integrity problem that propagates into model behavior and is hard to detect post-training.

Answer

Setting a high learning rate.

Answer

Using Python instead of R.

Answer

Using a GPU that is two generations old.

Question 16

A company is implementing 'Human-in-the-Loop' (HITL) for its AI decision system. What is the primary risk of relying on HITL?

Accepted Answer

The 'automation bias' or 'rubber stamping' of decisions by human reviewers.. Human fatigue and cognitive bias can lead to poor oversight, potentially creating a false sense of security.

Answer

It requires expensive API calls.

Answer

It causes the model to become slower.

Answer

It forces the model to use a specific programming language.

Question 17

When managing AI lifecycle risk for a model in a regulated industry, which artifact serves as the most important audit trail for the model's provenance?

Accepted Answer

The end-to-end model lineage and version control logs.. The end-to-end model lineage and version control logs provide the complete, traceable record of how the model was built — including data sources, training runs, hyperparameters, code commits, and artifact versions. This is the artifact auditors and regulators rely on to reconstruct the model's provenance and reproduce its behavior. Access logs, latency metrics, and billing statements document operational or financial activity, not the model's origin or derivation.

Answer

The user access logs for the dashboard.

Answer

The model's current latency metrics.

Answer

The cloud provider's billing statement.

Question 18

What is the primary risk objective of 'Model Versioning' in the AI lifecycle?

Accepted Answer

To ensure auditability, reproducibility, and the ability to roll back to a known stable state.. Model versioning exists to provide auditability (who changed what and when), reproducibility (recreate any past model and its results), and rollback capability (revert to a known stable version when a new release degrades). These properties are foundational for incident response, regulatory review, and safe continuous delivery of AI systems.

Answer

To prevent unauthorized users from viewing the code.

Answer

To reduce the amount of disk space used.

Answer

To increase the model's speed in production.

Question 19

You are assessing risk for an AI model that uses 'Transfer Learning'. What is the most critical risk to manage regarding the pre-trained base model?

Accepted Answer

The base model may have been trained on data that contains inherent, unidentifiable biases.. Transfer learning risks include the inheritance of latent biases from the pre-trained weights.

Answer

The base model cannot be used with custom datasets.

Answer

The base model will always run out of memory.

Answer

The base model requires a proprietary operating system.

Question 20

A team notices that an AI model's performance decreases when the input data contains abbreviations that were not present in the training set. This is a risk associated with which lifecycle stage?

Accepted Answer

Data preprocessing and training design.. Abbreviations absent from the training set are a vocabulary/coverage gap introduced during data collection and feature engineering. Because the model never learned representations for those tokens, it cannot generalize to them at inference — this is a data preprocessing and training design risk (B), not an infrastructure, monitoring, or deployment issue. The root cause is upstream of deployment: the training corpus did not represent the input distribution the model now sees.

Answer

Infrastructure provisioning.

Answer

Monitoring phase.

Answer

Deployment phase.