AAIR AI Lifecycle Risk Management Practice Test 3 — 15 Questions

Question 1

You are auditing a model's lifecycle and discover that the training data distribution changes every time the model retrains. What is the biggest risk here?

Accepted Answer

The model's behavior will be inconsistent and difficult to reproduce, making debugging impossible.. When the training data distribution changes on every retrain, the model's learned parameters and behavior shift unpredictably, making it impossible to reproduce results, compare versions, or debug regressions. This undermines model governance, auditability, and trust. Reproducibility requires stable, versioned training data and documented preprocessing.

Answer

The model will require more compute power.

Answer

The model will be too secure.

Answer

The model will be too small.

Question 2

Which TWO of the following are key components of a robust AI lifecycle risk management program?

Accepted Answer

Continuous model performance and data quality monitoring.. Option A is correct because continuous monitoring of model performance and data quality is essential for detecting drift, degradation, and data issues throughout the AI lifecycle, enabling timely retraining or remediation. Option D is correct because comprehensive versioning of data, code, and model artifacts provides traceability, reproducibility, and auditability, which are foundational to managing risk across the AI lifecycle. Options B, C, and E are not key components of AI lifecycle risk management: using only open-source libraries, adhering to a single coding language, and increasing team size do not by themselves address risk controls, governance, or model reliability.

Answer

Using only open-source libraries.

Answer

Strict adherence to a single coding language.

Answer

Increasing the number of developers on the team.

Question 3

What is the primary risk of 'Model Complexity' in the context of the AI lifecycle?

Accepted Answer

It becomes increasingly difficult to interpret decisions and perform root cause analysis during incidents.. As model complexity increases (deep neural networks, ensembles, large transformers), the internal decision logic becomes opaque, making it hard to explain why a specific output occurred. This directly impairs root cause analysis during incidents, because teams cannot trace which features or interactions drove a bad prediction. It also complicates bias detection and regulatory compliance.

Answer

The model is too fast for the hardware.

Answer

The model will always produce the same result.

Answer

It forces the team to use more expensive software licenses.

Question 4

Which TWO of the following techniques help mitigate bias in the AI lifecycle?

Accepted Answer

Conducting fairness audits using specialized metrics (e.g., Disparate Impact).. Option C is correct because conducting fairness audits with specialized metrics such as Disparate Impact (the ratio of favorable outcomes for a protected group to those of a reference group, ideally near 1.0 under the 80% rule) quantitatively detects and helps correct disparate treatment or outcomes across groups in the AI lifecycle. Option D is correct because sourcing training data from diverse, representative populations reduces sampling and representation bias, ensuring the model learns patterns that generalize fairly rather than overfitting to a dominant or historically skewed subgroup. Option A is incorrect because the programming language (e.g., Java) is irrelevant to bias; bias arises from data, objectives, and evaluation, not from the implementation language. Option B is incorrect because reducing model monitoring would hide drift and emerging bias, worsening rather than mitigating it. Option E is incorrect because simply adding more features can amplify bias through proxy variables and increase overfitting without addressing underlying data or fairness issues.

Answer

Switching the programming language to Java.

Answer

Reducing the frequency of model monitoring.

Answer

Increasing the number of features in the model.

Question 5

Which TWO metrics are standard in 'Monitoring' for identifying performance-related AI risks?

Accepted Answer

Inference request latency.. Latency and error rates are the primary operational indicators of model performance risk.

Answer

The number of lines of documentation.

Answer

The color of the deployment dashboard.

Answer

The total number of servers available.

Question 6

Which TWO of the following scenarios represent 'Data Quality Risk'?

Accepted Answer

Training data is from a 5-year-old dataset that is no longer representative.. Option A is correct because a 5-year-old dataset that is no longer representative introduces staleness and distribution drift, meaning the training data no longer reflects the current population or environment — a classic data quality risk affecting model validity. Option D is correct because inconsistent units (mixing meters and feet) are a data integrity/consistency defect that corrupts feature values and can cause the model to learn incorrect relationships, directly harming data quality. Option B does not belong because slow network deployment is an infrastructure/operational performance concern, not a property of the data itself. Option C does not belong because the programming language used to write the model is an implementation/engineering choice unrelated to data quality. Option E does not belong because using a popular open-source library is a technology/dependency decision, not a data quality issue.

Answer

The model is deployed on a slow network.

Answer

The model is written in a new programming language.

Answer

The model uses a popular open-source library.

Question 7

Which THREE artifacts should be reviewed during a post-incident AI risk analysis?

Accepted Answer

The original Model Card and documentation.. The original Model Card and documentation (C) must be reviewed because it defines the model's intended purpose, training data characteristics, performance metrics, and known limitations, which are essential to determine whether the incident stemmed from out-of-scope use or undocumented behavior. The raw input and prediction output logs (D) are critical because they let analysts reconstruct the exact inference requests and responses surrounding the incident, revealing drift, adversarial inputs, or anomalous outputs that triggered the risk event. The model versioning and lineage logs (E) are also required to trace which model artifact, training dataset, and pipeline commit were actually serving at the time, since a regression or unauthorized model swap is a common root cause in AI incidents. By contrast, employee training certificates (A) and building access logs for the data center (B) are general physical or HR compliance records that do not capture model behavior, data provenance, or inference-time evidence needed for an AI-specific post-incident analysis.

Answer

The employee training certificates.

Answer

The building access logs for the data center.

Question 8

Which THREE factors should be monitored to detect 'Model Drift' in the deployment phase?

Accepted Answer

The distribution of prediction outputs.. Model drift detection in the deployment phase focuses on changes in data and model behavior over time. Option A, the distribution of prediction outputs, is correct because a shift in the distribution of predictions (e.g., a classifier suddenly predicting one class far more often) is a direct symptom of model drift, often caused by changing input data or degraded decision boundaries. Option D, the confidence score of the model predictions, is correct because declining or abnormally shifting confidence scores indicate the model is becoming less certain and less aligned with the data it was trained on, a common signal of drift. Option E, the distribution of incoming input features, is correct because monitoring feature distributions (e.g., via statistical tests like PSI or KL divergence) detects data drift, which is the primary driver of model drift in production. Option B, the temperature of the server room, is incorrect because it is a physical infrastructure concern unrelated to statistical model behavior. Option C, the current market price of GPU hardware, is incorrect because it is a procurement/cost factor with no bearing on whether a deployed model's performance is degrading.

Answer

The temperature of the server room.

Answer

The current market price of GPU hardware.

Question 9

Which THREE risks are associated with 'Automated Deployment' of AI models without a human-in-the-loop?

Accepted Answer

Rapid propagation of faulty or biased models to production.. Option A is correct because automated deployment pipelines can push a faulty or biased model artifact straight to production without any manual review gate, so errors propagate at machine speed across all serving instances. Option C is correct because without a human reviewing the deployment configuration (IAM roles, exposed endpoints, secrets handling, container/image provenance), misconfigurations and security vulnerabilities can be merged and applied automatically. Option D is correct because human intuition and contextual judgment are often what catch edge-case or out-of-distribution failures that automated tests and metrics do not cover, and removing the human-in-the-loop eliminates that safeguard. Option B is not a risk of automation — automated deployment is generally faster, not slower, than manual release. Option E is not a risk of automated deployment; disk space consumption depends on model size and artifact retention, not on whether a human approves the deployment.

Answer

The deployment takes too long to run.

Answer

Increased use of disk space.

Question 10

Which TWO factors are essential when documenting a model for risk management?

Accepted Answer

The model's intended use cases and operational boundaries.. Option A is correct because documenting a model for risk management must clearly state its intended use cases and operational boundaries, as these define the context in which the model is valid and where its risk exposure begins and ends. Option D is correct because recording the performance metrics and limitations observed during testing provides the quantitative evidence needed to assess residual risk, monitor degradation, and set appropriate controls or thresholds. Together, A and D establish both the scope of applicability and the empirical behavior of the model, which are foundational to any risk assessment. Option B is incorrect because the developers' coffee brand is irrelevant to model risk. Option C is incorrect because personal hobbies of team members have no bearing on model risk documentation. Option E is incorrect because listing all users who access the model is an access-control or audit concern, not an essential factor for documenting the model itself for risk management.

Answer

The specific brand of coffee used by the developers.

Answer

The specific team member's personal hobbies.

Answer

The names of all users who access the model.

Question 11

Which THREE considerations must be addressed when designing a 'Training' pipeline to minimize data quality risk?

Accepted Answer

Recording full data lineage for every training run.. Option B is correct because recording full data lineage for every training run provides traceability of the exact datasets, transformations, and versions used, which is essential for detecting and remediating data quality issues and reproducing results. Option D is correct because automated outlier detection identifies noisy, anomalous, or corrupted records before they degrade model training, directly reducing data quality risk. Option E is correct because strict schema validation catches data type errors, missing fields, and format mismatches at ingestion time, preventing malformed data from entering the training pipeline. Options A and C are not correct because a high-end monitor and a specific server hardware brand are irrelevant to data quality; they concern display and procurement preferences rather than the integrity, validity, or traceability of training data.

Answer

Ensuring the training server has a high-end monitor.

Answer

Using a specific brand of server hardware.

Question 12

Which THREE strategies are effective for mitigating 'Adversarial Risk' in an AI lifecycle?

Accepted Answer

Rate limiting on API calls to prevent brute-force querying.. Option B is correct because rate limiting on API calls throttles the query volume an attacker can send, which directly counters brute-force and model-extraction attacks that rely on high-volume probing of the inference endpoint. Option D is correct because adversarial training explicitly augments the training set with perturbed examples, hardening the model's decision boundaries against evasion and adversarial-input attacks. Option E is correct because input sanitization and validation on all model endpoints rejects malformed, out-of-distribution, or injection-style payloads before they reach the model, reducing the attack surface for adversarial manipulation. Option A does not belong because adding documentation staff addresses process and knowledge management, not adversarial manipulation of the model. Option C does not belong because a more expensive cloud provider changes cost, not the security posture against adversarial inputs.

Answer

Hiring more documentation staff.

Answer

Using a more expensive cloud provider.

Question 13

A retail bank's credit-scoring AI was validated on 2023 applicant data and deployed in early 2024. By Q4 2024, portfolio default rates among approved loans have crept up while the model's input feature distributions remain statistically unchanged. The risk team wants to isolate the most probable lifecycle cause before retraining. Which risk management activity best addresses this situation?

Accepted Answer

Recompute the model's discrimination metrics against a fresh outcome-labeled holdout and compare them to the validation baseline.. When input distributions are stable but realized outcomes worsen, the lifecycle risk is concept drift rather than data drift. The only control listed that measures the changed feature-to-outcome relationship is re-evaluating discrimination and calibration against newly labeled outcomes and comparing to the validation baseline. That evidence determines whether retraining, recalibration, or redesign is warranted.

Answer

Recalibrate the model's probability outputs using isotonic regression on the original 2023 validation set.

Answer

Increase the decision threshold for approval to reduce the volume of higher-risk applicants entering the portfolio.

Answer

Run a Kolmogorov-Smirnov test on each input feature to detect data drift in the production population.

Question 14

An insurer deploys a claims-triage model built by an external vendor. The contract grants the insurer use rights but the vendor refuses to disclose training data composition or feature engineering logic, citing trade secrets. Internal audit must document residual risk before the model enters production. Which action best fulfills the AI risk management obligation given these constraints?

Accepted Answer

Document the opacity as a residual risk, obtain contractual attestations and independent testing rights, and set compensating controls such as challenger models and outcome monitoring.. Third-party opacity is a governance risk that must be recorded, not silently accepted. The defensible path is to document the limitation as residual risk, strengthen the contract with attestations and testing rights, and apply compensating controls such as challenger models and ongoing outcome monitoring. This preserves the business capability while giving audit and the risk committee visibility and enforceable levers.

Answer

Reject the vendor outright and rebuild the model internally using the insurer's own historical claims data.

Answer

Request the vendor's model card and treat its published performance metrics as the primary validation evidence.

Answer

Accept the vendor's position and rely on the vendor's SOC 2 report as sufficient third-party assurance.

Question 15

A retail bank's credit-scoring AI was trained on three years of historical loan data. During an ISACA AAIR-aligned risk review, the risk practitioner discovers that the model's training dataset included the final repayment outcomes of loans that were still open at the time of data extraction. Which risk is most directly introduced by this dataset construction?

Accepted Answer

Target leakage, because repayment outcomes encoded future information that would not be available at inference time.. The dataset includes repayment outcomes from loans that were still open when data was extracted, so the model sees information that would not exist at the moment a credit decision is made. That temporal contamination is target leakage, and it inflates validation performance while guaranteeing degradation in production. A lifecycle risk review must detect it during data preparation, before the model is approved for deployment.

Answer

Sampling bias, because the three-year window excludes applicants who never received a loan.

Answer

Concept drift, because credit behaviour changes over the three-year training window.

Answer

Label noise, because repayment outcomes were recorded inconsistently across legacy systems.