AAIR Hard Questions — 25 Difficult Questions

Question 1

What is the 'Accountability' principle in the NIST AI RMF intended to address?

Accepted Answer

Ensuring that there is clear ownership and responsibility for AI system performance and safety.. The Accountability principle in the NIST AI RMF (part of the Govern function) is about ensuring clear ownership and responsibility for AI system performance, safety, and outcomes throughout the lifecycle. It answers 'who is responsible?' so that risks are managed, issues are escalated, and redress is possible. It is not about explainability, bug-freeness, or speed.

Answer

Ensuring the AI can explain itself.

Answer

Making sure the model is free of bugs.

Answer

Ensuring that the AI system is fast.

Question 2

You are auditing the integration of AI risk into the Enterprise Risk Management (ERM) system. Which finding indicates a failure in the governance structure?

Accepted Answer

AI models deployed in production are missing from the centralized enterprise risk register.. A centralized enterprise risk register is the authoritative inventory of organizational risks; if production AI models are absent from it, the governance structure has failed to integrate AI risk into ERM — meaning those models are unmanaged, unreviewed, and invisible to executive and Board oversight. This is a structural governance failure because it breaks the linkage between AI risk identification and enterprise-level risk aggregation, violating the principle that all material risks must flow into the ERM.

Answer

The risk assessment template is 20 pages long.

Answer

The AI risk register is updated annually instead of monthly.

Answer

The AI risk committee meets quarterly rather than monthly.

Question 3

Which THREE actions should be taken when integrating AI risk into the broader ERM (Enterprise Risk Management) framework?

Accepted Answer

Establish aligned risk scoring criteria across all business functions.. Option B is correct because integrating AI risk into ERM requires a consistent, enterprise-wide risk scoring methodology so AI risks can be compared, aggregated, and prioritized alongside operational, financial, and compliance risks in the same register. Option D is correct because a shared risk taxonomy must be extended to capture AI-specific concepts such as model bias, data drift, hallucination, and model opacity; without common definitions, AI risks cannot be consistently identified, classified, or reported across functions. Option E is correct because AI risk owners must sit within the same governance and escalation channels (e.g., the same risk committee and reporting lines) as other risk owners, ensuring AI risk is not siloed and receives the same oversight, accountability, and board visibility as traditional risks. Option A does not belong because outsourcing all AI risk management removes internal ownership and accountability, which contradicts ERM principles of retained responsibility and integrated oversight. Option C does not belong because prohibiting every AI initiative with any inherent risk is an avoid-or-eliminate approach rather than risk-based management, and it would block innovation instead of treating, transferring, or accepting AI risk within defined tolerances.

Answer

Outsource all AI risk management to a third-party consultant.

Answer

Prohibit all AI initiatives that carry any inherent risk.

Question 4

You are mapping AI risks to the NIST AI RMF. Which step is essential when documenting the 'Map' function for a high-risk autonomous system?

Accepted Answer

Identifying and documenting the context and intended use. Identifying and documenting the context and intended use is the foundation of mapping AI risks to an enterprise framework.

Answer

Updating the SOC2 Type II report

Answer

Automating the model deployment pipeline

Answer

Conducting a quarterly penetration test

Question 5

An organization is evaluating 'Model Risk Management' (MRM) tools for its AI governance. What is the most critical feature to look for to ensure compliance with external regulatory requirements?

Accepted Answer

Comprehensive audit trails of model lineage, versions, and validation results.. External regulators require demonstrable traceability: who built the model, what data it used, which version is in production, and what validation it passed. Comprehensive audit trails of model lineage, versions, and validation results (B) provide exactly this evidence and are the cornerstone of MRM compliance. The other options are cosmetic or irrelevant to regulatory requirements.

Answer

Automatic social media integration.

Answer

The ability to run models on mobile devices.

Answer

The ability to add custom skins to the UI.

Question 6

A project lead argues that AI model performance metrics alone are sufficient for risk management. Why is this incorrect?

Accepted Answer

They do not capture social and compliance risk dimensions. Model performance metrics (accuracy, precision, recall, latency) measure technical behavior but say nothing about whether the model complies with regulations, treats protected groups fairly, respects privacy, or aligns with organizational values. Risk management must span social, ethical, legal, and compliance dimensions that performance metrics cannot capture. Relying on performance alone leaves material risk unmanaged.

Answer

They change too quickly to report

Answer

They are too difficult to calculate for small models

Answer

They are too subjective

Question 7

A developer is using 'Shadow Deployments' for a new AI model. What is the main risk being addressed?

Accepted Answer

Operational risk of deploying a poorly performing model to production.. Shadow deployments allow real-world data validation without exposing end-users to potential model failures.

Answer

Model documentation compliance.

Answer

Training data leakage.

Answer

Hardware infrastructure costs.

Question 8

To mitigate 'Training-Serving Skew' in the AI lifecycle, what is the best practice for data processing pipelines?

Accepted Answer

Use a feature store to ensure consistent feature engineering logic for both stages.. Training-serving skew occurs when the feature computation logic differs between the training pipeline and the inference pipeline, producing inconsistent feature values. A feature store (A) centralizes feature definitions and serves the same computed values to both training and online inference, eliminating the divergence. This is the industry-standard mitigation because it enforces a single source of truth for feature engineering.

Answer

Write separate code for training and inference to optimize for speed.

Answer

Use only raw data for both training and inference.

Answer

Train the model on the production server directly.

Question 9

An organization discovers that an AI model has learned a bias from training data. Per the AI Risk framework, what should be the immediate priority of the Governance function?

Accepted Answer

Suspend the model's usage and conduct a formal root cause analysis.. When bias is discovered in a model, the Governance function's immediate priority is to suspend the model's usage to prevent ongoing harm and then conduct a formal root cause analysis to determine how the bias entered and how to remediate it. This aligns with the AI risk framework's emphasis on containment before correction, protecting affected stakeholders and preserving evidence. Waiting or retraining on the same data would perpetuate the harm.

Answer

Wait for the quarterly board report to inform the board.

Answer

Fire the AI development team.

Answer

Retrain the model on the same data set to see if it fixes itself.

Question 10

The AI Risk Governance Committee is defining the risk appetite for a new generative AI chatbot. Which THREE factors must be considered to align with organizational risk tolerance?

Accepted Answer

Data privacy and residency requirements for training sets. Option B is correct because data privacy and residency requirements directly shape risk appetite: regulations like GDPR, HIPAA, or data-localization laws determine what training data may be used and where it can be stored, which bounds the acceptable risk for the chatbot. Option C is correct because model drift is a core AI risk factor — as data distributions shift, output accuracy degrades, so the committee must weigh the likelihood of drift against the organization's tolerance for degraded performance. Option D is correct because hallucination is a generative-AI-specific risk whose potential impact on brand reputation and customer trust must be assessed against the organization's risk tolerance. Option A is not a risk-appetite factor; sprint cadence is a delivery-process metric, not a measure of AI risk exposure. Option E is also not a risk-tolerance factor; GPU infrastructure cost is a budgeting concern, not a determinant of acceptable risk.

Answer

The speed of the development team's sprint cycles

Answer

The total cost of GPU infrastructure

Question 11

When defining KPIs for an AI Governance program, which metric is most predictive of long-term model robustness?

Accepted Answer

The frequency of model retraining triggered by performance drift alerts.. The frequency of model retraining triggered by performance drift alerts is the most predictive metric of long-term robustness because it reflects the organization's ability to detect degradation and respond with corrective action. A governance program that tracks this metric demonstrates closed-loop monitoring and remediation, which sustains model performance over time as data distributions shift. This aligns with MLOps best practices for drift detection and continuous improvement.

Answer

Number of users accessing the AI interface.

Answer

Total number of AI models currently in production.

Answer

The budget spent on GPU cloud computing resources.

Question 12

In the context of the AI RMF, what is the significance of the 'Measure' function for risk management?

Accepted Answer

To evaluate the model's performance and risk outcomes against established benchmarks.. The Measure function of the AI RMF is responsible for evaluating AI systems' performance, trustworthiness characteristics, and risk outcomes against defined benchmarks and metrics. It provides the quantitative and qualitative evidence needed to determine whether risks are within tolerance and whether the system meets its intended purpose. This function operationalizes the risk assessment by turning identified risks into measurable indicators.

Answer

To identify which employees have access to the AI system.

Answer

To set the budget for future AI projects.

Answer

To provide a score for the developers' performance.

Question 13

Which THREE strategies are effective for mitigating 'Adversarial Risk' in an AI lifecycle?

Accepted Answer

Rate limiting on API calls to prevent brute-force querying.. Option B is correct because rate limiting on API calls throttles the query volume an attacker can send, which directly counters brute-force and model-extraction attacks that rely on high-volume probing of the inference endpoint. Option D is correct because adversarial training explicitly augments the training set with perturbed examples, hardening the model's decision boundaries against evasion and adversarial-input attacks. Option E is correct because input sanitization and validation on all model endpoints rejects malformed, out-of-distribution, or injection-style payloads before they reach the model, reducing the attack surface for adversarial manipulation. Option A does not belong because adding documentation staff addresses process and knowledge management, not adversarial manipulation of the model. Option C does not belong because a more expensive cloud provider changes cost, not the security posture against adversarial inputs.

Answer

Hiring more documentation staff.

Answer

Using a more expensive cloud provider.

Question 14

When setting up an AI risk monitoring dashboard, what is the most important consideration for ensuring executive buy-in?

Accepted Answer

Map AI performance metrics to business impact indicators like customer churn or operational downtime.. Executive buy-in depends on demonstrating business value, not technical detail. Mapping AI performance metrics to business impact indicators such as customer churn, revenue, or operational downtime translates model behavior into outcomes executives already care about and are measured on. This makes the dashboard a decision-support tool rather than an engineering report.

Answer

Include the number of GPU hours utilized.

Answer

Display the raw training error rates.

Answer

Show a comparison of the organization's AI progress against competitors.

Question 15

When an AI model fails in production, what is the 'Post-Mortem' goal?

Accepted Answer

To identify root causes and update risk management practices. A post-mortem's goal is to identify root causes of the failure and feed those lessons back into risk management practices, controls, and monitoring. It is a blameless, learning-oriented exercise that improves future resilience rather than assigning fault. This aligns with AAIR's emphasis on continuous improvement of the AI risk management lifecycle.

Answer

To hide the failure from management

Answer

To fire the developer responsible

Answer

To delete all logs associated with the model

Question 16

In the context of the 'AI Lifecycle', what is the risk of 'Model Decay'?

Accepted Answer

The model's predictions become less accurate because the underlying environment or user behavior changes.. Model decay (also called model drift or concept drift) is the degradation of a model's predictive accuracy over time because the real-world environment, data distributions, or user behavior change after training. The model itself is unchanged, but its assumptions no longer match reality. This is why ongoing monitoring and retraining are essential in the AI lifecycle.

Answer

The model physically degrades over time.

Answer

The model's code becomes deprecated due to language updates.

Answer

The training hardware becomes too slow to run.

Question 17

An organization discovers that their AI model exhibits bias toward a specific demographic. What is the most important step in the Incident Response process?

Accepted Answer

Suspend the model's production use until the issue is investigated and remediated.. When an AI model is found to exhibit harmful bias, the immediate priority in incident response is containment — suspending production use prevents ongoing discriminatory decisions, legal exposure, and harm to affected individuals while the root cause is investigated. This mirrors the 'containment before eradication' principle in standard IR frameworks (NIST SP 800-61). Retraining or deleting data without first stopping the model leaves the biased system live and compounding harm.

Answer

Retrain the model immediately.

Answer

Delete all training data related to the demographic.

Answer

Issue a public press release.

Question 18

You are managing a model that utilizes 'Online Learning'. What is the most critical risk requiring constant lifecycle vigilance?

Accepted Answer

Adversarial data poisoning and instability.. Online learning continuously updates model parameters from streaming data, which means the model's training distribution is directly influenced by whatever data arrives in production. This creates a critical, ongoing risk of adversarial data poisoning — attackers can inject malicious samples into the live stream to corrupt the model — as well as instability from feedback loops and non-stationary data. Constant lifecycle vigilance is required because the model can degrade or be subverted between reviews.

Answer

Hardware failure in the training cluster.

Answer

Data drift from batch updates.

Answer

Memory exhaustion due to high throughput.

Question 19

You are utilizing Azure Machine Learning to manage a deployment. You detect a sudden drop in model performance due to 'concept drift'. Which specific configuration in the Azure ML Model Monitoring dashboard should be adjusted to better detect this?

Accepted Answer

Adjust the feature drift sensitivity threshold for categorical variables.. Concept drift detection in Azure Machine Learning model monitoring is configured through drift thresholds on features and predictions. For categorical variables, the relevant setting is the feature drift sensitivity threshold, which controls how much distribution shift triggers an alert. Adjusting this threshold for categorical features allows the monitor to detect concept drift manifesting as changes in categorical input distributions.

Answer

Increase the retraining frequency of the pipeline.

Answer

Disable the data lineage capture to improve latency.

Answer

Switch the model to a higher-capacity compute instance.

Question 20

You are configuring the NIST AI Risk Management Framework (AI RMF) 'Govern' function for a high-risk autonomous system. Which activity specifically fulfills the 'Govern' function's requirement for establishing accountability?

Accepted Answer

Formalizing the AI System Model Owner role with defined risk acceptance and oversight accountabilities.. The NIST AI RMF 'Govern' function (GV) establishes the policies, roles, and accountability structures for AI risk management. Formalizing the AI System Model Owner role with defined risk acceptance and oversight accountabilities directly fulfills GV 1.2 (accountability structures) and GV 1.4 (documented roles). The other options are Map, Measure, or Manage function activities, not Govern.

Answer

Performing a red-teaming exercise on the model inputs.

Answer

Configuring automated PII scrubbing in the data pipeline.

Answer

Reviewing the training dataset for bias using statistical sampling.