AAIA Practice Test 18 — 15 Questions

Question 1

You are auditing a financial institution's AI governance framework. The organization has a documented AI risk appetite statement, but you find that individual AI use cases are approved without any reference to it. Which action should you recommend to remediate this gap?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Require each AI use case to be mapped to the enterprise AI risk appetite and approved by the AI governance committee.. The correct remediation is to embed the enterprise AI risk appetite into the approval workflow for each AI use case. This ensures that risk tolerance is actively applied at the point of decision, and that the AI governance committee provides necessary oversight. Simply updating the statement or adding post-hoc reviews does not close the gap of use cases being approved without reference to the appetite.

Answer

Delegate AI use case approval to the data science team to speed up innovation.

Answer

Implement a post-implementation review process to assess AI use cases after deployment.

Answer

Update the AI risk appetite statement to include more granular risk categories and thresholds.

Question 2

An auditor is reviewing an organization's AI risk management process. The organization has identified several AI risks but has not prioritized them. Which TWO factors should the auditor recommend be used to prioritize AI risks? (Choose two.)

Accepted Answer

The likelihood of the risk materializing and its potential impact on business objectives.. AI risks should be prioritized based on their likelihood and potential impact, including regulatory and legal consequences. These factors help determine which risks pose the greatest threat to the organization and should be addressed first. Other factors like team size, popularity, or cloud costs are not relevant to risk prioritization and could lead to misallocation of resources.

Answer

The popularity of the AI model among end users.

Answer

The number of data scientists assigned to the AI project.

Answer

The cost of the AI project's cloud infrastructure.

Question 3

An auditor is testing whether a recommendation model leaks information about individual users in its training set. The team exposes only top-N ranked outputs through the API. Which technique is most appropriate to estimate membership inference risk under this constraint?

Accepted Answer

Shadow-model attack using the ranked outputs to train an inference classifier. Membership inference risk in a black-box setting is best estimated empirically by simulating an attacker who sees only outputs. A shadow-model attack builds surrogate models and a membership classifier from observed rankings, producing a measurable advantage over random guessing. White-box weight inspection, differential privacy accounting, and adversarial perturbation either require unavailable access or address an unrelated risk.

Answer

Adversarial example generation against the ranking endpoint

Answer

Direct inspection of the model's internal weight matrices for memorized user identifiers

Answer

Differential privacy accounting of the training procedure

Question 4

An audit team is reviewing a production AI system that performs real-time fraud detection. The system's output is a probability score between 0 and 1, and a threshold of 0.8 is applied to classify transactions as fraudulent. The team discovers that the threshold was set during initial model development and has not been adjusted since. Over the past six months, the false positive rate has increased from 2% to 15%, causing customer friction. The model's AUC-ROC has remained stable at 0.92. What is the MOST likely cause of the increased false positives?

Accepted Answer

The threshold of 0.8 is no longer optimal because the underlying distribution of predicted probabilities has shifted, even though the model's discriminative power is unchanged.. The model's stable AUC-ROC indicates that its ability to rank transactions by risk has not degraded. However, the fixed threshold of 0.8 may no longer be appropriate if the distribution of predicted probabilities for legitimate transactions has shifted upward. This can happen due to changes in the base rate of fraud or other factors, leading to more legitimate transactions crossing the threshold. Auditors should recommend periodic threshold tuning based on current data and business costs.

Answer

The model is suffering from concept drift, where the relationship between features and the target variable has changed, leading to more false positives.

Answer

The model is experiencing data drift in the input features, which has degraded its ranking ability.

Answer

The model's training data was not representative, causing it to systematically misclassify legitimate transactions as fraudulent over time.

Question 5

A global retailer has deployed an AI-powered dynamic pricing engine across 14 countries. The internal audit team discovers that the system adjusts prices based on inferred customer gender and ethnicity derived from browsing behavior. The organization has a formal AI ethics policy that prohibits discriminatory pricing. Which action should the lead AI auditor recommend FIRST?

Accepted Answer

Document the policy violation and escalate it through the AI governance committee for a formal risk assessment and remediation decision.. The auditor's role is to identify and escalate governance failures, not to remediate them directly. Escalating through the AI governance committee ensures the violation is evaluated against risk appetite and that remediation is properly authorized. This maintains auditor independence and triggers the organization's formal response mechanisms, which is the correct first action when a deployed AI system violates stated ethical policy.

Answer

Immediately suspend the pricing engine in all 14 countries until a full bias audit is completed by an independent third party.

Answer

Reconfigure the model to remove gender and ethnicity variables and revalidate it before reporting the finding to management.

Answer

Issue a qualified audit opinion on the AI ethics program and schedule a follow-up review in the next audit cycle.

Question 6

An AI audit team is validating a fraud-detection model deployed through a REST API. The model card claims 94% recall, but the auditor wants independent evidence that the live endpoint behaves as documented. The team has read access to the model registry and the API logs, but cannot access the training pipeline. Which technique should the auditor use to validate the deployed model's behavior?

Accepted Answer

Black-box API testing with a curated set of labeled probe inputs sent to the live endpoint.. When the auditor cannot inspect training internals but can reach the live endpoint, black-box testing with labeled probes provides independent, reproducible evidence about actual serving behavior. It validates the deployed artifact directly, detects discrepancies between documentation and production, and does not depend on the model owners' cooperation or internal artifacts.

Answer

Reviewing the training pipeline's source code and unit tests stored in the version control repository.

Answer

Recomputing the model's performance metrics from the original held-out test set.

Answer

Interviewing the data science team about their validation methodology and acceptance criteria.

Question 7

An auditor is examining a computer vision model used for medical image triage. The vendor provides only a compiled binary and no architecture documentation. The auditor wants to estimate how much the model's output changes when individual pixels are altered slightly, in order to assess decision boundary sharpness. Which technique is most appropriate?

Accepted Answer

Finite-difference sensitivity probing by perturbing individual pixels and measuring output change.. Finite-difference sensitivity probing measures how much the output changes when individual pixels are perturbed, producing an empirical estimate of local decision boundary sharpness. It requires no internal access, works on compiled binaries, and generates quantitative evidence. This makes it the most appropriate technique when architecture documentation is unavailable but output behavior must be characterized.

Answer

Training a surrogate decision tree to mimic the binary's outputs.

Answer

Gradient-based saliency mapping using the model's internal weights.

Answer

Reviewing the vendor's ISO/IEC 42001 certification and conformity assessment report.

Question 8

An AI governance committee is reviewing a proposal to deploy a generative AI chatbot for customer service. The committee wants to ensure that the chatbot's responses do not violate company policies or regulatory requirements. Which control should be implemented to provide the highest level of assurance?

Accepted Answer

Implement real-time content filtering and human-in-the-loop review for high-risk interactions.. The highest assurance is achieved through a combination of real-time content filtering and human-in-the-loop review for high-risk interactions. This provides continuous monitoring and intervention, ensuring that the chatbot's responses align with policies and regulations. One-time exercises or disclaimers do not offer ongoing protection, and relying on the foundation model provider does not transfer compliance accountability.

Answer

Publish a disclaimer that the chatbot's responses are not official company statements.

Answer

Rely on the chatbot's foundation model provider to ensure compliance.

Answer

Conduct a one-time red teaming exercise before deployment.

Question 9

An auditor is reviewing an AI-based loan underwriting system and wants to verify that the model's predictions are stable when small, non-semantic changes are made to input features. The auditor plans to submit the same applicant profile with trivial formatting variations. Which auditing technique does this describe?

Accepted Answer

Input perturbation testing for prediction consistency.. Input perturbation testing with semantically equivalent variations isolates inference-time stability. It reveals brittle preprocessing, encoding inconsistencies, and non-deterministic serving behavior that can cause identical applicants to receive different decisions. This makes it a direct and appropriate technique for verifying that the underwriting model behaves consistently under trivial input changes.

Answer

Backtesting the model against historical loan default rates.

Answer

Feature importance ranking using SHAP values.

Answer

Sensitivity analysis of the training loss curve.

Question 10

An auditor needs to confirm that a deployed model in production is bit-for-bit identical to the artifact that passed validation. Which artifact should the auditor use to verify integrity?

Accepted Answer

The model's cryptographic hash recorded in the model registry. Verifying that a production model equals the validated artifact requires a content-based fingerprint. A cryptographic hash stored at registration and recomputed on the deployed file gives objective, reproducible evidence of identity. Scripts, model cards, and run identifiers provide context and lineage but cannot detect a substituted or altered artifact, so they are insufficient for integrity assurance.

Answer

The model card summarizing intended use and limitations

Answer

The experiment tracking run identifier assigned during training

Answer

The training script stored in the source-control repository

Question 11

A retail company has deployed an AI-powered recommendation engine. The AI audit team is reviewing the system's operational metrics and finds that the model's prediction latency has gradually increased over three months. Which operational metric should the team analyze to determine if the increase is due to growing model complexity or infrastructure issues?

Accepted Answer

CPU and memory utilization of the inference service. To diagnose a gradual latency increase, the most relevant operational metrics are CPU and memory utilization of the inference service. These indicate whether the infrastructure is becoming a bottleneck due to increased load, memory leaks, or insufficient scaling. By analyzing these metrics alongside traffic patterns, the audit team can determine if the cause is infrastructure-related or requires model optimization.

Answer

Number of active users per day

Answer

Model accuracy over time

Answer

Model size and number of parameters

Question 12

An AI audit is being conducted for a natural language processing model used in a customer service chatbot. The model was trained on historical chat logs and deployed six months ago. The audit team notices that the model's responses have become increasingly irrelevant and sometimes offensive. The team reviews the model's input pipeline and finds that the tokenization step uses a fixed vocabulary that was built from the training data. However, new slang and abbreviations are frequently used by customers, and these are not in the vocabulary. What is the PRIMARY operational control that should have been implemented to prevent this issue?

Accepted Answer

Regular retraining of the model on a rolling window of recent chat logs to update the vocabulary and adapt to new language patterns.. The model's fixed vocabulary, built from historical data, cannot represent new slang and abbreviations, leading to out-of-vocabulary tokens and degraded performance. Regular retraining on recent data is the primary operational control to update the vocabulary and adapt to evolving language. This ensures the model remains effective and reduces the risk of irrelevant or offensive responses. Other controls like filters or parameter tuning are secondary and do not address the root cause.

Answer

Implementing a rule-based filter that blocks offensive words before they reach the model.

Answer

Increasing the model's temperature parameter to make its responses more diverse and less likely to be offensive.

Answer

Deploying a separate sentiment analysis model to flag potentially offensive responses for human review.

Question 13

During an audit of a natural language processing system, the auditor needs to determine whether a fine-tuned model reproduces verbatim passages from its training corpus. Which technique should the auditor apply to test for training-data extraction?

Accepted Answer

Prompt the model with corpus prefixes and measure longest common subsequence against the source text. Testing for training-data extraction requires prompting the model with corpus fragments and checking whether it regenerates the original text. Quantifying overlap with the longest common subsequence provides objective evidence of verbatim reproduction. Perplexity, fairness metrics, and performance profiling measure different properties and cannot demonstrate that specific training passages are recoverable from the model.

Answer

Run a fairness metric such as demographic parity across generated outputs

Answer

Profile inference latency and throughput under production load

Answer

Compute the model's perplexity on a held-out portion of the same corpus

Question 14

An AI auditor is evaluating a natural language processing model used to screen job applicants. The model was trained on five years of historical hiring data. The auditor finds that the model's selection rate for female candidates is 72% of the rate for male candidates. According to the EEOC Uniform Guidelines, what is the auditor's PRIMARY concern?

Accepted Answer

The selection rate disparity indicates potential disparate impact, and the auditor should investigate whether the model uses proxies for gender.. The EEOC four-fifths rule provides a benchmark: a selection rate below 80% of the highest group's rate suggests disparate impact. With a 72% ratio, the auditor must investigate potential proxy discrimination. The primary concern is the legal and ethical risk of disparate impact, not model accuracy or immediate retraining. The auditor's finding should trigger a deeper review of features, training data, and validation results.

Answer

The model's use of historical hiring data is acceptable because it reflects actual past outcomes and therefore is objective.

Answer

The model should be retrained immediately using a balanced dataset with equal numbers of male and female applicants.

Answer

The model's overall accuracy metric is below the 90% threshold required for employment decisions.

Question 15

An organization has deployed a recommendation model that suggests products to users on its e-commerce platform. The model is retrained weekly using the latest user interaction data. The audit team is reviewing the model's operational logs and finds that the retraining process sometimes fails silently, and the previous week's model continues to serve predictions. There is no alerting mechanism for retraining failures. What is the MOST appropriate control to implement to address this issue?

Accepted Answer

Implement automated monitoring and alerting for the retraining pipeline to notify the team of failures.. The retraining pipeline can fail silently, leaving a stale model in production. The most appropriate control is to implement automated monitoring and alerting for the pipeline. This ensures that failures are detected immediately, allowing the team to investigate and remediate. Other options like increasing retraining frequency or manual review do not provide timely detection. Rollback is a remediation step that depends on detection. Therefore, monitoring and alerting is the primary control.

Answer

Roll back to a previous model version if the current model's performance drops below a threshold.

Answer

Increase the frequency of retraining to daily to reduce the impact of any single failure.

Answer

Manually review the model's performance metrics each week to detect any degradation.