AAIA Practice Test 16 — 15 Questions

Question 1

In a financial services firm, you are auditing the 'Explainability' of a credit scoring model. The model uses complex non-linear features. Which technique should you verify is being used to provide local explanations for individual credit decisions?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

SHAP values. SHAP (SHapley Additive exPlanations) is a widely accepted, model-agnostic technique for providing local explanations of individual predictions.

Answer

Feature Importance Permutation

Answer

Model pruning

Answer

Data normalization

Question 2

During an audit of a model inference endpoint, you observe that 'Model Explanation' logs are missing. Which library or tool was likely omitted from the deployment?

Accepted Answer

SHAP integration. Tools like Captum (PyTorch), SHAP, or SageMaker Clarify are required to generate inference-time explanations.

Answer

Kubernetes manifest

Answer

TensorBoard logging

Answer

Prometheus exporter

Question 3

An auditor is evaluating an AI system's robustness against adversarial attacks. Which technique involves perturbing input data to see if the model output changes significantly?

Accepted Answer

Adversarial testing. Adversarial testing involves small perturbations to inputs to identify instability or security weaknesses.

Answer

Model pruning

Answer

Bias assessment

Answer

Data augmentation

Question 4

When auditing the 'training dataset' for a classification model, what should the auditor confirm regarding class balance?

Accepted Answer

That classes are equally represented or addressed. Class imbalance can cause the model to be biased toward the majority class; the auditor should check for techniques like re-sampling or cost-sensitive learning.

Answer

That all data is in CSV format

Answer

That the dataset size is at least 1TB

Answer

That training time is minimized

Question 5

An auditor is evaluating bias in a model deployed on AWS SageMaker. Which tool should be used to generate a report on pre-training and post-training bias metrics?

Accepted Answer

SageMaker Clarify. SageMaker Clarify provides specific bias detection capabilities for both training data and model predictions.

Answer

SageMaker Model Monitor

Answer

AWS Glue DataBrew

Answer

Amazon Inspector

Question 6

During an audit of an LLM-based application, which technique is most effective for testing the robustness against 'Prompt Injection'?

Accepted Answer

Red Teaming. Red Teaming involves manual and automated attempts to bypass safety filters and trigger malicious outputs.

Answer

Sentiment analysis

Answer

Cross-validation

Answer

Tokenization audit

Question 7

During an audit, you find that the AI training data includes personally identifiable information (PII) that was not anonymized. Which regulation does this most likely violate?

Accepted Answer

GDPR (General Data Protection Regulation). Handling PII without proper anonymization violates privacy regulations like GDPR or CCPA.

Answer

The Model Building Standards Act

Answer

The Sarbanes-Oxley Act

Answer

The Internal Server Maintenance Code

Question 8

When auditing fairness metrics, what does the 'Demographic Parity' metric measure?

Accepted Answer

Equal positive outcome rates. Demographic Parity requires that the probability of a positive outcome be equal across different demographic groups.

Answer

Equal training data representation

Answer

Equal error rates

Answer

Equal accuracy rates

Question 9

Which document is essential for an AI audit to identify the 'intended use' of an AI system?

Accepted Answer

Model Card. The Model Card is a standardized document that lists intended use, limitations, and performance metrics.

Answer

Server Hardware Specs

Answer

Financial Statement

Answer

Employee Training Log

Question 10

You are assessing a company's 'AI Governance Committee'. Which action best demonstrates effective board-level oversight?

Accepted Answer

Approving the annual AI strategy and corresponding risk mitigation plans. Board oversight must focus on strategic alignment and the establishment of high-level accountability frameworks.

Answer

Managing the day-to-day procurement of data labeling services

Answer

Reviewing individual code commits for AI model training

Answer

Selecting the machine learning framework used by data scientists

Question 11

When drafting an AI policy, you need to define the 'Human-in-the-Loop' (HITL) requirement. Which of the following represents a best-practice control for HITL?

Accepted Answer

Manual review of high-risk model outputs prior to execution. A requirement for human intervention in high-risk decisions ensures oversight and accountability.

Answer

Logging all model interactions for post-mortem analysis

Answer

Setting all model confidence thresholds to 100%

Answer

Automated approval for all decisions

Question 12

Which aspect of 'Data Drift' does the 'Population Stability Index' (PSI) measure?

Accepted Answer

Change in population distribution. PSI measures how much the distribution of a variable has changed between the training set and the production dataset.

Answer

Change in model parameters

Answer

Change in model accuracy

Answer

Increase in inference latency

Question 13

Which THREE actions are appropriate when you find a significant 'Bias' finding in an AI audit?

Accepted Answer

Recommend retraining with re-sampled data. Documenting the issue, notifying stakeholders, and recommending mitigation (re-sampling or retraining) are the appropriate steps.

Answer

Ignore the finding if the model is fast

Answer

Delete the audit evidence

Question 14

When auditing an AI system's 'Explainability', which issue is highlighted by 'Explanation Faithfulness'?

Accepted Answer

Whether the explanation accurately represents the model's logic. Faithfulness measures whether the explanation accurately reflects the model's internal decision process rather than just being a plausible simplification.

Answer

Whether the model is fast enough

Answer

Whether the user trusts the model

Answer

Whether the explanation is easy to read

Question 15

When auditing an AI pipeline, which THREE items should be part of the 'Evidence Collection' plan?

Accepted Answer

Access and error logs from the deployment server. Logs, configuration files, and model artifacts are the three core pieces of evidence needed for an audit.

Answer

The personal notes of the lead developer

Answer

A list of all team member salaries