ISACA Advanced in AI Audit (AAIA) (AAIA) AAIA Questions 226–230 | Page 4/4

226

MCQmedium

You are reviewing the AI Incident Response Plan. Which scenario constitutes a 'High' severity AI incident that requires immediate board notification under most enterprise risk frameworks?

A.A 200ms latency increase during inference

B.Systemic bias detected in a core automated lending model

C.A minor change in the training dataset size

D.A 2% drop in model prediction accuracy

AnswerB

Systemic bias in a core lending model causes discriminatory credit decisions across protected groups, breaching fair-lending regulation and affecting many customers. That enterprise-wide harm and legal exposure elevates it to High severity, triggering immediate board notification.

Why this answer

An incident where a model makes a biased decision impacting a protected class carries significant regulatory and reputational risk, requiring board visibility.

Full explanation →

227

MCQeasy

What is the goal of an AI 'Rollout' plan?

A.Reducing disk space

B.Deleting logs

C.Controlled deployment to users

D.Increasing training time

AnswerC

A rollout plan governs the controlled, staged release of an AI solution to users, moving it from pilot to production in manageable increments. This satisfies the stem's goal by ensuring adoption, monitoring and risk mitigation occur progressively, rather than exposing the entire user base to an untested deployment at once.

Why this answer

An AI rollout plan governs the controlled deployment of a model to users, typically through phased stages such as pilot, limited release, and general availability. The goal is to manage risk, gather feedback, and ensure readiness before full-scale exposure. It is fundamentally a deployment governance artifact, not a resource optimization or training activity.

Exam trap

AAIA often tests the misconception that rollout plans are about infrastructure or training optimization, when the core purpose is controlled, risk-managed deployment to users.

How to eliminate wrong answers

Option A is wrong because reducing disk space is an infrastructure optimization concern unrelated to rollout planning. Option B is wrong because deleting logs is a data retention action that would actually undermine rollout monitoring and auditability. Option D is wrong because increasing training time is a model development activity that occurs before rollout, not a goal of the rollout plan itself.

Full explanation →

228

Multi-Selectmedium

Which TWO of the following are examples of 'Data Leakage' that an auditor should look for in a pipeline?

Select 2 answers

A.Using cloud-based storage

B.Training on test data

C.Documenting model parameters

D.Using open-source libraries

E.Including target-related features that are not available at inference time

AnswersB, E

Training on test data leaks evaluation information into the model, inflating performance metrics because the model has already seen the holdout examples. This directly satisfies the stem's leakage criterion: the test set is no longer independent, so reported accuracy becomes an unreliable audit signal.

Why this answer

Option B is correct because training on test data is a classic form of data leakage: information from the evaluation set contaminates the training process, so the model's reported performance is optimistically biased and does not reflect true generalization to unseen data. Option E is correct because including target-related features that are not available at inference time (e.g., using the label or a post-outcome variable as an input) leaks future/target information into training, producing a model that cannot be reproduced or used in production. Option A is not leakage per se; using cloud-based storage is an infrastructure choice that may raise other security or compliance concerns but does not itself contaminate training with evaluation or target information.

Option C is not leakage; documenting model parameters is a governance and reproducibility practice. Option D is not leakage; using open-source libraries is a normal engineering decision that may introduce supply-chain risk but does not constitute data leakage.

Full explanation →

229

MCQhard

If a model's performance drops, and the input data distribution has changed, what operational process must be triggered?

A.Security audit

B.Model Deletion

C.Hardware Upgrade

D.Code Refactoring

E.Model Retraining

AnswerE

Distribution shift means the learned mapping no longer matches live inputs, so retraining on recent data restores alignment. Monitoring detects drift; retraining is the operational process that updates weights. Without it, accuracy degrades silently as the input distribution diverges from training.

Why this answer

Data distribution change triggers model retraining using updated data samples.

Full explanation →

230

MCQhard

When implementing drift detection in AWS SageMaker Model Monitor, which metric should be monitored to detect feature attribution shift for structured data?

A.Cosine Similarity

B.Mean Square Error

C.Feature Attribution Drift (SHAP)

D.Precision-Recall AUC

AnswerC

Feature attribution drift applies SHAP values to compare each feature's contribution between baseline and current data, revealing changes in how features influence predictions. Monitoring it satisfies the requirement to detect feature attribution shift for structured datasets in SageMaker Model Monitor.

Why this answer

Feature Attribution Drift, computed using SHAP (SHapley Additive exPlanations) values, measures how the contribution of each input feature to the model's prediction changes over time relative to a baseline. This is the correct metric for detecting feature attribution shift in structured data within SageMaker Model Monitor. It captures not just input distribution changes but changes in how the model weighs features.

Exam trap

The trap is confusing performance metrics (MSE, PR-AUC) with drift metrics — performance metrics require ground truth labels and measure accuracy, while attribution drift measures how the model's reasoning changes, independent of labels.

How to eliminate wrong answers

Option A is wrong because Cosine Similarity measures vector angle similarity and is used in NLP/embedding contexts, not for tabular feature attribution drift. Option B is wrong because Mean Square Error is a regression performance metric, not a drift detection metric — and it requires ground truth labels. Option D is wrong because Precision-Recall AUC is a classification performance metric, also requiring labels, and does not measure feature attribution shift.

Full explanation →

ISACA Advanced in AI Audit (AAIA) (AAIA) (AAIA) — Questions 226–230