ISACA Advanced in AI Audit (AAIA) (AAIA) (AAIA) — Questions 1–75

A full audit log of data provenance records every transformation and access event across the pipeline, giving the auditor verifiable traceability of data origin and movement. This directly satisfies the pipeline security evaluation by evidencing accountability and detecting tampering.

Environment objects capture the exact Python packages, environment variables and system dependencies used during training, so jobs can be reproduced identically across compute targets. This satisfies the reproducibility constraint by pinning the runtime state rather than relying on whatever libraries happen to be installed.

An Endpoint Configuration specifies the model variants to deploy and the compute resources backing each one, including instance type and count. This satisfies the stem's requirement by separating deployment settings from the model artefact itself, letting you host one or many models behind a single endpoint without retraining.

SHAP values apply a game-theoretic attribution that distributes each prediction's margin among input features, giving a locally faithful, additive explanation for a single scored applicant. This satisfies the regulator's need for case-by-case reasoning while remaining consistent with the ensemble's actual output, so the auditor can tie each feature contribution back to the decision.

Data source connectivity is a critical availability dependency: if the pipeline cannot reach its upstream sources, no inference or retraining can occur. Monitoring this connection detects outages before they cascade, directly supporting the operational availability the stem requires.

Counterfactuals generates perturbed feature values and shows the smallest change needed to flip a prediction, which is exactly the counterfactual analysis the audit requires. Other Responsible AI components (error analysis, fairness, interpretability) do not produce these what-if contrasts.

A stable AUC-ROC means the model still ranks fraudulent transactions higher than legitimate ones, but the calibration of predicted probabilities may have shifted. If the distribution of scores for legitimate transactions has moved higher, a fixed threshold of 0.8 will classify more legitimate transactions as fraudulent, increasing false positives. Regular threshold tuning is needed to align with current data and business costs.

Legal Counsel provides oversight of regulatory compliance, contractual obligations and liability exposure within the AI governance framework. Their involvement ensures AI systems align with applicable laws and industry regulations, forming one of the three required oversight roles.

Label drift means the distribution of the target variable itself shifts over time, so monitoring the target distribution with Population Stability Index detects that movement directly. Feature drift metrics would miss it, since inputs can remain stable while the label distribution changes.

A model repository functions as a versioned registry, storing each model iteration with immutable identifiers, lineage metadata and artefacts. This versioning satisfies the audit requirement for traceability, letting auditors reconstruct which exact model version produced a given decision and verify its approval history.

Epsilon quantifies the privacy loss bound in differential privacy: smaller values permit less distinguishability between neighbouring datasets, consuming more of the privacy budget. It directly controls how much information any single query may reveal about an individual.

The inference request ID uniquely identifies each model invocation, enabling end-to-end traceability across the AI system log. It satisfies the audit trail constraint by correlating inputs, outputs, model versions and timestamps to a single request, so investigators can reconstruct exactly what the system processed and returned.

A cryptographic hash of the model artifact, stored when the model was registered, provides a compact fingerprint that can be recomputed against the production artifact. If the hashes match, the auditor has objective evidence that the deployed file is byte-identical to the validated version, which is exactly the integrity assurance required.

LIME builds a locally faithful surrogate model around a single prediction, exposing which features drove that specific outcome. Auditors use this local decision transparency to verify individual decisions rather than global model behaviour, satisfying the need to explain why one case received its result.

Model training logs record datasets, hyperparameters, code versions and timestamps used to produce each model. They satisfy the audit-trail requirement by evidencing how a model was derived, enabling reproducibility and tracing of decisions back to the training process.

Adversarial perturbation testing deliberately applies small, crafted input changes to measure whether predictions shift, directly quantifying robustness. It targets the model's decision boundary under malicious manipulation, which is the specific threat the audit must assess.

The four-fifths rule states that a selection rate for any protected group that is less than 80% of the rate for the highest group may indicate disparate impact. A 72% ratio falls below this threshold, requiring the auditor to examine whether the model relies on features that correlate with gender. This investigation is the primary concern because it determines whether the model violates employment discrimination law.

Containerisation packages an application with its dependencies, libraries and runtime into a single immutable image, so the same artefact executes identically across development, testing and production. This directly satisfies the environment consistency requirement, eliminating "works on my machine" drift caused by differing host libraries or Python versions.

Model lineage records the provenance of the final model: its training data, code, parameters, and the transformations applied across each version. This documented chain lets auditors trace how the deployed model was produced and verify the artefacts behind it.

Request latency measures live system behaviour, unlike business or model-quality metrics. It captures the time to serve each inference request, directly reflecting operational health and user experience. This makes it an operational metric rather than an accuracy or revenue indicator.

SageMaker Model Monitor captures endpoint data and compares it against a baseline computed from training data, emitting violations when feature distributions shift. That statistical comparison of live versus baseline distributions is precisely the data drift detection the stem requires.

Insufficient model documentation is a recurring governance finding because auditors cannot verify training data, assumptions, or version history without it. Missing documentation breaks traceability and reproducibility requirements, making it a typical deficiency recorded in AI governance reviews.

Re-sampling adjusts the training data distribution so under-represented groups gain proportionate weight, directly correcting the skew that produced the bias finding. Retraining on this rebalanced dataset addresses the root cause rather than masking symptoms, making it an appropriate remediation action following the audit.

Adversarial attacks exploit model inputs to force misclassification, so susceptibility is a genuine model risk requiring documentation. It concerns the model's own robustness, distinct from data or operational risk, and must be recorded in the audit evidence.

Real-time content filtering can block policy-violating responses, while human-in-the-loop review for high-risk interactions adds a layer of human judgment. Together, they provide continuous assurance that the chatbot adheres to policies and regulations. This combination addresses both automated and nuanced cases, offering the highest level of assurance among the options.

Operational performance degradation covers model drift, latency spikes, throughput loss and availability failures once an AI system is in production. Governing it explicitly satisfies the requirement to manage risks affecting reliability, cost and service continuity across the model lifecycle.

SHAP assigns each input feature a Shapley value derived from cooperative game theory, quantifying its marginal contribution to a specific prediction. This yields consistent, locally faithful attributions even for deep networks, satisfying explainability requirements that simpler global feature-importance scores cannot meet.

Shadow deployment runs a new model version in parallel with the live production model, receiving the same real traffic but with its predictions discarded rather than served. This lets teams validate behaviour, latency and accuracy on real data before promotion, without affecting users.

Without version control, prompt edits propagate instantly and silently, so output shifts cannot be traced to a specific revision. This directly satisfies the audit's missing-versioning constraint: rollback is impossible, A/B comparisons are invalid, and behavioural drift becomes undetectable, making model responses unpredictable across deployments.

Concept drift means the statistical relationship between inputs and target outputs changes over time, so patterns learned during training no longer match live data. Operational teams should expect declining predictive accuracy and schedule retraining or monitoring alerts.

Decreasing model accuracy directly evidences data or concept drift, where live input distributions diverge from training data, degrading predictive performance. Auditors monitoring production models should track accuracy metrics against baselines, since this symptom satisfies the stem's requirement for a common drift indicator detectable through routine performance monitoring.

The DAG encodes the execution order of pipeline tasks, where each node is a task and each directed edge is a dependency that must complete before its successor runs. This dependency structure is what lets orchestrators like Airflow schedule tasks correctly and detect cycles.

Global Feature Importance aggregates average feature effects across the whole dataset, so it can mask cases where a feature drives one prediction strongly but not others. Relying on it alone therefore fails to explain individual decisions, which matters for auditing specific outcomes.

Feature skew arises when training features are computed differently from serving features, so the model receives inputs at inference that diverge from what it learned. This mismatch directly degrades prediction accuracy, producing incorrect outputs. It satisfies the stem's risk framing: the pipeline's training-serving inconsistency manifests as unreliable, wrong model predictions.

Data anonymisation irreversibly removes or alters personally identifiable information, so retraining datasets no longer expose individuals. This satisfies the stem's data privacy requirement by preventing re-identification, unlike pseudonymisation, which retains a mapping key and therefore remains personal data under GDPR. It directly enables lawful secondary use of user-generated data.

Data lineage logs record the origin, movement and transformation of each dataset through the pipeline, giving auditors direct documentary evidence of where training data came from. This satisfies Provenance verification by tracing data back to its source systems and processing steps.

A context faithfulness score measures whether generated output is grounded in the retrieved source documents, directly detecting hallucination in real time. Latency, token throughput, or cost metrics cannot reveal grounding failure, so this metric uniquely satisfies the stem's requirement to identify hallucination during production serving.

Ethics and compliance policies codify acceptable use, risk tolerance and regulatory obligations, giving the governance framework its enforceable baseline. Without documented policy, oversight bodies lack criteria to approve, restrict or retire AI systems, so this component directly satisfies the stem's demand for robust governance structure.

Access and error logs from the deployment server provide timestamped, tamper-evident records of who invoked the pipeline and what failed, forming the primary audit trail. This satisfies the evidence-collection requirement by supplying verifiable operational artefacts for the AI pipeline audit.

Retaining training data that remains re-identifiable, without access controls, breaches privacy principles because personal information persists beyond its lawful purpose. Anonymisation or strict access governance is required, so this practice directly violates data protection obligations during pre-processing.

Champion-Challenger routes a portion of live traffic to the challenger model while the champion continues serving most requests, comparing performance on real data. This satisfies the stem's safety constraint: the new model is validated against production traffic without exposing all users to unproven behaviour, enabling evidence-based promotion or rollback.

Azure Machine Learning Model Monitor continuously evaluates deployed models, detecting data drift and tracking data quality metrics, and surfaces them through a central dashboard. This directly satisfies the Chief Risk Officer's requirement for consolidated drift and quality oversight.

SageMaker Lineage Tracking automatically records each training job's inputs, outputs and associated entities, producing an auditable graph linking models to their data sources. This directly satisfies the reproducibility requirement by evidencing lineage and versioning for every training job during AI risk assessment.

GDPR governs processing of personal data and requires anonymisation or a lawful basis; unanonymised PII in training data breaches its data protection principles. The scenario names PII specifically, which is the exact category GDPR regulates, making it the applicable regulation.

Human in the loop is a governance principle that ensures meaningful human control over AI decisions. It can take forms such as human review of flagged cases, approval of high-risk decisions, or the ability to override or reverse automated outcomes. The key is that the human has real authority and information to intervene, not merely a nominal role. This definition aligns with ISACA's emphasis on accountability in AI governance.

Vertex AI ML Metadata stores the lineage graph tracking datasets, training runs, models and artefacts, letting an auditor trace a model's provenance through each pipeline step. Inspecting it reveals the exact artefacts and executions that produced the model.

SageMaker Model Monitor continuously captures production inference data and compares its statistical distribution against a baseline, detecting drift and data quality violations. It satisfies the requirement for automated alerts when input data deviates, without custom infrastructure.

Defined communication protocols ensure timely, coordinated disclosure to internal teams, regulators, affected users and the public during an AI incident. This satisfies the stem's requirement for a core response component, since containment and recovery depend on clear escalation paths and stakeholder notification obligations, particularly for AI systems with safety, privacy or compliance impact.

SageMaker Clarify generates pre-training and post-training bias metrics, directly satisfying the auditor's requirement for both measurement stages. It computes metrics such as class imbalance and disparate impact before training, then compares predicted labels against ground truth afterwards, producing a consolidated bias report within SageMaker.

A model card documents a model's intended use, limitations, training data and evaluation results, giving auditors the authoritative statement of purpose. Data sheets, system cards or privacy impact assessments address different concerns and do not define intended use as directly.

Submitting semantically equivalent inputs with trivial formatting variations and checking that predictions remain stable is exactly input perturbation testing for consistency. It detects brittle preprocessing, unstable feature encodings, and non-deterministic serving paths. For loan underwriting, an applicant should not receive a different decision because a date format or whitespace changed, so this technique directly supports fair and reliable outcomes.

PSI quantifies how much a variable's distribution has shifted between a baseline and a comparison dataset, summing weighted differences across bins. It therefore measures change in population distribution, not feature importance, accuracy or concept drift in the target label.

Model Validation follows Model Training in the AI model lifecycle. After training completes, the model is evaluated against held-out data to confirm accuracy, fairness and generalisation before deployment, ensuring it meets defined performance criteria prior to release.

A Pod Disruption Budget guarantees a minimum number of replicas stay running during voluntary disruptions such as node drains and cluster upgrades, directly satisfying the maintenance-availability constraint. It does not cover involuntary failures like hardware crashes, which ReplicaSets handle instead.

Independent validation subjects models to scrutiny by parties separate from developers, satisfying the stem's MRM pillar requirement. This separation of duties provides objective assessment of methodology, assumptions and outputs, which is the governance control that prevents developer bias from going unchallenged.

Post-deployment performance and drift monitoring validates that models continue performing within approved tolerances after release, detecting degradation or data drift. This satisfies MRM's requirement for ongoing monitoring controls, distinct from pre-deployment validation and development-stage testing.

Temporal cross-validation respects the time ordering of feature data, training on earlier periods and validating on later ones. This exposes leakage where future information contaminates past features, which random splits would mask, directly satisfying the audit's leakage-detection requirement.

Shifts in input feature importance after retraining typically signal upstream data drift or altered feature engineering, not model decay. Auditing source data distributions and feature derivation logic pinpoints the cause, satisfying the auditor's requirement to explain the change before any remediation or redeployment decision.

Inference latency and throughput determine whether the model meets the application's real-time or batch serving requirements under expected load. Auditors must verify these operational metrics against service-level objectives, since a model that is accurate but too slow or low-capacity is unsuitable for production deployment.

Human-in-the-loop inserts a person into the decision path so consequential or low-confidence outputs are reviewed before action. This oversight catches model errors, satisfies accountability requirements and mitigates harm, which purely automated pipelines cannot guarantee on their own.

Live accuracy metrics compare current predictions against ground-truth labels as they arrive, exposing degradation that static validation scores miss. This satisfies the dashboard's alerting purpose by surfacing performance decay early, before downstream business decisions are affected.

Dataset lineage records the origin, transformations and version history of the training data, satisfying the audit requirement to demonstrate what data produced the model. Without it, auditors cannot trace bias, consent or provenance, so it is a core registry field alongside model version and performance metrics.

Training data provenance and lineage records where data originated, how it was collected and transformed, and its movement through the pipeline. This satisfies the inventory's requirement to document data-related risk factors, enabling traceability, bias investigation and reproducibility across the model's lifecycle.

A pipeline architecture diagram documents the operational topology of the AI system — data ingestion, training, inference and monitoring components and their interactions. It satisfies the 'AI Operational' criterion because it captures how the deployed pipeline runs and is maintained, not design rationale or model metrics.

Prompt Guardrails inspect and filter both user inputs and model outputs, detecting and neutralising injected instructions before they alter model behaviour. This directly counters prompt injection, where malicious text attempts to override system prompts or hijack the LLM's intended task.

A Model Card is a structured document disclosing a model's features, limitations and training data, supporting AI compliance through transparency and accountability. It enables auditors and stakeholders to assess intended use, bias risk and provenance, satisfying governance and documentation obligations.

Recording the exact Git commit SHA pins the precise model code revision audited, so any reviewer can check out identical source and reproduce results. Without it, later commits make the audit unreproducible, violating the replication requirement in the stem.

Finite-difference probing perturbs individual pixels and records how the output score moves, yielding an empirical estimate of local sensitivity without needing internal weights. This works on a black-box binary and directly addresses decision boundary sharpness. For medical triage, sharp boundaries near clinically ambiguous images are a risk, so quantifying output change per pixel perturbation provides actionable audit evidence.

Class imbalance skews a classifier toward majority classes, inflating accuracy while harming minority-class recall. The auditor must verify either balanced representation or documented mitigation such as resampling, class weighting or stratified splits, confirming the training data supports fair, reliable model performance.

Environment variables often hold API keys and connection strings, so storing them as plaintext in pipeline configuration exposes credentials. A secrets manager encrypts these values at rest, enforces access controls and supports rotation, satisfying the audit requirement that sensitive configuration never lives in code or build definitions.

SHAP integration supplies the per-feature attribution values that populate Model Explanation logs, satisfying the audit's requirement for inference-time explainability records. Without it, the endpoint returns predictions but writes no explanation entries, so the logging gap points directly to its omission from the deployment.

Reviewing data provenance traces each training record's origin, collection method and transformation history, exposing tampering or unauthorised injection before the model learns from it. This directly satisfies the stem's requirement to verify integrity of the training set, since poisoning occurs at data ingestion rather than inference, making downstream output testing insufficient.

A staging environment mirrors production configuration, so model validation runs against realistic data and settings before release. This isolates testing from live workloads, letting you verify accuracy and catch regressions without affecting production users or data.

Without independent validation, high-impact models escape scrutiny by parties outside the development team, so bias, drift and failure modes remain undetected until production impact occurs. This directly satisfies the consequence of removing that control from the MRM framework.

The Model Registry entry stores the model artifact itself alongside its environment specification and associated metadata, giving auditors a complete, self-contained record of the version. This satisfies the audit constraint by binding the trained artefact to the exact runtime dependencies and provenance details needed to reproduce and verify that model version.