AAIA AI Auditing Tools And Techniques Practice Test 4 — 15 Questions

Question 1

An auditor is evaluating the data pipeline security. Which THREE controls should be verified?

Accepted Answer

Full audit log of data provenance. Option A (Full audit log of data provenance) is correct because auditors must verify an immutable, traceable record of where data originated, how it was transformed, and who accessed it, which is essential for detecting tampering and meeting compliance requirements such as those in SOC 2 or GDPR. Option B (Identity and Access Management (IAM) for the pipeline) is correct because IAM controls enforce least-privilege access, role-based permissions, and authentication for pipeline components, directly preventing unauthorized data access or modification. Option E (Encryption of data at rest) is correct because protecting stored pipeline data with encryption (e.g., AES-256 at rest via KMS-managed keys) is a fundamental confidentiality control that auditors must confirm to prevent exposure from compromised storage. Option C (number of lines of code) is not a security control—code volume has no bearing on pipeline security posture. Option D (office access badge logs) relates to physical facility security, not the data pipeline's logical security controls, so it is out of scope for this audit.

Answer

The number of lines of code in the pipeline

Answer

Office access badge logs

Question 2

When auditing an AI pipeline, which THREE items should be part of the 'Evidence Collection' plan?

Accepted Answer

Access and error logs from the deployment server. Option A is correct because access and error logs from the deployment server provide an auditable, timestamped record of who accessed the AI pipeline and when failures occurred, which is essential forensic evidence for tracing incidents and verifying operational integrity. Option C is correct because CI/CD pipeline configuration files document the build, test, and deployment steps, including environment variables, secrets handling, and approval gates, allowing auditors to verify that the pipeline followed approved change-management and security controls. Option D is correct because serialized model artifacts (e.g., pickle, SavedModel, or ONNX files) are the actual trained models deployed in production, and collecting them enables integrity checks, version verification, and reproducibility of the model's behavior during the audit. Option B does not belong because personal notes of the lead developer are informal, unverified, and not a controlled or reproducible artifact, so they lack evidentiary reliability. Option E does not belong because team member salary information is unrelated to the technical operation, security, or compliance of the AI pipeline and would introduce privacy concerns rather than relevant audit evidence.

Answer

The personal notes of the lead developer

Answer

A list of all team member salaries

Question 3

When scoping an audit, which TWO stakeholders should the auditor interview?

Accepted Answer

Model Owners. Model Owners (A) are correct because they hold accountability for the model's purpose, risk classification, and governance decisions, making them essential for defining the audit's scope and objectives. Data Scientists (D) are correct because they possess the technical detail on data sources, feature engineering, training pipelines, and model behavior needed to bound the audit's technical scope. The lobby receptionist (B), external marketing agency (C), and office cleaning staff (E) are not relevant stakeholders for scoping an audit, as they have no ownership, development, or operational responsibility for the model or its data.

Answer

The lobby receptionist

Answer

The company's external marketing agency

Answer

Office cleaning staff

Question 4

When documenting findings, which THREE elements should be included for each finding?

Accepted Answer

The root cause. Option C is correct because every documented finding must state the condition — the specific factual situation the auditor observed (what was found), which is the core of the finding. Option A is correct because the root cause explains why the condition exists, identifying the underlying reason so the finding is actionable rather than merely descriptive. Option B is correct because a recommendation for improvement provides the corrective or mitigating action needed to address the condition and its root cause, completing the finding's value. Options D and E are incorrect: font size is a formatting detail irrelevant to the substance of a finding, and listing employees who disagreed is not a required element and could introduce unnecessary or inappropriate personal information.

Answer

The font size used in the report

Answer

The names of all employees who disagreed

Question 5

When selecting testing techniques for an AI model, which THREE are considered 'Model-Agnostic'?

Accepted Answer

SHAP (SHapley Additive exPlanations). SHAP (A) is model-agnostic because it computes Shapley values from cooperative game theory using only the model's input-output behavior, so it can explain any classifier or regressor regardless of its internal architecture. Partial Dependence Plots (C) are also model-agnostic: they estimate the marginal effect of features on predictions by averaging model outputs over the dataset, requiring only a predict function rather than access to model internals. LIME (E) is explicitly model-agnostic, as its name states, since it explains individual predictions by fitting a simple interpretable surrogate (e.g., linear model) to perturbed samples around the instance, treating the original model as a black box. In contrast, neural network weight pruning (B) is a model-specific optimization technique that requires access to a neural network's weights and architecture, and linear regression coefficients (D) are intrinsic parameters only available for linear models, so neither generalizes across arbitrary model types.

Answer

Neural network weight pruning

Answer

Linear regression coefficients

Question 6

An auditor is evaluating the 'Model Monitoring' dashboard. Which TWO items should the dashboard display to alert the team of potential issues?

Accepted Answer

Real-time model accuracy metrics. Option B (Real-time model accuracy metrics) is correct because a Model Monitoring dashboard must continuously track how well the deployed model performs against ground truth or validation data, so degradation in accuracy (e.g., precision, recall, F1, or RMSE) can trigger timely alerts before business impact occurs. Option D (Data drift alerts) is correct because monitoring must detect distributional shifts between production input data and the training baseline—using statistical measures such as PSI, KL divergence, or KS tests—since drift is a leading indicator that model predictions will become unreliable. The remaining options do not belong: A (developer's vacation schedule) is an HR/administrative concern, C (public stock prices) is unrelated external market data, and E (office building temperature) is a facilities metric, none of which signal model performance or data integrity issues.

Answer

The model developer's vacation schedule

Answer

Public stock prices of the firm

Answer

Office building temperature

Question 7

Which TWO of the following are examples of 'Data Leakage' that an auditor should look for in a pipeline?

Accepted Answer

Training on test data. Option B is correct because training on test data is a classic form of data leakage: information from the evaluation set contaminates the training process, so the model's reported performance is optimistically biased and does not reflect true generalization to unseen data. Option E is correct because including target-related features that are not available at inference time (e.g., using the label or a post-outcome variable as an input) leaks future/target information into training, producing a model that cannot be reproduced or used in production. Option A is not leakage per se; using cloud-based storage is an infrastructure choice that may raise other security or compliance concerns but does not itself contaminate training with evaluation or target information. Option C is not leakage; documenting model parameters is a governance and reproducibility practice. Option D is not leakage; using open-source libraries is a normal engineering decision that may introduce supply-chain risk but does not constitute data leakage.

Answer

Using cloud-based storage

Answer

Documenting model parameters

Answer

Using open-source libraries

Question 8

Which THREE actions are appropriate when you find a significant 'Bias' finding in an AI audit?

Accepted Answer

Recommend retraining with re-sampled data. Option B is correct because retraining with re-sampled data directly addresses the root cause of bias by correcting class imbalance or underrepresentation in the training set, which is the standard remediation for a significant bias finding. Option C is correct because a significant bias finding represents a material risk to the organization, so management must be notified to enable informed decision-making and risk mitigation. Option D is correct because audit findings must be documented in the audit report to maintain an accurate, traceable record of the issue, its severity, and recommended actions. Option A is wrong because model speed is irrelevant to bias; ignoring a significant finding violates audit and governance principles. Option E is wrong because deleting audit evidence destroys the audit trail and is unethical and likely non-compliant with audit standards.

Answer

Ignore the finding if the model is fast

Answer

Delete the audit evidence

Question 9

Which THREE technical artifacts should the auditor collect to verify the 'Model Training Process'?

Accepted Answer

Environment requirements (e.g., requirements.txt). The auditor should collect the environment requirements file (A), such as requirements.txt, because it documents the exact library versions, dependencies, and runtime configuration used to reproduce the model training process. Training execution logs (C) are essential because they provide a timestamped record of the actual training runs, including metrics, errors, hyperparameters, and resource usage that prove the process occurred as described. The training script/code (E) is also required because it contains the model architecture, data preprocessing steps, loss function, optimizer settings, and training loop that define how the model was built. Options B and D do not belong: an office address is irrelevant to technical verification of model training, and employee performance reviews are HR records that do not evidence the training process itself.

Answer

The office address of the IT department

Answer

Employee performance reviews

Question 10

Which TWO monitoring tools are typically used to detect 'Data Drift' in production pipelines?

Accepted Answer

AWS SageMaker Model Monitor. AWS SageMaker Model Monitor (A) is correct because it is a purpose-built MLOps service that continuously monitors deployed endpoints and can detect data drift by comparing incoming inference data against a baseline computed from training data, using configurable statistics and constraints. Google Cloud Vertex AI Model Monitoring (D) is also correct because it natively tracks feature skew and drift for deployed models, comparing production prediction inputs to training baselines and emitting alerts when distributions diverge. These two are the only options designed for production ML monitoring. A web browser (B) is a generic client for viewing content, Microsoft Excel (C) is a spreadsheet application for manual data analysis, and Adobe Photoshop (E) is an image-editing tool — none of them provide automated data drift detection in production pipelines.

Answer

Web Browser

Answer

Microsoft Excel

Answer

Adobe Photoshop

Question 11

An audit team must verify that a credit-scoring model's predictions can be explained to a regulator on a case-by-case basis. The data science team used a gradient-boosted tree ensemble. Which technique should the auditor recommend to produce locally faithful explanations of individual decisions?

Accepted Answer

SHAP (SHapley Additive exPlanations) values. Explaining a single decision from a tree ensemble requires a local, additive attribution method. SHAP values decompose one prediction into per-feature contributions that sum to the model output, giving a faithful, auditable rationale for each applicant. Aggregate tools such as global importance, confusion matrices, or permutation importance describe population behaviour and cannot justify an individual score.

Answer

Confusion matrix computed on the holdout sample

Answer

Permutation importance measured on the validation set

Answer

Global feature-importance ranking from the training run

Question 12

An AI audit team is validating a fraud-detection model deployed through a REST API. The model card claims 94% recall, but the auditor wants independent evidence that the live endpoint behaves as documented. The team has read access to the model registry and the API logs, but cannot access the training pipeline. Which technique should the auditor use to validate the deployed model's behavior?

Accepted Answer

Black-box API testing with a curated set of labeled probe inputs sent to the live endpoint.. When the auditor cannot inspect training internals but can reach the live endpoint, black-box testing with labeled probes provides independent, reproducible evidence about actual serving behavior. It validates the deployed artifact directly, detects discrepancies between documentation and production, and does not depend on the model owners' cooperation or internal artifacts.

Answer

Reviewing the training pipeline's source code and unit tests stored in the version control repository.

Answer

Recomputing the model's performance metrics from the original held-out test set.

Answer

Interviewing the data science team about their validation methodology and acceptance criteria.

Question 13

An auditor needs to confirm that a deployed model in production is bit-for-bit identical to the artifact that passed validation. Which artifact should the auditor use to verify integrity?

Accepted Answer

The model's cryptographic hash recorded in the model registry. Verifying that a production model equals the validated artifact requires a content-based fingerprint. A cryptographic hash stored at registration and recomputed on the deployed file gives objective, reproducible evidence of identity. Scripts, model cards, and run identifiers provide context and lineage but cannot detect a substituted or altered artifact, so they are insufficient for integrity assurance.

Answer

The model card summarizing intended use and limitations

Answer

The experiment tracking run identifier assigned during training

Answer

The training script stored in the source-control repository

Question 14

An auditor is reviewing an AI-based loan underwriting system and wants to verify that the model's predictions are stable when small, non-semantic changes are made to input features. The auditor plans to submit the same applicant profile with trivial formatting variations. Which auditing technique does this describe?

Accepted Answer

Input perturbation testing for prediction consistency.. Input perturbation testing with semantically equivalent variations isolates inference-time stability. It reveals brittle preprocessing, encoding inconsistencies, and non-deterministic serving behavior that can cause identical applicants to receive different decisions. This makes it a direct and appropriate technique for verifying that the underwriting model behaves consistently under trivial input changes.

Answer

Backtesting the model against historical loan default rates.

Answer

Feature importance ranking using SHAP values.

Answer

Sensitivity analysis of the training loss curve.

Question 15

An auditor is examining a computer vision model used for medical image triage. The vendor provides only a compiled binary and no architecture documentation. The auditor wants to estimate how much the model's output changes when individual pixels are altered slightly, in order to assess decision boundary sharpness. Which technique is most appropriate?

Accepted Answer

Finite-difference sensitivity probing by perturbing individual pixels and measuring output change.. Finite-difference sensitivity probing measures how much the output changes when individual pixels are perturbed, producing an empirical estimate of local decision boundary sharpness. It requires no internal access, works on compiled binaries, and generates quantitative evidence. This makes it the most appropriate technique when architecture documentation is unavailable but output behavior must be characterized.

Answer

Training a surrogate decision tree to mimic the binary's outputs.

Answer

Gradient-based saliency mapping using the model's internal weights.

Answer

Reviewing the vendor's ISO/IEC 42001 certification and conformity assessment report.