AAIA Practice Test 15 — 15 Questions

Question 1

What is a 'Model Repository' in an AI audit context?

Accepted Answer

A versioned registry of models. A central, versioned store (like MLflow or Azure Model Registry) for managing trained models.

Answer

A backup hard drive

Answer

A list of developers

Answer

A documentation folder

Question 2

When using 'LIME' (Local Interpretable Model-agnostic Explanations) for auditing, what is the auditor looking for?

Accepted Answer

Local decision transparency. LIME creates a local surrogate model to explain individual predictions, helping the auditor understand why a specific decision was made.

Answer

Data distribution drift

Answer

Overall model performance

Answer

Adversarial robustness

Question 3

Which THREE documents are typically required for an AI Audit trail?

Accepted Answer

Model training logs. Model training logs (A) are required because they document the datasets, hyperparameters, training runs, and timestamps that establish how a model was built, which is essential for reproducibility and traceability in an AI audit trail. Model validation reports (B) are required because they record performance metrics, test results, bias/fairness evaluations, and sign-offs that demonstrate the model was assessed against acceptance criteria before use. Deployment approval records (C) are required because they provide the governance evidence of who authorized the release, when, and under what conditions, completing the chain of accountability from development to production. The remaining options do not belong: an employee vacation calendar (D) and cafeteria lunch receipts (E) are unrelated administrative records with no bearing on model provenance, validation, or deployment authorization.

Answer

Employee vacation calendar

Answer

Cafeteria lunch receipts

Question 4

In an AI Audit, what does 'Model Lineage' demonstrate?

Accepted Answer

Provenance of the final model. Model lineage is the documented, traceable record of a model's provenance — the datasets, code, hyperparameters, training runs, and transformations that produced the final artifact. In an AI audit, it demonstrates end-to-end traceability: which data trained the model, which code version was used, who approved each stage, and how the model evolved. This chain of custody is what allows auditors to verify reproducibility, detect unauthorized changes, and satisfy regulatory requirements.

Answer

The cloud provider's stock

Answer

The developer's name

Answer

The model's name

Question 5

Which of the following is an example of an 'operational metric' for an AI system?

Accepted Answer

Request latency. Operational metrics (system health) include latency, error rate, and throughput.

Answer

F1 Score

Answer

ROC AUC

Answer

Training accuracy

Question 6

Which TWO monitoring tools are typically used to detect 'Data Drift' in production pipelines?

Accepted Answer

AWS SageMaker Model Monitor. AWS SageMaker Model Monitor (A) is correct because it is a purpose-built MLOps service that continuously monitors deployed endpoints and can detect data drift by comparing incoming inference data against a baseline computed from training data, using configurable statistics and constraints. Google Cloud Vertex AI Model Monitoring (D) is also correct because it natively tracks feature skew and drift for deployed models, comparing production prediction inputs to training baselines and emitting alerts when distributions diverge. These two are the only options designed for production ML monitoring. A web browser (B) is a generic client for viewing content, Microsoft Excel (C) is a spreadsheet application for manual data analysis, and Adobe Photoshop (E) is an image-editing tool — none of them provide automated data drift detection in production pipelines.

Answer

Web Browser

Answer

Microsoft Excel

Answer

Adobe Photoshop

Question 7

Which of these is a typical 'audit finding' in an AI governance review?

Accepted Answer

Insufficient model documentation. Insufficient model documentation is a classic audit finding because governance frameworks require documented model purpose, training data, assumptions, limitations, validation results, and ownership. Without this documentation, auditors cannot verify that the model is fit for purpose, compliant, or properly governed. The other options describe normal technical or deployment characteristics, not deficiencies.

Answer

Too many GPU cores

Answer

Cloud-based deployment

Answer

Use of Python language

Question 8

When conducting an audit, which THREE of the following represent potential 'Model Risk' areas that require documentation?

Accepted Answer

Susceptibility to adversarial attacks. Option A is correct because susceptibility to adversarial attacks is a model risk area: AI/ML models can be manipulated by crafted inputs (e.g., evasion or poisoning attacks) that cause incorrect outputs, and this vulnerability must be documented during an audit. Option C is correct because inherent bias in decision-making is a core model risk: training data or algorithmic design can embed bias that leads to unfair or discriminatory outcomes, requiring documentation and mitigation. Option D is correct because susceptibility to data drift is a model risk: changes in input data distributions over time (data drift) or changes in the relationship between inputs and outputs (concept drift) degrade model performance and must be monitored and documented. Option B is incorrect because the color of server racks is a physical facility detail with no bearing on model behavior or risk. Option E is incorrect because local internet connection speed is an infrastructure/network performance factor, not a model risk area.

Answer

The color of the server racks

Answer

The speed of the local internet connection

Question 9

Which THREE risks should be explicitly managed in an 'AI Governance' program?

Accepted Answer

Operational performance degradation. Governance must manage bias, security, and performance risks.

Answer

The choice of font for the final report

Answer

The color of the server room lights

Question 10

What is 'Shadow Deployment' in AI MLOps?

Accepted Answer

Running a model in parallel without serving results. Running the new model alongside the production model, but without the new model's predictions influencing the final result, to validate performance.

Answer

Running only on local machine

Answer

Deleting the model

Answer

Replacing the production model

Question 11

When auditing an AI model, what does 'Concept Drift' signify to an operational team?

Accepted Answer

The model's predictive accuracy is likely degrading. Concept drift describes the phenomenon where the statistical relationship between input features and the target variable changes over time in the production environment, diverging from the distribution the model was trained on. Because the model's learned parameters are now misaligned with current data patterns, its predictive accuracy typically degrades. Operational teams monitor for drift (e.g., via PSI, KL divergence, or KS tests) to trigger retraining before business impact occurs.

Answer

The training data was leaked

Answer

The model is overfitted

Answer

The model is running too fast

Question 12

When auditing an ML pipeline, what does the 'DAG' (Directed Acyclic Graph) represent?

Accepted Answer

Pipeline task dependencies. The DAG defines the sequence and dependency of operational steps in the pipeline.

Answer

Model accuracy curve

Answer

The database schema

Answer

The network architecture

Question 13

Which operational process is required to ensure 'Data Privacy' when using user-generated data for model retraining?

Accepted Answer

Data anonymization. Anonymization or pseudonymization must be performed before training.

Answer

Increasing compute power

Answer

Model quantization

Answer

Removing the model

Question 14

Which TWO methods are commonly used by auditors to verify the 'Provenance' of training data?

Accepted Answer

Reviewing data lineage logs in the metadata store. Option C is correct because provenance verification relies on data lineage records maintained in a metadata store, which document the origin, transformations, and movement of each dataset through the pipeline, giving auditors a traceable chain of custody. Option D is correct because computing cryptographic hashes (e.g., SHA-256) of input files produces a unique fingerprint that can be compared against recorded values to confirm the data has not been altered and matches its claimed source. Options A, B, and E do not belong: watching a training video only shows content, not origin or integrity; interviewing janitorial staff has no bearing on data provenance controls; and checking whether a model is open source describes licensing or model availability, not the lineage or integrity of the training data itself.

Answer

Watching the training video

Answer

Interviewing the janitorial staff

Answer

Checking if the model is open source

Question 15

When auditing an AI pipeline, which THREE items should be part of the 'Evidence Collection' plan?

Accepted Answer

Access and error logs from the deployment server. Option A is correct because access and error logs from the deployment server provide an auditable, timestamped record of who accessed the AI pipeline and when failures occurred, which is essential forensic evidence for tracing incidents and verifying operational integrity. Option C is correct because CI/CD pipeline configuration files document the build, test, and deployment steps, including environment variables, secrets handling, and approval gates, allowing auditors to verify that the pipeline followed approved change-management and security controls. Option D is correct because serialized model artifacts (e.g., pickle, SavedModel, or ONNX files) are the actual trained models deployed in production, and collecting them enables integrity checks, version verification, and reproducibility of the model's behavior during the audit. Option B does not belong because personal notes of the lead developer are informal, unverified, and not a controlled or reproducible artifact, so they lack evidentiary reliability. Option E does not belong because team member salary information is unrelated to the technical operation, security, or compliance of the AI pipeline and would introduce privacy concerns rather than relevant audit evidence.

Answer

The personal notes of the lead developer

Answer

A list of all team member salaries