AAIA AI Auditing Tools And Techniques Practice Test 2 — 15 Questions

Question 1

Which aspect of 'Data Drift' does the 'Population Stability Index' (PSI) measure?

Accepted Answer

Change in population distribution. The Population Stability Index (PSI) measures how much the distribution of a population (typically the input features or scores) has shifted between two time periods, such as training versus current production data. It quantifies change in population distribution, not model parameters or accuracy. PSI is a standard drift-detection metric used in credit risk and other regulated modeling contexts.

Answer

Change in model parameters

Answer

Change in model accuracy

Answer

Increase in inference latency

Question 2

An auditor is using 'CleverHans' for model auditing. What kind of vulnerability is this library designed to detect?

Accepted Answer

Adversarial vulnerability. CleverHans is an open-source Python library developed by Google that implements reference implementations of adversarial attacks (e.g., FGSM, PGD, Carlini-Wagner) and defenses against machine learning models. It is specifically designed to help researchers and auditors probe models for adversarial vulnerabilities — inputs crafted to cause misclassification. Therefore, the vulnerability class it detects is adversarial vulnerability.

Answer

Memory leaks in model deployment

Answer

Infrastructure misconfiguration

Answer

PII leaks in training data

Question 3

Which metric is commonly used to audit classification models?

Accepted Answer

F1-score. The F1-score balances precision and recall, providing a single metric for classification performance.

Answer

Logarithmic growth

Answer

Mean Squared Error

Answer

CPU utilization

Question 4

Which of the following is a common 'drift' symptom an auditor should look for in production models?

Accepted Answer

Decreasing model accuracy. When data distribution changes (Concept Drift), the model's accuracy typically degrades over time.

Answer

Increasing server costs

Answer

Increased team size

Answer

Slow Git repository commits

Question 5

An auditor is evaluating an AI system for 'Model Inversion' risk. What is this?

Accepted Answer

Reconstructing training data from outputs. Model Inversion is an attack where an adversary reconstructs training data from model outputs.

Answer

Reducing inference latency

Answer

Increasing training bias

Answer

Overfitting the data

Question 6

What is the primary role of a 'Human-in-the-Loop' (HITL) audit requirement?

Accepted Answer

To review critical AI decisions. HITL ensures that sensitive or critical decisions made by AI are reviewed by human experts.

Answer

To automate model retraining

Answer

To generate marketing reports

Answer

To improve GPU performance

Question 7

In an audit of differential privacy implementations, what is the 'epsilon' parameter used for?

Accepted Answer

Privacy budget control. Epsilon is the privacy budget; lower values provide stronger privacy guarantees but potentially lower utility.

Answer

Model training speed

Answer

Learning rate scheduling

Answer

Database query limit

Question 8

When auditing data pipelines, what is the function of a 'Data Quality' assertion?

Accepted Answer

To validate data integrity before training. Assertions (e.g., in Great Expectations) check if data meets predefined schemas and constraints before ingestion.

Answer

To rename data columns

Answer

To increase prediction speed

Answer

To compress input files

Question 9

When auditing model versioning, what is the recommended practice for maintaining evidence?

Accepted Answer

Use immutable versioning. Immutable model repositories and versioning control (e.g., Git LFS, MLflow) ensure the exact state can be audited.

Answer

Delete old versions

Answer

Use mutable file names

Answer

Store models in cache

Question 10

Which tool provides visual confirmation of data lineage in an end-to-end AI project?

Accepted Answer

Apache Airflow. Apache Airflow is a workflow orchestration platform that provides a web-based UI showing DAG (Directed Acyclic Graph) runs, task dependencies, and data flow across pipeline stages. This visual representation allows stakeholders to trace data lineage from ingestion through transformation to model training and deployment, confirming end-to-end data provenance.

Answer

Docker

Answer

Jenkins

Answer

Kubernetes

Question 11

What is the purpose of 'Model Validation' in the audit lifecycle?

Accepted Answer

To verify model performance on test data. Model validation is the process of evaluating a trained model's performance on a separate test dataset to ensure it generalizes well and meets the intended objectives. It verifies metrics such as accuracy, precision, recall, or fairness before deployment, catching overfitting or data leakage issues.

Answer

To speed up data ingestion

Answer

To lower API costs

Answer

To increase training time

Question 12

What is a 'Model Repository' in an AI audit context?

Accepted Answer

A versioned registry of models. A central, versioned store (like MLflow or Azure Model Registry) for managing trained models.

Answer

A backup hard drive

Answer

A list of developers

Answer

A documentation folder

Question 13

When auditing model explainability, why is it risky to rely solely on 'Global Feature Importance'?

Accepted Answer

It fails to explain individual decisions. Global importance explains the model overall, but individual predictions (local) may be driven by different factors.

Answer

It lacks local prediction accuracy

Answer

It uses too much memory

Answer

It is only for regression models

Question 14

Which technique is recommended for auditing 'data leakage' in a feature engineering pipeline?

Accepted Answer

Temporal cross-validation. Cross-validation using temporal splits (rather than random splits) prevents training on future information.

Answer

Feature ablation

Answer

Normalization

Answer

Random sampling

Question 15

When using 'LIME' (Local Interpretable Model-agnostic Explanations) for auditing, what is the auditor looking for?

Accepted Answer

Local decision transparency. LIME creates a local surrogate model to explain individual predictions, helping the auditor understand why a specific decision was made.

Answer

Data distribution drift

Answer

Overall model performance

Answer

Adversarial robustness