AAIA AI Auditing Tools And Techniques — Timed 30 Questions

Question 1

An auditor is reviewing a production computer-vision system and must assess whether the team has implemented adequate monitoring for concept drift. Which TWO indicators should the auditor expect to be tracked on an ongoing basis? (Choose two.)

30 minute time limit — choose an answer to begin.

Accepted Answer

Population Stability Index comparing live feature distributions with the training baseline. Concept drift is a change in the input-to-label relationship, so detecting it requires either comparing live feature distributions against the training baseline or measuring live performance against delayed ground truth. Activity, latency, and access metrics describe operations and security rather than the statistical stability of the model's learned mapping, so they cannot substantiate a concept-drift monitoring claim.

Answer

Total inference latency measured at the API gateway

Answer

Count of active user accounts authorized to call the prediction endpoint

Answer

Number of pull requests merged into the model-serving repository each sprint

Question 2

An auditor is evaluating bias in a model deployed on AWS SageMaker. Which tool should be used to generate a report on pre-training and post-training bias metrics?

Accepted Answer

SageMaker Clarify. SageMaker Clarify is the AWS service purpose-built to detect bias and explainability issues in ML models, and it generates both pre-training bias metrics (computed on the dataset before training) and post-training bias metrics (computed on model predictions). It integrates directly with SageMaker training and endpoints and produces a bias report that auditors can review. The other services handle monitoring, data preparation, or security scanning, not bias reporting.

Answer

SageMaker Model Monitor

Answer

AWS Glue DataBrew

Answer

Amazon Inspector

Question 3

When auditing an Azure Machine Learning pipeline, which functionality within 'Responsible AI dashboard' is required to perform counterfactual analysis on model predictions?

Accepted Answer

Counterfactuals component. The 'Counterfactuals' component in the Azure Responsible AI dashboard allows auditors to see how input changes impact predictions.

Answer

Causal Inference component

Answer

Model Overview component

Answer

Error Analysis component

Question 4

You are auditing a model using IBM Watson OpenScale. Which feature should be configured to detect 'Disparate Impact' to ensure regulatory compliance?

Accepted Answer

Fairness monitor. The Fairness monitor in IBM Watson OpenScale is specifically designed to detect and measure bias in machine learning models, including metrics like Disparate Impact. It evaluates whether a model's predictions disproportionately favor or disadvantage certain groups based on protected attributes (e.g., gender, race). This directly supports regulatory compliance by identifying discriminatory outcomes, making it the correct choice for detecting Disparate Impact.

Answer

Quality monitor

Answer

Drift detection monitor

Answer

Explainability monitor

Answer

Payload logging

Question 5

To verify the provenance of a model in Google Cloud Vertex AI, which service should the auditor inspect to review the lineage graph of the artifacts?

Accepted Answer

Vertex AI ML Metadata. Vertex AI ML Metadata tracks the lineage of artifacts and executions within the pipeline.

Answer

Vertex AI Feature Store

Answer

Cloud Logging

Answer

Vertex AI Model Registry

Question 6

An auditor is evaluating an AI system's robustness against adversarial attacks. Which technique involves perturbing input data to see if the model output changes significantly?

Accepted Answer

Adversarial testing. Adversarial testing is the practice of deliberately perturbing inputs (adding noise, changing pixels, altering tokens) to probe whether a model's output changes in ways that reveal fragility or exploitable decision boundaries. It is the standard technique auditors use to assess robustness against adversarial attacks because it directly simulates the attacker's behavior. Pruning, bias assessment, and augmentation serve different purposes and do not test adversarial robustness.

Answer

Model pruning

Answer

Bias assessment

Answer

Data augmentation

Question 7

Which technique is best for verifying that a model has not been subject to 'training data poisoning'?

Accepted Answer

Review of data provenance. Verifying data provenance — tracing where training data came from, who handled it, and whether it was tampered with — is the most direct way to detect training data poisoning. Poisoning attacks insert malicious or corrupted samples into training data, so reviewing the origin, custody, and integrity of that data is the primary control. Accuracy or latency testing cannot reliably reveal deliberate poisoning because poisoned models often perform normally on standard benchmarks.

Answer

Latency testing

Answer

Model Accuracy testing

Answer

Hyperparameter tuning

Question 8

When auditing an MLflow experiment, which object allows the auditor to verify the exact parameters and code version used during training?

Accepted Answer

MLflow Run. An MLflow Run is the fundamental unit that captures all details of a single execution of a machine learning experiment, including parameters, metrics, artifacts, and the source code version (via Git commit or other tags). Therefore, the Run object allows an auditor to verify the exact parameters and code version used during training. The Model Registry manages model versions and stages but does not store the original training parameters or code version directly.

Answer

MLflow Model Registry

Answer

MLflow Experiment

Answer

MLflow Tracking Server

Question 9

When scoping an audit for a model using SHAP values, what is the primary objective of the auditor?

Accepted Answer

Explainability of model predictions. SHAP (SHapley Additive exPlanations) values are a game-theoretic method for attributing a model's prediction to its input features, so an audit scoped around SHAP is fundamentally about explainability of model predictions. The auditor's primary objective is to verify that the model's decisions can be understood and justified per-feature. This supports regulatory, fairness, and trust requirements rather than operational concerns.

Answer

Detecting data drift

Answer

Security vulnerability scanning

Answer

Infrastructure scaling

Question 10

When reviewing an AI system log, what is the 'inference request' ID used for?

Accepted Answer

Audit trail and traceability. The inference ID links the specific request to its model output, facilitating trace-ability during audits.

Answer

Data encryption

Answer

Hyperparameter storage

Answer

Model compression

Question 11

When auditing model deployment, what is the primary purpose of 'Shadow Mode' testing?

Accepted Answer

To compare model performance against production traffic without impacting users. Shadow mode allows running the new model in production alongside the old one to compare outputs without affecting users.

Answer

To encrypt model weights in memory

Answer

To hide model artifacts from unauthorized users

Answer

To automate hyperparameter optimization

Question 12

You are auditing a deployment on Kubernetes using Kubeflow. Which component should the auditor examine to ensure that the pipeline steps are reproducible?

Accepted Answer

Kubeflow Pipelines. Kubeflow Pipelines (KFP) ensure reproducibility through containerized steps and pipeline definitions stored in YAML.

Answer

Kubelet

Answer

Prometheus

Answer

Istio

Question 13

Which document is essential for an AI audit to identify the 'intended use' of an AI system?

Accepted Answer

Model Card. The Model Card explicitly documents a model's intended use, out-of-scope uses, limitations, and performance characteristics — making it the primary artifact auditors consult to determine whether a model is being used as designed. It is the canonical source for 'intended use' statements in AI governance. Without it, auditors cannot assess fitness-for-purpose or misuse risk.

Answer

Server Hardware Specs

Answer

Financial Statement

Answer

Employee Training Log

Question 14

During an audit of an LLM-based application, which technique is most effective for testing the robustness against 'Prompt Injection'?

Accepted Answer

Red Teaming. Red teaming is the most effective technique for testing LLM robustness against prompt injection because it involves adversarial humans (or automated adversaries) actively crafting malicious prompts — direct overrides, role-play jailbreaks, indirect injections via retrieved documents — to probe whether the model can be manipulated. It surfaces real-world attack paths that static analysis or standard validation cannot. Sentiment analysis, cross-validation, and tokenization audits do not test adversarial prompt handling.

Answer

Sentiment analysis

Answer

Cross-validation

Answer

Tokenization audit

Question 15

When auditing fairness using the 'Aequitas' toolkit, what is the first step an auditor should take?

Accepted Answer

Select bias metrics. The first step when auditing fairness using the Aequitas toolkit is to select the bias metrics that will be used to evaluate the model. Aequitas requires the auditor to choose from a set of fairness metrics (e.g., disparate impact, false positive rate parity) based on the context and protected attributes. Only after selecting metrics can the toolkit compute and visualize them. This step ensures the audit is tailored to the specific fairness concerns.

Answer

Encrypt the dataset

Answer

Train a new model

Answer

Delete sensitive attributes

Question 16

Which aspect of 'Data Drift' does the 'Population Stability Index' (PSI) measure?

Accepted Answer

Change in population distribution. The Population Stability Index (PSI) measures how much the distribution of a population (typically the input features or scores) has shifted between two time periods, such as training versus current production data. It quantifies change in population distribution, not model parameters or accuracy. PSI is a standard drift-detection metric used in credit risk and other regulated modeling contexts.

Answer

Change in model parameters

Answer

Change in model accuracy

Answer

Increase in inference latency

Question 17

An auditor is using 'CleverHans' for model auditing. What kind of vulnerability is this library designed to detect?

Accepted Answer

Adversarial vulnerability. CleverHans is an open-source Python library developed by Google that implements reference implementations of adversarial attacks (e.g., FGSM, PGD, Carlini-Wagner) and defenses against machine learning models. It is specifically designed to help researchers and auditors probe models for adversarial vulnerabilities — inputs crafted to cause misclassification. Therefore, the vulnerability class it detects is adversarial vulnerability.

Answer

Memory leaks in model deployment

Answer

Infrastructure misconfiguration

Answer

PII leaks in training data

Question 18

Which metric is commonly used to audit classification models?

Accepted Answer

F1-score. The F1-score balances precision and recall, providing a single metric for classification performance.

Answer

Logarithmic growth

Answer

Mean Squared Error

Answer

CPU utilization

Question 19

Which of the following is a common 'drift' symptom an auditor should look for in production models?

Accepted Answer

Decreasing model accuracy. When data distribution changes (Concept Drift), the model's accuracy typically degrades over time.

Answer

Increasing server costs

Answer

Increased team size

Answer

Slow Git repository commits

Question 20

An auditor is evaluating an AI system for 'Model Inversion' risk. What is this?

Accepted Answer

Reconstructing training data from outputs. Model Inversion is an attack where an adversary reconstructs training data from model outputs.

Answer

Reducing inference latency

Answer

Increasing training bias

Answer

Overfitting the data