AAIA AI Auditing Tools And Techniques — Timed 20 Questions

Question 1

An auditor is evaluating bias in a model deployed on AWS SageMaker. Which tool should be used to generate a report on pre-training and post-training bias metrics?

Accepted Answer

SageMaker Clarify. SageMaker Clarify provides specific bias detection capabilities for both training data and model predictions.

Answer

SageMaker Model Monitor

Answer

AWS Glue DataBrew

Answer

Amazon Inspector

Question 2

When auditing an Azure Machine Learning pipeline, which functionality within 'Responsible AI dashboard' is required to perform counterfactual analysis on model predictions?

Accepted Answer

Counterfactuals component. The 'Counterfactuals' component in the Azure Responsible AI dashboard allows auditors to see how input changes impact predictions.

Answer

Causal Inference component

Answer

Model Overview component

Answer

Error Analysis component

Question 3

You are auditing a model using IBM Watson OpenScale. Which feature should be configured to detect 'Disparate Impact' to ensure regulatory compliance?

Accepted Answer

Fairness monitor. Watson OpenScale features a 'Fairness' monitor that specifically tracks Disparate Impact ratios.

Answer

Quality monitor

Answer

Drift detection monitor

Answer

Explainability monitor

Answer

Payload logging

Question 4

To verify the provenance of a model in Google Cloud Vertex AI, which service should the auditor inspect to review the lineage graph of the artifacts?

Accepted Answer

Vertex AI ML Metadata. Vertex AI ML Metadata tracks the lineage of artifacts and executions within the pipeline.

Answer

Vertex AI Feature Store

Answer

Cloud Logging

Answer

Vertex AI Model Registry

Question 5

An auditor is evaluating an AI system's robustness against adversarial attacks. Which technique involves perturbing input data to see if the model output changes significantly?

Accepted Answer

Adversarial testing. Adversarial testing involves small perturbations to inputs to identify instability or security weaknesses.

Answer

Model pruning

Answer

Bias assessment

Answer

Data augmentation

Question 6

Which technique is best for verifying that a model has not been subject to 'training data poisoning'?

Accepted Answer

Review of data provenance. Analyzing data provenance and lineage ensures that only trusted sources were used for the training set.

Answer

Latency testing

Answer

Model Accuracy testing

Answer

Hyperparameter tuning

Question 7

When auditing an MLflow experiment, which object allows the auditor to verify the exact parameters and code version used during training?

Accepted Answer

MLflow Run. The MLflow Run object stores the code version (git commit), parameters, and metrics for a training session.

Answer

MLflow Model Registry

Answer

MLflow Experiment

Answer

MLflow Tracking Server

Question 8

When scoping an audit for a model using SHAP values, what is the primary objective of the auditor?

Accepted Answer

Explainability of model predictions. SHAP (SHapley Additive exPlanations) is used to quantify the contribution of each feature to a specific prediction, aiding interpretability.

Answer

Detecting data drift

Answer

Security vulnerability scanning

Answer

Infrastructure scaling

Question 9

When reviewing an AI system log, what is the 'inference request' ID used for?

Accepted Answer

Audit trail and traceability. The inference ID links the specific request to its model output, facilitating trace-ability during audits.

Answer

Data encryption

Answer

Hyperparameter storage

Answer

Model compression

Question 10

When auditing model deployment, what is the primary purpose of 'Shadow Mode' testing?

Accepted Answer

To compare model performance against production traffic without impacting users. Shadow mode allows running the new model in production alongside the old one to compare outputs without affecting users.

Answer

To encrypt model weights in memory

Answer

To hide model artifacts from unauthorized users

Answer

To automate hyperparameter optimization

Question 11

You are auditing a deployment on Kubernetes using Kubeflow. Which component should the auditor examine to ensure that the pipeline steps are reproducible?

Accepted Answer

Kubeflow Pipelines. Kubeflow Pipelines (KFP) ensure reproducibility through containerized steps and pipeline definitions stored in YAML.

Answer

Kubelet

Answer

Prometheus

Answer

Istio

Question 12

Which document is essential for an AI audit to identify the 'intended use' of an AI system?

Accepted Answer

Model Card. The Model Card is a standardized document that lists intended use, limitations, and performance metrics.

Answer

Server Hardware Specs

Answer

Financial Statement

Answer

Employee Training Log

Question 13

During an audit of an LLM-based application, which technique is most effective for testing the robustness against 'Prompt Injection'?

Accepted Answer

Red Teaming. Red Teaming involves manual and automated attempts to bypass safety filters and trigger malicious outputs.

Answer

Sentiment analysis

Answer

Cross-validation

Answer

Tokenization audit

Question 14

When auditing fairness using the 'Aequitas' toolkit, what is the first step an auditor should take?

Accepted Answer

Select bias metrics. The first step in Aequitas is to define the bias metrics (e.g., False Positive Rate Parity) relevant to the context.

Answer

Encrypt the dataset

Answer

Train a new model

Answer

Delete sensitive attributes

Question 15

Which aspect of 'Data Drift' does the 'Population Stability Index' (PSI) measure?

Accepted Answer

Change in population distribution. PSI measures how much the distribution of a variable has changed between the training set and the production dataset.

Answer

Change in model parameters

Answer

Change in model accuracy

Answer

Increase in inference latency

Question 16

An auditor is using 'CleverHans' for model auditing. What kind of vulnerability is this library designed to detect?

Accepted Answer

Adversarial vulnerability. CleverHans is a library for benchmarking the robustness of neural networks against adversarial examples.

Answer

Memory leaks in model deployment

Answer

Infrastructure misconfiguration

Answer

PII leaks in training data

Question 17

Which metric is commonly used to audit classification models?

Accepted Answer

F1-score. The F1-score balances precision and recall, providing a single metric for classification performance.

Answer

Logarithmic growth

Answer

Mean Squared Error

Answer

CPU utilization

Question 18

Which of the following is a common 'drift' symptom an auditor should look for in production models?

Accepted Answer

Decreasing model accuracy. When data distribution changes (Concept Drift), the model's accuracy typically degrades over time.

Answer

Increasing server costs

Answer

Increased team size

Answer

Slow Git repository commits

Question 19

An auditor is evaluating an AI system for 'Model Inversion' risk. What is this?

Accepted Answer

Reconstructing training data from outputs. Model Inversion is an attack where an adversary reconstructs training data from model outputs.

Answer

Reducing inference latency

Answer

Increasing training bias

Answer

Overfitting the data

Question 20

What is the primary role of a 'Human-in-the-Loop' (HITL) audit requirement?

Accepted Answer

To review critical AI decisions. HITL ensures that sensitive or critical decisions made by AI are reviewed by human experts.

Answer

To automate model retraining

Answer

To generate marketing reports

Answer

To improve GPU performance