AAIA AI Auditing Tools And Techniques — Timed 10 Questions

Question 1

An auditor is evaluating bias in a model deployed on AWS SageMaker. Which tool should be used to generate a report on pre-training and post-training bias metrics?

Accepted Answer

SageMaker Clarify. SageMaker Clarify provides specific bias detection capabilities for both training data and model predictions.

Answer

SageMaker Model Monitor

Answer

AWS Glue DataBrew

Answer

Amazon Inspector

Question 2

When auditing an Azure Machine Learning pipeline, which functionality within 'Responsible AI dashboard' is required to perform counterfactual analysis on model predictions?

Accepted Answer

Counterfactuals component. The 'Counterfactuals' component in the Azure Responsible AI dashboard allows auditors to see how input changes impact predictions.

Answer

Causal Inference component

Answer

Model Overview component

Answer

Error Analysis component

Question 3

You are auditing a model using IBM Watson OpenScale. Which feature should be configured to detect 'Disparate Impact' to ensure regulatory compliance?

Accepted Answer

Fairness monitor. Watson OpenScale features a 'Fairness' monitor that specifically tracks Disparate Impact ratios.

Answer

Quality monitor

Answer

Drift detection monitor

Answer

Explainability monitor

Answer

Payload logging

Question 4

To verify the provenance of a model in Google Cloud Vertex AI, which service should the auditor inspect to review the lineage graph of the artifacts?

Accepted Answer

Vertex AI ML Metadata. Vertex AI ML Metadata tracks the lineage of artifacts and executions within the pipeline.

Answer

Vertex AI Feature Store

Answer

Cloud Logging

Answer

Vertex AI Model Registry

Question 5

An auditor is evaluating an AI system's robustness against adversarial attacks. Which technique involves perturbing input data to see if the model output changes significantly?

Accepted Answer

Adversarial testing. Adversarial testing involves small perturbations to inputs to identify instability or security weaknesses.

Answer

Model pruning

Answer

Bias assessment

Answer

Data augmentation

Question 6

Which technique is best for verifying that a model has not been subject to 'training data poisoning'?

Accepted Answer

Review of data provenance. Analyzing data provenance and lineage ensures that only trusted sources were used for the training set.

Answer

Latency testing

Answer

Model Accuracy testing

Answer

Hyperparameter tuning

Question 7

When auditing an MLflow experiment, which object allows the auditor to verify the exact parameters and code version used during training?

Accepted Answer

MLflow Run. The MLflow Run object stores the code version (git commit), parameters, and metrics for a training session.

Answer

MLflow Model Registry

Answer

MLflow Experiment

Answer

MLflow Tracking Server

Question 8

When scoping an audit for a model using SHAP values, what is the primary objective of the auditor?

Accepted Answer

Explainability of model predictions. SHAP (SHapley Additive exPlanations) is used to quantify the contribution of each feature to a specific prediction, aiding interpretability.

Answer

Detecting data drift

Answer

Security vulnerability scanning

Answer

Infrastructure scaling

Question 9

When reviewing an AI system log, what is the 'inference request' ID used for?

Accepted Answer

Audit trail and traceability. The inference ID links the specific request to its model output, facilitating trace-ability during audits.

Answer

Data encryption

Answer

Hyperparameter storage

Answer

Model compression

Question 10

When auditing model deployment, what is the primary purpose of 'Shadow Mode' testing?

Accepted Answer

To compare model performance against production traffic without impacting users. Shadow mode allows running the new model in production alongside the old one to compare outputs without affecting users.

Answer

To encrypt model weights in memory

Answer

To hide model artifacts from unauthorized users

Answer

To automate hyperparameter optimization