AAIA Practice Question: An auditor is evaluating bias in a model deployed on AWS SageMaker
An auditor is evaluating bias in a model deployed on AWS SageMaker. Which tool should be used to generate a report on pre-training and post-training bias metrics?
⚠ Common exam trap
The trap is picking SageMaker Model Monitor because it also runs on SageMaker and produces reports — but Model Monitor is about drift and data quality, while Clarify is the only service that produces pre- and post-training bias metrics.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
SageMaker Clarify
SageMaker Clarify is the AWS service purpose-built to detect bias and explainability issues in ML models, and it generates both pre-training bias metrics (computed on the dataset before training) and post-training bias metrics (computed on model predictions). It integrates directly with SageMaker training and endpoints and produces a bias report that auditors can review. The other services handle monitoring, data preparation, or security scanning, not bias reporting.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
SageMaker Model Monitor
Why it's wrong here
Model Monitor detects data and model drift on deployed endpoints; it does not compute pre-training or post-training bias metrics. It is tempting because it is the SageMaker monitoring service, and would be correct for alerting on feature distribution shifts or quality degradation in production inference traffic.
- ✓
SageMaker Clarify
Why this is correct
SageMaker Clarify generates pre-training and post-training bias metrics, directly satisfying the auditor's requirement for both measurement stages. It computes metrics such as class imbalance and disparate impact before training, then compares predicted labels against ground truth afterwards, producing a consolidated bias report within SageMaker.
- ✗
AWS Glue DataBrew
Why it's wrong here
Glue DataBrew profiles and transforms datasets for preparation; it produces no pre-training or post-training bias metric report. It is tempting because bias work begins with data inspection, and DataBrew would be correct for cleaning, normalising, or detecting anomalies in the raw training dataset before model development.
- ✗
Amazon Inspector
Why it's wrong here
Inspector scans compute workloads and repositories for software vulnerabilities and unintended network exposure, not statistical bias in models. It is tempting because it is a security assessment service within AWS, and would be correct for identifying CVEs or configuration weaknesses in the infrastructure hosting the SageMaker endpoint.
About these practice questions
One of 230 original AAIA practice questions on Courseiva, each with a full explanation and wrong-answer analysis — not exam dumps or protected exam content. Learn why practice questions differ from exam dumps →
You might also like
Similar AAIA questions
- Which document is essential for an AI audit to identify the 'intended use' of an AI system?
- An auditor is reviewing an AI-based loan underwriting system and wants to verify that the model's predictions are stable…
- Which aspect of 'Data Drift' does the 'Population Stability Index' (PSI) measure?
- Which technique is recommended for auditing 'data leakage' in a feature engineering pipeline?
- Which THREE factors should an auditor consider when evaluating the suitability of an AI model for production?
Written and reviewed by Johnson Ajibi, MSc IT Security
Security Systems Architect · founder of Courseiva
Last reviewed September 2026 · checked against the official ISACA exam blueprint
This AAIA practice question is part of Courseiva's free ISACA certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the AAIA exam.