AAIA Practice Test 4 — 25 Questions

Question 1

You are auditing a model using IBM Watson OpenScale. Which feature should be configured to detect 'Disparate Impact' to ensure regulatory compliance?

Accepted Answer

Fairness monitor. The Fairness monitor in IBM Watson OpenScale is specifically designed to detect and measure bias in machine learning models, including metrics like Disparate Impact. It evaluates whether a model's predictions disproportionately favor or disadvantage certain groups based on protected attributes (e.g., gender, race). This directly supports regulatory compliance by identifying discriminatory outcomes, making it the correct choice for detecting Disparate Impact.

Answer

Quality monitor

Answer

Drift detection monitor

Answer

Explainability monitor

Answer

Payload logging

Question 2

When auditing an MLflow experiment, which object allows the auditor to verify the exact parameters and code version used during training?

Accepted Answer

MLflow Run. An MLflow Run is the fundamental unit that captures all details of a single execution of a machine learning experiment, including parameters, metrics, artifacts, and the source code version (via Git commit or other tags). Therefore, the Run object allows an auditor to verify the exact parameters and code version used during training. The Model Registry manages model versions and stages but does not store the original training parameters or code version directly.

Answer

MLflow Model Registry

Answer

MLflow Experiment

Answer

MLflow Tracking Server

Question 3

Which THREE factors must be evaluated when assessing the 'Fairness' of an AI model?

Accepted Answer

Disparities in model error rates across protected groups. Fairness assessment requires examining disparities in model error rates across protected groups (B), because unequal false-positive or false-negative rates (e.g., via equalized odds or error-rate parity) directly reveal discriminatory performance. Historical bias present in the training datasets (C) must be evaluated since biased or unrepresentative data propagates and amplifies unfair outcomes in the model. Demographic parity metrics for the model's predictions (D) are also essential, as they quantify whether favorable outcomes are distributed equally across groups (e.g., via disparate impact ratio). The programming language used to build the model (A) is irrelevant to fairness, as fairness depends on data, features, and evaluation metrics rather than implementation language. The number of CPU cores allocated for training (E) affects computational performance only, not the fairness of the model's predictions.

Answer

The programming language used to build the model

Answer

The number of CPU cores allocated for training

Question 4

What is the consequence of 'Pipeline Staleness' in an AI system?

Accepted Answer

Model degradation due to outdated data. If the pipeline is not refreshed, it trains on outdated data, leading to model degradation.

Answer

Unauthorized access

Answer

Increased hardware costs

Answer

System crash

Question 5

You are auditing a 'Model Monitoring' dashboard. Which metric is most indicative of a potential degradation in the model's reliability over time?

Accepted Answer

Statistical drift between the production input data and training baseline. Data drift and concept drift are primary indicators that a model's performance may be degrading in a production environment.

Answer

The physical location of the cloud server hosting the model

Answer

The number of lines of code in the model deployment script

Answer

The number of users accessing the model API

Question 6

In the context of AI Operations, what does 'CI/CD' stand for?

Accepted Answer

Continuous Integration and Continuous Deployment. Continuous Integration and Continuous Deployment/Delivery.

Answer

Computer Intelligence and Cloud Data

Answer

Continuous Inference and Cloud Deployment

Answer

Code Integration and Cloud Delivery

Question 7

Which THREE components are critical to monitor in an AI pipeline to ensure operational availability?

Accepted Answer

Data source connectivity. Data source connectivity (A) is critical because if the pipeline cannot reach its upstream data sources, ingestion fails and the pipeline becomes unavailable regardless of downstream health. Pipeline task execution status (B) is essential to detect failed, stalled, or retrying tasks so operators can remediate breaks in the workflow and maintain availability. Resource utilization (CPU/GPU) (D) must be monitored because saturation or exhaustion of compute resources causes task failures, timeouts, and degraded throughput that directly impact operational availability. Company stock price (C) and developer social media activity (E) are unrelated to the technical health or availability of an AI pipeline and therefore are not monitoring components for operational availability.

Answer

Company stock price

Answer

Developer social media activity

Question 8

During an audit of an LLM-based application, which technique is most effective for testing the robustness against 'Prompt Injection'?

Accepted Answer

Red Teaming. Red teaming is the most effective technique for testing LLM robustness against prompt injection because it involves adversarial humans (or automated adversaries) actively crafting malicious prompts — direct overrides, role-play jailbreaks, indirect injections via retrieved documents — to probe whether the model can be manipulated. It surfaces real-world attack paths that static analysis or standard validation cannot. Sentiment analysis, cross-validation, and tokenization audits do not test adversarial prompt handling.

Answer

Sentiment analysis

Answer

Cross-validation

Answer

Tokenization audit

Question 9

You are reviewing a model risk management (MRM) framework for an AI system using NVIDIA NeMo Guardrails. What is the primary purpose of this tool in a risk mitigation strategy?

Accepted Answer

To enforce safety boundaries on LLM inputs and outputs. NeMo Guardrails allows developers to define programmatic boundaries for LLM behavior, effectively serving as an automated control for AI governance.

Answer

To automate model retraining

Answer

To improve model training speed

Answer

To encrypt training datasets

Question 10

When auditing model explainability, why is it risky to rely solely on 'Global Feature Importance'?

Accepted Answer

It fails to explain individual decisions. Global importance explains the model overall, but individual predictions (local) may be driven by different factors.

Answer

It lacks local prediction accuracy

Answer

It uses too much memory

Answer

It is only for regression models

Question 11

A board of directors requests a quarterly report on AI risk exposure. Which component of the NIST AI Risk Management Framework should you prioritize to demonstrate that the board's AI policy is being operationally enforced?

Accepted Answer

Govern. The Govern function of the NIST AI RMF establishes the policies, roles, and accountability structures that ensure AI risk management is operationalized across the organization. Demonstrating that a board-level AI policy is actually enforced requires evidence of governance mechanisms — assigned responsibilities, documented processes, and oversight — which is exactly what Govern covers.

Answer

Measure

Answer

Manage

Answer

Map

Question 12

When auditing model fairness in a production environment using IBM Watson OpenScale, which metric would you specifically validate to ensure the model is not violating protected class regulations?

Accepted Answer

Disparate Impact. Disparate Impact is the fairness metric in IBM Watson OpenScale that quantifies whether a model's favorable outcomes are disproportionately distributed across protected groups (e.g., race, gender). It is computed as the ratio of favorable outcome rates between the unprivileged and privileged groups, with values below 0.8 (the EEOC four-fifths rule) typically flagging potential regulatory violations. This directly addresses protected class compliance requirements under laws like ECOA and Title VII.

Answer

Precision-Recall AUC

Answer

Model Latency

Answer

Root Mean Square Error

Question 13

An auditor is evaluating bias in a model deployed on AWS SageMaker. Which tool should be used to generate a report on pre-training and post-training bias metrics?

Accepted Answer

SageMaker Clarify. SageMaker Clarify is the AWS service purpose-built to detect bias and explainability issues in ML models, and it generates both pre-training bias metrics (computed on the dataset before training) and post-training bias metrics (computed on model predictions). It integrates directly with SageMaker training and endpoints and produces a bias report that auditors can review. The other services handle monitoring, data preparation, or security scanning, not bias reporting.

Answer

SageMaker Model Monitor

Answer

AWS Glue DataBrew

Answer

Amazon Inspector

Question 14

You are reviewing the AI Incident Response Plan. Which scenario constitutes a 'High' severity AI incident that requires immediate board notification under most enterprise risk frameworks?

Accepted Answer

Systemic bias detected in a core automated lending model. An incident where a model makes a biased decision impacting a protected class carries significant regulatory and reputational risk, requiring board visibility.

Answer

A 200ms latency increase during inference

Answer

A minor change in the training dataset size

Answer

A 2% drop in model prediction accuracy

Question 15

When implementing drift detection in AWS SageMaker Model Monitor, which metric should be monitored to detect feature attribution shift for structured data?

Accepted Answer

Feature Attribution Drift (SHAP). Feature Attribution Drift, computed using SHAP (SHapley Additive exPlanations) values, measures how the contribution of each input feature to the model's prediction changes over time relative to a baseline. This is the correct metric for detecting feature attribution shift in structured data within SageMaker Model Monitor. It captures not just input distribution changes but changes in how the model weighs features.

Answer

Cosine Similarity

Answer

Mean Square Error

Answer

Precision-Recall AUC

Question 16

When conducting an audit, what is the 'Model Registry' entry for a model version supposed to contain?

Accepted Answer

The model artifact, environment, and metadata. It should contain the model binary, the environment definition (requirements.txt), and the experiment metadata (the 'Run' ID).

Answer

Only the model binary

Answer

Only the training dataset

Answer

Only the developer's credentials

Question 17

An auditor is evaluating the data pipeline security. Which THREE controls should be verified?

Accepted Answer

Full audit log of data provenance. Option A (Full audit log of data provenance) is correct because auditors must verify an immutable, traceable record of where data originated, how it was transformed, and who accessed it, which is essential for detecting tampering and meeting compliance requirements such as those in SOC 2 or GDPR. Option B (Identity and Access Management (IAM) for the pipeline) is correct because IAM controls enforce least-privilege access, role-based permissions, and authentication for pipeline components, directly preventing unauthorized data access or modification. Option E (Encryption of data at rest) is correct because protecting stored pipeline data with encryption (e.g., AES-256 at rest via KMS-managed keys) is a fundamental confidentiality control that auditors must confirm to prevent exposure from compromised storage. Option C (number of lines of code) is not a security control—code volume has no bearing on pipeline security posture. Option D (office access badge logs) relates to physical facility security, not the data pipeline's logical security controls, so it is out of scope for this audit.

Answer

The number of lines of code in the pipeline

Answer

Office access badge logs

Question 18

Which aspect of 'Data Drift' does the 'Population Stability Index' (PSI) measure?

Accepted Answer

Change in population distribution. The Population Stability Index (PSI) measures how much the distribution of a population (typically the input features or scores) has shifted between two time periods, such as training versus current production data. It quantifies change in population distribution, not model parameters or accuracy. PSI is a standard drift-detection metric used in credit risk and other regulated modeling contexts.

Answer

Change in model parameters

Answer

Change in model accuracy

Answer

Increase in inference latency

Question 19

Which technique is recommended for auditing 'data leakage' in a feature engineering pipeline?

Accepted Answer

Temporal cross-validation. Cross-validation using temporal splits (rather than random splits) prevents training on future information.

Answer

Feature ablation

Answer

Normalization

Answer

Random sampling

Question 20

Which THREE items should be tracked in a 'Model Registry' for audit readiness?

Accepted Answer

Training dataset lineage. A is correct because training dataset lineage documents which data was used to train a model, which is essential for audit trails, reproducibility, and compliance with data governance requirements. B is correct because a source code reference such as a Git commit uniquely identifies the exact code version that produced the model, enabling auditors to trace and reproduce the training pipeline. E is correct because a model artifact hash provides cryptographic integrity verification, ensuring the deployed model has not been altered and matches the audited version. C is not appropriate because a cloud provider logo is merely branding and carries no audit or traceability value. D is not appropriate because employee birthday is personal data unrelated to model governance and would raise privacy concerns rather than support audit readiness.

Answer

Cloud provider logo

Answer

Employee birthday