AAIA Practice Test 9 — 10 Questions

Question 1

In the context of AI Operations, what does 'CI/CD' stand for?

Accepted Answer

Continuous Integration and Continuous Deployment. Continuous Integration and Continuous Deployment/Delivery.

Answer

Computer Intelligence and Cloud Data

Answer

Continuous Inference and Cloud Deployment

Answer

Code Integration and Cloud Delivery

Question 2

Which THREE components are critical to monitor in an AI pipeline to ensure operational availability?

Accepted Answer

Data source connectivity. Data source connectivity (A) is critical because if the pipeline cannot reach its upstream data sources, ingestion fails and the pipeline becomes unavailable regardless of downstream health. Pipeline task execution status (B) is essential to detect failed, stalled, or retrying tasks so operators can remediate breaks in the workflow and maintain availability. Resource utilization (CPU/GPU) (D) must be monitored because saturation or exhaustion of compute resources causes task failures, timeouts, and degraded throughput that directly impact operational availability. Company stock price (C) and developer social media activity (E) are unrelated to the technical health or availability of an AI pipeline and therefore are not monitoring components for operational availability.

Answer

Company stock price

Answer

Developer social media activity

Question 3

During an audit of an LLM-based application, which technique is most effective for testing the robustness against 'Prompt Injection'?

Accepted Answer

Red Teaming. Red teaming is the most effective technique for testing LLM robustness against prompt injection because it involves adversarial humans (or automated adversaries) actively crafting malicious prompts — direct overrides, role-play jailbreaks, indirect injections via retrieved documents — to probe whether the model can be manipulated. It surfaces real-world attack paths that static analysis or standard validation cannot. Sentiment analysis, cross-validation, and tokenization audits do not test adversarial prompt handling.

Answer

Sentiment analysis

Answer

Cross-validation

Answer

Tokenization audit

Question 4

You are reviewing a model risk management (MRM) framework for an AI system using NVIDIA NeMo Guardrails. What is the primary purpose of this tool in a risk mitigation strategy?

Accepted Answer

To enforce safety boundaries on LLM inputs and outputs. NeMo Guardrails allows developers to define programmatic boundaries for LLM behavior, effectively serving as an automated control for AI governance.

Answer

To automate model retraining

Answer

To improve model training speed

Answer

To encrypt training datasets

Question 5

When auditing model explainability, why is it risky to rely solely on 'Global Feature Importance'?

Accepted Answer

It fails to explain individual decisions. Global importance explains the model overall, but individual predictions (local) may be driven by different factors.

Answer

It lacks local prediction accuracy

Answer

It uses too much memory

Answer

It is only for regression models

Question 6

A board of directors requests a quarterly report on AI risk exposure. Which component of the NIST AI Risk Management Framework should you prioritize to demonstrate that the board's AI policy is being operationally enforced?

Accepted Answer

Govern. The Govern function of the NIST AI RMF establishes the policies, roles, and accountability structures that ensure AI risk management is operationalized across the organization. Demonstrating that a board-level AI policy is actually enforced requires evidence of governance mechanisms — assigned responsibilities, documented processes, and oversight — which is exactly what Govern covers.

Answer

Measure

Answer

Manage

Answer

Map

Question 7

When auditing model fairness in a production environment using IBM Watson OpenScale, which metric would you specifically validate to ensure the model is not violating protected class regulations?

Accepted Answer

Disparate Impact. Disparate Impact is the fairness metric in IBM Watson OpenScale that quantifies whether a model's favorable outcomes are disproportionately distributed across protected groups (e.g., race, gender). It is computed as the ratio of favorable outcome rates between the unprivileged and privileged groups, with values below 0.8 (the EEOC four-fifths rule) typically flagging potential regulatory violations. This directly addresses protected class compliance requirements under laws like ECOA and Title VII.

Answer

Precision-Recall AUC

Answer

Model Latency

Answer

Root Mean Square Error

Question 8

An auditor is evaluating bias in a model deployed on AWS SageMaker. Which tool should be used to generate a report on pre-training and post-training bias metrics?

Accepted Answer

SageMaker Clarify. SageMaker Clarify is the AWS service purpose-built to detect bias and explainability issues in ML models, and it generates both pre-training bias metrics (computed on the dataset before training) and post-training bias metrics (computed on model predictions). It integrates directly with SageMaker training and endpoints and produces a bias report that auditors can review. The other services handle monitoring, data preparation, or security scanning, not bias reporting.

Answer

SageMaker Model Monitor

Answer

AWS Glue DataBrew

Answer

Amazon Inspector

Question 9

You are reviewing the AI Incident Response Plan. Which scenario constitutes a 'High' severity AI incident that requires immediate board notification under most enterprise risk frameworks?

Accepted Answer

Systemic bias detected in a core automated lending model. An incident where a model makes a biased decision impacting a protected class carries significant regulatory and reputational risk, requiring board visibility.

Answer

A 200ms latency increase during inference

Answer

A minor change in the training dataset size

Answer

A 2% drop in model prediction accuracy

Question 10

When implementing drift detection in AWS SageMaker Model Monitor, which metric should be monitored to detect feature attribution shift for structured data?

Accepted Answer

Feature Attribution Drift (SHAP). Feature Attribution Drift, computed using SHAP (SHapley Additive exPlanations) values, measures how the contribution of each input feature to the model's prediction changes over time relative to a baseline. This is the correct metric for detecting feature attribution shift in structured data within SageMaker Model Monitor. It captures not just input distribution changes but changes in how the model weighs features.

Answer

Cosine Similarity

Answer

Mean Square Error

Answer

Precision-Recall AUC