AAIA Practice Test 10 — 10 Questions

Question 1

When conducting an audit, what is the 'Model Registry' entry for a model version supposed to contain?

Accepted Answer

The model artifact, environment, and metadata. It should contain the model binary, the environment definition (requirements.txt), and the experiment metadata (the 'Run' ID).

Answer

Only the model binary

Answer

Only the training dataset

Answer

Only the developer's credentials

Question 2

An auditor is evaluating the data pipeline security. Which THREE controls should be verified?

Accepted Answer

Full audit log of data provenance. Option A (Full audit log of data provenance) is correct because auditors must verify an immutable, traceable record of where data originated, how it was transformed, and who accessed it, which is essential for detecting tampering and meeting compliance requirements such as those in SOC 2 or GDPR. Option B (Identity and Access Management (IAM) for the pipeline) is correct because IAM controls enforce least-privilege access, role-based permissions, and authentication for pipeline components, directly preventing unauthorized data access or modification. Option E (Encryption of data at rest) is correct because protecting stored pipeline data with encryption (e.g., AES-256 at rest via KMS-managed keys) is a fundamental confidentiality control that auditors must confirm to prevent exposure from compromised storage. Option C (number of lines of code) is not a security control—code volume has no bearing on pipeline security posture. Option D (office access badge logs) relates to physical facility security, not the data pipeline's logical security controls, so it is out of scope for this audit.

Answer

The number of lines of code in the pipeline

Answer

Office access badge logs

Question 3

Which aspect of 'Data Drift' does the 'Population Stability Index' (PSI) measure?

Accepted Answer

Change in population distribution. The Population Stability Index (PSI) measures how much the distribution of a population (typically the input features or scores) has shifted between two time periods, such as training versus current production data. It quantifies change in population distribution, not model parameters or accuracy. PSI is a standard drift-detection metric used in credit risk and other regulated modeling contexts.

Answer

Change in model parameters

Answer

Change in model accuracy

Answer

Increase in inference latency

Question 4

Which technique is recommended for auditing 'data leakage' in a feature engineering pipeline?

Accepted Answer

Temporal cross-validation. Cross-validation using temporal splits (rather than random splits) prevents training on future information.

Answer

Feature ablation

Answer

Normalization

Answer

Random sampling

Question 5

Which THREE items should be tracked in a 'Model Registry' for audit readiness?

Accepted Answer

Training dataset lineage. A is correct because training dataset lineage documents which data was used to train a model, which is essential for audit trails, reproducibility, and compliance with data governance requirements. B is correct because a source code reference such as a Git commit uniquely identifies the exact code version that produced the model, enabling auditors to trace and reproduce the training pipeline. E is correct because a model artifact hash provides cryptographic integrity verification, ensuring the deployed model has not been altered and matches the audited version. C is not appropriate because a cloud provider logo is merely branding and carries no audit or traceability value. D is not appropriate because employee birthday is personal data unrelated to model governance and would raise privacy concerns rather than support audit readiness.

Answer

Cloud provider logo

Answer

Employee birthday

Question 6

During an audit of a model inference endpoint, you observe that 'Model Explanation' logs are missing. Which library or tool was likely omitted from the deployment?

Accepted Answer

SHAP integration. Model Explanation logs on an inference endpoint are produced by SHAP-based explainers integrated into the scoring pipeline; if those logs are missing, the SHAP integration was almost certainly omitted from the deployment. Azure ML's responsible-AI tooling and SageMaker Clarify both rely on SHAP to generate per-request or batch explanations, so the absence of explanation logs points directly to a missing SHAP component. The other options are infrastructure or monitoring tools that do not generate model explanations.

Answer

Kubernetes manifest

Answer

TensorBoard logging

Answer

Prometheus exporter

Question 7

When auditing an 'AI Model Lifecycle', which phase occurs immediately after the 'Model Training' phase?

Accepted Answer

Model Validation. After training, the model must be validated before it is deployed to production.

Answer

Writing the initial requirements

Answer

Marketing the model to customers

Answer

Decommissioning

Question 8

In the context of 'AI Compliance', what is a 'Model Card'?

Accepted Answer

A document providing transparency regarding model features, limitations, and training data. A Model Card is a structured transparency document (popularized by Mitchell et al., 2019) that describes a model's intended use, training data, evaluation metrics, limitations, ethical considerations, and performance across demographic groups. It is the standard artifact for AI compliance and responsible-AI reporting.

Answer

A physical access card for the data center

Answer

A credit card used for paying cloud compute bills

Answer

A list of all users authorized to access the AI

Question 9

When auditing an AI system's 'Explainability', which issue is highlighted by 'Explanation Faithfulness'?

Accepted Answer

Whether the explanation accurately represents the model's logic. Explanation Faithfulness measures whether the explanation produced by an interpretability method (e.g., SHAP, LIME, attention weights, counterfactual) actually reflects the true internal reasoning the model used to produce its output. An explanation can be plausible and readable yet still be unfaithful — for example, post-hoc saliency maps that highlight features the model did not actually rely on. Auditors care about faithfulness because unfaithful explanations create false assurance and can mask discriminatory or unsafe decision logic.

Answer

Whether the model is fast enough

Answer

Whether the user trusts the model

Answer

Whether the explanation is easy to read

Question 10

You are auditing a firm using LLMs (Large Language Models). Which risk is most specific to generative AI compared to traditional predictive models?

Accepted Answer

Hallucinations and generation of misinformation. Hallucination is a unique risk of generative models where they output plausible but factually incorrect information.

Answer

Database login credentials being shared

Answer

Server power consumption

Answer

Missing documentation for the training code