AAIA Hard Questions — 25 Difficult Questions

Question 1

When conducting an audit, which THREE of the following represent potential 'Model Risk' areas that require documentation?

Accepted Answer

Susceptibility to adversarial attacks. Adversarial vulnerability, data drift, and bias are three primary risks requiring documentation in an AI audit report.

Answer

The color of the server racks

Answer

The speed of the local internet connection

Question 2

Which TWO of the following are legitimate 'AI Governance' concerns regarding third-party AI service providers (SaaS)?

Accepted Answer

Data residency and compliance with local regulations. Auditors must ensure that third-party AI tools comply with internal data handling standards and that the vendor's own governance is adequate.

Answer

The programming language used by the vendor

Answer

Vendor's model update and retraining frequency

Answer

The physical location of the vendor's corporate office

Question 3

When selecting testing techniques for an AI model, which THREE are considered 'Model-Agnostic'?

Accepted Answer

SHAP (SHapley Additive exPlanations). LIME, SHAP, and Partial Dependence Plots (PDPs) are model-agnostic methods that work with any ML model.

Answer

Neural network weight pruning

Answer

Linear regression coefficients

Question 4

You are auditing a model's 'Robustness' against adversarial attacks. Which test is most appropriate?

Accepted Answer

Performing adversarial perturbation testing. Adversarial testing involves introducing perturbations to inputs to see if the model's output changes unexpectedly.

Answer

Checking if the server is plugged in

Answer

Checking the model's training accuracy

Answer

Asking the team to describe their favorite movies

Question 5

Which THREE criteria are essential when selecting an AI deployment strategy?

Accepted Answer

Service Level Agreements (SLAs). Strategy must balance risk, speed, and resource availability.

Answer

Color of the UI

Answer

Number of coffee machines

Question 6

An auditor finds that a model's 'input feature importance' has changed significantly after a retrain. What is the most appropriate action?

Accepted Answer

Investigate source data and feature logic. Investigate if the change is due to a shift in data or a feature engineering error.

Answer

Ignore the change

Answer

Delete the model

Answer

Upgrade the hardware

Question 7

Which operational process is required to ensure 'Data Privacy' when using user-generated data for model retraining?

Accepted Answer

Data anonymization. Anonymization or pseudonymization must be performed before training.

Answer

Increasing compute power

Answer

Model quantization

Answer

Removing the model

Question 8

When auditing a model deployment pipeline, which TWO aspects are critical to verify to ensure compliance with AI governance frameworks?

Accepted Answer

Validation of data processing against privacy policies. Governance requires both reproducibility and data privacy/security checks.

Answer

Whether the team uses a specific cloud provider

Answer

The speed of the training job

Answer

Usage of proprietary algorithms only

Question 9

When auditing model deployment, what is the primary purpose of 'Shadow Mode' testing?

Accepted Answer

To compare model performance against production traffic without impacting users. Shadow mode allows running the new model in production alongside the old one to compare outputs without affecting users.

Answer

To encrypt model weights in memory

Answer

To hide model artifacts from unauthorized users

Answer

To automate hyperparameter optimization

Question 10

When documenting audit findings for an AI system, which TWO of the following are critical to include to ensure the audit can be replicated?

Accepted Answer

Specific Git commit SHA for the model code. Version control information and the exact dataset version are essential for reproducing the audit results.

Answer

The office location of the data scientists

Answer

The brand of the coffee machine used

Answer

The developer's personal email address

Question 11

An auditor is evaluating an AI system for 'Model Inversion' risk. What is this?

Accepted Answer

Reconstructing training data from outputs. Model Inversion is an attack where an adversary reconstructs training data from model outputs.

Answer

Reducing inference latency

Answer

Increasing training bias

Answer

Overfitting the data

Question 12

An auditor is evaluating the 'Safety Filter' of an LLM. Which approach is most suitable for detecting 'jailbreak' vulnerabilities?

Accepted Answer

Adversarial red teaming. Red teaming with adversarial prompts (jailbreaks) is the standard method for testing safety filter resilience.

Answer

Running unit tests for code

Answer

Reviewing server logs

Answer

Checking network bandwidth

Question 13

When documenting findings, which THREE elements should be included for each finding?

Accepted Answer

The root cause. The condition, the root cause, and the recommendation are essential components of an audit finding.

Answer

The font size used in the report

Answer

The names of all employees who disagreed

Question 14

An organization is using 'Federated Learning' to maintain data privacy while training models. What is the primary audit risk associated with this architecture?

Accepted Answer

Model inversion attacks targeting the central aggregator. In Federated Learning, local model updates (gradients) can potentially reveal information about the underlying training data, necessitating 'Differential Privacy' as a control.

Answer

Increased training latency

Answer

Lack of access to training hardware

Answer

Centralized data breaches

Question 15

In an audit of differential privacy implementations, what is the 'epsilon' parameter used for?

Accepted Answer

Privacy budget control. Epsilon is the privacy budget; lower values provide stronger privacy guarantees but potentially lower utility.

Answer

Model training speed

Answer

Learning rate scheduling

Answer

Database query limit

Question 16

You are auditing a 'Model Monitoring' dashboard. Which metric is most indicative of a potential degradation in the model's reliability over time?

Accepted Answer

Statistical drift between the production input data and training baseline. Data drift and concept drift are primary indicators that a model's performance may be degrading in a production environment.

Answer

The physical location of the cloud server hosting the model

Answer

The number of lines of code in the model deployment script

Answer

The number of users accessing the model API

Question 17

You find that the 'Inference API' for a model allows 'Prompt Injection' attacks. Which governance failure is most likely?

Accepted Answer

Inadequate input validation controls and adversarial testing. Prompt injection is a failure of input validation and security design within the model governance framework.

Answer

Lack of a social media presence for the company

Answer

Using a low-cost cloud provider

Answer

Failure to update the company logo

Question 18

When using 'LIME' (Local Interpretable Model-agnostic Explanations) for auditing, what is the auditor looking for?

Accepted Answer

Local decision transparency. LIME creates a local surrogate model to explain individual predictions, helping the auditor understand why a specific decision was made.

Answer

Data distribution drift

Answer

Overall model performance

Answer

Adversarial robustness

Question 19

An auditor is evaluating the data pipeline security. Which THREE controls should be verified?

Accepted Answer

Full audit log of data provenance. Access control, encryption at rest, and data provenance are standard security controls for AI data pipelines.

Answer

The number of lines of code in the pipeline

Answer

Office access badge logs

Question 20

Which technique is most appropriate for mitigating 'Concept Drift' in a production model?

Accepted Answer

Implementing an automated re-training pipeline based on performance threshold triggers. Continuous re-training or fine-tuning based on recent data is the standard mitigation for concept drift.

Answer

Increasing the number of neurons in the hidden layers

Answer

Moving the model to a larger GPU cluster

Answer

Restricting access to the training dataset