AAIA AI Auditing Tools And Techniques Practice Test 3 — 15 Questions

Question 1

When auditing the 'training dataset' for a classification model, what should the auditor confirm regarding class balance?

Accepted Answer

That classes are equally represented or addressed. Class balance is a data-quality property: the auditor must confirm that the target classes are either roughly equally represented or that imbalance has been explicitly addressed (via resampling, class weights, or stratified splits). Severe imbalance can cause a model to achieve high accuracy by predicting the majority class while performing poorly on the minority class, which is a common source of biased or misleading performance. The other options concern format, size, or training efficiency, none of which address class balance.

Answer

That all data is in CSV format

Answer

That the dataset size is at least 1TB

Answer

That training time is minimized

Question 2

Which of these is a typical 'audit finding' in an AI governance review?

Accepted Answer

Insufficient model documentation. Insufficient model documentation is a classic audit finding because governance frameworks require documented model purpose, training data, assumptions, limitations, validation results, and ownership. Without this documentation, auditors cannot verify that the model is fit for purpose, compliant, or properly governed. The other options describe normal technical or deployment characteristics, not deficiencies.

Answer

Too many GPU cores

Answer

Cloud-based deployment

Answer

Use of Python language

Question 3

What is the primary benefit of using 'Containerization' for AI model auditability?

Accepted Answer

Environment reproducibility. Containerization packages an AI model along with its entire runtime environment—libraries, dependencies, and configuration files—into a single immutable image. This ensures that the model executes in exactly the same way regardless of where it is deployed, which is critical for auditability because auditors can reproduce the exact conditions under which a model was trained or inferred. Without such reproducibility, verifying model behavior or tracing decisions becomes unreliable due to environmental drift.

Answer

Reduced data storage costs

Answer

Increased hardware speed

Answer

Simplified UI design

Question 4

When auditing fairness metrics, what does the 'Demographic Parity' metric measure?

Accepted Answer

Equal positive outcome rates. Demographic Parity (also called statistical parity) requires that the probability of a positive prediction is equal across protected groups — i.e., P(Ŷ=1 | A=0) = P(Ŷ=1 | A=1). It measures outcome-rate parity, not error rates or accuracy. It is one of the simplest group fairness definitions and is often used in hiring, lending, and admissions audits.

Answer

Equal training data representation

Answer

Equal error rates

Answer

Equal accuracy rates

Question 5

An auditor is evaluating the 'Safety Filter' of an LLM. Which approach is most suitable for detecting 'jailbreak' vulnerabilities?

Accepted Answer

Adversarial red teaming. Red teaming with adversarial prompts (jailbreaks) is the standard method for testing safety filter resilience.

Answer

Running unit tests for code

Answer

Reviewing server logs

Answer

Checking network bandwidth

Question 6

During an AI audit, what is a 'PII Scrubber' tool used for?

Accepted Answer

Anonymizing sensitive data. A PII Scrubber is used to detect and anonymize personally identifiable information such as names, addresses, SSNs, and emails so that sensitive data can be used in AI training or analysis without exposing individuals. It supports privacy compliance by removing or masking identifiers before data flows into models or logs. This is a core privacy-preserving control in AI audits.

Answer

Cleaning up logs

Answer

Compressing images

Answer

Improving accuracy

Question 7

When conducting an audit, what is the 'Model Registry' entry for a model version supposed to contain?

Accepted Answer

The model artifact, environment, and metadata. It should contain the model binary, the environment definition (requirements.txt), and the experiment metadata (the 'Run' ID).

Answer

Only the model binary

Answer

Only the training dataset

Answer

Only the developer's credentials

Question 8

When auditing an AI system's 'Explainability', which issue is highlighted by 'Explanation Faithfulness'?

Accepted Answer

Whether the explanation accurately represents the model's logic. Explanation Faithfulness measures whether the explanation produced by an interpretability method (e.g., SHAP, LIME, attention weights, counterfactual) actually reflects the true internal reasoning the model used to produce its output. An explanation can be plausible and readable yet still be unfaithful — for example, post-hoc saliency maps that highlight features the model did not actually rely on. Auditors care about faithfulness because unfaithful explanations create false assurance and can mask discriminatory or unsafe decision logic.

Answer

Whether the model is fast enough

Answer

Whether the user trusts the model

Answer

Whether the explanation is easy to read

Question 9

When scoping an AI audit engagement, which TWO of the following documents should the auditor request to understand the AI model's governance structure?

Accepted Answer

Technical system architecture documentation. Option D, technical system architecture documentation, is correct because it describes how the AI model is built, deployed, and integrated with surrounding systems, revealing the governance-relevant controls, data flows, and ownership boundaries an auditor needs to assess. Option E, the Model Card, is correct because it is the standard artifact documenting a model's intended use, training data, performance metrics, limitations, and ethical considerations, which directly evidences the model's governance structure. The unmarked options do not belong: public social media posts (A) and company marketing brochures (B) are promotional or informal and not authoritative governance records, while developer salary structures (C) are HR compensation data unrelated to AI model governance.

Answer

Public social media posts

Answer

Company marketing brochures

Answer

Developer salary structures

Question 10

Which THREE factors should an auditor consider when evaluating the suitability of an AI model for production?

Accepted Answer

Inference latency and throughput. Option A (Inference latency and throughput) is correct because production suitability depends on whether the model can serve requests within required response-time SLAs and handle the expected request volume, making latency and throughput key operational metrics. Option C (Fairness and bias metrics) is correct because an auditor must verify that the model does not produce discriminatory or skewed outcomes across protected groups, which is a core governance and compliance concern. Option D (Model accuracy metrics) is correct because accuracy, precision, recall, F1, or similar performance measures determine whether the model reliably performs its intended task in production. Options B (Developer age demographics) and E (Operating system color schemes) are irrelevant to model suitability, as neither affects the model's performance, fairness, or operational viability.

Answer

Developer age demographics

Answer

Operating system color schemes

Question 11

When documenting audit findings for an AI system, which TWO of the following are critical to include to ensure the audit can be replicated?

Accepted Answer

Specific Git commit SHA for the model code. Option A is correct because recording the specific Git commit SHA pins the exact revision of the model code that was audited, so anyone can check out that same commit and reproduce the training or inference behavior without ambiguity from later changes. Option C is correct because capturing the version ID or hash of the training dataset uniquely identifies the exact data snapshot used, ensuring the same inputs can be retrieved and the results independently verified. Together, the code commit SHA and dataset version/hash form the minimal provenance pair needed for a replicable audit. Options B, D, and E are not relevant to reproducibility: office location, coffee machine brand, and a developer's personal email address have no bearing on the model's code, data, or computational results.

Answer

The office location of the data scientists

Answer

The brand of the coffee machine used

Answer

The developer's personal email address

Question 12

Which TWO techniques should an auditor employ to detect bias in a model where the training data has imbalanced demographic representation?

Accepted Answer

Implementation of fairness-aware loss functions. Fairness-aware loss functions (B) are correct because they explicitly incorporate group-fairness constraints or penalties (e.g., equalized odds, demographic parity regularization) into the model's objective during training, directly counteracting bias caused by imbalanced demographic representation. Stratified cross-validation (E) is correct because it preserves the proportional class and demographic distribution in each fold, ensuring that performance and bias metrics are evaluated reliably across underrepresented groups rather than being masked by skewed splits. The remaining options do not belong: updating the corporate logo (A) and changing the font size in the UI (C) are cosmetic and have no effect on model training or bias, while increasing the number of project meetings (D) is a process activity that does not alter data imbalance or model fairness.

Answer

Updating the corporate logo

Answer

Changing the font size in the UI

Answer

Increasing the number of project meetings

Question 13

When conducting an audit, which THREE of the following represent potential 'Model Risk' areas that require documentation?

Accepted Answer

Susceptibility to adversarial attacks. Option A is correct because susceptibility to adversarial attacks is a model risk area: AI/ML models can be manipulated by crafted inputs (e.g., evasion or poisoning attacks) that cause incorrect outputs, and this vulnerability must be documented during an audit. Option C is correct because inherent bias in decision-making is a core model risk: training data or algorithmic design can embed bias that leads to unfair or discriminatory outcomes, requiring documentation and mitigation. Option D is correct because susceptibility to data drift is a model risk: changes in input data distributions over time (data drift) or changes in the relationship between inputs and outputs (concept drift) degrade model performance and must be monitored and documented. Option B is incorrect because the color of server racks is a physical facility detail with no bearing on model behavior or risk. Option E is incorrect because local internet connection speed is an infrastructure/network performance factor, not a model risk area.

Answer

The color of the server racks

Answer

The speed of the local internet connection

Question 14

Which TWO methods are commonly used by auditors to verify the 'Provenance' of training data?

Accepted Answer

Reviewing data lineage logs in the metadata store. Option C is correct because provenance verification relies on data lineage records maintained in a metadata store, which document the origin, transformations, and movement of each dataset through the pipeline, giving auditors a traceable chain of custody. Option D is correct because computing cryptographic hashes (e.g., SHA-256) of input files produces a unique fingerprint that can be compared against recorded values to confirm the data has not been altered and matches its claimed source. Options A, B, and E do not belong: watching a training video only shows content, not origin or integrity; interviewing janitorial staff has no bearing on data provenance controls; and checking whether a model is open source describes licensing or model availability, not the lineage or integrity of the training data itself.

Answer

Watching the training video

Answer

Interviewing the janitorial staff

Answer

Checking if the model is open source

Question 15

Which TWO metrics provide the best insights into 'Model Quality' during an audit?

Accepted Answer

Recall score. Precision and Recall are fundamental metrics for assessing the quality of classification models.

Answer

Disk space usage

Answer

Number of server restarts

Answer

Team's average coffee intake