AI0-001 AI Security Practice Test 6 — 15 Questions

Question 1

An attacker repeatedly queries a public LLM API with carefully crafted inputs to reconstruct the model's architecture and approximate weights. This is an example of which attack?

Accepted Answer

Model extraction. Model extraction attacks involve querying a public API with carefully crafted inputs to reconstruct a target model's architecture and approximate weights. By analyzing the outputs (e.g., logits or probabilities), an attacker can train a substitute model that mimics the original, enabling offline exploitation or competitive intelligence. This directly matches the scenario described.

Answer

Data poisoning

Answer

Membership inference

Answer

Model inversion

Question 2

A healthcare organization is deploying an AI model to predict patient readmission risk. They must comply with regulations that protect patient privacy. Which TWO techniques should they implement to enhance privacy preservation?

Accepted Answer

Differential privacy. Differential privacy (B) is correct because it adds calibrated noise to the training data or model outputs, ensuring that the inclusion or exclusion of any single patient's record does not significantly affect the model's predictions. This provides a formal mathematical guarantee of privacy, which is essential for complying with regulations like HIPAA that protect patient data.

Answer

Data augmentation

Answer

Model quantization

Answer

Dropout regularization

Question 3

A company is developing an AI-powered recruitment tool. To prevent bias and ensure fairness, they want to audit the model's training data and outputs. Which TWO practices should they implement as part of secure AI development?

Accepted Answer

Threat modeling using STRIDE for AI-specific threats. Threat modeling for AI systems helps identify bias-related threats, and access controls on training data prevent unauthorized modifications that could introduce bias. Both are part of secure AI development practices.

Answer

Enabling model parallelism

Answer

Increasing the model's learning rate

Answer

Using a larger batch size

Question 4

A company deploys an LLM-based application that retrieves external web content to answer user queries. An attacker crafts a webpage that, when retrieved, injects a hidden instruction telling the LLM to ignore its system prompt and output sensitive internal data. What type of attack is this?

Accepted Answer

Indirect prompt injection. Indirect prompt injection occurs when malicious instructions are embedded in content retrieved by the LLM, as opposed to direct injection where the user themselves provides the malicious prompt.

Answer

Direct prompt injection

Answer

Jailbreaking

Answer

Model inversion attack

Question 5

A security engineer is conducting threat modeling for an AI system that uses a pre-trained image classifier. Applying STRIDE, which threat category most directly addresses an attacker manipulating the model's behavior by providing carefully crafted inputs that the model was not trained to handle robustly?

Accepted Answer

Tampering. Tampering involves unauthorized modification of data or systems. In this context, adversarial examples tamper with the input data to alter the model's behavior. Spoofing is about impersonation, Repudiation is about denying actions, and Information disclosure is about exposing sensitive data.

Answer

Repudiation

Answer

Information disclosure

Answer

Spoofing

Question 6

An AI team is concerned about their model leaking sensitive information from its training data when queried. Which privacy-preserving technique adds noise to the training process to limit what can be inferred about any individual record?

Accepted Answer

Differential privacy. Differential privacy (A) is the correct answer because it directly addresses the concern of leaking sensitive information from training data by adding calibrated noise to the training process or query responses. This noise ensures that the output of the model does not significantly change whether any single individual's record is included or excluded, thereby limiting what can be inferred about any specific record. The technique is formalized through a privacy budget (ε, epsilon) that quantifies the privacy guarantee, making it the standard approach for privacy-preserving machine learning.

Answer

Homomorphic encryption

Answer

Data sanitization

Answer

Federated learning

Question 7

A company uses an LLM API to generate customer support responses. They want to prevent the LLM from generating harmful content, even when users attempt jailbreaking. Which defense is MOST effective at the application layer?

Accepted Answer

Output filtering and content moderation. Output filtering and content moderation is the most effective defense at the application layer because it directly inspects the LLM's generated response before it reaches the user. This approach can catch and block harmful content that results from successful jailbreaking attempts, which input validation alone cannot prevent since the model may still produce undesirable outputs even with sanitized inputs.

Answer

Input validation and sanitization

Answer

Robust training techniques

Answer

Rate limiting

Question 8

A security analyst discovers that an attacker has been querying a production LLM API with thousands of carefully crafted prompts and using the responses to build a local copy of the model. Which attack is occurring?

Accepted Answer

Model extraction. Model extraction attacks involve querying a model to reconstruct its functionality or parameters. Membership inference determines if data was in training, data poisoning corrupts training, and prompt injection manipulates prompts.

Answer

Prompt injection

Answer

Data poisoning

Answer

Membership inference

Question 9

An organization is adopting a third-party pre-trained language model for internal use. To assess supply chain security, which document should they request to understand the components and dependencies of the model?

Accepted Answer

Software Bill of Materials (SBOM). A Software Bill of Materials (SBOM) is the correct document for assessing supply chain security because it provides a detailed, machine-readable inventory of all components, libraries, and dependencies used to build the model. This allows the organization to identify known vulnerabilities, licensing risks, and transitive dependencies, which is essential for evaluating the security posture of a third-party pre-trained model.

Answer

OWASP LLM Top 10

Answer

Model card

Answer

Data flow diagram

Question 10

A developer is building an AI-powered code completion tool. They want to ensure that the tool does not inadvertently suggest insecure code patterns. Which practice is MOST effective for reducing this risk?

Accepted Answer

Red teaming the AI system. Red teaming involves adversarial testing to find vulnerabilities. Output filtering can catch some insecure suggestions but may not cover all patterns. Secure data pipelines focus on training data security, not output. Rate limiting is unrelated.

Answer

Rate limiting

Answer

Secure data pipelines

Answer

Output filtering of insecure patterns

Question 11

An organization uses a fine-tuned LLM for generating financial reports. An attacker gains access to the model's API and sends a series of queries that gradually reconstruct the training data of the fine-tuned model. This is an example of which attack?

Accepted Answer

Model inversion. Model inversion attacks aim to reconstruct training data from model outputs. Data poisoning corrupts training, model extraction steals the model, and membership inference determines presence, not reconstruction.

Answer

Membership inference

Answer

Data poisoning

Answer

Model extraction

Question 12

A company deploys an LLM chatbot that has access to a database of customer orders. They want to prevent the LLM from revealing order details unless the user is authenticated as the owner. Which security control should be implemented?

Accepted Answer

Access controls on the model and API. Access controls on the model and API (Option D) are the correct security control because they enforce authentication and authorization at the API gateway or model endpoint level, ensuring that only the authenticated owner can query their own order details. This prevents unauthorized users from invoking the LLM to retrieve sensitive data, regardless of the prompt content. Without such access controls, the LLM would have no inherent mechanism to verify user identity before processing requests.

Answer

Output filtering

Answer

Rate limiting

Answer

Input validation and sanitization

Question 13

During a penetration test, a security analyst crafts a prompt that tricks an LLM into revealing its system prompt. Which OWASP LLM Top 10 vulnerability does this exploit?

Accepted Answer

Sensitive information disclosure. Tricking an LLM into revealing its system prompt directly exposes sensitive configuration details, which is a classic case of sensitive information disclosure. This vulnerability occurs when an attacker uses crafted inputs to bypass the model's intended boundaries and extract internal data that should remain confidential.

Answer

Model denial of service

Answer

Insecure output handling

Answer

Prompt injection

Question 14

A machine learning engineer wants to prevent data poisoning during the training of a model. Which practice is MOST effective for ensuring the integrity of the training data?

Accepted Answer

Secure data pipelines. Secure data pipelines include validation, checksums, and access controls to ensure data integrity. Output filtering is for outputs, red teaming tests the model, and differential privacy adds noise but does not prevent poisoning.

Answer

Differential privacy

Answer

Red teaming the model

Answer

Output filtering

Question 15

A security team is reviewing an LLM-powered application that can execute SQL queries based on user requests. They want to implement defenses against prompt injection that could lead to unauthorized database access. Which TWO controls are MOST effective? (Select two.)

Accepted Answer

Input validation and sanitization. Input validation and sanitization can filter malicious prompt content, and output filtering can block responses containing unauthorized data. Rate limiting does not prevent injection, access controls on the database are important but not a direct defense against injection (they limit impact), and robust training does not prevent injection at inference time.

Answer

Robust training techniques

Answer

Access controls on the database

Answer

Rate limiting