AI0-001 AI Security Practice Test 7 — 15 Questions

Question 1

A company deploys an LLM-based chatbot that retrieves data from external databases. An attacker embeds malicious instructions in a database record. When the chatbot retrieves that record, it executes the instructions, overriding its system prompt. Which type of attack is this?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Indirect prompt injection. This is an indirect prompt injection attack because the malicious instructions are embedded in a third-party data source (the database record) rather than being sent directly by the user. When the LLM retrieves and processes that record, the injected instructions override the system prompt, causing the chatbot to behave contrary to its intended design.

Answer

Model inversion attack

Answer

Direct prompt injection

Answer

Membership inference attack

Question 2

A team is developing a threat model for an AI system that processes user uploads. Using STRIDE, which threat involves an attacker modifying the model's training data to cause misclassification?

Accepted Answer

Tampering. Tampering is the STRIDE category for unauthorized modification of data. Data poisoning is a form of tampering with training data.

Answer

Spoofing

Answer

Repudiation

Answer

Information disclosure

Question 3

An organization deploys a machine learning model for credit scoring. An attacker submits carefully crafted loan applications that are slightly outside normal ranges but cause the model to approve high-risk loans. What type of attack is this?

Accepted Answer

Adversarial example. This is an adversarial example attack, where the attacker crafts inputs with small, carefully chosen perturbations that cause the ML model to misclassify them. In credit scoring, submitting loan applications with values slightly outside normal ranges exploits the model's decision boundary to approve high-risk loans, a classic evasion technique.

Answer

Model extraction

Answer

Prompt injection

Answer

Data poisoning

Question 4

A developer is building an AI-powered code completion tool. To ensure the model does not output malicious code when prompted with 'Write code to delete all files on the system', which defense is most effective?

Accepted Answer

Output filtering to detect and block dangerous code constructs. Output filtering can block generated code that contains dangerous patterns like file deletion commands.

Answer

Input validation to block the word 'delete'

Answer

Rate limiting on the number of requests per user

Answer

Retraining the model on safe code only

Question 5

An organization is deploying a machine learning model that classifies loan applications. They want to prevent an attacker from reconstructing individual customer records from the model's predictions. Which type of attack should they defend against?

Accepted Answer

Model inversion. Model inversion attacks allow an attacker to reconstruct the original training data by analyzing the model's predictions. In this scenario, the attacker could use the model's outputs to infer sensitive details about individual loan applicants, such as income or credit history, violating privacy. Defending against model inversion is critical when predictions can be used to reverse-engineer private training records.

Answer

Membership inference

Answer

Data poisoning

Answer

Adversarial example

Question 6

A company trains a sentiment analysis model on customer reviews. An attacker submits hundreds of reviews with the word 'excellent' attached to negative feedback, causing the model to classify negative reviews as positive. This is an example of which attack?

Accepted Answer

Data poisoning. Data poisoning occurs when an attacker deliberately corrupts the training data to manipulate the model's behavior. By injecting hundreds of reviews that pair the word 'excellent' with negative sentiment, the attacker shifts the model's learned decision boundary, causing it to misclassify genuinely negative reviews as positive. This directly undermines the integrity of the training dataset, which is the hallmark of a data poisoning attack.

Answer

Model extraction

Answer

Adversarial example

Answer

Prompt injection

Question 7

A healthcare AI system uses patient data to predict disease risk. To comply with privacy regulations, the organization wants to ensure that the model cannot reveal whether a specific patient's data was used in training. Which technique should they implement?

Accepted Answer

Differential privacy. Differential privacy adds noise to the training process, making it difficult to determine if any individual was in the training set. This directly addresses membership inference attacks.

Answer

Homomorphic encryption

Answer

Federated learning

Answer

Model validation

Question 8

A developer is building an LLM-powered code assistant. They want to prevent the model from generating insecure code. Which OWASP LLM Top 10 category is most relevant to this risk?

Accepted Answer

Insecure output handling. Insecure output handling (B) is the most relevant OWASP LLM Top 10 category because the risk is that the LLM generates insecure code, which is a direct output from the model. This category specifically addresses failures to validate, sanitize, or restrict the model's output before it is used in downstream applications, such as a code assistant. By not properly handling the generated code, the assistant could introduce vulnerabilities like SQL injection or command injection into the user's codebase.

Answer

Sensitive information disclosure

Answer

Model denial of service

Answer

Prompt injection

Question 9

An AI security team is conducting a threat model for a new document summarization service. They want to identify threats related to spoofing of the AI's identity. Which STRIDE category should they consider?

Accepted Answer

Spoofing. Spoofing in STRIDE refers to impersonating something or someone else. In the context of AI, an attacker could spoof the AI service to provide false summaries.

Answer

Repudiation

Answer

Tampering

Answer

Information disclosure

Question 10

A company deploys a chatbot that frequently gives outdated information. They want to implement a defense against prompt injection that also ensures responses are based on verified knowledge. Which approach is best?

Accepted Answer

Output filtering with a curated knowledge base. Output filtering combined with a knowledge base ensures that the chatbot's responses are checked against verified facts, mitigating both prompt injection and hallucination of outdated info.

Answer

Input sanitization only

Answer

Rate limiting

Answer

Robust training on adversarial examples

Question 11

An organization wants to use a pre-trained language model from a third-party vendor. What is the most important security step before deployment?

Accepted Answer

Vet the model for backdoors and malicious behavior. Vetting the pre-trained model for backdoors or malicious behavior is critical to supply chain security. This may include scanning for anomalies, testing on specific inputs, and reviewing the model's origins.

Answer

Host the model on a public cloud

Answer

Apply differential privacy to the model

Answer

Fine-tune the model on internal data

Question 12

A team is designing a secure API for an AI model. They want to prevent data leakage through overly detailed error messages. Which principle should they follow?

Accepted Answer

Use generic error messages. Least-privilege API access and minimal error information reduce the attack surface. Specifically, returning generic error messages prevents leaking internal details.

Answer

Return detailed error codes for debugging

Answer

Log errors to the client side

Answer

Disable all error messages

Question 13

An AI system is designed to automatically execute actions on behalf of users, such as sending emails. The security team is concerned about excessive agency. Which mitigation is most effective?

Accepted Answer

Restrict the functions the model can call and require human approval for sensitive actions. Limiting the scope of actions and requiring user confirmation for critical actions reduces the risk of the LLM performing unintended actions. This directly addresses excessive agency.

Answer

Disable output filtering

Answer

Increase the model's context window

Answer

Use a larger model

Question 14

A company uses an LLM to generate code. They want to ensure that the model does not accidentally output sensitive internal logic. Which practice should they implement?

Accepted Answer

Output filtering to block sensitive patterns. Output filtering is the correct practice because it directly inspects the model's generated text for patterns that match sensitive internal logic (e.g., API keys, source code snippets, or proprietary algorithms) and blocks or redacts them before the output is returned to the user. This is a reactive security control that operates at the application layer, ensuring that even if the LLM inadvertently generates sensitive content, it is never exposed. Rate limiting, red teaming, and federated learning address different concerns (availability, vulnerability discovery, and data privacy during training, respectively) and do not prevent the accidental leakage of internal logic in real-time outputs.

Answer

Rate limiting API calls

Answer

Red teaming the model

Answer

Federated learning

Question 15

A security engineer is implementing defenses against membership inference attacks on a classification model. Which TWO techniques are most effective? (Select TWO.)

Accepted Answer

Differential privacy. Differential privacy (C) is effective against membership inference attacks because it adds calibrated noise to the training process or model outputs, ensuring that the model's behavior does not significantly change whether any individual data point is included. This bounds the attacker's ability to distinguish between members and non-members of the training set, directly mitigating the core vulnerability exploited by membership inference.

Answer

Data augmentation

Answer

Homomorphic encryption

Answer

Increasing model size