AI0-001 AI Security Practice Test 5 — 15 Questions

Question 1

A company uses an LLM API to generate customer support responses. They want to prevent the LLM from generating harmful content, even when users attempt jailbreaking. Which defense is MOST effective at the application layer?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Output filtering and content moderation. Output filtering and content moderation is the most effective defense at the application layer because it directly inspects the LLM's generated response before it reaches the user. This approach can catch and block harmful content that results from successful jailbreaking attempts, which input validation alone cannot prevent since the model may still produce undesirable outputs even with sanitized inputs.

Answer

Input validation and sanitization

Answer

Robust training techniques

Answer

Rate limiting

Question 2

A security analyst discovers that an attacker has been querying a production LLM API with thousands of carefully crafted prompts and using the responses to build a local copy of the model. Which attack is occurring?

Accepted Answer

Model extraction. Model extraction (also called model stealing) occurs when an attacker queries a deployed model API repeatedly with crafted inputs and uses the input-output pairs to train a surrogate model that approximates the target's behavior. The scenario—thousands of queries used to build a local copy—is the textbook definition. The goal is to replicate the model's functionality, often to avoid API costs or to enable further attacks like adversarial example transfer.

Answer

Prompt injection

Answer

Data poisoning

Answer

Membership inference

Question 3

An organization is adopting a third-party pre-trained language model for internal use. To assess supply chain security, which document should they request to understand the components and dependencies of the model?

Accepted Answer

Software Bill of Materials (SBOM). A Software Bill of Materials (SBOM) is the correct document for assessing supply chain security because it provides a detailed, machine-readable inventory of all components, libraries, and dependencies used to build the model. This allows the organization to identify known vulnerabilities, licensing risks, and transitive dependencies, which is essential for evaluating the security posture of a third-party pre-trained model.

Answer

OWASP LLM Top 10

Answer

Model card

Answer

Data flow diagram

Question 4

An organization uses a fine-tuned LLM for generating financial reports. An attacker gains access to the model's API and sends a series of queries that gradually reconstruct the training data of the fine-tuned model. This is an example of which attack?

Accepted Answer

Model inversion. Model inversion is an attack where an adversary queries a model repeatedly to reconstruct the training data or infer sensitive features of the training set. In this scenario, the attacker uses API access to gradually reconstruct the fine-tuned model's training data, which is the defining characteristic of model inversion. The gradual querying pattern is typical of inversion attacks that exploit the model's memorization of training examples.

Answer

Membership inference

Answer

Data poisoning

Answer

Model extraction

Question 5

A company deploys an LLM chatbot that has access to a database of customer orders. They want to prevent the LLM from revealing order details unless the user is authenticated as the owner. Which security control should be implemented?

Accepted Answer

Access controls on the model and API. Access controls on the model and API (Option D) are the correct security control because they enforce authentication and authorization at the API gateway or model endpoint level, ensuring that only the authenticated owner can query their own order details. This prevents unauthorized users from invoking the LLM to retrieve sensitive data, regardless of the prompt content. Without such access controls, the LLM would have no inherent mechanism to verify user identity before processing requests.

Answer

Output filtering

Answer

Rate limiting

Answer

Input validation and sanitization

Question 6

A machine learning engineer wants to prevent data poisoning during the training of a model. Which practice is MOST effective for ensuring the integrity of the training data?

Accepted Answer

Secure data pipelines. Secure data pipelines enforce integrity controls — provenance tracking, access control, encryption, and validation — across the entire data ingestion and preprocessing flow, which directly prevents adversaries from injecting poisoned samples. Because poisoning attacks occur before or during training, protecting the pipeline is the most effective defense. Differential privacy, red teaming, and output filtering address different threat surfaces.

Answer

Differential privacy

Answer

Red teaming the model

Answer

Output filtering

Question 7

A company is deploying a new AI system that processes personal data. To comply with privacy regulations, they want to minimize the risk of membership inference attacks. Which THREE practices should they adopt? (Select three.)

Accepted Answer

Use differential privacy during training. Option A (Use differential privacy during training) is correct because differential privacy adds calibrated noise to the training process, which bounds how much any single individual's data can influence the model, directly reducing the signal that membership inference attacks exploit. Option B (Implement access controls on the model API) is correct because membership inference typically requires repeated, query-based probing of the model's outputs; restricting who can query the API and how often limits an adversary's ability to run the statistical tests needed to infer training-set membership. Option D (Enable audit logging of all model interactions) is correct because logging queries and responses enables detection of the anomalous, high-volume probing patterns characteristic of membership inference attempts, supporting timely investigation and response. Option C (Increase model size to improve accuracy) is not correct because larger, higher-capacity models tend to overfit training data more, which increases rather than minimizes membership inference risk. Option E (Use homomorphic encryption for model inference) is not correct because homomorphic encryption protects data confidentiality during computation but does not prevent an authorized client from analyzing the returned outputs to infer membership.

Answer

Increase model size to improve accuracy

Answer

Use homomorphic encryption for model inference

Question 8

An organization is planning to fine-tune an open-source LLM for internal use. To secure the supply chain, which TWO steps should they take before using the base model? (Select two.)

Accepted Answer

Verify the model's provenance and checksums. Option B is correct because verifying the model's provenance and checksums confirms that the base model was obtained from a trusted source and has not been tampered with or substituted during download, which is a foundational supply-chain control. Option C is correct because pre-trained models can contain hidden backdoors or malicious behaviors (e.g., triggered outputs or poisoned weights), so vetting the model before fine-tuning helps detect such threats prior to integrating it into internal systems. Option A is not appropriate because retraining from scratch is prohibitively expensive and unnecessary for supply-chain security. Option D is a runtime monitoring control that occurs after deployment, not a pre-use supply-chain step. Option E is incorrect because fine-tuning on sensitive internal data increases risk and does not secure the base model's supply chain.

Answer

Retrain the model from scratch

Answer

Set up audit logging of all interactions

Answer

Fine-tune the model on sensitive internal data

Question 9

A security team is evaluating the risk of adversarial examples against their image classification model. Which characteristic best describes an adversarial example?

Accepted Answer

An input modified by small, intentional perturbations designed to cause misclassification. An adversarial example is specifically crafted by adding small, often imperceptible perturbations to a legitimate input. These perturbations are designed to exploit the model's decision boundaries, causing it to output an incorrect classification with high confidence. This is a fundamental concept in AI security, highlighting the vulnerability of deep learning models to input manipulation.

Answer

A naturally occurring image that the model misclassifies due to poor training data

Answer

An image that has been resized incorrectly and appears distorted to the model

Answer

A corrupted image with missing pixels that the model cannot process

Question 10

A company deploys an LLM-based chatbot that retrieves data from external databases. An attacker embeds malicious instructions in a database record. When the chatbot retrieves that record, it executes the instructions, overriding its system prompt. Which type of attack is this?

Accepted Answer

Indirect prompt injection. This is an indirect prompt injection attack because the malicious instructions are embedded in a third-party data source (the database record) rather than being sent directly by the user. When the LLM retrieves and processes that record, the injected instructions override the system prompt, causing the chatbot to behave contrary to its intended design.

Answer

Model inversion attack

Answer

Direct prompt injection

Answer

Membership inference attack

Question 11

A data science team wants to train a model on sensitive medical records while minimizing the risk of leaking individual patient information. They need to ensure that the model's outputs do not reveal whether a specific patient's data was used in training. Which privacy-preserving technique directly addresses this requirement?

Accepted Answer

Differential privacy. Differential privacy directly addresses the requirement by adding calibrated noise to the training process or model outputs, ensuring that the inclusion or exclusion of any single patient's data does not significantly affect the final model. This provides a formal mathematical guarantee (ε-differential privacy) that an adversary cannot infer whether a specific individual's records were used, even with auxiliary information.

Answer

Homomorphic encryption

Answer

Data anonymization

Answer

Federated learning

Question 12

An organization wants to assess the security of its custom LLM application before production release. Which practice involves simulating attacks to identify vulnerabilities?

Accepted Answer

Red teaming. Red teaming (Option D) is the correct practice for simulating attacks to identify vulnerabilities in a custom LLM application. This involves ethical hackers or security experts actively probing the system with adversarial inputs, such as prompt injection, jailbreaking, or data poisoning attempts, to uncover weaknesses before production release. It directly tests the application's resilience against real-world attack vectors, aligning with the AI Security domain's focus on proactive threat assessment.

Answer

Blue teaming

Answer

Model validation

Answer

Data sanitization

Question 13

A security analyst notices that an LLM-based code assistant sometimes generates code snippets that appear to have been copied from its training data, including comments containing internal company names. Which type of attack could this inadvertently expose?

Accepted Answer

Model inversion. The LLM inadvertently reproducing verbatim training data, including internal company names, is a classic symptom of a model inversion attack. In this context, model inversion refers to an adversary extracting sensitive training data (e.g., proprietary code or comments) from the model's parameters by crafting prompts that cause the model to regurgitate memorized examples. This exposes confidential information that was never intended to be revealed, directly violating data confidentiality.

Answer

Model denial of service

Answer

Data poisoning

Answer

Prompt injection

Question 14

A company is fine-tuning a pre-trained open-source model for a sensitive application. They want to detect if the model contains a backdoor inserted by the original developers. Which supply chain security measure is most directly applicable?

Accepted Answer

Create a software bill of materials (SBOM) for the model and its dependencies. A Software Bill of Materials (SBOM) for the model and its dependencies provides a formal, machine-readable inventory of all components, including the base model, training data sources, and third-party libraries. This allows the security team to trace the provenance of each component and identify known vulnerabilities or suspicious artifacts that could indicate a backdoor inserted by the original developers. SBOMs are a key supply chain security measure recommended by frameworks like NIST SP 800-161 and are directly applicable to detecting unauthorized modifications in pre-trained models.

Answer

Apply input validation and sanitization techniques

Answer

Use homomorphic encryption for model weights

Answer

Implement differential privacy during fine-tuning

Question 15

A developer is deploying an AI service API. To protect against data leakage through API responses, which access control principle should be applied to API keys?

Accepted Answer

Implement least-privilege API access with scoped permissions. The least-privilege principle ensures that each API key is scoped to only the specific permissions required for its intended function, such as read-only access to a single endpoint. This minimizes the blast radius in case the key is compromised, preventing unauthorized access to other services or data. In AI service deployments, scoped permissions are often enforced via OAuth 2.0 scopes or IAM roles tied to the API key.

Answer

Disable API keys and rely on IP whitelisting only

Answer

Use a single shared API key for all services

Answer

Grant all API keys full access to simplify management