AI0-001 AI Security — 30 Questions

Question 1

A SOC analyst notices an unusually high number of model queries from a single API key, with inputs containing special characters and repeated prompt modifications. Which attack is MOST likely being attempted?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Jailbreaking. The high volume of queries with special characters and repeated prompt modifications is characteristic of jailbreaking attempts, where an attacker systematically probes the model for vulnerabilities to bypass safety guardrails. Unlike prompt injection, which typically involves a single crafted input, jailbreaking often involves iterative refinement of prompts to exploit model weaknesses.

Answer

Prompt injection

Answer

Model extraction

Answer

Membership inference

Question 2

A company is deploying a pre-trained image classification model from a third-party repository. Which supply chain security practice is MOST critical before integration?

Accepted Answer

Detecting backdoored models. Detecting backdoored models is the most critical practice because pre-trained models from third-party repositories can contain hidden malicious behaviors (backdoors) that trigger on specific inputs, compromising the integrity of the entire AI system. Unlike traditional software, models are opaque and can be tampered with during training or conversion, making backdoor detection essential before any integration.

Answer

Monitoring for anomalous inputs

Answer

Generating a software bill of materials (SBOM)

Answer

Performing red teaming

Question 3

Which OWASP LLM Top 10 vulnerability involves an attacker manipulating the LLM through crafted inputs that override the system's intended instructions?

Accepted Answer

Prompt injection. Prompt injection (Option B) is the correct answer because it directly describes an attack where crafted inputs override the system's intended instructions, causing the LLM to execute unauthorized actions or reveal restricted information. This vulnerability exploits the LLM's inability to distinguish between user-supplied content and system-level directives, effectively hijacking the model's behavior.

Answer

Sensitive information disclosure

Answer

Supply chain vulnerabilities

Answer

Model denial of service

Question 4

A data scientist suspects a model extraction attack on their deployed classifier. Which TWO indicators are MOST consistent with such an attack? (Select two.)

Accepted Answer

A large number of queries from a single IP address over a short period. A model extraction attack involves an adversary sending a large volume of queries from a single IP address in a short period to systematically probe the decision boundary of the classifier. This behavior is consistent with the attacker attempting to reconstruct a surrogate model by collecting enough input-output pairs to approximate the target model's function.

Answer

Queries that include SQL injection attempts

Answer

Queries that repeatedly ask for the same prediction

Answer

Queries with special characters attempting to reveal system prompts

Question 5

An organization deploys a large language model (LLM) to summarize confidential emails. They are concerned about sensitive information being exposed through the model's responses. Which attack should they be MOST worried about?

Accepted Answer

Prompt injection. Prompt injection is the most immediate threat because it allows an attacker to override the LLM's system instructions, potentially causing it to reveal confidential email content in its responses. Unlike other attacks, prompt injection directly exploits the model's inability to distinguish between user input and trusted instructions, making it the primary vector for leaking sensitive data from summarization tasks.

Answer

Data poisoning

Answer

Membership inference

Answer

Model extraction

Question 6

A company is deploying an LLM-based system that can execute API calls on behalf of users. Which TWO measures should they implement to prevent excessive agency?

Accepted Answer

Restrict the LLM to read-only or low-risk actions. Restricting the LLM to read-only or low-risk actions directly limits the scope of actions the model can execute, preventing it from performing unauthorized or destructive operations. This is a fundamental principle of least privilege applied to LLM-based systems, ensuring that even if the model is compromised or misused, it cannot escalate its agency to high-impact actions.

Answer

Implement strict output filtering

Answer

Apply rate limiting to API calls

Answer

Use input validation to sanitize user prompts

Question 7

A security analyst is investigating a potential adversarial attack on a production image classifier. The attack involves tiny perturbations that are invisible to the human eye but cause the model to misclassify a stop sign as a speed limit sign. Which type of attack is this?

Accepted Answer

Adversarial example. This is an adversarial example attack, where imperceptible perturbations are added to the input (e.g., a stop sign) to cause the model to misclassify it (e.g., as a speed limit sign). The perturbations are crafted using gradient-based methods (like FGSM or PGD) to maximize the model's loss, exploiting its linearity in high-dimensional spaces. This differs from other attacks because it targets the inference phase, not the training data or model parameters.

Answer

Data poisoning

Answer

Model inversion

Answer

Membership inference

Question 8

A machine learning engineer wants to prevent unauthorized users from querying a deployed AI model. Which access control measure is MOST appropriate to secure the API?

Accepted Answer

API key authentication. API key authentication is the most appropriate access control measure because it requires each request to include a unique key that identifies and authorizes the caller. This directly prevents unauthorized users from querying the model by validating the key against a pre-approved list before processing the request. Unlike other options, API keys provide a dedicated authentication layer for API access.

Answer

Rate limiting

Answer

Input sanitization

Answer

IP whitelisting

Question 9

A security team is red teaming an LLM-powered application. Which activity is MOST likely to be performed during red teaming?

Accepted Answer

Attempting jailbreaks to bypass safety guardrails. Red teaming an LLM-powered application focuses on adversarial testing to uncover security vulnerabilities, not on evaluating model performance or data quality. Attempting jailbreaks directly tests whether the LLM's safety guardrails can be bypassed to produce harmful or restricted outputs, which is the core objective of red teaming in AI security.

Answer

Calculating the model's accuracy on a test set

Answer

Reviewing the model's training data for bias

Answer

Auditing the model's inference latency

Question 10

A company uses a third-party AI model for sentiment analysis. They want to create a software bill of materials (SBOM) for this AI system. What is the PRIMARY purpose of an SBOM in this context?

Accepted Answer

To list all software components and dependencies used in the AI system. The primary purpose of an SBOM for an AI system is to provide a complete inventory of all software components, libraries, and dependencies that make up the system. This is critical for vulnerability management, license compliance, and supply chain risk assessment, especially when third-party AI models are integrated. It does not track performance metrics, training details, or user instructions.

Answer

To record the model's accuracy on benchmark datasets

Answer

To document the model's training hyperparameters

Answer

To provide a user manual for the AI model

Question 11

A company is implementing a guardrail system for their LLM chatbot. Which of the following is an example of a guardrail?

Accepted Answer

Rejecting requests that ask for illegal advice. A guardrail in an LLM system is a safety constraint that filters or rejects harmful inputs and outputs. Rejecting requests for illegal advice directly enforces policy compliance and prevents the model from generating prohibited content, which is the core function of a guardrail.

Answer

Using a larger context window

Answer

Increasing the model's temperature parameter

Answer

Enabling caching for frequent queries

Question 12

A security engineer is hardening an LLM-based API against OWASP LLM Top 10 risks. Which THREE risks should the engineer prioritize for mitigation?

Accepted Answer

Insecure output handling. Insecure output handling (A) is correct because LLM outputs can contain malicious content if the model is tricked via prompt injection or other attacks. Without proper output sanitization, an attacker can execute cross-site scripting (XSS) or server-side request forgery (SSRF) through the LLM's response. Training data poisoning (B) is correct because an attacker can inject malicious data into the training set, causing the model to produce biased, harmful, or backdoored outputs. Prompt injection (C) is correct because it directly exploits the LLM's input processing to bypass intended instructions, leading to unauthorized actions or data leakage.

Answer

Insecure deserialization

Answer

Model quantization errors

Question 13

An AI security analyst is evaluating a model that classifies images. The team wants to test whether small, imperceptible changes to input images can cause misclassification. Which type of attack are they testing?

Accepted Answer

Adversarial examples. Adversarial examples are specifically crafted inputs with small, imperceptible perturbations designed to cause a machine learning model to misclassify them. This directly matches the scenario of testing whether tiny changes to images can fool the classifier, which is a core concept in AI security for evaluating model robustness.

Answer

Data poisoning

Answer

Model inversion

Answer

Membership inference

Question 14

A company is deploying a new AI system that processes personal data. To comply with privacy regulations, they want to minimize the risk of membership inference attacks. Which THREE practices should they adopt? (Select three.)

Accepted Answer

Use differential privacy during training. Differential privacy (A) is correct because it adds calibrated noise to the training process or outputs, making it statistically difficult for an attacker to determine whether a specific individual's data was included in the training set. This directly mitigates membership inference attacks by bounding the influence of any single data point. Access controls (B) limit who can query the model, reducing the number of attempts an attacker can make to probe for membership. Audit logging (D) provides a record of all queries and responses, enabling detection of suspicious patterns that might indicate a membership inference attempt.

Answer

Increase model size to improve accuracy

Answer

Use homomorphic encryption for model inference

Question 15

An organization is planning to fine-tune an open-source LLM for internal use. To secure the supply chain, which TWO steps should they take before using the base model? (Select two.)

Accepted Answer

Verify the model's provenance and checksums. Verifying the model's provenance and checksums (Option B) ensures the base model has not been tampered with during download or transfer, confirming its integrity and authenticity. This is a critical supply chain security step before any fine-tuning, as it prevents the introduction of malicious modifications from untrusted sources.

Answer

Retrain the model from scratch

Answer

Set up audit logging of all interactions

Answer

Fine-tune the model on sensitive internal data

Question 16

A security team is evaluating the risk of adversarial examples against their image classification model. Which characteristic best describes an adversarial example?

Accepted Answer

An input modified by small, intentional perturbations designed to cause misclassification. An adversarial example is specifically crafted by adding small, often imperceptible perturbations to a legitimate input. These perturbations are designed to exploit the model's decision boundaries, causing it to output an incorrect classification with high confidence. This is a fundamental concept in AI security, highlighting the vulnerability of deep learning models to input manipulation.

Answer

A naturally occurring image that the model misclassifies due to poor training data

Answer

An image that has been resized incorrectly and appears distorted to the model

Answer

A corrupted image with missing pixels that the model cannot process

Question 17

A data science team wants to train a model on sensitive medical records while minimizing the risk of leaking individual patient information. They need to ensure that the model's outputs do not reveal whether a specific patient's data was used in training. Which privacy-preserving technique directly addresses this requirement?

Accepted Answer

Differential privacy. Differential privacy directly addresses the requirement by adding calibrated noise to the training process or model outputs, ensuring that the inclusion or exclusion of any single patient's data does not significantly affect the final model. This provides a formal mathematical guarantee (ε-differential privacy) that an adversary cannot infer whether a specific individual's records were used, even with auxiliary information.

Answer

Homomorphic encryption

Answer

Data anonymization

Answer

Federated learning

Question 18

An organization wants to assess the security of its custom LLM application before production release. Which practice involves simulating attacks to identify vulnerabilities?

Accepted Answer

Red teaming. Red teaming (Option D) is the correct practice for simulating attacks to identify vulnerabilities in a custom LLM application. This involves ethical hackers or security experts actively probing the system with adversarial inputs, such as prompt injection, jailbreaking, or data poisoning attempts, to uncover weaknesses before production release. It directly tests the application's resilience against real-world attack vectors, aligning with the AI Security domain's focus on proactive threat assessment.

Answer

Blue teaming

Answer

Model validation

Answer

Data sanitization

Question 19

A security analyst notices that an LLM-based code assistant sometimes generates code snippets that appear to have been copied from its training data, including comments containing internal company names. Which type of attack could this inadvertently expose?

Accepted Answer

Model inversion. The LLM inadvertently reproducing verbatim training data, including internal company names, is a classic symptom of a model inversion attack. In this context, model inversion refers to an adversary extracting sensitive training data (e.g., proprietary code or comments) from the model's parameters by crafting prompts that cause the model to regurgitate memorized examples. This exposes confidential information that was never intended to be revealed, directly violating data confidentiality.

Answer

Model denial of service

Answer

Data poisoning

Answer

Prompt injection

Question 20

A company is fine-tuning a pre-trained open-source model for a sensitive application. They want to detect if the model contains a backdoor inserted by the original developers. Which supply chain security measure is most directly applicable?

Accepted Answer

Create a software bill of materials (SBOM) for the model and its dependencies. A Software Bill of Materials (SBOM) for the model and its dependencies provides a formal, machine-readable inventory of all components, including the base model, training data sources, and third-party libraries. This allows the security team to trace the provenance of each component and identify known vulnerabilities or suspicious artifacts that could indicate a backdoor inserted by the original developers. SBOMs are a key supply chain security measure recommended by frameworks like NIST SP 800-161 and are directly applicable to detecting unauthorized modifications in pre-trained models.

Answer

Apply input validation and sanitization techniques

Answer

Use homomorphic encryption for model weights

Answer

Implement differential privacy during fine-tuning