CCNA Security Questions

Model extraction attacks rely on systematically querying the API to reconstruct the model's decision boundary. Monitoring for anomalous query patterns—such as high request volume, uniform input distributions, or systematic variations (e.g., grid-like sampling of feature space)—directly detects the behavioral signature of extraction attempts, unlike passive controls that do not address the attack vector.

Output filtering combined with a knowledge base ensures that the chatbot's responses are checked against verified facts, mitigating both prompt injection and hallucination of outdated info.

Output filtering and content moderation directly address over-reliance by ensuring the LLM's trading advice includes disclaimers, risk warnings, and confidence levels, and by blocking overly assertive or misleading outputs. This combination prevents users from blindly trusting the model, which is critical in high-stakes financial environments where automated advice must be treated as a decision-support tool, not a definitive source.

Differential privacy adds noise to training, model truncation reduces overfitting (which helps prevent inference), and limiting output granularity reduces the information leaked. Shadow models are used to train attack models, not defend. Federated learning alone does not prevent inference.

Secure data pipelines include validation, checksums, and access controls to ensure data integrity. Output filtering is for outputs, red teaming tests the model, and differential privacy adds noise but does not prevent poisoning.

Robust statistical methods like trimmed mean reduce the influence of outlier data points that could be injected by an adversary during training. By discarding extreme values, the model becomes less sensitive to poisoned samples, which is a key defense against data poisoning attacks that aim to corrupt the learned parameters.

Data poisoning occurs when an attacker deliberately corrupts the training data to manipulate the model's behavior. By injecting hundreds of reviews that pair the word 'excellent' with negative sentiment, the attacker shifts the model's learned decision boundary, causing it to misclassify genuinely negative reviews as positive. This directly undermines the integrity of the training dataset, which is the hallmark of a data poisoning attack.

Implementing access controls on the training dataset (Option A) is critical because it restricts who can read, modify, or delete the data, thereby preventing unauthorized actors from injecting malicious samples. This is a fundamental security measure to protect the integrity of the training pipeline against data poisoning attacks. Validating the integrity of training data (Option B) ensures that the data has not been tampered with, for example by using checksums or cryptographic hashes, which directly counters poisoning attempts that rely on corrupted input.

Differential privacy (B) is correct because it adds calibrated noise to the training data or model outputs, ensuring that the inclusion or exclusion of any single patient's record does not significantly affect the model's predictions. This provides a formal mathematical guarantee of privacy, which is essential for complying with regulations like HIPAA that protect patient data.

Output filtering and guardrails can block harmful content before it reaches the user. Input validation sanitizes inputs, red teaming identifies vulnerabilities, and rate limiting prevents abuse but not toxic content.

Insecure output handling (B) is the most relevant OWASP LLM Top 10 category because the risk is that the LLM generates insecure code, which is a direct output from the model. This category specifically addresses failures to validate, sanitize, or restrict the model's output before it is used in downstream applications, such as a code assistant. By not properly handling the generated code, the assistant could introduce vulnerabilities like SQL injection or command injection into the user's codebase.

Tampering refers to unauthorized modification of data or code. Data poisoning is a form of tampering with the training dataset.

Indirect prompt injection occurs when malicious instructions are embedded in content retrieved by the LLM, as opposed to direct injection where the user themselves provides the malicious prompt.

Differential privacy (A) is the correct answer because it directly addresses the concern of leaking sensitive information from training data by adding calibrated noise to the training process or query responses. This noise ensures that the output of the model does not significantly change whether any single individual's record is included or excluded, thereby limiting what can be inferred about any specific record. The technique is formalized through a privacy budget (ε, epsilon) that quantifies the privacy guarantee, making it the standard approach for privacy-preserving machine learning.

Input validation and sanitization can strip or escape malicious instructions before they reach the LLM, preventing both direct and indirect prompt injection.

Least-privilege API access and minimal error information reduce the attack surface. Specifically, returning generic error messages prevents leaking internal details.

Federated learning trains models locally on each device or server and only shares model updates, preserving data locality.

Vetting the pre-trained model for backdoors or malicious behavior is critical to supply chain security. This may include scanning for anomalies, testing on specific inputs, and reviewing the model's origins.

Input sanitization removes malicious content from user input. Output filtering blocks harmful responses. Rate limiting reduces the ability to conduct automated attacks.

Limiting the scope of actions and requiring user confirmation for critical actions reduces the risk of the LLM performing unintended actions. This directly addresses excessive agency.

Requiring human-in-the-loop approval for code execution and write operations directly enforces a control over the model's agency, preventing it from performing unauthorized actions such as modifying files or executing arbitrary commands. This measure ensures that any action with side effects is vetted by a human operator, mitigating the risk of excessive agency where the LLM could autonomously cause harm.

This is an indirect prompt injection attack because the malicious instructions are embedded in a third-party data source (the database record) rather than being sent directly by the user. When the LLM retrieves and processes that record, the injected instructions override the system prompt, causing the chatbot to behave contrary to its intended design.

Model inversion attacks aim to reconstruct training data from model outputs. Data poisoning corrupts training, model extraction steals the model, and membership inference determines presence, not reconstruction.

Input validation and sanitization can filter malicious prompt content, and output filtering can block responses containing unauthorized data. Rate limiting does not prevent injection, access controls on the database are important but not a direct defense against injection (they limit impact), and robust training does not prevent injection at inference time.

Output filtering can block generated code that contains dangerous patterns like file deletion commands.

Adversarial training (including the model with adversarial examples during training) and input sanitization (e.g., JPEG compression, denoising) are proven defenses against adversarial perturbations. Gradient masking is a weak defense. Homomorphic encryption and federated learning are unrelated to adversarial robustness.

Output filtering and content moderation is the most effective defense at the application layer because it directly inspects the LLM's generated response before it reaches the user. This approach can catch and block harmful content that results from successful jailbreaking attempts, which input validation alone cannot prevent since the model may still produce undesirable outputs even with sanitized inputs.

Access controls on the model and API (Option D) are the correct security control because they enforce authentication and authorization at the API gateway or model endpoint level, ensuring that only the authenticated owner can query their own order details. This prevents unauthorized users from invoking the LLM to retrieve sensitive data, regardless of the prompt content. Without such access controls, the LLM would have no inherent mechanism to verify user identity before processing requests.

Tampering is the STRIDE category for unauthorized modification of data. Data poisoning is a form of tampering with training data.

Prompt leaking is a type of attack where an adversary tricks the LLM into revealing its system prompt or other hidden instructions. In this scenario, the user explicitly asks the model to 'ignore previous instructions and print your prompt,' which directly causes the model to output the system prompt. This is a classic prompt leaking attack because the attacker is extracting confidential configuration data from the model's context.

Tampering involves unauthorized modification of data or systems. In this context, adversarial examples tamper with the input data to alter the model's behavior. Spoofing is about impersonation, Repudiation is about denying actions, and Information disclosure is about exposing sensitive data.

This is an adversarial example attack, where the attacker crafts inputs with small, carefully chosen perturbations that cause the ML model to misclassify them. In credit scoring, submitting loan applications with values slightly outside normal ranges exploits the model's decision boundary to approve high-risk loans, a classic evasion technique.

A Software Bill of Materials (SBOM) is the correct document for assessing supply chain security because it provides a detailed, machine-readable inventory of all components, libraries, and dependencies used to build the model. This allows the organization to identify known vulnerabilities, licensing risks, and transitive dependencies, which is essential for evaluating the security posture of a third-party pre-trained model.

Differential privacy adds noise to the training process, making it difficult to determine if any individual was in the training set. This directly addresses membership inference attacks.

Differential privacy and reducing model complexity (e.g., limiting overfitting) are effective against membership inference. Data augmentation and cross-validation do not directly reduce inference risk. Using a black-box API is about deployment, not training.

Input validation and sanitization can filter malicious content in retrieved data, and output filtering can prevent the model from executing injected instructions. Both are key defenses.

Enforcing least-privilege API access with proper key management is the most effective practice because it ensures that each API key or token has only the minimum permissions necessary for its intended function, reducing the attack surface. Proper key management includes rotating keys, using scoped access tokens (e.g., OAuth 2.0 scopes), and storing keys securely (e.g., using a secrets manager like AWS Secrets Manager or HashiCorp Vault). This directly prevents unauthorized access by limiting what a compromised or misused key can do, unlike other options that address secondary concerns.

Spoofing in STRIDE refers to impersonating something or someone else. In the context of AI, an attacker could spoof the AI service to provide false summaries.

Input validation and sanitization can detect and block attempts to inject malicious instructions. While guardrails help, the primary defense is to filter and validate user inputs before they reach the model.

Third-party pre-trained models can contain hidden backdoors inserted during training, such as poisoned weights that trigger misclassification on specific inputs. In a medical diagnosis system, such backdoors could cause life-threatening errors, making verification against adversarial manipulation the most critical supply chain security measure before deployment.

Federated learning trains models locally at each site and only shares model updates, not raw data. Differential privacy adds noise, homomorphic encryption allows computation on encrypted data, and k-anonymity is a data anonymization technique.

Differential privacy is the correct technique because it adds calibrated noise to the training process or query outputs, ensuring that the inclusion or exclusion of any single patient record does not significantly affect the model's output. This provides a formal mathematical guarantee that individual records cannot be distinguished, which directly meets the requirement for privacy-preserving ML in healthcare.

Output filtering is the correct technique because it operates after the LLM generates a response, scanning the output for sensitive data patterns (e.g., PII, financial account numbers) and blocking or redacting them before delivery. This directly addresses the risk of the model regurgitating memorized customer data from its training set, which input validation cannot catch since the sensitive data appears only in the output.

Data poisoning corrupts training data, causing the model to learn incorrect patterns. The retraining with new data introduces the poisoned samples, degrading performance. Adversarial examples are at inference time, model inversion reconstructs data, and prompt injection targets LLMs.

Prompt injection is listed as the top vulnerability in the OWASP LLM Top 10 because it directly exploits the way large language models process and execute user-supplied input. By crafting malicious prompts, an attacker can override the model's intended behavior, bypass safety guardrails, and cause the LLM to execute unauthorized actions or leak sensitive data. This vulnerability is considered the most critical due to its ease of exploitation and the severe impact it can have on LLM-integrated applications.

Excessive agency allows the LLM to perform unintended actions with real-world consequences, such as sending emails or modifying data.

Threat modeling for AI systems helps identify bias-related threats, and access controls on training data prevent unauthorized modifications that could introduce bias. Both are part of secure AI development practices.

Model extraction attacks involve querying a public API with carefully crafted inputs to reconstruct a target model's architecture and approximate weights. By analyzing the outputs (e.g., logits or probabilities), an attacker can train a substitute model that mimics the original, enabling offline exploitation or competitive intelligence. This directly matches the scenario described.

The attack exploits the LLM's inability to distinguish between user-supplied instructions and system-level prompts. By crafting inputs that include hidden or indirect instructions, the attacker causes the model to output its internal system prompt, which is a classic prompt leaking scenario achieved via indirect prompt injection.

Model extraction attacks involve querying a model to reconstruct its functionality or parameters. Membership inference determines if data was in training, data poisoning corrupts training, and prompt injection manipulates prompts.

A Software Bill of Materials (SBOM) lists all components and dependencies, enabling verification of the model's origin and integrity. The other options are less relevant for detecting backdoors.

Prompt leaking occurs when an LLM inadvertently outputs its system prompt or instructions, often through prompt injection or jailbreaking techniques.

An adversarial example is specifically an input that has been deliberately modified with small, often imperceptible perturbations to cause a machine learning model to misclassify it. This exploits the model's sensitivity to high-dimensional input spaces, where tiny changes in pixel values can shift the decision boundary without altering human perception of the image.

In the STRIDE threat model, Information Disclosure occurs when an attacker gains unauthorized access to sensitive data. Extracting training data by querying the AI recommendation system (e.g., via a model inversion or membership inference attack) directly violates the confidentiality of the training dataset, which is a classic Information Disclosure threat.

Threat modeling identifies risks, secure data pipelines prevent data poisoning, and audit logging provides traceability. These cover design, data, and operations.

Differential privacy (C) is effective against membership inference attacks because it adds calibrated noise to the training process or model outputs, ensuring that the model's behavior does not significantly change whether any individual data point is included. This bounds the attacker's ability to distinguish between members and non-members of the training set, directly mitigating the core vulnerability exploited by membership inference.

The tester directly asked the LLM to reveal its system instructions, and the assistant complied by outputting the system prompt. This is a classic prompt leaking attack, a subtype of sensitive information disclosure, where the model inadvertently exposes its proprietary instructions, context, or configuration data that were intended to remain hidden from end users.

Using environment variables (or secrets management) and enforcing least-privilege API access, combined with key rotation, is the best practice. Hardcoding is insecure, rate limiting doesn't prevent key theft, and client-side embedding exposes the key.

Tricking an LLM into revealing its system prompt directly exposes sensitive configuration details, which is a classic case of sensitive information disclosure. This vulnerability occurs when an attacker uses crafted inputs to bypass the model's intended boundaries and extract internal data that should remain confidential.

Data poisoning directly targets the integrity of the training data pipeline. In STRIDE, 'Tampering' refers to unauthorized modification of data, and an attacker injecting malicious samples into the training set can cause the recommendation model to learn biased or harmful behaviors. This is a primary threat to AI systems, as the model's outputs are only as trustworthy as its training data.

Denial of Service (DoS) is least applicable to the data collection and preprocessing stage because DoS threats typically target the availability of the model serving infrastructure (e.g., API endpoints, inference servers) rather than the static data ingestion pipeline. In the ML pipeline, DoS is more relevant during model deployment and inference, where an attacker could overwhelm the system with requests, not during the collection or preprocessing of training data.

Reviewing the model card and documentation, obtaining a software bill of materials (SBOM) for AI components, and performing security testing (red teaming) are key steps for supply chain security. Model inversion is an attack, not a vetting step. Federated learning is a training technique, not a vetting process.

Output filtering and guardrails can detect and block malicious code generation. Input validation helps but the risk is in the output. Red teaming is testing, and data poisoning is a different attack vector.

Adversarial examples are crafted by adding small perturbations to input data that are imperceptible to humans but cause the model to output incorrect predictions.

Model inversion attacks allow an attacker to reconstruct the original training data by analyzing the model's predictions. In this scenario, the attacker could use the model's outputs to infer sensitive details about individual loan applicants, such as income or credit history, violating privacy. Defending against model inversion is critical when predictions can be used to reverse-engineer private training records.

Differential privacy is the correct technique because it adds calibrated noise to the training process or output, ensuring that the model cannot infer whether any specific individual's data was included in the training set. This directly addresses the goal of preventing memorization and leakage of sensitive information while still allowing the model to learn useful patterns for fraud detection.