AI Security practice questions

Tampering with training data

Why it fails: Data poisoning is a tampering threat during preprocessing.

Spoofing of data sources

Why it fails: Spoofing is a valid threat at data collection if sources are impersonated.

Information disclosure via data leakage

Why it fails: Sensitive data could be leaked during collection or preprocessing.

Red teaming the model

Why it fails: Red teaming identifies vulnerabilities but does not provide runtime protection against over-reliance.

Rate limiting and input validation

Why it fails: These address abuse and injection, not over-reliance.

Differential privacy

Why it fails: Differential privacy protects training data, not over-reliance.

Data poisoning

Why it fails: Data poisons the training data, but the immediate concern for deployed LLM is prompt injection leaking secrets.

Membership inference

Why it fails: Membership inference determines if a record was in training data, but does not directly leak email contents.

Model extraction

Why it fails: Model extraction aims to replicate the model, not directly leak confidential emails.

Prompt injection

Why it fails: Prompt injection is specific to LLMs, not fraud detection models.

Adversarial examples

Why it fails: Adversarial examples affect individual predictions, not a systematic degradation across many samples after retraining.

Model inversion

Why it fails: Model inversion reconstructs training data, not degrade performance.

Spoofing of data sources

Why it fails: Spoofing is about authentication, not directly about training data integrity.

Repudiation of data provenance

Why it fails: Repudiation is about non-repudiation logging, not a primary data threat.

Reviewing the model's software bill of materials (SBOM)

Why it fails: SBOM lists components but does not detect backdoors inserted into model weights.

Ensuring the model is hosted on a private cloud

Why it fails: Hosting location does not guarantee the model hasn't been tampered with.

Checking for proper API key management

Why it fails: API key management is important for deployment security but doesn't address supply chain integrity of the model itself.

Rate limiting API requests

Why it fails: Rate limiting prevents abuse but does not stop a single crafted injection attack.

Output filtering

Why it fails: Output filtering catches harmful responses but doesn't prevent the injection from being processed.

Using a larger model

Why it fails: Model size does not inherently prevent prompt injection.

Rate limiting

Why it fails: Rate limiting controls request frequency but does not prevent insecure suggestions.

Secure data pipelines

Why it fails: Secure data pipelines protect training data but do not directly address output safety.

Output filtering of insecure patterns

Why it fails: Output filtering can block known insecure patterns but may miss novel ones; red teaming is more comprehensive.

Model denial of service

Why it fails: Model denial of service involves overwhelming the model with requests, not information disclosure.

Insecure output handling

Why it fails: Insecure output handling refers to the application failing to filter or validate LLM outputs, not the LLM leaking information.

Prompt injection

Why it fails: Prompt injection is the attack vector, but the OWASP category for leaking system prompts is sensitive information disclosure.

Robust training techniques

Why it fails: Robust training makes the model less vulnerable to some attacks, but not to inference-time prompt injection.

Access controls on the database

Why it fails: Access controls limit what the SQL execution can do, but do not prevent injection itself.

Rate limiting

Why it fails: Rate limiting reduces abuse but does not prevent injection attacks.

Sensitive information disclosure

Why it fails: Sensitive information disclosure is about leaking data, not generating insecure code.

Model denial of service

Why it fails: Model denial of service is about resource exhaustion, not output security.

Prompt injection

Why it fails: Prompt injection focuses on input manipulation, not output validation.

Input sanitization only

Why it fails: Sanitising input strips obvious injection payloads but cannot stop novel phrasing, and it does nothing to ground answers in verified knowledge, so outdated responses persist. It is tempting because sanitisation is a genuine, low-cost first layer against injection, yet retrieval-augmented generation is what supplies current, authoritative content.

Rate limiting

Why it fails: Rate limiting throttles request volume, not content; it neither neutralises injected instructions nor connects the model to verified knowledge, so stale answers continue. It is tempting because rate limiting genuinely mitigates abuse and denial-of-service against an API, but prompt injection and outdated data are unaffected.

Robust training on adversarial examples

Why it fails: Adversarial training hardens the model against known attack patterns but cannot guarantee grounding in current verified sources, and new injections still bypass it. It is tempting because adversarial examples genuinely improve robustness, yet retrieval-augmented generation is what anchors responses to authoritative, up-to-date knowledge.

Increasing model temperature

Why it fails: Temperature controls sampling randomness, so raising it increases output variability without blocking injected instructions. It tempts because higher values are used for creative generation tasks, and would be correct when diverse responses are wanted, but prompt injection defence requires input filtering and output validation instead.

Disabling system prompts

Why it fails: Disabling system prompts removes the instruction layer that constrains model behaviour, so injected text in user or retrieved content faces no competing directive — it worsens injection risk rather than mitigating it. It is tempting because system prompts can themselves be leaked or overridden, and omitting them is defensible when a model needs no behavioural guardrails.

Data poisoning attack

Why it fails: Data poisoning involves corrupting training data to manipulate model behavior, not extracting system prompts from a deployed model.

Membership inference attack

Why it fails: Membership inference determines if a specific record was used in training, not extracting prompts from the model's response.

Model inversion attack

Why it fails: Model inversion aims to reconstruct private training data from model outputs, not to steal system prompts.

Model inversion to reconstruct training data

Why it fails: Model inversion is a privacy attack, not a jailbreak technique for bypassing safety guardrails.

Payload splitting across multiple user messages

Why it fails: Payload splitting is more of a prompt injection technique than a jailbreak; jailbreaks typically aim to override safety rules, not split payloads.

Few-shot prompting with benign examples

Why it fails: Few-shot prompting with benign examples is a legitimate technique to guide model behavior, not a jailbreak technique.

Defensive distillation to smooth model outputs

Why it fails: Defensive distillation was initially proposed to reduce adversarial transferability, but it has been shown to be ineffective against many attacks and can even increase vulnerability. It does not provide a principled defense and is not recommended as a primary mitigation. The technique often fails under adaptive attacks where the adversary knows the defense.

Gradient masking by obfuscating model gradients

Why it fails: Gradient masking attempts to hide gradient information from attackers, but it often provides a false sense of security. Attackers can use transferability or black-box methods to bypass it. Moreover, it can make the model harder to train and debug. It is not considered a robust defense because it does not eliminate the underlying vulnerability; it only obscures it temporarily.

Input sanitization by removing outliers

Why it fails: Removing outliers may help with data quality but is not a reliable defense against adversarial evasion. Adversarial perturbations are often small and crafted to be within normal input ranges, so they may not appear as outliers. This approach can also degrade model performance by discarding legitimate edge cases, and it does not address the core vulnerability to crafted inputs.

Require the provider to publish a model card describing intended use and limitations.

Why it fails: A model card documents intended use, training data characteristics, and known limitations, which supports informed selection. It is a transparency artifact, not a cryptographic proof of integrity, so it cannot confirm that the deployed weights match the evaluated version or that the artifact has not been altered after publication.

Enforce a strict acceptable use policy for employees interacting with the assistant.

Why it fails: An acceptable use policy governs user behavior and reduces misuse risk, but it says nothing about whether the provider's model artifact is authentic. The concern here is supply-chain integrity of the model itself, which requires technical verification mechanisms rather than procedural rules aimed at end users.

Run a red-team exercise against the integrated assistant before production release.

Why it fails: Red teaming uncovers exploitable behaviors such as prompt injection or unsafe outputs, which is valuable for risk assessment. However, it does not attest to artifact integrity or provenance; a tampered model could pass or fail adversarial testing without revealing that its weights differ from the evaluated version.

Perform gradient clipping to limit weight updates

Why it fails: Gradient clipping prevents exploding gradients and can improve training stability, but it does not specifically address poisoning attacks. Poisoned samples can still influence the model within the clipped gradient bounds. While it may limit the impact of individual samples, it does not detect or remove malicious data, so it is not a targeted mitigation.

Increase the learning rate to quickly adapt to new data

Why it fails: Increasing the learning rate would make the model more sensitive to recent samples, including poisoned ones, potentially amplifying the attack. It does not help detect or mitigate poisoning; instead, it could worsen the model's performance and security. A higher learning rate is generally not recommended when data integrity is in question.

Apply k-nearest neighbors outlier detection on training data

Why it fails: k-nearest neighbors outlier detection can identify anomalous data points, but it may not effectively detect poisoning samples that are deliberately crafted to appear normal. Attackers can design poisoned samples to be close to benign instances, evading such detection. This method is not robust against sophisticated poisoning attacks that aim to blend in.

Model inversion

Why it fails: Model inversion reconstructs training data from a model's outputs, not perturbing inputs. It is tempting because it also targets model confidentiality, and would be correct when the goal is recovering representative features or sensitive training samples from prediction confidence scores.

Membership inference

Why it fails: Membership inference determines whether a specific record was in the training set; it does not alter images to force misclassification. It is tempting because both are privacy and adversarial concerns for deployed models, and would be correct when assessing training-data leakage.

Data poisoning

Why it fails: Data poisoning corrupts the training dataset so the model learns wrong patterns; it does not perturb inference-time inputs. It is tempting because both attacks degrade classifier accuracy, and would be correct when an adversary controls or contaminates the data used to train or fine-tune the model.

Differential privacy during training

Why it fails: Differential privacy adds noise during model training to protect individual records in the training set; it does nothing about malicious instructions embedded in runtime prompts. It is tempting because it is a genuine privacy defence, and would be correct if the requirement were preventing training-data memorisation rather than blocking injection.

Rate limiting API calls

Why it fails: Rate limiting caps request volume; it does nothing to stop injected instructions inside user or retrieved content. It is tempting because rate limiting is a standard API abuse control, and would be correct when defending against brute-force or denial-of-service traffic rather than adversarial prompt content.

Output filtering of model responses

Why it fails: Filtering responses catches leaked output but cannot stop injected instructions from altering the model's behaviour or triggering tool calls, since the malicious input is processed before any output exists. It is tempting because output filtering genuinely mitigates harmful or sensitive content generation, which is a different problem from instruction hijacking.