AI Security practice questions

Prompt injection

Why wrong: Prompt injection could be one goal, but the pattern of repeated modifications suggests jailbreaking.

Model extraction

Why wrong: Model extraction typically involves many diverse inputs to replicate model behavior, not special characters.

Membership inference

Why wrong: Membership inference requires many queries but not special characters.

Monitoring for anomalous inputs

Why wrong: Monitoring is a runtime defense, not a pre-deployment check.

Generating a software bill of materials (SBOM)

Why wrong: SBOM lists components but does not verify model integrity.

Performing red teaming

Why wrong: Red teaming tests behavior but may not catch subtle backdoors.

Sensitive information disclosure

Why wrong: This involves leaking data, not overriding instructions.

Supply chain vulnerabilities

Why wrong: Supply chain issues relate to third-party components.

Model denial of service

Why wrong: DoS floods the model, not manipulates instructions.

Queries that include SQL injection attempts

Why wrong: SQL injection targets databases, not model extraction.

Queries that repeatedly ask for the same prediction

Why wrong: Repeated identical queries are more indicative of a denial-of-service or probing, not extraction.

Queries with special characters attempting to reveal system prompts

Why wrong: This suggests prompt injection or jailbreaking, not extraction.

Data poisoning

Why wrong: Data poisons the training data, but the immediate concern for deployed LLM is prompt injection leaking secrets.

Membership inference

Why wrong: Membership inference determines if a record was in training data, but does not directly leak email contents.

Model extraction

Why wrong: Model extraction aims to replicate the model, not directly leak confidential emails.

Implement strict output filtering

Why wrong: Output filtering blocks harmful content but does not limit the LLM's ability to take actions.

Apply rate limiting to API calls

Why wrong: Rate limiting controls frequency, not the scope of actions.

Use input validation to sanitize user prompts

Why wrong: Input validation prevents injection but does not control the LLM's actions.

Data poisoning

Why wrong: Data poisoning corrupts training data, not inference-time inputs.

Model inversion

Why wrong: Model inversion reconstructs training data from model outputs.

Membership inference

Why wrong: Membership inference determines if a record was in the training set.

Rate limiting

Why wrong: Rate limiting controls usage volume but does not authenticate users.

Input sanitization

Why wrong: Input sanitization prevents attacks but does not enforce access control.

IP whitelisting

Why wrong: IP whitelisting can be bypassed with VPNs and doesn't provide per-user granularity.

Calculating the model's accuracy on a test set

Why wrong: Accuracy evaluation is part of model validation, not red teaming.

Reviewing the model's training data for bias

Why wrong: Bias review is important but is not typically the focus of red teaming; red teaming simulates adversarial attacks.

Auditing the model's inference latency

Why wrong: Latency auditing is a performance test, not a security red team activity.

To record the model's accuracy on benchmark datasets

Why wrong: Benchmark accuracy is not part of an SBOM.

To document the model's training hyperparameters

Why wrong: Hyperparameters are not typically part of an SBOM; they are model-specific metadata.

To provide a user manual for the AI model

Why wrong: A user manual is different from an SBOM.

Using a larger context window

Why wrong: Context window size is a technical parameter, not a guardrail.

Increasing the model's temperature parameter

Why wrong: Temperature controls randomness, not safety guardrails.

Enabling caching for frequent queries

Why wrong: Caching improves performance but does not enforce safety.

Insecure deserialization

Why wrong: Insecure deserialization is a general web security risk but not specific to the OWASP LLM Top 10.

Model quantization errors

Why wrong: Model quantization errors are not part of the OWASP LLM Top 10.

Data poisoning

Why wrong: Data poisoning corrupts training data, not test-time inputs.

Model inversion

Why wrong: Model inversion reconstructs training data from model outputs.

Membership inference

Why wrong: Membership inference determines if a data point was in the training set.

Increase model size to improve accuracy

Why wrong: Larger models may memorize more, increasing membership inference risk.

Use homomorphic encryption for model inference

Why wrong: Homomorphic encryption allows computation on encrypted data but does not inherently prevent membership inference.

Retrain the model from scratch

Why wrong: Retraining from scratch is costly and unnecessary if the pre-trained model is properly vetted.

Set up audit logging of all interactions

Why wrong: Audit logging is important but is an operational control after deployment, not a supply chain security step before using the model.

Fine-tune the model on sensitive internal data

Why wrong: Fine-tuning should only be done after supply chain checks, and may introduce risks if the base model is compromised.

A naturally occurring image that the model misclassifies due to poor training data

Why wrong: This describes a model weakness but not an adversarial example, which requires deliberate perturbation.

An image that has been resized incorrectly and appears distorted to the model

Why wrong: Resize errors are preprocessing issues, not adversarial attacks.

A corrupted image with missing pixels that the model cannot process

Why wrong: Missing pixels cause input corruption, not adversarial manipulation.

Homomorphic encryption

Why wrong: Homomorphic encryption allows computation on encrypted data but does not address membership inference after model release.

Data anonymization

Why wrong: Anonymization of the dataset helps but may be reversed; differential privacy provides a mathematical guarantee.

Federated learning

Why wrong: Federated learning keeps data on local devices but does not prevent membership inference from the model itself.

Blue teaming

Why wrong: Blue teaming focuses on defense, not simulated attacks.

Model validation

Why wrong: Model validation checks performance, not security vulnerabilities.

Data sanitization

Why wrong: Data sanitization cleans inputs but is not an assessment practice.

Model denial of service

Why wrong: Model denial of service overwhelms the model, not extract data.

Data poisoning

Why wrong: Data poising corrupts training data to manipulate behavior, not to extract data.

Prompt injection

Why wrong: Prompt injection manipulates the model's instructions, not extracts training data.

Apply input validation and sanitization techniques

Why wrong: Input validation defends against injection, not supply chain backdoors.

Use homomorphic encryption for model weights

Why wrong: Homomorphic encryption protects data in use but does not detect backdoors.

Implement differential privacy during fine-tuning

Why wrong: Differential privacy protects training data privacy, not detect backdoors.