AI Security practice questions

Model inversion

Why wrong: Model inversion reconstructs training data from model outputs.

Membership inference

Why wrong: Membership inference determines if a record was in the training set.

Data poisoning

Why wrong: Data poisons the training set, not the inference input.

Differential privacy during training

Why wrong: Differential privacy protects training data privacy, not inference-time injection.

Rate limiting API calls

Why wrong: Rate limiting prevents abuse but does not filter malicious prompts.

Output filtering of model responses

Why wrong: Output filtering catches harmful responses but does not prevent injection from occurring.

Tampering with training data

Why wrong: Data poisoning is a tampering threat during preprocessing.

Spoofing of data sources

Why wrong: Spoofing is a valid threat at data collection if sources are impersonated.

Information disclosure via data leakage

Why wrong: Sensitive data could be leaked during collection or preprocessing.

Homomorphic encryption

Why wrong: Homomorphic encryption allows computation on encrypted data but is not specifically a decentralized training approach.

Secure multi-party computation

Why wrong: Secure multi-party computation enables joint computation without revealing inputs but is not the standard decentralized training method.

Differential privacy

Why wrong: Differential privacy adds noise to protect individual records but does not keep data decentralized.

Prompt injection

Why wrong: Prompt injection could be one goal, but the pattern of repeated modifications suggests jailbreaking.

Model extraction

Why wrong: Model extraction typically involves many diverse inputs to replicate model behavior, not special characters.

Membership inference

Why wrong: Membership inference requires many queries but not special characters.

Monitoring for anomalous inputs

Why wrong: Monitoring is a runtime defense, not a pre-deployment check.

Generating a software bill of materials (SBOM)

Why wrong: SBOM lists components but does not verify model integrity.

Performing red teaming

Why wrong: Red teaming tests behavior but may not catch subtle backdoors.

Insecure output handling

Why wrong: Insecure output handling is about validating model outputs, not leaking prompts.

Model inversion

Why wrong: Model inversion reconstructs training data, not prompts.

Excessive agency

Why wrong: Excessive agency refers to the model performing unintended actions.

Sensitive information disclosure

Why wrong: This involves leaking data, not overriding instructions.

Supply chain vulnerabilities

Why wrong: Supply chain issues relate to third-party components.

Model denial of service

Why wrong: DoS floods the model, not manipulates instructions.

Homomorphic encryption

Why wrong: Homomorphic encryption protects data in use but does not prevent membership inference.

Model pruning

Why wrong: Pruning reduces model size, not privacy.

Federated learning

Why wrong: Federated learning keeps data local but does not guarantee indistinguishability.

Red teaming the model

Why wrong: Red teaming identifies vulnerabilities but does not provide runtime protection against over-reliance.

Rate limiting and input validation

Why wrong: These address abuse and injection, not over-reliance.

Differential privacy

Why wrong: Differential privacy protects training data, not over-reliance.

Using a larger context window

Why wrong: Larger context window does not affect authorization.

Enabling response logging

Why wrong: Logging is detective, not preventive.

Implementing rate limiting

Why wrong: Rate limiting throttles usage but does not prevent unauthorized access.

Prompt injection

Why wrong: Prompt injection is a vector, but the risk is amplified if the model has agency.

Model denial of service

Why wrong: DoS prevents access, not unauthorized actions.

Hallucinations producing dangerous advice

Why wrong: Hallucinations are about inaccurate information, not actions.

Rate limiting

Why wrong: Rate limiting reduces attack volume but does not prevent injection.

Differential privacy

Why wrong: Differential privacy does not address injection.

Federated learning

Why wrong: Federated learning is a training paradigm, not a runtime defense.

Deploying the model on the fastest hardware available

Why wrong: Hardware speed is a performance concern, not security.

Using homomorphic encryption for all data at rest

Why wrong: Homomorphic encryption is not practical for all data at rest; standard encryption suffices.

Queries that include SQL injection attempts

Why wrong: SQL injection targets databases, not model extraction.

Queries that repeatedly ask for the same prediction

Why wrong: Repeated identical queries are more indicative of a denial-of-service or probing, not extraction.

Queries with special characters attempting to reveal system prompts

Why wrong: This suggests prompt injection or jailbreaking, not extraction.

A technique that reconstructs training data from the model's outputs

Why wrong: This describes model inversion, not adversarial examples.

An attack that injects malicious data into the training set to corrupt the model

Why wrong: This describes data poisoning, not adversarial examples.

A method to determine if a specific data point was used in the training set

Why wrong: This describes membership inference attacks.

Data poisoning

Why wrong: Data poisons the training data, but the immediate concern for deployed LLM is prompt injection leaking secrets.

Membership inference

Why wrong: Membership inference determines if a record was in training data, but does not directly leak email contents.

Model extraction

Why wrong: Model extraction aims to replicate the model, not directly leak confidential emails.

Prompt injection

Why wrong: Prompt injection is specific to LLMs, not fraud detection models.

Adversarial examples

Why wrong: Adversarial examples affect individual predictions, not a systematic degradation across many samples after retraining.

Model inversion

Why wrong: Model inversion reconstructs training data, not degrade performance.

Monitor API usage for anomalous patterns

Why wrong: Monitoring can detect attacks but not prevent supply chain backdoors.

Use federated learning to train the model

Why wrong: Federated learning trains across decentralized data, but does not verify pre-trained model integrity.

Implement differential privacy during training

Why wrong: Differential privacy protects training data privacy, not detect backdoors.

Input validation and sanitization

Why wrong: Input validation helps prevent injection but does not stop the model from generating toxic outputs.

Rate limiting on API requests

Why wrong: Rate limiting limits request volume but does not filter content.

Red teaming the AI system

Why wrong: Red teaming is a testing methodology, not a runtime defense.