AI Risk Management practice questions

A mandate to use the vendor's cloud region exclusively.

Why it fails: Cloud region usage is a data residency concern, not a shadow AI control.

A request for a free trial period.

Why it fails: This is a commercial term, not a risk mitigation control.

A requirement for 99.99% uptime SLA.

Why it fails: Uptime is an availability metric, not a risk/shadow AI control.

Overfitting of the LLM on the training set.

Why it fails: This happens during training, not during retrieval.

Gradient-based evasion attacks on the LLM.

Why it fails: This is a standard adversarial ML attack, not RAG-specific.

Prompt injection targeting the LLM model weights.

Why it fails: This is a standard LLM vulnerability, not specific to the RAG retrieval mechanism.

Increasing the learning rate during training.

Why it fails: Learning rate changes do not improve robustness to evasion.

Switching to a larger neural network.

Why it fails: Larger networks can often be more vulnerable to adversarial examples.

Implementing input denoising.

Why it fails: Denoising helps with random noise but not deliberate gradient-based attacks.

To gain access to the raw training database.

Why it fails: This is a data exfiltration attack.

To cause the model to crash.

Why it fails: This is a DoS attack.

To inject malicious weights into the model.

Why it fails: This is a poisoning attack.

AI systems should be retired once they reach end of life in the technology portfolio.

Why it fails: End-of-life retirement is a lifecycle management concern that applies after a system is no longer needed or supportable, but the scenario concerns how risks are recorded and treated while systems are active. Decommissioning does not address ownership, rating, or treatment documentation and is not the principle the described practice reflects.

Model performance metrics should be reported to the board on a quarterly basis.

Why it fails: Board reporting is a governance communication activity that may follow from risk management, but the scenario describes the mechanics of recording and treating individual risks. Quarterly performance reporting does not by itself ensure that each risk has an owner, a rating, and a documented treatment decision.

All AI risks must be eliminated before any system can be deployed to production.

Why it fails: Risk management recognizes that residual risk is acceptable when it falls within tolerance, so universal elimination is neither required nor realistic. The described practice of documenting treatments, including acceptance, contradicts the idea that every risk must be eliminated and reflects proportionate management instead.

Decommission the model immediately and revert to the previous manual readmission assessment process until a new model is built from scratch.

Why it fails: Immediate decommissioning is disproportionate when the disparity can be mitigated with targeted controls. A full revert may introduce greater clinical risk from inconsistent manual assessments and disrupt care operations. ISACA's risk-based approach favors proportionate response: contain the harm, document the issue, and remediate systematically rather than abandoning a functional system.

Remove the age variable from the model's feature set to eliminate the source of the disparity.

Why it fails: Removing age does not eliminate the disparity because correlated features such as diagnosis codes or medication history can serve as proxies. This approach also degrades model performance for the elderly without addressing the underlying data representation problem. ISACA guidance warns against superficial fairness fixes that mask rather than resolve discriminatory outcomes in AI systems.

Increase the model's overall accuracy threshold from 85% to 95% to ensure better predictions for all patients including those over 65.

Why it fails: Raising a global accuracy threshold does not fix subgroup disparities and may be unachievable without architectural changes. Aggregate accuracy can remain high while a specific cohort is systematically underserved, which is exactly the pattern observed. This action addresses the symptom superficially and could delay meaningful remediation while consuming resources on an ineffective adjustment.

Encrypt the postal code field at rest and in transit so the model cannot use it for pricing decisions.

Why it fails: Encryption protects data confidentiality but does not prevent a model from using a feature during inference. If postal code is an input, the model can consume it whether or not it is encrypted at rest. This control addresses a security concern, not proxy discrimination. Confusing data protection with fairness mitigation is a common governance error that leaves the discriminatory mechanism intact.

Increase the model's regularization strength to reduce overfitting to local pricing patterns.

Why it fails: Regularization controls model complexity and can improve generalization, but it does not target the specific mechanism of proxy discrimination. A well-regularized model can still discriminate if a proxy feature carries predictive signal correlated with a protected characteristic. This option addresses overfitting, a different risk, and would not provide evidence of fairness to a regulator or internal audit.

Add a disclaimer to the checkout page stating that prices are generated by an automated system and may vary.

Why it fails: A disclosure informs customers but does not reduce discrimination. Transparency about automation does not change the model's inputs or outputs, nor does it satisfy fair-pricing obligations. In many jurisdictions, disclosing an unfair practice does not cure the unfairness. This option addresses notice and consent, which is a separate governance concern from proxy discrimination mitigation.

The model produces hallucinatory facts.

Why it fails: Hallucination is the model generating unsupported content from its own parameters; indirect prompt injection instead arrives through external data the agent ingests, hijacking its tool calls and outputs. Hallucination is the right answer when accuracy of generated facts, without any adversarial input, is the concern.

The API key for the LLM is exposed in the code.

Why it fails: Indirect prompt injection embeds adversarial instructions in content the agent retrieves, such as web pages or documents, causing it to execute attacker-chosen actions. Exposed API keys are a secrets-management failure. Key exposure is the correct finding when credentials are committed to repositories or embedded in client-side code.

The training data includes PII of users.

Why it fails: Training-data PII is a privacy and data-governance exposure arising before deployment; indirect prompt injection is a runtime attack delivered via retrieved content that subverts the agent's instructions. PII in training corpora is the correct concern when assessing model provenance and regulatory compliance obligations.

Unauthorized modification of model weights.

Why it fails: Membership inference attacks leak information about training data through prediction confidence; they do not alter stored parameters, since the adversary only queries the deployed model. Weight modification describes a poisoning or tampering attack, relevant when an adversary can write to the model store or training pipeline.

Unauthorized access to the model training algorithm.

Why it fails: Membership inference attacks exploit model outputs to infer training-set membership; they do not expose the training algorithm itself, which is typically proprietary code rather than a learned artefact. Algorithm theft is the concern of model-extraction attacks, where an adversary reconstructs the model's functionality through queries.

Complete system shutdown via resource exhaustion.

Why it fails: Membership inference attacks infer whether a specific record was in the training set, leaking privacy; they do not exhaust compute or memory to halt the system. Resource exhaustion describes denial-of-service, which would be the concern when an adversary floods the model with queries to degrade availability.

Requiring all models to be open-source.

Why it fails: Mandating open-source models constrains procurement rather than addressing NIST AI RMF's risk-governance outcomes, which apply equally to proprietary and open models. It is tempting because open weights permit inspection and independent evaluation, and would be the right choice where auditability of model internals is a hard requirement.

Implementing a strict 'no external API' policy for all models.

Why it fails: A blanket no-external-API rule blocks legitimate third-party model access rather than governing it, so it cannot satisfy NIST AI RMF's Govern, Map, Measure and Manage functions. It is tempting because restricting external calls reduces data-leak exposure, and would suit a fully air-gapped deployment where no external model dependency is permitted.

Retrain the model immediately with all available data.

Why it fails: Retraining immediately skips the assessment: without first quantifying drift, identifying affected features and checking data quality, retraining may bake in corrupted or shifted data. Retraining is the correct response once assessment confirms genuine, persistent drift and clean labelled data are available.

Shut down the application to prevent incorrect predictions.

Why it fails: Shutting down the application is a containment action, not a risk assessment step, and it halts business value before the drift's cause, scope and impact are understood. It would be justified only once assessment shows predictions are unsafe and immediate harm is likely.

Change the model architecture to a more complex design.

Why it fails: Changing the architecture is a remediation decision that presupposes the assessment's findings; drift often stems from input data changes rather than model capacity. Architecture redesign would be warranted after assessment confirms the model itself, not the data pipeline, is the limiting factor.

Apply data masking to the input images.

Why it fails: Data masking replaces or redacts sensitive fields in structured records; it does not alter the spatial pixel relationships an adversarial perturbation exploits, so the manipulated image still reaches the model. It is tempting as a recognised privacy control, and would be correct where the requirement is hiding personally identifiable information in stored or displayed datasets.

Implement differential privacy during data preprocessing.

Why it fails: Differential privacy adds calibrated noise to aggregate outputs or training data to protect individual records, so it does not harden the inference path against adversarially crafted pixel perturbations. It is tempting because it is a genuine privacy-enhancing technique, and it would be the right control when the requirement is preventing membership inference or reconstruction of training data.

Increase the model epoch count to improve generalization.

Why it fails: More epochs change how long training runs, not the model's sensitivity to small input gradients, so a perturbed image still crosses the decision boundary. It is tempting because extra training is often assumed to harden models generally, and would be the right response when the actual problem is underfitting on the legitimate training distribution.

Implement role-based access control for the training pipeline.

Why it fails: RBAC governs who may push artefacts to the pipeline, not whether the downloaded base weights themselves were tampered with before arrival. It is tempting because pipeline access control is a genuine supply chain safeguard, and it would be correct for protecting your own training infrastructure from unauthorised modification.

Run a sensitivity analysis on the model output.

Why it fails: Sensitivity analysis measures how output varies with input perturbations, revealing nothing about embedded triggers or backdoors in the pre-trained weights. It is tempting because it is a real model-validation technique, and it would be correct for assessing robustness, fairness or feature influence after integration.

Enable MFA for the model registry access.

Why it fails: MFA hardens authentication to the registry but cannot detect malicious weights already published by the third party. It is tempting because identity controls are a recognised supply chain mitigation, and MFA would be correct for preventing credential-based compromise of your own registry or repository accounts.

Model training speed vs. model cost.

Why it fails: Differential privacy's cost is accuracy: noise calibrated to epsilon degrades model utility as privacy guarantees strengthen. Training speed and cost are unaffected by the privacy mechanism itself. That trade-off applies when selecting instance types or distributed training strategies, not when tuning privacy budgets.

Security of the training pipeline vs. developer productivity.

Why it fails: Differential privacy manages the trade-off between analytical utility and individual privacy: adding noise reduces identifiability but degrades accuracy. Security of the pipeline versus developer productivity describes DevSecOps friction, not the privacy-utility balance differential privacy governs.

Deployment latency vs. model scalability.

Why it fails: Differential privacy bounds how much any single record influences output by adding calibrated noise, trading model accuracy against privacy budget (epsilon). Deployment latency and scalability are unrelated axes; that trade-off belongs to infrastructure sizing or inference architecture decisions, not to the noise-injection mechanism itself.

To verify the model meets legal data storage requirements.

Why it fails: A bias audit evaluates whether model outputs disadvantage particular groups; legal data storage requirements concern retention and residency, not fairness. It is tempting because compliance is part of AI governance, but storage obligations are verified through data-handling and privacy audits, not bias testing.

To reduce the cost of model deployment.

Why it fails: A bias audit examines model outputs for discriminatory or unfair treatment across groups; deployment cost is unrelated to that objective. It is tempting because cost reduction is a legitimate AI programme goal, but it is achieved through cost-optimisation reviews, not through assessing fairness or disparate impact.

To ensure the model is computationally efficient.

Why it fails: A bias audit assesses fairness and disparate impact across protected groups; computational efficiency concerns runtime and resource use, not bias. It is tempting because efficiency is a valid model concern, but it is addressed by performance profiling and optimisation, not by the fairness testing a bias audit performs.

The model is overwritten with malicious logic.

Why it fails: Model inversion reconstructs training data or sensitive attributes by querying a model's outputs; overwriting the model with malicious logic describes a poisoning or backdoor attack instead. It is tempting because both are model-focused threats, and would be correct if the scenario involved tampered training data or injected malicious weights.

The model becomes unavailable to users.

Why it fails: Model inversion exposes training data through query access; unavailability to users describes a denial-of-service or availability attack instead. It is tempting because both are AI security risks, and would be correct if the scenario involved resource exhaustion, model theft of service, or an outage preventing legitimate predictions.

The model predicts the wrong output due to noisy inputs.

Why it fails: Model inversion is an inference-time confidentiality attack that recovers training data through repeated queries, not mere inaccurate prediction from noisy inputs. It is tempting because both concern model outputs, and would be correct if the scenario described degraded accuracy or robustness against adversarial or noisy input.

Encrypting the model weights at rest in the cloud.

Why it fails: Encrypting weights at rest protects stored model artefacts, not the API endpoint that serves inference requests. It is correct for securing model files in object storage or volumes, but does nothing against unauthorised API calls, abuse or data interception in transit.

Removing all logging to prevent log injection.

Why it fails: Removing logging eliminates the audit trail needed to detect prompt injection, abuse and data exfiltration against the endpoint, so it directly weakens detection rather than securing it. It tempts because log injection is a genuine risk, but the correct remedy is sanitising and validating log input, not deleting telemetry entirely.

The physical office building security.

Why it fails: Physical office building security protects facilities and staff, not the AI system's model, data, or pipeline components requiring individual threat assessment. It is tempting because physical controls genuinely matter for overall AI risk, but they fall under organisational security rather than the AI system boundary itself.

The personal devices of the developers.

Why it fails: Developer personal devices sit outside the AI system boundary; they are endpoint and personnel concerns, not model, data, or pipeline components needing individual threat assessment. It is tempting because compromised developer endpoints can poison training data, but that is a supply-chain risk, not an AI system component.

The internal organizational culture.

Why it fails: Internal organisational culture is a governance and human-factors consideration, not a technical AI system component requiring individual threat assessment. It is tempting because culture shapes how AI risks are managed, but the assessment targets the model, data, and pipeline elements themselves.

Check for compliance with data privacy laws.

Why it fails: Data privacy compliance checks examine lawful processing of personal data, not adversarial prompt handling, so they never exercise injection defences. It is tempting because red teaming sits within governance programmes, and privacy auditing would be the right activity for a regulatory compliance assessment.

Review the training data logs for errors.

Why it fails: Reviewing training data logs is a static quality check that cannot reveal how a live model responds to crafted adversarial prompts. It is tempting because data quality strongly influences model behaviour, and log review would be correct when investigating bias or labelling errors.

Scan the code repository for hardcoded secrets.

Why it fails: Scanning repositories finds committed credentials, not adversarial prompts; it cannot exercise the model's instruction-following boundary. Secret scanning is the right control when the risk is exposed API keys or tokens in source code, which is a supply-chain concern rather than prompt injection testing.

The aggregator becoming a single point of failure.

Why it fails: Aggregator availability is a resilience concern, not the security risk posed by model updates themselves; those updates can leak training data through gradient inversion or poisoning. Single-point-of-failure planning suits infrastructure redundancy design, whereas the stem asks about the confidentiality and integrity of transmitted updates.

Encryption of the communication channel.

Why it fails: Channel encryption is a control, not the concern; the updates themselves can still leak training data via gradient inversion or be poisoned. Encrypting transport suits protecting data in transit generally, but federated learning's distinctive risk lies in what the model updates reveal or corrupt once decrypted.

Slow network speeds at the edge.

Why it fails: Network latency is an availability and performance issue, not the confidentiality or integrity threat posed by model updates, which can be poisoned or inverted to reconstruct client training data. Slow links matter when tuning aggregation frequency or bandwidth budgets, but they do not expose gradients to adversarial manipulation.