Scenario practice questions

Implement rate limiting on API calls to prevent denial-of-service attacks.

Why it fails: Rate limiting is a security control to ensure availability and prevent abuse, but it has no bearing on model interpretability or trust. The scenario is about explaining predictions to medical professionals, not about protecting the API from attacks. Thus, this control, while generally good practice, is not relevant to the core need described.

Apply encryption to the model weights to prevent unauthorized access.

Why it fails: Encrypting model weights protects intellectual property and prevents theft, but it does not provide explanations for predictions. While security is important, the scenario emphasizes trust and explainability for medical professionals. Encryption alone does not address the need to understand model decisions, making it insufficient as the most critical control here.

Use federated learning to train the model across multiple hospitals without sharing patient data.

Why it fails: Federated learning enhances privacy by keeping data local, which is valuable in healthcare. However, it does not directly provide interpretability or explanations for predictions. The scenario focuses on trust and explanation for radiologists, so while federated learning is beneficial, it is not the most critical control for this specific requirement.

Insufficient adversarial robustness testing against prompt injection attacks on the API endpoint.

Why it fails: Prompt injection is an attack technique where malicious input manipulates model behavior, but nothing in the scenario indicates an attacker is involved. The observed change results from a legitimate vendor update, not adversarial manipulation, so robustness testing against injection would not address the root cause of silently shifting model behavior.

Failure to implement differential privacy on the prompts sent to the third-party model.

Why it fails: Differential privacy protects individual data records from being inferred from model outputs and is relevant to training data privacy, not to detecting vendor-side model changes. Applying it to prompts would not have revealed the weight update and does not address the governance gap that allowed an unannounced change to affect regulated outputs.

Lack of model explainability tooling to interpret why outputs changed after the update.

Why it fails: Explainability tools help interpret model reasoning but are downstream of the real problem, which is that the firm had no visibility or notification that a change occurred at all. Even perfect explainability would not remedy the absence of contractual change-control and version-pinning provisions with the model provider.

A quarterly vendor performance dashboard reporting incident counts for AI-enabled equipment already deployed across plants.

Why it fails: A performance dashboard monitors existing vendors but does not govern how new AI-enabled suppliers are selected and approved. It is a downstream monitoring tool, not an upstream control. The scenario concerns creating actionable rules for vendor selection and deployment, so a dashboard leaves the primary governance gap unaddressed and provides no criteria for evaluating prospective suppliers.

A training deck explaining the board's AI principles so that procurement staff can interpret them when negotiating vendor contracts.

Why it fails: Training improves awareness but leaves interpretation to individual procurement staff, producing inconsistent outcomes and no auditable criteria. Without a standard defining required evidence and approval gates, two buyers could reach opposite conclusions from the same principles. Governance requires enforceable, documented criteria rather than relying on staff judgment to translate principles during negotiations.

A revised AI principles statement with additional clauses addressing vendor responsibility and ethical sourcing.

Why it fails: Adding clauses to the principles statement keeps the guidance at the same abstract level and does not give business units actionable vendor evaluation criteria. Principles express values and intent; they are not designed to specify evidence, approval gates, or assessment procedures. Revising them delays practical guidance and risks continued inconsistent vendor decisions across business units.

Acquiring a subsidiary always requires retraining all enterprise AI models from scratch.

Why it fails: There is no universal requirement to retrain all models from scratch after an acquisition. The risk management lesson is to assess how new data sources affect model behavior and governance, not to mandate wholesale retraining. This option overstates a specific remediation as a general rule and misses the underlying data drift and ethical risk issue.

Recommendation engines are inherently unbiased and only reflect objective customer preferences.

Why it fails: Recommendation engines learn patterns from historical data and can encode cultural, social, and commercial biases present in that data. The scenario directly contradicts this claim by showing culturally insensitive outputs resulting from data changes, so asserting inherent objectivity mischaracterizes how these models work and ignores the observed harm.

Ethical risks in AI are only relevant during the initial design phase and not after deployment.

Why it fails: Ethical risks can emerge or intensify after deployment, as this scenario demonstrates. The culturally insensitive recommendations appeared only after new training data was introduced post-deployment, proving that ethical risk management must continue throughout the AI lifecycle rather than being confined to initial design.

The AI system running slower than usual.

Why it fails: Degraded performance indicates a reliability or capacity fault, not compromise of the model's integrity, confidentiality or behaviour. It is tempting because slowdowns often accompany attacks, and performance degradation would be the correct finding in an availability-monitoring or SLA-breach investigation rather than a security incident report.

A scheduled server maintenance window.

Why it fails: Scheduled maintenance is planned operational downtime, involving no adversarial manipulation of an AI system and no security consequence. It is tempting because any interruption to an AI service can look like an incident, and a maintenance window would be the correct explanation for planned unavailability in a change-management record.

A user simply disliking the AI output.

Why it fails: A user disliking output is subjective dissatisfaction, with no attacker, policy violation or security impact on the AI system. It is tempting because poor output can signal manipulation, and user dissatisfaction would be the correct subject of a quality-feedback or model-tuning review, not an AI security incident.

Retire the model permanently and revert to fully manual claim review until a replacement is validated.

Why it fails: Permanent retirement is a disproportionate response to a detected distribution shift. It discards a functioning capability and imposes a large operational cost on the claims function, while a validated replacement may take months. The sharper issue is that the model's operating threshold and monitoring are no longer matched to current data, which can often be corrected faster than a full replacement.

Suppress reporting of the increased false-positive rate until the next scheduled annual model validation to avoid disrupting operations.

Why it fails: Withholding a known degradation from governance bodies defeats the purpose of monitoring and delays treatment. It also creates a documentation gap that would be difficult to defend in an audit or regulatory inquiry. Material changes in model performance should be escalated when detected, not held until a calendar milestone, regardless of operational inconvenience.

Leave the model unchanged and rely on the existing appeals process to correct any erroneous fraud flags after claimants complain.

Why it fails: Relying on appeals shifts the burden of detecting errors onto affected customers and delays correction until after harm has occurred. It also fails to address the underlying cause, so false positives will continue at an elevated rate. Using the appeals channel as the primary control converts a preventable model degradation into a customer-experience and fairness problem.

Immediately decommission the model and replace it with a rules-based scoring system to eliminate drift risk entirely.

Why it fails: Replacing the model is a disproportionate response. Drift is an inherent, manageable risk in any production ML system, and a rules-based system introduces different risks while discarding the model's value. The scenario asks for risk management, not risk elimination, and decommissioning without cause would be poor governance.

Document the drift risk in the register as accepted, noting that the model's current accuracy exceeds the business requirement.

Why it fails: Accepting the risk without an owner or monitoring is not a valid treatment. Risk acceptance must be a conscious, authorized decision made by an accountable party, typically with defined review intervals. Simply recording acceptance while leaving the entry unowned and unmonitored perpetuates the exact governance gap the analyst identified.

Wait until the next scheduled annual model retraining to reassess drift, since the model currently meets performance targets.

Why it fails: Deferring drift reassessment to an annual retraining cycle leaves the credit-scoring model exposed to silent degradation for up to a year. In risk management, an unowned and unmonitored risk cannot be governed, and performance on a static validation set says nothing about live data distribution shifts. This defers rather than treats the risk.

Require all AI systems to be rated 'high' residual risk until they have completed a full independent audit, then downgrade them once the audit report is signed off by the CISO.

Why it fails: This inflates every system to 'high' and destroys the ranking value the committee needs for remediation budgeting. It also still treats the post-audit rating as static, so the same drift that produced the biased credit-scoring outcomes would recur. The scenario demands ongoing re-evaluation, not a one-time audit gate that collapses the risk scale.

Assign each AI system an owner accountable for the residual risk rating and require that owner to attest annually that the rating remains accurate and that no material model changes have occurred.

Why it fails: Annual attestation is too infrequent to catch a bias drift that materialized within six months and depends on the owner self-reporting changes they may not detect. Accountability is necessary but insufficient; without defined re-evaluation triggers and register updates, the rating stays stale between attestations, and the biased credit-scoring behavior would continue undetected.

Move the AI risk register into the enterprise risk management (ERM) system and adopt the ERM likelihood-by-impact scoring scale so that AI risks are comparable to operational and financial risks.

Why it fails: Harmonizing scales improves enterprise comparability but does nothing to catch post-deployment drift or emergent bias. A medium rating on a shared ERM scale would still have been assigned once and left unchanged. The scenario's failure is temporal, not lexical, so changing the scoring scale without adding re-evaluation triggers leaves the root cause intact.

Model inversion attack

Why it fails: Model inversion attempts to reconstruct training data or extract sensitive information from a model's outputs. While it can be performed in a black-box setting, the goal here is to cause misclassification (evasion), not to infer training data. The scenario describes an attacker aiming to evade detection, not to extract information, so model inversion is not the correct category.

Data poisoning attack

Why it fails: Data poisoning involves manipulating the training data to corrupt the model's learning process. In this scenario, the attacker has no access to the training pipeline or data; they only query the deployed model. Thus, poisoning is not feasible. The attack occurs at inference time, not training time, so this option does not fit the described adversary capabilities.

White-box evasion attack

Why it fails: A white-box evasion attack requires full knowledge of the model's architecture, parameters, and gradients. In this scenario, the attacker only has black-box query access to the API and cannot compute gradients directly. Therefore, white-box techniques such as Fast Gradient Sign Method (FGSM) or Projected Gradient Descent (PGD) are not applicable. The attacker must rely on query-based methods, making this option incorrect.

Re-run the pre-deployment validation test suite and confirm the model still passes its original acceptance thresholds.

Why it fails: Re-running the original validation suite only confirms the model meets the thresholds set at deployment. Because the drift is demographic and masked by stable aggregate accuracy, the original suite will likely pass and provide false assurance. Post-deployment monitoring must evaluate subgroup performance continuously rather than relying on a static pre-launch baseline that cannot detect emerging fairness harms.

Increase the human review rate for all denied claims from ten percent to one hundred percent and pause model retraining indefinitely.

Why it fails: Full human review of denials addresses the symptom but not the governance gap, and pausing retraining freezes the model in a degraded state. It also creates unsustainable operational cost without producing the subgroup metrics needed to understand root cause. Governance requires a detection and response capability, not a permanent manual workaround that masks the underlying drift and leaves the model unimproved.

Document the drift in the model risk register and schedule a comprehensive model revalidation for the next annual review cycle.

Why it fails: Documenting drift without immediate detection and response controls leaves affected claimants exposed for up to a year. Annual revalidation cycles are too slow for a live model producing potentially discriminatory outcomes. Governance requires timely monitoring and escalation proportional to harm, and deferring action to a scheduled cycle fails to meet the expectation of ongoing oversight for high-impact AI decisions.

Enforce strict input validation to reject transactions that fall outside the historical range.

Why it fails: Input validation can filter out malformed data but cannot handle legitimate transactions that are simply different due to a promotion. Rejecting such transactions would cause more false positives or block valid customers. This control is too rigid and does not adapt to changing data patterns, which is the core issue here.

Apply differential privacy during model training to reduce the model's sensitivity to individual transactions.

Why it fails: Differential privacy adds noise to protect individual data points but does not address distributional shifts in input data. It would not prevent the model from misclassifying legitimate transactions during a promotional period. While valuable for privacy, it is not the appropriate control for handling data drift or concept drift in this scenario.

Use adversarial training to make the model robust against malicious inputs.

Why it fails: Adversarial training improves robustness against deliberately crafted inputs designed to fool the model, not against natural shifts in data distribution from business events. The promotional period is not an adversarial attack; it is a legitimate change in customer behavior. This control would not address the false positive spike caused by distributional shift.

Use a cloud-based facial recognition API with strong encryption for data in transit and at rest.

Why it fails: Cloud-based processing with encryption protects data during transmission and storage, but it still involves sending sensitive biometric data to a third party, which may violate privacy regulations. Encryption does not prevent unauthorized access or surveillance by the cloud provider or attackers. This approach increases the attack surface and may not meet strict government privacy requirements.

Regularly audit the facial recognition model for bias and accuracy.

Why it fails: Auditing for bias and accuracy is important for fairness and performance, but it does not directly address privacy risks or unauthorized surveillance. These audits focus on model behavior, not on data protection or surveillance minimization. They are complementary but not the primary control for this privacy-centric scenario.

Obtain consent from all individuals entering the public space before processing their faces.

Why it fails: Obtaining consent from all individuals in a public space is impractical and often impossible. It does not scale and may not be legally sufficient in many jurisdictions. While consent is important, it is not a technical control that can be enforced reliably. This approach fails to address the operational realities of public surveillance.

Reject the vendor outright, since undisclosed training data composition makes any risk assessment impossible.

Why it fails: Outright rejection is an absolutist position that ignores compensating controls. Risk management permits accepting residual risk when it is understood, owned, and mitigated to an acceptable level. A third-party audit attestation is itself a form of assurance that can partially substitute for direct disclosure, so blanket rejection overstates the impossibility of assessment.

Accept the vendor's accuracy metrics as sufficient evidence and deploy the model without additional controls.

Why it fails: Relying solely on vendor-reported accuracy metrics is inadequate because those metrics are self-selected and may not reflect the hospital's patient population. Without independent validation or contractual assurance, the committee cannot claim the residual risk is understood or bounded. This approach effectively transfers trust to the vendor without any governance mechanism.

Require the vendor to sign a liability waiver transferring all clinical decision responsibility to the hospital.

Why it fails: A liability waiver addresses legal exposure, not the underlying risk of unknown model provenance. It does nothing to improve assurance, validate performance on local data, or establish oversight. Shifting liability does not reduce the chance of patient harm, and it leaves the hospital bearing the clinical and reputational consequences without added controls.

Accuracy

Why it fails: Accuracy measures how often the model makes correct predictions. While high accuracy is desirable, it does not provide explanations for individual decisions. A highly accurate model can still be a black box, failing to meet the need for explainability to applicants and auditors.

Latency

Why it fails: Latency refers to the time it takes for the model to produce a prediction. Low latency is important for real-time applications, but it does not contribute to explaining decisions. The requirement is about transparency and compliance, not speed.

Scalability

Why it fails: Scalability is about the model's ability to handle increasing amounts of data or users. While important for performance, it does not help explain decisions to applicants or auditors. The scenario emphasizes explainability for compliance, not the ability to scale.

Commission a full adversarial red-team exercise against the credit-scoring model before any documentation changes are made.

Why it fails: Red-teaming probes exploitable weaknesses such as evasion or extraction, but it does not produce the missing feature rationale or intended-use statement. Running it first consumes scarce security resources while the underlying governance gap—no documented model rationale or limitations—remains unaddressed. The scenario describes a documentation and accountability deficiency, not a demonstrated exploitable vulnerability, so adversarial testing is a downstream activity rather than the first corrective step.

Immediately retrain the model with a different algorithm that offers inherently higher interpretability.

Why it fails: Swapping algorithms may improve interpretability, but it discards the existing validation evidence and does not by itself create the required documentation of intended use and limitations. The firm would still lack a defensible rationale for the new model's feature weighting. Retraining is a disproportionate, disruptive response when the immediate deficiency is governance documentation rather than measured model performance or a proven technical flaw.

Encrypt the training dataset at rest and restrict access to the model's API endpoint.

Why it fails: Encryption at rest and API access restriction are sound data-protection controls, but they address confidentiality of data and model access, not the missing rationale for feature weighting or the absent intended-use and limitation statements. Those controls would leave the accountability gap intact, so the model would still fail an AI risk review that requires documented model purpose, constraints, and design justification.

Implement differential privacy during training to ensure individual data points cannot be inferred.

Why it fails: Differential privacy protects individual privacy by adding noise, but it does not address group fairness or disparate impact. It could even exacerbate bias by adding noise that disproportionately affects minority groups. This technique is unrelated to detecting or mitigating bias in model decisions, and thus is not the appropriate control for this scenario.

Use SHAP (SHapley Additive exPlanations) values to interpret model predictions and identify biased features.

Why it fails: SHAP values explain individual predictions by attributing them to features, which can reveal if sensitive attributes influence outcomes. However, in this scenario, the sensitive attributes were removed from the training data, so they are not directly present. SHAP might still show correlations through proxy features, but it is primarily an interpretation tool, not a mitigation technique. It helps detect bias but does not directly mitigate it.

Use model watermarking to embed ownership information and track unauthorized use.

Why it fails: Model watermarking is a technique for intellectual property protection, embedding a hidden signature in the model to verify ownership. It has no role in detecting or mitigating bias in model predictions. This option is a distractor that confuses security controls for model theft with fairness controls.

Membership inference attack

Why it fails: A membership inference attack determines whether a specific data point was part of the training set. It does not involve perturbing inputs to cause misclassification. The scenario focuses on altering malicious files to evade detection, which is unrelated to membership inference.

Poisoning attack

Why it fails: A poisoning attack occurs during training, where an attacker injects malicious data into the training set to corrupt the model's behavior. The scenario describes testing perturbations at inference time on already trained model, not manipulating the training data. Therefore, poisoning is not the appropriate simulation.

Model inversion attack

Why it fails: A model inversion attack attempts to reconstruct training data or extract sensitive information from a model by querying it. It does not aim to cause misclassification of inputs. The team's goal is to test misclassification through perturbations, so model inversion is irrelevant here.

Enforce role-based access control (RBAC) to limit who can view the surveillance footage.

Why it fails: RBAC is a standard access control mechanism applicable to any IT system. It is not AI-specific and does not mitigate risks unique to AI, such as adversarial attacks or unintended bias. While essential for overall security, it does not qualify as an AI-specific control in this context.

Use HTTPS to encrypt video feeds in transit between cameras and the processing server.

Why it fails: HTTPS encryption is a general network security control, not specific to AI. While important for protecting data in transit, it does not address AI-specific risks such as model evasion or privacy leakage through inference. The scenario asks for AI-specific controls, so this option, though good practice, is not the best fit.

Regularly update the AI model with new data to improve accuracy.

Why it fails: Regular model updates can improve performance but are not a specific control for privacy or security risks. In fact, without proper validation, updates could introduce new vulnerabilities or biases. This is a maintenance activity, not a targeted AI security or privacy control, and thus is not the correct choice.