Hard Difficulty Questions

Writing a blog post about the failure

Why it fails: Publishing a blog post does not remediate the model or satisfy the governance policy's required corrective workflow, such as retraining, documenting, or suspending deployment. Transparency write-ups suit post-incident communication for public accountability, not the compliance remediation itself.

Ignoring the finding if the model is profitable

Why it fails: Profitability is not a governance criterion; the policy requires remediation regardless of financial performance, so ignoring the finding leaves the model deployed in violation. Ignoring findings is never sanctioned, though deprioritising low-risk items may occur under a documented risk-acceptance process.

Giving the model a new name

Why it fails: Renaming the model changes no configuration, data, or behaviour, so the underlying non-compliance persists and the policy's remediation requirements remain unmet. Renaming is useful for versioning or lifecycle housekeeping, not for resolving a governance finding.

Man-in-the-middle attack

Why it fails: Man-in-the-middle intercepts traffic between communicating parties, which does not alter training data or model weights. It is tempting because AI pipelines move data across networks, and would be correct when assessing interception of model updates or inference requests in transit.

Denial-of-service attack

Why it fails: Denial-of-service targets availability at inference time by exhausting compute, memory or request quotas, not the training phase. It is tempting because AI endpoints are expensive to query, and would be correct when assessing runtime resilience of a deployed model against resource exhaustion.

Phishing attack

Why it fails: Phishing harvests credentials from users, giving no access to the training pipeline or dataset. It is tempting because it is a common initial-access vector against organisations, and would be correct when assessing human-targeted compromise preceding lateral movement toward AI assets, not training-phase manipulation.

Increasing the number of neurons in the hidden layers

Why it fails: Adding neurons expands model capacity, which addresses underfitting, not the changing statistical relationship between inputs and target over time. Retraining on recent data is what counters drift; widening hidden layers suits a model that cannot capture the complexity already present in stationary data.

Moving the model to a larger GPU cluster

Why it fails: Extra GPU capacity raises throughput and shortens training time; it does not alter the input distribution or retrain the model, so predictions still degrade as the relationship between features and labels shifts. Larger clusters suit scaling compute for training runs, not correcting drift.

Restricting access to the training dataset

Why it fails: Restricting dataset access is a governance and privacy control; it neither detects nor corrects the shift in input distribution or label relationship that defines drift. Scheduled retraining on fresh data is the remedy; access restriction suits protecting sensitive training records from unauthorised use.

Printing to logs

Why it fails: Logs are widely readable and often shipped to third-party aggregators, so printing keys leaks credentials permanently. Printing tempts during debugging, but production deployments should fetch secrets at runtime from a managed vault such as Azure Key Vault, never writing them to log streams.

Hardcoding in the script

Why it fails: Hardcoded keys are exposed to anyone with source or image access, and rotation demands code changes. Hardcoding tempts for quick prototypes, but production deployments should retrieve secrets at runtime from a managed vault such as Azure Key Vault, keeping credentials out of the artefact.

Storing in a Git repo

Why it fails: Git history preserves committed secrets indefinitely, and repository clones or forks expose them broadly. Storing keys in Git tempts because it is convenient for configuration, but production deployments should retrieve secrets at runtime from a managed vault such as Azure Key Vault.

Running unit tests for code

Why it fails: Unit tests verify deterministic code paths, not adversarial natural-language prompts, so they cannot surface jailbreak bypasses. They are tempting because testing is a recognised assurance activity, and they would be correct for validating application logic, but safety-filter evaluation needs red-team prompt suites measuring refusal behaviour.

Reviewing server logs

Why it fails: Server logs record requests after processing, so they reveal jailbreaks only retrospectively and cannot systematically probe filter bypasses. They are tempting because logs expose attempted prompts, and log review would be correct for incident forensics or auditing historical usage rather than actively testing filter robustness.

Checking network bandwidth

Why it fails: Network bandwidth measures throughput, not prompt content or filter behaviour, so it cannot reveal jailbreak bypasses. It is tempting because large adversarial payloads shift traffic, and bandwidth monitoring would be correct for capacity planning or detecting anomalous data transfer volumes.

Company CEO name

Why it fails: A CEO's name has no bearing on pipeline execution time; latency is governed by model inference, data transfer and compute provisioning. It tempts because leadership decisions shape AI strategy, but strategy does not determine per-request pipeline latency.

Office building height

Why it fails: Office building height does not affect AI pipeline latency, which depends on compute, network and storage performance. It tempts because physical premises appear in some infrastructure discussions, yet building geometry has no bearing on request processing time.

Feature Importance Permutation

Why it fails: Feature importance permutation yields global rankings averaged across the dataset, not per-decision attributions for one applicant. It is tempting because it interprets complex non-linear models, but auditing individual credit decisions requires local methods such as SHAP or LIME that explain a single prediction's feature contributions.

Model pruning

Why it fails: Model pruning removes weights or neurons to shrink a network, producing no per-decision attribution for individual credit outcomes. It is tempting because pruning simplifies complex non-linear models, but simplification is not explanation; local explanations require techniques such as SHAP or LIME that attribute a single prediction to its input features.

Data normalization

Why it fails: Data normalisation rescales feature values before training and produces no per-instance attribution for individual credit decisions. It is tempting because it tames complex non-linear feature ranges, but preprocessing affects model inputs, not post-hoc explanation; local explanations need SHAP, LIME, or counterfactual methods that attribute a single prediction.

Reducing inference latency

Why it fails: Reducing inference latency is a performance optimisation, not a privacy attack; model inversion reconstructs training data by querying the model's outputs. Latency tuning would be the right concern when the requirement is real-time scoring throughput, not confidentiality of training samples.

Increasing training bias

Why it fails: Training bias concerns skewed or unrepresentative data producing unfair predictions, not reconstruction of training samples from model outputs. Bias mitigation would be the correct choice when the requirement is fairness across demographic groups rather than confidentiality.

Overfitting the data

Why it fails: Overfitting raises inversion susceptibility but is not inversion itself; inversion is the attack that exploits confidence outputs to reconstruct training records. Overfitting would be the correct focus when tuning regularisation to improve generalisation on held-out data.

Increased training latency

Why it fails: This is a performance risk, not a fundamental security or governance risk.

Lack of access to training hardware

Why it fails: Hardware access is a capacity planning issue, not an architectural risk.

Centralized data breaches

Why it fails: Federated learning is designed to avoid centralizing data, reducing this specific risk.

Ask developer for coffee

Why it fails: Decommissioning covers technical and governance tasks such as archiving model artefacts, revoking endpoint access, and notifying stakeholders; fetching coffee is unrelated to any AI governance lifecycle. It is tempting as a joke answer, but it satisfies no decommissioning requirement and addresses no model, data, or access concern.

Delete all company records

Why it fails: Decommissioning retires a model from production by archiving artefacts, revoking access, and updating documentation; deleting all company records destroys unrelated business data and violates retention obligations. It is tempting as a clean-up step, but record deletion is a data-governance decision, not part of the model retirement workflow.

Security audit

Why it fails: A security audit examines access controls, vulnerabilities, and compliance posture; it does not address statistical drift. Distribution shift requires retraining or drift-monitoring remediation. Security audits are the right trigger after suspected credential compromise or unauthorised model access, not performance decay.

Model Deletion

Why it fails: Deletion discards the model entirely, so it cannot restore accuracy when only the input distribution has shifted; the drift requires retraining or recalibration on current data. Deletion is tempting as genuine decommissioning of an obsolete or non-compliant model, but that scenario involves retirement, not recoverable data drift.

Hardware Upgrade

Why it fails: Added CPU or memory cannot correct a statistical shift between training and production inputs; the model's learned parameters no longer match the data, so retraining is required. Hardware upgrades are tempting when latency or throughput degrades, but that is a capacity problem, not distributional drift.

Code Refactoring

Why it fails: Refactoring restructures existing code without changing the model's learned parameters, so predictions remain miscalibrated against the shifted distribution; retraining on current data is needed. Refactoring is tempting when code quality or maintainability suffers, but that scenario concerns engineering hygiene, not model accuracy.

The model version

Why it fails: A PDB governs voluntary evictions, guaranteeing a minimum count of available replicas during node drains or upgrades; it says nothing about which model version runs. It is tempting because PDBs are core availability controls in Kubernetes AI deployments, and would be correct when the audit question concerns disruption tolerance rather than artefact versioning.

Maximum allowed CPU usage

Why it fails: Resource limit.

The cost of the cluster

Why it fails: A PDB governs voluntary eviction tolerances during node drains and upgrades, expressed as minAvailable or maxUnavailable replica counts. Cluster cost is a billing concern handled by cost management tooling, not by workload disruption policy. PDBs would be the correct answer when the question asks how many replicas may go offline during maintenance.

Defines the database password

Why it fails: An Endpoint Configuration specifies the model variant, instance type and count for hosting; it holds no database credentials, which belong to the model container's environment or secrets. It is tempting because configuration objects often carry connection settings, but here the artefact merely defines how SageMaker provisions the inference fleet.

Defines the data cleaning logic

Why it fails: An Endpoint Configuration specifies which model artefacts and instance types back an endpoint, plus variant weights. Data cleaning belongs to preprocessing scripts or pipelines, not to endpoint provisioning, so it plays no part in serving infrastructure.

Defines the training budget

Why it fails: An Endpoint Configuration specifies the model, instance type and variant settings used to deploy one or more endpoints; it has no bearing on training spend. Training budgets are governed by Training Jobs and their resource limits. Endpoint Configuration is the right artefact when you need to launch a production endpoint from a trained model.

Higher cloud bills

Why it fails: Missing system prompt versioning causes behavioural drift and irreproducible outputs, not higher cloud spend. Cost is tempting because token usage and model choice drive billing, but prompt versioning affects traceability and rollback, so the operational risk is uncontrolled changes to model behaviour.

Increased latency

Why it fails: Latency stems from model size, token volume, and infrastructure, not prompt versioning. Latency is tempting because pipeline performance is a common audit concern, but unversioned prompts cause inconsistent outputs and lost rollback capability, which is a correctness and governance risk rather than a speed one.

Database corruption

Why it fails: Database corruption arises from storage faults, failed writes, or hardware issues, none of which prompt versioning controls. It is tempting because audit findings often touch data integrity, but missing prompt versions affect model behaviour reproducibility, not the underlying database's structural integrity.

Increasing the size of the marketing team

Why it fails: Marketing headcount does not establish accountability, oversight, or risk controls, so it cannot satisfy governance requirements such as defined roles, policies, and audit mechanisms. It is tempting because AI governance needs organisational buy-in and communication, but those are outcomes of governance structures, not a critical framework component.

Choosing the most expensive cloud platform

Why it fails: Cost does not deliver the policies, risk assessments, or oversight bodies that AI governance requires; an expensive platform can still lack the necessary controls. It is tempting because procurement decisions feel consequential and budget signals commitment, but governance frameworks are defined by accountability and process, not by the price of the underlying cloud service.

Custom prediction routines

Why it fails: Used for preprocessing, not monitoring skew.

Artifact metadata locks

Why it fails: Prevents accidental overwrites.

Pipeline concurrency limits

Why it fails: Controls resource usage, not skew.

Database corruption

Why it fails: Feature skew is a mismatch between feature values or distributions at training and serving time, producing degraded predictions. Database corruption is a storage integrity fault, so it would be the concern when diagnosing damaged pages or failed writes, not pipeline skew.

Increased hardware costs

Why it fails: Feature skew concerns inconsistent feature computation or distributions between training and inference, not compute expenditure. Increased hardware costs would be the relevant risk when scaling infrastructure for larger models or throughput, which is unrelated to skew.

System crash

Why it fails: Feature skew is a training/serving mismatch: features computed differently in the pipeline than at inference, degrading predictions. A system crash is an availability failure, not a statistical divergence. It tempts because pipeline faults can cause outages, but skew manifests as silent accuracy loss, so crash detection would not surface it.

Reducing training time

Why it fails: Reducing training time alters nothing about inference-time inputs, so adversarial perturbations still reach the model. It is tempting because shorter training is a genuine efficiency lever for compute cost and iteration speed, but adversarial robustness requires input validation, perturbation detection or adversarial training, not less training.

Increasing model depth

Why it fails: Deepening the network adds parameters and non-linearity, which increases the attack surface for gradient-based perturbations rather than filtering them; adversarial robustness needs input sanitisation, adversarial training or certified defences. Depth is tempting because it raises clean accuracy on complex tasks, where extra capacity genuinely helps.

Ignoring the input

Why it fails: Discarding the input removes the attack vector but also prevents any prediction, so the model cannot serve legitimate requests. Rejecting or sanitising suspicious inputs is correct when the requirement is filtering adversarial examples while still returning valid inferences.

Throughput (TPS)

Why it fails: Throughput counts requests served per second, revealing nothing about whether outputs are grounded in retrieved context. It is tempting because throughput monitoring is genuinely useful for capacity planning and autoscaling decisions, where it would be the right metric.

Token generation latency

Why it fails: Latency measures response speed, not factual accuracy; a fluent, fast answer can still be ungrounded. It is tempting because latency monitoring is genuinely useful for detecting performance degradation and capacity problems, where it would be the right metric.

Error code distribution

Why it fails: Error codes capture transport and API failures, not semantically plausible but ungrounded content, which returns HTTP 200. It is tempting because error distribution monitoring is genuinely useful for detecting service faults and integration breakage, where it would be the right metric.

Memory footprint

Why it fails: Memory footprint tracks resource consumption, not whether generated claims are supported by source context. It is tempting because memory monitoring is genuinely useful for detecting leaks and sizing inference hosts, where it would be the right metric.

Vertex AI Model Monitoring for data drift

Why it fails: This monitors drift, not PII leakage during inference.

Vertex AI Feature Store access logs

Why it fails: This would log features but does not provide a control to prevent PII logging during inference.

Cloud IAM role binding for Vertex AI

Why it fails: IAM controls access permissions but does not scrub content from inference logs.