Hard Difficulty Questions

AWS CloudHSM

Why it fails: CloudHSM provides dedicated hardware security modules for key storage and cryptographic operations, not detection or masking of PHI within text. It is tempting because HIPAA compliance demands strong key custody, and CloudHSM is correct where regulatory mandates single-tenant FIPS 140-2 Level 3 key protection.

Amazon Macie

Why it fails: Macie discovers and classifies sensitive data such as PHI in S3, alerting on exposure; it does not redact or mask identified elements before inference. It is tempting because Macie is the correct choice when the requirement is to find where PHI resides and flag unsecured buckets, rather than transform it.

AWS Key Management Service (AWS KMS)

Why it fails: KMS manages encryption keys, not de-identification.

Increase the confidence threshold to 99%

Why it fails: Raising the threshold reduces false positives but may also reduce true positives – it does not improve the underlying model.

Submit a request to AWS to retrain the base Rekognition model

Why it fails: AWS does not retrain base models on customer demand; the service provides Custom Labels for this purpose.

Use Amazon SageMaker to build a new object detection model from scratch

Why it fails: Building a new model from scratch with Amazon SageMaker would require curating a labelled dataset, training, and tuning a custom architecture, which is disproportionate effort when the existing Rekognition model only needs its confidence threshold adjusted for a single category. This option is tempting because SageMaker is the correct choice when no pre-trained API exists for the required object class, and full control over model architecture and training data is necessary.

Increase the number of retrieved documents

Why it fails: Retrieving more documents adds context but does not force the model to rely solely on it, so hallucination remains possible. It is tempting because richer retrieval often improves answer quality, and it would be correct when relevant documents are being missed by the retriever.

Use a larger context window model

Why it fails: A larger context window changes how much text fits, not whether the model is constrained to it, so unsupported answers persist. It is tempting because more retrieved context can improve grounding, and it would help when documents are being truncated before reaching the model.

Adjust the temperature parameter to 0

Why it fails: Temperature 0 makes sampling deterministic but does not restrict generation to the retrieved context, so unsupported content can still appear. It is tempting because low temperature reduces randomness, and it would be the right choice when you need reproducible, consistent outputs.

Use distributed training across multiple GPUs

Why it fails: Distributed training across multiple GPUs shortens wall-clock training time but splits the workload without preventing catastrophic forgetting of the model's original knowledge. It is the right technique when training duration alone is the constraint, not when preserving prior knowledge matters.

Use prompt engineering instead of fine-tuning

Why it fails: Prompt engineering leaves model weights untouched, so it cannot adapt the model to the new dataset's task, which is the stated goal of fine-tuning. It is tempting because it avoids training entirely, and it would be correct when the base model already handles the task and only output steering is needed.

Full fine-tuning on the new dataset

Why it fails: Full fine-tuning updates every parameter, which maximises training time and risks catastrophic forgetting of the model's original knowledge — the two things the team wants to avoid. It is tempting because it can yield the highest task accuracy, and it would be correct when ample compute is available and preserving prior knowledge is irrelevant.

Increase the maximum depth of trees

Why it fails: Increasing depth tends to increase overfitting, widening the gap.

Increase the number of features

Why it fails: Adding more features can increase overfitting if they are noisy.

Increase the subsample ratio to 1.0

Why it fails: Using all data each iteration (subsample=1.0) can increase overfitting; stochastic gradient boosting uses lower subsample to reduce overfitting.

Aggregation bias; mitigate by using a single model for all groups

Why it fails: Aggregation bias arises when one model is fitted across heterogeneous subgroups, not when a dataset contains a single gender. Using a single model for all groups is tempting as a fairness approach, but the stem describes sampling skew, which requires rebalancing or augmenting the training data instead.

Measurement bias; mitigate by using more precise measurement tools

Why it fails: Measurement bias concerns distorted or inaccurate feature values, whereas the stem describes a training set containing only one gender. More precise measurement tools are tempting because they address noisy labels, but they cannot introduce the missing gender group; resampling or augmenting the dataset is required.

Historical bias; mitigate by removing sensitive attributes from the model

Why it fails: Removing sensitive attributes does not correct a training set drawn from one gender; the label correlations remain skewed, so predictions stay biased. Historical bias is tempting because it names skewed sampling, yet the effective mitigation is rebalancing or augmenting the dataset, since proxy variables still encode gender.

Decrease the batch size

Why it fails: Smaller batch sizes can slow convergence and increase training time.

Use a larger instance type with more vCPUs

Why it fails: Larger instances cost more per hour; while training time may decrease, cost often increases.

Increase the number of hyperparameter tuning jobs

Why it fails: More tuning jobs increase total compute time and cost.

Increase the model's precision by raising the classification threshold

Why it fails: Raising the threshold would likely decrease recall, increasing false negatives and cost.

Use a different algorithm that trades off recall for precision

Why it fails: Trading recall for precision would hurt the goal of reducing false negatives.

Add more features to the model without changing the threshold

Why it fails: Adding features may or may not improve recall; it is not a direct lever and might not reduce false negatives efficiently.

Use Amazon Bedrock API

Why it fails: Amazon Bedrock is a fully managed service that runs inference on AWS-operated infrastructure, so it cannot satisfy the requirement to host the model on the company's own hardware. It would be the right choice when the team prefers managed APIs over self-hosted deployment.

Use Amazon Comprehend

Why it fails: Amazon Comprehend performs NLP tasks such as entity, sentiment and keyphrase detection; it neither generates code nor hosts large language models. Comprehend would be the right choice for extracting insights from text, not for self-hosted generative inference.

Use Amazon Lex

Why it fails: Amazon Lex builds conversational chatbots using intents and slots; it does not execute large language models for code generation. Lex would be correct for voice or text customer-service bots, not for running generative inference on the company's own infrastructure.

Switch to a larger model with more parameters

Why it fails: Larger models may have stronger pre-training knowledge and could be even more prone to ignoring context.

Increase the number of retrieved documents in the context window

Why it fails: More documents may confuse the model further; the issue is the model ignoring context, not lack of context.

Decrease the top-p value to 0.1 to restrict token selection

Why it fails: Top-p controls cumulative probability; while it can reduce diversity, lowering temperature is the more direct fix for adherence to context.

Label encoding

Why it fails: Label encoding assigns each of the 50,000 zip codes an arbitrary integer, implying a false ordinal relationship and producing a sparse, meaningless numeric feature. It is intended for low-cardinality or genuinely ordered categories, not high-cardinality nominal identifiers where target or frequency encoding is used.

Ordinal encoding

Why it fails: Ordinal encoding imposes a rank order on zip codes that does not exist, so the model treats 90210 as greater than 10001, distorting distance calculations. It is correct for genuinely ordered categories such as education level or severity ratings, not for nominal high-cardinality identifiers.

One-hot encoding

Why it fails: One-hot encoding creates 50,000 binary columns, causing extreme dimensionality, memory blow-up and sparse features that degrade training. It works well for low-cardinality nominal variables, but for high-cardinality identifiers target encoding, hashing or embeddings are used instead.

Use SageMaker Clarify to generate SHAP values and adjust the model's feature importance by removing biased features.

Why it fails: SHAP values explain predictions but do not automatically adjust the model; feature removal may not be optimal.

Use SMOTE (Synthetic Minority Oversampling Technique) to balance the training dataset before retraining.

Why it fails: SMOTE addresses class imbalance, but the issue here is fairness in precision, not just class imbalance; also synthetic data may not guarantee fairness.

Use SageMaker Model Monitor to detect feature drift and automatically retrain the model with updated data.

Why it fails: Model Monitor detects drift but does not mitigate bias; retraining with same imbalance may not fix bias.

Apply chain-of-thought prompting to encourage reasoning

Why it fails: Chain-of-thought prompting improves multi-step reasoning accuracy but supplies no brand guidelines or product facts, so off-brand wording and factual errors persist. It is tempting because it genuinely helps arithmetic and logic tasks, where reasoning transparency rather than grounding in source material is the actual problem.

Use zero-shot prompting without any examples

Why it fails: Zero-shot prompting provides no examples or reference material, leaving the model to invent brand voice and product details, which is precisely what causes the errors. It is tempting for speed on simple, well-specified tasks, but here the model needs grounding in approved content.

Decrease the temperature to 0.0 to eliminate randomness

Why it fails: Lowering temperature reduces randomness in word selection but does not inject brand guidelines or correct product facts; a confident wrong statement remains wrong. Temperature tuning is the right control when output variety or creativity is the problem, not when accuracy and brand alignment are.

The model was not pre-trained on enough data; use a larger base model

Why it fails: Pre-training size is not the issue; overfitting is more likely.

The training data was imbalanced; collect more diverse data

Why it fails: Data imbalance may cause bias but not necessarily high confidence in wrong answers.

The inference temperature is too low; increase it

Why it fails: Low temperature increases determinism but does not cause high confidence in wrong answers.

Decrease temperature and decrease top-p

Why it fails: Lowering temperature sharpens the probability distribution and lowering top-p truncates it to fewer tokens, both pushing output toward the highest-probability, most repetitive phrasing. It is tempting because these settings reduce hallucination, and they would be correct for factual, deterministic responses.

Decrease temperature and increase top-p

Why it fails: Lowering temperature alone narrows sampling toward likely tokens, so repetition persists regardless of raising top-p, which merely widens the candidate pool. It is tempting because the two parameters are often tuned together, and it would suit tasks needing mild variation with high factual reliability.

Increase temperature and decrease top-p

Why it fails: Raising temperature increases randomness, but lowering top-p narrows the sampling pool to fewer tokens, cancelling that diversity and often worsening repetition. Top-p is intended to truncate improbable tokens for coherence, so it suits factual or deterministic tasks where you want controlled, focused output.

{"Effect":"Allow","Action":"bedrock:GetFoundationModel","Resource":"*"}

Why it fails: GetFoundationModel only retrieves model metadata; invoking Claude Instant requires the InvokeModel action on the model's ARN. It is tempting because discovering which foundation models exist is a genuine prerequisite task, and this statement would be correct for a role that only lists available models.

{"Effect":"Allow","Action":"bedrock:InvokeModel","Resource":"arn:aws:bedrock:us-east-1::foundation-model/*"}

Why it fails: The resource ARN omits the region-specific inference profile or model identifier required for Claude Instant invocation, so the call still fails authorisation. It is tempting because InvokeModel is the right action, and this statement would work for models addressable directly by foundation-model ARN.

{"Effect":"Allow","Action":"bedrock:*","Resource":"*"}

Why it fails: Granting bedrock:* on * violates least privilege and still may not match the inference-profile ARN the invocation targets. It is tempting because wildcards resolve many permission errors quickly, and this would be acceptable for a temporary administrative role rather than a scoped application role.

Deploy the model without any content filters to maximize creativity

Why it fails: Removing content filters lets the model emit harmful, unsafe or non-compliant output, directly violating responsible AI requirements. Filters are tempting to disable when they block legitimate creative prompts, yet they are the correct choice precisely for open-ended creative tools where misuse and harmful generation must be constrained.

Increase model size to improve accuracy at the expense of interpretability

Why it fails: Scaling parameters trades transparency for marginal accuracy, and interpretability is a responsible AI requirement, not an optional sacrifice. Larger models are the right choice when raw predictive accuracy on a benchmark is the sole objective and explainability is not mandated by regulation or stakeholder need.

Use only synthetic data for training to avoid privacy issues

Why it fails: Synthetic-only training discards real-world distribution, producing models that generalise poorly and still risk leakage if generation is flawed. Synthetic data is genuinely useful for augmenting scarce or sensitive datasets, but responsible deployment requires provenance controls, consent and evaluation on representative real data.

AWS WAF

Why it fails: AWS WAF filters HTTP requests at the edge against IP reputation and web exploit signatures; it cannot parse model completions for PII or investment-advice topics. It is tempting as a content control, but it belongs in front of web traffic, not inside Bedrock's guardrail evaluation.

Amazon Macie

Why it fails: Macie discovers and classifies sensitive data held in S3 buckets; it cannot inspect Bedrock prompt or completion traffic, so it never sees generated PII or topics. It is tempting because it does detect credit card and Social Security numbers, but only at rest in storage, not in model responses.

AWS Identity and Access Management (IAM) policies

Why it fails: IAM policies govern which principals may invoke models and resources, not what the model outputs; they cannot detect generated credit card numbers or block investment-advice topics. They are tempting because they enforce access control, which is the right tool for restricting who calls Bedrock, not for content filtering.

AWS CloudTrail

Why it fails: CloudTrail records API activity for audit and compliance, but it does not version models, detect data or model drift, or retire them. It is the correct service when the requirement is capturing account-level API calls and event history, not managing the model lifecycle itself.

SageMaker Role Manager

Why it fails: SageMaker Role Manager builds IAM roles and permissions for users, so it grants access rather than versioning, monitoring or decommissioning models. It is the correct feature when defining persona-based permissions for SageMaker domain users, not for governing model artefacts and drift.

Topic denial

Why it fails: Topic denial blocks prohibited subjects, not ungrounded responses.

Content filtering

Why it fails: Content filtering blocks harmful content, not factual accuracy.

PII detection

Why it fails: PII detection identifies personal data, not factual correctness.