Hard Difficulty Questions

Train the model with a loss function that weights profit more heavily than customer satisfaction.

Why it fails: Weighting profit above customer satisfaction in the loss function directly optimises the behaviour the marketing team is complaining about, worsening the dissatisfaction rather than aligning recommendations with preference. Such weighting suits scenarios where revenue maximisation is the sole objective and preference signals are deliberately subordinate.

Adjust the model's hyperparameters to reduce the influence of profit features.

Why it fails: Hyperparameters govern learning dynamics such as learning rate and regularisation strength, not the semantic weighting of individual input features like profit. Tuning them cannot rebalance profit against preference. Feature weighting belongs to feature engineering or the loss function, so this is the right lever only when the model architecture itself needs tuning.

Remove profit data from the training set and only use customer preference data.

Why it fails: Deleting profit data removes the profitability constraint entirely, so recommendations can no longer be balanced against margin and the stated requirement to maintain profitability fails. Preference-only training fits pure relevance or engagement ranking, where commercial return is not a modelling objective at all.

Provide a few-shot example of a valid JSON response in the prompt

Why it fails: A single few-shot example demonstrates format but does not guarantee every response parses as JSON; the model may still emit prose or malformed brackets. Few-shot prompting is the right choice when teaching a task pattern or style, not when downstream systems require guaranteed syntactic validity.

Include a system prompt that says 'You are a helpful coding assistant.'

Why it fails: A persona instruction sets tone and role, not output syntax; it cannot constrain token generation to valid JSON. It is tempting because system prompts do steer behaviour, and this one would be the right choice when the goal is shaping a general assistant's helpfulness rather than enforcing a machine-readable schema.

Use chain-of-thought prompting to have the model reason step-by-step before answering

Why it fails: Chain-of-thought elicits intermediate reasoning tokens, which typically appear in the output and break JSON parsing. It is the right choice when the task needs multi-step arithmetic or logic, not when the requirement is a strictly parseable structured payload.

Use class weights during training

Why it fails: Class weights rebalance the training loss, altering the learned model rather than the decision threshold applied at inference; the question asks which threshold adjustment to make. Class weighting is the right approach when retraining is possible and the training set is imbalanced.

Apply SMOTE to the training data

Why it fails: SMOTE synthesises minority-class samples during training, changing the fitted model instead of the inference-time threshold the question asks about. It is appropriate when retraining with an imbalanced dataset is acceptable and synthetic minority examples are wanted.

Increase classification threshold

Why it fails: Raising the threshold makes the model more conservative, predicting fewer positives and reducing recall further while increasing precision. Lowering the threshold is what captures more fraud. Increasing the threshold suits scenarios where false positives are the costly error, not false negatives.

Implement a simple Retrieval-Augmented Generation (RAG) pipeline

Why it fails: A RAG pipeline retrieves documents to ground generated text; it cannot select APIs, sequence calls or act on results. It is tempting because RAG is the standard pattern for knowledge-grounded answers, and would be correct where the task is answering from an internal corpus rather than executing multi-step tool workflows.

Fine-tune a model to output API call sequences directly

Why it fails: Fine-tuning bakes fixed API sequences into model weights, so it cannot adapt call order to novel requests or react to API responses mid-task. It suits stable, repetitive classification or extraction tasks with labelled examples, not dynamic multi-step orchestration where the model must choose tools at runtime.

Use function calling with a fixed sequence of API calls

Why it fails: A fixed call sequence cannot adapt to user requests, so the agent cannot choose which API to invoke or reorder steps. It is tempting because fixed chains are deterministic and easy to test, and would be correct where the workflow is invariant and every request follows identical steps.

The dataset is too small for the model complexity

Why it fails: Small data can cause overfitting, but the described workflow strongly indicates leakage as the primary issue.

The learning rate is too high

Why it fails: High learning rate would cause divergence, not large train-validation gap after convergence.

The model is overfitting due to high variance

Why it fails: Overfitting is a symptom; the root cause here is leakage from improper data splitting practices.

Ignore the disparity because the model is accurate overall

Why it fails: Ignoring disparity may violate anti-discrimination laws.

Remove all demographic attributes from the dataset

Why it fails: Removing attributes may not eliminate bias due to proxy variables, and can hinder fairness analysis.

Train the model using accuracy as the performance metric

Why it fails: Accuracy as a metric treats all correct predictions equally, so a model that classifies every transaction as legitimate achieves 99.9% accuracy without detecting any fraud. This fails because the 0.1% fraudulent class is the target of detection, and accuracy is blind to class distribution. It is tempting because accuracy is the default metric for balanced datasets, where misclassification costs are symmetric and each class is equally represented.

Undersample the legitimate transactions to match the number of fraudulent ones

Why it fails: Undersampling loses valuable data and may degrade model performance.

Increase the regularization strength in the model

Why it fails: Regularization does not address class imbalance.

Deploy stateless containers without session persistence

Why it fails: Stateless containers discard the preprocessing session state the stem requires to persist across requests, so horizontal scaling breaks continuity. It is tempting because stateless replicas scale cleanly behind a load balancer, and it would be correct if each request were fully self-contained with no session affinity needed.

Use a single larger GPU instance to handle all requests

Why it fails: One larger GPU instance raises per-node capacity but remains a single scaling unit, so it cannot scale horizontally and still caps concurrency. It is tempting because bigger hardware often lowers latency, and it would be correct for a workload that cannot be partitioned across replicas.

Implement a message queue (e.g., Kafka) to buffer requests

Why it fails: Kafka buffers and decouples request intake but does not scale the inference workers or preserve preprocessing session state across replicas. It is tempting because queues absorb traffic spikes, and it would be correct for smoothing bursty asynchronous workloads rather than maintaining per-session state during horizontal scaling.

Replace the convolutional layers with transformer layers to improve attention.

Why it fails: Changing architecture typically requires full retraining and may not address low-light specifically.

Use adversarial training to add imperceptible perturbations to training images.

Why it fails: Adversarial training improves resistance to malicious perturbations but not necessarily to natural low-light conditions.

Increase the model's depth by adding more convolutional layers.

Why it fails: Increasing depth changes architecture and requires retraining; it may not directly address low-light performance.

Full fine-tuning of all model parameters

Why it fails: Full fine-tuning updates every weight, so gradients and optimiser states for a large model far exceed limited GPU memory. It suits abundant data and ample hardware. With 500 examples, parameter-efficient methods such as LoRA train small adapters instead, avoiding that memory cost.

Instruction tuning with the full dataset

Why it fails: Instruction tuning still updates the full parameter set, so optimiser and gradient memory exceed the limited GPU. It suits large, diverse instruction datasets teaching general task-following. With only 500 examples, parameter-efficient fine-tuning such as LoRA adapts the model within memory.

Retrieval-Augmented Generation (RAG) without fine-tuning

Why it fails: RAG retrieves external context at inference but leaves model weights unchanged, so it cannot adapt behaviour to the labelled examples the task requires. It suits knowledge-intensive question answering over a document corpus, not domain-specific fine-tuning of a small labelled set under memory limits.

Data encryption at rest

Why it fails: Encryption at rest protects stored data from physical theft or unauthorised disk access, but training pipelines read decrypted records into memory, so it does nothing against re-identification, memorisation or inference leakage. It is tempting because it is a genuine privacy control for storage compliance, and would be correct where data at rest is the exposure.

Data replication

Why it fails: Replication copies data to additional stores, multiplying exposure points and retention obligations without removing or masking any personal identifiers. It tempts because replication supports availability and backup, and it would be correct for resilience or disaster recovery — but privacy requires anonymisation, consent management or access controls, not duplication.

Explicit consent from all data subjects

Why it fails: Consent is not always required; other legal bases may apply.

Data retention period of at least 10 years

Why it fails: Retention periods vary; no universal 10-year requirement.

Compare the distribution of predictions to the training set

Why it fails: Prediction distribution comparison can indicate shift but is less reliable than performance metrics.

Monitor the model's training loss

Why it fails: Training loss can stay low even if the underlying data distribution changes.

Retrain the model daily on new data

Why it fails: Retraining is a remedy, not a monitoring method for detecting drift.

Auto-scale the number of API endpoints

Why it fails: Auto-scaling increases capacity during high traffic, but it may not reduce per-request latency and can increase costs due to more resources.

Quantize the model to FP16

Why it fails: Quantization is a model optimization performed by the provider, not something the client can implement for a cloud-hosted API.

The vector database has low recall

Why it fails: Low recall would cause missing some relevant documents, but the generated answer might still be based on retrieved documents, not hallucination.

The temperature setting is too high

Why it fails: High temperature increases randomness but does not cause hallucination of facts not in the context; it still conditions on provided text.

The embedding model is not accurately representing document semantics

Why it fails: Poor embeddings could cause retrieval of irrelevant documents, but if retrieved documents are correct, hallucinations are due to other causes.

The policy allows access from a wide private IP range

Why it fails: While the policy allows a wide private IP range, it is still restricted to corporate networks, making this less significant than unlimited inference requests.

The policy does not require multi-factor authentication

Why it fails: Multi-factor authentication is explicitly required in the policy, so this is not a weakness.

The policy grants 'audit_log' access to data scientists

Why it fails: Granting audit_log access to data scientists is a minor access control issue but not as critical as model extraction risk.

Reduce the learning rate during training

Why it fails: Learning rate governs optimisation convergence, not the input distribution the model encounters; a lower rate cannot reconcile training and deployment data. Tuning it is tempting when training loss is unstable or the model overfits, but distribution shift requires detecting and correcting the mismatch, for example through drift monitoring or retraining on current data.

Add more features to the model

Why it fails: Adding features cannot correct a covariate or concept shift between training and deployment distributions; it risks overfitting the original sample further. Feature engineering is tempting because it raises accuracy when the training data genuinely lacks predictive signal, but here the test set already scores 98%, so the gap is distributional, not representational.

Increase the number of cross-validation folds

Why it fails: Cross-validation folds estimate model performance on held-out splits of the same training distribution, so they cannot correct a mismatch between training and deployment data. It is tempting because more folds appear to improve rigour, but cross-validation addresses variance in evaluation, not distribution shift.

Use a simpler model to reduce variability

Why it fails: A simpler model may underfit and not solve the drift issue.

Increase the frequency of model retraining

Why it fails: Retraining more often increases compute spend and reacts to noise rather than slow drift, worsening the problem described. Frequent retraining suits fast, high-impact distribution shifts, whereas gradual drift calls for smoothing the trigger, for example requiring sustained degradation before retraining.

Reduce the batch size for inference

Why it fails: Batch size governs inference throughput and memory, not when retraining fires, so it leaves the fluctuating accuracy trigger untouched. Reducing batch size helps fit large models into constrained memory during serving, not controlling retraining frequency.

Monolithic API with serverless functions

Why it fails: Serverless functions introduce cold-start delays and per-invocation overheads that undermine the sub-second response times a customer-facing real-time application demands. The pattern suits sporadic, event-driven workloads where elasticity and pay-per-use billing outweigh consistent latency. Continuous generative inference needs provisioned throughput, not scale-to-zero compute.

Edge deployment on user devices

Why it fails: Edge deployment runs inference on user devices, whose constrained compute cannot host large generative models, and distributing updates across heterogeneous hardware raises cost and versioning overhead. It tempts because on-device inference minimises network latency, and it would suit small models on controlled hardware.

Batch processing with synchronous requests

Why it fails: Batch processing accumulates requests and runs them on a schedule, so responses are not returned within the interactive window a customer-facing application demands. It tempts because batching maximises throughput per unit cost, and it would be correct for offline scoring such as overnight report generation.

Label encoding

Why it fails: Label encoding assigns arbitrary integers, implying order that misleads tree splits.

One-hot encoding

Why it fails: One-hot encoding creates high-dimensional sparse data, which can degrade tree performance.

Frequency encoding

Why it fails: Frequency encoding uses category counts; target encoding is more directly predictive.