Implementing AI Solutions practice questions

Train a custom model from scratch on the policy documents each month

Why it fails: Training from scratch each month demands labelled data, compute and ML expertise, and cannot keep pace with monthly policy edits. It is tempting because bespoke training suits stable domains with abundant proprietary data, but here retrieval-augmented generation over an updated index avoids retraining entirely.

Fine-tune a base LLM on the policy documents monthly

Why it fails: Fine-tuning bakes policy text into weights, so each monthly update still requires a fresh training run and redeployment, exactly the recurring cost the team cannot afford. It is tempting because fine-tuning adapts tone and task behaviour, which suits stable formats rather than frequently changing factual content.

Use a larger foundation model with a longer context window and paste all documents into each prompt

Why it fails: Pasting every policy document into each prompt exhausts the context window, inflates latency and cost per query, and cannot scale as the corpus grows. It is tempting because long-context models handle small, static document sets well, but retrieval of only relevant passages satisfies the monthly-update requirement.

Random chunking with varying sizes

Why it fails: Random chunking would likely corrupt logical units.

Hierarchical chunking combining small and large chunks

Why it fails: Hierarchical chunking helps with different levels of detail but may still split cross-references if not semantically aware.

Fixed-size chunking with 256 tokens

Why it fails: Fixed-size chunking may split sentences and cross-references, breaking context.

BLEU score

Why it fails: BLEU measures n-gram overlap with reference text, so syntactically invalid code sharing tokens with correct answers still scores highly, failing to isolate the syntax problem. It is tempting because BLEU is standard for machine translation and summarisation, where surface phrasing overlap is the intended quality signal.

Perplexity

Why it fails: Perplexity measures token-level likelihood, not whether generated code parses or compiles, so it cannot capture syntax failures. It is tempting because it is a common language-model metric, but it would be correct when assessing fluency or comparing language models on next-token prediction.

Exact match accuracy

Why it fails: Exact match accuracy demands the whole output match a reference string, so it cannot quantify syntactically broken yet plausible code. It is tempting because it is a standard code-generation metric, but it would be correct when outputs are short and a single canonical answer exists.

Deploy a synchronous REST API with auto-scaling

Why it fails: Synchronous APIs can still cause timeouts if processing takes long; they block the caller.

Stream responses directly from the model to the client

Why it fails: Streaming may help latency but does not handle buffering during spikes.

Pre-compute all possible outputs and cache them

Why it fails: Pre-computation is impractical for dynamic multi-modal inputs.

Logging all input data for manual review

Why it fails: Manual review is not scalable and does not provide automated drift detection.

Monitoring prediction confidence scores over time

Why it fails: Confidence scores may not reliably indicate drift; a model can be confident yet wrong.

Retraining the model weekly as a routine

Why it fails: Retraining is an action, not a monitoring strategy.

LoRA (Low-Rank Adaptation)

Why it fails: LoRA freezes the base weights and trains small low-rank matrices, so optimiser state stays tiny and fits 8GB. It is tempting because it is the standard parameter-efficient method, but the stem asks which technique is most feasible, and LoRA is precisely that technique.

Instruction tuning with a smaller model

Why it fails: Swapping to a smaller model changes the base model rather than the fine-tuning technique, so it does not answer how to fine-tune the given 7B model within 8GB. It is tempting when quality targets are modest, but the stem fixes the model at 7B parameters.

Full fine-tuning of all parameters

Why it fails: Full fine-tuning updates every weight and stores optimiser states for all 7B parameters, exceeding 8GB of VRAM by a wide margin. It is tempting because it can yield the highest task accuracy when compute is unconstrained, but here memory, not quality, is the binding limit.

Fine-tune a base LLM on the policy documents monthly

Why it fails: Fine-tuning bakes the policy content into model weights, so every monthly document update requires another training run, which the stem rules out. It is tempting for teaching a model a fixed style or domain, but retrieval-augmented generation retrieves current documents at query time instead.

Train a custom model from scratch on the policy documents each month

Why it fails: Training from scratch monthly is prohibitively expensive and slow, and the stem explicitly rules out retraining each time documents change. It is tempting when a domain's vocabulary is genuinely unique, but retrieval-augmented generation over the documents satisfies monthly updates without any model training.

Use a larger foundation model with a longer context window and paste all documents into each prompt

Why it fails: Pasting all policy documents into each prompt would quickly exceed the context window’s token limit as the document set grows, causing truncation and loss of relevant information. This approach is tempting because large foundation models excel at processing extensive single-turn inputs, making it ideal for tasks like analysing a single long legal contract where all content fits within the window.

Apply random undersampling of the majority class until the dataset is balanced

Why it fails: Undersampling discards many legitimate transactions, losing valuable information and reducing model generalization.

Remove all duplicate transactions from the dataset

Why it fails: Deduplication does not address class imbalance; fraud examples are rare and duplicates may be legitimate.

Use the raw dataset without any resampling, relying on class weights during training

Why it fails: Class weights help but raw imbalance can still cause the model to favor the majority class, harming recall.

Perform data cleaning and normalization on the production data before inference

Why it fails: Data cleaning is part of data preparation, not real-time monitoring.

Periodically retrain the model with newly labeled data

Why it fails: Retraining is a response to monitoring findings, not a monitoring step itself.

Automatically roll back to the previous model version if a metric drops below a threshold

Why it fails: Rollback is an action triggered by monitoring, not a monitoring step.

Fine-tune a base LLM on the policy documents monthly

Why it fails: Monthly fine-tuning still requires retraining whenever documents change, which the team explicitly cannot afford, and it bakes policy content into weights rather than retrieving current text. Fine-tuning suits teaching a model a fixed style or domain behaviour, not frequently updated factual documents.

Train a custom model from scratch on the policy documents each month

Why it fails: Training from scratch monthly discards the pretrained weights and requires labelled data plus compute each cycle, so it cannot track monthly document changes affordably. It is tempting because custom training tailors a model to domain content, but retrieval-augmented generation over an indexed document store updates answers by reindexing, with no retraining.

Use a larger foundation model with a longer context window and paste all documents into each prompt

Why it fails: Pasting every policy document into each prompt consumes the context window and scales poorly as the corpus grows, and it cannot guarantee retrieval of the relevant passage. Long-context prompting suits small, stable document sets, not a monthly-updated policy corpus.

The production data has a different distribution than the training data (concept drift)

Why it fails: Concept drift typically occurs over time, not immediately upon deployment; the question describes poor performance right away.

The model's hyperparameters were not properly tuned

Why it fails: Improper tuning usually leads to lower evaluation scores, not high ones, so this does not explain the discrepancy.

The model is overfitting to the training data

Why it fails: Overfitting would typically show a large gap between training and validation scores, but here the test score is high, suggesting leakage or distribution shift.

Unit tests for the data pipeline

Why it fails: Unit tests verify individual code components, not the model's ability to classify new categories.

Integration tests for API calls

Why it fails: Integration tests ensure APIs work together, but they do not evaluate model performance on new data.

Evaluation framework for LLM output quality

Why it fails: LLM evaluation frameworks are for text generation quality, not for classification accuracy on new categories.

Use LoRA (Low-Rank Adaptation) with the base model in full precision

Why it fails: Keeping the base model in full precision retains fp32 weights and activations, so memory stays high despite LoRA's small trainable adapters. LoRA with a quantised base (QLoRA) is the memory-saving configuration; full-precision LoRA suits ample-VRAM setups where numerical fidelity outweighs footprint.

Use PEFT (Parameter-Efficient Fine-Tuning) without specifying a specific method

Why it fails: PEFT names a family of methods, not a concrete configuration, so it does not itself specify the low-rank adapter mechanism the scenario requires; the team must select LoRA explicitly. Generic PEFT guidance is correct when comparing adapter families or when any parameter-efficient approach satisfies the memory constraint.

Full fine-tuning of the entire model

Why it fails: Full fine-tuning updates every weight and stores full gradients, optimiser states and activations, so memory scales with total parameters and exceeds a limited GPU budget. It is the right choice when maximum task accuracy matters and abundant multi-GPU capacity is available, not when memory must be minimised.

Use a general-purpose embedding model like text-embedding-ada-002

Why it fails: While it may work, a domain-specific embedding model fine-tuned on technical manuals would likely yield better retrieval accuracy.

Use dot product as the similarity metric for non-normalized embeddings

Why it fails: Dot product can be used but is less common; if embeddings are normalized, cosine similarity is equivalent but more interpretable.

Use Euclidean distance as the similarity metric for vector search

Why it fails: Euclidean distance is sensitive to vector magnitude; embeddings are often normalized, making cosine similarity more effective.

Regression testing only after model updates

Why it fails: Regression testing is important but typically performed after changes; the question asks for testing types to include in the plan, and the three selected (A, C, D) are core to a comprehensive test strategy.

Load testing to determine maximum concurrent users

Why it fails: Load testing is for performance and scalability, not for functional correctness or output quality.

Chain-of-thought prompting with step-by-step reasoning

Why it fails: Chain-of-thought is more suited for reasoning tasks; for code generation, providing examples of correct code is more direct.

System prompt instructing the model to output JSON

Why it fails: JSON mode structures output but does not ensure syntactic correctness of the code itself.

Instruction fine-tuning on a large Python corpus

Why it fails: Fine-tuning is effective but requires extensive resources and data; prompt engineering is a lighter-weight solution.

Roll back the model to the previous version and ignore the new sensors

Why it fails: Ignoring new sensors defeats the purpose of monitoring the whole plant; it's not a long-term solution.

Increase the anomaly threshold to reduce false positives

Why it fails: Increasing the threshold may reduce false positives but also reduce true positives, missing real anomalies.

Use an ensemble of the old and a new model without retraining

Why it fails: Ensembling without retraining on new data does not address the distribution shift; the old model still fails on new sensors.

Fine-tune a base LLM on the policy documents monthly

Why it fails: Fine-tuning bakes the documents into model weights, so monthly policy changes force a fresh training run each cycle, exactly the cost the team must avoid. It is tempting because fine-tuning genuinely suits adapting a model's tone, format or task behaviour, not injecting frequently changing factual content.

Train a custom model from scratch on the policy documents each month

Why it fails: Training from scratch demands large labelled datasets, substantial compute and ML expertise, and rebuilding the model every month directly contradicts the no-retraining constraint. It is tempting because bespoke pretraining genuinely suits organisations building domain-specific foundation models where no existing LLM fits.

Use a larger foundation model with a longer context window and paste all documents into each prompt

Why it fails: Pasting every policy document into each prompt consumes the context window and scales poorly as the corpus grows, while monthly updates still require re-supplying the whole set. It is tempting because long-context prompting genuinely suits small, stable document sets queried occasionally without any indexing infrastructure.

Delete all normal transactions until the dataset is balanced

Why it fails: Deleting normal transactions discards 99.9% of legitimate behaviour, so the model never learns the true decision boundary and will flag ordinary activity as fraud. Deletion is tempting because it balances classes instantly, and suits scenarios where the majority class is genuinely noisy or mislabelled.

Duplicate the fraudulent transactions 1,000 times

Why it fails: Duplicating fraud cases 1,000 times creates exact copies, so the model memorises those specific transactions rather than learning general fraud patterns, and validation leaks. Duplication is tempting as a quick rebalance, and is acceptable only with varied synthetic generation, not verbatim copies.

Train the model on the original dataset; class imbalance does not affect model performance

Why it fails: Training on the raw 1000:1 ratio lets the model minimise loss by predicting the majority class, yielding high accuracy but near-zero fraud recall. Leaving data untouched is defensible when classes are already balanced or when the algorithm natively handles skew.