Experimentation practice questions

Increase the temperature to 1.5

Why it fails: Increasing the temperature to 1.5 makes the model more creative and stochastic, which significantly increases the risk of hallucinations. For grounding tasks, you generally want to decrease the temperature to ensure the model behaves deterministically and sticks strictly to the provided input context rather than generating arbitrary information.

Run the model on a larger GPU cluster

Why it fails: More compute resources do not improve the logical reasoning or factual accuracy of the model. Scaling the infrastructure is irrelevant to solving the problem of model hallucinations. The solution must come from improving the prompt or the context provided to the model during the experimentation phase.

Enable more training epochs

Why it fails: Training for more epochs typically leads to overfitting, which worsens the tendency to hallucinate. The issue of factual inaccuracy usually stems from the model's inability to prioritize retrieved context over its internal weights. Further training without changing the data strategy will not solve the grounding problem.

Increase the learning rate

Why it fails: Increasing the learning rate usually exacerbates overfitting, as the model takes larger steps that can lead to aggressive memorization of training samples. It does not provide the regularization needed to penalize complexity. Proper mitigation of overfitting requires methods that constrain the model's weight updates, not increase their speed.

Add more layers to the network

Why it fails: Adding more layers increases the model's capacity, which makes it even more prone to overfitting. If a model is already struggling with overfitting, making it deeper or wider will only provide more parameters for the model to memorize the training data rather than learning generalized language patterns.

Switch to a larger dataset

Why it fails: While having more data is generally good, it is often not an immediate option in an ongoing experiment. If the dataset size is fixed, you must adjust the model parameters or regularization techniques first. Changing the dataset is a structural change that falls outside the scope of hyperparameter tuning.

Only evaluate prompts on a small, synthetic dataset.

Why it fails: Using a small or purely synthetic dataset risks over-fitting the prompts to specific data patterns that may not exist in the real world. A representative and diverse validation set is necessary to ensure the results generalize well to actual user inputs and complex, unpredictable queries.

Change the underlying model architecture every five trials.

Why it fails: Changing the model architecture mid-experiment introduces a confounding variable, making it impossible to perform a fair comparison of prompt performance. The model must remain constant throughout the experimentation phase to ensure that any observed changes in quality or accuracy are exclusively attributed to the prompt engineering.

Use the longest possible prompt for every test case.

Why it fails: Prompt length does not inherently correlate with response quality and can increase latency and cost. Testing only the longest possible prompts fails to evaluate the efficiency and robustness of more concise prompts, leading to biased results that prioritize verbosity over effective, high-quality communication and model interaction.

Exhaustive Grid Search for all possible parameter combinations.

Why it fails: Grid search is computationally prohibitive for deep learning models with many hyperparameters. It evaluates every combination regardless of performance, leading to massive waste of compute resources. For LLMs, the parameter space is too large for this method to be viable in any practical experimentation scenario.

Randomly assign hyperparameters without tracking results.

Why it fails: Randomly assigning parameters without tracking is equivalent to guessing. Without logging performance data and iteratively narrowing the search space, the researcher cannot learn from the experiments, resulting in a disorganized, non-reproducible process that provides no guarantee of finding an optimal or even a functional model configuration.

The job will proceed with a warning regarding sequence length.

Why it fails: The policy explicitly forbids configuration overrides. If the infrastructure is governed by this policy, the system will not allow the job to initiate in a non-compliant state. The job must be adjusted to meet the policy requirements before it can successfully start on the cluster.

The system will automatically downscale the sequence length.

Why it fails: Systems usually do not perform automatic, silent modifications to user configurations to prevent unintended side effects on the model's mathematical behavior. Instead, the job will typically fail or be blocked by the policy engine, requiring the user to explicitly correct their configuration to meet the defined constraints.

The job will run but with downgraded performance.

Why it fails: If the policy is strictly enforced, the job will not run at all. It is not a matter of degraded performance but of outright denial of access to the cluster's compute resources. The user must resolve the policy conflict before the experiment can be executed in the environment.

Convert all text to lowercase to ensure consistency.

Why it fails: Converting all text to lowercase removes important semantic cues, such as proper nouns and sentence beginnings. This can significantly reduce the quality of the model's language understanding. LLMs are generally trained to handle casing, so preserving it is important for maintaining the nuances of natural language.

Tokenize the data using a random, unverified tokenizer.

Why it fails: The tokenizer must match the specific model architecture being fine-tuned. Using an unverified or random tokenizer will lead to incorrect token IDs, causing the model to receive nonsense input. This renders the entire experiment invalid, as the model cannot learn to map tokens to meaningful linguistic patterns.

Decrease the number of workers in the data loader.

Why it fails: Reducing the number of workers in the data loader will likely worsen the bottleneck by making the data supply even slower. The goal should be to increase data ingestion throughput, not decrease it, so that the GPUs are constantly fed with the data required for computation.

Reduce the number of GPUs used for the experiment.

Why it fails: Reducing the number of GPUs does not solve the underlying efficiency problem; it simply reduces the total amount of hardware being underutilized. The researcher should aim to maximize the utility of the existing cluster capacity before deciding to downsize the distributed training job.

Enable CPU-based offloading for all model layers.

Why it fails: Offloading to the CPU is a technique used to resolve OOM issues when model weights exceed VRAM, but it significantly slows down training because the CPU-to-GPU interconnect is much slower than GPU memory. This would further decrease performance and efficiency, not improve it, for an underutilized system.

Increase the gradient clipping value to 5.0.

Why it fails: Increasing the gradient clipping threshold allows for larger updates, which will likely exacerbate the current numerical instability. If the experiment is already producing NaNs at 1.0, relaxing the constraint will allow even larger, more destructive updates to occur, worsening the divergence of the model's loss function.

Switch the precision to FP32.

Why it fails: While FP32 offers more headroom, the issue is likely rooted in the optimization dynamics (learning rate or weight initialization), not just the precision type. Switching to FP32 increases memory usage significantly, which might trigger OOM errors before the stability issue is resolved, making this a secondary strategy at best.

Change the optimizer to SGD.

Why it fails: AdamW is typically more stable than SGD for training large models due to its adaptive learning rate properties. Switching to SGD, which lacks these adaptive mechanisms, would likely lead to slower convergence or even worse instability, as SGD is less forgiving of poorly tuned hyperparameters in large language model architectures.

The learning rate schedule itself.

Why it fails: The learning rate schedule is the independent variable being tested. Keeping it constant would defeat the purpose of the experiment, as no comparison could be made. The engineer must vary the schedule across runs to evaluate its impact on convergence, so this factor should not be held constant.

The evaluation metrics and validation set.

Why it fails: Evaluation metrics and validation set should be consistent to measure convergence fairly, but they are not the primary factors that directly affect the training dynamics. The experiment's goal is to compare learning rate schedules, so the training setup must be controlled. While keeping evaluation consistent is good practice, it is not one of the two most critical factors to hold constant for valid comparison of schedules.

The number of epochs and batch size.

Why it fails: While these are important hyperparameters, they are not the focus of this experiment. However, they should typically be held constant to avoid confounding effects. The question asks for factors that must be held constant; the number of epochs and batch size are indeed factors to control, but they are not the independent variable. Yet, the stem specifies 'which TWO factors should be held constant' and the correct answers are architecture/weights and dataset/order, as they are fundamental to isolating the learning rate schedule. Changing epochs or batch size would also confound, but they are less directly tied to the schedule's effect on convergence. The two most critical are architecture/weights and dataset/order.

Increase `model.optim.weight_decay` to 1.0 so the rising validation loss is penalized.

Why it fails: Weight decay regularizes parameters during optimization, but an extreme value of 1.0 would crush the weights and harm both training and validation performance. Even a moderate increase only slows overfitting; it does not automatically identify or retain the checkpoint with the lowest validation loss. The scenario asks for automatic retention of the best checkpoint, which weight decay alone cannot provide.

Set `model.optim.lr = 0.0` after epoch 3 so the weights freeze at their best point.

Why it fails: Zeroing the learning rate freezes weight updates but does not restore or save an earlier checkpoint. The model file already written at the end of epoch 3 would remain the reference artifact, and validation loss would simply plateau rather than improve. It also gives no mechanism to compare epochs or select the lowest validation-loss state, which is the actual goal here.

Reduce `trainer.max_epochs` to 3 so training stops before validation loss rises.

Why it fails: Shortening the run to three epochs prevents observing the later degradation, but the final checkpoint from epoch 3 is not guaranteed to be the best one. Validation loss could still be lower at an intermediate step within epoch 3. This is a manual, brittle workaround that discards training signal instead of using monitored checkpoint selection.

Increase the learning rate significantly

Why it fails: Increasing the learning rate does not impact memory consumption directly. In fact, it often leads to training instability or divergence, requiring more frequent checkpointing and longer experimentation cycles. Memory optimization requires structural changes to the computation graph rather than hyperparameter adjustments that affect the optimization convergence behavior.

Reduce the batch size to one

Why it fails: Reducing batch size to one is an extreme measure that destabilizes gradient estimates. While it lowers memory usage, it degrades the statistical quality of the updates and significantly slows down convergence. Gradient checkpointing is a more robust solution that maintains training efficiency while managing resource constraints effectively.

Switch to a smaller model architecture

Why it fails: Switching to a smaller model changes the fundamental scope of the experiment. If the objective is to fine-tune a specific model, changing the architecture invalidates the experiment's goal. Resource optimization should occur within the context of the chosen model before resorting to changing the underlying model size or complexity.

Inference latency

Why it fails: Inference latency measures speed, not quality. A very fast model that generates hallucinations or irrelevant content will result in poor user experience. While performance matters for product usability, quality metrics focus on the relevance, accuracy, and tone of the model output, which are the primary drivers of user satisfaction.

Tokens per second

Why it fails: Tokens per second is a throughput metric, not a quality metric. Optimizing for high throughput can sometimes lead to lower-quality outputs if the quantization or sampling parameters are overly aggressive. User satisfaction is derived from the meaningfulness of the content rather than the raw speed of the generation process.

GPU memory utilization

Why it fails: GPU memory utilization is an infrastructure metric used for capacity planning and performance optimization. It has no correlation with the quality of text generated by the model. Relying on resource metrics during A/B testing ignores the primary objective of understanding how well the model satisfies the end user's needs.

Model quantization bit-width

Why it fails: Quantization affects inference speed and memory usage, not the retrieval accuracy of the RAG system. While it is important for deployment, it does not change the semantic search results or the relevance of the retrieved documents, making it a secondary concern when tuning for core retrieval accuracy performance.

System prompt length

Why it fails: System prompt length is a design choice related to instruction following, not document retrieval. While it affects how the model interprets retrieved context, it does not influence the search process itself. It should be tuned separately from the retrieval pipeline to optimize the model's overall reasoning and output quality.

GPU clock speed

Why it fails: GPU clock speed is a hardware configuration parameter that impacts the speed of computation. It does not affect the logical accuracy of the RAG retrieval process. Focusing on hardware settings during the retrieval experimentation phase misdirects efforts away from the algorithmic improvements that actually impact the quality of results.

To increase the total number of training epochs

Why it fails: Mixing data is not about the epoch count; it is about maintaining the distribution of knowledge. Simply increasing epochs without original data would likely accelerate the degradation of the model's general abilities, as it would overfit exclusively to the new, smaller fine-tuning dataset while ignoring base-level general intelligence.

To reduce the computational time of fine-tuning

Why it fails: Including more data in the training set increases, rather than decreases, the total computational time. The motivation for data mixing is strictly about preserving model quality and general capabilities, not performance optimization. Efficient training is achieved through hardware acceleration and batching, not by omitting data to save time.

To improve the hardware utilization rates

Why it fails: Data mixing has no impact on hardware utilization. Utilization is dictated by the model size, batch size, and the underlying GPU architecture. While data management is part of the experimentation pipeline, the primary goal of mixing pre-training data is algorithmic preservation, not the optimization of compute utilization metrics.

Deleting logs to conserve disk space

Why it fails: Deleting logs destroys the history of your experiments, making it impossible to perform comparative analysis. Experiment tracking relies on maintaining a long-term record of successes and failures. In a production environment, this audit trail is also necessary for debugging issues that arise after a model has been deployed.

Manually calculating gradients during training

Why it fails: Calculating gradients manually is not part of an experiment tracking workflow; it is an error-prone task handled by the deep learning framework itself. Tracking should focus on metadata, metrics, and configurations, not the low-level mathematical implementation details that the framework handles during the automatic differentiation process.

Deploying the model to production

Why it fails: Deployment is the final stage of the lifecycle, occurring only after experimentation, validation, and testing are complete. Attempting to deploy without a successful experimentation phase leads to unpredictable model behavior, performance bottlenecks, and potential quality issues that could have been resolved through early testing and iterative refinement.

Purchasing new hardware for the team

Why it fails: Hardware procurement is a logistical task that supports AI work, but it is not a technical part of the experimentation phase itself. Experimentation is about the algorithmic and configuration-based refinement of the AI pipeline, not about the acquisition of the compute resources used during the training process.

Writing marketing copy for the product

Why it fails: Marketing is a business function, not a technical experimentation task. While the output of an LLM project may be used for marketing, the experimentation process is strictly concerned with the technical evaluation and optimization of the model's performance, safety, and reliability in a controlled, data-driven environment.

Change activation function to ReLU

Why it fails: Changing the activation function is a major architectural change that would likely degrade the model's performance on modern architectures like Llama. ReLU is generally less effective than SwiGLU for deep models. The current overfitting problem should be addressed through regularization, not by changing the fundamental activation layer design.

Change optimizer to SGD

Why it fails: AdamW is the standard optimizer for LLMs. Switching to SGD, which is less efficient for large-scale models, would likely lead to slower, less effective convergence. It would not address the generalization issue between the training and validation sets, making it an ineffective strategy for fixing the observed overfitting behavior.

Decrease weight decay

Why it fails: Decreasing weight decay would reduce regularization, which is the opposite of what is needed when the model is overfitting. If validation loss is rising, you need to increase, not decrease, the constraints on the model's weight growth. This action would only worsen the observed overfitting gap during the training.

The color scheme of the monitoring dashboard

Why it fails: The visual design of a dashboard is entirely irrelevant to the cost-benefit analysis of an AI experiment. While usability matters, it does not impact the training outcome or the compute costs. Focusing on trivialities like UI elements distracts from the core metrics that actually define the success of an experiment.

The popularity of the LLM framework used

Why it fails: The popularity of a framework does not indicate its performance or cost-effectiveness for a specific task. While community support is useful, the decision should be based on technical capability and resource efficiency. Relying on popularity metrics ignores the specific requirements of the project and the actual compute-to-accuracy ratio.

The number of research papers published by the team

Why it fails: The number of publications is a measure of academic output, not the operational success of an AI project. In enterprise experimentation, success is measured by the reliability and effectiveness of the deployed model, not by the volume of literature produced, making this an irrelevant metric for cost-benefit analysis.

Stopping training after a fixed number of days

Why it fails: Stopping training based on time is an arbitrary constraint that ignores the model's actual performance. A model might need more or less time depending on the learning rate and data. Stop-loss techniques should be dynamic and based on performance thresholds, not on a hard-coded time limit that ignores convergence status.

Increasing the learning rate when loss is high

Why it fails: This is a reactive optimization strategy, not a stop-loss technique. While it aims to improve convergence, it does not stop the experiment. Stop-loss is specifically concerned with the termination of ineffective experiments to preserve resources, rather than the manual adjustment of hyperparameters during the training process itself.

Saving a checkpoint every 100 iterations

Why it fails: Saving checkpoints is a fault-tolerance strategy, not a stop-loss technique. It allows for recovery in case of hardware failure or for comparing model states, but it does not proactively stop an experiment. These are distinct operational concepts, and confusing them can lead to significant waste of compute resources.

Power consumption versus disk space

Why it fails: While quantization reduces the size of the model on disk and slightly lowers power usage, these are secondary benefits. The primary technical risk of quantization is the potential loss of model accuracy or reasoning capability. Focusing on power or disk space ignores the most critical impact on the LLM application.

Training time versus model size

Why it fails: Quantization is typically applied to the model after training, so it does not directly affect training time. While the resulting model size is smaller, this is a storage consideration. The actual trade-off during inference is between performance speed and the logical output quality of the quantized, production-ready model.

GPU clock speed versus CPU utilization

Why it fails: Quantization does not change GPU clock speeds or CPU utilization in any direct, meaningful way. These metrics are tied to the underlying hardware execution and load balancing. The critical trade-off in quantization is algorithmic precision versus utility, which has nothing to do with the physical clock speed of the hardware.