Select Two (Multi-Select) Questions

A line chart of the moving average of total training loss across epochs

Why it fails: A moving average of total loss tracks overall optimization progress but aggregates away per-example information. It cannot identify which individual training examples produce large gradient updates, so it does not answer the team's specific diagnostic question.

A pie chart showing the proportion of examples in each loss decile

Why it fails: A pie chart of loss deciles summarizes the distribution of losses but discards gradient information entirely. Even if high-loss examples are overrepresented, the chart cannot show which examples have unusually large gradient norms, making it unsuitable for the stated task.

A heatmap of the model's attention weights for a single randomly chosen example

Why it fails: Attention weights for one random example describe internal model behavior on that example, not the distribution of gradient norms across the training set. It provides no ranking or comparison that would help the team locate unusually influential examples.

The ambient temperature of the datacenter

Why it fails: While hardware performance might fluctuate slightly with temperature, it is not a factor that researchers can or should control for experiment reproducibility. Focusing on irrelevant physical parameters distracts from the necessary software-level controls, such as code versions, dependency management, and random seed setting, which are the real drivers of reproducibility.

The number of hours the researcher works

Why it fails: The researcher's working hours are irrelevant to the technical reproducibility of the experiment. Reproducibility is a function of the code, data, and environment settings. Managing the team's time is a management task that has no relationship to the deterministic nature of the training and validation processes in AI.

The learning rate used during fine-tuning

Why it fails: Learning rates are typically tuned per model size because different architectures have unique optimal convergence rates. Enforcing the exact same learning rate across vastly different model sizes would be detrimental, as the smaller model might underfit while the larger one might diverge, leading to an unfair experimental comparison.

The hardware GPU generation (e.g., A100 vs H100)

Why it fails: Hardware generation influences throughput and latency, but it does not change the model's generative logic or classification accuracy. As long as the precision is consistent, the model's output should remain identical regardless of the underlying GPU architecture, making this factor unnecessary to control for accuracy metrics.

The color scheme of the monitoring dashboard

Why it fails: The visual design of a dashboard is entirely irrelevant to the cost-benefit analysis of an AI experiment. While usability matters, it does not impact the training outcome or the compute costs. Focusing on trivialities like UI elements distracts from the core metrics that actually define the success of an experiment.

The popularity of the LLM framework used

Why it fails: The popularity of a framework does not indicate its performance or cost-effectiveness for a specific task. While community support is useful, the decision should be based on technical capability and resource efficiency. Relying on popularity metrics ignores the specific requirements of the project and the actual compute-to-accuracy ratio.

The number of research papers published by the team

Why it fails: The number of publications is a measure of academic output, not the operational success of an AI project. In enterprise experimentation, success is measured by the reliability and effectiveness of the deployed model, not by the volume of literature produced, making this an irrelevant metric for cost-benefit analysis.

The number of hidden layers in the model

Why it fails: While architecture size matters, the number of layers is a technical detail, not a top-level selection criterion for business applications. Performance on the target task, total cost of ownership, and ease of deployment are far more important than the specific depth of the model's architecture itself.

The specific programming language used to build the model

Why it fails: Whether a model was built in PyTorch, TensorFlow, or JAX is largely irrelevant as long as it can be deployed within the enterprise's inference stack (e.g., NVIDIA Triton). Modern deployment containers abstract these differences. The underlying framework code is less important than model performance, cost, and compliance.

Average token generation speed

Why it fails: While important for production latency, token generation speed does not measure the quality or accuracy of the generative output. This is an operational performance metric rather than an experimentation metric for model behavior or prompt effectiveness, and focusing on it ignores the goal of improving response quality.

Number of model parameters

Why it fails: The number of parameters is a static architectural property of the model, not a metric for evaluating prompting strategy effectiveness. It does not fluctuate during prompt engineering experiments and provides no feedback on how well the model is interpreting the retrieved context or user intent.

Total GPU memory consumption

Why it fails: GPU memory consumption is a hardware and infrastructure constraint. While vital for system stability, it is irrelevant for evaluating the quality or logic of the text generated by the model. Relying on this during prompt experimentation fails to inform the developer about the semantic correctness of the output.

Model quantization bit-width

Why it fails: Quantization affects inference speed and memory usage, not the retrieval accuracy of the RAG system. While it is important for deployment, it does not change the semantic search results or the relevance of the retrieved documents, making it a secondary concern when tuning for core retrieval accuracy performance.

System prompt length

Why it fails: System prompt length is a design choice related to instruction following, not document retrieval. While it affects how the model interprets retrieved context, it does not influence the search process itself. It should be tuned separately from the retrieval pipeline to optimize the model's overall reasoning and output quality.

GPU clock speed

Why it fails: GPU clock speed is a hardware configuration parameter that impacts the speed of computation. It does not affect the logical accuracy of the RAG retrieval process. Focusing on hardware settings during the retrieval experimentation phase misdirects efforts away from the algorithmic improvements that actually impact the quality of results.

Deleting logs to conserve disk space

Why it fails: Deleting logs destroys the history of your experiments, making it impossible to perform comparative analysis. Experiment tracking relies on maintaining a long-term record of successes and failures. In a production environment, this audit trail is also necessary for debugging issues that arise after a model has been deployed.

Manually calculating gradients during training

Why it fails: Calculating gradients manually is not part of an experiment tracking workflow; it is an error-prone task handled by the deep learning framework itself. Tracking should focus on metadata, metrics, and configurations, not the low-level mathematical implementation details that the framework handles during the automatic differentiation process.

Sharing the entire training dataset publicly to increase model transparency.

Why it fails: Publicly sharing training datasets containing sensitive information is a major security violation. Transparency should be achieved through metadata and documentation rather than exposing raw, sensitive user data. Proper data governance requires keeping training sets secure and protected from unauthorized access at all times.

Increasing the model parameter count to improve internal data encryption.

Why it fails: Increasing parameter count does not provide encryption or security. In fact, larger models may be more prone to memorizing sensitive training data. Security must be managed through external controls, data hygiene, and policy-based guardrails, not through the architectural size or complexity of the underlying LLM itself.

Vary multiple architectural hyperparameters simultaneously in every single experimental iteration.

Why it fails: Varying several hyperparameters at once confounds the results, because no single change can be attributed to the observed effect. Ablation isolates one variable per run while holding others fixed. Simultaneous sweeping suits coarse hyperparameter search, where interaction effects are the goal, not causal attribution.

Discard all experimental runs that fail to meet performance expectations without logging the failure.

Why it fails: Discarding failed runs without logging them introduces survivorship bias and destroys reproducibility, since the negative results are exactly what the ablation is measuring. Logging every run, including failures, is required. Discarding is tempting to keep result tables clean, but that suits reporting, not scientific validity.

Automating all processes to eliminate the potential for human error.

Why it fails: Eliminating human oversight is often dangerous in complex or high-stakes environments. AI models can exhibit unpredictable behavior in edge cases, and removing humans entirely creates a single point of failure. Human judgment is necessary for handling nuances and unexpected scenarios that the AI has not been trained to manage.

Restricting human access to the model's internal weights and architecture.

Why it fails: Human oversight should focus on the output and the reasoning process, not necessarily the raw weights. Restricting access to internal documentation often hinders the ability of human reviewers to understand why a model is failing. Transparency and access to information are key components of effective oversight for Trustworthy AI.

Word count distribution analysis.

Why it fails: Word count distribution measures the length of the output but provides no insight into the semantic correctness or factuality of the content. While useful for evaluating formatting requirements, it is irrelevant for detecting hallucinations, as a long, descriptive answer can be just as factually incorrect as a short one.

Model training loss convergence tracking.

Why it fails: Training loss measures how well the model predicts the next token on the training dataset, not the factual accuracy of the generative output. A model can have low training loss while still hallucinating during inference, so this metric is ineffective for identifying factual errors in generative AI applications.

The names of the engineers who reviewed the training logs.

Why it fails: Reviewer identity is an audit-trail detail, not a factor that shapes the trained weights. Recording it does not help another person recreate the model, because the outcome depends on data, configuration, and randomness rather than on who inspected the logs. This item is not part of the essential reproducibility metadata for a fine-tuning run.

The wall-clock duration of each training epoch.

Why it fails: Epoch duration depends on hardware, cluster load, and batch scheduling, and it does not influence the learned parameters. Two runs with identical seeds, data, and hyperparameters can have very different epoch times yet produce equivalent models. Capturing this metric is useful for performance analysis but is not essential for reproducing the model itself.

The GPU model and driver version used during training.

Why it fails: Hardware and driver details can affect numerical determinism at the margins, but they are not the primary determinants of the learned result. Identical configuration on different but compatible hardware usually yields a functionally equivalent model. For the essential reproducibility metadata set, the seed and dataset version carry far more weight than the specific GPU model.

Retrain the model after quantization so the weights adapt to the lower precision.

Why it fails: Quantization-aware training or fine-tuning is a separate technique, not a requirement for a valid inference comparison. The experiment measures the effect of quantization as applied, so retraining would change the object of study. It also adds cost and time without being necessary for the comparison.

Measure latency only on the first generated token, since subsequent tokens are less affected by precision.

Why it fails: First-token latency reflects prefill and prompt processing, while later tokens reflect decoding, and both are affected by precision. Restricting measurement to one phase would miss the effect on overall throughput and quality. A valid comparison covers the full generation path.

Apply the same random seed to both configurations so token sampling is identical.

Why it fails: A shared seed does not make FP16 and INT8 outputs identical because the numerical representations differ. Seeding is useful for reproducibility within a configuration, but it cannot remove precision-induced differences. Requiring it here misunderstands the source of variation being studied.

Deploy the resulting checkpoints to NVIDIA Triton Inference Server for latency benchmarking.

Why it fails: Triton deployment measures inference latency and throughput after training, which is unrelated to reproducing the fine-tuning comparison. Serving benchmarks do not capture training seeds, dataset versions, or container tags, so they cannot help another engineer recreate the experiment. This activity belongs to a deployment evaluation phase, not the reproducibility of the training study.

Increase the number of GPUs in every run so that throughput is maximized.

Why it fails: Maximizing GPU count changes the effective global batch size and gradient accumulation behavior, which can alter convergence independently of the learning-rate schedule. It is an optimization choice, not a reproducibility requirement, and applying it inconsistently across runs would confound the comparison rather than make the experiments repeatable.

Enable mixed precision training to reduce memory usage on each DGX node.

Why it fails: Mixed precision affects numerical behavior and memory footprint but is a performance and efficiency setting, not a guarantee of reproducibility. If one run uses FP16 and another uses FP32, results may diverge for reasons unrelated to the learning-rate schedule. Consistency of precision matters, but enabling it is not itself a required reproducibility practice.

Log the loss curve to TensorBoard so the team can visually compare runs.

Why it fails: Visualization helps humans spot differences but does not change the underlying stochasticity of training. The runs would still diverge numerically; the team would simply see the divergence more clearly. Logging is an observability improvement, not a reproducibility fix.

Increase the number of GPUs per run so each experiment finishes faster.

Why it fails: Adding GPUs changes the effective global batch size and the reduction order in distributed training, which can introduce additional numerical variance rather than reduce it. Faster completion does not make results more reproducible. Without adjusting batch size and seeds, this change could actually worsen the inconsistency the team is observing.

Reduce the learning rate by a factor of ten across all sweep configurations.

Why it fails: Lowering the learning rate changes the optimization trajectory but does not eliminate the stochastic sources causing identical configs to diverge. It may even mask the problem while making the sweep less informative about the intended hyperparameter range. This is a hyperparameter change, not a reproducibility control.

Allow each trial to choose its own prompt template so the model can show its best behavior.

Why it fails: Letting trials use different prompt templates introduces a second variable that can easily dominate the effect of temperature and top-p. A favorable template might make one configuration look better even if the decoding setting is not the cause. Controlled experiments require identical prompts so the sampling parameters are the only differences.

Increase the model's parameter count for trials with higher temperature to compensate for randomness.

Why it fails: Changing the model between trials destroys comparability because model capability becomes a confound alongside temperature. Temperature affects the shape of the sampling distribution, not the model's knowledge, so compensating with more parameters is conceptually wrong. A sweep over decoding parameters must keep the underlying model fixed across all trials.

Disable logging of per-trial parameters to reduce storage overhead during the sweep.

Why it fails: Omitting parameter logs makes it impossible to reconstruct which configuration produced which output, undermining reproducibility and auditability. Storage savings are trivial compared with the cost of an uninterpretable sweep. Recording temperature, top-p, seed, and prompt identifiers for every trial is essential for later analysis and replication.

Tune each recipe's learning rate separately until its score is maximized.

Why it fails: Giving each recipe its own extensive tuning budget confounds the comparison: differences may reflect unequal tuning effort rather than the recipe design. A defensible comparison either fixes the tuning protocol across recipes or reports the search budget used for each so the reviewer can judge fairness.

Reuse the evaluation set during training as an additional validation signal.

Why it fails: Using the evaluation set during training contaminates it, so its scores no longer estimate performance on unseen data. The comparison would then reward recipes that adapted most to those specific examples, which is exactly the leakage that undermines the credibility the team is trying to establish.

Report only the single best run for each recipe to keep the summary concise.

Why it fails: Reporting only the best run hides run-to-run variance and overstates expected performance. A reviewer cannot tell whether a recipe is reliably better or simply benefited from a favorable random draw. Summarizing across seeds with mean and spread gives a far more honest and defensible picture.

Vary the temperature for each template to explore a wider range of outputs.

Why it fails: Varying temperature alongside the prompt template introduces a second independent variable. Differences in output quality could then be due to sampling randomness rather than the prompt. To isolate the template effect, decoding parameters such as temperature and top-p must remain fixed across conditions.

Fine-tune the LLM separately for each prompt template before comparing them.

Why it fails: Fine-tuning separately for each template changes the model weights, so any performance difference could come from the fine-tuning run rather than the prompt. This confounds the experiment. If fine-tuning is part of the study, it should be a separate factor, not mixed into a prompt-template comparison.

Use a different evaluation metric for each template to capture their unique strengths.

Why it fails: Different metrics for each template make the results incomparable. A fair experiment applies the same metric to both conditions. If multiple metrics are of interest, they should be computed for both templates. Using different metrics would bias the comparison and prevent a valid conclusion about which prompt performs better.

Evaluate each prompt variant on a different dataset tailored to its strengths

Why it fails: Tailoring datasets to each variant guarantees that scores reflect dataset differences rather than prompt quality. A variant evaluated on questions aligned with its phrasing will look better regardless of true factuality. Controlled experiments require a common evaluation set so that every condition is measured under identical conditions.

Increase the temperature for variants that produce shorter answers to equalize response length

Why it fails: Adjusting temperature per variant deliberately changes the decoding distribution, which directly affects factuality and variability. It also confounds the comparison because temperature is a known driver of hallucination. Equalizing length by altering sampling is a manipulation, not a control, and it undermines the goal of attributing differences to prompt phrasing alone.

Allow the retrieval index to be rebuilt with different embedding models for each prompt variant

Why it fails: Changing the embedding model changes which passages are retrieved, so differences in factuality could stem from better or worse evidence rather than from prompt phrasing. This introduces a confounding variable. To isolate the prompt's effect, the retrieval component must remain identical, including the index, embeddings, and retrieval parameters.