# Reproducibility, Versioning, and Experiment Tracking

> Chapter 9 of the Courseiva GOOGLE-PMLE curriculum — https://courseiva.com/learn/google-pmle/reproducibility-and-versioning

**Official objective:** 3.2 — Manage experiments, models, and datasets for reproducibility and traceability.

## Introduction

Reproducibility, versioning, and experiment tracking are the safety net that prevents machine learning work from becoming a chaotic mess. They matter for the PMLE exam because Google expects you to know how to prove that a model's results are trustworthy and can be repeated by anyone, anywhere, at any time — a core requirement for production-grade machine learning systems.

## The Head Chef's Recipe Book Analogy

A head chef runs a busy professional kitchen. Every dish that leaves the pass must taste identical every time, whether the chef cooks it or a junior line cook does. To guarantee this, the chef keeps a meticulous, bound recipe book. Each recipe is a single, unambiguous version of a dish, with exact ingredient weights measured in grams, precise cooking temperatures, and step-by-step timings recorded in seconds. The book has a table of contents that lists every version of every dish ever created, with dates and handwritten notes about why a particular version was modified. If a new line cook decides to experiment by adding a pinch more salt to the bolognese, the chef does not allow anyone to scribble in the margins of the master book. Instead, the chef photocopies the current recipe, writes "Experiment 42: double salt" on the copy, and staples it into a separate section labelled "Trials and Revisions." The chef dates this copy and files it next to the original. Later, when the team tastes the saltier bolognese and decides it is better, the chef does not throw away the old recipe. The chef writes a new master recipe in the book, numbers it Version 2.0, and leaves the original Version 1.0 intact for reference. Every ingredient batch is labelled with a lot number so that if a tomato supplier sends a bad crate, the chef can trace which dishes were affected and which recipes used that batch. This system ensures that any cook can walk into the kitchen today, pick up the recipe book, and produce a plate identical to one served six months ago. The alternative is chaos: guesswork, inconsistent dishes, and no way to learn from past mistakes.

This is exactly what reproducibility, versioning, and experiment tracking mean for a machine learning engineer. The chef is the engineer. The recipe book is the version control system for code, data, and model configurations. The trials section is the experiment tracking log. The lot numbers on ingredients are the dataset hashes and environment snapshots. Every successful machine learning project depends on this same discipline: a way to lock down exactly what was done, with which ingredients, under which conditions, so that the results can be recreated on demand.

## Core explanation

Let's break this down from the ground up. You are building a machine learning model. You write code, you collect data, you train the model, you get a result. If you run the same code again tomorrow, will you get the exact same result? If the answer is no, your work is not reproducible. Reproducibility means that someone else, using the same code, the same data, and the same environment, can produce the exact same outcome. This is fundamental in science and engineering because it builds trust. Without it, you cannot tell if a model improvement came from a genuine insight or from random luck.

Versioning is the practice of assigning unique identifiers to every version of your code, your data, and your model. Think of it like saving a file not as "final_version_final_3.docx" but as a snapshot with a timestamp and a hash — a mathematical fingerprint that uniquely identifies that specific version. Every time you make a change, you create a new snapshot. You never overwrite the old one. This means you can always go back to a previous version if something breaks. In software development, we use tools like Git to version code. In machine learning, we also need to version data sets and trained model files because even a tiny change in the data — like one extra row — can produce a completely different model.

Experiment tracking is the process of logging every detail about a training run: the hyperparameters you used (things like learning rate or number of layers), the exact code commit hash, the dataset version, the training duration, the final accuracy, and any notes you wrote. This log lives in a centralised system, like a shared spreadsheet or a dedicated tool such as MLflow or Vertex AI Experiments. The goal is that when you look back two months later and see a model with 89% accuracy, you can instantly see exactly how that model was built, not just what the final number was.

Why does all this matter? Imagine you are at a company that builds a fraud detection model. One week, the model performs brilliantly. The next week, it suddenly performs terribly. Without reproducibility, versioning, and experiment tracking, you have no idea what changed. Did someone update a library? Did the data pipeline introduce a new field? Did a team member accidentally train with different hyperparameters? You would waste days or weeks trying to figure out the root cause. With these practices in place, you can compare the successful experiment's log with the failing one and spot the difference immediately.

What does this replace? Before these practices became standard, machine learning was done in a very ad-hoc way. Engineers would run scripts on their laptops, save models to random folders, and share results via email. If a model broke, nobody could reproduce the original result. This was fine for academic research where the focus was on novelty, but it is unacceptable in a business environment where models affect real people's money, health, or safety. The shift to systematic reproducibility, versioning, and experiment tracking is what allows machine learning to be treated as a reliable engineering discipline rather than a wild science experiment.

Here are the core components you need to know for the PMLE exam:

- A version control system for code, such as Git, records every change to your codebase, who made it, and why. Each change gets a unique commit hash.

- A dataset versioning tool, such as DVC (Data Version Control) or the dataset registry in Vertex AI, takes a snapshot of your data at a specific point in time. It stores the data itself in cloud storage but keeps a pointer and a hash so you can always retrieve the exact same data later.

- An experiment tracking system, like Vertex AI Experiments or MLflow, logs all the parameters, metrics, and artifacts from a training run. It organises these logs so you can compare runs side by side.

- A model registry is a central place where you store, version, and manage your trained models. Each model version has metadata attached: who trained it, on what data, with what code, and with what evaluation results.

- Environment reproducibility means locking down the versions of every software library, the operating system, and even the hardware configuration. Tools like Docker containers package the entire environment into a single image that can be run identically on any machine.

When you put all these pieces together, you create a chain of provenance. You can trace a deployed model all the way back to the raw data it was trained on, the exact version of the training script, and the experiment log that produced it. This traceability is what the exam calls "lineage." It is the backbone of trustworthy machine learning.

## Real-world context

Let's walk through a concrete scenario at a fictional company called "FinSecure," which builds a machine learning model to approve or decline small business loans. The model is already in production, but the team wants to improve its accuracy. A senior machine learning engineer named Priya is tasked with running experiments to find a better model. Here is exactly what she does, step by step.

First, Priya writes a new feature engineering function that extracts a new signal from the application data. She does not just save the new script over the old one. She commits it to a Git repository with a clear message: "Add cash-flow volatility feature." This commit gets a unique hash, like "a1b2c3d". The entire team can see this change, and if the new feature causes problems later, they can revert to the previous version.

Next, Priya needs to run her experiment. She opens the experiment tracking dashboard in Vertex AI Experiments. She creates a new experiment called "cash-flow-volatility-v1" and defines the parameters she wants to vary: the learning rate, the number of trees in the model, and the window size for calculating volatility. She logs these as parameters. She also notes the current Git commit hash and the dataset version she is using. The dataset is versioned via a separate data versioning tool — each time the raw loan data is updated, a new version number is assigned, and the old data is not overwritten.

Priya launches the training job. The system automatically captures the start time, the full environment (Python version, library versions), and the performance metrics every few minutes. When the job finishes, the system logs the final accuracy, precision, recall, and the model file itself. The model file is automatically saved to the model registry under a new version. The registry records that this model version was produced from experiment "cash-flow-volatility-v1" with Git hash "a1b2c3d" and dataset version "v15".

A week later, Priya tries a different experiment. She modifies the data preprocessing steps. She commits the change, notes the new commit hash, and runs another experiment. This second model performs worse. Without proper tracking, she might not know why. But with the logs, she can compare the two experiments side by side. She sees that in the second experiment, the data preprocessing accidentally removed a column that was critical for the previous model. She fixes the bug, commits the corrected code, and runs a third experiment that performs better than the first.

Now, the team decides to deploy the best model to production. The operations team does not simply grab any model file. They look at the model registry, select the version that corresponds to the successful experiment, and promote it to production. The registry automatically records the exact code, data, and environment that produced that model. Six months later, an auditor asks FinSecure to prove that their loan approval model does not discriminate. The team can use the lineage information to retrieve the exact training data, the exact model, and the exact evaluation metrics. They can rerun the evaluation on that exact model version with new test data to verify fairness. Without reproducibility, versioning, and experiment tracking, this audit would be impossible.

Priya also uses a practice called a "golden test set." She versions the test data separately and never uses it for training. When she wants to compare two experiments fairly, she always evaluates them on the same version of the test data. The experiment tracking system makes this transparent because it logs which test dataset version was used for each evaluation.

Finally, every month the team performs a "reproducibility bake-off." They pick a random past experiment from three months ago, read the full log, and try to rerun it from scratch — pulling the exact code version from Git, the exact data version from the data registry, and the exact environment from a Docker container. If they cannot reproduce the results within an acceptable margin of error, they treat it as a critical incident and fix the gaps in their pipeline. This is not an optional nice-to-have in a regulated industry; it is a requirement for passing compliance audits.

## Exam focus

The PMLE exam tests your understanding of reproducibility, versioning, and experiment tracking in several specific ways. First, expect scenario-based multiple-choice questions. The exam will describe a situation where a team has a problem — like a model that suddenly degrades in production — and you must choose which practice would have prevented or solved the problem. The answer pattern almost always involves pointing to a lack of versioning for data or a missing lineage trail. They love to set a trap where they describe a team that versions code but not data, and ask why they cannot reproduce an old result. The correct answer is always that the data version was not captured.

Second, you will face questions about experiment tracking tools and their features. The exam expects you to know that Vertex AI Experiments is the managed experiment tracking service on Google Cloud. You do not need to memorise every click in the UI, but you do need to understand what it does: log hyperparameters, metrics, and artifacts, and compare runs. A common trap question asks you to choose between Vertex AI Experiments and a simpler tool like a spreadsheet. The correct answer is the managed service because it provides automatic logging, lineage, and retrieval, which a spreadsheet cannot do at scale.

Third, the exam tests your knowledge of model registries. They want you to recognise that a model registry is the authoritative source for which model version is in production, and that it stores metadata such as training data version, evaluation metrics, and model lineage. A trap question might ask you to deploy a model by manually copying a file from an engineer's laptop. The correct answer will always involve using the model registry to promote the correct version.

Fourth, expect questions about environment reproducibility. The exam will test your understanding of containerisation with Docker. They might present a scenario where a model works on a data scientist's laptop but fails in production because of library version differences. The correct answer is to containerise the environment using a Docker image that includes all dependencies, and to version that image. A common trap is suggesting that you just install the libraries again in production — that is fragile because you cannot guarantee you will install the exact same versions.

Here are the specific concepts the exam loves to test:

- Lineage: the ability to trace a model back through every step, from raw data to deployed model. Understand that lineage includes code, data, environment, and hyperparameters.

- Hash-based versioning: the use of cryptographic hashes (like SHA-256) to fingerprint datasets and models. Knowing that even a single byte change in a file produces a completely different hash is crucial.

- Immutable versioning: the principle that once a version is created, it is never changed. If you need to correct an error, you create a new version. This is the opposite of overwriting files.

- Experiment comparison: the ability to view multiple experiment runs side by side to identify which parameters caused a change in performance. The exam might ask you to interpret a comparison table.

- Parameter logging versus metric logging: parameters are the inputs you control (learning rate, number of layers), and metrics are the outputs you measure (accuracy, loss). A question might confuse the two and ask what should be logged.

Trap pattern: a question describes a team that "saves every model to a shared drive with filenames like model_v3_final.pkl." The exam wants you to recognise that this is not real versioning because there is no link to the code and data that produced it. The correct answer will involve creating a proper experiment log that ties the model file to a specific commit and dataset hash.

Trap pattern: a question says, "an engineer modified the training script but did not commit the change to Git, then ran an experiment with good results. How can the team reproduce those results later?" The correct answer is that they cannot, because the code version was not captured. The trap answer is that they can look at the experiment log, but if the log does not include a Git hash, it is useless.

Definitions to memorise for the exam:

- Reproducibility: the ability to obtain the same results using the same code, data, and environment.
- Replicability: the ability to obtain consistent results across different experiments but using the same method. (This is a related but distinct concept; the exam focuses on reproducibility.)
- Versioning: the practice of assigning unique, immutable identifiers to each version of code, data, and models.
- Experiment tracking: the systematic logging of all parameters, metrics, artifacts, and environment details for each training run.
- Model registry: a central repository that stores, versions, and manages trained machine learning models, along with their metadata.
- Lineage: the complete history of a model's creation, including all data, code, and environment dependencies.

## Step by step

1. **Set Up Version Control for Code** — Initialise a Git repository for all your training scripts, configuration files, and any custom code. Every time you make a change, commit it with a descriptive message. This creates a permanent, timestamped record of what code was used for every experiment. Without this step, you cannot later prove which code produced a given model.
2. **Version Your Dataset with a Hash** — Before starting any training, take a snapshot of your dataset using a tool like DVC or the Vertex AI dataset service. The tool computes a cryptographic hash of the data file and stores the data in a reliable location like Cloud Storage. You record this hash in your experiment log. This step ensures that even if the raw data changes later, you can always retrieve the exact data used for a particular training run.
3. **Define and Log Experiment Parameters** — Create a structured record of all the hyperparameters and configuration choices for the experiment. Use an experiment tracking system like Vertex AI Experiments to log each parameter and its value. This includes the learning rate, batch size, number of layers, and any other settings. Logging these allows you to compare runs and identify which parameter changes affected performance.
4. **Run Training and Automatically Log Metrics** — Execute the training job in an environment that automatically captures performance metrics at regular intervals (loss, accuracy, precision). The tracking system records these metrics alongside the parameters, the code commit hash, the dataset hash, and the environment details. This creates a complete, auditable record of the entire training process in a single place.
5. **Save the Trained Model to a Model Registry** — After training finishes, push the model artifact to a model registry. The registry assigns a unique version number and stores metadata linking this model to its parent experiment, code commit, and dataset hash. This ensures that when you later want to deploy or audit this model, you can trace it back to exactly how it was built.
6. **Document the Experiment Outcome with Notes** — Add a human-readable note in the experiment tracking system explaining why you ran this experiment, what the results mean, and whether the model should be considered for deployment. This step is often overlooked but is critical for future team members — or your future self — to understand the reasoning behind decisions.
7. **Verify Reproducibility by Rerunning a Past Experiment** — Periodically, pick a past experiment from the log and attempt to reproduce it exactly. Pull the exact code version, data version, and environment container, and rerun the training. If the results differ significantly, investigate the cause and fix the gaps in your tracking pipeline. This step validates that your system actually works and catches untracked dependencies early.

## Comparisons

### Software Version Control (Git) vs Data Version Control (DVC)

**Software Version Control (Git):**
- Versions text-based code files efficiently using diffs
- Designed for collaboration on code with branching and merging
- Stores the entire history locally and on remote servers

**Data Version Control (DVC):**
- Versions large binary datasets using pointers and cloud storage
- Does not store data directly; stores metadata and hashes
- Primarily used to track dataset snapshots for machine learning pipelines

### Experiment Tracking (Vertex AI Experiments) vs Model Registry (Vertex AI Model Registry)

**Experiment Tracking (Vertex AI Experiments):**
- Logs parameters, metrics, and artifacts for each training run
- Allows comparison of multiple runs side by side
- Provides a searchable history of all experiments

**Model Registry (Vertex AI Model Registry):**
- Stores and versions trained model files
- Manages deployment lifecycle (staging, production, archived)
- Links each model version back to its experiment tracking entry

### Reproducibility vs Replicability

**Reproducibility:**
- Requires exact same code, data, and environment
- Goal is to produce identical results every time
- Essential for debugging and compliance in production ML

**Replicability:**
- Requires same method but can use different data or environment
- Goal is to achieve consistent results across different settings
- More common in academic research where exact replication is not always possible

### Immutable Versioning vs Mutable File Management

**Immutable Versioning:**
- Once created, a version is never modified or deleted
- Each version gets a unique hash that acts as a permanent identifier
- Provides a reliable audit trail because history cannot be altered

**Mutable File Management:**
- Files are overwritten or renamed, destroying history
- No permanent identifier; relies on filenames and folder locations
- Leads to confusion about which version is correct and what data was used

## Diagram

_This diagram shows how raw data, code, and experiment tracking inputs feed into a training job, producing logged metrics and a model artifact that flows to the model registry and deployment, with all components contributing to a unified audit trail._

```mermaid
flowchart TD
    A[Raw Data Source] --> B[Data Versioning Tool]
    B --> C[Versioned Dataset Hash]
    C --> D[Training Job]
    E[Code Repository Git] --> D
    F[Experiment Tracking System] --> D
    D --> G[Logged Metrics and Params]
    D --> H[Trained Model Artifact]
    H --> I[Model Registry]
    I --> J[Deployment to Production]
    B --> K[Dataset Lineage]
    E --> K
    F --> K
    G --> K
    I --> K
    K --> L[Audit Trail]
```

## Common misconceptions

- **Misconception:** If I save my Jupyter notebook with all the outputs, that counts as experiment tracking. **Reality:** Saving a notebook with outputs is not experiment tracking. Experiment tracking requires logging structured parameters and metrics in a searchable system that allows comparison across runs and ties to specific code and data versions. (Beginners often rely on notebooks because they are interactive and produce immediate visual outputs. But notebooks are fragile — they can accidentally execute cells out of order, and they do not capture the environment or data version automatically. The exam expects you to know that a notebook alone is insufficient for reproducibility.)
- **Misconception:** Versioning a dataset just means keeping a copy of the CSV file in a folder with a date in the filename. **Reality:** Proper dataset versioning uses a hash of the file contents to create a unique, immutable identifier, and stores the file in a reliable location with metadata about its origin. A date-based filename is unreliable because someone could overwrite the file, and two files with different contents could have the same date. (This misconception comes from how non-technical professionals manage files on their personal computers. In a team environment, filenames are not enforceable; a hashing system is. The exam tests this by presenting scenarios where a team used filename-based versioning and then could not tell which data version was used for a particular model.)
- **Misconception:** If I use the same random seed, my model training will always be perfectly reproducible on any machine. **Reality:** Setting a random seed helps but does not guarantee full reproducibility because different hardware (GPU versus CPU, different GPU architectures) and different library versions can introduce floating-point arithmetic differences that produce slightly different results. Full reproducibility requires controlling the entire environment, not just the seed. (Many tutorials highlight setting a random seed as the key to reproducibility, which leads beginners to believe it is sufficient. In reality, machine learning frameworks are not deterministic across different hardware, and the exam tests your understanding that environment reproducibility (via containers) is necessary.)
- **Misconception:** Experiment tracking is only needed for teams that are collaborating; a solo data scientist does not need it. **Reality:** Even a single engineer needs experiment tracking because memory is fallible. After a few weeks, you will not remember exactly which hyperparameters you used or why a particular experiment failed. Tracking saves time and prevents repeating mistakes, regardless of team size. (People who work alone often feel they have no need for formal processes because they can "just remember." This is a classic overconfidence bias. The exam sets questions where a lone engineer cannot reproduce a past result because they did not log the details, reinforcing that tracking is for everyone.)
- **Misconception:** The model registry and the experiment tracking system are the same thing, so I only need to use one of them. **Reality:** They are separate but complementary. An experiment tracking system logs the details of training runs (parameters, metrics, environment), while a model registry stores and versions the resulting model artifacts and manages their deployment lifecycle. The model registry should reference the experiment log, but they serve different functions. (The terms "registry" and "tracking" both involve storing information, so beginners lump them together. The exam distinguishes them by asking which system to use for a specific task: logging a new training run (experiment tracking) versus promoting a model to production (model registry).)

## Key takeaways

- Reproducibility requires capturing the exact code version, data version, library versions, and hardware configuration — not just the final model file.
- Versioning uses cryptographic hashes to create immutable snapshots of code, data, and models, enabling you to retrieve the exact state at any point in time.
- Experiment tracking logs hyperparameters, metrics, and artifacts for every training run, and allows side-by-side comparison to identify what caused a performance change.
- A model registry manages the lifecycle of trained models, tracking which version is in production, and linking it back to its training code and data.
- Lineage is the chain of provenance that connects raw data, feature engineering, model training, evaluation, and deployment into a traceable story.
- Containerising your environment with Docker is the only reliable way to ensure that training runs identically across different machines and over time.
- Never rely on memory, filenames, or spreadsheets for tracking — use purpose-built tools like Git, DVC, and Vertex AI Experiments for auditable reproducibility.
- For the PMLE exam, you must understand the difference between reproducibility (same inputs, same results) and replicability (same method, different inputs, consistent results) — the exam focuses on reproducibility.
- A common exam trap is a scenario where a team versions code but not data — the correct answer always identifies the missing data version as the root cause of irreproducibility.
- Immutable versioning is non-negotiable: once a version is created, never modify it; always create a new version to preserve the historical record.

## FAQ

**What is the difference between experiment tracking and a model registry?**

Experiment tracking logs the details of training runs, such as hyperparameters and metrics, while a model registry stores and versions the resulting model files and manages their deployment lifecycle. They are complementary: the model registry typically links back to the experiment tracking entry that produced the model.

**Do I need to version my data if it never changes?**

Yes, even if the data never changes, you still need to version it because you need to prove which data was used. Without a version, you cannot be certain that the model was trained on the expected data. A hash of the data file provides that proof and acts as a permanent reference.

**Can I use Git to version my datasets directly?**

Git is not designed for large binary files like datasets and will become slow and bloated. Use a data versioning tool like DVC (Data Version Control) or a cloud service like Vertex AI Datasets, which stores the data externally and uses pointers in Git to track versions.

**What is a lineage, and why does it matter for the exam?**

Lineage is the end-to-end history of a model: from raw data, through feature engineering, training, evaluation, to deployment. It matters because the PMLE exam tests your ability to trace a deployed model back to its exact training conditions, which is essential for debugging and compliance.

**What happens if I forget to log a hyperparameter?**

If you forget to log a hyperparameter, you lose the ability to reproduce that experiment precisely. You can still guess the value later, but you cannot prove it. The exam expects you to log all relevant parameters systematically, not just the ones you think are important at the time.

**Is it okay to use a simple spreadsheet for experiment tracking?**

For a single person working on a small project, a spreadsheet might work temporarily, but it is not a proper solution for production systems. Spreadsheets do not enforce structured fields, cannot automatically capture metrics, and do not link to code and data versions. The exam expects you to recognise that managed tools like Vertex AI Experiments are the correct answer for scalable, auditable tracking.

## Check your understanding

1. **A team uses Git to version their code but does not version their datasets. During an audit, they cannot prove which data was used to train a model that is now in production. Which practice did they neglect?**

   Answer: They neglected data versioning. Without a hash or snapshot of the dataset, they cannot link the model to the exact data it was trained on, breaking the chain of lineage.

2. **What is the primary purpose of a model registry in a machine learning workflow?**

   Answer: The model registry stores, versions, and manages the lifecycle of trained models, tracking which version is in production and linking it to its training code, data, and experiment log.

3. **Why does setting a random seed alone not guarantee full reproducibility of model training?**

   Answer: Because different hardware (like GPU versus CPU) and different library versions can introduce floating-point arithmetic differences that produce slightly different results, even with the same seed.

4. **What is the role of a cryptographic hash in dataset versioning?**

   Answer: A cryptographic hash creates a unique, fixed-length fingerprint of the entire dataset. It ensures that if even one byte changes, the hash changes, allowing you to verify that you are using the exact same data as a previous experiment.

5. **An engineer records hyperparameters and final accuracy in a spreadsheet after each training run, but does not record the code commit hash or dataset version. What is the biggest risk of this approach?**

   Answer: The biggest risk is that someone cannot reproduce the model because they do not know which code version and which dataset version produced that accuracy. The spreadsheet lacks the lineage information needed to link the results back to the inputs.

---

Interactive version with quiz and diagrams: https://courseiva.com/learn/google-pmle/reproducibility-and-versioning