Exam domain 3, "Designing and implementing ML pipelines," might sound like a daunting engineering task, but at its heart it solves a simple problem: how do you make your machine learning process repeatable, reliable, and automatic? Without a pipeline, every small change to your data or code forces you to manually re-run a long chain of steps, inviting errors and wasted time. Mastering Vertex AI Pipelines on Google Cloud is essential for the PMLE exam because it tests whether you can design the automation that turns messy, manual ML work into an industrial-strength system.
Jump to a section
A simple way to picture Building End-to-End ML Pipelines with Vertex AI Pipelines
Have you ever wanted to cook a healthy dinner every night, but found yourself ordering takeaway because chopping vegetables and washing dishes feels overwhelming?
That is exactly the problem that machine learning engineers face when building ML models. You do not just train a model once — you have to clean data, transform it, train the model, evaluate it, and deploy it, over and over again, every time new data arrives. Doing each step manually is like cooking each meal from scratch every single evening: chaotic, prone to mistakes, and exhausting.
Sunday meal prep is the solution. On Sunday, you wash all your vegetables, cook your rice, grill your chicken, and portion everything into containers. Throughout the week, you just grab a container and heat it up. You have turned a complex, multi-step process into a repeatable, automated pipeline.
Vertex AI Pipelines is exactly that for machine learning. It is a kitchen where you define each step of your ML process — getting data, cleaning it, training a model, testing it, deploying it — and then automate the whole sequence to run reliably every time. Just like meal prep saves your evenings, Vertex AI Pipelines saves data scientists from manual, repetitive work and ensures the final ML "meal" is consistent and high quality.
Let us break down what an end-to-end machine learning pipeline actually is. Imagine you have a recipe for baking a cake. The recipe has several steps: mix dry ingredients, mix wet ingredients, combine them, pour into a pan, bake, cool, and frost. If you do these steps manually every time, you might forget the baking powder, overmix the batter, or burn the cake. An automated pipeline is like a smart oven that follows your recipe automatically, checks each step for quality, and produces a perfect cake every time.
In the world of machine learning, the "cake" is a trained and deployed model that makes predictions. The "recipe" is a series of steps, which we call components. Each component does one specific job. Typical components in an ML pipeline include:
Data ingestion: pulling raw data from a storage bucket or database.
Data validation: checking that the data is not missing values or full of errors.
Data preprocessing: cleaning and transforming the data into a format the model can understand.
Model training: feeding the data into a learning algorithm to create a model.
Model evaluation: testing the model on new data to see how accurate it is.
Model deployment: putting the model into production so it can start making predictions.
Vertex AI Pipelines is a service on Google Cloud that lets you build and run these pipelines. It is built on two open-source frameworks: Kubeflow Pipelines and TensorFlow Extended (TFX). Kubeflow Pipelines is the orchestrator — think of it as the conductor of an orchestra, telling each musician when to play. TFX is a set of pre-built components designed specifically for TensorFlow models, like special instruments that already know how to play certain tunes. You are not forced to use TFX; you can write custom components using any Python code you like.
Why do we need this infrastructure? Before pipelines, data scientists often worked in notebooks (like Jupyter) on their laptops. They would write code to load data, clean it, train a model, and save it — all in one long, messy document. When they wanted to retrain the model with new data, they had to manually re-run the notebook, hoping nothing broke. If a colleague wanted to reproduce the results, they had to borrow the exact same laptop and notebook. It was fragile and unscalable.
A pipeline solves that by packaging each step into a container. A container is like a shipping crate that holds all the software and code needed for a single step. This crate can be moved to any computer — your laptop, a giant server in the cloud, or a cluster of machines — and it will run exactly the same way. Every step becomes a container, and the pipeline connects them, passing data from one container to the next.
Vertex AI Pipelines adds two powerful features on top of this basic idea. First, it automatically tracks what is called lineage — a complete record of where every piece of data came from, which code was used to process it, and which model it produced. This is crucial for auditing and compliance. Second, it integrates deeply with other Google Cloud services. For example, the data validation step can use Google Cloud's Dataflow service to process massive datasets, and the deployment step can use Vertex AI Endpoints to host your model.
For the PMLE exam, you need to understand both Kubeflow Pipelines and TFX. Kubeflow Pipelines is about the orchestration: defining the graph of steps, handling dependencies, and retrying failed steps. TFX is about the specific ML components: libraries like TensorFlow Data Validation (TFDV) for checking data quality, TensorFlow Transform (TFT) for preprocessing data at scale, and TensorFlow Model Analysis (TFMA) for evaluating model performance. The exam wants you to know which tool to use for which job and how to wire them together in a pipeline.
You will also need to understand how components communicate. In a pipeline, data is passed as Artifacts — typed references to data stored in cloud storage. For example, a dataset artifact points to a CSV file. A model artifact points to a saved model in a bucket. When one component finishes, it produces artifacts that the next component consumes. This loose coupling means you can swap out components without breaking the pipeline, as long as the artifact types match.
Finally, pipelines can be triggered to run automatically. You can schedule them to run every night when new data arrives, or trigger them when a new file appears in cloud storage. This automation is the end goal: the ML system runs itself, freeing humans to do more creative work.
Define the pipeline graph
You write a Python script using the Kubeflow Pipelines SDK. You define each step as a function decorated with @component. The function's inputs and outputs become the pipeline's data flow. For example, a 'train' component might take a dataset as input and produce a model as output.
Compile the pipeline
The Python definition is compiled into a JSON or YAML file that describes the entire graph, including how components connect and what resources they need. This file is the recipe that Vertex AI Pipelines will follow.
Upload the pipeline
You upload the compiled file to Vertex AI Pipelines via the Google Cloud console, the gcloud command line tool, or the Python client. This registers the pipeline so you can run it multiple times with different parameters.
Trigger a run
You start a run either manually, on a schedule, or in response to an event (like a new file landing in Cloud Storage). Each run is an independent execution of the pipeline, and its metadata is stored in the Vertex AI Metadata store for later analysis.
Monitor and handle failures
The pipeline executes step by step. If a step fails, Vertex AI Pipelines can retry it (if configured) or stop the run. You can inspect logs and outputs in the console. After fixing the issue, you re-run the pipeline, and previously successful steps are cached — they do not re-execute.
Promote the model
If the final step (often called Pusher in TFX or a custom deploy component) succeeds, the model is registered in Vertex AI Model Registry. From there, you can deploy it to an endpoint for online predictions or use it for batch predictions.
Consider a real company, let us call it ShopSmart, an online retailer that wants to recommend products to customers. They have a team of three data scientists who spend most of their time manually retraining a recommendation model every week. The process is painful. Bob downloads the latest sales data from a database and copies it to his laptop. Alice runs a Python script to clean the data and engineer features like "time since last purchase." Carol trains a model using a Jupyter notebook. Then Bob manually uploads the new model to a server and restarts the prediction service. If any step fails — say the database is slow or Alice's script has a bug — the whole process stalls, and the team spends hours debugging.
ShopSmart decides to adopt Vertex AI Pipelines. Here is exactly what happens step by step:
A data engineer writes a pipeline definition using the Kubeflow Pipelines SDK (a Python library). They define six components: ingest_data, validate_data, preprocess, train, evaluate, and deploy. Each component is a Python function decorated with @component, which tells the pipeline system that this function should run in its own container.
The team packages each component with its dependencies. For example, the train component includes a requirements.txt file specifying libraries like TensorFlow 2.10 and pandas. The pipeline system builds a Docker container image for each component automatically.
The pipeline is compiled into a JSON-like file called a pipeline job spec. This file describes the entire graph: ingest_data runs first, then validate_data and preprocess run in parallel, then train runs only after preprocess finishes, and so on.
A schedule is created using Google Cloud Scheduler to trigger the pipeline every Monday at 6 AM. The trigger sends a request to Vertex AI Pipelines, which creates a new pipeline run.
On Monday at 6 AM, the pipeline starts. Ingest_data pulls the latest sales data from BigQuery (Google's data warehouse) and stores a CSV file in Cloud Storage. Validate_data uses TensorFlow Data Validation to check for problems like missing values or distribution shifts. If the data looks bad — say sales dropped 90% due to a website outage — the pipeline can be configured to stop early and alert the team via email.
The preprocess component uses TensorFlow Transform to scale numerical features and convert categories into numeric IDs. This step is computationally heavy, so it runs on Google Cloud Dataflow, which automatically uses hundreds of machines to speed things up.
The train component trains a neural network on the preprocessed data using a GPU machine. It saves the trained model and its evaluation metrics as artifacts.
The evaluate component loads the model and runs it against a held-out test dataset. It computes metrics like precision and recall. If the new model is not better than the current one, the pipeline stops and does not deploy.
If the model passes evaluation, the deploy component uploads it to Vertex AI Model Registry and creates a new endpoint version. The new model starts serving recommendations to customers within minutes — all without any human intervention.
The result? ShopSmart's recommendation engine now updates itself every week reliably. The data scientists are free to experiment with new features and algorithms instead of fighting broken scripts. The company even sets up a second pipeline that retrains the model daily during the holiday season, automatically adjusting to changed shopping behaviour.
The PMLE exam tests your understanding of two frameworks: Kubeflow Pipelines (KFP) and TensorFlow Extended (TFX). You need to know the differences, when to use each, and how they integrate with Vertex AI. The exam will not ask you to write code line by line, but it will ask scenario-based questions that require you to choose the right tool or design pattern.
Here are the exact topics to focus on:
- Kubeflow Pipelines (KFP) concepts: You must understand components, pipelines, runs, experiments, and artifacts. A common question gives you a scenario with three steps and asks which one should be a separate component. The answer is any step that could change independently or that uses different compute resources. - TFX component library: The exam loves to ask which TFX component to use for specific jobs. Memorise these: * ExampleGen: ingests data and splits it into training and evaluation sets. * StatisticsGen: computes descriptive statistics of the data. * SchemaGen: infers the data schema (column types and ranges). * ExampleValidator: detects anomalies in data compared to the schema. * Transform: performs feature engineering at scale. * Trainer: trains the model. * Evaluator: evaluates the model performance. * Pusher: deploys the validated model to production. - Difference between TFX and KFP: TFX provides pre-built components for ML tasks. KFP is a generic orchestrator that can run any container. The exam might present a scenario where you need to run a custom Python script for data cleaning — the correct answer would be to write a custom KFP component, not to misuse a TFX component. - Conditional execution and branching: You need to know that pipelines can include conditional statements. For example, "if model accuracy > 0.95, then deploy; else, send a notification." The exam tests this by showing a pipeline DAG and asking what happens under certain conditions. - Caching and usage tracking: Vertex AI Pipelines caches component outputs if the component code and inputs have not changed. This means re-running a pipeline after a small change only re-executes the changed steps. The exam might ask how to optimise cost or speed — the answer is to take advantage of caching by making components granular and deterministic. - Pipeline parameters: A pipeline can accept parameters, like the learning rate or the location of the input data. The exam tests your understanding of how to pass these parameters and override them at runtime.
Common traps on the exam:
Confusing TFX's Transform with Trainer. Transform is for feature engineering, Trainer is for model training. A question might describe a step that normalises numeric features — that is Transform, not Trainer.
Thinking that a pipeline must use TFX. It does not. You can build a KFP pipeline entirely with custom Python components. The exam will test whether you know when to use TFX (when you want pre-built, production-tested components) versus custom code (when you have a unique or simple step).
Forgetting that the Evaluator component does not just compute metrics — it also compares the new model against a baseline and enforces thresholds. The Pusher component only deploys if the Evaluator says the model is good enough.
Mixing up the order of TFX components. The correct order is: ExampleGen, StatisticsGen, SchemaGen, ExampleValidator, Transform, Trainer, Evaluator, Pusher. The exam loves to scramble this order and ask you to identify the error.
Ignoring the concept of a metadata store. Vertex AI Pipelines uses a metadata store (managed by the service) to track all artifacts and executions. The exam might ask how lineage is tracked — the answer is through the metadata store, not through custom logging.
A Vertex AI Pipeline converts your manual ML workflow into an automated, repeatable graph of containerised steps.
Kubeflow Pipelines (KFP) is the orchestrator; TensorFlow Extended (TFX) is a library of pre-built ML components that run inside it.
Each step in a pipeline is a container, making the pipeline portable and ensuring consistent execution across environments.
Pipeline outputs are automatically cached, so re-running after a fix only reprocesses the changed steps and their dependents.
Data lineage is tracked in a metadata store, giving you a complete record of which data and code produced each model.
TFX components have a fixed order: ExampleGen, StatisticsGen, SchemaGen, ExampleValidator, Transform, Trainer, Evaluator, then Pusher.
You can use any ML framework in a KFP pipeline — it is not limited to TensorFlow.
Pipelines can accept parameters and include conditional logic to decide whether to deploy a model based on evaluation metrics.
These come up on the exam all the time. Here's how to tell them apart.
Kubeflow Pipelines (KFP)
A generic orchestrator that can run any containerised step
Focuses on the 'pipeline' — the graph of execution and dependencies
Steps can be custom Python code or any pre-built container
TensorFlow Extended (TFX)
A library of pre-built components specifically for ML tasks
Focuses on the 'components' — reusable building blocks for data validation, training, etc.
All components are TensorFlow-based, though they can wrap other frameworks
Pipeline Run
A single execution of the pipeline with a specific set of inputs and parameters
Produces concrete artifacts (data, models, metrics)
Can be triggered manually, on a schedule, or by an event
Pipeline Experiment
A grouping of multiple related runs, often with different parameters or code versions
Used for tracking the history of experiments (e.g., different hyperparameters)
Helps you compare runs and choose the best model
Component Caching
Stores the output of a component after execution
Reuses stored output if code and input have not changed
Operates at the pipeline level, saving compute time
Container Reuse
Reuses the same Docker image across multiple pipeline runs
Reduces image build time, but does not skip component execution
Operates at the infrastructure level, saving storage and build time
Metadata Store
Stores structured metadata about artifacts, executions, and lineage
Enables tracing which data and code produced a specific model
Accessed via the Vertex AI Pipelines UI or API
Cloud Logging
Stores unstructured log text from container stdout/stderr
Used for debugging errors and inspecting runtime behaviour
Accessed via the Cloud Logging console
Mistake
A pipeline is just a script that runs steps in order, like a Python script with multiple function calls.
Correct
A pipeline is a graph of containerised components that can run in parallel, handle failures gracefully, cache results, and track metadata. A simple script cannot do parallel execution or automatic retry of failed steps.
Beginners often think of pipelines as fancy scripts because that is the most familiar concept. They do not realise that the orchestration and containerisation are what make pipelines powerful.
Mistake
TFX and Kubeflow Pipelines are two completely separate things that you cannot use together.
Correct
TFX components are designed to run inside Kubeflow Pipelines on Vertex AI. You can mix TFX components with custom KFP components in a single pipeline.
The names are different and come from different open-source projects, so beginners assume they are competitors. In reality, TFX components are often used inside KFP pipelines as ready-made building blocks.
Mistake
Once a pipeline is deployed, you cannot change it. You have to create a new one from scratch.
Correct
You can update a pipeline definition and recompile it. Vertex AI Pipelines will show both versions, and you can run the new version while keeping the old one for reference.
In software development, deploying a new version often means overwriting the old one. But pipelines are versioned and experiments are separate, so you can have many versions without conflict.
Mistake
You must use TensorFlow models to use Vertex AI Pipelines.
Correct
Vertex AI Pipelines supports any ML framework, including PyTorch, scikit-learn, XGBoost, and custom code. TFX is TensorFlow-focused, but KFP can run any containerised code.
The exam and documentation often highlight TFX, which is TensorFlow-specific. Beginners may conclude that TensorFlow is mandatory, but the underlying orchestrator is framework-agnostic.
Mistake
If one step in the pipeline fails, the whole pipeline has to start from the beginning when you fix it.
Correct
Vertex AI Pipelines caches the outputs of successful steps. When you fix the failed step and re-run, only that step and its downstream dependents re-execute. Cached outputs are reused.
This sounds like magic, and many beginners have never used caching in a workflow system. They assume a linear script model where failure means redoing all work.
Mistake
Pipelines are only useful for large companies with huge datasets.
Correct
Even a simple pipeline with two steps — data preprocessing and model training — saves time and prevents errors compared to running manually. The benefits scale with complexity, but the basics apply to any project.
Beginners think automation is only for big-budget teams because they see complex demos. In reality, a simple pipeline is faster to debug and easier to reproduce than a notebook.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
Kubeflow Pipelines is an open-source framework for building and running ML pipelines. Vertex AI Pipelines is Google Cloud's managed service that runs Kubeflow Pipelines and TFX pipelines in a fully serverless environment, so you do not need to manage your own Kubernetes cluster.
No. You can use any machine learning framework, including PyTorch, scikit-learn, XGBoost, or custom Python code. Vertex AI Pipelines runs containers, so as long as your step is containerised, it works.
Data is passed as Artifacts — typed references to data stored in Google Cloud Storage. A component declares its input and output artifacts, and the pipeline system ensures the output of one step is available as input to the next.
A TFX component is a pre-built building block for common ML tasks, such as data ingestion (ExampleGen), data validation (ExampleValidator), feature engineering (Transform), model training (Trainer), and model evaluation (Evaluator). You can use them as-is or customise them.
Yes. Vertex AI Pipelines caches the outputs of every step. If you re-run the pipeline and the code and inputs for a step have not changed, the cached output is reused. Only the step you changed and any steps that depend on it will re-execute.
The Evaluator component in TFX checks the new model's performance against a threshold (like minimum accuracy) and also compares it to a previously deployed baseline model. It decides whether the model is good enough to be sent to the Pusher for deployment.
You've finished Building End-to-End ML Pipelines with Vertex AI Pipelines. Continue through the PMLE study guide to build a complete picture of the exam.
Done with this chapter?