Exam objective 2.4 is all about making sure your machine learning project is not a one-off lab experiment that only you can run. The problem is that real ML workflows have many steps — getting data, cleaning it, training a model, testing it, and deploying it — and doing these steps by hand is slow, error-prone, and impossible to repeat. For the MLA-C01 exam, you need to understand how SageMaker Pipelines lets you build a self-running assembly line for your ML projects, so you can automate the boring stuff and focus on improving the model.
Jump to a section
A simple way to picture Building and Automating ML Pipelines with SageMaker Pipelines
Have you ever helped plan a large family dinner or a wedding reception? You have to coordinate the menu, the cooking, the serving, and the cleanup. Doing that manually, one dish at a time, is a recipe for disaster. You would be running around the kitchen, checking on the roast, stirring the sauce, and answering calls from the florist, all while the guests start arriving. It is chaotic, and you will likely burn the garlic bread.
Now imagine having a master event planner. You tell this planner the final goal: a four-course meal for 100 guests, served at 7pm. The planner then automatically triggers the grocery order on Tuesday, sets the ovens to preheat at 4pm, tells the salad chef to start at 5pm, alerts the roast chef for 6pm, and checks that the dishwasher is empty before dessert. If the grocery delivery is late, the planner automatically reschedules the prep times and texts you a notification. This planner is SageMaker Pipelines. You define the recipe (your ML steps: data loading, training, tuning, deployment), and SageMaker Pipelines orchestrates the entire kitchen automatically, handling delays and errors without you needing to stand over the stove. You just check the final result and enjoy the meal.
SageMaker Pipelines is a service from AWS that lets you define, automate, and manage end-to-end machine learning workflows. Think of a workflow as a recipe with multiple steps: first you get your ingredients (raw data), then you clean and chop them (data processing), then you cook them (train a model), then you taste and adjust (evaluate and tune the model), and finally you plate and serve (deploy the model). A pipeline is the automated system that runs these steps in the correct order, every time, without you having to press buttons manually.
Before SageMaker Pipelines, data scientists often wrote custom scripts to chain steps together. They might use a shell script that calls one Python file after another, or they would manually start each step in the SageMaker console. This approach is fragile: if step 2 crashes, the whole thing stops, and it is hard to reproduce the exact same run later. SageMaker Pipelines solves this by providing a managed service that tracks every run, logs everything, and can automatically retry failed steps or send alerts.
The core building block of a pipeline is a step. There are several types of steps:
ProcessingStep: runs a script to clean, transform, or analyse your data. For example, you might use a ProcessingStep to remove missing values and normalise numerical features.
TrainingStep: starts a training job on SageMaker. You give it an algorithm (like XGBoost or a custom PyTorch script) and your training data location in S3.
TuningStep: runs a hyperparameter tuning job. This automatically tries different model settings (like learning rate or tree depth) to find the best combination.
ModelStep: creates a SageMaker model object from the trained artefacts. This is a necessary step before you can deploy.
TransformStep: runs a batch transform job, which makes predictions on a large set of data all at once, without needing a live endpoint.
ConditionStep: adds logic to your pipeline. For example, "If the model accuracy is above 90%, then deploy it; otherwise, do not."
CreateModelStep: registers the model in the SageMaker Model Registry, which is a central catalogue of all your models.
EndpointStep: deploys the model to a real-time endpoint so applications can get predictions instantly.
All of these steps connect in a directed acyclic graph (DAG). That is a fancy way of saying you define which steps depend on which. For instance, you cannot train a model until the data is processed, and you cannot evaluate it until the model is trained. SageMaker Pipelines figures out the order and runs steps in parallel if they are independent. If a step fails, you can see exactly where it failed in the execution history.
You define the pipeline using the SageMaker Python SDK. You write code that looks like this: you import the necessary step classes, create step objects, chain them together, and then create a Pipeline object with a name and the steps list. When you call pipeline.start(), SageMaker takes over and runs everything in the cloud. You can schedule the pipeline to run on a cron schedule (e.g., every night at midnight).
Why does this matter? Because in a real business, data changes over time. A model that worked last month might be outdated now. You want to retrain the model regularly with fresh data. Manually doing that is not scalable. SageMaker Pipelines automates that retraining, with approval gates and notifications. The exam tests your understanding of these step types, the DAG concept, and how to handle errors and conditional branching.
Define Pipeline Parameters
You start by creating parameter objects for your pipeline, like the path to the raw data in S3 or the instance type for training. This makes your pipeline reusable for different input data or configurations without editing code.
Create ProcessingStep for Data Preparation
You define a ProcessingStep that runs a script (e.g., a Python or Spark script) to clean, transform, and split your data into training, validation, and test sets. This step outputs the processed data to a specified S3 location.
Create TrainingStep for Model Training
You create a TrainingStep that takes the processed training data and trains an ML model. You specify the algorithm (built-in or custom), hyperparameters, and compute resources. The output is the model artefacts saved to S3.
Add ConditionStep for Model Evaluation
You code a ConditionStep that checks a metric (e.g., accuracy or F1 score) calculated on the validation data. If the metric meets your threshold, the pipeline proceeds to deployment; otherwise, it stops or triggers a notification.
Define ModelStep and Deploy with EndpointStep
If the condition passes, you use a ModelStep to create a SageMaker model object from the trained artefacts. Then you add an EndpointStep to deploy that model to a real-time endpoint, making it available for predictions. Optionally, you use a RegisterModel step to add it to the Model Registry.
Create the Pipeline and Start Execution
You assemble all the steps into a Pipeline object, specifying the pipeline name, parameters, and the list of steps in dependency order. You then call pipeline.start() or trigger it via an EventBridge rule using the SDK, CLI, or console. SageMaker orchestrates and monitors the execution.
Consider Sarah, a machine learning engineer at a retail company that sells shoes online. Her boss wants the website to suggest shoes to customers based on their browsing history. Sarah built a recommendation model that works well in testing, but she needs to make sure it keeps working well as new shoes arrive and customer tastes change.
Sarah uses SageMaker Pipelines to automate her workflow. Here is what she does step by step:
She stores the raw click-stream data in an S3 bucket. Every night, new data arrives.
She creates a SageMaker Pipeline definition using the SageMaker Python SDK. The first step is a ProcessingStep. This step runs a Spark script that reads the raw data, filters out bot traffic, groups clicks by user, and creates feature vectors (lists of numerical characteristics for each user and each shoe). The output is written to a clean S3 location.
The second step is a TrainingStep. It takes the clean data and trains an XGBoost model to predict which shoe a user will click on next. The trained model artefacts are saved to S3.
The third step is a ConditionStep. It checks: is the accuracy of the new model at least 2% better than the current live model? To do this, it calls an EvaluateModel Lambda function that compares the new and old models on a holdout test dataset.
If the condition passes, the pipeline runs a ModelStep to create a SageMaker model and then an EndpointStep to deploy it to a real-time endpoint, replacing the old model. If the condition fails, the pipeline sends an alert to Sarah via Amazon SNS (Simple Notification Service) email, and does not deploy the worse model.
Sarah sets up an EventBridge rule to trigger the pipeline every night at 3 AM. The next morning, she checks the pipeline execution history. If a step failed because the data was malformed, she can inspect the logs in CloudWatch.
In her role, Sarah does not sit clicking buttons. She writes the pipeline definition code once, tests it thoroughly, and then lets it run automatically. She spends her time improving the model algorithm and handling edge cases, not babysitting training jobs. This is what IT professionals actually do with SageMaker Pipelines: they automate retraining, model evaluation, and deployment, ensuring the production model is always up-to-date without manual intervention.
The MLA-C01 exam tests objective 2.4 by giving you scenario-based questions where you must design or troubleshoot a SageMaker Pipeline. You will not be asked to write code in the exam, but you need to know the purpose and behaviour of each step type. Here are the exact concepts they love to test:
Step types and their roles: You must know that ProcessingStep is for data processing, TrainingStep is for training, ModelStep creates a model object, and EndpointStep deploys. They might ask: "Which step would you use to run a data cleaning script?" The answer is ProcessingStep.
Conditional branching: ConditionStep is a frequent trap. They will ask: "How do you deploy a model only if it meets a certain accuracy threshold?" The correct answer is to use a ConditionStep that checks the metric and then either proceeds to deployment or stops.
Pipeline execution order (DAG): They give you a list of steps and ask which runs first. You need to understand dependencies. For example, a TrainingStep cannot run before a ProcessingStep that provides its input data.
Model Registry integration: They test whether you know you can register a model using a CreateModelStep or a dedicated RegisterModel step. The exam expects you to know that registering the model in the Model Registry allows for versioning and approvals.
Error handling and retries: They might ask: "If a step fails due to a transient error, what is the best practice?" The answer is to configure retry policies on the step, not to ignore the error or manually restart the pipeline.
Scheduling: They will ask about using Amazon EventBridge or AWS Lambda to trigger a pipeline on a schedule. They will not ask about cron syntax details, but they will expect you to know that pipelines can be run on a set schedule.
Trap: They give you a scenario where the user manually runs each step in the console. The correct answer is to recommend SageMaker Pipelines for automation.
Trap: They suggest using a single large script for all steps. The correct answer is to break it into separate pipeline steps for modularity.
Trap: They say "use a TrainingStep to clean data." That is wrong — data cleaning needs a ProcessingStep.
Key definitions to memorise:
Pipeline: A directed acyclic graph (DAG) of steps that run an ML workflow.
Step: A single unit of work in a pipeline (processing, training, etc.).
Execution: A single run of a pipeline, with a unique ID.
Cache hit: If step inputs and parameters are identical to a previous run, the pipeline can reuse the cached result instead of re-running the step (this saves cost and time).
SageMaker Pipelines automates the entire ML workflow from data processing to model deployment using a directed acyclic graph (DAG) of steps.
Each step in a pipeline must be a specific SageMaker step type: ProcessingStep, TrainingStep, TuningStep, ModelStep, TransformStep, ConditionStep, or EndpointStep.
A ConditionStep allows you to add if-then-else logic to your pipeline, such as deploying only if a model meets a performance threshold.
Pipelines can be triggered on a schedule using Amazon EventBridge, enabling automatic retraining of models with fresh data.
SageMaker Pipelines integrates with the SageMaker Model Registry so you can version, approve, and deploy models in a controlled manner.
Pipeline executions are traceable: you can view logs, metrics, and step status in the SageMaker console or via the AWS CLI.
Using pipeline caching, you can avoid re-running steps whose inputs and parameters have not changed, saving time and cost.
These come up on the exam all the time. Here's how to tell them apart.
ProcessingStep
Used for data cleaning, transformation, and feature engineering.
Runs a script (e.g., Python, Spark) inside a container.
Outputs processed data to S3, not a model artefact.
TrainingStep
Used for training an ML model using an algorithm.
Runs a training job that produces a model artefact.
Output is the trained model saved to S3, ready for deployment.
ConditionStep
Adds if-then-else logic to the pipeline flow.
Does not run any compute itself; it just checks a condition.
Commonly used to decide whether to deploy a model or not.
ModelStep
Creates a SageMaker Model resource from trained artefacts.
Does not evaluate any condition; it just packages the model.
Required before you can use an EndpointStep or TransformStep.
SageMaker Pipelines (managed orchestration)
Designed specifically for ML workflows with native step types.
Tightly integrated with SageMaker training, processing, and endpoints.
Runs inside the SageMaker environment with built-in logging.
Step Functions (general-purpose orchestration)
General-purpose state machine for any AWS service workflow.
Requires you to manually build integrations for ML tasks.
More flexible but more code is needed to replicate SageMaker step types.
Pipeline Caching (reuse previous output)
Compares input parameters and data to see if a step can be skipped.
Saves cost and time by not re-running unchanged steps.
Automatic; you enable it in the step definition.
Re-running a pipeline from scratch
Every step runs fresh, regardless of whether inputs changed.
Ensures consistency but costs more and takes longer.
Simulates a completely new pipeline execution.
SageMaker Model Registry (catalogue of models)
Stores model versions, metadata, and approval statuses.
Acts as a central repository for managing model lifecycle.
Used as a destination or source for pipeline steps.
SageMaker Pipeline steps (individual actions)
Individual actions like training, processing, or deployment.
Each pipeline execution creates a sequence of steps.
The registry is where pipeline outputs (models) are logged.
Mistake
SageMaker Pipelines is just a fancy way to run a Jupyter notebook in the cloud.
Correct
SageMaker Pipelines is a managed workflow orchestrator that runs individual steps as separate, containerised jobs. It is not a notebook runner. You define the pipeline in code, and it executes on SageMaker infrastructure, not inside a notebook cell.
Beginners often confuse SageMaker Pipelines with SageMaker Notebooks because both use Python SDKs. The difference is that notebooks are interactive development environments, while pipelines are automated production workflow engines.
Mistake
If one step in the pipeline fails, the entire pipeline is ruined and you have to start from scratch.
Correct
Pipelines can have retry policies and conditional logic. You can configure retries for transient errors (like a timeout). After fixing the issue, you can also resume from the failed step using a cached execution or by starting a new execution that skips previously successful steps if the inputs haven't changed.
This misconception comes from experience with simple shell scripts where a failure means starting over. SageMaker Pipelines is more resilient because it tracks outputs and can reuse previous results.
Mistake
SageMaker Pipelines can only run inside the AWS region where it was created, and you cannot use data from other regions.
Correct
Pipelines can reference data in any S3 bucket in any region. The pipeline runs in the region where it is defined, but it can read data from cross-region buckets (though you might incur data transfer costs).
New learners sometimes think AWS services are locked to one region because many consoles show per-region resources. In reality, S3 buckets are region-specific but accessible from anywhere via API calls.
Mistake
You must deploy a model to an endpoint inside the pipeline definition itself, or the pipeline is incomplete.
Correct
Deploying to an endpoint is optional. Many pipelines stop at model evaluation or registration. The pipeline can create a model and register it in the Model Registry, and then a separate process (like a CI/CD pipeline) handles the actual deployment.
People think an ML pipeline must end with a deployed model because that seems like the natural endpoint. In real-world workflows, deployment is often gated by manual approval or separate deployment systems.
Mistake
SageMaker Pipelines only works with built-in SageMaker algorithms, not custom ones.
Correct
Pipelines can run any custom Docker container. You can bring your own algorithm, framework, or even a custom inference script. The TrainingStep and ProcessingStep both support custom images.
Early SageMaker documentation emphasised built-in algorithms, leading beginners to think custom code was hard to integrate. The SDK has made it straightforward for years.
Mistake
Pipelines are free; you only pay for the underlying compute resources they use.
Correct
While there is no additional fee for creating the pipeline definition, you do pay for the SageMaker compute instances used by each step (training, processing, etc.) and for data transfer between steps. There is also a per-execution fee for certain step types like TuningStep.
Many assume that since the pipeline orchestration is a managed service, it must be free. AWS does charge for pipeline executions, though the cost is usually small compared to compute.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
SageMaker Studio is an integrated development environment (IDE) for building, training, and deploying models. SageMaker Pipelines is a workflow orchestrator that runs automated, repeatable ML pipelines. You can create and manage pipelines from within SageMaker Studio, but they are separate services.
Yes. You can bring your own Docker image for training, processing, or inference. You just specify the image URI in the step definition, and SageMaker will pull and run your custom container.
You can use Amazon EventBridge to create a rule that triggers the pipeline on a cron schedule. The rule calls the StartPipelineExecution API from the SageMaker Pipelines service principal. You set the schedule expression in EventBridge.
You can configure a retry policy for the step to automatically retry if the failure is due to a transient issue. If it keeps failing, you need to increase the instance type or memory allocation for that step and restart the pipeline execution.
Not necessarily. The pipeline can be fully automated to deploy without approval if the ConditionStep passes. However, you can integrate with the Model Registry and use an approval gate there. The pipeline can register the model and then a separate approval process from the registry triggers the deployment.
Yes. If two steps have no dependencies on each other, the pipeline engine will run them concurrently. For example, you could run two separate training jobs in parallel on different data subsets.
You've finished Building and Automating ML Pipelines with SageMaker Pipelines. Continue through the MLA-C01 study guide to build a complete picture of the exam.
Done with this chapter?