A company uses SageMaker Pipelines to automate their ML workflow. They notice that the pipeline reruns all steps even when the input data has not changed. Which feature should they enable to avoid unnecessary recomputation?
Pipeline caching stores step outputs keyed on input signatures, so unchanged data and code let SageMaker skip re-execution and reuse prior artefacts. This directly removes the unnecessary recomputation the company observes, cutting cost and runtime without altering pipeline structure.
Why this answer
Pipeline caching in SageMaker Pipelines automatically reuses the output of a step if its inputs (including parameters, data, and code) have not changed since the last successful execution. This avoids recomputation by comparing a hash of the step's dependencies against previous runs, making it the correct feature to prevent unnecessary reruns when input data remains identical.
Exam trap
The trap here is that candidates confuse caching with conditional branching or parallel execution, assuming that skipping steps via conditions or running steps in parallel will avoid recomputation, when in fact only caching directly reuses prior outputs based on input immutability.
How to eliminate wrong answers
Option B is wrong because a Lambda step is used for custom processing or integration (e.g., invoking external APIs), not for detecting input changes or caching step outputs; it would add complexity without solving the core caching requirement. Option C is wrong because a Conditional step evaluates a condition to branch the pipeline (e.g., skip a step based on a metric), but it does not automatically detect unchanged inputs or cache results; it requires manual logic and still incurs overhead for the condition check. Option D is wrong because setting the pipeline execution mode to 'Parallel' controls whether steps run sequentially or concurrently, but it does not prevent recomputation of steps whose inputs have not changed; it only affects execution order, not caching.