An ML pipeline uses SageMaker Processing to run a feature engineering script. The script takes a long time and the team wants to speed up pipeline execution. What is the MOST effective approach?
Trap 1: Enable pipeline caching for the Processing step
Caching skips the step if inputs unchanged, but does not speed up execution when inputs change.
Trap 2: Use a larger instance type with more vCPUs
While a larger instance helps, parallelization with multiple instances is more effective for scalable speedup.
Trap 3: Use a Tuning step instead
Tuning is for hyperparameter optimization, not feature engineering.
- A
Increase the instance count for the Processing step
Increasing the instance count for the Processing step distributes the workload across multiple instances running in parallel, directly shortening the long-running script's wall-clock duration. This satisfies the stem's goal of speeding up pipeline execution, provided the script can be partitioned across shards.
- B
Enable pipeline caching for the Processing step
Why it fails: Caching skips the step if inputs unchanged, but does not speed up execution when inputs change.
- C
Use a larger instance type with more vCPUs
Why it fails: While a larger instance helps, parallelization with multiple instances is more effective for scalable speedup.
- D
Use a Tuning step instead
Why it fails: Tuning is for hyperparameter optimization, not feature engineering.