MLA-C01 Deployment and Orchestration of ML Workflows Practice Question

A company runs a real-time fraud detection model on a SageMaker endpoint. The model is updated weekly, and each update must be validated against live traffic without affecting existing predictions. The team wants to compare the new model's performance against the current model using a small percentage of incoming requests, while ensuring that the current model continues to serve the majority of traffic. Which SageMaker deployment strategy should they use?

⚠ Common exam trap

Candidates often confuse shadow testing with A/B testing, as shadow testing does not return predictions to the caller and thus cannot be used for live performance validation.

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

A/B testing with production variants, where the new model variant receives a small portion of traffic.

SageMaker production variants enable A/B testing by allowing multiple models on a single endpoint with configurable traffic weights. This lets the team direct a small percentage of live traffic to the new model while the current model serves the rest, facilitating performance comparison without disrupting service. Other strategies either do not return predictions or switch all traffic at once.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✓
A/B testing with production variants, where the new model variant receives a small portion of traffic.
Why this is correct
A/B testing with production variants allows you to deploy multiple models to the same endpoint and distribute traffic between them. By assigning a small weight to the new variant, you can evaluate its performance on live traffic while the existing model handles the rest. This meets the requirement of validating the new model without impacting the majority of predictions.
✗
Canary deployment, where the new model is deployed to a separate endpoint and traffic is gradually shifted using a load balancer.
Why it's wrong here
Canary deployment in SageMaker is not a native feature; it typically requires manual traffic shifting via a load balancer or Route 53. While it allows gradual traffic shifting, it does not provide the built-in variant weighting and performance monitoring that SageMaker production variants offer, making it less suitable for this scenario.
✗
Blue/green deployment, where traffic is shifted all at once from the old model to the new model after validation.
Why it's wrong here
Blue/green deployment shifts all traffic to the new environment after validation, which does not allow for gradual evaluation on live traffic. The scenario requires the new model to serve a small percentage of requests while the old model continues to serve the majority, so an all-at-once switch is not appropriate.
✗
Shadow testing, where the new model receives a copy of live traffic but its predictions are not returned to the caller.
Why it's wrong here
Shadow testing is used to evaluate a new model by mirroring live traffic to it without affecting responses. However, the scenario requires comparing performance while the new model serves a portion of actual predictions. Shadow testing does not return predictions to the caller, so it cannot be used to validate the model's real-world impact on business metrics.

Go deeper

MLA-C01 Deployment and Orchestration of ML Workflows Practice Question

Why each option matters

Option-by-option breakdown

Related to this question