MLA-C01 Deployment and Orchestration of ML Workflows Practice Question

A machine learning team is deploying a model to a SageMaker endpoint and needs to implement A/B testing between two model versions. They want to split traffic 80/20 and monitor performance metrics for each variant. Which two actions should they take? (Choose two.)

⚠ Common exam trap

The trap here is assuming that Model Monitor automatically compares variant performance, but it only monitors individual models for drift and quality.

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

Create a SageMaker endpoint configuration with two production variants, each specifying a different model and initial weight.

To perform A/B testing on SageMaker, you create an endpoint configuration with multiple production variants, each with a different model and weight to split traffic. Enabling data capture logs the requests and responses for each variant, allowing you to analyze performance metrics. Other options either do not provide native A/B testing or add unnecessary complexity.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✗
Configure an Application Load Balancer to distribute traffic between two separate endpoints.
Why it's wrong here
An Application Load Balancer can distribute traffic between endpoints, but it does not integrate with SageMaker's variant weights or provide native A/B testing capabilities. Using separate endpoints and an ALB increases complexity and cost, and it lacks the built-in monitoring and data capture features of SageMaker endpoints.
✗
Use SageMaker Model Monitor to automatically compare the accuracy of the two variants.
Why it's wrong here
Model Monitor detects data drift and model quality issues but does not automatically compare accuracy between two variants. It can monitor model quality if ground truth is available, but it does not provide built-in A/B comparison. The team would need to analyze the captured data separately.
✓
Create a SageMaker endpoint configuration with two production variants, each specifying a different model and initial weight.
Why this is correct
An endpoint configuration with multiple production variants allows you to deploy multiple models to a single endpoint. Each variant can have a different model and an initial weight that determines the traffic split. This is the foundational step for A/B testing, as it enables simultaneous serving of both model versions.
✗
Create two separate SageMaker endpoints and use AWS Lambda to route requests based on a random number.
Why it's wrong here
Creating separate endpoints and using a Lambda function for routing is a custom solution that requires additional infrastructure and management. It does not leverage SageMaker's native traffic splitting and data capture. This approach is more complex and error-prone compared to using production variants.
✓
Enable data capture on the endpoint to log request and response data for analysis.
Why this is correct
Data capture logs the input and output data for each inference request, which is essential for analyzing the performance of each variant. By enabling data capture, the team can later evaluate metrics such as accuracy or latency per variant and make informed decisions. This is a key component of A/B testing.

Go deeper

MLA-C01 Deployment and Orchestration of ML Workflows Practice Question

Why each option matters

Option-by-option breakdown

Related to this question