mediumMultiple Select
MLA-C01 Practice Question: A company uses Amazon SageMaker to deploy a model…
A company uses Amazon SageMaker to deploy a model for real-time inference. They want to perform A/B testing between two model versions. Which TWO actions should the company take to set up A/B testing? (Choose TWO.)
⚠ Common exam trap
Candidates often confuse the separate service Amazon CloudWatch Evidently with SageMaker's native traffic splitting, or think that auto scaling or zero-weight strategies are prerequisites for A/B testing.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Create an endpoint configuration with multiple production variants, each with a different model.
Option A is correct because SageMaker A/B testing is implemented by creating an endpoint configuration that contains multiple production variants, where each variant references a different model (via its ModelName), allowing the endpoint to serve both model versions behind a single endpoint. Option C is correct because each production variant has an InitialVariantWeight, and setting these weights establishes the desired traffic split (e.g., 80/20) so the endpoint routes the corresponding proportion of invocations to each model. Option B is incorrect because CloudWatch Evidently is a feature-flagging/experimentation service for applications, not the mechanism for splitting traffic between SageMaker production variants. Option D is incorrect because auto scaling adjusts instance counts for capacity, not traffic distribution between model versions, so it does not set up A/B testing. Option E is incorrect because setting a variant's weight to 0 means it receives no traffic, and shifting to 100 later is a blue/green-style cutover rather than a concurrent A/B test with a defined split.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✓
Create an endpoint configuration with multiple production variants, each with a different model.
Why this is correct
Creating an endpoint configuration with multiple production variants lets each model version receive a defined share of inference traffic, satisfying the A/B testing requirement. SageMaker routes requests across variants according to assigned weights, so you can compare live performance. Deploying both models behind one endpoint is the mechanism that enables controlled traffic splitting.
- ✗
Use Amazon CloudWatch Evidently to split traffic between models.
Why it's wrong here
Evidently splits traffic for feature experimentation and metrics, not SageMaker endpoint variants. It is tempting because it performs A/B testing, and it would be right for web or application experiments. SageMaker A/B testing instead requires creating multiple production variants on one endpoint and configuring traffic distribution between them.
- ✓
Set the initial weight of each production variant to the desired traffic split.
Why this is correct
Variant weights define the traffic distribution across the endpoint; setting them establishes the desired split between the two model versions. Without explicit weights, SageMaker defaults to equal routing, so this action controls the A/B experiment's proportions.
- ✗
Enable auto scaling for each production variant individually.
Why it's wrong here
Auto scaling adjusts instance counts to match traffic demand; it does not split inference requests between variants. A/B testing requires configuring multiple production variants within one endpoint and assigning them traffic weights. Auto scaling would be the right action for handling variable load on an already-configured endpoint, not for distributing traffic across model versions.
- ✗
Set the second production variant's weight to 0 and update later to 100.
Why it's wrong here
Setting a variant's weight to 0 removes it from traffic rotation entirely, so no requests reach that model and no comparison data is collected. A/B testing needs both variants serving simultaneously with non-zero weights. This approach suits a staged blue/green rollout, where traffic shifts fully once the new version is validated, rather than continuous A/B measurement.
Go deeper
Related to this question
About these practice questions
One of 665 original MLA-C01 practice questions on Courseiva, each with a full explanation and wrong-answer analysis — not exam dumps or protected exam content. Learn why practice questions differ from exam dumps →
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This MLA-C01 practice question is part of Courseiva's free Amazon Web Services certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the MLA-C01 exam.