Automated Retraining on Model Quality Degradation: SageMaker Model Monitor with CloudWatch Alarms
A machine learning engineer observes that model performance on a SageMaker endpoint has degraded over the past week. Ground truth labels are available with a 2-day delay. The engineer wants to automatically trigger a retraining pipeline when prediction quality drops below an acceptable threshold. Which approach is most appropriate?
Quick Answer
The scenario has three requirements stacked on top of each other: detect quality degradation using real ground truth rather than a proxy signal, do it automatically, and kick off retraining without a human in the loop, and the correct answer is the only option that chains a tool for each requirement together. SageMaker Model Quality Monitor is purpose-built to compare live predictions against ground truth labels once they arrive, computing metrics like accuracy on a schedule, which fits the detail that labels lag by two days. A CloudWatch alarm on that metric is what turns a passive measurement into an actionable trigger the moment the value crosses an unacceptable threshold, and a Lambda function is the standard lightweight way to respond to that alarm by kicking off a retraining pipeline. Each piece matters: monitoring without an alarm would require someone to keep checking dashboards, and an alarm without ground-truth-based quality monitoring would only catch input data drift, not the actual prediction degradation described here. When a question combines delayed ground truth, a need to catch performance drops rather than data drift, and a requirement for automatic action, expect the answer to link a quality monitor, a CloudWatch alarm, and an automated trigger like Lambda.
⚠ Common exam trap
A common mix-up: candidates confuse Data Quality Monitor (which monitors input data drift) with Model Quality Monitor (which monitors prediction accuracy against ground truth), leading them to choose Option C incorrectly.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Use SageMaker Model Monitor - Model Quality Monitor with ground truth, create a CloudWatch alarm on the metric, and trigger an AWS Lambda function to start retraining
SageMaker Model Monitor's Model Quality Monitor is specifically designed to compare model predictions against ground truth labels (available with a 2-day delay) and track metrics like accuracy, precision, recall, or F1 score. You can configure a CloudWatch alarm on a metric such as 'accuracy' dropping below a threshold, which triggers an AWS Lambda function to start the retraining pipeline. This automates the detection of prediction quality degradation and the retraining response without manual intervention.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✓
Use SageMaker Model Monitor - Model Quality Monitor with ground truth, create a CloudWatch alarm on the metric, and trigger an AWS Lambda function to start retraining
Why this is correct
Model Quality Monitor evaluates predictions against ground truth; CloudWatch alarm on quality metric triggers retraining.
- ✗
Manually evaluate the model weekly and retrain as needed
Why it's wrong here
Manual process is not automated and may miss degradation between evaluations.
- ✗
Use SageMaker Model Monitor - Data Quality Monitor to detect drift, then trigger retraining
Why it's wrong here
Data quality monitor detects input drift, not prediction quality. Ground truth is required for model quality.
- ✗
Use SageMaker Clarify to monitor bias drift and trigger retraining
Why it's wrong here
Bias drift monitors fairness, not overall prediction quality.
Quick reference
Cloud Service Model Comparison
| Model | You Manage | Provider Manages | Examples |
|---|---|---|---|
| IaaS | OS, runtime, apps, data | Hardware, hypervisor, networking | EC2, Azure VMs, GCP Compute Engine |
| PaaS | Apps and data | OS, runtime, middleware, hardware | Elastic Beanstalk, Azure App Service |
| SaaS | Data and settings only | Everything else | Microsoft 365, Salesforce, Workday |
| FaaS / Serverless | Function code only | Infra, scaling, runtime | Lambda, Azure Functions, Cloud Run |
| CaaS | Containers and apps | Kubernetes, OS, hardware | EKS, AKS, GKE |
Go deeper
Related to this question
About these practice questions
One of 835 original MLA-C01 practice questions on Courseiva, each with a full explanation and wrong-answer analysis — not exam dumps or protected exam content. Learn why practice questions differ from exam dumps →
Same concept, more angles
2 more ways this is tested on MLA-C01
These questions test the same concept from different angles. Work through them to make sure you can recognise it however the exam phrases it.
Variation 1. A machine learning engineer is setting up a retraining pipeline that triggers when concept drift is detected. They plan to use CloudWatch Alarms to monitor the model's accuracy metric. When drift is detected, they want to automatically start a SageMaker training job. Which architecture should they use?
medium- A.CloudWatch Alarm → SQS → Lambda → SageMaker Training Job
- B.CloudWatch Alarm → EventBridge → SageMaker Training Job
- ✓ C.CloudWatch Alarm → SNS → Lambda → SageMaker Training Job
- D.CloudWatch Alarm → Lambda directly (without SNS)
Why C: CloudWatch Alarms cannot directly invoke SageMaker training jobs; they require an intermediary like SNS to trigger a Lambda function, which then calls the SageMaker API to start the training job. This pattern ensures reliable decoupling and allows the Lambda function to handle any preprocessing or conditional logic before launching the job.
Variation 2. A machine learning engineer wants to set up a retraining pipeline that triggers when model quality degrades. Which TWO components are essential for this automated retraining pipeline? (Select TWO)
easy- ✓ A.CloudWatch Alarm on model quality metric
- ✓ B.SNS topic to send notification to a Lambda function
- C.SageMaker Ground Truth to collect new labels
- D.SageMaker Data Wrangler to preprocess data
- E.EventBridge rule to schedule retraining weekly
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This MLA-C01 practice question is part of Courseiva's free Amazon Web Services certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the MLA-C01 exam.