MLA-C01 ML Solution Monitoring, Maintenance and Security Practice Test 8 — 15 Questions

Question 1

A company uses SageMaker Model Monitor to detect drift. They want to receive notifications when drift is detected. Which TWO services can be used together to send notifications? (Choose TWO.)

Accepted Answer

Amazon SNS topics to send email or SMS.. Options B and D are correct. Amazon CloudWatch Alarm can trigger on a Model Monitor metric, and it can publish to an SNS topic to send email or SMS. Option A (Lambda) could be used but is not a direct notification service. Option C (SQS) is for queues. Option E (EventBridge) can route events but not directly send notifications; usually triggers Lambda.

Answer

Amazon EventBridge to trigger a notification.

Answer

AWS Lambda to process the drift and send an email via SES.

Answer

Amazon SQS to queue the notification.

Question 2

A retail company has deployed a real-time recommendation model on a SageMaker endpoint. The model is trained daily using SageMaker Pipelines that process user interaction data from a large S3 bucket. Recently, the operations team noticed that the endpoint's predictions have become stale; users are seeing recommendations based on data from days ago. The pipeline runs successfully every day at 2 AM UTC, but the endpoint continues to serve the old model version. The team checks the pipeline and finds no errors. The model registry contains multiple model versions approved automatically. The endpoint is configured with production variants, but only one variant is active. The team suspects the issue is with the deployment step in the pipeline. They want to automatically deploy new model versions to the endpoint as soon as they are registered and approved. What should they do?

Accepted Answer

Set up the SageMaker Model Registry to trigger a Lambda function on approval that updates the endpoint using the new model version.. Option B is correct because using a SageMaker Model Registry with an automatic deployment pipeline (via EventBridge or Lambda triggered by approval) ensures new models are deployed when approved. Option A (manual approval) is not automatic. Option C (test on batch transform) doesn't deploy to endpoint. Option D (change endpoint configuration manually) is not automated and suggested they do currently? Actually they need automation.

Answer

Configure the pipeline to stop automatic approval and require manual approval before deployment.

Answer

Modify the pipeline to run batch transforms on the new model and compare metrics, then update the endpoint.

Answer

Create a daily cron job that checks for new model versions and manually updates the endpoint configuration.

Question 3

A healthcare company is subject to HIPAA and uses SageMaker to train models on patient data. The data is stored in an S3 bucket with server-side encryption using a customer-managed KMS key. The training job uses a custom Docker container that needs to read the data. The security team is concerned about unauthorized access to the data during training. They want to ensure that only the specific training job can access the decryption key. The training runs in a VPC. What should they do?

Accepted Answer

Configure the training job to run in a VPC with an S3 VPC endpoint and attach an IAM role that has kms:Decrypt permission only for the key and only for that role.. Option C is correct because using a VPC and configuring the training job to use a VPC with a VPC endpoint for S3 and KMS ensures data stays within the VPC. Also, the training role should have strict permissions to the KMS key. Option A (bucket policy) alone is not enough. Option B (use only private subnet) but still need S3 access. Option D (S3 access points) is not the primary security measure.

Answer

Place the training job in a private subnet and use a NAT gateway for S3 access.

Answer

Configure the S3 bucket policy to allow only the SageMaker training role's ARN.

Answer

Use S3 access points with a policy that restricts access to the training job's IP address.

Question 4

A startup is using SageMaker to train a deep learning model. They use GPU instances for training. The training job takes about 8 hours. The team notices that sometimes the training job fails with an error message indicating that the instance was terminated due to Amazon EBS volume underprovisioned. The team is using the default EBS volume size for the training instance. They want to avoid this error without over-provisioning. What should they do?

Accepted Answer

Specify a larger EBS volume size in the training job's resource configuration.. Option B is correct because increasing the EBS volume size to accommodate the dataset and intermediate checkpoint files prevents the volume full error. Option A (use compute-optimized instances) doesn't fix storage. Option C (Amazon EFS) is a file system but may add latency and is not directly attached to training instances; requires mount. Option D (FSx for Lustre) is high-performance but complex and overkill; also requires separate setup.

Answer

Mount an Amazon EFS file system to the training instance and store all data there.

Answer

Switch to compute-optimized (C5) instances to reduce storage usage.

Answer

Configure the training job to use Amazon FSx for Lustre as a scratch file system.

Question 5

A media company uses SageMaker endpoints to serve a model that predicts video engagement. They have two production variants: Variant A (ml.c5.large) for regular traffic and Variant B (ml.c5.xlarge) for burst traffic. They use weighted routing (90% to A, 10% to B). Recently, during peak hours, Variant A's latency increase causes many requests to time out. The metrics show that both variants are under similar CPU load, but the number of concurrent requests to Variant A is very high. The team wants to ensure that burst traffic is handled properly without manual intervention. What should they do?

Accepted Answer

Configure Application Auto Scaling for each variant with a target tracking scaling policy based on the number of concurrent requests per instance.. Option B is correct because changing to target tracking scaling based on the number of concurrent requests (or InvocationsPerInstance) ensures each variant scales based on its load. Option A (swap weights) doesn't fix scaling. Option C (p99 latency alarm) might trigger too late. Option D (separate endpoint) is not necessary.

Answer

Increase the traffic weight to Variant B to 70% and reduce Variant A to 30%.

Answer

Set a CloudWatch alarm on Variant A's p99 latency and trigger a step scaling policy to add instances.

Answer

Create a separate endpoint for burst traffic and route peak traffic to it via DNS.

Question 6

A company has deployed a SageMaker endpoint for real-time inference. The security team needs to monitor for potential security threats such as unauthorized access attempts and tampering with the model configuration. Which TWO actions should the team take? (Choose TWO.)

Accepted Answer

Enable AWS CloudTrail for the SageMaker endpoint API calls. CloudTrail logs all API calls to the SageMaker endpoint, including who made the call and from where, which helps identify unauthorized access. AWS Config continuously monitors endpoint configuration changes and can trigger alerts when changes are made without authorization. SageMaker Model Monitor is for data drift, not security. Data Capture captures input/output for monitoring model performance, not security. GuardDuty is a threat detection service for AWS accounts and workloads, but it does not directly monitor SageMaker endpoints specifically.

Answer

Enable SageMaker Data Capture on the endpoint

Answer

Enable SageMaker Model Monitor for the endpoint

Answer

Enable Amazon GuardDuty for the endpoint

Question 7

A machine learning team at a retail company has deployed a product recommendation model using Amazon SageMaker. The model is updated weekly with new data. Recently, the team noticed that the model's accuracy on a holdout evaluation set has been declining over the past month. The data pipeline that feeds the training job has not changed. The team suspects data drift. They have SageMaker Model Monitor enabled on the inference endpoint and have set up Amazon CloudWatch metrics for feature distribution distances. Upon reviewing the CloudWatch dashboards, they see that the feature distribution distance metric for the most important feature 'product_category' has increased significantly. However, the team is unsure if this is the root cause. Which remediation step should the team take FIRST?

Accepted Answer

Investigate the data pipeline that feeds the training job to ensure consistent data collection and encoding of the 'product_category' feature. Before retraining the model or deploying a new endpoint, the team should investigate the source of the data drift by checking the input data pipeline. The data pipeline might have introduced a systematic error, such as a change in how 'product_category' is encoded or collected. Option A (retrain the model with more recent data) might not help if the data itself is corrupted. Option B (reduce the number of features) could ignore the problem. Option D (rebuild the endpoint) would not address the data drift. Therefore, the first step is to investigate the data pipeline.

Answer

Retrain the model using the most recent week of data and redeploy to the endpoint

Answer

Rebuild the SageMaker endpoint with a different instance type to improve performance

Answer

Reduce the number of features in the model by removing 'product_category'

Question 8

A financial services company uses an Amazon SageMaker endpoint for real-time credit scoring. The endpoint is deployed with an ml.c5.2xlarge instance. Recently, the data science team has received complaints from users about slow response times. The team monitors the endpoint using CloudWatch metrics. They observe that the InvocationsPerSecond metric averages 50, the ModelLatency metric averages 200 milliseconds, and the CPUUtilization metric averages 95%. The team has also noticed that the endpoint occasionally returns HTTP 503 (Service Unavailable) errors during peak hours. The team needs to reduce latency and eliminate 503 errors while minimizing cost increase. Which solution should the team implement?

Accepted Answer

Create a SageMaker endpoint with multiple instances behind a load balancer and configure automatic scaling based on CPUUtilization or InvocationsPerSecond. CPUUtilization at 95% indicates that the instance is overloaded, causing high latency and 503 errors. Scaling out (adding more instances) will distribute the load and reduce latency, and using automatic scaling ensures that the number of instances adjusts to demand, minimizing cost by scaling down when traffic is low. Option A (larger instance) may not be as cost-effective as scaling out, and Option B (enable data capture) would not help latency. Option D (increase timeout) does not address the root cause of overloading.

Answer

Enable SageMaker Data Capture to collect inference data for later analysis to identify slow requests

Answer

Replace the endpoint instance type with a more powerful compute-optimized instance, such as ml.c5.4xlarge

Answer

Increase the endpoint invocation timeout from 60 seconds to 120 seconds in the application configuration

Question 9

A healthcare startup has deployed a machine learning model on Amazon SageMaker that predicts patient readmission risks. The model uses sensitive health data stored in an S3 bucket encrypted with AWS KMS. The SageMaker endpoint is configured with an IAM role that has the following policy attached: {    "Version": "2012-10-17",    "Statement": [        {            "Effect": "Allow",            "Action": "s3:*",            "Resource": "arn:aws:s3:::healthcare-data/*",            "Condition": {                "Bool": {                    "aws:SecureTransport": "true"                }            }        },        {            "Effect": "Allow",            "Action": "kms:Decrypt",            "Resource": "*"        }    ] }. During a security audit, the team discovers that the IAM role's KMS permission is too permissive because it allows decryption of any KMS key in the account. The team needs to modify the policy to follow the principle of least privilege while still allowing the SageMaker endpoint to read the encrypted data. Which modification should the team make?

Accepted Answer

Change the KMS statement to: "Action": "kms:Decrypt", "Resource": "arn:aws:kms:us-east-1:123456789012:key/1234abcd-12ab-34cd-56ef-1234567890ab". The current policy allows kms:Decrypt on any KMS key (*). To follow least privilege, the team should restrict the Resource to the specific KMS key used to encrypt the S3 bucket. Option A (change the Action to kms:Decrypt and restrict Resource to the specific key ARN) is correct. Option B (remove the KMS statement entirely) would break the endpoint because it cannot decrypt the data. Option C (add a condition for specific encryption context) is good practice but still allows decryption of any key if condition is met, not least privilege. Option D (use kms:DescribeKey instead of kms:Decrypt) does not allow decryption.

Answer

Change the KMS statement Action to "kms:DescribeKey" instead of "kms:Decrypt"

Answer

Add a condition to the KMS statement: "Condition": { "StringEquals": { "kms:ViaService": "s3.us-east-1.amazonaws.com" } }

Answer

Remove the KMS statement entirely, as S3 bucket policies with SSE-KMS do not require KMS permissions

Question 10

An e-commerce company uses a SageMaker endpoint to serve a product recommendation model. The model is retrained every month using batch transforms. The ML team has set up a retraining pipeline using SageMaker Processing jobs and Step Functions. Recently, the Step Functions workflow has been failing at the retraining step with an error: 'AccessDeniedException: User: arn:aws:sts::123456789012:assumed-role/RetrainingRole/abc123 is not authorized to perform: s3:GetObject on resource: arn:aws:s3:::training-data/processed/latest.parquet'. The team confirms that the S3 bucket exists and the object is present. The retraining role has the following policy: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject" ], "Resource": "arn:aws:s3:::training-data/*" } ] }. The team also verifies that the bucket policy does not explicitly deny access. What is the MOST likely cause of the AccessDenied error?

Accepted Answer

The training data object uses server-side encryption with AWS KMS (SSE-KMS), and the retraining role lacks kms:Decrypt permission on the KMS key. The error indicates that the retraining role is not authorized to GetObject on the specific object. Even though the policy allows 'arn:aws:s3:::training-data/*', if the object is encrypted with SSE-KMS, the role also needs kms:Decrypt permission on the KMS key. The bucket policy might also require encryption. Option B is the most likely cause. Option A (wrong region) would give a different error. Option C (lack of S3 bucket policy) is not the issue if there is no explicit deny. Option D (path typo) would result in a 404 Not Found error, not AccessDenied.

Answer

The Step Functions execution role does not have permission to invoke the SageMaker Processing job

Answer

The path in the error message is misspelled; the actual object is at a different key

Answer

The S3 bucket has a bucket policy that denies access to the retraining role based on a condition like aws:SourceIp

Question 11

A gaming company uses a SageMaker endpoint for real-time player churn prediction. The model is updated weekly. After a recent retraining, the team notices that the endpoint's predicted probabilities for churn have shifted dramatically: the average predicted probability dropped from 0.3 to 0.05. The team suspects concept drift (the relationship between features and target changed) rather than data drift. They have SageMaker Model Monitor set up for data drift and quality metrics, but not for bias or explainability. The team needs to confirm concept drift and take corrective action. Which approach should the team take FIRST?

Accepted Answer

Configure SageMaker Model Monitor's model quality monitoring to compare predictions against actual outcomes collected from a week of production traffic. To detect concept drift, the team needs to compare the model's predictions against actual observed outcomes (ground truth). SageMaker Model Monitor's quality monitoring can track prediction accuracy over time if ground truth is provided. Option D (set up Model Monitor's model quality monitoring) is the correct first step. Option A (retrain with more recent data) might help but does not confirm drift. Option B (data drift monitoring) checks feature distribution, not concept drift. Option C (use Clarify for SHAP values) is for feature importance, not drift detection.

Answer

Immediately retrain the model using the most recent month of data and redeploy to the endpoint

Answer

Use Amazon SageMaker Clarify to compute SHAP values and understand which features are driving the new predictions

Answer

Investigate data drift by reviewing the Model Monitor feature distribution constraints and comparing recent input data to the baseline

Question 12

A machine learning team has deployed a model using Amazon SageMaker and wants to set up continuous monitoring for data drift. Which TWO actions are essential for ongoing data drift detection?

Accepted Answer

Enable data capture on the SageMaker endpoint to store inference data in Amazon S3.. Option A (Enable data capture) is essential because data capture collects inference requests and responses, which are required for monitoring. Option C (Configure Model Monitor to run hourly) is essential because Model Monitor analyzes the captured data against a baseline to detect drift. Option B is a prerequisite but not an ongoing action. Options D and E are unrelated to data drift monitoring.

Answer

Set up Amazon CloudWatch alarms on the endpoint's invocation latency metric.

Answer

Deploy a shadow endpoint to compare predictions from the current model and a challenger model.

Answer

Create a baseline from the training data to serve as a reference distribution.

Question 13

A financial services company has deployed a machine learning model using Amazon SageMaker to predict loan default risk. The model is hosted on a real-time endpoint and uses a SageMaker Model Monitor schedule to check for data drift every hour. The monitoring schedule has been running for a month without issues. Starting last week, the data science team noticed that the endpoint's invocation latency has increased by 300% and error rates have spiked to 5% from a baseline of 0.1%. The team suspects the model is receiving out-of-distribution data that is causing longer processing times and occasional timeouts. They have active CloudWatch alarms on latency and error rates but no alarms on data drift. The Model Monitor schedule shows no failures in its status. The team needs to quickly identify whether data drift is the root cause and take corrective action. Which course of action should the team take to diagnose and address the issue?

Accepted Answer

Use the Model Monitor's built-in baseline drift analysis on the captured inference data stored in Amazon S3, and run an Amazon CloudWatch Logs Insights query on the endpoint logs to identify specific input features that have changed distribution.. Option A is correct because analyzing the captured inference data against the baseline using Model Monitor's built-in drift analysis will directly determine if data drift exists, and the log insights query can pinpoint which features have changed. Option B is wrong because SageMaker Debugger is for training-time debugging, not for inference data drift. Option C is wrong because retraining without diagnosing wastes resources if drift is not the cause. Option D is wrong because increasing endpoint capacity addresses symptoms but not the root cause, and may not fix errors due to out-of-distribution data.

Answer

Retrain the model using the latest training data from the last month and deploy a new endpoint to replace the current one.

Answer

Increase the endpoint's instance count and enable auto-scaling to handle the increased latency and errors.

Answer

Enable SageMaker Debugger on the endpoint to capture inference tensors and compare them to training tensor distributions.

Question 14

A company wants to monitor its Amazon SageMaker real-time endpoint for data quality issues. Which TWO actions should the company take?

Accepted Answer

Create a baseline from the training data to compare against live data.. To monitor data quality with SageMaker Model Monitor, you need to enable data capture on the endpoint and create a baseline from the training data. The other options are not directly required for data quality monitoring.

Answer

Use SageMaker Debugger to analyze training jobs.

Answer

Set up an AWS Lambda function to preprocess incoming requests.

Answer

Configure Amazon S3 bucket notifications for model artifacts.

Question 15

A machine learning engineer sees the above error in Amazon CloudWatch Logs for a SageMaker endpoint. What is the most likely cause?

Accepted Answer

The inference code in the Docker container outputs a different response format than expected by the endpoint.. The error indicates the model returned a response with an unexpected structure. The expected format was a JSON with a 'predictions' array, but the model output a single 'prediction' field. This mismatch is typically due to a bug in the inference code within the Docker container, not corruption, overload, or misconfiguration of data capture.

Answer

The model file is corrupted during deployment.

Answer

The data capture configuration is incorrectly set to capture only the response body.

Answer

The endpoint is overloaded and dropping requests.