MLA-C01 ML Solution Monitoring, Maintenance and Security Practice Test 5 — 15 Questions

Question 1

An e-commerce company uses a machine learning model to predict customer churn. They notice that the model's performance degrades after a major marketing campaign changes customer behavior. Which approach is MOST effective to detect and respond to this type of concept drift?

Accepted Answer

Use SageMaker Model Monitor to track prediction distribution and trigger retraining.. SageMaker Model Monitor can automatically detect drift in prediction distributions and trigger retraining pipelines.

Answer

Deploy an A/B test to compare the current model with a baseline.

Answer

Manually review model accuracy each month.

Answer

Set up a weekly batch transform job to compute accuracy against historical data.

Answer

Increase the number of instances for the endpoint.

Question 2

A company's ML pipeline runs in multiple AWS accounts (dev, test, prod). They want to enforce that only approved models from a central Model Registry can be deployed to the production account. Which combination of services is MOST appropriate to implement this governance?

Accepted Answer

AWS Organizations with SCPs, AWS CodePipeline with cross-account actions, and SageMaker Model Registry with approval status.. AWS Organizations SCPs restrict actions, CodePipeline automates cross-account deployment, and Model Registry provides approval gates.

Answer

AWS Config, Amazon GuardDuty, and AWS Security Hub.

Answer

Amazon API Gateway, AWS Step Functions, and Amazon DynamoDB.

Answer

AWS Service Catalog, AWS KMS, and AWS CloudTrail.

Answer

AWS CloudFormation StackSets, Amazon EventBridge, and AWS Lambda.

Question 3

A SageMaker training job has been running for several hours but shows no progress. The job is using a custom Docker container. The engineer suspects a bug in the training script. Which tool is BEST to debug the training job without stopping it?

Accepted Answer

SageMaker Debugger. SageMaker Debugger can monitor training in real-time, capture tensors, and detect anomalies without interrupting the job.

Answer

Amazon CloudWatch Logs

Answer

SageMaker Local Mode

Answer

SageMaker Processing

Answer

SageMaker Profiler

Question 4

A company runs a real-time inference endpoint with an auto-scaling policy based on average CPU utilization. During a traffic spike, the endpoint scales out but takes several minutes to become healthy, causing increased latency. The endpoint uses a large instance type. Which change would MOST effectively reduce the time to scale out?

Accepted Answer

Use a pre-warmed endpoint with a target tracking scaling policy.. Pre-warming the endpoint with a target tracking scaling policy maintains a baseline of ready instances, reducing cold start time.

Answer

Switch to a smaller instance type.

Answer

Enable SageMaker Inference Recommender to optimize instance type.

Answer

Implement a canary deployment with a blue/green strategy.

Answer

Set a lower scaling cooldown period.

Question 5

A company wants to automate the deployment of a SageMaker model into production whenever a new model version is approved in the Model Registry. Which service can be used to trigger the deployment pipeline?

Accepted Answer

Amazon CloudWatch Events (EventBridge). Amazon EventBridge can respond to Model Registry events (e.g., approval status change) and start an automated pipeline.

Answer

AWS Lambda

Answer

Amazon S3 Events

Answer

Amazon SNS

Answer

AWS Config

Question 6

A data science team deployed a model on Amazon SageMaker and enabled Model Monitor to detect data drift. After a week, they receive alerts indicating that the distribution of a key feature has shifted significantly. However, the model's accuracy on the recent production data remains high. Which action should the team take next?

Accepted Answer

Investigate the root cause of the drift as it may be benign or may lead to future degradation.. Option A is correct because data drift does not always immediately impact accuracy; it's important to investigate. Option B is wrong because retraining without investigation may be wasteful. Option C is wrong because drift can become problematic later. Option D is wrong because sample size is not the issue; the drift is real.

Answer

Disable the data drift alert since accuracy is not affected.

Answer

Increase the sample size for monitoring to reduce false positives.

Answer

Retrain the model immediately because data drift always degrades performance.

Question 7

A company's SageMaker real-time endpoint is experiencing high latency under load. The CloudWatch metrics show that the ModelLatency is acceptable, but the OverheadLatency is spiking. What is the most likely cause?

Accepted Answer

The endpoint is under-provisioned with insufficient instance count.. Option C is correct because OverheadLatency includes SageMaker framework overhead, which increases when the endpoint is scaled improperly. Option A would affect ModelLatency. Option B would increase latency but not specifically OverheadLatency. Option D would affect network latency but not OverheadLatency.

Answer

The request payload size is too large.

Answer

The SageMaker endpoint is not in the same VPC as the client.

Answer

The model inference code is inefficient.

Question 8

A machine learning engineer wants to encrypt model artifacts stored in Amazon S3. The artifacts are created and used by SageMaker training jobs and endpoints. What is the simplest way to ensure encryption at rest?

Accepted Answer

Create an S3 bucket with default encryption using SSE-S3 and allow SageMaker access.. Option A is correct because SSE-S3 is the simplest encryption method and works seamlessly with SageMaker. Option B is not a thing; SageMaker does not have default encryption for artifacts. Option C is possible but not simplest. Option D is for versioning, not encryption.

Answer

Use SageMaker's default encryption with an AWS managed key.

Answer

Enable S3 bucket versioning and MFA delete.

Answer

Use a custom KMS key and grant SageMaker permission to use it.

Question 9

A team uses SageMaker Clarify to monitor bias drift in production. They schedule weekly analysis. After a month, Clarify reports a significant increase in a bias metric. What should the team do first?

Accepted Answer

Review the analysis report to understand which feature and segment contributed to the drift.. Option D is correct because reviewing the report helps understand which feature and segment contributed to the drift. Option A is premature without understanding the cause. Option B is ignoring the issue. Option C does not address the drift source.

Answer

Disable the bias monitor because the metric may be noisy.

Answer

Immediately retrain the model with a balanced dataset.

Answer

Increase the frequency of analysis to daily.

Question 10

A model deployed on SageMaker uses custom inference code. The endpoint is showing intermittent 500 errors. CloudWatch logs reveal 'TimeoutError: Request timed out after 60 seconds'. The model takes on average 55 seconds to process. What is the most effective solution?

Accepted Answer

Optimize the inference code to reduce latency.. Option A is correct because the timeout is at the container level, but the issue is latency near limit; optimizing code is most effective. Option B might help with load but not per-request latency. Option C does not exist (invocation timeout is set by client but server-side timeout is 60s default). Option D: container timeout can be increased, but default is 60s; increasing might mask performance issues.

Answer

Increase the invocation timeout in the SageMaker API call.

Answer

Increase the SageMaker endpoint's model container timeout setting.

Answer

Increase the endpoint's instance count.

Question 11

A company requires that all SageMaker notebook instances be created within a private VPC without internet access. Which configuration step is mandatory?

Accepted Answer

Configure VPC settings when creating the notebook instance, choosing a private subnet.. Option B is correct because you must choose a private subnet when creating the notebook instance. Option A would enable internet access. Option C is not a valid setting. Option D would assign a public IP, which would provide internet access.

Answer

Use a SageMaker Studio notebook instead.

Answer

Enable SageMaker direct internet access.

Answer

Assign a public IP to the notebook instance.

Question 12

A team uses SageMaker Model Monitor to track data quality. They notice that the monitor's constraint violations are increasing but the model performance remains good. What should they do?

Accepted Answer

Investigate the specific features that are violating constraints to see if they are still relevant.. Option A is correct because investigating specific violating features helps determine if constraints are still relevant. Option B is reactive without analysis. Option C ignores the issue. Option D may be unnecessary if performance is still good.

Answer

Disable the monitor because it is not affecting performance.

Answer

Relax the constraint thresholds to reduce alerts.

Answer

Retrain the model using the latest data.

Question 13

A machine learning engineer is setting up automated retraining for a model using SageMaker Pipelines. The pipeline should trigger when a data drift alert is received from Model Monitor. Which event source should the engineer use to initiate the pipeline?

Accepted Answer

Amazon CloudWatch Events (Amazon EventBridge) rule that captures Model Monitor outcome.. Option D is correct because SageMaker Model Monitor publishes drift results as Amazon CloudWatch Events (now EventBridge). Option A: S3 events on monitoring output are not directly linked to drift alerts. Option B: no webhook exists. Option C: polling is inefficient.

Answer

AWS Lambda function that polls CloudWatch logs.

Answer

S3 event notification on the monitoring output bucket.

Answer

SageMaker model monitor webhook.

Question 14

A data scientist wants to version control trained models and manage approvals for deployment. Which SageMaker feature should they use?

Accepted Answer

SageMaker Model Registry.. Option B is correct because SageMaker Model Registry is purpose-built for model versioning and approval workflows. Option A is for experiment tracking, not deployment. Option C is for labeling data. Option D is for feature storage.

Answer

SageMaker Experiments.

Answer

SageMaker Feature Store.

Answer

SageMaker Ground Truth.

Question 15

A company wants to secure access to a SageMaker real-time endpoint. Which TWO actions should be taken? (Select two.)

Accepted Answer

Attach a resource-based policy to the endpoint.. Options A and D are correct. A: Attaching a resource-based policy allows fine-grained control over who can invoke the endpoint. D: Configuring the endpoint within a VPC and using VPC endpoints improves network security. B is incorrect because IAM roles with sts:AssumeRole are not typically used for endpoint invocation. C is incorrect because AWS WAF does not integrate with SageMaker endpoints. E is about logging, not access control.

Answer

Use an IAM role with sts:AssumeRole for invocation.

Answer

Enable AWS WAF on the endpoint.

Answer

Use AWS CloudTrail to log all invocations.