MLA-C01 ML Solution Monitoring, Maintenance and Security Practice Test 1 — 15 Questions

Question 1

A machine learning engineer at a retail company is monitoring a production model that predicts inventory demand. The model's prediction accuracy has dropped significantly over the past week. The engineer checks the model's input data and notices a new product category was introduced with a different distribution. Which concept is most likely causing the performance degradation?

Accepted Answer

Covariate shift. B is correct because covariate shift occurs when the distribution of the input features changes while the relationship between features and the target remains the same. In this scenario, the introduction of a new product category with a different distribution alters the input data distribution, causing the model to encounter unseen patterns and degrade in prediction accuracy.

Answer

Concept drift

Answer

Data leakage

Answer

Model decay

Question 2

A data science team is using Amazon SageMaker to train and deploy a binary classification model. They want to continuously monitor the model for data drift in production. Which combination of AWS services and SageMaker features should they use to implement automated drift detection with minimal operational overhead?

Accepted Answer

SageMaker Model Monitor and Amazon CloudWatch. SageMaker Model Monitor is the native SageMaker feature designed specifically for continuously monitoring deployed models for data drift, bias drift, and feature attribution drift. It automatically captures inference requests and responses, computes statistics, and publishes metrics to Amazon CloudWatch, which can trigger alarms for drift detection. This combination provides automated drift detection with minimal operational overhead because it requires no custom infrastructure or manual scheduling.

Answer

SageMaker Debugger and Amazon SNS

Answer

SageMaker Pipelines and AWS Lambda

Answer

SageMaker Clarify and AWS Config

Question 3

A financial services company uses a custom container on Amazon SageMaker to serve a fraud detection model. The model's inference latency has recently increased, causing timeouts for some requests. The team reviews the SageMaker logs and finds that the container is consuming more memory than allocated. What should the team do to maintain service quality while ensuring cost-effectiveness?

Accepted Answer

Change the instance type to a memory-optimized instance, such as r5.large. The correct answer is D because the root cause is that the container is consuming more memory than allocated, leading to increased latency and timeouts. Switching to a memory-optimized instance like r5.large directly addresses the memory constraint by providing more memory per vCPU, which resolves the performance issue without over-provisioning compute resources. This approach is cost-effective because it targets the specific bottleneck (memory) rather than scaling out or changing unrelated parameters.

Answer

Decrease the model's batch size to reduce memory usage

Answer

Increase the number of instances in the endpoint to distribute the load

Answer

Implement an auto-scaling policy based on memory utilization

Question 4

A machine learning team is building a CI/CD pipeline for model deployment using Amazon SageMaker. They need to ensure that all model artifacts are encrypted at rest and in transit, and that access to the models is controlled via IAM. Which TWO actions should the team take to meet these requirements? (Choose TWO.)

Accepted Answer

Configure the SageMaker notebook instance to use a KMS key for encryption. Option D is correct because configuring a SageMaker notebook instance to use a KMS key ensures that data at rest on the notebook's storage volume (e.g., EBS) is encrypted. This directly addresses the requirement for encryption at rest for model artifacts during development. Option E is correct because using HTTPS endpoints for invoking the SageMaker model ensures encryption in transit via TLS, protecting data as it moves between clients and the model endpoint.

Answer

Set the SageMaker model's 'EnableNetworkIsolation' parameter to true

Answer

Enable default encryption on the S3 bucket that stores model artifacts

Answer

Enable AWS CloudTrail to log all API calls to SageMaker

Question 5

A healthcare company deploys a model to predict patient readmission risk. The model was trained on historical data and is now showing signs of concept drift. The team needs to implement a monitoring solution that can detect drift and automatically retrain the model when drift is detected. Which THREE steps should the team take to build this solution? (Choose THREE.)

Accepted Answer

Deploy SageMaker Model Monitor to track prediction quality over time. A is correct because Amazon SageMaker Model Monitor can continuously track prediction quality metrics (e.g., accuracy, precision) over time by analyzing data captured from the endpoint. This allows the team to detect concept drift by comparing live predictions against a baseline, triggering alerts when performance degrades. It provides a managed, automated way to monitor model quality without manual intervention.

Answer

Disable the existing endpoint to prevent stale predictions during retraining

Answer

Manually compare the model's predictions against a holdout validation set each week

Question 6

A company is using Amazon SageMaker to host a real-time inference endpoint. They want to restrict access to the endpoint to only a specific VPC and require authentication using AWS IAM. Which TWO configuration steps should they take to achieve this? (Choose TWO.)

Accepted Answer

Configure the endpoint to be deployed in a private subnet within the VPC. Option A is correct because deploying the SageMaker endpoint in a private subnet within the VPC ensures that the endpoint is not publicly accessible and can only be reached from within that VPC. This is achieved by using a VPC interface endpoint (AWS PrivateLink) or by placing the endpoint directly in the VPC, which restricts network traffic to the VPC boundary.

Answer

Attach a resource-based policy to the endpoint that denies all traffic except from the VPC

Answer

Place the endpoint behind Amazon CloudFront to act as a proxy

Answer

Use a public subnet and configure a security group to allow only the company's IP range

Question 7

A machine learning engineer is troubleshooting a model that is producing unexpectedly low accuracy in production. The engineer examines the model's training data and finds that the distribution of the target variable in production is significantly different from the training set. What type of drift is the model experiencing?

Accepted Answer

Concept drift. Option B is correct because a change in the target variable distribution is concept drift. Option A is wrong because covariate shift is input distribution change. Option C is wrong because prior probability shift is a type of concept drift, but not the best answer here. Option D is wrong because data drift is a general term.

Answer

Prior probability shift

Answer

Data drift

Answer

Covariate shift

Question 8

A team deploys a machine learning model using a SageMaker endpoint with an ML.T4 instance. After a week, they notice that the endpoint's CPU utilization is consistently below 10% and latency is low. However, the endpoint is incurring high costs. Which action should the team take to reduce costs while maintaining the ability to serve traffic?

Accepted Answer

Migrate to a SageMaker Serverless Inference endpoint. Option C is correct because a serverless inference endpoint scales to zero when not in use, reducing cost. Option A is wrong because multi-model endpoints still have always-on instances. Option B is wrong because reducing instances may cause throttling. Option D is wrong because asynchronous inference is for batch, not real-time.

Answer

Switch to a multi-model endpoint to share instances across models

Answer

Reduce the number of instances to one

Answer

Implement an asynchronous inference endpoint

Question 9

A company is using Amazon SageMaker to train a model on sensitive customer data. The security team requires that all data be encrypted in transit and at rest, and that the training job does not have internet access. Which configuration should the team use to meet these requirements?

Accepted Answer

Configure the training job to run in a private subnet with no internet access, and use a KMS key for encryption. Option C is correct because running the SageMaker training job in a private subnet with no internet access ensures the job cannot reach the public internet, satisfying the no-internet-access requirement. Using an AWS KMS key for encryption at rest (for the S3 bucket and EBS volumes) and enforcing encryption in transit (via HTTPS/TLS for SageMaker and S3 endpoints) meets the encryption requirements. SageMaker training jobs in a private subnet use VPC endpoints (e.g., S3 and SageMaker API endpoints) to communicate securely without internet access.

Answer

Configure the training job to run in a public subnet with a security group that blocks outbound traffic

Answer

Configure the training job to run in a private subnet, but disable encryption to reduce latency

Answer

Configure the training job to run in a VPC with a NAT gateway, and use default SageMaker encryption

Question 10

A company has a SageMaker endpoint that uses a trained model to classify images. The endpoint is experiencing high latency and the team suspects it is due to the model size. Which action can the team take to reduce latency without significantly impacting accuracy?

Accepted Answer

Use SageMaker Neo to compile the model for the target instance. SageMaker Neo compiles trained models into an optimized binary for the target hardware, applying techniques like operator fusion, memory layout optimization, and quantization. This reduces model size and inference latency while preserving accuracy, making it the correct choice for addressing high latency caused by model size.

Answer

Switch to a compute-optimized instance type

Answer

Reduce the batch size of inference requests

Answer

Convert the model to ONNX format

Question 11

A data science team deploys a regression model using Amazon SageMaker. After one week, the model's prediction accuracy drops significantly. The team needs to detect this degradation automatically and trigger retraining. Which AWS service should they use to monitor the model's performance over time and set up alerts?

Accepted Answer

Amazon SageMaker Model Monitor. Amazon SageMaker Model Monitor is the correct choice because it is purpose-built to continuously monitor machine learning models deployed on SageMaker endpoints for data drift, feature attribution drift, and prediction quality degradation. It automatically compares live inference data against a baseline, triggers alerts when performance drops, and can be configured to initiate retraining pipelines via AWS Lambda or Step Functions, directly addressing the need to detect accuracy degradation and trigger retraining.

Answer

AWS CloudWatch

Answer

Amazon Inspector

Answer

AWS Config

Question 12

A company uses Amazon SageMaker to host a real-time inference endpoint for a fraud detection model. The endpoint is deployed with three instances of ml.m5.large. The model processes each request in about 200 ms. Lately, users report occasional timeouts (requests taking >5 seconds). The team suspects model drift or data skew. What is the MOST likely cause and solution?

Accepted Answer

A recent change increased the average input size, causing longer inference time; investigate input preprocessing.. Option B is correct because the symptom of occasional timeouts (>5 seconds) on a model that normally processes requests in ~200 ms suggests that a recent change in input data characteristics (e.g., larger payloads or more complex features) is causing sporadic latency spikes. Investigating input preprocessing can identify if data skew or increased input size is overwhelming the model's inference path, which is a common monitoring concern in SageMaker real-time endpoints.

Answer

The instances are under-provisioned; switch to ml.m5.xlarge instances.

Answer

The endpoint is experiencing too many concurrent requests; add more instances.

Answer

Model drift caused the model to become computationally heavier; retrain the model.

Question 13

A machine learning engineer deploys a model to an Amazon SageMaker endpoint with data capture enabled. The endpoint uses a production variant with initial instance count of 2. After a week, they notice that the captured data is not being sent to the specified Amazon S3 bucket. The IAM role used by the endpoint has the following policy attached. What is the MOST likely reason for the failure?

Accepted Answer

The IAM role does not have permission to write to the correct S3 prefix.. Option C is correct because the IAM role attached to the SageMaker endpoint must have write permissions to the exact S3 prefix where data capture is configured. The policy shown likely grants access to a broader bucket or a different prefix, but not the specific path (e.g., s3://bucket-name/prefix/) that the endpoint's DataCaptureConfig specifies. Without s3:PutObject on that exact prefix, the captured data fails to upload silently.

Answer

The S3 bucket does not exist.

Answer

The S3 bucket uses AWS KMS encryption and the role lacks kms:Decrypt permission.

Answer

The IAM role does not have s3:ListBucket permission.

Question 14

A company uses Amazon Rekognition to moderate user-generated images. They want to set up a monitoring system that alerts the team if the number of inappropriate images flagged by the model exceeds a threshold. Which combination of AWS services should they use?

Accepted Answer

Amazon CloudWatch to publish custom metrics and create an alarm, and AWS Lambda to process images and publish metrics.. Option A is correct because Amazon CloudWatch can publish custom metrics (number of flagged images) and trigger alarms. AWS Lambda can process the images and publish the metric. Option B is incomplete because CloudWatch Logs alone cannot trigger alarms on custom metrics. Option C is incorrect because SNS alone does not monitor metrics. Option D is incorrect because AWS Config is for configuration tracking.

Answer

Amazon CloudWatch Logs to store inference logs and create a metric filter.

Answer

AWS Config to track resource changes and trigger an SNS notification.

Answer

Amazon Simple Notification Service (SNS) to send alerts when threshold is exceeded.

Question 15

A team deploys a PyTorch model on Amazon SageMaker for real-time inference. They notice that inference latency is higher than expected. They suspect the serialization format used for input data is inefficient. Which approach would MOST likely reduce latency?

Accepted Answer

Change the input serialization format to Protocol Buffers.. Protocol Buffers (protobuf) are a binary serialization format that is significantly more compact and faster to parse than text-based formats like JSON or CSV. By reducing the size of the input data and the CPU overhead of deserialization, switching to protobuf directly addresses the root cause of high inference latency on SageMaker real-time endpoints.

Answer

Use Amazon SageMaker Batch Transform instead of real-time inference.

Answer

Enable automatic scaling on the endpoint.

Answer

Increase the instance type to a compute-optimized instance.