MLA-C01 ML Solution Monitoring, Maintenance and Security — 30 Questions

Question 1

A machine learning engineer at a retail company is monitoring a production model that predicts inventory demand. The model's prediction accuracy has dropped significantly over the past week. The engineer checks the model's input data and notices a new product category was introduced with a different distribution. Which concept is most likely causing the performance degradation?

Accepted Answer

Covariate shift. B is correct because covariate shift occurs when the distribution of the input features changes while the relationship between features and the target remains the same. In this scenario, the introduction of a new product category with a different distribution alters the input data distribution, causing the model to encounter unseen patterns and degrade in prediction accuracy.

Answer

Concept drift

Answer

Data leakage

Answer

Model decay

Question 2

A data science team is using Amazon SageMaker to train and deploy a binary classification model. They want to continuously monitor the model for data drift in production. Which combination of AWS services and SageMaker features should they use to implement automated drift detection with minimal operational overhead?

Accepted Answer

SageMaker Model Monitor and Amazon CloudWatch. SageMaker Model Monitor is the native SageMaker feature designed specifically for continuously monitoring deployed models for data drift, bias drift, and feature attribution drift. It automatically captures inference requests and responses, computes statistics, and publishes metrics to Amazon CloudWatch, which can trigger alarms for drift detection. This combination provides automated drift detection with minimal operational overhead because it requires no custom infrastructure or manual scheduling.

Answer

SageMaker Debugger and Amazon SNS

Answer

SageMaker Pipelines and AWS Lambda

Answer

SageMaker Clarify and AWS Config

Question 3

A financial services company uses a custom container on Amazon SageMaker to serve a fraud detection model. The model's inference latency has recently increased, causing timeouts for some requests. The team reviews the SageMaker logs and finds that the container is consuming more memory than allocated. What should the team do to maintain service quality while ensuring cost-effectiveness?

Accepted Answer

Change the instance type to a memory-optimized instance, such as r5.large. The correct answer is D because the root cause is that the container is consuming more memory than allocated, leading to increased latency and timeouts. Switching to a memory-optimized instance like r5.large directly addresses the memory constraint by providing more memory per vCPU, which resolves the performance issue without over-provisioning compute resources. This approach is cost-effective because it targets the specific bottleneck (memory) rather than scaling out or changing unrelated parameters.

Answer

Decrease the model's batch size to reduce memory usage

Answer

Increase the number of instances in the endpoint to distribute the load

Answer

Implement an auto-scaling policy based on memory utilization

Question 4

A machine learning team is building a CI/CD pipeline for model deployment using Amazon SageMaker. They need to ensure that all model artifacts are encrypted at rest and in transit, and that access to the models is controlled via IAM. Which TWO actions should the team take to meet these requirements? (Choose TWO.)

Accepted Answer

Configure the SageMaker notebook instance to use a KMS key for encryption. Option D is correct because configuring a SageMaker notebook instance to use a KMS key ensures that data at rest on the notebook's storage volume (e.g., EBS) is encrypted. This directly addresses the requirement for encryption at rest for model artifacts during development. Option E is correct because using HTTPS endpoints for invoking the SageMaker model ensures encryption in transit via TLS, protecting data as it moves between clients and the model endpoint.

Answer

Set the SageMaker model's 'EnableNetworkIsolation' parameter to true

Answer

Enable default encryption on the S3 bucket that stores model artifacts

Answer

Enable AWS CloudTrail to log all API calls to SageMaker

Question 5

A healthcare company deploys a model to predict patient readmission risk. The model was trained on historical data and is now showing signs of concept drift. The team needs to implement a monitoring solution that can detect drift and automatically retrain the model when drift is detected. Which THREE steps should the team take to build this solution? (Choose THREE.)

Accepted Answer

Deploy SageMaker Model Monitor to track prediction quality over time. A is correct because Amazon SageMaker Model Monitor can continuously track prediction quality metrics (e.g., accuracy, precision) over time by analyzing data captured from the endpoint. This allows the team to detect concept drift by comparing live predictions against a baseline, triggering alerts when performance degrades. It provides a managed, automated way to monitor model quality without manual intervention.

Answer

Disable the existing endpoint to prevent stale predictions during retraining

Answer

Manually compare the model's predictions against a holdout validation set each week

Question 6

A company is using Amazon SageMaker to host a real-time inference endpoint. They want to restrict access to the endpoint to only a specific VPC and require authentication using AWS IAM. Which TWO configuration steps should they take to achieve this? (Choose TWO.)

Accepted Answer

Configure the endpoint to be deployed in a private subnet within the VPC. Option A is correct because deploying the SageMaker endpoint in a private subnet within the VPC ensures that the endpoint is not publicly accessible and can only be reached from within that VPC. This is achieved by using a VPC interface endpoint (AWS PrivateLink) or by placing the endpoint directly in the VPC, which restricts network traffic to the VPC boundary.

Answer

Attach a resource-based policy to the endpoint that denies all traffic except from the VPC

Answer

Place the endpoint behind Amazon CloudFront to act as a proxy

Answer

Use a public subnet and configure a security group to allow only the company's IP range

Question 7

A machine learning engineer is troubleshooting a model that is producing unexpectedly low accuracy in production. The engineer examines the model's training data and finds that the distribution of the target variable in production is significantly different from the training set. What type of drift is the model experiencing?

Accepted Answer

Concept drift. Option B is correct because a change in the target variable distribution is concept drift. Option A is wrong because covariate shift is input distribution change. Option C is wrong because prior probability shift is a type of concept drift, but not the best answer here. Option D is wrong because data drift is a general term.

Answer

Prior probability shift

Answer

Data drift

Answer

Covariate shift

Question 8

A team deploys a machine learning model using a SageMaker endpoint with an ML.T4 instance. After a week, they notice that the endpoint's CPU utilization is consistently below 10% and latency is low. However, the endpoint is incurring high costs. Which action should the team take to reduce costs while maintaining the ability to serve traffic?

Accepted Answer

Migrate to a SageMaker Serverless Inference endpoint. Option C is correct because a serverless inference endpoint scales to zero when not in use, reducing cost. Option A is wrong because multi-model endpoints still have always-on instances. Option B is wrong because reducing instances may cause throttling. Option D is wrong because asynchronous inference is for batch, not real-time.

Answer

Switch to a multi-model endpoint to share instances across models

Answer

Reduce the number of instances to one

Answer

Implement an asynchronous inference endpoint

Question 9

A company is using Amazon SageMaker to train a model on sensitive customer data. The security team requires that all data be encrypted in transit and at rest, and that the training job does not have internet access. Which configuration should the team use to meet these requirements?

Accepted Answer

Configure the training job to run in a private subnet with no internet access, and use a KMS key for encryption. Option C is correct because running the SageMaker training job in a private subnet with no internet access ensures the job cannot reach the public internet, satisfying the no-internet-access requirement. Using an AWS KMS key for encryption at rest (for the S3 bucket and EBS volumes) and enforcing encryption in transit (via HTTPS/TLS for SageMaker and S3 endpoints) meets the encryption requirements. SageMaker training jobs in a private subnet use VPC endpoints (e.g., S3 and SageMaker API endpoints) to communicate securely without internet access.

Answer

Configure the training job to run in a public subnet with a security group that blocks outbound traffic

Answer

Configure the training job to run in a private subnet, but disable encryption to reduce latency

Answer

Configure the training job to run in a VPC with a NAT gateway, and use default SageMaker encryption

Question 10

A company has a SageMaker endpoint that uses a trained model to classify images. The endpoint is experiencing high latency and the team suspects it is due to the model size. Which action can the team take to reduce latency without significantly impacting accuracy?

Accepted Answer

Use SageMaker Neo to compile the model for the target instance. SageMaker Neo compiles trained models into an optimized binary for the target hardware, applying techniques like operator fusion, memory layout optimization, and quantization. This reduces model size and inference latency while preserving accuracy, making it the correct choice for addressing high latency caused by model size.

Answer

Switch to a compute-optimized instance type

Answer

Reduce the batch size of inference requests

Answer

Convert the model to ONNX format

Question 11

A data science team deploys a regression model using Amazon SageMaker. After one week, the model's prediction accuracy drops significantly. The team needs to detect this degradation automatically and trigger retraining. Which AWS service should they use to monitor the model's performance over time and set up alerts?

Accepted Answer

Amazon SageMaker Model Monitor. Amazon SageMaker Model Monitor is the correct choice because it is purpose-built to continuously monitor machine learning models deployed on SageMaker endpoints for data drift, feature attribution drift, and prediction quality degradation. It automatically compares live inference data against a baseline, triggers alerts when performance drops, and can be configured to initiate retraining pipelines via AWS Lambda or Step Functions, directly addressing the need to detect accuracy degradation and trigger retraining.

Answer

AWS CloudWatch

Answer

Amazon Inspector

Answer

AWS Config

Question 12

A company uses Amazon SageMaker to host a real-time inference endpoint for a fraud detection model. The endpoint is deployed with three instances of ml.m5.large. The model processes each request in about 200 ms. Lately, users report occasional timeouts (requests taking >5 seconds). The team suspects model drift or data skew. What is the MOST likely cause and solution?

Accepted Answer

A recent change increased the average input size, causing longer inference time; investigate input preprocessing.. Option B is correct because the symptom of occasional timeouts (>5 seconds) on a model that normally processes requests in ~200 ms suggests that a recent change in input data characteristics (e.g., larger payloads or more complex features) is causing sporadic latency spikes. Investigating input preprocessing can identify if data skew or increased input size is overwhelming the model's inference path, which is a common monitoring concern in SageMaker real-time endpoints.

Answer

The instances are under-provisioned; switch to ml.m5.xlarge instances.

Answer

The endpoint is experiencing too many concurrent requests; add more instances.

Answer

Model drift caused the model to become computationally heavier; retrain the model.

Question 13

A machine learning engineer deploys a model to an Amazon SageMaker endpoint with data capture enabled. The endpoint uses a production variant with initial instance count of 2. After a week, they notice that the captured data is not being sent to the specified Amazon S3 bucket. The IAM role used by the endpoint has the following policy attached. What is the MOST likely reason for the failure?

Accepted Answer

The IAM role does not have permission to write to the correct S3 prefix.. Option C is correct because the IAM role attached to the SageMaker endpoint must have write permissions to the exact S3 prefix where data capture is configured. The policy shown likely grants access to a broader bucket or a different prefix, but not the specific path (e.g., s3://bucket-name/prefix/) that the endpoint's DataCaptureConfig specifies. Without s3:PutObject on that exact prefix, the captured data fails to upload silently.

Answer

The S3 bucket does not exist.

Answer

The S3 bucket uses AWS KMS encryption and the role lacks kms:Decrypt permission.

Answer

The IAM role does not have s3:ListBucket permission.

Question 14

A company uses Amazon Rekognition to moderate user-generated images. They want to set up a monitoring system that alerts the team if the number of inappropriate images flagged by the model exceeds a threshold. Which combination of AWS services should they use?

Accepted Answer

Amazon CloudWatch to publish custom metrics and create an alarm, and AWS Lambda to process images and publish metrics.. Option A is correct because Amazon CloudWatch can publish custom metrics (number of flagged images) and trigger alarms. AWS Lambda can process the images and publish the metric. Option B is incomplete because CloudWatch Logs alone cannot trigger alarms on custom metrics. Option C is incorrect because SNS alone does not monitor metrics. Option D is incorrect because AWS Config is for configuration tracking.

Answer

Amazon CloudWatch Logs to store inference logs and create a metric filter.

Answer

AWS Config to track resource changes and trigger an SNS notification.

Answer

Amazon Simple Notification Service (SNS) to send alerts when threshold is exceeded.

Question 15

A team deploys a PyTorch model on Amazon SageMaker for real-time inference. They notice that inference latency is higher than expected. They suspect the serialization format used for input data is inefficient. Which approach would MOST likely reduce latency?

Accepted Answer

Change the input serialization format to Protocol Buffers.. Protocol Buffers (protobuf) are a binary serialization format that is significantly more compact and faster to parse than text-based formats like JSON or CSV. By reducing the size of the input data and the CPU overhead of deserialization, switching to protobuf directly addresses the root cause of high inference latency on SageMaker real-time endpoints.

Answer

Use Amazon SageMaker Batch Transform instead of real-time inference.

Answer

Enable automatic scaling on the endpoint.

Answer

Increase the instance type to a compute-optimized instance.

Question 16

A financial services company deploys a credit risk model using an Amazon SageMaker endpoint with data capture enabled. The model uses a custom container. The compliance team requires that all inference requests and responses are logged to an S3 bucket with server-side encryption using AWS KMS. The IAM role for the endpoint has the following policy. What must be added to meet the compliance requirement?

Accepted Answer

Add kms:GenerateDataKey and kms:Decrypt permissions to the IAM role.. The correct answer is A because the IAM role for the SageMaker endpoint needs permissions to generate a data key (kms:GenerateDataKey) for encrypting captured data and to decrypt (kms:Decrypt) the KMS key when writing to the S3 bucket. Without these, the endpoint cannot use the customer-managed KMS key for server-side encryption, even if the bucket policy allows it.

Answer

Add s3:PutObjectAcl permission to the IAM role.

Answer

Enable S3 default encryption on the bucket.

Answer

Modify the container to handle encryption internally.

Question 17

Which TWO actions are recommended best practices for securing an Amazon SageMaker notebook instance? (Select TWO.)

Accepted Answer

Use AWS KMS to encrypt the notebook instance's storage volume.. Option C is correct because encrypting the notebook instance's storage volume with AWS KMS ensures data-at-rest protection, which is a fundamental security best practice. SageMaker notebook instances use Amazon EBS volumes for storage, and KMS encryption safeguards sensitive code, datasets, and model artifacts stored on that volume against unauthorized access.

Answer

Use network ACLs to restrict API calls to the SageMaker API.

Answer

Enable Multi-AZ deployment for the notebook instance.

Answer

Associate the notebook instance with a public subnet that has an internet gateway.

Question 18

Which THREE components are required to set up automated model retraining in response to performance degradation using Amazon SageMaker? (Select THREE.)

Accepted Answer

A CloudWatch alarm that triggers when a quality metric falls below a threshold.. Option B is correct because a CloudWatch alarm can monitor a SageMaker Model Monitor quality metric (e.g., accuracy, precision) and trigger an alarm when the metric falls below a defined threshold. This alarm acts as the event source to initiate automated retraining, forming the monitoring and alerting backbone of the retraining pipeline.

Answer

An Amazon SNS topic with a subscription to send a manual approval email.

Answer

A production variant with a canary traffic shift configuration.

Question 19

A company operates an e-commerce platform that uses a machine learning model to recommend products to users. The model is deployed on an Amazon SageMaker endpoint with automatic scaling enabled based on average CPU utilization. The model was trained on historical data and is updated weekly. Recently, the platform experienced a flash sale event that caused a sudden spike in traffic. During the event, the endpoint's latency increased dramatically, and many requests timed out. After the event, the team reviews the CloudWatch metrics and notices that the CPU utilization never exceeded 70%, and the scaling policy was triggered but instances took several minutes to become available. The team wants to prevent similar issues in future flash sales. Which course of action would be MOST effective?

Accepted Answer

Implement scheduled scaling to add capacity ahead of known flash sales.. Option D is correct because scheduled scaling allows you to proactively add capacity ahead of known traffic events like flash sales, eliminating the cold-start delay that occurs when reactive scaling policies (like those based on CPU utilization) must launch new instances. During the flash sale, the scaling policy was triggered but instances took minutes to become available, causing timeouts; scheduled scaling pre-warms the endpoint by adjusting the desired instance count before the traffic spike hits.

Answer

Use predictive scaling based on historical traffic patterns.

Answer

Lower the CPU utilization threshold for the scaling policy to 40%.

Answer

Switch to larger instance types to handle higher CPU loads.

Question 20

A healthcare company deploys a model that predicts patient readmission risk. The model is deployed using a SageMaker real-time endpoint with data capture enabled. The compliance team requires that all inference data be encrypted at rest in S3 using AWS KMS with a customer managed key. The team has configured the endpoint to use an IAM role that includes the necessary KMS permissions. However, after deployment, the captured data is not being written to the S3 bucket. The team checks the CloudWatch logs for the endpoint and finds no errors. The S3 bucket policy is as follows:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::my-bucket/*",
      "Condition": {
        "Bool": {
          "aws:SecureTransport": "false"
        }
      }
    }
  ]
}

The bucket also has a default KMS key. What is the MOST likely reason that the captured data is not being written?

Accepted Answer

The KMS key policy does not grant the SageMaker execution role the kms:GenerateDataKey permission.. The correct answer is C because SageMaker data capture encrypts captured data at rest in S3 using server-side encryption with AWS KMS (SSE-KMS). When a customer managed KMS key is used, the SageMaker execution role must have the kms:GenerateDataKey permission to encrypt the data before writing it to S3. Even if the IAM role has other KMS permissions, without kms:GenerateDataKey, the data capture write operation fails silently, and CloudWatch logs may not show errors because the failure occurs at the KMS encryption step before the S3 PutObject call.

Answer

The bucket policy includes an explicit deny that overrides any allow.

Answer

The bucket policy denies all PutObject requests because aws:SecureTransport is false.

Answer

The S3 bucket does not exist.