MLA-C01 ML Solution Monitoring, Maintenance, and Security Practice Test 6 — 15 Questions

Question 1

An ML team has deployed a model to a SageMaker real-time endpoint and wants to set up automated monitoring for model quality. Which TWO elements are required to configure SageMaker Model Monitor for model quality? (Select TWO.)

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

A ground truth labels dataset for comparison. SageMaker Model Monitor for model quality requires a ground truth labels dataset to compare the model's predictions against actual outcomes. This comparison is essential for calculating quality metrics like accuracy, precision, recall, or F1 score, which indicate how well the model is performing over time.

Answer

SHAP values for feature attribution

Answer

A constraints file with allowed deviation thresholds

Answer

A baseline statistics file derived from the training data

Question 2

A machine learning team deploys a fraud detection model on a SageMaker endpoint. The model's predictions are used in real-time. The team wants to monitor for data drift by comparing incoming data distributions against a baseline created from the training data. Which SageMaker capability should they use?

Accepted Answer

SageMaker Model Monitor - Data Quality Monitor. SageMaker Model Monitor's Data Quality Monitor is specifically designed to detect data drift by comparing the statistical distribution of incoming inference data against a baseline computed from the training dataset. This capability tracks metrics like mean, variance, and quantiles for each feature, alerting when significant deviations occur. For a fraud detection model requiring real-time monitoring of input distributions, this is the correct choice.

Answer

SageMaker Model Monitor - Model Quality Monitor

Answer

SageMaker Model Monitor - Feature Attribution Drift Monitor

Answer

SageMaker Model Monitor - Bias Drift Monitor

Question 3

A data scientist notices that a SageMaker endpoint is returning HTTP 5XX errors under high load. The endpoint uses a single ml.m5.large instance. The team wants to reduce these errors without changing the instance type. What is the most cost-effective step?

Accepted Answer

Configure auto-scaling for the endpoint with a target tracking policy. Configuring auto-scaling with a target tracking policy allows the endpoint to dynamically add more instances under high load, distributing the traffic and reducing HTTP 5XX errors. Since the team cannot change the instance type, scaling out is the most cost-effective way to handle increased demand, as it only adds capacity when needed and avoids over-provisioning.

Answer

Increase the endpoint's invocation timeout to 120 seconds

Answer

Deploy the model on a SageMaker batch transform job

Answer

Create a new endpoint with multiple instances and use weighted routing

Question 4

A financial services company needs to deploy a SageMaker endpoint that processes sensitive customer data. The security policy requires that all data in transit between the endpoint and the application must be encrypted, and that the endpoint cannot be accessed from the public internet. Additionally, model containers must not be able to initiate outbound internet requests. Which combination of settings meets these requirements?

Accepted Answer

Enable VPC-only mode for the endpoint and set EnableInterContainerTrafficEncryption to True. Enabling VPC-only mode for the SageMaker endpoint ensures the endpoint is not accessible from the public internet, and setting EnableInterContainerTrafficEncryption to True encrypts data in transit between containers within the endpoint. This combination directly satisfies the requirements for no public internet access and encrypted data in transit, while the model containers are isolated from outbound internet requests by the VPC configuration.

Answer

Attach a public endpoint with an SSL certificate and restrict access via IAM

Answer

Use a private subnet with a NAT Gateway and set EnableNetworkIsolation to True

Answer

Deploy on a multi-model endpoint with encryption at rest using KMS

Question 5

A machine learning engineer observes that model performance on a SageMaker endpoint has degraded over the past week. Ground truth labels are available with a 2-day delay. The engineer wants to automatically trigger a retraining pipeline when prediction quality drops below an acceptable threshold. Which approach is most appropriate?

Accepted Answer

Use SageMaker Model Monitor - Model Quality Monitor with ground truth, create a CloudWatch alarm on the metric, and trigger an AWS Lambda function to start retraining. SageMaker Model Monitor's Model Quality Monitor is specifically designed to compare model predictions against ground truth labels (available with a 2-day delay) and track metrics like accuracy, precision, recall, or F1 score. You can configure a CloudWatch alarm on a metric such as 'accuracy' dropping below a threshold, which triggers an AWS Lambda function to start the retraining pipeline. This automates the detection of prediction quality degradation and the retraining response without manual intervention.

Answer

Manually evaluate the model weekly and retrain as needed

Answer

Use SageMaker Model Monitor - Data Quality Monitor to detect drift, then trigger retraining

Answer

Use SageMaker Clarify to monitor bias drift and trigger retraining

Question 6

A company uses SageMaker Studio for collaborative ML development. The security team requires that all SageMaker Studio notebooks run within a VPC and cannot access the public internet. Which configuration should the administrator set?

Accepted Answer

Enable VPC-only mode for the SageMaker Studio domain. Enabling VPC-only mode for the SageMaker Studio domain ensures that all Studio notebooks and apps are launched within the specified VPC and cannot access the public internet. This mode enforces that all network traffic, including internet-bound traffic, is routed through the VPC, and it blocks direct internet access by default, meeting the security team's requirement.

Answer

Use SageMaker notebook instances instead of Studio

Answer

Apply an SCP that denies internet access for all IAM users

Answer

Set the SageMaker Studio domain to use a public subnet with a NAT Gateway

Question 7

A data science team wants to track the lineage of models, including datasets, training jobs, and endpoints, for reproducibility and audit. They need a solution that captures relationships between artifacts automatically during training and deployment. Which service should they use?

Accepted Answer

SageMaker ML Lineage Tracking. SageMaker ML Lineage Tracking automatically records the relationships between artifacts (datasets, jobs, models, endpoints) and tracks the provenance of ML workflows.

Answer

Amazon S3 object versioning

Answer

SageMaker Experiments

Answer

SageMaker Model Registry

Question 8

A company deploys a real-time inference endpoint and wants to be alerted if the number of 4XX errors exceeds 10 per minute over a 5-minute period. Which steps should they take?

Accepted Answer

Create a CloudWatch alarm on the 4XXError metric with a threshold of 10 and an evaluation period of 5 minutes, and configure SNS notification. A CloudWatch alarm on the `4XXError` metric with a threshold of 10 and an evaluation period of 5 minutes directly monitors the rate of HTTP 4XX errors from the SageMaker real-time inference endpoint. When the alarm state transitions to ALARM (i.e., the average 4XX errors per minute exceeds 10 over the 5-minute window), it triggers an SNS notification to alert the team. This is the standard approach for real-time metric-based alerting in AWS.

Answer

Create a CloudWatch alarm on the Invocations metric and set a threshold

Answer

Enable endpoint auto-scaling with a target tracking policy

Answer

Use SageMaker Model Monitor to capture invocations and trigger an SNS topic

Question 9

A team is using SageMaker Clarify to detect bias drift in a deployed model's predictions. They run weekly bias monitoring jobs. The team wants to be notified when the bias metric for a sensitive feature exceeds a threshold. What is the most efficient method to achieve this?

Accepted Answer

After each Clarify job, run a custom Lambda that parses the report and publishes a custom CloudWatch metric; create an alarm on that metric. SageMaker Clarify bias monitoring jobs output a JSON report to S3 but do not natively publish CloudWatch metrics. By using a custom Lambda to parse the report and publish a custom CloudWatch metric, you can then create a CloudWatch alarm that triggers notifications when the bias metric exceeds a threshold. This is the most efficient automated method because it leverages CloudWatch's native alarm and notification capabilities without manual intervention.

Answer

Configure the Clarify monitoring job to send results to an SNS topic directly

Answer

Manually review the bias report in SageMaker Studio each week

Answer

Use SageMaker Model Monitor - Bias Drift Monitor which automatically creates CloudWatch metrics

Question 10

A company wants to reduce costs for a SageMaker real-time endpoint that receives predictable traffic patterns: high during business hours and low at night. The model is a small PyTorch model. Which cost-saving strategy is most suitable?

Accepted Answer

Configure auto-scaling with a scheduled scaling policy to add instances during business hours and reduce at night. Auto-scaling with a schedule can adjust instance count based on time, matching capacity to demand. This is more efficient than manual scaling or using a larger instance.

Answer

Use a single large instance to handle peak load

Answer

Use a multi-model endpoint with multiple models

Answer

Switch to batch transform jobs and run nightly

Question 11

A machine learning engineer is deploying a model using a SageMaker endpoint and needs to ensure that the model artifacts are encrypted at rest using a customer-managed KMS key. Which configuration should they set?

Accepted Answer

Set the KMS key when creating the model using the CreateModel API. The `CreateModel` API in SageMaker accepts a `ModelKmsKeyId` parameter that specifies a customer-managed KMS key for encrypting the model artifacts at rest. This key is used when SageMaker copies the artifacts from S3 to the inference instance's Amazon EBS volume, ensuring encryption at rest. The other options either apply to different resources or do not control the encryption of the model artifacts themselves.

Answer

Set the KMS key in the endpoint configuration's ProductionVariant

Answer

Enable default encryption on the S3 bucket containing the model artifacts

Answer

Use SageMaker Studio's KMS integration

Question 12

A team is monitoring a SageMaker endpoint and notices that the average latency (ModelLatency) is increasing over time, but the number of invocations is steady. They suspect that the model's inference code is becoming slower due to memory leaks. Which metric should they also examine to confirm this hypothesis?

Accepted Answer

MemoryUtilization metric (if custom). SageMaker publishes memory utilization metrics if the container emits them via CloudWatch. A memory leak would show increasing memory usage over time, correlating with increasing latency.

Answer

Invocations metric

Answer

OverheadLatency metric

Answer

5XXError metric

Question 13

A company wants to share a trained model across multiple AWS accounts for inference. The model is stored in a central account's S3 bucket and needs to be deployed in other accounts' SageMaker endpoints. What is the recommended approach?

Accepted Answer

Attach a resource policy to the model in the central account allowing the other accounts' SageMaker service principals to access it. SageMaker allows you to attach a resource-based policy directly to the model resource in the central account, granting the SageMaker service principal from other accounts permission to call `sagemaker:CreateModel` and `sagemaker:CreateEndpointConfig` using the shared model. This approach avoids copying artifacts and leverages AWS Identity and Access Management (IAM) cross-account trust, where the central account's model policy explicitly allows the remote account's SageMaker service role to access the model and its underlying S3 objects.

Answer

Use AWS RAM to share the model artifact S3 bucket

Answer

Use SageMaker Model Registry with cross-account sharing enabled

Answer

Copy the model artifacts to each account's S3 bucket and create separate models

Question 14

A machine learning team needs to automatically retrain a model when concept drift is detected in the deployed endpoint's predictions. Which TWO steps should they take? (Choose TWO.)

Accepted Answer

Create a CloudWatch alarm on a model quality metric (e.g., accuracy) and trigger a Lambda function to start a retraining job. Model Quality Monitor compares predictions with ground truth to detect concept drift. When an alarm triggers, a Lambda function can start a retraining pipeline. Data Quality Monitor is for data drift, not concept drift.

Answer

Schedule retraining with Amazon EventBridge on a fixed schedule

Answer

Configure SageMaker Model Monitor - Data Quality Monitor to detect input drift

Answer

Use SageMaker Clarify to monitor bias drift

Question 15

A company is deploying a SageMaker real-time endpoint and needs to monitor inference latency. Which THREE metrics are available from SageMaker for this purpose? (Choose THREE.)

Accepted Answer

OverheadLatency. OverheadLatency is a SageMaker metric that measures the time taken by the SageMaker infrastructure to process the request before and after invoking the model, including network overhead and framework overhead. It is one of the three metrics specifically designed to monitor inference latency for real-time endpoints.

Answer

Invocations

Answer

MemoryUtilization