MLA-C01 ML Solution Monitoring, Maintenance, and Security Practice Test 3 — 15 Questions

Question 1

A machine learning team deploys a model for loan approval. They want to monitor data drift on the real-time endpoint using SageMaker Model Monitor. Which set of actions should they take to set up data quality monitoring?

Accepted Answer

Enable data capture on the endpoint, generate a baseline from training data, create a data quality monitoring schedule, and set up a CloudWatch Alarm on violations. SageMaker Model Monitor requires a baseline from training data, then schedules monitoring jobs that compare live endpoint captures against that baseline. Alerts are sent via CloudWatch Alarms.

Answer

Use SageMaker Clarify to detect data drift on the endpoint

Answer

Create a model quality monitoring schedule directly on the endpoint without any baseline

Answer

Enable data capture and rely on SageMaker Model Monitor to automatically infer drift without a baseline

Question 2

A financial services company deploys a fraud detection model with a SageMaker endpoint. They need to ensure that all data sent to the endpoint is encrypted in transit and at rest, and that the endpoint cannot be accessed from the public internet. Which combination of settings should they use?

Accepted Answer

Enable VPC-only mode, inter-container traffic encryption, and use a KMS key for endpoint encryption. VPC-only mode ensures the endpoint is private. Inter-container traffic encryption is needed for data in transit between containers in multi-model endpoints. KMS encryption secures data at rest on the instance storage.

Answer

Use endpoint data encryption with an AWS managed key and enable public endpoint access

Answer

Enable inter-container traffic encryption and disable VPC-only mode

Answer

Deploy the endpoint in a private subnet without SageMaker VPC-only mode

Question 3

A data scientist notices that a production model's accuracy has degraded over the past week. The training data distribution remains unchanged, but the relationship between features and the target has shifted. Which type of drift is occurring, and which monitoring approach should be used?

Accepted Answer

Concept drift; use SageMaker Model Monitor model quality monitoring with ground truth labels. Concept drift occurs when the underlying relationship between features and target changes. Model quality monitoring (comparing predictions against ground truth) detects this. Data drift monitors feature distribution changes, which are not present here.

Answer

Bias drift; use SageMaker Clarify post-deployment bias monitoring

Answer

Data drift; use SageMaker Model Monitor data quality monitoring

Answer

Feature attribution drift; use SageMaker Clarify

Question 4

A team wants to automatically retrain a model whenever data drift is detected on their SageMaker endpoint. Which AWS service should they use to invoke a retraining pipeline in response to a CloudWatch Alarm?

Accepted Answer

Amazon SNS topic triggering an AWS Lambda function. Option D is correct because Amazon SNS can directly subscribe to a CloudWatch Alarm and, upon alarm state change, publish a message to an SNS topic. That topic can then trigger an AWS Lambda function, which invokes the retraining pipeline (e.g., SageMaker Processing or training job). This creates a fully managed, serverless event-driven workflow without needing custom polling or additional orchestration services.

Answer

AWS Step Functions directly from CloudWatch Alarm

Answer

SageMaker Processing job scheduled via EventBridge

Answer

Amazon SQS queue polling by a custom application

Question 5

A company uses SageMaker Model Monitor to track feature attribution drift with SHAP. They notice that the SHAP values have changed significantly for a feature, while the model performance remains stable. What is the MOST likely interpretation?

Accepted Answer

The model's behavior has changed, which may lead to future performance degradation and warrants investigation. Option D is correct because a significant change in SHAP values indicates that the model's internal feature importance has shifted, even if overall performance metrics like accuracy or loss remain stable. This is a classic sign of concept drift or model behavior drift, where the model's decision boundary has changed for that feature, which can lead to future performance degradation as the drift accumulates. SageMaker Model Monitor tracks feature attribution drift separately from data drift, and a change in SHAP values without data drift suggests the model is relying on the feature differently, warranting investigation.

Answer

The ground truth labels are incorrect, causing the drift

Answer

Data drift has occurred for that feature

Answer

The model is learning equally from all features, so no action is needed

Question 6

A machine learning engineer deploys a multi-model endpoint using SageMaker. They need to track which model version was used for each inference request for compliance purposes. Which service should they integrate to capture this lineage?

Accepted Answer

SageMaker ML Lineage Tracking. SageMaker ML Lineage Tracking is the correct service because it is specifically designed to capture and query the lineage of machine learning artifacts, including model versions, datasets, and inference requests. By integrating with SageMaker endpoints, it automatically records the model version used for each inference, enabling compliance auditing without custom code.

Answer

AWS CloudTrail

Answer

SageMaker Model Monitor

Answer

Amazon DynamoDB with custom logging

Question 7

A company wants to reduce costs for a real-time inference endpoint that experiences predictable traffic spikes during business hours and low traffic at night. Which auto-scaling policy is MOST cost-effective while maintaining performance?

Accepted Answer

Scheduled scaling that increases instances before business hours and decreases after. Option C is correct because scheduled scaling directly aligns capacity with the predictable traffic pattern (business hours vs. night), allowing you to proactively add instances before demand increases and remove them afterward. This avoids the cost of over-provisioning during low-traffic periods and the latency of reactive scaling, making it the most cost-effective approach for a known, recurring schedule.

Answer

Step scaling based on CPU utilization

Answer

Manual scaling by the operations team

Answer

Target tracking with a custom metric for response time

Question 8

A team trains a model using SageMaker and wants to ensure that the training job cannot access the internet, but needs to access a private S3 bucket in the same VPC. Which configuration should they use?

Accepted Answer

Enable network isolation and provide VPC subnet and security group. SageMaker training jobs can be configured with VPC settings. Enabling network isolation prevents internet access, but the job still needs VPC connectivity to reach S3 via VPC endpoints. Setting the VPC config allows access to private resources.

Answer

Enable network isolation only, without VPC configuration

Answer

Disable network isolation but restrict security group rules

Answer

Use a public S3 bucket with bucket policies

Question 9

A data scientist deploys a model and wants to monitor the endpoint's invocation latency. They notice that the CloudWatch metric 'ModelLatency' is high, but 'OverheadLatency' is low. Which statement correctly interprets these metrics?

Accepted Answer

The model inference time is the bottleneck; consider optimizing the model or using a faster instance type. The 'ModelLatency' metric measures the time taken by the SageMaker model container to process a single request, including inference and any preprocessing/postprocessing within the container. 'OverheadLatency' measures the time spent on SageMaker infrastructure (e.g., network I/O, request queuing, and response handling). When ModelLatency is high and OverheadLatency is low, the bottleneck is clearly the model inference time itself, not the infrastructure overhead. Therefore, optimizing the model (e.g., quantization, pruning) or upgrading to a faster instance type (e.g., GPU vs. CPU) is the correct remediation.

Answer

The SageMaker overhead is causing the delay; check endpoint configuration

Answer

The endpoint is overloaded; increase the number of instances

Answer

The network latency is high; move the endpoint closer to clients

Question 10

An organization uses SageMaker Studio and needs to restrict Studio's internet access while allowing users to install custom packages from a private PyPI mirror hosted in a VPC. Which networking configuration should they use?

Accepted Answer

Disable internet access for Studio and configure VPC-only mode, then use a VPC endpoint to the private PyPI mirror. SageMaker Studio in VPC-only mode (no internet access) combined with VPC-only mode for the domain and VPC endpoints to private PyPI mirror enables package installation from the private mirror. NAT gateway or internet gateway would allow internet access, which is not desired.

Answer

Use a NAT gateway to allow outbound traffic to the private PyPI mirror

Answer

Disable internet access for Studio and rely on SageMaker's default VPC configuration

Answer

Enable internet access for Studio and use a VPC endpoint to the private PyPI mirror

Question 11

A machine learning team wants to monitor bias in a deployed model's predictions on an ongoing basis. Which AWS service should they use to schedule bias monitoring jobs and generate reports?

Accepted Answer

SageMaker Clarify with bias drift monitoring. SageMaker Clarify is the correct choice because it provides built-in bias drift monitoring capabilities that can be scheduled to run on a recurring basis. It evaluates predictions against pre-training and post-training bias metrics (e.g., DPL, DI, CDDL) and generates detailed reports, making it the only service designed specifically for ongoing bias monitoring in deployed models.

Answer

Amazon QuickSight with Athona queries

Answer

AWS CloudTrail for prediction logging

Answer

SageMaker Model Monitor with data quality monitoring

Question 12

A company uses SageMaker Inference Recommender to select the optimal endpoint configuration. After running the recommender, they receive a recommendation for a specific instance type and initial instance count. What should they do next to optimize costs over time?

Accepted Answer

Set up auto-scaling with a target tracking policy based on the recommended metric. SageMaker Inference Recommender provides a baseline configuration. To optimize costs, they should apply auto-scaling with a target tracking policy based on the recommended metric, such as invocation count or latency.

Answer

Use the recommended configuration without changes, as it is already optimal

Answer

Purchase a Savings Plan for the recommended instance type to reduce hourly cost

Answer

Manually adjust the instance count daily based on observed traffic

Question 13

A machine learning team wants to detect concept drift in a production model. Which TWO actions should they take? (Choose TWO)

Accepted Answer

Enable data capture on the endpoint to collect ground truth labels. Concept drift is detected by comparing model predictions to actual outcomes (ground truth). Capturing ground truth and using model quality monitoring is essential. Data quality monitoring would not detect concept drift.

Answer

Set up SageMaker Model Monitor data quality monitoring schedule

Answer

Use SageMaker Clarify for feature attribution drift

Answer

Enable daily retraining to automatically correct drift

Question 14

A company needs to secure a SageMaker real-time endpoint such that only authorized applications within a VPC can invoke the model, and all data in transit is encrypted. Which THREE configuration steps should they implement? (Choose THREE)

Accepted Answer

Use a KMS key for endpoint data encryption (at rest). Option C is correct because using a KMS key for endpoint data encryption at rest ensures that the model data and artifacts stored on the endpoint's ML storage volumes are encrypted. This is a required step for meeting encryption compliance, though the question specifically asks about data in transit; however, KMS encryption at rest is a standard security best practice and is listed as a correct answer in the context of overall data protection for the endpoint.

Answer

Assign a public IP to the endpoint instances

Answer

Disable SageMaker's API gateway logs

Question 15

A data science team wants to automate the retraining of a model when data drift is detected. Which TWO AWS services should they use in combination to achieve this? (Choose TWO)

Accepted Answer

SageMaker Model Monitor. SageMaker Model Monitor detects data drift and can publish to CloudWatch Metrics. CloudWatch Alarms can trigger an SNS topic, which invokes a Lambda function that starts a SageMaker Pipeline for retraining.

Answer

AWS Cloud9

Answer

Amazon DynamoDB

Answer

Amazon Kinesis Data Analytics