MLA-C01 ML Solution Monitoring, Maintenance, and Security — 25 Questions

Question 1

A machine learning engineer is monitoring a deployed model for data drift. The input features are a mix of categorical and numerical columns. The baseline is from the training data. Which SageMaker Model Monitor feature should they enable to detect changes in the distribution of each feature over time?

Accepted Answer

Data quality monitoring. Data quality monitoring in SageMaker Model Monitor detects schema and statistical drift (including distribution changes) for input features. Model quality monitors predictions vs. ground truth, not input features.

Answer

Bias drift monitoring

Answer

Model quality monitoring

Answer

Feature attribution drift monitoring

Question 2

A team receives alerts that their SageMaker endpoint latency has increased significantly. They check CloudWatch metrics and see Invocations rising, but ModelLatency remains stable. Which metric should they investigate to find the source of the increased latency?

Accepted Answer

OverheadLatency. OverheadLatency measures the time taken by the SageMaker infrastructure to handle requests before and after model inference, including request routing, authentication, and response processing. Since ModelLatency is stable but total endpoint latency has increased, the extra time must be in the overhead component, making OverheadLatency the correct metric to investigate.

Answer

ModelLatency

Answer

5XXError

Answer

4XXError

Question 3

A data scientist wants to track the lineage of models, datasets, and training jobs in SageMaker. Which SageMaker feature should they use to capture these relationships as artifacts and actions?

Accepted Answer

SageMaker ML Lineage Tracking. SageMaker ML Lineage Tracking creates a graph of artifacts (datasets, models) and actions (training jobs, endpoints) to track the provenance of ML workflows.

Answer

SageMaker Model Registry

Answer

SageMaker Experiments

Answer

SageMaker Feature Store

Question 4

A financial services company must deploy a SageMaker endpoint that processes sensitive customer data. They require that all traffic between the endpoint and the model containers be encrypted, and that the endpoint cannot be accessed from outside a specific VPC. Which combination of settings should they use?

Accepted Answer

Enable inter-container traffic encryption and configure the endpoint with VPC-only mode. Option B is correct because inter-container traffic encryption ensures that data between the SageMaker endpoint and the model containers is encrypted in transit, typically using TLS. Configuring the endpoint with VPC-only mode restricts all inference traffic to the specified VPC, preventing any access from outside that VPC. This combination directly addresses the requirements for encrypted inter-container traffic and VPC-restricted access.

Answer

Use a private VPC and enable data encryption at rest using KMS

Answer

Enable network isolation mode and inter-container traffic encryption

Answer

Deploy the endpoint in a private subnet and use a VPC endpoint for SageMaker API

Question 5

A team has deployed a real-time inference endpoint and wants to automatically scale based on CPU utilization. Which scaling policy type should they use with Application Auto Scaling for SageMaker endpoints?

Accepted Answer

Target tracking scaling. Target tracking scaling adjusts the number of instances based on a target metric value (e.g., CPU utilization at 50%). Step scaling uses step adjustments, and simple scaling is deprecated. Predictive scaling is not supported for SageMaker endpoints.

Answer

Step scaling

Answer

Predictive scaling

Answer

Simple scaling

Question 6

A company deploys a model for fraud detection. They need to monitor for bias after deployment, specifically whether the model's false positive rate changes across demographic groups over time. Which SageMaker feature should they use?

Accepted Answer

SageMaker Clarify (post-deployment bias monitoring). SageMaker Clarify provides post-deployment bias monitoring by analyzing predictions against ground truth labels for defined facets. It can track metrics like false positive rate differences over time.

Answer

SageMaker Model Monitor – Model Quality

Answer

SageMaker Model Monitor – Feature Attribution Drift

Answer

SageMaker Model Monitor – Data Quality

Question 7

A company wants to reduce costs for a production SageMaker endpoint that has predictable traffic patterns. They have purchased a Savings Plan. What additional step can they take to further optimize costs while maintaining performance?

Accepted Answer

Use SageMaker Inference Recommender to right-size the endpoint. SageMaker Inference Recommender provides instance type and configuration recommendations to right-size endpoints, balancing cost and performance. It is the appropriate tool for cost optimization beyond a Savings Plan.

Answer

Reduce the number of instances to one, regardless of load

Answer

Switch from real-time to batch inference

Answer

Disable auto-scaling

Question 8

A machine learning engineer is setting up a retraining pipeline that triggers when concept drift is detected. They plan to use CloudWatch Alarms to monitor the model's accuracy metric. When drift is detected, they want to automatically start a SageMaker training job. Which architecture should they use?

Accepted Answer

CloudWatch Alarm → SNS → Lambda → SageMaker Training Job. Option C is correct because CloudWatch Alarms cannot directly invoke SageMaker training jobs; they require an intermediary like SNS to trigger a Lambda function, which then calls the SageMaker API to start the training job. This pattern ensures reliable decoupling and allows the Lambda function to handle any preprocessing or conditional logic before launching the job.

Answer

CloudWatch Alarm → SQS → Lambda → SageMaker Training Job

Answer

CloudWatch Alarm → EventBridge → SageMaker Training Job

Answer

CloudWatch Alarm → Lambda directly (without SNS)

Question 9

A company needs to give a data science team in another AWS account access to deploy a model from a shared model registry. Which approach should they use to grant cross-account access?

Accepted Answer

Attach a resource-based policy to the model registry granting the other account access. Resource-based policies (also called resource policies) can be attached to the model registry to allow cross-account access. IAM roles are used for intra-account access, not for granting access to resources in another account.

Answer

Create an IAM role in the other account with permissions to access the registry

Answer

Use AWS Organizations SCP to allow cross-account access

Answer

Share the model artifacts via Amazon S3 bucket policy and use an IAM role

Question 10

A company uses SageMaker Model Monitor for data quality. They notice that monitoring jobs are failing intermittently with constraint violations. Upon review, they see that some features have different data types in production compared to the baseline (e.g., string instead of integer). Which type of drift is this?

Accepted Answer

Schema drift. Option A is correct because schema drift occurs when the structure or data types of features in production data differ from the baseline used during model training. In this scenario, a feature that was an integer in the baseline is now a string in production, which is a classic example of schema drift. SageMaker Model Monitor detects this by comparing the inferred schema of production data against the baseline schema, flagging any type mismatches as constraint violations.

Answer

Concept drift

Answer

Statistical drift

Answer

Bias drift

Question 11

A team wants to monitor the number of requests and latency of their SageMaker endpoint using a unified dashboard. Which AWS service should they use to create a custom dashboard with these metrics?

Accepted Answer

Amazon CloudWatch Dashboards. Amazon CloudWatch Dashboards allow you to create custom views of metrics from any source, including SageMaker endpoint metrics like Invocations and Latency. SageMaker itself does not provide a dashboard for these metrics.

Answer

AWS CloudTrail

Answer

AWS Config

Answer

SageMaker Studio

Question 12

A company uses SageMaker JumpStart to deploy a foundation model for a summarization task. They want to minimize costs while still meeting a latency requirement of under 2 seconds. Which option should they consider?

Accepted Answer

Use SageMaker Inference Recommender to select the cheapest instance that meets latency. SageMaker Inference Recommender runs load tests against your model on various instance types and provides latency and cost metrics. By selecting the cheapest instance that still meets the sub-2-second latency requirement, you directly minimize cost while satisfying the performance constraint. This is the most systematic and cost-effective approach for this scenario.

Answer

Deploy the model on a serverless endpoint

Answer

Enable auto-scaling to handle variable traffic

Answer

Use the largest GPU instance to ensure fast inference

Question 13

A company wants to automatically trigger model retraining when SageMaker Model Monitor detects data drift. Which TWO services should they integrate to achieve this automation? (Choose two.)

Accepted Answer

AWS Lambda. EventBridge can schedule or trigger events based on monitoring job completions, and Lambda can execute the retraining code. SNS is used for notifications but not direct invocation of retraining. Step Functions can orchestrate but is not required for this simple trigger.

Answer

AWS Step Functions

Answer

Amazon CloudWatch Alarms

Answer

Amazon SNS

Question 14

A company wants to secure data in transit between the client and SageMaker endpoint, and between containers in the same endpoint. Which THREE configurations should they apply? (Choose three.)

Accepted Answer

Enable network isolation for the model containers. Enforce inter-container traffic encryption, use HTTPS for the endpoint, and enable network isolation to prevent internet exposure. VPC-only mode controls network access but does not directly encrypt traffic.

Answer

Enable data encryption at rest using KMS

Answer

Use a VPC-only endpoint configuration

Question 15

A machine learning engineer is setting up model quality monitoring for a binary classification model. They have ground truth labels available in Amazon S3. Which TWO steps are required to configure model quality monitoring? (Choose two.)

Accepted Answer

Create a schedule for the monitoring job to run at regular intervals. A baseline must be computed from training data and ground truth, and a schedule for monitoring jobs must be defined. The monitoring job then compares production predictions against ground truth.

Answer

Create a CloudWatch Alarm for accuracy degradation

Answer

Use Amazon SageMaker Clarify for bias detection

Answer

Enable data capture on the endpoint to collect predictions

Question 16

A data science team uses Amazon SageMaker Model Monitor to detect data drift in production. They notice that the schema of incoming data (number of features) has changed compared to the training baseline. Which type of monitor is BEST suited to detect this issue?

Accepted Answer

Data quality monitor. The Data quality monitor in SageMaker Model Monitor is specifically designed to detect violations in the input data schema, such as changes in the number of features, feature types, or missing values, by comparing incoming data against a baseline computed from the training dataset. Since the issue is a structural change in the schema (number of features), the Data quality monitor is the correct choice.

Answer

Bias drift monitor

Answer

Feature attribution drift monitor

Answer

Model quality monitor

Question 17

An ML engineer wants to be notified when the average inference latency of a SageMaker endpoint exceeds 500 ms for 2 consecutive evaluation periods. Which AWS service combination should they use?

Accepted Answer

CloudWatch Alarm + SNS. CloudWatch collects the 'Latency' metric from SageMaker endpoints. A CloudWatch Alarm can be set on this metric with a threshold of 500 ms for 2 consecutive periods. SNS then sends notification (email, SMS, etc.). Lambda is for automated remediation, not required for notification only.

Answer

SageMaker Model Monitor + SNS

Answer

EventBridge + Lambda

Answer

SageMaker Clarify + SNS

Question 18

A company deploys a model for credit risk assessment on a SageMaker endpoint. To comply with internal policies, they must ensure that the endpoint only allows inference requests from within a specific VPC and that the data is encrypted at rest. Which configuration meets these requirements?

Accepted Answer

Configure endpoint in VPC-only mode and enable KMS encryption for the endpoint. Option D is correct because configuring the SageMaker endpoint in VPC-only mode ensures that inference requests can only originate from within the specified VPC, satisfying the network restriction requirement. Enabling KMS encryption for the endpoint encrypts the model data and inference data at rest using a customer-managed key, meeting the encryption-at-rest policy.

Answer

Enable network isolation mode and use KMS for endpoint encryption

Answer

Configure endpoint in VPC-only mode and enable inter-container traffic encryption

Answer

Deploy endpoint in a public subnet and use a security group to restrict traffic

Question 19

A machine learning team uses SageMaker Pipelines and wants to automatically retrain a model when data drift is detected. They have set up Model Monitor to publish drift violations to CloudWatch. Which approach provides a COMPLETE serverless retraining pipeline triggered by drift detection?

Accepted Answer

Configure a CloudWatch Alarm on drift metric → SNS topic → Lambda function that starts the SageMaker Pipeline execution. The recommended pattern: CloudWatch Alarm triggers on drift metric → SNS message → Lambda function (receives SNS) → starts SageMaker Pipeline execution. EventBridge could also trigger on SNS events, but Lambda is simplest. EventBridge can schedule retraining but does not directly react to specific drift alarms. Step Functions would add unnecessary complexity.

Answer

Use SageMaker Model Monitor to directly invoke a SageMaker Pipeline when drift is detected

Answer

Use EventBridge to schedule retraining daily regardless of drift

Answer

Create an EventBridge rule that triggers on Model Monitor drift events to start the pipeline

Question 20

A financial institution uses SageMaker to train and deploy models. They need to track every experiment, model version, and deployment step for audit purposes. Which SageMaker feature should they use to capture the full lineage of artifacts, actions, and contexts?

Accepted Answer

SageMaker ML Lineage Tracking. SageMaker ML Lineage Tracking creates a graph of artifacts (datasets, models), actions (training jobs, deployment), and contexts (experiments). It provides a complete audit trail. Experiments alone track trials but not lineage. Model Registry tracks model versions but not full pipeline lineage. Clarify is for bias monitoring.

Answer

SageMaker Clarify

Answer

SageMaker Model Registry

Answer

SageMaker Experiments