MLA-C01 ML Solution Monitoring, Maintenance, and Security — Timed 50 Questions

Question 1

A company needs to secure a SageMaker real-time endpoint such that only authorized applications within a VPC can invoke the model, and all data in transit is encrypted. Which TWO configuration steps should they implement? (Choose TWO)

50 minute time limit — choose an answer to begin.

Accepted Answer

Enable VPC-only mode for the endpoint. To ensure only authorized applications within a VPC can invoke the SageMaker endpoint, enable VPC-only mode (D). To encrypt all data in transit, enable inter-container traffic encryption (E). Option C (KMS for at-rest encryption) does not address the requirement of encrypting data in transit; it protects data at rest, which is not asked for here. Options A and B do not contribute to the stated requirements.

Answer

Assign a public IP to the endpoint instances

Answer

Disable SageMaker's API gateway logs

Answer

Use a KMS key for endpoint data encryption (at rest)

Question 2

A team has deployed a real-time inference endpoint. They need to monitor the latency experienced by end users, including network overhead. Which CloudWatch metric should they use?

Accepted Answer

OverheadLatency. OverheadLatency measures the additional time added by SageMaker infrastructure, including request handling, pre-processing, post-processing, and internal network overhead. It does not include model inference time (captured by ModelLatency) or client-side network latency. For monitoring end-user experience within the AWS environment, OverheadLatency is the metric that accounts for network overhead inside the service. If total end-to-end latency including client-side network is required, custom client-side metrics should be used.

Answer

ModelLatency

Answer

Latency

Answer

Invocations

Question 3

A company wants to automatically trigger a retraining pipeline when concept drift is detected in their deployed model. Which combination of services should they use?

Accepted Answer

SageMaker Model Monitor → CloudWatch Alarm → SNS → Lambda. SageMaker Model Monitor detects concept drift by analyzing model predictions against a baseline, then publishes metrics to CloudWatch. A CloudWatch Alarm triggers when drift exceeds a threshold, sending a notification via SNS to invoke a Lambda function, which starts the retraining pipeline. This end-to-end integration ensures automated, event-driven retraining without manual intervention.

Answer

SageMaker Model Monitor → Lambda

Answer

CloudWatch Events → SageMaker Training Job

Answer

SageMaker Clarify → SNS → Step Functions

Question 4

A data scientist uses SageMaker Model Monitor to track feature attribution drift. Which technique does SageMaker Model Monitor use to compute feature attributions?

Accepted Answer

SHAP. SageMaker Model Monitor uses SHAP (SHapley Additive exPlanations) to compute feature attributions for model explainability and drift detection. SHAP provides a unified measure of feature importance based on cooperative game theory, ensuring consistent and locally accurate attributions across all features.

Answer

Permutation Feature Importance

Answer

Partial Dependence Plots

Answer

LIME

Question 5

A team monitors a production endpoint and notices a sudden increase in 5XXError count. Which of the following is the most likely cause?

Accepted Answer

The model container is out of memory or crashing. A sudden increase in 5XX errors, particularly HTTP 503 or 502, typically indicates that the model container is failing to process requests due to resource exhaustion (e.g., OOM kills) or a crash in the inference process. In a production ML endpoint, such errors often stem from the container running out of memory, leading to the container being terminated by the orchestrator (e.g., Kubernetes OOMKill) or the application crashing internally, which directly causes 5XX responses.

Answer

The endpoint is under-provisioned and requests are throttled

Answer

The input data format has changed

Answer

The model is returning predictions with high latency

Question 6

Which TWO steps should they take? (Select TWO.)

Accepted Answer

Attach a resource policy to the endpoint that restricts access to the VPC endpoint. To restrict access to a specific VPC, create a VPC endpoint for SageMaker (com.amazonaws.region.sagemaker.api) and attach a resource policy that denies requests unless they originate from that VPC endpoint. Options B, D, and E do not achieve this goal: network isolation mode prevents training jobs from accessing the internet but does not restrict endpoint invocation; a NAT gateway is used for outbound traffic, not inbound; inter-container traffic encryption secures communication between containers, not client-to-endpoint communication.

Answer

Enable network isolation mode on the endpoint

Answer

Use a VPC with a NAT gateway and configure the endpoint to use the VPC

Answer

Enable inter-container traffic encryption on the endpoint

Question 7

A company uses a SageMaker endpoint for real-time inference. To reduce costs, they want to implement auto-scaling based on the number of invocations per instance. They also need to ensure that scaling actions are recorded for audit. Which TWO steps should they take? (Select TWO.)

Accepted Answer

Create a target tracking scaling policy based on the InvocationsPerInstance metric. To implement auto-scaling for a SageMaker endpoint based on invocations per instance, you must first register the endpoint as a scalable target with Application Auto Scaling (Option C). Then, create a target tracking scaling policy using the predefined InvocationsPerInstance metric (Option A). This policy automatically adjusts the number of instances to maintain the target invocations per instance, reducing costs by scaling down during low demand. Option B (enabling CloudTrail) is not required for the auto-scaling implementation itself; while it can record API calls for auditing, it is not a direct step in setting up scaling. Options D and E are incorrect because Model Monitor monitors data quality and custom metrics are unnecessary as the metric is already emitted.

Answer

Enable AWS CloudTrail to log auto-scaling API calls

Answer

Enable SageMaker Model Monitor to track invocation counts

Answer

Publish custom metrics to CloudWatch for endpoint utilization

Question 8

A company needs to give a data science team from another AWS account access to deploy models to a SageMaker endpoint in the company's account. The company wants to minimize administrative overhead while ensuring security. Which TWO steps should the company take? (Select TWO.)

Accepted Answer

Create a new SageMaker execution role in the company account and grant the external account's IAM role permission to pass it. Creating a SageMaker execution role in the company account and granting the external account's IAM role permission to pass it is a standard and secure method for cross-account access to SageMaker resources. Option A is incorrect because SageMaker models do not support resource-based policies for cross-account access; such policies are only available for SageMaker notebook instances. Options C, D, and E are also incorrect: sharing credentials or providing root user credentials violates security best practices, and a VPN is unnecessary for this use case.

Answer

Attach a resource-based policy to the SageMaker model granting access to the external account

Answer

Create an IAM role in the company account and share the credentials with the external team

Answer

Provide the external team with the company's AWS account root user credentials

Answer

Set up a VPN connection between the two accounts for network access

Question 9

A company is deploying a foundation model using SageMaker JumpStart. They want to minimize inference costs while maintaining low latency. Which TWO strategies should they consider? (Select TWO)

Accepted Answer

Enable auto-scaling with a target tracking policy based on Invocations per instance. Auto-scaling with a target tracking policy based on Invocations per instance dynamically adjusts the number of instances to match demand, ensuring you only pay for the compute capacity you need while maintaining low latency. This avoids over-provisioning and reduces idle costs, directly addressing the goal of minimizing inference costs.

Answer

Enable data capture for all requests to analyze usage patterns

Answer

Use SageMaker Savings Plans for discounted compute rates

Answer

Deploy the model on a single large instance to maximize throughput

Question 10

A machine learning engineer is monitoring a deployed model for data drift. The input features are a mix of categorical and numerical columns. The baseline is from the training data. Which SageMaker Model Monitor feature should they enable to detect changes in the distribution of each feature over time?

Accepted Answer

Data quality monitoring. Data quality monitoring in SageMaker Model Monitor detects schema and statistical drift (including distribution changes) for input features. Model quality monitors predictions vs. ground truth, not input features.

Answer

Bias drift monitoring

Answer

Model quality monitoring

Answer

Feature attribution drift monitoring

Question 11

A team receives alerts that their SageMaker endpoint latency has increased significantly. They check CloudWatch metrics and see Invocations rising, but ModelLatency remains stable. Which metric should they investigate to find the source of the increased latency?

Accepted Answer

OverheadLatency. OverheadLatency measures the time taken by the SageMaker infrastructure to handle requests before and after model inference, including request routing, authentication, and response processing. Since ModelLatency is stable but total endpoint latency has increased, the extra time must be in the overhead component, making OverheadLatency the correct metric to investigate.

Answer

ModelLatency

Answer

5XXError

Answer

4XXError

Question 12

A data scientist wants to track the lineage of models, datasets, and training jobs in SageMaker. Which SageMaker feature should they use to capture these relationships as artifacts and actions?

Accepted Answer

SageMaker ML Lineage Tracking. SageMaker ML Lineage Tracking creates a graph of artifacts (datasets, models) and actions (training jobs, endpoints) to track the provenance of ML workflows.

Answer

SageMaker Model Registry

Answer

SageMaker Experiments

Answer

SageMaker Feature Store

Question 13

A financial services company must deploy a SageMaker endpoint that processes sensitive customer data. They require that all traffic between the endpoint and the model containers be encrypted, and that the endpoint cannot be accessed from outside a specific VPC. Which combination of settings should they use?

Accepted Answer

Enable inter-container traffic encryption and configure the endpoint with VPC-only mode. Inter-container traffic encryption ensures that data between the SageMaker endpoint and the model containers is encrypted in transit, typically using TLS. Configuring the endpoint with VPC-only mode restricts all inference traffic to the specified VPC, preventing any access from outside that VPC. This combination directly addresses the requirements for encrypted inter-container traffic and VPC-restricted access.

Answer

Use a private VPC and enable data encryption at rest using KMS

Answer

Enable network isolation mode and inter-container traffic encryption

Answer

Deploy the endpoint in a private subnet and use a VPC endpoint for SageMaker API

Question 14

A team has deployed a real-time inference endpoint and wants to automatically scale based on CPU utilization. Which scaling policy type should they use with Application Auto Scaling for SageMaker endpoints?

Accepted Answer

Target tracking scaling. Target tracking scaling adjusts the number of instances based on a target metric value (e.g., CPU utilization at 50%). Step scaling uses step adjustments, and simple scaling is deprecated. Predictive scaling is not supported for SageMaker endpoints.

Answer

Step scaling

Answer

Predictive scaling

Answer

Simple scaling

Question 15

A company deploys a model for fraud detection. They need to monitor for bias after deployment, specifically whether the model's false positive rate changes across demographic groups over time. Which SageMaker feature should they use?

Accepted Answer

SageMaker Clarify (post-deployment bias monitoring). SageMaker Clarify provides post-deployment bias monitoring by analyzing predictions against ground truth labels for defined facets. It can track metrics like false positive rate differences over time.

Answer

SageMaker Model Monitor – Model Quality

Answer

SageMaker Model Monitor – Feature Attribution Drift

Answer

SageMaker Model Monitor – Data Quality

Question 16

A company wants to reduce costs for a production SageMaker endpoint that has predictable traffic patterns. They have purchased a Savings Plan. What additional step can they take to further optimize costs while maintaining performance?

Accepted Answer

Use SageMaker Inference Recommender to right-size the endpoint. SageMaker Inference Recommender provides instance type and configuration recommendations to right-size endpoints, balancing cost and performance. It is the appropriate tool for cost optimization beyond a Savings Plan.

Answer

Reduce the number of instances to one, regardless of load

Answer

Switch from real-time to batch inference

Answer

Disable auto-scaling

Question 17

A machine learning engineer is setting up a retraining pipeline that triggers when concept drift is detected. They plan to use CloudWatch Alarms to monitor the model's accuracy metric. When drift is detected, they want to automatically start a SageMaker training job. Which architecture should they use?

Accepted Answer

CloudWatch Alarm → SNS → Lambda → SageMaker Training Job. CloudWatch Alarms cannot directly invoke SageMaker training jobs; they require an intermediary like SNS to trigger a Lambda function, which then calls the SageMaker API to start the training job. This pattern ensures reliable decoupling and allows the Lambda function to handle any preprocessing or conditional logic before launching the job.

Answer

CloudWatch Alarm → SQS → Lambda → SageMaker Training Job

Answer

CloudWatch Alarm → EventBridge → SageMaker Training Job

Answer

CloudWatch Alarm → Lambda directly (without SNS)

Question 18

A company needs to give a data science team in another AWS account access to deploy a model from a shared model registry. Which approach should they use to grant cross-account access?

Accepted Answer

Attach a resource-based policy to the model registry granting the other account access. Resource-based policies (also called resource policies) can be attached to the model registry to allow cross-account access. IAM roles are used for intra-account access, not for granting access to resources in another account.

Answer

Create an IAM role in the other account with permissions to access the registry

Answer

Use AWS Organizations SCP to allow cross-account access

Answer

Share the model artifacts via Amazon S3 bucket policy and use an IAM role

Question 19

A company uses SageMaker Model Monitor for data quality. They notice that monitoring jobs are failing intermittently with constraint violations. Upon review, they see that some features have different data types in production compared to the baseline (e.g., string instead of integer). Which type of drift is this?

Accepted Answer

Schema drift. Schema drift occurs when the structure or data types of features in production data differ from the baseline used during model training. In this scenario, a feature that was an integer in the baseline is now a string in production, which is a classic example of schema drift. SageMaker Model Monitor detects this by comparing the inferred schema of production data against the baseline schema, flagging any type mismatches as constraint violations.

Answer

Concept drift

Answer

Statistical drift

Answer

Bias drift

Question 20

A team wants to monitor the number of requests and latency of their SageMaker endpoint using a unified dashboard. Which AWS service should they use to create a custom dashboard with these metrics?

Accepted Answer

Amazon CloudWatch Dashboards. Amazon CloudWatch Dashboards allow you to create custom views of metrics from any source, including SageMaker endpoint metrics like Invocations and Latency. SageMaker itself does not provide a dashboard for these metrics.

Answer

AWS CloudTrail

Answer

AWS Config

Answer

SageMaker Studio