ML Solution Monitoring, Maintenance, and Security practice questions

Assign a public IP to the endpoint instances

Why wrong: Would make the endpoint potentially accessible from internet.

Disable SageMaker's API gateway logs

Why wrong: Irrelevant to encryption or VPC access.

Use a KMS key for endpoint data encryption (at rest)

Why wrong: Encrypts data on the instance's storage.

ModelLatency

Why wrong: ModelLatency only measures the time inside the model container, excluding overhead.

Latency

Why wrong: Latency is not a standard SageMaker endpoint metric; the correct term is ModelLatency or OverheadLatency.

Invocations

Why wrong: Invocations counts requests, not latency.

SageMaker Model Monitor → Lambda

Why wrong: Missing CloudWatch Alarm and SNS; Lambda cannot directly subscribe to Model Monitor output.

CloudWatch Events → SageMaker Training Job

Why wrong: This lacks the drift detection step; it would trigger on a schedule, not on drift.

SageMaker Clarify → SNS → Step Functions

Why wrong: SageMaker Clarify is not designed for drift detection; it handles bias and explainability.

Permutation Feature Importance

Why wrong: Permutation importance is global and not used by SageMaker Model Monitor for drift.

Partial Dependence Plots

Why wrong: Partial dependence plots show average marginal effects, not per-feature attributions.

LIME

Why wrong: LIME is a local explanation method, but SageMaker Model Monitor uses SHAP.

The endpoint is under-provisioned and requests are throttled

Why wrong: Throttling results in 4XX errors (TooManyRequests), not 5XX.

The input data format has changed

Why wrong: Input format change typically causes 4XX errors, not 5XX.

The model is returning predictions with high latency

Why wrong: High latency does not cause 5XX errors; it may increase latency metrics.

Enable network isolation mode on the endpoint

Why wrong: Network isolation mode prevents SageMaker training jobs from accessing the internet, but does not apply to endpoint invocation or restrict access to a specific VPC.

Use a VPC with a NAT gateway and configure the endpoint to use the VPC

Why wrong: A NAT gateway provides outbound internet access from a VPC, but does not restrict inbound access to the endpoint from within the VPC.

Enable inter-container traffic encryption on the endpoint

Why wrong: Inter-container traffic encryption secures communication between containers inside a SageMaker endpoint, but does not encrypt the client-to-endpoint traffic nor restrict VPC access.

Enable AWS CloudTrail to log auto-scaling API calls

Why wrong: Incorrect. While enabling CloudTrail can log auto-scaling API calls, it is not one of the two steps needed to implement auto-scaling itself. The required steps are registering the scalable target and creating the scaling policy.

Enable SageMaker Model Monitor to track invocation counts

Why wrong: Incorrect. SageMaker Model Monitor is used for monitoring data and model quality, not for tracking invocation counts for scaling.

Publish custom metrics to CloudWatch for endpoint utilization

Why wrong: Incorrect. SageMaker automatically emits the InvocationsPerInstance metric, so publishing custom metrics is unnecessary for this scaling policy.

Attach a resource-based policy to the SageMaker model granting access to the external account

Why wrong: Incorrect. SageMaker models do not support resource-based policies for cross-account access. This is a common misconception; resource-based policies are only available for SageMaker notebook instances, not models or endpoints.

Create an IAM role in the company account and share the credentials with the external team

Why wrong: Incorrect. Sharing IAM role credentials is insecure and introduces administrative overhead; AWS recommends using IAM roles with trust policies instead.

Provide the external team with the company's AWS account root user credentials

Why wrong: Incorrect. Providing root user credentials is a severe security risk and violates AWS best practices.

Set up a VPN connection between the two accounts for network access

Why wrong: Incorrect. A VPN connection is not required for cross-account access via IAM roles; it adds unnecessary complexity.

Enable data capture for all requests to analyze usage patterns

Why wrong: Data capture incurs additional costs and does not directly minimize inference costs.

Use SageMaker Savings Plans for discounted compute rates

Why wrong: Savings Plans require a 1-3 year commitment and are not a direct strategy for JumpStart deployments.

Deploy the model on a single large instance to maximize throughput

Why wrong: A single large instance may be cost-inefficient and create a bottleneck.

Bias drift monitoring

Why wrong: Bias drift monitoring detects changes in fairness metrics, not the distribution of individual input features.

Model quality monitoring

Why wrong: Model quality monitoring tracks metrics like accuracy or precision using ground truth labels, not input feature distributions.

Feature attribution drift monitoring

Why wrong: Feature attribution drift monitors changes in SHAP values over time, not the raw feature distributions.

ModelLatency

Why wrong: ModelLatency is stable according to the scenario, so it is not the source of the increase.

5XXError

Why wrong: 5XXError counts server errors; while it may correlate, it is not a latency metric.

4XXError

Why wrong: 4XXError counts client errors; it does not directly affect latency.

SageMaker Model Registry

Why wrong: Model Registry manages model versions and approvals but does not track the full lineage of datasets and training jobs.

SageMaker Experiments

Why wrong: Experiments track trial parameters and metrics but do not create a lineage graph of artifacts and actions.

SageMaker Feature Store

Why wrong: Feature Store manages feature definitions and offline/online storage, not lineage of ML artifacts.

Use a private VPC and enable data encryption at rest using KMS

Why wrong: Encryption at rest does not encrypt inter-container traffic nor restrict access to the VPC.

Enable network isolation mode and inter-container traffic encryption

Why wrong: Network isolation prevents internet access but does not restrict endpoint access to the VPC; also, it does not encrypt inter-container traffic by itself.

Deploy the endpoint in a private subnet and use a VPC endpoint for SageMaker API

Why wrong: A private subnet and VPC endpoint restrict API calls but do not encrypt inter-container traffic.

Step scaling

Why wrong: Step scaling uses step adjustments based on breach magnitude, not a target metric value.

Predictive scaling

Why wrong: Predictive scaling forecasts future load, but it is not supported for SageMaker endpoints.

Simple scaling

Why wrong: Simple scaling is deprecated and not recommended for production use.

SageMaker Model Monitor – Model Quality

Why wrong: Model quality monitors overall model performance metrics, not fairness across groups.

SageMaker Model Monitor – Feature Attribution Drift

Why wrong: Feature attribution drift monitors changes in SHAP values, not bias metrics.

SageMaker Model Monitor – Data Quality

Why wrong: Data quality monitors input feature distributions, not bias metrics.

Reduce the number of instances to one, regardless of load

Why wrong: Arbitrarily reducing instances may cause performance degradation and throttling.

Switch from real-time to batch inference

Why wrong: Batch inference is not suitable for real-time, low-latency requirements.

Disable auto-scaling

Why wrong: Disabling auto-scaling may result in over-provisioning or under-provisioning, increasing costs or harming performance.

CloudWatch Alarm → SQS → Lambda → SageMaker Training Job

Why wrong: SQS (queue) is unnecessary; SNS is the standard target for CloudWatch Alarms to trigger actions.

CloudWatch Alarm → EventBridge → SageMaker Training Job

Why wrong: EventBridge cannot directly invoke a SageMaker training job; it requires a Lambda or Step Functions target.

CloudWatch Alarm → Lambda directly (without SNS)

Why wrong: CloudWatch Alarms cannot directly invoke Lambda; they must go through SNS or another service.

Create an IAM role in the other account with permissions to access the registry

Why wrong: The IAM role in the other account must be assumed, but the registry must also allow access via a resource policy or trust policy.

Use AWS Organizations SCP to allow cross-account access

Why wrong: SCPs are used to set permission boundaries, not to grant specific cross-account access.

Share the model artifacts via Amazon S3 bucket policy and use an IAM role

Why wrong: S3 bucket policies can share model artifacts, but the model registry access must still be granted separately.

Concept drift

Why wrong: Concept drift is a change in the relationship between features and the target variable, not in the input data structure.

Statistical drift

Why wrong: Statistical drift involves changes in the distribution (e.g., mean, variance) of feature values, not their data types.

Bias drift

Why wrong: Bias drift refers to changes in fairness metrics, not data types.

AWS CloudTrail

Why wrong: CloudTrail records API calls, not metrics.

AWS Config

Why wrong: AWS Config tracks resource configuration changes, not metrics.

SageMaker Studio

Why wrong: SageMaker Studio does not have a built-in dashboard for endpoint metrics; it can view CloudWatch logs but not create custom dashboards.