MLA-C01 ML Solution Monitoring, Maintenance, and Security Practice Test 2 — 15 Questions

Question 1

A company wants to reduce costs for a production SageMaker endpoint that has predictable traffic patterns. They have purchased a Savings Plan. What additional step can they take to further optimize costs while maintaining performance?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use SageMaker Inference Recommender to right-size the endpoint. SageMaker Inference Recommender provides instance type and configuration recommendations to right-size endpoints, balancing cost and performance. It is the appropriate tool for cost optimization beyond a Savings Plan.

Answer

Reduce the number of instances to one, regardless of load

Answer

Switch from real-time to batch inference

Answer

Disable auto-scaling

Question 2

A machine learning engineer is setting up a retraining pipeline that triggers when concept drift is detected. They plan to use CloudWatch Alarms to monitor the model's accuracy metric. When drift is detected, they want to automatically start a SageMaker training job. Which architecture should they use?

Accepted Answer

CloudWatch Alarm → SNS → Lambda → SageMaker Training Job. CloudWatch Alarms cannot directly invoke SageMaker training jobs; they require an intermediary like SNS to trigger a Lambda function, which then calls the SageMaker API to start the training job. This pattern ensures reliable decoupling and allows the Lambda function to handle any preprocessing or conditional logic before launching the job.

Answer

CloudWatch Alarm → SQS → Lambda → SageMaker Training Job

Answer

CloudWatch Alarm → EventBridge → SageMaker Training Job

Answer

CloudWatch Alarm → Lambda directly (without SNS)

Question 3

A company needs to give a data science team in another AWS account access to deploy a model from a shared model registry. Which approach should they use to grant cross-account access?

Accepted Answer

Attach a resource-based policy to the model registry granting the other account access. Resource-based policies (also called resource policies) can be attached to the model registry to allow cross-account access. IAM roles are used for intra-account access, not for granting access to resources in another account.

Answer

Create an IAM role in the other account with permissions to access the registry

Answer

Use AWS Organizations SCP to allow cross-account access

Answer

Share the model artifacts via Amazon S3 bucket policy and use an IAM role

Question 4

A company uses SageMaker Model Monitor for data quality. They notice that monitoring jobs are failing intermittently with constraint violations. Upon review, they see that some features have different data types in production compared to the baseline (e.g., string instead of integer). Which type of drift is this?

Accepted Answer

Schema drift. Schema drift occurs when the structure or data types of features in production data differ from the baseline used during model training. In this scenario, a feature that was an integer in the baseline is now a string in production, which is a classic example of schema drift. SageMaker Model Monitor detects this by comparing the inferred schema of production data against the baseline schema, flagging any type mismatches as constraint violations.

Answer

Concept drift

Answer

Statistical drift

Answer

Bias drift

Question 5

A team wants to monitor the number of requests and latency of their SageMaker endpoint using a unified dashboard. Which AWS service should they use to create a custom dashboard with these metrics?

Accepted Answer

Amazon CloudWatch Dashboards. Amazon CloudWatch Dashboards allow you to create custom views of metrics from any source, including SageMaker endpoint metrics like Invocations and Latency. SageMaker itself does not provide a dashboard for these metrics.

Answer

AWS CloudTrail

Answer

AWS Config

Answer

SageMaker Studio

Question 6

A company uses SageMaker JumpStart to deploy a foundation model for a summarization task. They want to minimize costs while still meeting a latency requirement of under 2 seconds. Which option should they consider?

Accepted Answer

Use SageMaker Inference Recommender to select the cheapest instance that meets latency. SageMaker Inference Recommender runs load tests against your model on various instance types and provides latency and cost metrics. By selecting the cheapest instance that still meets the sub-2-second latency requirement, you directly minimize cost while satisfying the performance constraint. This is the most systematic and cost-effective approach for this scenario.

Answer

Deploy the model on a serverless endpoint

Answer

Enable auto-scaling to handle variable traffic

Answer

Use the largest GPU instance to ensure fast inference

Question 7

A company wants to automatically trigger model retraining when SageMaker Model Monitor detects data drift. Which TWO services should they integrate to achieve this automation? (Choose two.)

Accepted Answer

AWS Lambda. EventBridge can schedule or trigger events based on monitoring job completions, and Lambda can execute the retraining code. SNS is used for notifications but not direct invocation of retraining. Step Functions can orchestrate but is not required for this simple trigger.

Answer

AWS Step Functions

Answer

Amazon CloudWatch Alarms

Answer

Amazon SNS

Question 8

A company wants to secure data in transit between the client and SageMaker endpoint, and between containers in the same endpoint. Which THREE configurations should they apply? (Choose three.)

Accepted Answer

Enable network isolation for the model containers. Enforce inter-container traffic encryption, use HTTPS for the endpoint, and enable network isolation to prevent internet exposure. VPC-only mode controls network access but does not directly encrypt traffic.

Answer

Enable data encryption at rest using KMS

Answer

Use a VPC-only endpoint configuration

Question 9

A machine learning engineer is setting up model quality monitoring for a binary classification model. They have ground truth labels available in Amazon S3. Which TWO steps are required to configure model quality monitoring? (Choose two.)

Accepted Answer

Create a schedule for the monitoring job to run at regular intervals. A baseline must be computed from training data and ground truth, and a schedule for monitoring jobs must be defined. The monitoring job then compares production predictions against ground truth.

Answer

Create a CloudWatch Alarm for accuracy degradation

Answer

Use Amazon SageMaker Clarify for bias detection

Answer

Enable data capture on the endpoint to collect predictions

Question 10

A data science team uses Amazon SageMaker Model Monitor to detect data drift in production. They notice that the schema of incoming data (number of features) has changed compared to the training baseline. Which type of monitor is BEST suited to detect this issue?

Accepted Answer

Data quality monitor. The Data quality monitor in SageMaker Model Monitor is specifically designed to detect violations in the input data schema, such as changes in the number of features, feature types, or missing values, by comparing incoming data against a baseline computed from the training dataset. Since the issue is a structural change in the schema (number of features), the Data quality monitor is the correct choice.

Answer

Bias drift monitor

Answer

Feature attribution drift monitor

Answer

Model quality monitor

Question 11

An ML engineer wants to be notified when the average inference latency of a SageMaker endpoint exceeds 500 ms for 2 consecutive evaluation periods. Which AWS service combination should they use?

Accepted Answer

CloudWatch Alarm + SNS. CloudWatch collects the 'Latency' metric from SageMaker endpoints. A CloudWatch Alarm can be set on this metric with a threshold of 500 ms for 2 consecutive periods. SNS then sends notification (email, SMS, etc.). Lambda is for automated remediation, not required for notification only.

Answer

SageMaker Model Monitor + SNS

Answer

EventBridge + Lambda

Answer

SageMaker Clarify + SNS

Question 12

A company deploys a model for credit risk assessment on a SageMaker endpoint. To comply with internal policies, they must ensure that the endpoint only allows inference requests from within a specific VPC and that the data is encrypted at rest. Which configuration meets these requirements?

Accepted Answer

Configure endpoint in VPC-only mode and enable KMS encryption for the endpoint. Configuring the SageMaker endpoint in VPC-only mode ensures that inference requests can only originate from within the specified VPC, satisfying the network restriction requirement. Enabling KMS encryption for the endpoint encrypts the model data and inference data at rest using a customer-managed key, meeting the encryption-at-rest policy.

Answer

Enable network isolation mode and use KMS for endpoint encryption

Answer

Configure endpoint in VPC-only mode and enable inter-container traffic encryption

Answer

Deploy endpoint in a public subnet and use a security group to restrict traffic

Question 13

A machine learning team uses SageMaker Pipelines and wants to automatically retrain a model when data drift is detected. They have set up Model Monitor to publish drift violations to CloudWatch. Which approach provides a COMPLETE serverless retraining pipeline triggered by drift detection?

Accepted Answer

Configure a CloudWatch Alarm on drift metric → SNS topic → Lambda function that starts the SageMaker Pipeline execution. The recommended pattern: CloudWatch Alarm triggers on drift metric → SNS message → Lambda function (receives SNS) → starts SageMaker Pipeline execution. EventBridge could also trigger on SNS events, but Lambda is simplest. EventBridge can schedule retraining but does not directly react to specific drift alarms. Step Functions would add unnecessary complexity.

Answer

Use SageMaker Model Monitor to directly invoke a SageMaker Pipeline when drift is detected

Answer

Use EventBridge to schedule retraining daily regardless of drift

Answer

Create an EventBridge rule that triggers on Model Monitor drift events to start the pipeline

Question 14

A financial institution uses SageMaker to train and deploy models. They need to track every experiment, model version, and deployment step for audit purposes. Which SageMaker feature should they use to capture the full lineage of artifacts, actions, and contexts?

Accepted Answer

SageMaker ML Lineage Tracking. SageMaker ML Lineage Tracking creates a graph of artifacts (datasets, models), actions (training jobs, deployment), and contexts (experiments). It provides a complete audit trail. Experiments alone track trials but not lineage. Model Registry tracks model versions but not full pipeline lineage. Clarify is for bias monitoring.

Answer

SageMaker Clarify

Answer

SageMaker Model Registry

Answer

SageMaker Experiments

Question 15

A company notices that the prediction distribution of their deployed model has shifted significantly from the training data distribution, but the input data distribution remains unchanged. Which type of drift is occurring, and what is the MOST likely cause?

Accepted Answer

Concept drift; the underlying relationship between features and target has changed. Concept drift occurs when the relationship between input features and target variable changes, causing prediction distribution shift. Data drift would be input distribution change. The scenario specifically says input distribution unchanged but predictions shifted, indicating concept drift. Possible cause: change in customer behavior or market conditions.

Answer

Data drift; the training data is no longer representative of the current environment

Answer

Bias drift; the model is making unfair predictions against a protected group

Answer

Model drift; the model has degraded due to software issues