MLS-C01 Machine Learning Implementation and Operations — 20 Questions

Question 1

Drag and drop the steps to evaluate a trained model using SageMaker Model Monitor in the correct order.

Accepted Answer

Step 1: Enable data capture on the endpoint. Step 2: Create a baseline from training data. Step 3: Schedule the monitoring job. Step 4: Review the monitoring reports.. Model Monitor requires enabling data capture, creating baseline, schedule, and reviewing reports.

Answer

Step 1: Create a baseline from training data. Step 2: Enable data capture on the endpoint. Step 3: Schedule the monitoring job. Step 4: Review the monitoring reports.

Answer

Step 1: Enable data capture on the endpoint. Step 2: Schedule the monitoring job. Step 3: Create a baseline from training data. Step 4: Review the monitoring reports.

Answer

Step 1: Create a baseline from training data. Step 2: Schedule the monitoring job. Step 3: Enable data capture on the endpoint. Step 4: Review the monitoring reports.

Question 2

An ML team is using Amazon SageMaker to train a model. They notice that the training job is taking longer than expected and the CloudWatch metrics show high GPU utilization but low CPU utilization. Which action is MOST likely to improve training speed?

Accepted Answer

Use SageMaker Pipe mode to stream data from S3 to reduce I/O bottleneck. High GPU utilization with low CPU utilization typically indicates that the GPU is actively computing and the CPU is not a bottleneck for data preprocessing. However, this does not rule out I/O inefficiencies at the storage layer. SageMaker Pipe mode streaming eliminates the need to download data to the local filesystem first, reducing I/O wait times. Even when the GPU is busy, reducing I/O overhead can speed up the training by decreasing the time between batches and allowing the GPU to process more data per unit time.

Answer

Switch to a CPU-only instance to avoid GPU overhead

Answer

Increase the number of training instances

Answer

Use a larger GPU instance with more GPU memory

Question 3

A company uses SageMaker to run training jobs on a schedule. The training data is stored in an S3 bucket that receives new data every hour. Which TWO approaches can the company use to trigger a training job when new data arrives?

Accepted Answer

Set up an Amazon EventBridge rule that triggers on S3 Object Created events and targets a Lambda function. Both options B and E are correct because they use an event-driven approach to trigger a training job when new data arrives. Option B uses Amazon EventBridge to capture S3 Object Created events and route them to a Lambda function, which starts the SageMaker training job via the AWS SDK. Option E uses the native S3 event notification feature to directly invoke a Lambda function on object creation, which then starts the training job. Both methods are valid and eliminate the need for polling or scheduled jobs.

Answer

Use an SQS queue to buffer S3 events and poll from a training instance

Answer

Use AWS Step Functions with a scheduled execution

Answer

Use CloudWatch Logs to monitor S3 access logs and trigger a Lambda function

Question 4

A data scientist has trained a model using SageMaker and wants to deploy it to an endpoint. Which step is required before deployment?

Accepted Answer

Register the model in SageMaker Model Registry. Before deploying a trained model to a SageMaker endpoint, you must create a SageMaker Model object. This can be done via the 'Create model' API or by using the Model Registry, but the registry is optional. None of the provided options describe the required step: uploading training data (A) is done before training, creating a custom Docker image (B) is only needed for custom containers, retraining (C) is unrelated to deployment, and registering in the Model Registry (D) is not mandatory because you can create a model directly.

Answer

Upload the training data to S3

Answer

Create a custom Docker image

Answer

Retrain the model with more data

Question 5

Which TWO actions can help reduce inference latency for a SageMaker endpoint?

Accepted Answer

Use SageMaker Neo to optimize the model. SageMaker Neo optimizes trained models for a specific target hardware platform by compiling the model graph, fusing operations, and applying quantization and pruning techniques. This reduces the model's memory footprint and computational requirements, directly decreasing inference latency at the endpoint. Option D, 'Enable SageMaker Endpoint Cache,' is not a real SageMaker feature; there is no managed endpoint cache service. Batch transform (A) is used for offline inference, not for reducing endpoint latency. Larger instance types (C) primarily increase throughput and may not improve per-request latency. Multi-model endpoints (E) reduce model loading overhead but do not directly affect inference latency.

Answer

Switch to batch transform

Answer

Use a larger instance type

Answer

Enable SageMaker Endpoint Cache

Answer

Use a multi-model endpoint

Question 6

Refer to the exhibit. A SageMaker training job is launched with the CLI command shown. The job fails with an error 'S3 data distribution type not supported for File mode'. What is the most likely fix?

Accepted Answer

Change TrainingInputMode to Pipe. The error 'S3 data distribution type not supported for File mode' occurs because SageMaker's File mode downloads the entire dataset to the training instance's local storage before training begins, which does not support the ShardedByS3Key data distribution type. Changing TrainingInputMode to Pipe mode resolves this, as Pipe mode streams data directly from S3 to the algorithm, allowing each instance to receive a unique shard of the data without requiring local storage.

Answer

Change InstanceCount to 2

Answer

Increase VolumeSizeInGB to 100

Answer

Increase MaxRuntimeInSeconds to 7200

Question 7

A data scientist is reviewing the training logs from a SageMaker training job. The model's loss decreases steadily and accuracy increases. However, when the model is evaluated on a holdout test set, the accuracy is only 0.65. Which issue does this behavior suggest?

Accepted Answer

The model is overfitting to the training data.. The model shows strong performance on the training set (loss decreases, accuracy increases) but poor generalization to the holdout test set (accuracy 0.65). This classic symptom indicates overfitting, where the model has memorized the training data rather than learning generalizable patterns. In SageMaker, this can occur when the model is too complex relative to the amount of training data, or when regularization techniques like dropout or L2 weight decay are insufficient.

Answer

The learning rate is too high.

Answer

The model is underfitting the training data.

Answer

The training data is corrupted.

Question 8

A company is using Amazon SageMaker to deploy a model. The model is a large ensemble that requires 8 GB of memory. The company wants to minimize endpoint cost. Which instance type should they choose?

Accepted Answer

ml.m5.large. The ml.m5.large instance provides 8 GB of memory (2 vCPUs, 8 GiB RAM), which meets the model's requirement of 8 GB. It is typically priced lower than ml.c5.xlarge in AWS (e.g., $0.115/hr vs $0.17/hr), making it the most cost-effective choice among instances that can accommodate the memory. The ml.c5.xlarge also provides 8 GB but at a higher cost, while smaller instances lack sufficient memory and larger instances incur unnecessary expense.

Answer

ml.c5.xlarge

Answer

ml.t2.medium

Answer

ml.c5.large

Answer

ml.c5.2xlarge

Question 9

A data scientist is deploying a machine learning model on Amazon SageMaker for real-time inference. The model requires low-latency predictions and must be able to handle up to 1000 requests per second. Which TWO actions should the data scientist take to ensure the endpoint can meet the performance requirements? (Choose 2.)

Accepted Answer

Configure an auto scaling policy for the endpoint based on invocation metrics.. Configuring an auto scaling policy based on invocation metrics (e.g., SageMakerVariantInvocationsPerInstance) allows the endpoint to dynamically scale to handle variable loads up to 1000 requests per second while maintaining low latency. Option E is correct because deploying on a single large instance (e.g., ml.p3.16xlarge) can provide enough compute capacity to meet the peak load of 1000 req/s without the complexity of scaling, assuming the instance can handle the throughput. Option B (data capture) is for monitoring and retraining, not directly for meeting latency or throughput requirements. Option A (multi-model endpoint) is designed for cost savings with many small models, not for high-throughput single-model serving. Option C (serverless endpoint) may have cold start delays and is typically not recommended for sustained high-throughput low-latency workloads.

Answer

Use a multi-model endpoint to host multiple models on the same instance.

Answer

Enable data capture to Amazon S3 for model monitoring and retraining.

Answer

Use a serverless inference endpoint to automatically scale.

Question 10

An ML team is deploying a model to a SageMaker endpoint for real-time inference. The model is large (2 GB) and requires GPU for low-latency inference. The team wants to minimize cost while maintaining a response time of under 200 ms. Which instance configuration and SageMaker feature would be best?

Accepted Answer

Use a GPU instance (ml.p3.2xlarge) with SageMaker Elastic Inference.. The best choice. By using a GPU instance (ml.p3.2xlarge) combined with SageMaker Elastic Inference, the team can attach a fractional GPU accelerator to the instance, providing the necessary GPU compute for low-latency inference while minimizing cost compared to using a full GPU instance alone. Option B (batch transform) is designed for offline inference on large datasets, not real-time. Option C (serverless inference) typically uses CPU instances and may incur cold starts, which can exceed the 200 ms response time requirement. Option D (multi-model endpoint) is optimized for hosting multiple models on a single instance, but does not directly address GPU acceleration or cost minimization for a single large model.

Answer

Use a batch transform job on GPU instances.

Answer

Use a serverless inference endpoint with a CPU instance.

Answer

Use a multi-model endpoint on a GPU instance.

Question 11

A data scientist is training a model using Amazon SageMaker and wants to track hyperparameter tuning jobs, training jobs, and model metrics. The team also needs to compare experiments visually. Which AWS service should be used?

Accepted Answer

Amazon SageMaker Experiments. Amazon SageMaker Experiments is purpose-built for tracking and comparing machine learning experiments, including hyperparameter tuning jobs, training jobs, and model metrics. It provides a visual interface for comparison. Option A is wrong because Amazon CloudWatch Logs is for storing logs and monitoring, not for experiment tracking and comparison. Option B is wrong because AWS Glue DataBrew is a visual data preparation tool, not for experiment management. Option C is wrong because AWS Step Functions orchestrates workflows but does not provide experiment tracking or visual comparison.

Answer

Amazon CloudWatch Logs

Answer

AWS Glue DataBrew

Answer

AWS Step Functions

Question 12

A company uses Amazon SageMaker to train machine learning models. The training data contains personally identifiable information (PII). The company needs to ensure that the data is encrypted in transit between S3 and SageMaker. Which configuration is REQUIRED?

Accepted Answer

Use SageMaker in a VPC with VPC endpoints for S3. Amazon SageMaker uses HTTPS by default for all communication with Amazon S3, ensuring data is encrypted in transit. No additional configuration is required to achieve this. Option A is not required because VPC endpoints provide private connectivity but do not add encryption beyond what HTTPS already provides. Option B (S3 server-side encryption) protects data at rest. Option C (bucket policy requiring SSL) is already satisfied as SageMaker uses HTTPS. Option D (custom Docker image with TLS) is unnecessary as SageMaker handles TLS for S3 traffic by default.

Answer

Enable S3 server-side encryption

Answer

Set the S3 bucket policy to require SSL

Answer

Use a custom Docker image with TLS configured

Question 13

A company is using Amazon Forecast for demand forecasting. The data includes time series data for multiple items. The company wants to ensure that the forecast is updated daily as new data arrives. Which approach should be used to automate this process?

Accepted Answer

Enable the 'AutoPredictor' feature in Forecast to automatically update the predictor when new data arrives.. Amazon Forecast's AutoPredictor feature automatically retrains the predictor when new data is imported via CreateDatasetImportJob, eliminating the need for manual scheduling. Option A is incorrect because it mentions CreatePredictorBacktestExportJob, which is used for exporting backtest results, not for retraining the predictor. The correct API for retraining is CreatePredictor. Option B is wrong because Forecast does not support real-time data ingestion via Kinesis. Option C is wrong because Amazon SageMaker is not used for Forecast retraining; Forecast handles its own model training.

Answer

Use AWS Lambda to invoke the Forecast CreateDatasetImportJob and CreatePredictorBacktestExportJob APIs on a daily schedule triggered by Amazon CloudWatch Events.

Answer

Use Amazon Kinesis Data Streams to stream new data directly into Forecast.

Answer

Use Amazon SageMaker to retrain the model daily and replace the forecast endpoint.

Question 14

A company is using Amazon SageMaker to train a deep learning model. The training job is failing with an error 'CUDA out of memory'. The training instance is an ml.p3.2xlarge with 16 GB GPU memory. The model architecture and batch size are appropriate for this instance size. What is the most likely cause of this error?

Accepted Answer

Enable automatic mixed precision (AMP) training to reduce memory usage.. Enabling automatic mixed precision (AMP) training reduces GPU memory usage by storing tensors in half-precision (FP16) where possible, while keeping critical operations in full precision (FP32). This directly addresses the 'CUDA out of memory' error on an ml.p3.2xlarge instance (16 GB GPU memory) without changing the model architecture or batch size, which are already appropriate.

Answer

Reduce the number of epochs.

Answer

Increase the number of GPUs by using a distributed training instance type.

Answer

Use a smaller instance type to force lower memory usage.

Question 15

A data scientist is deploying a model using Amazon SageMaker. The model endpoint needs to handle real-time inference requests with low latency. The model is a large ensemble of 10 deep learning models, each approximately 500 MB. What is the most cost-effective deployment strategy that meets the low-latency requirement?

Accepted Answer

Use a SageMaker multi-model endpoint to host all models on one or more instances.. A SageMaker multi-model endpoint (MME) allows hosting multiple models on a single or few instances, dynamically loading them from Amazon S3 into memory as needed. This is the most cost-effective option for a large ensemble of 500 MB models because it avoids the expense of separate endpoints or multiple instances per model, while still supporting low-latency real-time inference by keeping frequently used models cached.

Answer

Deploy each model to a separate endpoint and use a load balancer.

Answer

Use a single endpoint with multiple instances behind it.

Answer

Use a SageMaker batch transform job to process inference requests in batches.

Question 16

A company is using Amazon SageMaker to train a model with a custom algorithm. The training script reads data from an S3 bucket using boto3. The training job fails with an 'AccessDenied' error when trying to access the S3 bucket. The IAM role attached to the SageMaker notebook instance has full S3 access. What is the most likely cause?

Accepted Answer

The SageMaker execution role used for the training job does not have S3 access permissions.. The IAM role attached to the SageMaker notebook instance is used for interactive development, but training jobs run under a separate SageMaker execution role. Even if the notebook role has full S3 access, the training job's execution role must also have explicit S3 permissions. The 'AccessDenied' error indicates that the execution role lacks the necessary s3:GetObject or s3:ListBucket actions for the S3 bucket.

Answer

The S3 bucket has a bucket policy that denies access from the SageMaker service.

Answer

The training script is using an incorrect S3 bucket name.

Answer

The SageMaker training job is not configured to use the S3 VPC endpoint.

Question 17

A machine learning engineer is deploying a model using AWS Lambda for real-time inference. The model is a scikit-learn RandomForestClassifier with 100 trees, serialized as a pickle file of 150 MB. The Lambda function has 3 GB memory allocated. However, the inference requests are timing out after 30 seconds. What is the most likely cause?

Accepted Answer

The model is loaded from S3 on every invocation, causing high latency.. The default behavior of loading a model from S3 on every Lambda invocation introduces significant latency. Each invocation must download the 150 MB pickle file from S3 over the network, deserialize it, and then run inference, which easily exceeds the 30-second timeout. The model should be loaded once outside the handler (in global scope) and reused across invocations to avoid this overhead.

Answer

scikit-learn is not compatible with AWS Lambda.

Answer

The Lambda function does not have enough memory to load the model.

Answer

The Lambda function timeout is set too low; increase it to 5 minutes.

Question 18

A data scientist is using Amazon SageMaker for hyperparameter tuning. The tuning job uses a Bayesian optimization strategy. After 10 training jobs, the objective metric (validation accuracy) has plateaued at 0.85. The data scientist wants to explore more diverse hyperparameter combinations. What should the data scientist do?

Accepted Answer

Increase the exploration weight in the tuning job configuration.. In Bayesian optimization, the exploration weight controls the trade-off between exploring new hyperparameter regions and exploiting known good regions. Increasing this weight encourages the acquisition function to sample more diverse combinations, which can help escape a plateau. Option C is correct because it directly addresses the need for greater diversity in the search space.

Answer

Decrease the exploration weight in the tuning job configuration.

Answer

Switch to random search strategy.

Answer

Increase the number of parallel training jobs.

Question 19

An IAM policy is attached to a SageMaker execution role. A data scientist tries to create a training job using a custom algorithm stored in an ECR repository. The training job fails with an 'AccessDenied' error when pulling the Docker image from ECR. What is the missing permission?

Accepted Answer

ecr:GetDownloadUrlForLayer and ecr:BatchGetImage on the ECR repository. When SageMaker pulls a custom Docker image from ECR during training job creation, the execution role needs permissions to download the image layers. The required actions are ecr:GetDownloadUrlForLayer (to generate pre-signed URLs for each layer) and ecr:BatchGetImage (to retrieve image metadata and layer manifests). Without these, the 'AccessDenied' error occurs because SageMaker cannot authenticate or fetch the container image from the ECR repository.

Answer

ecr:PutImage on the ECR repository

Answer

s3:GetObject on the ECR repository

Answer

sagemaker:CreateTrainingJob on the ECR resource

Question 20

A DevOps engineer created a SageMaker notebook instance using the Terraform configuration shown. The notebook instance is in a VPC with a public subnet. However, the notebook instance cannot access the internet. What is the most likely cause?

Accepted Answer

The subnet does not have a route to an internet gateway.. A SageMaker notebook instance in a VPC with a public subnet requires a route to an internet gateway (IGW) in the subnet's route table to access the internet. Without that route, traffic from the notebook cannot reach the internet, even if `direct_internet_access` is enabled. The Terraform configuration likely omitted the route to the IGW, causing the connectivity failure.

Answer

The role_arn is incorrect or missing permissions.

Answer

The instance type ml.t2.medium does not support internet access.

Answer

The direct_internet_access parameter is set to 'Enabled' but should be 'Disabled'.