MLS-C01 Modeling Practice Test 27 — 15 Questions

Question 1

A data scientist is training a binary classification model using SageMaker XGBoost and notices that training loss decreases but validation loss increases after a few epochs. Which action should the data scientist take to address this issue?

Accepted Answer

Set early stopping based on validation loss. The increasing validation loss while training loss decreases is a classic sign of overfitting. Setting early stopping based on validation loss halts training when the validation loss stops improving, preventing the model from memorizing noise in the training data. SageMaker XGBoost's `early_stopping_rounds` parameter monitors the evaluation metric on the validation set and stops training if no improvement is seen for a specified number of rounds.

Answer

Increase the number of rounds

Answer

Increase the learning rate

Answer

Increase the maximum tree depth

Question 2

A company wants to deploy a machine learning model that requires very low latency predictions (under 10ms). The model is a small ensemble of decision trees. Which SageMaker deployment option is most suitable?

Accepted Answer

SageMaker endpoint with a single instance. C is correct because a SageMaker endpoint with a single instance provides a persistent, real-time inference API that can achieve sub-10ms latency for a small ensemble of decision trees. The endpoint keeps the model loaded in memory and uses synchronous HTTP requests, minimizing cold start and network overhead, which is essential for low-latency predictions.

Answer

SageMaker Notebook instance

Answer

AWS Lambda function with the model packaged

Answer

SageMaker Batch Transform

Question 3

A data scientist is using SageMaker to train a deep learning model with a large dataset stored in S3. The training is taking a long time. Which action would most likely reduce training time without sacrificing accuracy?

Accepted Answer

Use SageMaker Pipe Input mode. SageMaker Pipe Input mode streams training data directly from S3 into the algorithm without first downloading it to the local EBS volume. This eliminates the I/O bottleneck caused by large dataset downloads, significantly reducing training time while preserving accuracy because the model sees the same data.

Answer

Increase the batch size

Answer

Use a smaller instance type

Answer

Reduce the number of epochs

Question 4

A data scientist is tuning a linear regression model and observes that the model has high bias and low variance. Which action is most likely to improve model performance?

Accepted Answer

Add more features. High bias and low variance indicate underfitting, meaning the model is too simple to capture the underlying patterns in the data. Adding more features increases model complexity, allowing it to learn more relevant relationships and reduce bias. This directly addresses the core issue of underfitting in linear regression.

Answer

Reduce the number of features

Answer

Increase regularization

Answer

Reduce the amount of training data

Question 5

A company is building a recommendation system for an e-commerce platform. The data includes user IDs and item IDs. Which SageMaker built-in algorithm is most appropriate?

Accepted Answer

Factorization Machines. Factorization Machines (FM) are specifically designed for recommendation tasks with sparse, high-dimensional categorical data like user IDs and item IDs. They model pairwise interactions between features (e.g., user-item interactions) using factorized parameters, making them highly effective for collaborative filtering and implicit feedback scenarios in e-commerce.

Answer

BlazingText

Answer

XGBoost

Answer

Image Classification

Question 6

A data scientist is using principal component analysis (PCA) for dimensionality reduction before training a classifier. The classifier's performance on the test set is poor. What is the most likely cause?

Accepted Answer

Too few principal components were retained, losing important information. C is correct because PCA is an unsupervised dimensionality reduction technique that projects data onto principal components capturing the maximum variance. If too few components are retained, the reduced representation may discard features that are critical for the classifier to distinguish between classes, leading to poor test performance due to underfitting.

Answer

The classifier is overfitting

Answer

The data was not scaled before applying PCA

Answer

Too many principal components were retained, including noise

Question 7

A company is using a custom Docker container in SageMaker for training. The training job fails with 'ResourceLimitExceeded' error. Which action should the data scientist take?

Accepted Answer

Request a limit increase for the instance type. The 'ResourceLimitExceeded' error in SageMaker indicates that the AWS account has reached a service quota for the specified instance type (e.g., ml.p3.2xlarge). This is a quota limit, not a performance or resource exhaustion issue within the training job itself. The correct action is to request a limit increase via the AWS Service Quotas console or AWS Support, which raises the maximum number of concurrent instances or total vCPUs allowed for that instance family.

Answer

Use a smaller instance type

Answer

Reduce the number of epochs

Answer

Use a pre-built SageMaker container instead

Question 8

A data scientist needs to evaluate a binary classification model's performance. Which metric is most appropriate when the cost of false positives is very high?

Accepted Answer

Precision. Precision is the most appropriate metric when the cost of false positives is very high because it measures the proportion of positive identifications that were actually correct. In binary classification, precision = TP / (TP + FP), so a high precision means very few false positives occur, directly minimizing the costly error type.

Answer

Accuracy

Answer

F1 score

Answer

Recall

Question 9

A data scientist is training a model using SageMaker and wants to automatically stop training when the model stops improving. Which TWO options can be used?

Accepted Answer

Built-in early stopping in XGBoost. Built-in early stopping in XGBoost (Option B) is correct because XGBoost natively supports an `early_stopping_rounds` parameter that halts training when the validation metric stops improving for a specified number of rounds. SageMaker Debugger (Option C) is correct because it can monitor training metrics in real time and trigger a stop action via a built-in or custom rule (e.g., `VanishingGradient` or `LossNotDecreasing`) when the model stops improving, integrating with SageMaker's `StopTraining` API.

Answer

AWS Step Functions

Answer

CloudWatch Alarms

Answer

SageMaker Model Monitor

Question 10

A company is deploying a real-time inference endpoint with SageMaker. The model is a large neural network that requires GPU acceleration. Which TWO configurations must be set?

Accepted Answer

Instance type with GPU. Option A is correct because deploying a real-time inference endpoint with a large neural network that requires GPU acceleration necessitates selecting an instance type with a GPU, such as the ml.p3 or ml.g4dn series, to provide the parallel processing power needed for low-latency inference. Without a GPU instance, the model would fall back to CPU, leading to unacceptable inference times for large neural networks.

Answer

Batch transform job

Answer

Production variant

Answer

Training container image

Question 11

A data scientist is building a binary classifier and wants to evaluate model performance. Which THREE metrics are most commonly used?

Accepted Answer

Precision. Precision is a core metric for binary classifiers, measuring the proportion of true positive predictions among all positive predictions. It is especially important when the cost of false positives is high, such as in spam detection or fraud alert systems.

Answer

Mean Absolute Error

Answer

RMSE

Question 12

Refer to the exhibit. A data scientist is trying to create a SageMaker training job but receives an access denied error. The IAM policy shown is attached to their role. What is the most likely reason for the error?

Accepted Answer

The policy only allows CreateTrainingJob when the training job status is 'Failed', which is never true initially. Option A is correct because the IAM policy uses a `Condition` block with `sagemaker:TrainingJobStatus` set to `Failed`. When a `CreateTrainingJob` API call is made, the training job status is not yet set (it is `Creating` or `InProgress`), so the condition evaluates to false, and the request is denied. The policy only grants permission when the status equals `Failed`, which never occurs at creation time.

Answer

The Action is not allowed because 'CreateTrainingJob' is misspelled

Answer

There is an explicit deny in another policy

Answer

The Resource is set to '*' which does not include the specific training job ARN

Question 13

Refer to the exhibit. A data scientist runs the AWS CLI command to create a SageMaker training job. The job fails immediately with 'ValidationException: Invalid instance type'. What is the most likely issue?

Accepted Answer

The training image URI is for a different AWS region. The error 'ValidationException: Invalid instance type' occurs because the training image URI specified in the command points to an Amazon ECR repository in a different AWS region than where the SageMaker training job is being created. SageMaker validates that the image URI is accessible from the current region; if the URI references a region that does not contain the XGBoost image or the instance type is not supported in that region's ECR, the validation fails. The instance type itself (ml.m5.large) is valid for XGBoost, but the mismatch between the image's region and the job's region triggers the exception.

Answer

The IAM role ARN is invalid

Answer

The S3 bucket 'my-bucket' does not exist or the role lacks permissions

Answer

The instance type ml.m5.large does not support the XGBoost image

Question 14

Refer to the exhibit. What is the recall of the model?

Accepted Answer

0.80. Recall is calculated as True Positives divided by the sum of True Positives and False Negatives. From the confusion matrix, True Positives = 80 and False Negatives = 20, so recall = 80 / (80 + 20) = 0.80. Option B is correct.

Answer

0.85

Answer

0.89

Answer

0.90

Question 15

A data scientist is training a classification model on an imbalanced dataset where the positive class represents only 5% of the data. Which technique would BEST address the class imbalance without discarding data?

Accepted Answer

Use SMOTE to generate synthetic samples for the minority class. SMOTE (Synthetic Minority Oversampling Technique) is the best choice because it generates synthetic samples for the minority class by interpolating between existing minority instances, effectively balancing the dataset without discarding any data. This avoids the information loss of undersampling and the overfitting risk of simple random oversampling, making it ideal for a 5% positive class scenario.

Answer

Randomly undersample the majority class

Answer

Adjust the decision threshold to 0.95

Answer

Randomly oversample the minority class with replacement