MLS-C01 Modeling Practice Test 9 — 15 Questions

Question 1

A machine learning engineer is evaluating a multi-class classification model that predicts product categories. The model outputs probabilities for 10 classes. The engineer wants to improve the model's calibration so that the predicted probabilities reflect the true likelihood of each class. Which THREE techniques can help?

Accepted Answer

Use temperature scaling. Platt scaling and isotonic regression are common calibration methods for classification models. Temperature scaling is a variant of Platt scaling for neural networks. Using a different loss function like cross-entropy helps but is not a calibration technique per se.

Answer

Increase model complexity

Answer

Use focal loss

Question 2

A data scientist is training a text classification model using Amazon SageMaker's built-in BlazingText algorithm. The dataset contains 1 million documents. Which TWO hyperparameters are most important to tune for improving model accuracy?

Accepted Answer

Learning rate. Learning rate and number of epochs are critical hyperparameters for training neural networks like BlazingText. They control how quickly the model learns and how long it trains.

Answer

Batch size

Answer

Loss function

Answer

Type of optimizer

Question 3

A company is using Amazon SageMaker to train a large language model with billions of parameters. The training job uses multiple GPU instances in a distributed fashion. The training is converging but the loss is not decreasing as expected. The data scientist suspects that the learning rate is too high. Which technique should the data scientist use to automatically adjust the learning rate during training?

Accepted Answer

Implement learning rate scheduling with a cosine annealing schedule. Learning rate scheduling, such as a cosine annealing schedule, can automatically reduce the learning rate over time. This helps the model converge better. SageMaker's built-in algorithms support learning rate scheduling, or the user can implement it in custom training scripts.

Answer

Use a fixed learning rate and train for more epochs

Answer

Increase the batch size to reduce variance

Answer

Use gradient clipping

Question 4

A data scientist is building a recommendation system using collaborative filtering. The dataset contains user-item interactions in a sparse matrix. The model will be trained on Amazon SageMaker using the built-in Factorization Machines algorithm. Which data format should the scientist use for the training data?

Accepted Answer

RecordIO-protobuf format with sparse features. Amazon SageMaker's Factorization Machines algorithm expects input in the 'application/x-recordio-protobuf' format for sparse data, or in CSV format for dense data. For sparse data, Protobuf is recommended for performance.

Answer

CSV format with all features as columns

Answer

JSON format with nested arrays

Answer

Parquet format

Question 5

A data scientist is training a binary classification model and wants to evaluate its performance using a metric that is robust to class imbalance. Which metric should be used?

Accepted Answer

F1 score. The F1 score is the harmonic mean of precision and recall and is robust to class imbalance because it considers both false positives and false negatives. Accuracy can be misleading with imbalanced classes.

Answer

Mean squared error

Answer

Area under the ROC curve (AUC)

Answer

Accuracy

Question 6

A machine learning engineer is deploying a model to an Amazon SageMaker endpoint for real-time inference. The model requires a preprocessing step that involves tokenizing text and converting it to a numerical format. To minimize latency, where should the preprocessing logic be implemented?

Accepted Answer

Inside the SageMaker inference container using the inference.py script. To minimize latency, it's best to include the preprocessing logic inside the inference container that serves the model. This avoids additional network calls to separate preprocessing services.

Answer

Using Amazon SageMaker batch transform

Answer

As a separate AWS Lambda function called before the endpoint

Answer

On the client side before sending the request

Question 7

A data scientist is training a deep learning model for image classification on Amazon SageMaker. The dataset consists of 10,000 images of size 224x224 pixels. The training job uses a single ml.p3.2xlarge instance. The data scientist notices that the GPU utilization is very low (~20%) and the training is slow. Which change would most likely improve GPU utilization?

Accepted Answer

Increase the number of data loader workers to load data in parallel. Low GPU utilization often indicates that the data loading pipeline is bottlenecked. Increasing the number of data loader workers can improve data throughput to the GPU, keeping it busy.

Answer

Use gradient accumulation

Answer

Use a larger instance type with more GPUs

Answer

Increase the batch size

Question 8

A data scientist is using Amazon SageMaker to train a model and wants to automatically stop the training job if the loss does not improve for a certain number of epochs. Which SageMaker feature can be used for this purpose?

Accepted Answer

Custom early stopping callback in the training script. SageMaker provides built-in early stopping via the 'StoppingCondition' parameter in the training job definition, or through custom training scripts that use callbacks. The simplest way is to set MaxRuntimeInSeconds, but for early stopping based on loss, the data scientist should implement a custom callback in the training script.

Answer

SageMaker Experiments

Answer

SageMaker Automatic Model Tuning

Answer

SageMaker Debugger

Question 9

A data scientist is training a binary classification model on an imbalanced dataset where the positive class represents 5% of the data. Which metric is most appropriate for evaluating model performance?

Accepted Answer

AUC-ROC. Option B is correct because AUC-ROC is robust to class imbalance and measures the trade-off between true positive rate and false positive rate. Option A is wrong because accuracy can be misleading with imbalanced data. Option C is wrong because RMSE is for regression. Option D is wrong because R-squared is for regression.

Answer

Accuracy

Answer

Root Mean Squared Error (RMSE)

Answer

R-squared

Question 10

A team is using Amazon SageMaker to train a deep learning model for image classification. The training job is taking too long, and they want to reduce training time without sacrificing model accuracy. Which approach is most effective?

Accepted Answer

Use transfer learning with a pre-trained model and fine-tune on the target dataset. Option C is correct because using a pre-trained model (transfer learning) leverages existing feature representations, reducing training time while maintaining accuracy. Option A is wrong because reducing epochs may harm accuracy. Option B is wrong because reducing batch size can increase training time. Option D is wrong because reducing image size may lose information.

Answer

Reduce the batch size

Answer

Reduce the number of training epochs

Answer

Reduce the image resolution

Question 11

A machine learning engineer is tuning hyperparameters for a gradient boosting model using Amazon SageMaker Automatic Model Tuning. The objective metric is validation accuracy. After several tuning jobs, the best accuracy achieved is 0.85, but the engineer suspects the model is overfitting. Which hyperparameter adjustment is most likely to reduce overfitting?

Accepted Answer

Increase the regularization parameter (e.g., lambda or alpha). Option D is correct because increasing the regularization parameter (e.g., lambda or alpha in XGBoost) penalizes model complexity and reduces overfitting. Option A is wrong because increasing learning rate can cause overfitting. Option B is wrong because increasing max depth increases model complexity, leading to overfitting. Option C is wrong because decreasing subsample might reduce overfitting but increasing it introduces more data, which could increase overfitting.

Answer

Increase the maximum depth of trees

Answer

Increase the subsample ratio

Answer

Increase the learning rate

Question 12

A data scientist needs to choose an algorithm for a regression problem with 50 features and 1 million training examples. The model must be interpretable and the training data fits in memory. Which algorithm is most appropriate?

Accepted Answer

Linear regression. Option A is correct because linear regression is interpretable, scales well to large datasets, and is suitable for regression. Option B is wrong because XGBoost is less interpretable. Option C is wrong because k-NN is computationally expensive at inference and not interpretable. Option D is wrong because PCA is dimensionality reduction, not a regression algorithm.

Answer

Principal Component Analysis (PCA)

Answer

XGBoost

Answer

k-Nearest Neighbors

Question 13

A company is building a sentiment analysis model for customer reviews. The dataset includes 10,000 positive and 10,000 negative reviews. The data scientist splits the data into 70% training, 15% validation, and 15% test sets. After training, the model achieves 99% accuracy on training set but only 82% on validation set. What is the most likely issue?

Accepted Answer

The model is overfitting. Option B is correct because a large gap between training and validation accuracy indicates overfitting. Option A is wrong because the dataset is balanced. Option C is wrong because underfitting would show low training accuracy. Option D is wrong because data leakage is not indicated by accuracy gap alone.

Answer

There is data leakage from validation to training

Answer

The dataset is imbalanced

Answer

The model is underfitting

Question 14

A machine learning team is deploying a real-time inference endpoint for a recommendation model using Amazon SageMaker. The model takes a long time to load (several minutes) due to its size (5 GB). Which deployment strategy minimizes the cold start latency?

Accepted Answer

Use Multi-Model Endpoints to keep the model loaded between invocations. Option D is correct because Multi-Model Endpoints allow loading models on demand, but with a large model, it may still be slow. However, Multi-Model Endpoints are designed to reduce cold start by keeping models loaded. Option A is wrong because increasing instance count doesn't reduce load time per instance. Option B is wrong because Serverless Inference has cold starts. Option C is wrong because a single instance with larger memory doesn't reduce load time significantly.

Answer

Use a single instance with a large memory size

Answer

Use SageMaker Serverless Inference

Answer

Use a larger instance type with more vCPUs

Question 15

A data scientist is training a neural network for time series forecasting. The training loss decreases initially but then starts to increase after 20 epochs. Which action should the scientist take to address this?

Accepted Answer

Implement early stopping based on validation loss. Option C is correct because early stopping monitors validation loss and stops training when it starts increasing, preventing overfitting. Option A is wrong because increasing learning rate may cause divergence. Option B is wrong because adding more layers could worsen overfitting. Option D is wrong because dropout helps with overfitting but the issue is that loss is increasing, not just overfitting; early stopping is direct.

Answer

Increase the dropout rate

Answer

Increase the learning rate

Answer

Add more layers to the network