MLS-C01 Modeling — 10 Questions

Question 1

A data scientist is training a binary classification model using Amazon SageMaker. The dataset is highly imbalanced (99% negative class, 1% positive class). The model currently achieves 99% accuracy but fails to detect most positive cases. Which metric should the data scientist primarily use to evaluate model performance?

Accepted Answer

F1 score. In highly imbalanced datasets (99% negative, 1% positive), accuracy is misleading because a model can achieve 99% accuracy by simply predicting the majority class for all instances, failing to detect any positive cases. The F1 score (option B) is the harmonic mean of precision and recall, providing a balanced measure that penalizes models that trade off recall for precision or vice versa. This makes it the primary metric for evaluating binary classification performance on imbalanced data, as it directly reflects the model's ability to correctly identify positive cases while minimizing false positives.

Answer

ROC AUC

Answer

Recall

Answer

Accuracy

Question 2

A team is building a product recommendation system using matrix factorization in Amazon SageMaker. They notice that the model's training loss decreases steadily but validation loss starts increasing after 5 epochs. What is the most likely cause?

Accepted Answer

Overfitting. In matrix factorization for recommendation systems, a decreasing training loss with an increasing validation loss after several epochs is a classic sign of overfitting. The model is memorizing the training data (including noise) rather than learning generalizable patterns, which degrades its performance on unseen validation data.

Answer

Underfitting

Answer

Not enough training data

Answer

Learning rate too high

Question 3

A company is using Amazon SageMaker to train a deep learning model on a large dataset. The training job is taking too long. The team wants to reduce training time without changing the model architecture. Which action should they take?

Accepted Answer

Use SageMaker's distributed training with multiple instances. SageMaker's distributed training with multiple instances splits the dataset and model computations across several machines, enabling parallel processing that significantly reduces wall-clock training time. This approach leverages data parallelism or model parallelism without altering the model architecture, directly addressing the need for faster training.

Answer

Increase the learning rate by a factor of 10

Answer

Reduce the number of epochs

Answer

Reduce the batch size

Question 4

A data scientist is deploying a regression model in Amazon SageMaker that predicts housing prices. The model shows high bias (underfitting). Which action is most likely to reduce bias?

Accepted Answer

Add more features or increase model complexity. High bias (underfitting) means the model is too simple to capture the underlying patterns in the data. Adding more features or increasing model complexity (e.g., using polynomial features, deeper trees, or a more flexible algorithm) directly addresses underfitting by giving the model greater capacity to learn from the data. In Amazon SageMaker, this could involve using a more complex built-in algorithm like XGBoost with deeper trees or adding feature engineering transformations in a processing job.

Answer

Reduce the amount of training data

Answer

Increase regularization strength

Answer

Use a simpler model

Question 5

A machine learning engineer is training a neural network on Amazon SageMaker using a custom Docker container. The training job fails with an error: 'CUDA out of memory.' The training instance is an ml.p3.2xlarge with 16 GB GPU memory. The model and data fit into memory when using batch size 32, but the engineer wants to maximize GPU utilization. Which approach should the engineer use to fix the out-of-memory error while maintaining efficient training?

Accepted Answer

Implement gradient accumulation with a larger effective batch size. Gradient accumulation allows the engineer to simulate a larger effective batch size by accumulating gradients over multiple forward/backward passes before performing an optimizer step. This keeps the per-step memory footprint low (avoiding CUDA out-of-memory) while maintaining training dynamics similar to a larger batch, thus maximizing GPU utilization without crashing.

Answer

Enable mixed precision training

Answer

Reduce batch size to 1

Answer

Use a CPU-only instance

Question 6

A data scientist is training a deep learning model using Amazon SageMaker. The training loss is decreasing, but the validation loss starts increasing after 10 epochs. The model is overfitting. Which TWO actions should the data scientist take to reduce overfitting? (Choose 2.)

Accepted Answer

Add dropout layers. Option D is correct because dropout layers randomly deactivate a fraction of neurons during training, which forces the network to learn more robust features and reduces co-adaptation, a common cause of overfitting. This technique is particularly effective in deep learning models trained on SageMaker, where large architectures can quickly memorize training data.

Answer

Increase the number of layers

Answer

Remove L2 regularization

Answer

Increase the number of training steps

Question 7

A company is using Amazon SageMaker to tune hyperparameters for a gradient boosting model. The objective is to minimize root mean squared error (RMSE). The data scientist wants to explore the hyperparameter space efficiently. Which THREE hyperparameter tuning strategies should the data scientist consider? (Choose 3.)

Accepted Answer

Bayesian optimization. Bayesian optimization is correct because it builds a probabilistic model of the objective function (RMSE) and uses an acquisition function to select the next hyperparameter combination to evaluate. This approach is sample-efficient, making it ideal for expensive-to-evaluate models like gradient boosting, as it balances exploration and exploitation to find optimal hyperparameters with fewer trials.

Answer

Grid search

Answer

Manual search

Question 8

A data scientist is training a binary classifier on an imbalanced dataset where the positive class represents 1% of the data. The model currently achieves 99% accuracy but a recall of only 10% on the positive class. Which metric combination should the data scientist prioritize to evaluate model improvements?

Accepted Answer

F1 score and AUC-ROC. With a highly imbalanced dataset (1% positive class), 99% accuracy is misleading because the model can achieve it by simply predicting the majority class. The low recall (10%) indicates the model fails to identify most positive instances. The F1 score balances precision and recall, providing a single metric for minority class performance, while AUC-ROC evaluates the model's ability to distinguish between classes across all thresholds, making it robust to class imbalance.

Answer

Precision and recall at 90% precision

Answer

Accuracy and RMSE

Answer

Precision and RMSE

Question 9

An e-commerce company uses a linear regression model to predict customer lifetime value (LTV). The model shows high variance on the test set, with training RMSE much lower than test RMSE. Which of the following is the MOST effective approach to reduce overfitting?

Accepted Answer

Apply L2 regularization (Ridge regression). High variance (low training RMSE, high test RMSE) indicates overfitting. L2 regularization (Ridge regression) adds a penalty proportional to the square of the coefficients, shrinking them toward zero without eliminating them, which reduces model complexity and improves generalization. This directly addresses overfitting by constraining the model's sensitivity to noise in the training data.

Answer

Use a polynomial kernel in a support vector regressor

Answer

Add more features, including interaction terms

Answer

Increase training data size by duplicating existing samples

Question 10

A company wants to use Amazon SageMaker to train a deep learning model using a custom TensorFlow script. The data is stored in an S3 bucket. Which SageMaker API operation should be used to launch the training job?

Accepted Answer

CreateTrainingJob. The correct API operation to launch a training job in Amazon SageMaker is CreateTrainingJob. This operation specifies the training algorithm (or custom script), resource configuration (instance type and count), input data configuration (pointing to S3), and output location for the model artifacts. It directly initiates the training process on SageMaker-managed infrastructure.

Answer

CreateHyperParameterTuningJob

Answer

CreateEndpoint

Answer

CreateTransformJob