MLS-C01 Modeling Practice Test 38 — 15 Questions

Question 1

A data scientist is working on a regression problem with a dataset that contains outliers. The data scientist is choosing between mean squared error (MSE) and mean absolute error (MAE) as the loss function. Which loss function is more robust to outliers?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

MAE, because it treats all errors equally.. MAE is more robust to outliers because it uses the absolute difference between predicted and actual values, which does not disproportionately penalize large errors. In contrast, MSE squares the errors, causing outliers to have a much larger influence on the loss and model updates. This makes MAE less sensitive to extreme values in regression tasks.

Answer

Both are equally robust to outliers.

Answer

MSE, because it penalizes large errors more heavily.

Answer

Neither is robust; use Huber loss instead.

Question 2

A team is using Amazon SageMaker to train a model. The training job repeatedly fails with a 'ResourceLimitExceeded' error. Which action should the team take to resolve this issue?

Accepted Answer

Request a service limit increase for SageMaker resources.. The 'ResourceLimitExceeded' error in Amazon SageMaker indicates that the AWS account has reached a service quota for SageMaker resources, such as the number of concurrent training jobs, total instance count, or specific instance types. The correct action is to request a service limit increase via the AWS Service Quotas console or by contacting AWS Support, as this directly addresses the quota cap causing the failure.

Answer

Reduce the size of the training dataset.

Answer

Switch to using Spot Instances.

Answer

Use a different instance type with less memory.

Question 3

A data scientist is using a decision tree algorithm for a classification task. The tree is very deep and achieves 100% accuracy on the training set but performs poorly on the test set. Which technique should the data scientist use to improve generalization?

Accepted Answer

Prune the decision tree.. A deep decision tree that achieves 100% training accuracy but poor test accuracy is overfitting the training data. Pruning the tree removes branches that have little statistical power, reducing complexity and improving generalization to unseen data.

Answer

Add more features to the dataset.

Answer

Reduce the number of training samples.

Answer

Increase the maximum depth of the tree.

Question 4

A machine learning team is using Amazon SageMaker to train a model with a large dataset stored in S3. The training job is taking too long. Which THREE of the following actions can reduce training time? (Choose three.)

Accepted Answer

Use a GPU instance with more powerful GPUs.. Using a GPU instance with more powerful GPUs (Option B) reduces training time because it increases the parallel compute capacity for matrix operations, which are the core of deep learning. Amazon SageMaker allows you to select instances like p3.16xlarge with NVIDIA V100 GPUs, which offer significantly higher FLOPS compared to smaller GPU instances, directly accelerating model training.

Answer

Decrease the batch size.

Answer

Increase the batch size.

Question 5

A data scientist is evaluating a classification model. The confusion matrix shows that the model has 50 true positives, 100 true negatives, 20 false positives, and 30 false negatives. Which TWO metrics can be calculated from this confusion matrix? (Choose two.)

Accepted Answer

Recall. Recall (also known as sensitivity) is calculated as TP / (TP + FN) = 50 / (50 + 30) = 0.625, measuring the proportion of actual positives correctly identified. Precision is calculated as TP / (TP + FP) = 50 / (50 + 20) = 0.714, measuring the proportion of positive predictions that are correct. Both metrics are directly derived from the four values in the confusion matrix.

Answer

R-squared

Answer

F1 score

Answer

Root mean squared error

Question 6

A company is using Amazon SageMaker to train a deep learning model for image segmentation. The training job uses a single ml.p3.2xlarge instance and takes 48 hours to complete. The team needs to reduce training time to under 12 hours to meet a deadline. The dataset is 50 GB of images stored in S3. The team currently uses File mode to download the data to the training instance. The model architecture is a convolutional neural network (CNN) with 50 layers. The team has access to multiple instances of the same type. Which approach will most effectively reduce training time?

Accepted Answer

Use SageMaker's distributed data parallelism with multiple instances.. SageMaker's distributed data parallelism splits the 50 GB dataset across multiple ml.p3.2xlarge instances, allowing each instance to process a subset of the data in parallel. This can reduce training time from 48 hours to under 12 hours, assuming near-linear scaling with the number of instances (e.g., 4 instances for a 4x speedup). The approach directly addresses the need to reduce wall-clock time without altering the model architecture or data loading method.

Answer

Reduce the number of layers in the CNN to speed up training.

Answer

Increase the batch size on the single instance to process more data per iteration.

Answer

Switch to Pipe mode to stream data from S3, reducing data loading time.

Question 7

A financial services company is developing a fraud detection model using a highly imbalanced dataset where fraudulent transactions are only 0.1% of the data. The data scientist has trained a gradient boosting model that achieves 99.9% accuracy but only detects 20% of actual fraud cases. The business requirement is to detect at least 80% of fraud while minimizing false positives. The data scientist has access to SageMaker and can use any built-in algorithm or custom script. Which approach should the data scientist take to meet the business requirement?

Accepted Answer

Use Amazon SageMaker Random Cut Forest (RCF) algorithm for anomaly detection.. Amazon SageMaker Random Cut Forest (RCF) is an unsupervised anomaly detection algorithm that is well-suited for highly imbalanced datasets like this one (0.1% fraud). Unlike supervised methods that struggle with extreme class imbalance, RCF isolates anomalies by measuring how many random cuts are needed to separate a point from the rest of the data, making it effective at detecting rare fraud cases without requiring balanced training data. This approach can meet the 80% fraud detection requirement while minimizing false positives by tuning the anomaly score threshold.

Answer

Keep the model but adjust the classification threshold to increase recall.

Answer

Use random under-sampling of the majority class to balance the dataset and retrain the model.

Answer

Use random oversampling of the minority class to balance the dataset and retrain the model.

Question 8

A data scientist is building a regression model to predict house prices. The dataset includes features such as square footage, number of bedrooms, year built, and location. After training a linear regression model, the data scientist notices that the residuals have a clear pattern when plotted against predicted values: they increase with predicted values. The model also has high RMSE. Which action should the data scientist take to improve the model?

Accepted Answer

Apply a log transformation to the target variable.. A pattern in residuals indicates non-linearity, and transforming the target variable (e.g., log transformation) can stabilize variance and linearize relationships. Option A is wrong because removing outliers does not address the underlying non-linearity or heteroscedasticity; it may even discard useful data. Option B is wrong because L1 regularization helps reduce overfitting by penalizing large coefficients, but it does not fix non-constant variance or non-linearity. Option D is wrong because while interaction terms can model relationships between features, they do not directly address the pattern of increasing residuals (heteroscedasticity) and may not resolve the non-linearity in the target.

Answer

Remove outliers from the dataset.

Answer

Use L1 regularization (Lasso) to reduce overfitting.

Answer

Add interaction terms between features.

Question 9

A company is building a text classification model to categorize customer support tickets. The dataset is highly imbalanced with 95% of tickets belonging to 'General Inquiry' and 5% to 'Complaint'. The data scientist is using a random forest classifier. Which metric is most appropriate for evaluating model performance on the minority class?

Accepted Answer

F1-score for 'Complaint'. In a highly imbalanced dataset (95% General Inquiry, 5% Complaint), accuracy is misleading because a model that predicts 'General Inquiry' for every ticket would achieve 95% accuracy but completely fail on the minority class. The F1-score for 'Complaint' is the harmonic mean of precision and recall, providing a balanced evaluation of the model's ability to correctly identify complaints without being skewed by the majority class. For a random forest classifier, this metric directly addresses the minority class performance, which is the primary concern.

Answer

Accuracy

Answer

ROC AUC

Answer

Precision for 'Complaint'

Question 10

A data scientist is training a deep learning model for image segmentation using a U-Net architecture. The model overfits severely. The scientist tries L2 regularization, dropout, and data augmentation, but validation loss remains high while training loss approaches zero. Which additional strategy is most likely to reduce overfitting?

Accepted Answer

Implement early stopping based on validation loss. Early stopping monitors validation loss and halts training when it stops improving, directly addressing overfitting by preventing the model from memorizing noise after it has learned generalizable features. Since the training loss is near zero but validation loss remains high, the model has already started overfitting, and early stopping can cut training at the point just before overfitting worsens.

Answer

Increase the batch size

Answer

Use a larger learning rate

Answer

Add more convolutional layers to increase model capacity

Question 11

A company wants to build a real-time anomaly detection system for IoT sensor data. The data arrives as a stream of numerical values. The model should adapt to concept drift over time. Which approach is most suitable?

Accepted Answer

Train an online learning model, such as stochastic gradient descent (SGD) with a sliding window. Online learning with stochastic gradient descent (SGD) using a sliding window allows the model to continuously update its parameters as new IoT sensor data arrives, adapting to concept drift without retraining from scratch. The sliding window ensures that the model focuses on the most recent data distribution, discarding outdated patterns, which is essential for real-time anomaly detection in streaming environments.

Answer

Use a static deep learning model trained once on historical data

Answer

Use a stateful LSTM with fixed weights

Answer

Batch train a random forest model monthly

Question 12

A data scientist is using Amazon SageMaker to train a linear regression model. The training data has 10 features and 100,000 observations. The model's training loss is decreasing, but the validation loss starts increasing after a few epochs. Which step should the data scientist take first to address this issue?

Accepted Answer

Reduce the learning rate. The increasing validation loss while training loss decreases is a classic sign of overfitting. Reducing the learning rate (Option B) is the first step to stabilize training by allowing the optimizer to take smaller, more controlled steps, which can help the model converge to a better local minimum and reduce validation loss. In SageMaker, this is typically adjusted via the `learning_rate` hyperparameter in the estimator.

Answer

Add more features to the model

Answer

Increase the batch size

Answer

Increase the number of epochs

Question 13

A data scientist is training a binary classifier on a dataset with 1 million rows and 500 features. The model uses XGBoost and achieves an AUC of 0.95 on the training set but only 0.72 on the test set. The scientist suspects overfitting. Which combination of hyperparameter adjustments is most likely to improve generalization?

Accepted Answer

Decrease 'max_depth' and increase 'min_child_weight'. Decreasing 'max_depth' reduces the complexity of individual trees, preventing the model from learning overly specific patterns in the training data. Increasing 'min_child_weight' forces the algorithm to require a higher sum of instance weights (hessian) before further partitioning, which acts as a regularization mechanism that discourages splits on noisy or sparse data. Together, these adjustments directly combat overfitting in XGBoost by limiting tree depth and requiring more evidence for splits, which improves generalization from the training AUC of 0.95 to a higher test AUC.

Answer

Increase 'max_depth' and decrease 'learning_rate'

Answer

Increase 'subsample' and decrease 'colsample_bytree'

Answer

Decrease 'gamma' and increase 'learning_rate'

Question 14

A company uses Amazon SageMaker to deploy a model that predicts customer churn. The model is retrained weekly. The data scientist notices that the model's accuracy remains high, but the business reports that the model is not capturing new churn patterns. What is the most likely cause?

Accepted Answer

The model is experiencing concept drift. Concept drift occurs when the statistical properties of the target variable change over time, causing the model's predictions to become less relevant even if accuracy metrics remain high. In this scenario, the model is retrained weekly but still fails to capture new churn patterns because the underlying customer behavior has shifted—a classic sign of concept drift rather than a data or overfitting issue. Amazon SageMaker's built-in Model Monitor can detect such drift by comparing inference data distributions against a baseline.

Answer

The model is underfitting the data

Answer

The model has data leakage from future data

Answer

The model is overfitting to the training data

Question 15

A data scientist is building a recommendation system for an e-commerce platform. The dataset includes user-item interactions (clicks, purchases, ratings). The scientist wants to use matrix factorization. Which approach is most appropriate for handling implicit feedback (e.g., clicks) rather than explicit ratings?

Accepted Answer

Use weighted alternating least squares (WALS) with confidence weights. Weighted Alternating Least Squares (WALS) is specifically designed for implicit feedback scenarios because it treats unobserved interactions as negative signals with low confidence, rather than missing values. By assigning confidence weights (e.g., based on click frequency or dwell time), WALS can factorize the implicit feedback matrix effectively, avoiding the bias introduced by treating all zeros as true negatives.

Answer

Use k-means clustering to segment users and then use item popularity within clusters

Answer

Use singular value decomposition (SVD) on the interaction matrix with missing values filled with 0

Answer

Use a deep neural network with a softmax output to predict item probabilities