MLS-C01 Modeling Practice Test 31 — 15 Questions

Question 1

A data scientist ran a hyperparameter tuning job for an XGBoost model. The tuning job completed, but the best validation RMSE is 2.34. The data scientist believes the model can perform better. Based on the exhibit, which change to the tuning strategy is most likely to improve the model's performance?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Increase the static num_round hyperparameter to 500. Increasing the static `num_round` hyperparameter to 500 allows the model to train for more boosting rounds, which can reduce underfitting and lower the RMSE further. The current best validation RMSE of 2.34 suggests the model may not have converged, and additional rounds can help the XGBoost model learn more complex patterns, provided overfitting is monitored with early stopping.

Answer

Use random search instead of Bayesian optimization

Answer

Change the objective to binary:logistic

Answer

Increase the maximum value of eta to 1.0

Question 2

A data scientist is training a binary classification model for fraud detection. The dataset is highly imbalanced with only 1% fraudulent transactions. The model currently achieves 99% accuracy but only catches 5% of actual fraud cases. Which metric should the data scientist focus on to better evaluate model performance?

Accepted Answer

Recall. In fraud detection with highly imbalanced data (1% fraud), accuracy is misleading because a model can achieve 99% accuracy by simply predicting 'not fraud' for all transactions. Recall (true positive rate) measures the proportion of actual fraud cases correctly identified, which is critical when the cost of missing fraud is high. The model currently catches only 5% of fraud, so improving recall is the primary goal to reduce false negatives.

Answer

Precision

Answer

Accuracy

Answer

Root Mean Squared Error (RMSE)

Question 3

A team is using SageMaker to train a deep learning model for image classification. The training job is failing with a 'CUDA out of memory' error. The team is using a p3.2xlarge instance (1 GPU, 16 GB GPU memory). The dataset consists of 256x256 RGB images. Which action is MOST likely to resolve the error without changing the instance type?

Accepted Answer

Reduce the batch size. The 'CUDA out of memory' error indicates that the GPU memory is exhausted. Reducing the batch size directly decreases the memory footprint per training step, allowing the model to fit within the 16 GB GPU memory of the p3.2xlarge instance. This is the most direct and effective fix without changing the instance type.

Answer

Increase the batch size to utilize GPU more efficiently

Answer

Enable automatic model tuning to optimize hyperparameters

Answer

Use Spot Instances to reduce cost

Question 4

A company is using SageMaker to deploy a real-time inference endpoint for a natural language processing model. The model receives input text and returns predictions. The data scientist notices that the endpoint latency increases significantly under load. Which design change would MOST effectively reduce latency?

Accepted Answer

Use an inference pipeline to combine preprocessing and model inference. An inference pipeline in SageMaker allows you to chain preprocessing logic directly with the model inference within the same endpoint container. This eliminates the need for separate Lambda functions or client-side preprocessing, which reduces network round-trips and serialization overhead, thereby lowering latency under load.

Answer

Enable data capture for monitoring

Answer

Switch to batch transform for real-time predictions

Answer

Increase the number of instances behind the endpoint

Question 5

A machine learning engineer is training a linear regression model on a dataset with 50 features. After training, the model achieves high accuracy on the training set but poor accuracy on the test set. Which technique should the engineer use to address this issue?

Accepted Answer

Apply L1 or L2 regularization. The model exhibits overfitting: high training accuracy but poor test accuracy. L1 (Lasso) or L2 (Ridge) regularization penalizes large coefficients, reducing model complexity and improving generalization. This directly addresses the variance problem without requiring more data or features.

Answer

Train a deeper neural network with more layers

Answer

Add more features through feature engineering

Answer

Increase the size of the training dataset

Question 6

A data scientist is using SageMaker to train an XGBoost model for regression. The training data contains categorical features with high cardinality (e.g., zip code with over 10,000 unique values). Which feature engineering approach is MOST appropriate to avoid overfitting while preserving predictive power?

Accepted Answer

Use target encoding with smoothing. Target encoding with smoothing is the most appropriate approach because it replaces each high-cardinality category with the mean of the target variable for that category, regularized by a smoothing factor that pulls estimates toward the global mean. This preserves predictive power by capturing the relationship between the category and the target while preventing overfitting on rare categories that have few samples. In SageMaker XGBoost, this avoids the curse of dimensionality from one-hot encoding and the arbitrary ordering from label encoding.

Answer

One-hot encode the categorical features

Answer

Apply frequency encoding based on category occurrence

Answer

Label encode the categorical features

Question 7

A company is building a recommender system using matrix factorization. The dataset contains user-item interactions. The model is trained on a large dataset, but the recommendations for new users are poor. Which approach would MOST effectively address this cold-start problem?

Accepted Answer

Incorporate user demographic features as side information. Matrix factorization models learn latent factors only from user-item interactions. For new users with no history, the model cannot compute a meaningful latent vector, leading to poor recommendations. Incorporating user demographic features as side information allows the model to initialize or infer latent factors for new users based on their attributes, directly addressing the cold-start problem.

Answer

Switch to item-based collaborative filtering only

Answer

Increase the number of latent factors in the model

Answer

Use only implicit feedback signals for training

Question 8

A data scientist is training a deep learning model for object detection. The training loss decreases rapidly in the first few epochs but then plateaus at a high value. The validation loss starts increasing after a few epochs. Which adjustment is MOST likely to improve generalization?

Accepted Answer

Implement early stopping with a patience parameter. The described behavior—training loss plateauing at a high value while validation loss increases—is a classic sign of overfitting. Early stopping with a patience parameter halts training when validation performance stops improving, preventing the model from memorizing noise and thus improving generalization. This directly addresses the overfitting without altering the model architecture or data distribution.

Answer

Add more convolutional layers

Answer

Use more aggressive data augmentation

Answer

Increase the learning rate

Question 9

A company uses SageMaker to deploy a model for predicting customer churn. The model was trained on historical data and achieves 85% accuracy on the test set. After deployment, the model's predictions are significantly worse on new data due to changes in customer behavior. What is the MOST likely cause?

Accepted Answer

Concept drift in the underlying data distribution. The model's performance degradation on new data, despite high accuracy on the test set, is a classic symptom of concept drift. Concept drift occurs when the statistical properties of the target variable (customer churn) change over time due to shifts in customer behavior, making the trained model's decision boundary obsolete. SageMaker deployed the model as a persistent endpoint, but the underlying data distribution has evolved, so the model no longer generalizes to the current environment.

Answer

Data leakage during training

Answer

The training dataset was too small

Answer

The model is overfitting to the training data

Question 10

A data scientist is building a model to predict insurance claim amounts. The target variable is right-skewed with many small claims and a few very large claims. The scientist wants to minimize the impact of outliers. Which loss function or transformation is MOST appropriate?

Accepted Answer

Apply a log transformation to the target variable. Applying a log transformation to the target variable reduces skewness and mitigates the impact of outliers by compressing the scale of large values. This makes the distribution more symmetric and suitable for models like linear regression. Option A (mean squared error) is sensitive to outliers. Option B (quantile loss) predicts the median, which is robust but not typical for mean prediction. Option C (Poisson loss) is designed for count data, not continuous skewed targets. Option D (log transformation) is the standard approach for right-skewed continuous targets.

Answer

Use mean squared error loss without any transformation

Answer

Use quantile loss to predict the median

Answer

Use Poisson loss assuming the target follows a Poisson distribution

Question 11

A data scientist is training a random forest model for a binary classification task. The dataset has 100,000 samples and 500 features. The model is overfitting. Which TWO actions are MOST likely to reduce overfitting?

Accepted Answer

Reduce the maximum depth of each tree. Reducing the maximum depth of each tree limits the complexity of individual trees, preventing them from memorizing noise and specific patterns in the training data. This is a standard regularization technique for random forests that directly combats overfitting by controlling the variance of the model.

Answer

Increase the number of trees in the forest

Answer

Increase the number of features considered at each split

Answer

Use all features for each tree

Question 12

A data scientist is using Amazon SageMaker to train a deep learning model. The training job is taking too long. Which THREE actions can reduce training time?

Accepted Answer

Use incremental training to continue from a previous model. Incremental training allows you to start from a previously trained model, which reduces training time because the model does not need to learn from scratch. SageMaker's incremental training loads the existing model artifacts and continues training on new data, significantly cutting down the time required to converge compared to full retraining.

Answer

Use Spot Instances to reduce cost

Answer

Decrease the batch size to reduce memory usage

Question 13

A company is building a sentiment analysis model for customer reviews. The dataset is balanced with 10,000 positive and 10,000 negative reviews. The model achieves 95% accuracy on the test set but fails to generalize to new reviews from a different product category. Which TWO techniques can improve generalization?

Accepted Answer

Use stratified k-fold cross-validation during training. Stratified k-fold cross-validation ensures that each fold maintains the same class distribution as the original dataset, which helps the model learn more robust patterns across different subsets of data. This technique reduces variance in the evaluation and improves generalization to unseen data from different product categories by preventing overfitting to idiosyncrasies of a single train-test split.

Answer

Increase the training dataset size by collecting more reviews

Answer

Add more features like review length and word count

Answer

Use a more complex model with more layers

Question 14

Refer to the exhibit. A data scientist wants to use SageMaker to train a model using data stored in 'my-bucket'. The training job fails with an access denied error. What is the MOST likely cause?

Accepted Answer

The training job is not requesting server-side encryption with AES256. SageMaker training jobs require server-side encryption with AES256 when the S3 bucket uses default encryption with AES256. If the training job does not explicitly request SSE-S3 (AES256) in its S3 data source configuration, the S3 service denies access, resulting in an 'access denied' error even if the IAM role has full S3 permissions.

Answer

The bucket uses AWS KMS key encryption instead of AES256

Answer

The bucket name in the policy does not match the actual bucket name

Answer

The bucket is publicly accessible but the IAM role lacks permissions

Question 15

Refer to the exhibit. A data scientist is evaluating a binary classification model for spam detection. The exhibit shows a single prediction instance. What is the model's prediction for this instance?

Accepted Answer

Spam. The model's prediction is 'Spam' because the prediction instance shows a probability of 0.95 for the 'Spam' class, which exceeds the typical decision threshold of 0.5 used in binary classification. Since the probability for 'Spam' is higher than for 'Ham' (0.05), the model assigns the instance to the class with the highest probability, which is Spam.

Answer

Ham

Answer

0.95

Answer

The model is unsure because probability is not 1.0