MLS-C01 Modeling Practice Test 31 — 15 Questions

Question 1

A data scientist is using SageMaker to train an XGBoost model for regression. The training data contains categorical features with high cardinality (e.g., zip code with over 10,000 unique values). Which feature engineering approach is MOST appropriate to avoid overfitting while preserving predictive power?

Accepted Answer

Use target encoding with smoothing. Target encoding with smoothing is the most appropriate approach because it replaces each high-cardinality category with the mean of the target variable for that category, regularized by a smoothing factor that pulls estimates toward the global mean. This preserves predictive power by capturing the relationship between the category and the target while preventing overfitting on rare categories that have few samples. In SageMaker XGBoost, this avoids the curse of dimensionality from one-hot encoding and the arbitrary ordering from label encoding.

Answer

One-hot encode the categorical features

Answer

Apply frequency encoding based on category occurrence

Answer

Label encode the categorical features

Question 2

A company is building a recommender system using matrix factorization. The dataset contains user-item interactions. The model is trained on a large dataset, but the recommendations for new users are poor. Which approach would MOST effectively address this cold-start problem?

Accepted Answer

Incorporate user demographic features as side information. Matrix factorization models learn latent factors only from user-item interactions. For new users with no history, the model cannot compute a meaningful latent vector, leading to poor recommendations. Incorporating user demographic features as side information allows the model to initialize or infer latent factors for new users based on their attributes, directly addressing the cold-start problem.

Answer

Switch to item-based collaborative filtering only

Answer

Increase the number of latent factors in the model

Answer

Use only implicit feedback signals for training

Question 3

A data scientist is training a deep learning model for object detection. The training loss decreases rapidly in the first few epochs but then plateaus at a high value. The validation loss starts increasing after a few epochs. Which adjustment is MOST likely to improve generalization?

Accepted Answer

Implement early stopping with a patience parameter. The described behavior—training loss plateauing at a high value while validation loss increases—is a classic sign of overfitting. Early stopping with a patience parameter halts training when validation performance stops improving, preventing the model from memorizing noise and thus improving generalization. This directly addresses the overfitting without altering the model architecture or data distribution.

Answer

Add more convolutional layers

Answer

Use more aggressive data augmentation

Answer

Increase the learning rate

Question 4

A company uses SageMaker to deploy a model for predicting customer churn. The model was trained on historical data and achieves 85% accuracy on the test set. After deployment, the model's predictions are significantly worse on new data due to changes in customer behavior. What is the MOST likely cause?

Accepted Answer

Concept drift in the underlying data distribution. The model's performance degradation on new data, despite high accuracy on the test set, is a classic symptom of concept drift. Concept drift occurs when the statistical properties of the target variable (customer churn) change over time due to shifts in customer behavior, making the trained model's decision boundary obsolete. SageMaker deployed the model as a persistent endpoint, but the underlying data distribution has evolved, so the model no longer generalizes to the current environment.

Answer

Data leakage during training

Answer

The training dataset was too small

Answer

The model is overfitting to the training data

Question 5

A data scientist is building a model to predict insurance claim amounts. The target variable is right-skewed with many small claims and a few very large claims. The scientist wants to minimize the impact of outliers. Which loss function or transformation is MOST appropriate?

Accepted Answer

Apply a log transformation to the target variable. Using a log transformation or modeling with a log-link function can reduce skewness and impact of outliers. Option A (Mean squared error) is sensitive to outliers. Option B (Quantile loss) is robust but less common for mean prediction. Option D (Poisson loss) is for count data. Option C (Log transformation of target) is standard for skewed continuous targets.

Answer

Use mean squared error loss without any transformation

Answer

Use quantile loss to predict the median

Answer

Use Poisson loss assuming the target follows a Poisson distribution

Question 6

A data scientist is training a random forest model for a binary classification task. The dataset has 100,000 samples and 500 features. The model is overfitting. Which TWO actions are MOST likely to reduce overfitting?

Accepted Answer

Reduce the maximum depth of each tree. Reducing the maximum depth of each tree limits the complexity of individual trees, preventing them from memorizing noise and specific patterns in the training data. This is a standard regularization technique for random forests that directly combats overfitting by controlling the variance of the model.

Answer

Increase the number of trees in the forest

Answer

Increase the number of features considered at each split

Answer

Use all features for each tree

Question 7

A data scientist is using Amazon SageMaker to train a deep learning model. The training job is taking too long. Which THREE actions can reduce training time?

Accepted Answer

Use incremental training to continue from a previous model. Incremental training allows you to start from a previously trained model, which reduces training time because the model does not need to learn from scratch. SageMaker's incremental training loads the existing model artifacts and continues training on new data, significantly cutting down the time required to converge compared to full retraining.

Answer

Use Spot Instances to reduce cost

Answer

Decrease the batch size to reduce memory usage

Question 8

A company is building a sentiment analysis model for customer reviews. The dataset is balanced with 10,000 positive and 10,000 negative reviews. The model achieves 95% accuracy on the test set but fails to generalize to new reviews from a different product category. Which TWO techniques can improve generalization?

Accepted Answer

Use stratified k-fold cross-validation during training. Option B is correct because stratified k-fold cross-validation ensures that each fold maintains the same class distribution as the original dataset, which helps the model learn more robust patterns across different subsets of data. This technique reduces variance in the evaluation and improves generalization to unseen data from different product categories by preventing overfitting to idiosyncrasies of a single train-test split.

Answer

Increase the training dataset size by collecting more reviews

Answer

Add more features like review length and word count

Answer

Use a more complex model with more layers

Question 9

Refer to the exhibit. A data scientist wants to use SageMaker to train a model using data stored in 'my-bucket'. The training job fails with an access denied error. What is the MOST likely cause?

Accepted Answer

The training job is not requesting server-side encryption with AES256. Option C is correct because SageMaker training jobs require server-side encryption with AES256 when the S3 bucket uses default encryption with AES256. If the training job does not explicitly request SSE-S3 (AES256) in its S3 data source configuration, the S3 service denies access, resulting in an 'access denied' error even if the IAM role has full S3 permissions.

Answer

The bucket uses AWS KMS key encryption instead of AES256

Answer

The bucket name in the policy does not match the actual bucket name

Answer

The bucket is publicly accessible but the IAM role lacks permissions

Question 10

Refer to the exhibit. A data scientist is evaluating a binary classification model for spam detection. The exhibit shows a single prediction instance. What is the model's prediction for this instance?

Accepted Answer

Spam. The model's prediction is 'Spam' because the prediction instance shows a probability of 0.95 for the 'Spam' class, which exceeds the typical decision threshold of 0.5 used in binary classification. Since the probability for 'Spam' is higher than for 'Ham' (0.05), the model assigns the instance to the class with the highest probability, which is Spam.

Answer

Ham

Answer

0.95

Answer

The model is unsure because probability is not 1.0

Question 11

Refer to the exhibit. A data scientist runs the above CLI command to create a SageMaker training job. The job fails with an error 'Unable to read data from s3://bucket/train/'. What is the MOST likely cause?

Accepted Answer

The IAM role does not have permissions to read from the S3 bucket. The error 'Unable to read data from s3://bucket/train/' indicates that the SageMaker training job cannot access the S3 input data. The most common cause is that the IAM role specified in the command does not have the necessary s3:GetObject permission on the S3 bucket or objects. SageMaker uses the IAM role to assume permissions for reading training data, and without proper S3 read access, the job fails at the data loading stage.

Answer

The training image is not accessible

Answer

The instance type does not support the required memory

Answer

The training job is in a different region than the S3 bucket

Question 12

A data scientist is training a binary classification model on a dataset where the positive class represents only 1% of the data. The model's accuracy is 99%, but the recall for the positive class is 0%. Which metric should the scientist use to evaluate the model's performance effectively?

Accepted Answer

Area under the Precision-Recall curve (PR AUC). In a highly imbalanced dataset where the positive class is only 1%, accuracy is misleading because a model can achieve 99% accuracy by simply predicting the negative class for all samples, resulting in 0% recall for the positive class. The Area under the Precision-Recall curve (PR AUC) is the correct metric because it focuses on the performance of the positive class by evaluating the trade-off between precision and recall, making it sensitive to changes in the minority class. Unlike ROC AUC, which can be overly optimistic in imbalanced settings due to the large number of true negatives, PR AUC provides a more realistic assessment of model performance for rare events.

Answer

Area under the ROC curve (ROC AUC)

Answer

Accuracy

Answer

F1 score

Question 13

A machine learning team is deploying a real-time inference endpoint for a fraud detection model using Amazon SageMaker. The model requires low latency (<100 ms) and the team expects a steady stream of requests with occasional spikes. Which instance type and deployment strategy should they use to minimize cost while meeting latency requirements?

Accepted Answer

Use ml.c5 instances with a single endpoint and provisioned concurrency.. Option D is correct because ml.c5 instances are compute-optimized for low-latency inference, and provisioned concurrency pre-warms the endpoint to handle steady traffic with spikes without cold starts, meeting the <100 ms requirement cost-effectively. This combination avoids over-provisioning while ensuring consistent performance.

Answer

Use ml.p3 instances with a multi-model endpoint.

Answer

Use AWS Lambda with a container image for serverless inference.

Answer

Use ml.m5 instances with a production variant and auto-scaling.

Question 14

A data scientist is training a deep learning model on Amazon SageMaker using a PyTorch estimator. The training job runs on a single ml.p3.2xlarge instance but is taking too long. The scientist wants to reduce training time by using distributed data parallelism across multiple GPUs. Which change to the training script and SageMaker estimator is required?

Accepted Answer

Modify the script to use torch.nn.parallel.DistributedDataParallel and set instance_count to 2 in the estimator.. Option C is correct because to achieve distributed data parallelism across multiple GPUs on multiple instances with PyTorch, you must modify the training script to use `torch.nn.parallel.DistributedDataParallel` (DDP), which handles gradient synchronization across nodes. Additionally, you must set `instance_count` to 2 (or more) in the SageMaker PyTorch estimator to launch multiple instances, each with its own GPU, enabling true multi-node distributed training.

Answer

Add the SageMaker distributed data parallelism configuration in the estimator and modify the script to use the SageMaker distributed library.

Answer

Change the framework to TensorFlow and use tf.distribute.MirroredStrategy with instance_count=2.

Answer

Modify the script to use torch.nn.DataParallel and keep instance_count as 1.

Question 15

A team is training a linear regression model to predict house prices. After training, they observe that the model has high bias (underfitting). Which action is most likely to reduce bias?

Accepted Answer

Add more relevant features and increase model complexity.. High bias (underfitting) means the model is too simple to capture the underlying patterns in the data. Adding more relevant features and increasing model complexity (e.g., using polynomial features or more interaction terms) gives the linear regression model greater capacity to fit the training data, directly reducing bias. This aligns with the bias-variance tradeoff, where increasing complexity lowers bias at the cost of potentially increasing variance.

Answer

Increase the regularization strength.

Answer

Reduce the amount of training data.

Answer

Decrease the number of model parameters.