MLS-C01 Modeling Practice Test 20 — 15 Questions

Question 1

A company has deployed a real-time inference endpoint using SageMaker for a fraud detection model. The model uses a Random Forest classifier. The endpoint receives predictions but the latency is too high. The metric shows p99 latency of 500ms, but the requirement is under 200ms. The team has already optimized the instance type to the maximum allowed by their budget. The data scientist suggests: A) Reducing the number of trees in the Random Forest model. B) Switching to a linear model like Logistic Regression. C) Enabling SageMaker's batch transform instead of real-time endpoint. D) Adding more instances to the endpoint behind a load balancer. Which option will MOST effectively reduce latency while maintaining acceptable accuracy?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Reduce the number of trees in the Random Forest model. (Reducing the number of trees) is the most effective method to reduce latency while maintaining acceptable accuracy. Fewer trees directly decrease inference time of the Random Forest model, although it may slightly impact accuracy. Switching to a linear model (Option A) would reduce latency but likely result in significant accuracy loss. Batch transform (Option C) is not suitable for real-time inference. Adding more instances (Option D) improves throughput but not per-request latency.

Answer

Switch to a linear model like Logistic Regression

Answer

Enable SageMaker's batch transform

Answer

Add more instances to the endpoint

Question 2

A company is building a binary classifier to predict customer churn. The dataset has 10,000 samples with 500 churners (5% positive class). After training a logistic regression model, the precision is 0.8 and recall is 0.2. Which metric should the data scientist focus on to improve the model's ability to identify churners while minimizing false positives?

Accepted Answer

Increase recall. The model's recall is very low (0.2), meaning it misses most churners. Since the goal is to identify churners (positive class), improving recall should be the primary focus. Option A (accuracy) is misleading due to class imbalance. Option B (precision) is already high (0.8), so further improvement would likely reduce recall. Option D (F1 score) balances precision and recall, but the immediate need is to address the low recall, not to balance both.

Answer

Increase accuracy

Answer

Increase precision

Answer

Increase F1 score

Question 3

A data scientist is training a deep learning model on Amazon SageMaker using a custom Docker container. The training job fails with an error 'OutOfMemoryError: CUDA out of memory'. The instance type is ml.p3.2xlarge (8 GB GPU memory). The model has 50 million parameters. What is the most likely cause and solution?

Accepted Answer

The model uses FP32 precision; enable mixed precision training. A model with 50 million parameters in FP32 precision requires approximately 200 MB per parameter (4 bytes each = 200 MB for 50M), plus additional memory for activations, gradients, and optimizer states, which can easily exceed the 8 GB GPU memory of ml.p3.2xlarge. Mixed precision training (FP16) halves the memory usage for tensors, reducing the overall footprint and often fitting the model within GPU limits. Option A (instance type) may solve the problem but is more expensive and unnecessary if mixed precision works. Option B (batch size) is a contributing factor but not the most likely root cause, as even a batch size of 1 may still cause OOM due to parameter storage. Option C (gradient checkpointing) trades compute for memory by recomputing activations, but does not address the primary issue of parameter storage in FP32. Therefore, enabling mixed precision is the most direct and cost-effective solution.

Answer

The instance type is insufficient; switch to ml.p3.8xlarge

Answer

The batch size is too large; reduce batch size

Answer

Enable gradient checkpointing to reduce memory

Question 4

A team deployed a SageMaker endpoint for real-time inference using a PyTorch model. After monitoring, they notice that the latency is highly variable, with p99 latency 10x the p50 latency. The endpoint uses a single ml.c5.2xlarge instance with auto-scaling based on average CPU utilization. Which change is most likely to reduce latency variability?

Accepted Answer

Change the auto-scaling metric to 'InvocationsPerInstance'. Scaling based on InvocationsPerInstance allows the endpoint to react more quickly to changes in request volume, reducing the queueing that causes high p99 latency. Option A (increasing batch size) would actually increase latency. Option B (pre-warming) helps with cold starts but not queueing from traffic spikes. Option C (GPU instance) is unlikely to help if the model is CPU-bound and would not address the root cause of latency variability.

Answer

Increase the batch size for inference

Answer

Pre-warm the model by sending dummy requests every minute

Answer

Switch to a GPU instance type

Question 5

A data scientist is using Amazon SageMaker to train a model with the built-in XGBoost algorithm. The dataset contains missing values. What is the default behavior of SageMaker XGBoost regarding missing values?

Accepted Answer

It automatically learns the best direction (left or right) for missing values during training. SageMaker's built-in XGBoost algorithm treats missing values (i.e., NaN, None, or 0 by default) as a separate category. During training, it automatically learns the best direction (left or right) to assign missing values at each split, based on the reduction in loss. This is the default behavior (Option D). Option A is incorrect because XGBoost does not raise an error; it handles missing values internally. Option B is incorrect because XGBoost does not impute with the column mean; that would require data preprocessing. Option C is incorrect because rows with missing values are not removed; they are included in training with the learned direction.

Answer

It raises an error and stops training

Answer

It imputes missing values with the column mean

Answer

It removes rows with missing values

Question 6

A company is fine-tuning a BERT model on Amazon SageMaker for a text classification task. The training script uses PyTorch and Hugging Face Transformers. The training job completes successfully, but the final model accuracy is low. The dataset has 10,000 labeled samples. What is the most likely cause and solution?

Accepted Answer

The model is overfitting due to small dataset; use a pre-trained checkpoint and fine-tune only top layers. Fine-tuning the entire BERT model on only 10,000 samples leads to overfitting, resulting in low accuracy. The recommended approach is to use a pre-trained checkpoint and fine-tune only the top layers, which leverages transfer learning and reduces the risk of overfitting. Option A (instance type) impacts training speed, not accuracy directly. Option C (learning rate) could be a factor but overfitting is the most likely given the dataset size. Option D (data loader bug) would typically cause errors, not low accuracy without errors.

Answer

The instance type is insufficient; use a larger instance

Answer

The learning rate is too high; reduce it

Answer

The training script has a bug in the data loader

Question 7

A data scientist is using Amazon SageMaker Autopilot to automatically build a model for a regression problem. The dataset has 100 features and 50,000 rows. Autopilot recommends a model with an R² of 0.85 on the validation set. However, when deployed to production, the model performs poorly (R² of 0.2). What is the most likely cause?

Accepted Answer

The production data distribution has shifted from the training data distribution. A large discrepancy between validation and production performance often indicates data drift. Option A (overfitting) is possible but less likely given validation performance. Option C (feature importance) is not the direct cause. Option D (Autopilot bug) is rare.

Answer

The model is overfitting to the training data

Answer

The model is underfitting due to insufficient training

Answer

Autopilot selected the wrong features

Question 8

A machine learning team is using SageMaker to train a model with the built-in Linear Learner algorithm. The dataset has 1 million rows and 20 features. The training completes, but the model's mean squared error (MSE) is high. Which parameter adjustment is most likely to reduce MSE?

Accepted Answer

Increase the number of epochs. Increasing the number of epochs allows the model to see the data more times, helping it converge to a lower training error, thus reducing MSE. Option A is incorrect: increasing mini-batch size typically improves computational efficiency but can make convergence slower per epoch, potentially requiring even more epochs to converge; it may not directly reduce MSE. Option B is incorrect: cross-entropy is a loss function for classification problems, not regression; Linear Learner with MSE is appropriate for regression. Option D is incorrect: increasing the learning rate can cause the optimizer to overshoot the minimum or diverge, often increasing rather than decreasing MSE.

Answer

Increase the mini-batch size

Answer

Change the loss function to cross-entropy

Answer

Increase the learning rate

Question 9

A data scientist is training a recurrent neural network (RNN) for time series forecasting. The training loss decreases steadily for the first 10 epochs, then plateaus. The validation loss starts increasing after epoch 10. What is the most appropriate action?

Accepted Answer

Stop training early and use the model from epoch 10. The validation loss increasing while training loss decreases indicates overfitting. Early stopping halts training before overfitting worsens, preserving the model that performed best on validation data (epoch 10). Option B (continuing training) would increase overfitting. Option C (adding layers) may exacerbate overfitting. Option D (increasing batch size) is not directly addressing overfitting and may not help.

Answer

Continue training for more epochs

Answer

Add more layers to the network

Answer

Increase the batch size

Question 10

A company is using Amazon SageMaker to host a model for real-time inference. The model is a large ensemble of 10 XGBoost models, each 2 GB. The endpoint uses a single ml.c5.18xlarge instance. The inference latency is high (average 2 seconds). Which change would most effectively reduce latency?

Accepted Answer

Use SageMaker Multi-Model Endpoints to serve each model independently. Serialization/deserialization of large models is a bottleneck; SageMaker Multi-Model Endpoints can reduce overhead by loading only the requested model. Option B (GPU) may not help if the bottleneck is CPU. Option C (Add more instances) helps throughput but not per-request latency. Option D (Batch Transform) is for offline inference.

Answer

Switch to a GPU instance type

Answer

Add more instances behind a load balancer

Answer

Use SageMaker Batch Transform instead of real-time endpoint

Question 11

A data scientist is using Amazon SageMaker to train a linear regression model. The dataset has outliers. Which TWO techniques can help reduce the impact of outliers? (Choose TWO.)

Accepted Answer

Trim the dataset to remove extreme values. Options A and D are correct. Huber loss is robust to outliers, and trimming the dataset removes extreme values. Option B (more features) is not relevant for handling outliers. Option C (L1 regularization) reduces overfitting but not outlier impact. Option E (standardization) does not handle outliers.

Answer

Add more features

Answer

Apply L1 regularization

Answer

Standardize the features

Question 12

A data scientist is tuning a random forest model using SageMaker Hyperparameter Tuning. The objective metric is validation:accuracy. Which THREE hyperparameters are most commonly tuned for random forest? (Choose THREE.)

Accepted Answer

Minimum samples per leaf (min_samples_leaf). Options B, C, and D are correct. Common tunable hyperparameters for random forest include number of trees (n_estimators), maximum depth (max_depth), and minimum samples per leaf (min_samples_leaf). Option A (learning rate) is for gradient boosting. Option E (batch size) is for neural networks, not random forest.

Answer

Learning rate

Answer

Batch size

Question 13

A data science team is using Amazon SageMaker to train a deep learning model for object detection using the built-in SSD algorithm. The dataset consists of 100,000 labeled images stored in a SageMaker Pipe Mode input. The training job uses a single ml.p3.2xlarge instance. After 2 hours, the training job fails with the error 'ResourceLimitExceeded: The account-level service limit for ml.p3.2xlarge for training job usage is 1. Contact AWS Support to request a limit increase'. However, the team has already submitted a limit increase request and it was approved for 5 instances. What is the most likely cause of the error?

Accepted Answer

The service limit increase has not yet been applied to the account in the current region. The error 'ResourceLimitExceeded' indicates that the account's service limit for ml.p3.2xlarge training instances has been exceeded. Even though the team requested and received approval for a limit increase to 5 instances, the increase may not have taken effect yet in the current region. AWS service limit increases are applied per region, and there can be a propagation delay after approval. Option A (GPU memory) would cause a different error such as 'OutOfMemory'. Option B (algorithm requirement) is unrelated because the SSD algorithm does run on the chosen instance. Option D (S3 permissions) would result in an 'AccessDenied' error, not a limit error. Therefore, option C is the correct answer.

Answer

The instance is running out of GPU memory

Answer

The built-in SSD algorithm requires a GPU instance type with at least 16 GB of GPU memory

Answer

The IAM role does not have permission to access the S3 bucket for model artifacts

Question 14

A financial services company is building a fraud detection model using Amazon SageMaker. The dataset has 10 million transactions, with 0.1% fraudulent. They train an XGBoost model with default hyperparameters. The model achieves 99.9% accuracy on the test set, but only catches 10% of actual fraud cases. The company wants to maximize the number of fraud cases caught while keeping the false positive rate below 5%. The data scientist has already tried adjusting the class weights and threshold, but the recall is still low. What should the data scientist do next?

Accepted Answer

Use a different algorithm such as a balanced random forest or SMOTE with XGBoost. The model's high accuracy but low recall indicates it's predicting the majority class (non-fraud) most of the time due to extreme class imbalance (0.1% fraud). Default XGBoost does not handle such imbalance well. Using a balanced random forest (which under-samples the majority class during training) or SMOTE (synthetic oversampling of the minority class) with XGBoost can improve recall by making the model more sensitive to fraud. Option A (collecting more data) may not help if the imbalance ratio stays the same. Option C (PCA) reduces dimensionality but does not address class imbalance. Option D (larger instance) only affects training speed, not model performance.

Answer

Collect more data, especially fraudulent transactions, to balance the dataset

Answer

Apply PCA to reduce the number of features and prevent overfitting

Answer

Use a larger instance type to train for more epochs

Question 15

A startup is deploying a machine learning model for real-time recommendation on Amazon SageMaker. The model is a TensorFlow model (1 GB) and the endpoint uses a single ml.c5.2xlarge instance. The inference latency is currently 500 ms per request. The startup expects traffic to increase 10x in the next month. They want to maintain latency under 500 ms. What is the most cost-effective solution?

Accepted Answer

Set up auto-scaling for the endpoint based on average latency or request count. Auto-scaling dynamically adds instances based on demand, handling a 10x increase in traffic while maintaining latency under 500 ms. This is more cost-effective than over-provisioning a larger instance (D) or switching to an expensive GPU instance (B). SageMaker Batch Transform (A) is not suitable for real-time inference.

Answer

Use SageMaker Batch Transform to process requests in batches

Answer

Switch to a GPU instance type for faster inference

Answer

Upgrade to a larger CPU instance type, such as ml.c5.4xlarge