MLS-C01 Modeling Practice Test 50 — 15 Questions

Question 1

A financial institution is training a binary fraud detection model on a dataset with 0.2% positive cases. After training an XGBoost model in Amazon SageMaker, the data scientist observes that the model predicts 'not fraud' for every validation sample, achieving 99.8% accuracy. The team wants the model to identify fraud cases while keeping false positives manageable. Which approach should the data scientist take?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use the SageMaker XGBoost built-in algorithm with scale_pos_weight set to the ratio of negative to positive examples.. The core issue is severe class imbalance: with 0.2% positives, a model can achieve high accuracy by always predicting the majority class. The SageMaker XGBoost algorithm supports the scale_pos_weight hyperparameter, which multiplies the gradient of positive examples, effectively rebalancing the loss. This makes the model pay attention to fraud cases, improving recall, and the team can then adjust the classification threshold to control false positives.

Answer

Change the objective to reg:squarederror so the model can output continuous fraud probability scores.

Answer

Increase the number of boosting rounds and lower the learning rate to allow the model to learn minority patterns.

Answer

Apply L2 regularization with a high lambda value to prevent overfitting to the majority class.

Question 2

A data scientist is using Amazon SageMaker to train a k-means clustering model on a dataset with 1 million records and 50 features. The goal is to segment customers into 10 groups. After training, the data scientist notices that the clusters are unbalanced, with one cluster containing 90% of the data points. Which action should the data scientist take to improve cluster quality?

Accepted Answer

Apply feature scaling to the input data before training.. K-means clustering relies on distance calculations, which are dominated by features with larger scales if data is not normalized. This can cause clusters to be unbalanced. Applying feature scaling (e.g., standardization) ensures each feature contributes equally, leading to more meaningful clusters. Other actions like increasing clusters or epochs do not address the root cause, and SageMaker's k-means does not allow changing the distance metric.

Answer

Increase the number of epochs to allow better convergence.

Answer

Increase the number of clusters to 20.

Answer

Change the distance metric to Manhattan distance.

Question 3

A financial institution is building a fraud detection model using Amazon SageMaker. The dataset is highly imbalanced, with only 0.5% of transactions labeled as fraudulent. The data scientist trains a binary classification model using the XGBoost algorithm and notices that the model predicts almost all transactions as non-fraudulent, achieving 99.5% accuracy but failing to detect any fraud. Which evaluation metric should the data scientist use to better assess the model's performance in this scenario?

Accepted Answer

Precision-Recall Area Under the Curve (AUC-PR). For highly imbalanced classification problems, precision-recall AUC is the preferred metric because it evaluates the model's performance on the positive (minority) class. It combines precision and recall, showing how well the model identifies fraud without being overwhelmed by the large number of true negatives. This makes it far more informative than accuracy or ROC AUC in this context.

Answer

Accuracy

Answer

Mean Squared Error (MSE)

Answer

Area Under the ROC Curve (AUC-ROC)

Question 4

A fraud detection team trains an XGBoost binary classifier in Amazon SageMaker using the built-in algorithm. The label distribution is 99.5% legitimate and 0.5% fraudulent. The team wants to maximize the model's ability to rank fraudulent transactions highly without manually resampling the data. Which SageMaker built-in XGBoost hyperparameter should they tune to achieve this?

Accepted Answer

scale_pos_weight. Class imbalance is addressed at the loss level by weighting the positive class, and the SageMaker built-in XGBoost algorithm provides scale_pos_weight for exactly that purpose. Setting it near the negative-to-positive ratio makes errors on fraudulent transactions costlier during boosting, improving the model's ability to rank rare events highly without altering the dataset.

Answer

num_round

Answer

subsample

Answer

max_depth

Question 5

A data scientist is using Amazon SageMaker to train a model on a dataset with a highly imbalanced binary target (99% negative, 1% positive). The data scientist wants to improve the model's ability to detect the positive class without collecting more data. Which two of the following approaches should be used? (Choose two.)

Accepted Answer

Use the 'scale_pos_weight' hyperparameter in the XGBoost algorithm.. For imbalanced binary classification, using 'scale_pos_weight' in XGBoost adjusts the loss to focus on the minority class, and applying SMOTE generates synthetic minority samples to balance the training data. Both methods directly address the imbalance without discarding data. Increasing epochs or using accuracy does not help, and undersampling discards valuable negative examples.

Answer

Increase the number of epochs in the training job.

Answer

Use accuracy as the evaluation metric for model selection.

Answer

Remove some negative class samples to balance the dataset.

Question 6

A machine learning engineer is using Amazon SageMaker to train a model with the built-in Linear Learner algorithm. The engineer wants to perform hyperparameter tuning to improve model performance. Which two hyperparameters are most appropriate to tune for the Linear Learner algorithm? (Choose two.)

Accepted Answer

`wd` (weight decay). The correct answers are `learning_rate` and `wd` (weight decay). For Linear Learner, learning_rate controls the optimization step size, which is fundamental to how well the model converges. wd is the L2 regularization parameter that controls overfitting by penalizing large weights. Both directly influence the model's learned parameters and generalization performance. Other hyperparameters like num_epochs, mini_batch_size, and target_recall are either less critical for performance tuning or serve post-training adjustments.

Answer

`mini_batch_size`

Answer

`num_epochs`

Answer

`target_recall`

Question 7

A machine learning engineer is training a linear regression model with Amazon SageMaker's built-in Linear Learner algorithm on a dataset where the target variable has a skewed distribution and contains a few extreme outliers. The engineer wants the model to be robust to these outliers without manually transforming the target. Which setting should the engineer configure?

Accepted Answer

Set predictor_type to 'regressor' and use the 'huber' loss function.. The huber loss is a robust regression loss that behaves like squared error for small errors and like absolute error for large errors, limiting the influence of outliers. Configuring the built-in Linear Learner with predictor_type='regressor' and loss='huber' directly addresses the skewed target and extreme values without manual transformation, aligning with the requirement.

Answer

Set predictor_type to 'regressor' and use the 'logistic' loss function.

Answer

Set predictor_type to 'regressor' and use the 'squared' loss function with L1 regularization.

Answer

Set predictor_type to 'regressor' and enable 'normalize_label' to True.

Question 8

A machine learning team is using Amazon SageMaker to train a custom model with a deep neural network. They notice that the training loss decreases steadily, but the validation loss starts to increase after a few epochs. The team wants to improve the model's generalization performance without changing the model architecture. Which technique should they apply?

Accepted Answer

Implement early stopping based on validation loss.. The scenario describes classic overfitting: training loss decreases while validation loss increases. Early stopping is a regularization technique that halts training at the point of best validation performance, preventing the model from memorizing noise in the training data. It requires no architectural changes and is straightforward to implement, making it the ideal choice.

Answer

Decrease the batch size.

Answer

Increase the learning rate.

Answer

Add dropout layers to the neural network.

Question 9

A computer vision team trains an image classifier with SageMaker and finds the model fits the training set almost perfectly but validation accuracy is 20 points lower. They want to reduce this generalization gap using only hyperparameters and regularization, without collecting more data. (Choose two.)

Accepted Answer

Add dropout layers with a rate between 0.2 and 0.5 in the network architecture.. A large train-validation gap signals overfitting, and the standard remedies that require no new data are capacity constraints applied during training. Dropout injects noise into activations, and early stopping limits how long the optimizer can chase training-set idiosyncrasies. Both are hyperparameter-level interventions that reduce variance and are directly tunable in SageMaker training jobs.

Answer

Increase the learning rate substantially so the model escapes sharp minima associated with overfitting.

Answer

Raise the batch size to the full training set so each gradient step is exact.

Answer

Reduce the input image resolution by half before feeding the network.

Question 10

A machine learning engineer is tuning a SageMaker training job and wants to reduce the risk of overfitting a gradient boosted tree model on a small tabular dataset. The engineer is considering hyperparameter changes in the SageMaker built-in XGBoost algorithm. (Choose two.)

Accepted Answer

Increase the value of lambda (L2 regularization).. Overfitting is controlled by constraining model capacity. In the SageMaker built-in XGBoost algorithm, increasing lambda penalizes large leaf weights through L2 regularization, and decreasing max_depth limits how finely each tree can partition the data. Both changes reduce variance and improve generalization on a small tabular dataset, unlike raising learning rate, boosting rounds, or disabling row subsampling.

Answer

Increase the value of subsample to 1.0.

Answer

Increase the value of num_round.

Answer

Increase the value of eta (learning rate).

Question 11

A data scientist is using Amazon SageMaker to train a model on a dataset that contains categorical features with high cardinality (e.g., user IDs). The data scientist wants to use the built-in Factorization Machines algorithm. Which data format should the training data be in to handle these categorical features effectively?

Accepted Answer

RecordIO-Protobuf format with sparse encoding.. The correct answer is RecordIO-Protobuf format with sparse encoding. Factorization Machines are designed to handle sparse data, especially high-cardinality categorical features, by learning latent vectors. The RecordIO-Protobuf format efficiently stores sparse data by only recording non-zero entries, reducing memory and compute. CSV with one-hot encoding would be dense and inefficient, while JSON Lines and Parquet are not natively supported by the algorithm. Using the correct format ensures the model can leverage the sparsity for effective training.

Answer

CSV format with one-hot encoding for all categorical features.

Answer

JSON Lines format with nested categorical fields.

Answer

Parquet format with dictionary encoding for categorical columns.

Question 12

A machine learning engineer is using Amazon SageMaker Automatic Model Tuning to optimize a gradient boosting model. The tuning job has completed, and the engineer wants to identify the best hyperparameter configuration. Which of the following should the engineer do to find the best configuration?

Accepted Answer

Examine the DescribeHyperParameterTuningJob response and look for the BestTrainingJob field.. After a hyperparameter tuning job completes, the best training job is automatically identified based on the objective metric. The DescribeHyperParameterTuningJob API returns this information in the BestTrainingJob field, which includes the hyperparameters and the metric value. This is the most reliable and direct way to obtain the best configuration.

Answer

Query the SageMaker Experiments API to retrieve the trial component with the highest metric value.

Answer

Sort the training jobs by the objective metric in Amazon CloudWatch Logs.

Answer

Use the SageMaker console to view the tuning job and manually compare the hyperparameters of each training job.

Question 13

A financial services company is training a fraud detection model using Amazon SageMaker. The dataset contains 10 million transactions, with only 0.2% labeled as fraudulent. The team wants to optimize the model's ability to identify fraud while minimizing false positives. Which SageMaker built-in algorithm and evaluation metric should they use to achieve this goal?

Accepted Answer

Use the XGBoost algorithm with the area under the precision-recall curve (AUPRC) as the evaluation metric.. For highly imbalanced fraud detection, XGBoost combined with AUPRC is optimal. XGBoost can model complex non-linear relationships and supports class weighting to handle imbalance. AUPRC focuses on the precision-recall trade-off for the positive class, providing a more informative evaluation than ROC AUC or accuracy when the positive class is rare. This directly aligns with the need to catch fraud while controlling false positives.

Answer

Use the Linear Learner algorithm with the F1 score as the evaluation metric.

Answer

Use the K-Nearest Neighbors algorithm with accuracy as the evaluation metric.

Answer

Use the XGBoost algorithm with the area under the ROC curve (AUC) as the evaluation metric.

Question 14

A data scientist runs a SageMaker automatic model tuning job for a deep neural network. The tuning job launches 40 training trials in parallel with Bayesian strategy, but after 10 trials the best objective has barely improved and several trials report out-of-memory on GPU. The search space includes a learning rate range of 1e-6 to 0.1 on a log scale and batch sizes from 8 to 4096. Which action is MOST appropriate?

Accepted Answer

Narrow the batch size range to values that fit the GPU memory and reduce the learning-rate range to a tighter log interval.. Two defects are visible: batch sizes that exceed GPU memory cause launch failures, and a learning-rate range spanning five orders of magnitude means most trials are uninformative. Constraining the batch-size range to fit the hardware and narrowing the learning-rate interval around values known to train stably focuses the Bayesian search where improvements are possible.

Answer

Switch the strategy to Random so the search space is explored more broadly than Bayesian search allows.

Answer

Increase max_parallel_training_jobs to 80 so more trials complete before the tuning budget is exhausted.

Answer

Set early_stopping to true and keep the search space unchanged so failing trials terminate quickly.

Question 15

A data scientist is using Amazon SageMaker to train a custom TensorFlow model with distributed training across multiple GPU instances. The training job uses the SageMaker distributed data parallel library. They observe that the training throughput does not scale linearly with the number of instances; adding more instances yields diminishing returns. Which factor is most likely causing this scalability issue?

Accepted Answer

The model has a large number of parameters, causing high communication overhead during gradient synchronization.. In distributed data parallel training, gradient synchronization across workers introduces communication overhead. For models with many parameters, the all-reduce operation can dominate training time, leading to sublinear scaling. The SageMaker distributed data parallel library mitigates this with communication overlap and optimized algorithms, but large models still face limits. Other factors like data loading are less likely to cause diminishing returns.

Answer

The training dataset is stored in Amazon S3 and not in Amazon FSx for Lustre.

Answer

The learning rate is too high, causing the model to diverge.

Answer

The instances are of different types, causing load imbalance.