MLS-C01 Modeling Practice Test 19 — 15 Questions

Question 1

A machine learning engineer is using SageMaker's built-in XGBoost algorithm for a multi-class classification problem. The training job completes but the model accuracy is low. Which THREE hyperparameters should the engineer tune to improve performance?

Accepted Answer

eta (learning rate). XGBoost hyperparameters: 'num_round' (number of boosting rounds), 'eta' (learning rate), and 'max_depth' (tree depth) are key for improving accuracy. 'subsample' can help but is less direct. 'min_child_weight' also important but these three are most common. 'colsample_bytree' is for feature subsampling.

Answer

subsample

Answer

colsample_bytree

Question 2

A data scientist is evaluating a binary classification model. The model's AUC-ROC is 0.95. Which TWO statements are true?

Accepted Answer

The model has excellent discriminative ability. AUC-ROC measures the model's ability to distinguish between classes across all thresholds. A high AUC (close to 1) indicates good performance. AUC-ROC is threshold-independent. It does not directly indicate accuracy or calibration.

Answer

The model has no false positives

Answer

The model is well-calibrated

Answer

The model's accuracy is at least 95%

Question 3

A company uses Amazon SageMaker to train a deep learning model for image classification. The training dataset consists of 500,000 images, each 256x256 pixels, stored in S3. The team uses a single ml.p3.2xlarge instance for training. The training time is unacceptably long (over 48 hours). The team wants to reduce training time without sacrificing model accuracy. They have already optimized the data pipeline by using SageMaker Pipe mode and sharding the S3 dataset. The model is a ResNet-50 implemented in TensorFlow. The team is considering the following options: A) Switch to a ml.p3.16xlarge instance which has 8 GPUs and more memory. B) Implement distributed data parallelism using Horovod across multiple instances. C) Use SageMaker's built-in Hyperparameter Tuning to find optimal hyperparameters. D) Reduce the image resolution to 128x128 to speed up training. Which option will MOST effectively reduce training time while maintaining accuracy?

Accepted Answer

Implement distributed data parallelism using Horovod across multiple instances. Using multiple instances with Horovod for distributed data parallelism can scale training linearly with the number of GPUs, significantly reducing time. A larger single instance (ml.p3.16xlarge) provides 8 GPUs but still limited by single instance. Hyperparameter tuning does not directly reduce training time. Reducing resolution may lose accuracy.

Answer

Switch to a ml.p3.16xlarge instance

Answer

Reduce the image resolution to 128x128

Answer

Use SageMaker's built-in Hyperparameter Tuning

Question 4

A data scientist is working on a regression problem to predict house prices. The dataset has 80 features, including categorical variables with high cardinality (e.g., zip code with 10,000 unique values). The target variable is log-transformed. The data scientist trains a linear regression model and obtains an R² of 0.45 on the test set. To improve performance, the data scientist considers: A) Applying one-hot encoding to all categorical features and using Ridge regression. B) Using target encoding for high-cardinality features and using a tree-based model like XGBoost. C) Removing all categorical features and using polynomial features for numerical features. D) Using principal component analysis (PCA) on all features before training a linear model. Which approach is MOST likely to improve the model's performance?

Accepted Answer

Target encoding + XGBoost. Target encoding efficiently handles high-cardinality features, and tree-based models like XGBoost can capture non-linear relationships and interactions, likely improving R². One-hot encoding would create too many features, causing sparsity. Removing categories loses information. PCA may discard important information.

Answer

Remove categorical features and use polynomial features

Answer

One-hot encoding + Ridge regression

Answer

PCA on all features before linear regression

Question 5

A company has deployed a real-time inference endpoint using SageMaker for a fraud detection model. The model uses a Random Forest classifier. The endpoint receives predictions but the latency is too high. The metric shows p99 latency of 500ms, but the requirement is under 200ms. The team has already optimized the instance type to the maximum allowed by their budget. The data scientist suggests: A) Reducing the number of trees in the Random Forest model. B) Switching to a linear model like Logistic Regression. C) Enabling SageMaker's batch transform instead of real-time endpoint. D) Adding more instances to the endpoint behind a load balancer. Which option will MOST effectively reduce latency while maintaining acceptable accuracy?

Accepted Answer

Reduce the number of trees in the Random Forest model. Reducing the number of trees directly reduces inference time of Random Forest, but may degrade accuracy. However, it's the most direct method for latency reduction while keeping the model type. Switching to Logistic Regression may reduce latency but likely reduces accuracy significantly. Batch transform is not suitable for real-time. Adding instances helps throughput but not per-request latency.

Answer

Switch to a linear model like Logistic Regression

Answer

Enable SageMaker's batch transform

Answer

Add more instances to the endpoint

Question 6

A company is building a binary classifier to predict customer churn. The dataset has 10,000 samples with 500 churners (5% positive class). After training a logistic regression model, the precision is 0.8 and recall is 0.2. Which metric should the data scientist focus on to improve the model's ability to identify churners while minimizing false positives?

Accepted Answer

Increase recall. Option A is correct because recall is very low (0.2), meaning the model misses most churners. Improving recall will capture more churners. Option B (precision) is already high but recall is low. Option C (accuracy) is misleading due to class imbalance. Option D (F1) balances precision and recall, but the primary concern is low recall.

Answer

Increase accuracy

Answer

Increase precision

Answer

Increase F1 score

Question 7

A data scientist is training a deep learning model on Amazon SageMaker using a custom Docker container. The training job fails with an error 'OutOfMemoryError: CUDA out of memory'. The instance type is ml.p3.2xlarge (8 GB GPU memory). The model has 50 million parameters. What is the most likely cause and solution?

Accepted Answer

The model uses FP32 precision; enable mixed precision training. Option B is correct because 50M parameters likely exceed GPU memory when using full precision. Mixed precision (FP16) reduces memory usage. Option A (batch size) could help but is secondary. Option C (instance type) may be unnecessary if mixed precision works. Option D (checkpointing) doesn't address memory during training.

Answer

The instance type is insufficient; switch to ml.p3.8xlarge

Answer

The batch size is too large; reduce batch size

Answer

Enable gradient checkpointing to reduce memory

Question 8

A team deployed a SageMaker endpoint for real-time inference using a PyTorch model. After monitoring, they notice that the latency is highly variable, with p99 latency 10x the p50 latency. The endpoint uses a single ml.c5.2xlarge instance with auto-scaling based on average CPU utilization. Which change is most likely to reduce latency variability?

Accepted Answer

Change the auto-scaling metric to 'InvocationsPerInstance'. Option C is correct because high p99 latency often results from cold starts or queueing when traffic spikes. Scaling on invocations per instance ensures more instances are ready. Option A (GPU) may not help if model is CPU-bound. Option B (batch size) can increase latency. Option D (warm-up) helps cold starts but not queueing.

Answer

Increase the batch size for inference

Answer

Pre-warm the model by sending dummy requests every minute

Answer

Switch to a GPU instance type

Question 9

A data scientist is using Amazon SageMaker to train a model with the built-in XGBoost algorithm. The dataset contains missing values. What is the default behavior of SageMaker XGBoost regarding missing values?

Accepted Answer

It automatically learns the best direction (left or right) for missing values during training. Option A is correct because XGBoost treats missing values as a separate category and learns the best direction to handle them (by default). Option B (mean imputation) is not default; XGBoost handles missingness internally. Option C (removing rows) is not default. Option D (fail) is not default.

Answer

It raises an error and stops training

Answer

It imputes missing values with the column mean

Answer

It removes rows with missing values

Question 10

A company is fine-tuning a BERT model on Amazon SageMaker for a text classification task. The training script uses PyTorch and Hugging Face Transformers. The training job completes successfully, but the final model accuracy is low. The dataset has 10,000 labeled samples. What is the most likely cause and solution?

Accepted Answer

The model is overfitting due to small dataset; use a pre-trained checkpoint and fine-tune only top layers. Option C is correct because BERT is large and 10,000 samples may not be enough; using a pre-trained checkpoint and doing transfer learning is standard. Option A (learning rate) is possible but not most likely. Option B (SageMaker error) is unlikely. Option D (instance type) doesn't affect accuracy directly.

Answer

The instance type is insufficient; use a larger instance

Answer

The learning rate is too high; reduce it

Answer

The training script has a bug in the data loader

Question 11

A data scientist is using Amazon SageMaker Autopilot to automatically build a model for a regression problem. The dataset has 100 features and 50,000 rows. Autopilot recommends a model with an R² of 0.85 on the validation set. However, when deployed to production, the model performs poorly (R² of 0.2). What is the most likely cause?

Accepted Answer

The production data distribution has shifted from the training data distribution. Option B is correct because a large discrepancy between validation and production performance often indicates data drift. Option A (overfitting) is possible but less likely given validation performance. Option C (feature importance) is not the direct cause. Option D (Autopilot bug) is rare.

Answer

The model is overfitting to the training data

Answer

The model is underfitting due to insufficient training

Answer

Autopilot selected the wrong features

Question 12

A machine learning team is using SageMaker to train a model with the built-in Linear Learner algorithm. The dataset has 1 million rows and 20 features. The training completes, but the model's mean squared error (MSE) is high. Which parameter adjustment is most likely to reduce MSE?

Accepted Answer

Increase the number of epochs. Option D is correct because increasing the number of epochs allows the model to converge better. Option A (learning rate increase) may cause instability. Option B (batch size increase) can slow convergence. Option C (loss function change) is not straightforward.

Answer

Increase the mini-batch size

Answer

Change the loss function to cross-entropy

Answer

Increase the learning rate

Question 13

A data scientist is training a recurrent neural network (RNN) for time series forecasting. The training loss decreases steadily for the first 10 epochs, then plateaus. The validation loss starts increasing after epoch 10. What is the most appropriate action?

Accepted Answer

Stop training early and use the model from epoch 10. Option B is correct because validation loss increasing while training loss decreases indicates overfitting. Early stopping halts training before overfitting worsens. Option A (more epochs) would increase overfitting. Option C (batch size) is not directly addressing overfitting. Option D (more layers) may exacerbate overfitting.

Answer

Continue training for more epochs

Answer

Add more layers to the network

Answer

Increase the batch size

Question 14

A company is using Amazon SageMaker to host a model for real-time inference. The model is a large ensemble of 10 XGBoost models, each 2 GB. The endpoint uses a single ml.c5.18xlarge instance. The inference latency is high (average 2 seconds). Which change would most effectively reduce latency?

Accepted Answer

Use SageMaker Multi-Model Endpoints to serve each model independently. Option A is correct because serialization/deserialization of large models is a bottleneck; SageMaker Multi-Model Endpoints can reduce overhead by loading only the requested model. Option B (GPU) may not help if the bottleneck is CPU. Option C (batch transform) is for offline inference. Option D (more instances) helps throughput but not per-request latency.

Answer

Switch to a GPU instance type

Answer

Add more instances behind a load balancer

Answer

Use SageMaker Batch Transform instead of real-time endpoint

Question 15

A data scientist has this IAM policy attached to their role. When trying to create a SageMaker endpoint using the AWS CLI, they get an 'AccessDenied' error. What is the most likely reason?

Accepted Answer

The policy does not include 'sagemaker:InvokeEndpoint' action. Option B is correct because the policy only allows creating the endpoint (CreateEndpoint), but not invoking it (InvokeEndpoint). The error may be due to missing permissions for other actions like sagemaker:DescribeEndpoint. Option A (S3) is not shown. Option C (KMS) not shown. Option D (missing resources) is possible but policy uses *.

Answer

The role does not have permission to use KMS to decrypt the model artifacts

Answer

The role does not have permission to access the S3 bucket containing the model artifacts

Answer

The policy uses a wildcard '*' for resources instead of specific ARNs