MLS-C01 Modeling Practice Test 34 — 15 Questions

Question 1

A data scientist is training a binary classification model on imbalanced data (95% negative, 5% positive). Which metric is most appropriate for evaluating model performance?

Accepted Answer

Area Under the ROC Curve (AUC-ROC). AUC-ROC is the most appropriate metric for imbalanced binary classification because it evaluates the model's ability to distinguish between positive and negative classes across all classification thresholds, without being biased by the 95% negative majority. It measures the trade-off between true positive rate and false positive rate, making it robust to class imbalance.

Answer

R-squared

Answer

Mean Squared Error (MSE)

Answer

Accuracy

Question 2

A data scientist is training a linear regression model and notices high bias in the training set. What action is most likely to reduce bias?

Accepted Answer

Add more relevant features to the model.. High bias indicates that the model is underfitting the training data, meaning it is too simple to capture the underlying patterns. Adding more relevant features increases the model's capacity to learn complex relationships, directly reducing bias. This is a standard approach in linear regression to address underfitting.

Answer

Apply L1 regularization.

Answer

Increase the learning rate.

Answer

Increase the amount of training data.

Question 3

A machine learning engineer is tuning a gradient boosting model using SageMaker Hyperparameter Tuning. The objective is to minimize MAE. The tuning job uses 20 training jobs. After 10 jobs, the best objective value is 5.2. Which action should the engineer take to potentially improve the result?

Accepted Answer

Increase the total number of training jobs to 50.. Increasing the total number of training jobs from 20 to 50 gives the Bayesian optimization algorithm more opportunities to explore the hyperparameter space and exploit promising regions. With only 10 jobs completed, the tuning job may not have converged to the global minimum of MAE, and additional jobs can refine the search, especially since Bayesian search builds a probabilistic model that improves with more observations.

Answer

Set early stopping to avoid overfitting.

Answer

Change the objective metric to RMSE.

Answer

Switch the tuning strategy from Bayesian to Random search.

Question 4

A company has a time series forecasting problem with daily sales data. The data shows both trend and seasonality. Which Amazon SageMaker built-in algorithm is most suitable?

Accepted Answer

DeepAR. DeepAR is a supervised learning algorithm for time series forecasting that explicitly models both trend and seasonality using recurrent neural networks (RNNs). It is designed to handle multiple related time series, incorporate additional features like holidays or promotions, and produce probabilistic forecasts, making it the most suitable choice for daily sales data with trend and seasonal patterns.

Answer

K-Means

Answer

Linear Learner

Answer

XGBoost

Question 5

A data scientist is using SageMaker to train a model. The training data is stored in an S3 bucket in a different AWS account. What is required to allow SageMaker to access the data?

Accepted Answer

Configure the SageMaker execution role with a policy that grants cross-account access to the S3 bucket.. SageMaker uses an IAM execution role to access resources. To allow cross-account access to an S3 bucket, the SageMaker execution role must have an IAM policy that grants s3:GetObject and s3:ListBucket permissions for the bucket, and the S3 bucket policy must also grant cross-account access to that role. This is the standard AWS mechanism for cross-account resource access.

Answer

Set up VPC peering between the two accounts.

Answer

Create a SageMaker notebook instance in the same account as the S3 bucket.

Answer

Launch the training job from a SageMaker notebook in the account containing the S3 bucket.

Question 6

A data scientist is training a neural network for image classification. The training loss decreases but validation loss increases after a few epochs. Which TWO actions should be taken to address this?

Accepted Answer

Implement early stopping based on validation loss.. The scenario describes overfitting: training loss keeps falling while validation loss rises after a few epochs, meaning the model is memorizing training data rather than generalizing. Option A is correct because early stopping monitors validation loss and halts training at the point where validation performance stops improving, preventing the model from continuing to overfit. Option C is correct because increasing the dropout rate regularizes the network by randomly deactivating neurons during training, which reduces co-adaptation and improves generalization, directly countering the overfitting trend. Option B is not appropriate because raising the learning rate would likely destabilize training or worsen convergence, not fix overfitting. Option D is wrong because training for more epochs would let the model overfit even further as validation loss continues to climb. Option E is wrong because adding more convolutional layers increases model capacity, which typically exacerbates overfitting rather than resolving it.

Answer

Increase the learning rate.

Answer

Increase the number of epochs.

Answer

Add more convolutional layers.

Question 7

A company uses SageMaker to train a model. The training job fails with 'ResourceLimitExceeded' error. Which TWO actions should the company take to resolve this?

Accepted Answer

Use a different instance type that is not at its limit.. The 'ResourceLimitExceeded' error indicates that the requested instance type has reached its concurrent usage limit in the current AWS region. Switching to a different instance type that is not at its limit allows the training job to proceed without exceeding the service quota. Option E is correct because requesting a service limit increase for SageMaker training job resources directly raises the cap on the number of concurrent instances or total instance count, resolving the underlying quota issue.

Answer

Launch the training job in a different AWS region.

Answer

Use SageMaker Managed Spot Training to reduce cost.

Answer

Compress the training data to reduce storage requirements.

Question 8

A data scientist is building a recommender system using Amazon SageMaker. The dataset contains user-item interactions with implicit feedback (clicks). Which THREE evaluation metrics are appropriate for this use case?

Accepted Answer

Precision@k. For implicit feedback (e.g., clicks) in recommender systems, evaluation metrics should focus on ranking quality without requiring explicit negative labels. Precision@k measures the proportion of relevant items among top-k recommendations, Recall@k measures the proportion of relevant items retrieved in top-k, and Mean Average Precision (MAP) averages precision across multiple recall levels, all suitable for top-k recommendation tasks. RMSE is for explicit rating prediction and requires ground truth ratings, not applicable to implicit feedback. AUC-ROC is for binary classification with both positive and negative samples, but implicit feedback lacks explicit negatives, making it unsuitable.

Answer

Root Mean Squared Error (RMSE)

Answer

Area Under the ROC Curve (AUC-ROC)

Question 9

A data scientist is trying to create a SageMaker endpoint using an IAM role with the attached policy. The operation fails with 'AccessDenied'. What is the MOST likely cause?

Accepted Answer

The policy does not allow sagemaker:CreateEndpoint.. The error 'AccessDenied' when creating a SageMaker endpoint is most likely due to missing sagemaker:CreateEndpoint permission. To create an endpoint, the IAM role must have the sagemaker:CreateEndpoint action. While InvokeEndpoint is needed for invoking the endpoint after creation, it is not required during creation. S3 bucket access may be needed for model artifacts, but that would typically cause a different error or occur earlier in the process (when creating the model). Therefore, option C is the most likely cause.

Answer

The policy does not allow sagemaker:InvokeEndpoint.

Answer

The policy does not allow access to the S3 bucket for model artifacts.

Answer

The policy does not allow sagemaker:CreateModel.

Question 10

A training job fails with the error shown. The training script expects a file named 'train.csv' in the 'training' channel. What is the most likely cause?

Accepted Answer

The 'train.csv' file is located inside a subfolder within 's3://my-bucket/data/', and the script expects it directly in the channel path.. The error indicates that the training script cannot find 'train.csv' in the expected location. When SageMaker copies data from an S3 channel path (e.g., 's3://my-bucket/data/') to the training instance, it places the contents of that S3 prefix directly into the channel directory (e.g., '/opt/ml/input/data/training/'). If the CSV file is inside a subfolder (e.g., 's3://my-bucket/data/subfolder/train.csv'), the script will not find it at the top level of the channel path, causing a 'file not found' error.

Answer

The S3 bucket policy denies access to the 'train.csv' file.

Answer

The channel name in the input data configuration does not match the script's expected channel name.

Answer

The training script has a bug that prevents it from reading the file.

Question 11

A company is building a binary classifier to detect fraudulent transactions. The dataset is highly imbalanced with only 0.1% positive cases. The data scientist uses logistic regression and obtains 99.9% accuracy on the test set. Which metric should the data scientist use to evaluate the model's performance?

Accepted Answer

Precision-recall curve. With only 0.1% positive cases, accuracy is misleading because a model that always predicts 'not fraudulent' achieves 99.9% accuracy. The precision-recall curve focuses on the positive class and is robust to extreme class imbalance, showing the trade-off between precision and recall across thresholds. This makes it the best choice for evaluating a binary classifier on highly imbalanced fraud detection data.

Answer

ROC AUC

Answer

Precision

Answer

F1 score

Question 12

A data scientist trains a gradient boosting model on a large dataset using SageMaker. The training completes successfully, but when deploying the model to a real-time endpoint, inference latency is too high. Which change is MOST likely to reduce latency without significant accuracy loss?

Accepted Answer

Prune the trees by removing nodes with low importance. Pruning trees by removing nodes with low importance reduces the model's complexity, which directly decreases inference latency because fewer decision paths need to be evaluated. In gradient boosting, this can be done with minimal accuracy loss if the removed nodes correspond to splits that contribute little to the overall prediction, as measured by feature importance or gain.

Answer

Use a larger instance type for the endpoint

Answer

Increase the number of trees in the ensemble

Answer

Use SageMaker Batch Transform instead of real-time

Question 13

A data scientist is training a linear regression model and notices that the model performs well on training data but poorly on validation data. Which technique should be applied to reduce overfitting?

Accepted Answer

Apply L2 regularization (Ridge). L2 regularization (Ridge) adds a penalty term proportional to the square of the magnitude of the coefficients to the loss function. This shrinks the weights toward zero, reducing the model's sensitivity to individual features and preventing it from fitting noise in the training data, which directly addresses overfitting.

Answer

Increase the number of epochs

Answer

Add more features

Answer

Remove training examples

Question 14

A data scientist is using SageMaker to train a deep learning model for image classification. The training job is taking too long. Which approach can reduce training time?

Accepted Answer

Use SageMaker's distributed data parallelism. SageMaker's distributed data parallelism splits the training data across multiple GPUs or instances, allowing each worker to process a different subset of the data simultaneously. This reduces the wall-clock time per epoch by parallelizing the computation, which directly addresses the 'taking too long' issue for deep learning image classification models.

Answer

Use SageMaker Neo to compile the model

Answer

Increase the number of epochs

Answer

Use a smaller image size

Question 15

A data scientist is building a model to predict customer churn. The dataset contains categorical features with high cardinality (e.g., ZIP code, customer ID). Which encoding method is MOST suitable?

Accepted Answer

Target encoding. Target encoding is most suitable for high-cardinality categorical features because it replaces each category with the mean of the target variable for that category, effectively capturing the predictive signal while keeping the feature space dense. This avoids the curse of dimensionality from one-hot encoding and the arbitrary ordinality of label encoding, which can mislead tree-based models.

Answer

One-hot encoding

Answer

Label encoding

Answer

Hashing encoding