MLS-C01 Modeling Practice Test 5 — 15 Questions

Question 1

A machine learning engineer is deploying a model using Amazon SageMaker and wants to automatically scale the endpoint based on the number of incoming requests. Which scaling policy should be used?

Accepted Answer

Target tracking scaling. Amazon SageMaker endpoints support Application Auto Scaling. A target tracking scaling policy (Option C) is the recommended approach when you want to automatically scale based on a metric like InvocationsPerInstance. It adjusts capacity to maintain the target value of the metric. Step scaling (Option A) requires defining step adjustments and thresholds. Simple scaling is no longer recommended by AWS. Scheduled scaling (Option B) is for predictable traffic patterns. Therefore, Option C is correct.

Answer

Step scaling

Answer

Scheduled scaling

Answer

Simple scaling

Question 2

A data scientist is training a neural network on a dataset with 1 million images. The training loss decreases steadily but the validation loss starts to increase after 10 epochs. Which action should the scientist take to improve generalization?

Accepted Answer

Implement early stopping. Increasing validation loss while training loss decreases indicates overfitting. Early stopping (Option A) halts training when validation loss stops improving, directly preventing overfitting. Option B (adding more layers) increases model capacity and typically worsens overfitting. Option C (reducing learning rate) might slow training but does not directly stop overfitting. Option D (increasing epochs) would continue training and likely worsen overfitting.

Answer

Add more layers to the network

Answer

Reduce the learning rate

Answer

Increase the number of epochs

Question 3

A company uses Amazon SageMaker to train a model using the built-in Linear Learner algorithm. The training data contains missing values in some features. What is the best practice for handling missing values with this algorithm?

Accepted Answer

Impute missing values using mean or median imputation. Linear Learner expects dense input; it cannot handle missing values. The best practice is to impute missing values before training, such as using mean or median imputation. Removing rows with missing values (Option A) may lose valuable data. Setting missing values to zero (Option C) could bias the model. The algorithm does not have a built-in `handle_missing` parameter (Option D). Therefore, Option B (Impute missing values using mean or median imputation) is correct.

Answer

Remove rows with missing values

Answer

Set missing values to zero

Answer

Use the `handle_missing` parameter in the algorithm

Question 4

Which TWO metrics are appropriate for evaluating a binary classification model when the cost of false negatives is high?

Accepted Answer

Recall. When false negatives are costly, we want to minimize them, so recall (true positive rate) is important. Precision is also important to avoid too many false positives, but F1 score balances both. Recall directly measures false negatives, and F1 combines precision and recall. AUC-ROC is a general measure, and accuracy can be misleading. Therefore, the two appropriate metrics are Recall (option C) and F1 score (option D).

Answer

Accuracy

Answer

AUC-ROC

Answer

Precision

Question 5

Which THREE techniques are effective for reducing overfitting in a deep neural network?

Accepted Answer

Early stopping. Dropout (C) randomly drops neurons during training, forcing the network to learn redundant representations and reducing overfitting. L2 regularization (E) adds a penalty on large weights, discouraging complex models. Early stopping (B) monitors validation loss and halts training before the model starts overfitting. Increasing model complexity (A) would worsen overfitting, and reducing training data (D) also increases overfitting risk due to less generalization. Thus, the correct techniques are B, C, and E.

Answer

Increasing model complexity

Answer

Reducing the amount of training data

Question 6

Which TWO actions are best practices for tuning hyperparameters using Amazon SageMaker Automatic Model Tuning?

Accepted Answer

Use Bayesian optimization strategy. Amazon SageMaker Automatic Model Tuning supports Bayesian optimization, random search, and grid search strategies. Bayesian optimization (Option C) is efficient for finding optimal hyperparameters by exploring promising regions. Random search (Option E) is effective for high-dimensional spaces and often outperforms grid search. Grid search (Option D) is not recommended for many hyperparameters due to combinatorial explosion. Setting a very large number of training jobs (Option A) is costly and unnecessary. Using the same hyperparameters as the baseline model (Option B) does not perform tuning. Therefore, the best practices are Options C and E.

Answer

Set the number of training jobs to a very large value

Answer

Use the same hyperparameters as the baseline model

Answer

Use grid search strategy

Question 7

Refer to the exhibit. A data scientist is assigned an IAM policy to deploy a SageMaker model. When the scientist tries to create an endpoint, the action fails with an authorization error. What is the missing permission?

Accepted Answer

iam:PassRole. The error occurs because the IAM policy does not include the `iam:PassRole` permission. When creating a SageMaker endpoint, the service must assume an IAM role to access resources (e.g., S3 buckets, CloudWatch). The `iam:PassRole` permission allows the user to pass that role to SageMaker. The other actions listed are either for inference (`InvokeEndpoint`), listing endpoints (`ListEndpoints`), or updating endpoints (`UpdateEndpoint`), which are not relevant to the creation process. Therefore, the missing permission is `iam:PassRole` (Option A).

Answer

sagemaker:ListEndpoints

Answer

sagemaker:InvokeEndpoint

Answer

sagemaker:UpdateEndpoint

Question 8

Refer to the exhibit. A data scientist runs the above AWS CLI command to create a SageMaker training job using the built-in Linear Learner algorithm. The training job fails with an error. What is the most likely cause?

Accepted Answer

The S3 data type is AugmentedManifestFile, but Linear Learner requires RecordIO or CSV. The command uses `S3DataType` as `AugmentedManifestFile`, but the Linear Learner algorithm only supports `RecordIO` or `CSV` as the S3 data type. AugmentedManifestFile is used for algorithms like object detection that require additional labels. The content type `application/x-recordio` is correct for RecordIO, but since the data type is set to AugmentedManifestFile, the training job fails. The IAM role, instance type, and MaxRuntimeInSeconds are all valid and would not cause this specific error. Therefore, the most likely cause is the incorrect S3 data type, which corresponds to option A.

Answer

The IAM role does not have sufficient permissions

Answer

The instance type ml.m5.large does not support the Linear Learner algorithm

Answer

The MaxRuntimeInSeconds is too short

Question 9

Refer to the exhibit. A data scientist is configuring SageMaker Model Monitor for data quality checks. The configuration above is used. What is the purpose of the `ProbabilityThresholdAttribute` set to "0.5"?

Accepted Answer

It defines the probability threshold used to convert model output to binary predictions for monitoring. In SageMaker Model Monitor, the `ProbabilityThresholdAttribute` parameter is used for binary classification models to define the probability threshold for converting model output probabilities (e.g., 0.7) to binary predictions (0 or 1). This threshold is used to monitor drift in the distribution of predictions over time, not to set the endpoint inference threshold. Option D correctly identifies this purpose. Option A is incorrect because it does not filter input data; it only defines the threshold for converting probabilities to labels for monitoring. Option B is incorrect as it does not specify a sampling threshold; sampling is configured separately. Option C is incorrect because it does not set the accuracy metric threshold; accuracy is a separate metric.

Answer

It filters the input data to only include predictions above the threshold

Answer

It specifies the threshold for sampling data for monitoring

Answer

It sets the threshold for the accuracy metric

Question 10

A data scientist is training a binary classifier using logistic regression on a dataset that is highly imbalanced (95% negative class, 5% positive class). The model achieves 95% accuracy but only predicts the negative class. Which metric should the scientist use to evaluate the model's performance on the positive class?

Accepted Answer

Recall. The model only predicts the negative class, so recall (true positives / (true positives + false negatives)) is 0, directly revealing the failure to detect positives. Precision is undefined (or 0/0) because there are no positive predictions. Accuracy is misleading (95% but useless). Therefore, recall is the most informative metric for evaluating performance on the positive class.

Answer

Precision

Answer

F1 Score

Answer

Accuracy

Question 11

A team is training a deep learning model on Amazon SageMaker using a large dataset stored in S3. The training job is taking a long time, and the team suspects that data loading is the bottleneck. The dataset consists of many small files (average size 10KB). Which change would most effectively reduce the I/O bottleneck?

Accepted Answer

Combine the small files into larger files (e.g., TFRecord format). Combining small files into larger ones (e.g., TFRecord or Parquet) reduces the number of S3 GET requests and improves throughput. Using Pipe mode reads data sequentially, which is less efficient for random access. Increasing instance count or using P3 instances addresses compute, not I/O. Amazon EFS is not recommended for training jobs due to higher latency.

Answer

Use SageMaker Pipe mode instead of File mode

Answer

Increase the number of training instances

Answer

Use a P3 instance type for better GPU performance

Question 12

A company is using Amazon SageMaker to deploy a machine learning model for real-time inference. The model was trained using XGBoost and achieves high accuracy. However, during deployment, the endpoint returns a 'ModelError' when receiving input data. The input is a CSV string. What is the most likely cause?

Accepted Answer

The input data format does not match the model's expected format (e.g., CSV vs JSON). The most common cause of ModelError during inference is that the input format does not match what the model expects. XGBoost models typically expect CSV without headers. The serializer setting in SageMaker must be configured correctly. If the model expects text/csv but the endpoint is configured as JSON, the error occurs. The other options are less likely: model weights are loaded correctly if the model deployed, and the instance type affects latency not errors.

Answer

The inference instance type is too small

Answer

The model is not properly loaded into memory

Answer

The model weights are corrupted during deployment

Question 13

A data scientist is using Amazon SageMaker to perform hyperparameter tuning for a neural network. The tuning job uses the 'Random' search strategy. After 10 training jobs, the best objective metric has plateaued. The scientist wants to improve the results without increasing the total number of training jobs. Which approach should they take?

Accepted Answer

Switch the hyperparameter tuning strategy to 'Bayesian'. Switching to Bayesian search (e.g., 'Bayesian' strategy) is more efficient because it uses past results to choose the next hyperparameters, potentially finding better values in fewer jobs. Increasing the number of jobs would increase cost. Random search might get lucky but is less efficient. Changing the objective metric or scaling features would not directly improve the tuning process.

Answer

Use a different objective metric that is easier to optimize

Answer

Normalize the input features to have zero mean and unit variance

Answer

Increase the maximum number of training jobs

Question 14

A company is building a recommendation system using collaborative filtering on Amazon SageMaker. The dataset contains user-item interactions with a long-tail distribution: a few items have millions of interactions, while most items have very few. The model currently uses matrix factorization with ALS. The recall@20 metric is low for niche items. Which modification would most likely improve recall for long-tail items?

Accepted Answer

Use implicit feedback with confidence weighting to downweight popular items. Implicit feedback models can incorporate confidence weights that downweight popular items, helping the model focus on less frequent items. Adding explicit features would not directly address the long-tail. Increasing the number of factors might help but could also overfit. Regularization is already present; adjusting it might not target the issue specifically.

Answer

Increase the regularization parameter to prevent overfitting

Answer

Add explicit features like item category and user demographics

Answer

Increase the number of latent factors in the matrix

Question 15

A data scientist is training a random forest model on a dataset with 50 features. After training, the model achieves 98% accuracy on the training set but only 85% on the test set. Which technique is most appropriate to reduce the generalization error?

Accepted Answer

Reduce the maximum depth of each tree. The gap indicates overfitting. Random forest can overfit if trees are too deep or if the number of trees is too high. Reducing the maximum depth of trees limits model complexity and helps generalization. Increasing the number of trees typically reduces overfitting but can also increase computational cost; however, reducing depth is more direct. Feature selection or PCA might help but are less direct than controlling tree complexity.

Answer

Apply Principal Component Analysis (PCA) to reduce dimensionality

Answer

Add more training data

Answer

Increase the number of trees in the forest