MLS-C01 Modeling Practice Test 22 — 15 Questions

Question 1

A data scientist is working on a multi-class classification problem with 10 classes. The model outputs probabilities and the scientist wants to evaluate the model's ability to rank classes correctly. Which metric is most appropriate?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Area Under the ROC Curve (AUC-ROC). AUC-ROC is the most appropriate metric because it evaluates the model's ability to rank classes correctly by measuring the trade-off between true positive rate and false positive rate across all thresholds. For multi-class problems, AUC-ROC can be computed using one-vs-rest or one-vs-one strategies, and it directly assesses ranking quality independent of a specific decision threshold.

Answer

F1 score

Answer

Accuracy

Answer

Log loss

Question 2

A machine learning engineer is deploying a model using Amazon SageMaker and wants to automatically scale the endpoint based on the number of incoming requests. Which scaling policy should be used?

Accepted Answer

Target tracking scaling. Target tracking scaling is correct because it automatically adjusts the number of endpoint instances to keep a specified metric (like InvocationsPerInstance) at a target value, directly responding to incoming request load. SageMaker's target tracking policy uses a predefined or custom metric and a target value, then scales in or out to maintain that target, which is ideal for request-based scaling. This is the only policy among the options that continuously adjusts capacity based on real-time demand without manual threshold definitions.

Answer

Step scaling

Answer

Scheduled scaling

Answer

Simple scaling

Question 3

A data scientist is training a neural network on a dataset with 1 million images. The training loss decreases steadily but the validation loss starts to increase after 10 epochs. Which action should the scientist take to improve generalization?

Accepted Answer

Implement early stopping. Increasing validation loss while training loss decreases indicates overfitting. Early stopping (Option A) halts training when validation loss stops improving, directly preventing overfitting. Option B (adding more layers) increases model capacity and typically worsens overfitting. Option C (reducing learning rate) might slow training but does not directly stop overfitting. Option D (increasing epochs) would continue training and likely worsen overfitting.

Answer

Add more layers to the network

Answer

Reduce the learning rate

Answer

Increase the number of epochs

Question 4

Which THREE techniques are effective for reducing overfitting in a deep neural network?

Accepted Answer

Early stopping. Early stopping (B) is correct because it halts training when validation loss stops improving, preventing the network from continuing to fit noise in the training set. Dropout (C) is correct because randomly deactivating neurons during training forces the network to learn redundant, robust features instead of relying on specific co-adapted units. L2 regularization (E) is correct because it adds a penalty proportional to the squared magnitude of weights to the loss function, discouraging large weights and thus reducing variance. Increasing model complexity (A) is wrong because more parameters and capacity make overfitting worse, not better. Reducing the amount of training data (D) is wrong because less data gives the model fewer examples to generalize from, typically increasing overfitting.

Answer

Increasing model complexity

Answer

Reducing the amount of training data

Question 5

Refer to the exhibit. A data scientist is assigned an IAM policy to deploy a SageMaker model. When the scientist tries to create an endpoint, the action fails with an authorization error. What is the missing permission?

Accepted Answer

iam:PassRole. When creating a SageMaker endpoint, the service must assume an IAM role that has permissions to access required resources (e.g., S3, ECR). The user needs iam:PassRole permission to pass that role to SageMaker. Without it, the CreateEndpoint call fails with an authorization error. The other options are not required for endpoint creation.

Answer

sagemaker:ListEndpoints

Answer

sagemaker:InvokeEndpoint

Answer

sagemaker:UpdateEndpoint

Question 6

A data scientist is training a binary classifier using logistic regression on a dataset that is highly imbalanced (95% negative class, 5% positive class). The model achieves 95% accuracy but only predicts the negative class. Which metric should the scientist use to evaluate the model's performance on the positive class?

Accepted Answer

Recall. Recall (also called sensitivity or true positive rate) measures the proportion of actual positive cases that are correctly identified by the model. In a highly imbalanced dataset where the model predicts only the negative class, recall for the positive class would be zero, making it the most informative metric to evaluate performance on the positive class.

Answer

Precision

Answer

F1 Score

Answer

Accuracy

Question 7

A team is training a deep learning model on Amazon SageMaker using a large dataset stored in S3. The training job is taking a long time, and the team suspects that data loading is the bottleneck. The dataset consists of many small files (average size 10KB). Which change would most effectively reduce the I/O bottleneck?

Accepted Answer

Combine the small files into larger files (e.g., TFRecord format). Combining small files into larger ones (e.g., TFRecord or Parquet) reduces the number of S3 GET requests and improves throughput. Using Pipe mode reads data sequentially, which is less efficient for random access. Increasing instance count or using P3 instances addresses compute, not I/O. Amazon EFS is not recommended for training jobs due to higher latency.

Answer

Use SageMaker Pipe mode instead of File mode

Answer

Increase the number of training instances

Answer

Use a P3 instance type for better GPU performance

Question 8

A company is using Amazon SageMaker to deploy a machine learning model for real-time inference. The model was trained using XGBoost and achieves high accuracy. However, during deployment, the endpoint returns a 'ModelError' when receiving input data. The input is a CSV string. What is the most likely cause?

Accepted Answer

The input data format does not match the model's expected format (e.g., CSV vs JSON). A 'ModelError' on a SageMaker endpoint receiving a CSV string most commonly occurs when the input format does not match what the model's container expects. For example, an XGBoost model trained with a specific content type (e.g., text/csv or application/json) will fail if the InvokeEndpoint call sends a mismatched ContentType or a malformed CSV (wrong number of columns, headers included when not expected). The container's input_fn parses the payload and raises an error on mismatch.

Answer

The inference instance type is too small

Answer

The model is not properly loaded into memory

Answer

The model weights are corrupted during deployment

Question 9

A data scientist is using Amazon SageMaker to perform hyperparameter tuning for a neural network. The tuning job uses the 'Random' search strategy. After 10 training jobs, the best objective metric has plateaued. The scientist wants to improve the results without increasing the total number of training jobs. Which approach should they take?

Accepted Answer

Switch the hyperparameter tuning strategy to 'Bayesian'. Bayesian search uses a probabilistic model of the objective function to choose the next hyperparameter configuration, focusing trials in promising regions based on prior results. This is more sample-efficient than random search, so it can find better hyperparameters within the same number of training jobs.

Answer

Use a different objective metric that is easier to optimize

Answer

Normalize the input features to have zero mean and unit variance

Answer

Increase the maximum number of training jobs

Question 10

A data scientist is training a random forest model on a dataset with 50 features. After training, the model achieves 98% accuracy on the training set but only 85% on the test set. Which technique is most appropriate to reduce the generalization error?

Accepted Answer

Reduce the maximum depth of each tree. The model shows a large gap between training accuracy (98%) and test accuracy (85%), which is a classic sign of overfitting. Reducing the maximum depth of each tree limits the model's ability to memorize training data, thereby reducing variance and improving generalization. This directly addresses the overfitting problem without discarding useful features or requiring more data.

Answer

Apply Principal Component Analysis (PCA) to reduce dimensionality

Answer

Add more training data

Answer

Increase the number of trees in the forest

Question 11

A company uses Amazon SageMaker to host a model for real-time predictions. The model endpoint is experiencing high latency during peak hours. The data scientist wants to reduce latency without increasing cost. Which action should they take?

Accepted Answer

Enable auto-scaling for the endpoint based on latency metrics. Auto-scaling adds endpoint instances dynamically when latency (or invocations) crosses a threshold, spreading the load across more instances and reducing per-request latency without permanently provisioning larger or more expensive capacity. Because scaling is triggered only during peak hours, cost stays proportional to actual demand. This directly addresses the 'reduce latency without increasing cost' constraint.

Answer

Enable data capture for the endpoint to log requests

Answer

Switch to a larger instance type

Answer

Reduce the number of instances behind the endpoint

Question 12

A data scientist is training a linear regression model on a dataset with 10 numerical features. After training, the model's R-squared value is 0.99 on the training set but only 0.60 on the test set. Which TWO of the following are appropriate actions to reduce overfitting? (Choose TWO.)

Accepted Answer

Use a subset of the most important features. Option C is correct because using a subset of the most important features (feature selection) reduces model complexity and variance, which directly combats overfitting when the model has learned noise from irrelevant or redundant features. Option E is correct because L2 regularization (Ridge regression) adds a penalty term equal to lambda times the sum of squared coefficients to the loss function, shrinking coefficients and constraining model complexity to improve generalization. Option A is not correct because normalizing features only rescales inputs to comparable ranges; it aids convergence and fair regularization but does not by itself reduce overfitting. Option B is not correct because adding more features increases model complexity and typically worsens overfitting. Option D is not correct because increasing training epochs lets the model fit the training data even more closely, worsening the train-test gap.

Answer

Normalize the features

Answer

Add more features to the model

Answer

Increase the number of training epochs

Question 13

A data scientist is training a k-means clustering model on a dataset with 1,000 points. The scientist uses the elbow method to choose the number of clusters. The elbow plot shows a clear bend at k=4. After running k-means with k=4, the scientist wants to evaluate the quality of the clustering. Which THREE of the following are suitable internal clustering validation metrics? (Choose THREE.)

Accepted Answer

Calinski-Harabasz index. The Calinski-Harabasz index (option C) is a suitable internal metric because it evaluates cluster quality using only the data and the resulting labels, computing the ratio of between-cluster dispersion to within-cluster dispersion, so it works without ground-truth labels. The silhouette score (option D) is also internal: it measures how similar each point is to its own cluster versus the nearest other cluster using distances, requiring no external labels. The Davies-Bouldin index (option E) is internal as well, averaging each cluster's worst-case similarity to other clusters based on scatter and separation, with lower values indicating better clustering. Options A and B are not suitable here because the Adjusted Rand index and Rand index are external validation metrics that compare the clustering against a known ground-truth labeling, which the scientist does not have in this unsupervised scenario.

Answer

Adjusted Rand index

Answer

Rand index

Question 14

A data scientist is training a binary classification model on imbalanced data (95% negative, 5% positive). The model achieves 99% accuracy on the test set but fails to detect any positive cases. Which metric should the scientist focus on to evaluate model performance?

Accepted Answer

Recall. Recall measures the proportion of actual positives that were correctly identified (TP / (TP + FN)). With 95% negatives, a model that predicts 'negative' for every sample achieves 99% accuracy while catching zero positives — recall exposes this failure directly by returning 0.0. Accuracy is misleading on imbalanced data because the majority class dominates the metric, so recall (often paired with precision/F1 or AUC-PR) is the right lens.

Answer

Accuracy

Answer

RMSE

Answer

Precision

Question 15

A machine learning engineer is deploying a model for real-time inference using Amazon SageMaker. The model is a large ensemble that requires 8 GB of memory and 4 vCPUs. The expected traffic is 100 requests per second with a 200 ms latency requirement. Which instance configuration should they choose?

Accepted Answer

ml.c5.2xlarge (8 vCPU, 16 GB). (ml.c5.2xlarge) is correct because it offers 8 vCPUs and 16 GB memory, meeting both the compute and memory requirements while being cost-effective for CPU-based inference. Option A (ml.t2.medium) is wrong because it has only 2 vCPUs and 4 GB, insufficient for the 8 GB memory need. Option C (ml.p3.2xlarge) is wrong because it includes a GPU, which is unnecessary and overkill for this CPU-bound workload, and costs more. Option D (ml.m5.large) is wrong because it has only 2 vCPUs and 8 GB memory, lacking the required vCPUs and barely meeting memory.

Answer

ml.t2.medium (2 vCPU, 4 GB)

Answer

ml.p3.2xlarge (8 vCPU, 61 GB GPU)

Answer

ml.m5.large (2 vCPU, 8 GB)