MLS-C01 Modeling — Timed 25 Questions

Question 1

A data scientist is training a binary classification model using Amazon SageMaker. The dataset is highly imbalanced (99% negative class, 1% positive class). The model currently achieves 99% accuracy but fails to detect most positive cases. Which metric should the data scientist primarily use to evaluate model performance?

25 minute time limit — choose an answer to begin.

Accepted Answer

F1 score. In highly imbalanced datasets (99% negative, 1% positive), accuracy is misleading because a model can achieve 99% accuracy by simply predicting the majority class for all instances, failing to detect any positive cases. The F1 score (option B) is the harmonic mean of precision and recall, providing a balanced measure that penalizes models that trade off recall for precision or vice versa. This makes it the primary metric for evaluating binary classification performance on imbalanced data, as it directly reflects the model's ability to correctly identify positive cases while minimizing false positives.

Answer

ROC AUC

Answer

Recall

Answer

Accuracy

Question 2

A machine learning engineer is training a neural network on Amazon SageMaker using a custom Docker container. The training job fails with an error: 'CUDA out of memory.' The training instance is an ml.p3.2xlarge with 16 GB GPU memory. The model and data fit into memory when using batch size 32, but the engineer wants to maximize GPU utilization. Which approach should the engineer use to fix the out-of-memory error while maintaining efficient training?

Accepted Answer

Implement gradient accumulation with a larger effective batch size. Gradient accumulation allows the engineer to simulate a larger effective batch size by accumulating gradients over multiple forward/backward passes before performing an optimizer step. This keeps the per-step memory footprint low (avoiding CUDA out-of-memory) while maintaining training dynamics similar to a larger batch, thus maximizing GPU utilization without crashing.

Answer

Enable mixed precision training

Answer

Reduce batch size to 1

Answer

Use a CPU-only instance

Question 3

A company is using Amazon SageMaker to tune hyperparameters for a gradient boosting model. The objective is to minimize root mean squared error (RMSE). The data scientist wants to explore the hyperparameter space efficiently. Which THREE hyperparameter tuning strategies should the data scientist consider? (Choose 3.)

Accepted Answer

Bayesian optimization. Bayesian optimization (A) is correct because SageMaker's built-in Bayesian strategy builds a probabilistic model of the objective (RMSE) and uses it to choose the next hyperparameter combination, making the search efficient when each training job is expensive. Random search (B) is correct because it samples hyperparameter combinations from the defined ranges, which is simple, parallelizable, and often finds good configurations faster than exhaustive methods in high-dimensional spaces. Hyperband (E) is correct because SageMaker supports Hyperband as a multi-fidelity strategy that early-stops poorly performing training jobs and reallocates resources to promising configurations, efficiently exploring the space. Grid search (C) is not among the correct answers because it exhaustively evaluates every combination in a predefined grid, which scales poorly and is inefficient for large or continuous hyperparameter spaces. Manual search (D) is not correct because it relies on human trial-and-error rather than an automated tuning strategy, so it does not efficiently explore the hyperparameter space.

Answer

Grid search

Answer

Manual search

Question 4

A data scientist is training a binary classification model using Amazon SageMaker. The dataset is highly imbalanced (99% negative class, 1% positive class). The model currently achieves 99% accuracy but fails to detect most positive cases. Which metric should the data scientist primarily use to evaluate model performance?

25 minute time limit — choose an answer to begin.

Accepted Answer

F1 score and AUC-ROC. With a highly imbalanced dataset (1% positive class), 99% accuracy is misleading because the model can achieve it by simply predicting the majority class. The low recall (10%) indicates the model fails to identify most positive instances. The F1 score balances precision and recall, providing a single metric for minority class performance, while AUC-ROC evaluates the model's ability to distinguish between classes across all thresholds, making it robust to class imbalance.

Answer

Precision and recall at 90% precision

Answer

Accuracy and RMSE

Answer

Precision and RMSE

Question 5

A data scientist is building a multi-class classification model with 10 classes. The dataset has 100,000 samples. After training a random forest with 100 trees, the model achieves 85% accuracy on the test set. However, the data scientist notices that for one rare class (1% of data), recall is only 5%. Which technique is MOST likely to improve recall for the rare class without significantly reducing overall accuracy?

Accepted Answer

Apply SMOTE to oversample the rare class in the training data. SMOTE (Synthetic Minority Oversampling Technique) generates synthetic samples for the rare class by interpolating between existing minority instances, which directly addresses the class imbalance. This increases the model's exposure to the rare class during training, improving recall without discarding data or significantly altering the overall class distribution, thus preserving overall accuracy.

Answer

Increase the number of trees to 500

Answer

Use stratified sampling only for the test set

Answer

Reduce the decision threshold for the rare class to 0.1

Question 6

Which TWO metrics are MOST appropriate for evaluating a regression model that predicts house prices, where the business is most sensitive to large errors?

Accepted Answer

Root Mean Squared Error (RMSE). RMSE (A) is correct because it squares the residuals before averaging and taking the square root, so large errors are penalized disproportionately—exactly matching a business that is most sensitive to large errors in predicted house prices. MAPE (B) is correct because it expresses errors as a percentage of actual prices, giving a scale-independent, relative measure of error that is directly interpretable for stakeholders comparing mispricing across differently priced homes. Accuracy (C) is not appropriate because it is a classification metric (fraction of correct labels) and regression outputs are continuous values, not discrete classes. MAE (D) is a valid regression metric but treats all errors linearly, so it does not emphasize large errors as strongly as RMSE does. R-squared (E) measures the proportion of variance explained by the model and does not directly quantify prediction error magnitude, so it is less suited to a large-error-sensitive business objective.

Answer

Accuracy

Answer

Mean Absolute Error (MAE)

Answer

R-squared

Question 7

A data scientist runs a SageMaker training job that fails with the above error. The S3 bucket and object exist, and the IAM role has s3:GetObject permission. What is the MOST likely cause?

Accepted Answer

The S3 object was uploaded with incorrect checksum or is corrupted. The error indicates that SageMaker cannot read the S3 object, even though the bucket and object exist and the IAM role has s3:GetObject permission. The most likely cause is that the object was uploaded with an incorrect checksum or is corrupted, which prevents SageMaker from verifying the integrity of the data during the training job initialization. SageMaker uses ETag (MD5 checksum) validation when reading objects, and a mismatch triggers a failure.

Answer

The training instance does not have internet access

Answer

The S3 bucket has versioning enabled and the object version is not specified

Answer

The IAM role lacks kms:Decrypt permission for an encrypted S3 object

Question 8

A data scientist is trying to run a SageMaker training job that writes output to an S3 bucket 'my-bucket'. The IAM policy is shown. The training job fails with an AccessDenied error when trying to write to S3. What is the reason?

Accepted Answer

The S3 bucket is encrypted with AWS KMS and the policy does not include kms:GenerateDataKey. The training job fails with an AccessDenied error when writing to an S3 bucket that is encrypted with AWS KMS. The IAM policy shown must include the `kms:GenerateDataKey` permission to allow the SageMaker training job to generate a data key for encrypting the output objects. Without this KMS permission, the S3 PutObject operation is denied even if the policy allows `s3:PutObject`, as KMS encryption requires explicit authorization to use the customer master key (CMK).

Answer

The policy does not allow s3:ListBucket

Answer

The policy does not allow s3:PutObject

Answer

The policy does not allow s3:PutObjectAcl

Question 9

A data scientist is training a binary classification model using Amazon SageMaker. The dataset is highly imbalanced (99% negative class, 1% positive class). The model currently achieves 99% accuracy but fails to detect most positive cases. Which metric should the data scientist primarily use to evaluate model performance?

25 minute time limit — choose an answer to begin.

Accepted Answer

Apply SMOTE to oversample the minority class. The dataset is highly imbalanced (500 churners out of 10,000 samples, a 5% positive rate). Logistic regression trained on such imbalance tends to bias toward the majority class, resulting in low recall for the minority class and a poor F1-score. SMOTE (Synthetic Minority Oversampling Technique) generates synthetic samples for the minority class by interpolating between existing minority instances, which balances the class distribution and allows the model to learn a better decision boundary, directly improving recall and F1-score.

Answer

Increase the regularization strength (C)

Answer

Apply PCA to reduce feature dimensionality

Answer

Use the original dataset without any modification

Question 10

A company is deploying a real-time fraud detection system using a gradient boosting model on AWS SageMaker. The model uses 200 features and is trained on 50 GB of data. The inference latency requirement is under 10 ms per request. During load testing, the endpoint shows average latency of 15 ms. Which change is MOST likely to reduce latency below 10 ms?

Accepted Answer

Reduce the number of features to the top 50 based on feature importance. Reducing the number of features from 200 to the top 50 directly decreases the amount of data each inference request must process, which lowers both feature engineering overhead and model evaluation time. For gradient boosting models on SageMaker, fewer features mean fewer decision tree splits to traverse per prediction, which can significantly reduce latency without requiring hardware changes. This is the most direct and cost-effective way to meet the 10 ms requirement.

Answer

Switch to a GPU-based instance type

Answer

Increase the number of trees in the model

Answer

Use a larger batch size for inference

Question 11

A company is building a recommendation system for an e-commerce platform. They have user-item interaction data (clicks, purchases) and want to use matrix factorization. They plan to use Amazon SageMaker to train the model. Which dataset format is MOST appropriate for the built-in Factorization Machines algorithm?

Accepted Answer

RecordIO-protobuf with user_id, item_id, and label fields. The built-in Factorization Machines algorithm in Amazon SageMaker requires the RecordIO-protobuf format for optimal performance, as it allows efficient binary serialization and direct integration with SageMaker's distributed training infrastructure. This format supports sparse data representation, which is critical for high-dimensional user-item interaction data, and enables faster I/O and reduced memory overhead compared to text-based formats.

Answer

Libsvm format with user_id and item_id as features

Answer

CSV file with user_id, item_id, and label columns

Answer

JSON lines file with user_id, item_id, and label fields

Question 12

A data scientist is training a binary classification model using Amazon SageMaker. The dataset is highly imbalanced (99% negative class, 1% positive class). The model currently achieves 99% accuracy but fails to detect most positive cases. Which metric should the data scientist primarily use to evaluate model performance?

25 minute time limit — choose an answer to begin.

Accepted Answer

F1 score. The F1 score is the harmonic mean of precision and recall, making it robust to class imbalance. With 95% accuracy but only 10% recall on the positive class, the model is essentially a trivial classifier that predicts the majority class. F1 score captures both false positives and false negatives, providing a balanced view of performance on the minority class.

Answer

Log loss

Answer

Accuracy

Answer

Area under the ROC curve (AUC)

Question 13

A research team is developing a deep learning model to classify medical images into 10 disease categories. They have a dataset of 50,000 labeled images, but the class distribution is highly imbalanced: the most common class has 20,000 images, while the rarest class has only 200 images. To address this, they apply data augmentation (random rotations, flips, and brightness adjustments) to the minority classes until each class has 20,000 images. They then train a convolutional neural network (CNN) from scratch using cross-entropy loss. The model achieves 95% overall accuracy but only 30% recall on the rarest class. Which change is MOST likely to improve recall on the rarest class without significantly reducing overall accuracy?

Accepted Answer

Replace cross-entropy loss with focal loss. Focal loss is specifically designed to address class imbalance by down-weighting the loss contribution from well-classified examples (majority classes) and focusing training on hard, misclassified examples (minority classes). This directly improves recall on the rarest class, while cross-entropy loss treats all classes equally, causing the model to be biased toward the majority classes.

Answer

Increase dropout rate from 0.2 to 0.5 to reduce overfitting

Answer

Switch from Adam optimizer to SGD with momentum

Answer

Reduce the batch size from 64 to 16 to increase stochasticity

Question 14

A financial services company is building a fraud detection model using a large dataset of credit card transactions. The dataset contains 10 million rows with 50 features, including transaction amount, merchant category, time of day, and customer historical features. The label is binary: fraudulent (1% of data) or legitimate. The company wants to deploy a real-time inference endpoint using Amazon SageMaker that can score transactions with sub-100ms latency. The current model is a gradient boosting model (XGBoost) trained on a sample of 1 million rows due to memory constraints. The model achieves 0.95 AUC on a held-out test set but the fraud recall (sensitivity) is only 0.4, which is unacceptable because the cost of missing a fraud is high. The data science team has access to a larger compute instance (ml.m5.24xlarge) for training. Which course of action is most likely to improve fraud recall while maintaining latency requirements?

Accepted Answer

Train the XGBoost model on the full 10 million rows using an ml.p3.2xlarge instance with GPU support, and apply SMOTE oversampling to the minority class before training.. Training on the full 10 million rows with a GPU-accelerated instance (ml.p3.2xlarge) allows the XGBoost model to learn from the complete data distribution, addressing the bias introduced by the 1 million row sample. Applying SMOTE oversampling to the minority class (fraud) directly tackles the class imbalance (1% fraud), which is the root cause of the low recall (0.4). SMOTE generates synthetic fraudulent examples, improving the model's ability to detect fraud without significantly increasing inference latency, as the model architecture and deployment remain unchanged.

Answer

Engineer additional features from transaction time and merchant category, then retrain the XGBoost model on the same 1 million row sample.

Answer

Downsample the majority class to 1% of the original size to create a balanced dataset of 200,000 rows, then retrain the XGBoost model on this balanced sample.

Answer

Replace XGBoost with a logistic regression model trained on the full dataset, as linear models are faster to train and may generalize better on large data.

Question 15

Drag and drop the steps to use Amazon SageMaker Debugger to debug a training job in the correct order.

Accepted Answer

Configure SageMaker Debugger hooks → Set up training job with Debugger rules → Execute training job → Analyze debug data in SageMaker Studio. Debugger requires hook configuration, job setup with rules, execution, and analysis.

Answer

Configure SageMaker Debugger hooks → Execute training job → Set up training job with Debugger rules → Analyze debug data

Answer

Execute training job → Configure SageMaker Debugger hooks → Set up training job with Debugger rules → Analyze debug data

Answer

Set up training job with Debugger rules → Configure SageMaker Debugger hooks → Execute training job → Analyze debug data

Question 16

Match each AWS AI service to its capability.

Accepted Answer

Amazon Comprehend: Extract insights and relationships from text. The correct matches are: Amazon Comprehend for extracting insights from text, Amazon Lex for building conversational interfaces, and Amazon Polly for text-to-speech. Distractors swap capabilities between services.

Answer

Amazon Lex: Translate text between languages

Answer

Amazon Polly: Convert speech to text

Question 17

A data scientist is training a binary classification model using Amazon SageMaker. The dataset is highly imbalanced (99% negative class, 1% positive class). The model currently achieves 99% accuracy but fails to detect most positive cases. Which metric should the data scientist primarily use to evaluate model performance?

25 minute time limit — choose an answer to begin.

Accepted Answer

F1 score. The F1 score is the harmonic mean of precision and recall, providing a balanced measure that considers both false positives and false negatives. On a highly imbalanced dataset, accuracy is misleading, and F1 score ensures the model is effectively detecting the positive class without being overwhelmed by the majority class.

Answer

Accuracy

Answer

Precision

Answer

Recall

Question 18

A company uses Amazon SageMaker to train a classification model. The training job fails with an error indicating that the algorithm requires a GPU but the instance type does not have one. The scientist used the built-in XGBoost algorithm. What should the scientist do to resolve the issue?

Accepted Answer

Choose a CPU instance type for the training job. SageMaker's built-in XGBoost algorithm is a CPU-based implementation and does not require a GPU — the error is triggered because the training job was launched on an instance type that the algorithm's container cannot use, or the job was misconfigured. The correct fix is to select a CPU instance type (e.g., ml.m5 or ml.c5 family) that matches the built-in XGBoost container's requirements. Built-in XGBoost in SageMaker is designed to run on CPU instances and does not support GPU acceleration out of the box.

Answer

Install a GPU-enabled version of XGBoost in the training container

Answer

Change the algorithm to a deep learning algorithm

Answer

Use a larger GPU instance type

Question 19

A company uses Amazon SageMaker to train a model using the built-in Linear Learner algorithm. The training data contains missing values in some features. What is the best practice for handling missing values with this algorithm?

Accepted Answer

Impute missing values using mean or median imputation. Mean or median imputation is the best practice because Linear Learner expects numeric input and cannot handle missing values natively; imputing with mean or median preserves the dataset size and avoids bias introduced by dropping rows. Median is preferred for skewed distributions, while mean works for roughly symmetric data. This approach is standard in SageMaker's own documentation and examples for Linear Learner.

Answer

Remove rows with missing values

Answer

Set missing values to zero

Answer

Use the `handle_missing` parameter in the algorithm

Question 20

Which TWO metrics are appropriate for evaluating a binary classification model when the cost of false negatives is high?

Accepted Answer

Recall. When false negatives are costly, we want to minimize them, so recall (true positive rate) is important. Precision is also important to avoid too many false positives, but F1 score balances both. Recall directly measures false negatives, and F1 combines precision and recall. AUC-ROC is a general measure, and accuracy can be misleading. Therefore, the two appropriate metrics are Recall (option C) and F1 score (option D).

Answer

Accuracy

Answer

AUC-ROC

Answer

Precision