MLS-C01 Modeling Practice Test 40 — 15 Questions

Question 1

A data scientist is training a model using SageMaker's built-in XGBoost algorithm. The dataset has 500 features and 1 million rows. The training job is taking too long. The scientist wants to reduce training time without sacrificing accuracy. Which action is LIKELY to be most effective?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Apply Principal Component Analysis (PCA) to reduce the number of features. PCA reduces the dimensionality of the dataset from 500 features to a smaller set of principal components, which directly decreases the computational complexity of training the XGBoost model. With fewer features, each tree split requires fewer operations, and the overall training time drops significantly. Since PCA retains the variance in the data, it can preserve model accuracy while speeding up training.

Answer

Use a smaller instance type to reduce time

Answer

Reduce the number of trees in XGBoost

Answer

Use a larger instance type with more vCPUs

Question 2

A data scientist is building a binary classifier to predict customer churn. The dataset is highly imbalanced (5% churn). Which TWO techniques can help improve the model's ability to detect churn?

Accepted Answer

Use Synthetic Minority Over-sampling Technique (SMOTE). SMOTE generates synthetic samples for the minority class by interpolating between existing minority instances, effectively balancing the dataset and providing the model with more diverse churn examples to learn from. This directly addresses the class imbalance problem without losing information from the majority class.

Answer

Downsample the majority class to balance the dataset

Answer

Use accuracy as the evaluation metric

Answer

Increase the model complexity by adding more layers

Question 3

A company is deploying a machine learning model on SageMaker for real-time inference. The model requires GPU for low latency. Which THREE steps are necessary to set up the endpoint?

Accepted Answer

Create a SageMaker model object that points to the S3 bucket containing the model artifacts and the inference container image. To deploy a model for real-time inference on SageMaker, you must first create a SageMaker model object that references the model artifacts stored in S3 and the inference container image (e.g., a GPU-enabled Docker image). This object is the foundational resource that SageMaker uses to launch instances for serving predictions.

Answer

Train the model using a SageMaker training job

Answer

Create a SageMaker batch transform job

Question 4

A data scientist is evaluating a regression model. Which TWO metrics are appropriate for evaluating regression performance?

Accepted Answer

Root Mean Squared Error (RMSE). Root Mean Squared Error (RMSE) is a standard metric for regression models because it measures the average magnitude of prediction errors in the same units as the target variable. It penalizes larger errors more heavily due to squaring, making it sensitive to outliers, which is useful for evaluating model accuracy in continuous value prediction.

Answer

F1 score

Answer

Area Under the ROC Curve (AUC)

Answer

Precision

Question 5

A company runs a real-time recommendation system on SageMaker with a model that uses a deep neural network. The endpoint uses a single ml.p3.2xlarge instance. Recently, the number of users has grown, and the endpoint's latency has increased from 50ms to 200ms, exceeding the SLA of 100ms. The model inference code is optimized and cannot be improved further. The company wants to reduce latency while minimizing cost. The data scientist has the following options: A. Switch to a larger instance type with more GPU memory, such as ml.p3.8xlarge. B. Use SageMaker's Elastic Inference to attach an EI accelerator to the existing instance. C. Deploy the model on multiple smaller instances (e.g., ml.p3.2xlarge) behind a load balancer and distribute traffic. D. Convert the model to use TensorFlow Lite and deploy on a CPU-based instance. Which option is the MOST cost-effective and meets the latency requirement?

Accepted Answer

Use SageMaker's Elastic Inference. The most cost-effective option is B (Use SageMaker's Elastic Inference) because it provides dedicated GPU acceleration at a fraction of the cost of a full GPU instance, reducing inference latency without requiring a larger instance. Option C (Switch to a larger instance type) would increase cost significantly. Option D (Deploy on multiple smaller instances behind a load balancer) would increase complexity and cost, and may not guarantee latency reduction. Option A (Convert to TensorFlow Lite on CPU) could reduce cost but may not meet the latency requirement as CPU inference is slower than GPU for deep neural networks, and model conversion might impact accuracy.

Answer

Convert to TensorFlow Lite on CPU

Answer

Switch to a larger instance type, e.g., ml.p3.8xlarge

Answer

Deploy on multiple smaller instances behind a load balancer

Question 6

A data scientist is training a binary classification model on a dataset with 100 features and 10,000 rows. The model overfits significantly: training accuracy is 99%, but validation accuracy is 80%. The data scientist has tried L1 and L2 regularization without improvement. The dataset is clean and representative. Which approach is MOST likely to reduce overfitting? A. Increase the number of training epochs. B. Add more training data by generating synthetic samples using SMOTE. C. Reduce the number of features using PCA. D. Use a simpler model like logistic regression instead of a decision tree ensemble. The data scientist needs to maintain a validation accuracy above 85%, but the current model is too complex. The company has limited budget for data labeling. Which option is BEST?

Accepted Answer

Use a simpler model like logistic regression. The current model (likely a decision tree ensemble like Random Forest or XGBoost) is too complex for the dataset, causing overfitting. Switching to a simpler model like logistic regression reduces variance by limiting the hypothesis space, which directly addresses overfitting without requiring additional data or feature engineering. Given the limited labeling budget, this approach is cost-effective and can improve generalization, potentially achieving the required >85% validation accuracy.

Answer

Add more training data by generating synthetic samples using SMOTE

Answer

Reduce the number of features using PCA

Answer

Increase the number of training epochs

Question 7

A data scientist is training a binary classification model on an imbalanced dataset (95% negative class, 5% positive class). The model achieves 95% accuracy but only predicts the negative class for all examples. Which metric should the scientist use to evaluate model performance more appropriately?

Accepted Answer

AUC-ROC. AUC-ROC is robust to class imbalance because it evaluates the model's ability to discriminate between positive and negative classes across all classification thresholds, rather than relying on a single threshold. In this scenario, the model predicts only the negative class, so its true positive rate is 0 and false positive rate is 0, yielding an AUC-ROC of 0.5 (random performance), which correctly reflects the model's lack of predictive power.

Answer

F1 score

Answer

Mean squared error

Answer

Accuracy

Question 8

A company uses Amazon SageMaker to train a deep learning model for image classification. The training job is taking longer than expected. The data scientist observes that GPU utilization is low (around 30%) and CPU utilization is high. Which action is most likely to reduce training time?

Accepted Answer

Increase the number of data loading workers. Low GPU utilization with high CPU utilization indicates a data loading bottleneck where the CPU cannot prepare batches fast enough to keep the GPU busy. Increasing the number of data loading workers (e.g., SageMaker's `sagemaker.session.Session` or PyTorch `DataLoader` `num_workers`) allows parallel data preprocessing and I/O, reducing idle GPU time and overall training duration.

Answer

Reduce the batch size

Answer

Increase the batch size

Answer

Increase the learning rate

Question 9

A machine learning engineer is deploying a model that predicts customer churn. The model outputs probabilities between 0 and 1. The business requires that at least 90% of customers flagged for churn actually churn (precision >= 0.9). Currently, the model's precision is 0.85 at the default threshold of 0.5. Which threshold adjustment should the engineer consider?

Accepted Answer

Increase the threshold to 0.7. Increasing the threshold to 0.7 raises the probability cutoff for classifying a customer as churning. This means only customers with a high predicted probability (strong model confidence) are flagged, which reduces false positives and increases precision. Since the current precision at 0.5 is 0.85 and the goal is ≥0.9, moving the threshold higher is the correct direction to achieve the required precision.

Answer

Decrease the threshold to 0.4

Answer

Decrease the threshold to 0.3

Answer

Keep the threshold at 0.5

Question 10

A data scientist is building a regression model to predict house prices. The dataset contains 10 features, including 'number_of_bedrooms' and 'square_footage'. The scientist observes that the model has high variance. Which TWO actions are most appropriate to reduce overfitting? (Choose TWO.)

Accepted Answer

Reduce model complexity by using a simpler model. A is correct because reducing model complexity, such as using a simpler model (e.g., linear regression instead of a high-degree polynomial), directly decreases variance by limiting the model's capacity to fit noise in the training data. This aligns with the bias-variance tradeoff, where simpler models have higher bias but lower variance, making them less prone to overfitting.

Answer

Increase the number of training epochs

Answer

Decrease the amount of training data

Answer

Add more polynomial features

Question 11

A company uses Amazon SageMaker to train a deep learning model using TensorFlow. The training job is failing with an 'OutOfMemory' error. The instance type is ml.p3.2xlarge with 16 GB GPU memory. The model has 10 million parameters. Which THREE actions should be taken to resolve the memory issue? (Choose THREE.)

Accepted Answer

Reduce the batch size. Reducing the batch size directly decreases the memory footprint per training step because fewer samples are loaded into GPU memory simultaneously. With 10 million parameters and 16 GB GPU memory, the default batch size may exceed available memory for activations and gradients. This is the most straightforward fix for an OutOfMemory error in TensorFlow on SageMaker.

Answer

Increase the number of epochs

Answer

Increase the batch size

Question 12

A data scientist is performing hyperparameter optimization for a gradient boosting model using Amazon SageMaker Automatic Model Tuning. The objective metric is 'validation:logloss'. Which TWO strategies can help the tuning job converge faster? (Choose TWO.)

Accepted Answer

Use Bayesian optimization strategy. Bayesian optimization is a hyperparameter tuning strategy that builds a probabilistic model of the objective function and uses it to select the most promising hyperparameter combinations to evaluate next. By focusing on regions of the hyperparameter space that are likely to yield better validation:logloss, it converges to an optimal configuration in fewer training jobs compared to uninformed search methods, thus speeding up the tuning process.

Answer

Increase the number of tuning jobs

Answer

Increase the resource limits for each training job

Answer

Use random search strategy

Question 13

A data scientist has this IAM policy attached to an IAM role used by SageMaker. When trying to create a training job, the scientist gets an access denied error. The training data is in 's3://my-bucket/training-data/'. What is the most likely cause?

Accepted Answer

Missing s3:ListBucket permission. The error occurs because the IAM policy grants s3:GetObject permission on the training data objects but lacks s3:ListBucket permission on the bucket itself. SageMaker's CreateTrainingJob API first performs a ListBucket call to verify the bucket exists and to enumerate objects, even if the exact object key is known. Without s3:ListBucket, the ListBucket call fails, resulting in an access denied error.

Answer

The bucket name is misspelled

Answer

The S3 resource ARN is incorrect

Answer

The sagemaker:CreateTrainingJob action is not allowed

Question 14

A data scientist ran a SageMaker training job that failed with the error shown. The training script expects the data in '/opt/ml/input/data/training/train.csv'. What is the most likely issue?

Accepted Answer

The S3 data path should point to the exact file, not the folder. The SageMaker training job expects the S3 data path to point directly to the CSV file (e.g., s3://bucket/train.csv), not to a folder containing the file. When the path points to a folder, SageMaker downloads the folder contents but the training script's hardcoded path '/opt/ml/input/data/training/train.csv' fails because the file is not placed at that exact location—SageMaker copies the file into the channel directory with its original name, but the folder path causes the file to be nested or missing, leading to a file-not-found error.

Answer

The hyperparameter 'sagemaker_program' is misspelled

Answer

The training script has a bug in reading the file

Answer

The channel name should be 'train' instead of 'training'

Question 15

A data scientist is setting up a SageMaker training job and has attached this IAM policy to the execution role. The training job fails with an access denied error when trying to write to the output path 's3://my-bucket/output/model.tar.gz'. What additional permission is needed?

Accepted Answer

s3:ListBucket. The training job fails because SageMaker needs to verify that the output S3 bucket exists before writing to it. The s3:ListBucket permission is required to list the contents of the bucket (or confirm its existence) as part of the write operation. Without this permission, the service cannot validate the bucket, resulting in an access denied error even if s3:PutObject is allowed.

Answer

s3:GetObject for the output path

Answer

s3:DeleteObject

Answer

iam:PassRole on the role itself