MLS-C01 Modeling Practice Test 8 — 15 Questions

Question 1

A company is building a binary classifier to predict customer churn. The dataset has 10,000 samples with 500 churners (5% positive class). After training a logistic regression model, the precision is 0.8 and recall is 0.2. Which metric should the data scientist focus on to improve the model's ability to identify churners while minimizing false positives?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Increase recall. The model's recall is very low (0.2), meaning it misses most churners. Since the goal is to identify churners (positive class), improving recall should be the primary focus. Option A (accuracy) is misleading due to class imbalance. Option B (precision) is already high (0.8), so further improvement would likely reduce recall. Option D (F1 score) balances precision and recall, but the immediate need is to address the low recall, not to balance both.

Answer

Increase accuracy

Answer

Increase precision

Answer

Increase F1 score

Question 2

A data scientist is training a deep learning model on Amazon SageMaker using a custom Docker container. The training job fails with an error 'OutOfMemoryError: CUDA out of memory'. The instance type is ml.p3.2xlarge (8 GB GPU memory). The model has 50 million parameters. What is the most likely cause and solution?

Accepted Answer

The model uses FP32 precision; enable mixed precision training. A model with 50 million parameters in FP32 precision requires approximately 200 MB per parameter (4 bytes each = 200 MB for 50M), plus additional memory for activations, gradients, and optimizer states, which can easily exceed the 8 GB GPU memory of ml.p3.2xlarge. Mixed precision training (FP16) halves the memory usage for tensors, reducing the overall footprint and often fitting the model within GPU limits. Option A (instance type) may solve the problem but is more expensive and unnecessary if mixed precision works. Option B (batch size) is a contributing factor but not the most likely root cause, as even a batch size of 1 may still cause OOM due to parameter storage. Option C (gradient checkpointing) trades compute for memory by recomputing activations, but does not address the primary issue of parameter storage in FP32. Therefore, enabling mixed precision is the most direct and cost-effective solution.

Answer

The instance type is insufficient; switch to ml.p3.8xlarge

Answer

The batch size is too large; reduce batch size

Answer

Enable gradient checkpointing to reduce memory

Question 3

A team deployed a SageMaker endpoint for real-time inference using a PyTorch model. After monitoring, they notice that the latency is highly variable, with p99 latency 10x the p50 latency. The endpoint uses a single ml.c5.2xlarge instance with auto-scaling based on average CPU utilization. Which change is most likely to reduce latency variability?

Accepted Answer

Change the auto-scaling metric to 'InvocationsPerInstance'. Scaling based on InvocationsPerInstance allows the endpoint to react more quickly to changes in request volume, reducing the queueing that causes high p99 latency. Option A (increasing batch size) would actually increase latency. Option B (pre-warming) helps with cold starts but not queueing from traffic spikes. Option C (GPU instance) is unlikely to help if the model is CPU-bound and would not address the root cause of latency variability.

Answer

Increase the batch size for inference

Answer

Pre-warm the model by sending dummy requests every minute

Answer

Switch to a GPU instance type

Question 4

A company is fine-tuning a BERT model on Amazon SageMaker for a text classification task. The training script uses PyTorch and Hugging Face Transformers. The training job completes successfully, but the final model accuracy is low. The dataset has 10,000 labeled samples. What is the most likely cause and solution?

Accepted Answer

The model is overfitting due to small dataset; use a pre-trained checkpoint and fine-tune only top layers. Fine-tuning the entire BERT model on only 10,000 samples leads to overfitting, resulting in low accuracy. The recommended approach is to use a pre-trained checkpoint and fine-tune only the top layers, which leverages transfer learning and reduces the risk of overfitting. Option A (instance type) impacts training speed, not accuracy directly. Option C (learning rate) could be a factor but overfitting is the most likely given the dataset size. Option D (data loader bug) would typically cause errors, not low accuracy without errors.

Answer

The instance type is insufficient; use a larger instance

Answer

The learning rate is too high; reduce it

Answer

The training script has a bug in the data loader

Question 5

A machine learning team is using SageMaker to train a model with the built-in Linear Learner algorithm. The dataset has 1 million rows and 20 features. The training completes, but the model's mean squared error (MSE) is high. Which parameter adjustment is most likely to reduce MSE?

Accepted Answer

Increase the number of epochs. Increasing the number of epochs allows the model to see the data more times, helping it converge to a lower training error, thus reducing MSE. Option A is incorrect: increasing mini-batch size typically improves computational efficiency but can make convergence slower per epoch, potentially requiring even more epochs to converge; it may not directly reduce MSE. Option B is incorrect: cross-entropy is a loss function for classification problems, not regression; Linear Learner with MSE is appropriate for regression. Option D is incorrect: increasing the learning rate can cause the optimizer to overshoot the minimum or diverge, often increasing rather than decreasing MSE.

Answer

Increase the mini-batch size

Answer

Change the loss function to cross-entropy

Answer

Increase the learning rate

Question 6

A company is using Amazon SageMaker to host a model for real-time inference. The model is a large ensemble of 10 XGBoost models, each 2 GB. The endpoint uses a single ml.c5.18xlarge instance. The inference latency is high (average 2 seconds). Which change would most effectively reduce latency?

Accepted Answer

Use SageMaker Multi-Model Endpoints to serve each model independently. Serialization/deserialization of large models is a bottleneck; SageMaker Multi-Model Endpoints can reduce overhead by loading only the requested model. Option B (GPU) may not help if the bottleneck is CPU. Option C (Add more instances) helps throughput but not per-request latency. Option D (Batch Transform) is for offline inference.

Answer

Switch to a GPU instance type

Answer

Add more instances behind a load balancer

Answer

Use SageMaker Batch Transform instead of real-time endpoint

Question 7

A data scientist has this IAM policy attached to their role. When trying to create a SageMaker endpoint using the AWS CLI, they get an 'AccessDenied' error. What is the most likely reason?

Accepted Answer

The role does not have permission to access the S3 bucket containing the model artifacts. The role does not have permission to access the S3 bucket containing the model artifacts. When creating a SageMaker endpoint, the service needs to access the model artifacts stored in S3. The IAM role attached to the data scientist must have s3:GetObject permission on the bucket. Without this, the CreateEndpoint API call will fail with AccessDenied. Option B is incorrect because sagemaker:InvokeEndpoint is not required to create an endpoint; it is needed for invoking the endpoint after creation. Option A is less likely because KMS decryption is only needed if the model artifacts are encrypted with a customer-managed KMS key. Option D is not a typical cause because using a wildcard '*' for resources is acceptable and does not cause AccessDenied errors.

Answer

The role does not have permission to use KMS to decrypt the model artifacts

Answer

The policy does not include 'sagemaker:InvokeEndpoint' action

Answer

The policy uses a wildcard '*' for resources instead of specific ARNs

Question 8

A financial services company is building a fraud detection model using Amazon SageMaker. The dataset has 10 million transactions, with 0.1% fraudulent. They train an XGBoost model with default hyperparameters. The model achieves 99.9% accuracy on the test set, but only catches 10% of actual fraud cases. The company wants to maximize the number of fraud cases caught while keeping the false positive rate below 5%. The data scientist has already tried adjusting the class weights and threshold, but the recall is still low. What should the data scientist do next?

Accepted Answer

Use a different algorithm such as a balanced random forest or SMOTE with XGBoost. The dataset is extremely imbalanced (0.1% fraud), and the model's high accuracy is misleading—it's likely predicting the majority class almost exclusively. Adjusting class weights and thresholds hasn't improved recall, indicating the model isn't learning the minority class patterns. Using a different algorithm designed for imbalanced data, such as a balanced random forest (which undersamples the majority class in each tree) or applying SMOTE (which synthetically generates minority class samples) with XGBoost, directly addresses the imbalance and can significantly improve recall while controlling false positives.

Answer

Collect more data, especially fraudulent transactions, to balance the dataset

Answer

Apply PCA to reduce the number of features and prevent overfitting

Answer

Use a larger instance type to train for more epochs

Question 9

A company is training a deep learning model on Amazon SageMaker using a large dataset stored in S3. The training job is failing with an error indicating insufficient memory. The model architecture and hyperparameters are fixed. Which change is MOST likely to resolve the issue without modifying the model code?

Accepted Answer

Enable SageMaker's distributed data parallelism.. Enable SageMaker's distributed data parallelism. Since the model architecture and hyperparameters are fixed, the insufficient memory error likely arises because the dataset is too large to fit into the memory of a single instance. Distributed data parallelism splits the training data across multiple instances, allowing each instance to process a smaller subset, thereby reducing per-instance memory usage without any code modifications. Option B (managed Spot training) reduces cost but does not address memory. Option C (using a larger instance) could provide more memory but may be more expensive and does not directly solve the root cause of data size relative to fixed hyperparameters. Option D (Pipe mode) improves data streaming efficiency but does not reduce the memory required for model parameters or intermediate activations.

Answer

Use managed Spot training to get cheaper compute.

Answer

Use a larger instance type with more memory.

Answer

Use Pipe mode for input data instead of File mode.

Question 10

A data scientist is using Amazon SageMaker to train a gradient boosting model on a dataset with categorical features. The dataset contains a column 'UserID' with over 1 million unique values. The training is taking very long and the model size is large. Which technique would MOST effectively reduce training time and model size while maintaining accuracy?

Accepted Answer

Apply feature hashing to UserID.. Hashing reduces the number of distinct categories to a fixed number of buckets, controlling dimensionality. Option A is wrong because one-hot encoding would explode the feature space. Option C is wrong because label encoding creates ordinal relationships that may mislead the model. Option D is wrong because removing UserID likely loses important signal.

Answer

Use one-hot encoding on UserID.

Answer

Use label encoding for UserID.

Answer

Remove UserID from the dataset.

Question 11

A machine learning engineer is using Amazon SageMaker to deploy a model for real-time inference. The model must respond within 100 milliseconds. The initial deployment uses a single ml.m5.large instance, but latency is too high. Which change should the engineer make to reduce latency?

Accepted Answer

Switch to a compute-optimized instance like ml.c5.2xlarge.. A compute-optimized instance like ml.c5.2xlarge provides more CPU and memory, reducing inference latency. Option B is wrong because batch transform is for offline predictions, not real-time; it does not reduce latency for real-time inference. Option C is wrong because using a smaller instance (ml.t2.medium) reduces resources and would likely increase latency, not reduce it. Option D is wrong because multi-model endpoints share resources among models and can lead to contention, potentially increasing latency.

Answer

Use batch transform instead of real-time endpoint.

Answer

Deploy to a single ml.t2.medium instance to reduce cost.

Answer

Deploy the model on a multi-model endpoint.

Question 12

A data scientist is training a binary classification model using Amazon SageMaker's built-in XGBoost algorithm. The dataset is highly imbalanced (95% negative class, 5% positive class). The model achieves high accuracy but poor recall on the positive class. Which TWO actions should the data scientist take to improve recall without significantly sacrificing precision?

Accepted Answer

Set scale_pos_weight to the ratio of negative to positive samples.. Option B is correct because SageMaker's built-in XGBoost supports the scale_pos_weight hyperparameter, and setting it to the ratio of negative to positive samples (approximately 95/5 = 19) directly reweights the positive class's contribution to the loss, counteracting the imbalance and improving recall on the minority class. Option E is correct because SMOTE (Synthetic Minority Over-sampling Technique) generates synthetic minority-class samples by interpolating between existing positive examples, which balances the class distribution and gives the model more positive signal to learn from, improving recall. Option A is not correct because random undersampling discards a large portion of the majority class, which can lose valuable information and hurt overall precision and generalization. Option C is not correct because increasing max_depth makes the trees more complex and prone to overfitting; it does not specifically address class imbalance and may worsen precision. Option D is not correct because reducing eta and increasing num_round is a general regularization/boosting tuning strategy that affects convergence and overfitting, not the class-imbalance problem causing poor positive-class recall.

Answer

Perform random undersampling of the majority class.

Answer

Increase the max_depth hyperparameter.

Answer

Reduce the learning rate (eta) and increase num_round.

Question 13

A machine learning engineer is deploying a model using Amazon SageMaker. The model requires preprocessing steps (e.g., scaling, encoding) that were applied during training. Which TWO options can ensure the same preprocessing is applied at inference?

Accepted Answer

Create a Scikit-learn pipeline that includes preprocessing and the model, then deploy it.. Option D is correct because a Scikit-learn pipeline bundles the preprocessing transformers (e.g., StandardScaler, OneHotEncoder) together with the estimator into a single serialized artifact, so SageMaker invokes the exact same fitted transformations at inference time as during training. Option E is correct because a SageMaker Inference Pipeline chains a preprocessing container (which can host the fitted transformer logic) directly to the model container, so each request is automatically passed through the same preprocessing before scoring, without the client managing separate calls. Option A is not ideal because an AWS Lambda function invoked before inference is external glue code that must be kept in sync with the training transformations and adds latency and operational overhead, rather than guaranteeing identical fitted preprocessing. Option B is not ideal because a separate preprocessing endpoint still requires the client to orchestrate two calls and does not inherently guarantee the same fitted transformer state is reused. Option C is wrong because retraining the model on every inference request is computationally wasteful, non-deterministic, and not how SageMaker endpoints are designed to serve predictions.

Answer

Implement preprocessing as an AWS Lambda function invoked before inference.

Answer

Deploy a separate preprocessing endpoint and call it before the model endpoint.

Answer

Retrain the model in each inference request with the preprocessing applied.

Question 14

A data scientist is using Amazon SageMaker Autopilot to automatically build a binary classification model. The dataset has 50 features and 100,000 rows. After the experiment completes, the best candidate model achieves an F1 score of 0.85 on the validation set. However, when deployed to a real-time endpoint, the model's F1 score drops to 0.72 on production data. The data distributions between training and production are similar. What is the MOST likely cause of the performance drop?

Accepted Answer

The Autopilot pipeline used features that are not available at inference time (data leakage).. The most likely cause is data leakage: Autopilot's feature engineering pipeline included features that are derived from or correlated with the target in a way that is not reproducible at inference time, so validation F1 is inflated relative to production. Because the distributions are stated to be similar, drift and missing-value issues are ruled out, and instance type does not affect model quality. Data leakage is the classic explanation for a large train/validation-to-production gap with stable distributions.

Answer

Concept drift occurred between training and production.

Answer

The production data contains missing values that were not present in training.

Answer

The inference endpoint uses a different instance type than training.

Question 15

A company wants to use Amazon SageMaker to train a model, but the training data contains personally identifiable information (PII). The data scientist needs to ensure that the PII is not accessible during training. The data is stored in S3. What is the MOST secure approach?

Accepted Answer

Use SageMaker's built-in mechanisms to encrypt data at rest and in transit, and ensure the training container does not have direct S3 access.. Using SageMaker with a SageMaker-managed KMS key encrypts data at rest and in transit, and the training container cannot access S3 directly. Option A is wrong because encryption alone does not prevent access. Option B is wrong because Redshift adds complexity. Option C is wrong because a VPC alone does not encrypt data.

Answer

Configure a VPC for the SageMaker notebook and training job.

Answer

Load the data into Amazon Redshift and use Redshift ML.

Answer

Use server-side encryption with S3-managed keys (SSE-S3).