Modeling practice questions

Impute missing values with zeros and use singular value decomposition (SVD)

Why wrong: Imputing zeros biases the model because missing does not mean zero preference.

Remove all users and items with fewer than 10 interactions to reduce sparsity

Why wrong: This reduces data and may lose valuable patterns.

Use item-based collaborative filtering with cosine similarity

Why wrong: Collaborative filtering also suffers from sparsity; it does not inherently handle missing interactions better.

Use a smaller instance type

Why wrong: Incorrect: Smaller instance type reduces compute resources, increasing training time.

Reduce the number of epochs

Why wrong: Incorrect: Reducing epochs shortens training time but may lead to underfitting, sacrificing accuracy. The goal is to maintain accuracy.

Use managed spot training with checkpointing

Why wrong: Incorrect: Managed spot training primarily reduces cost, not training time. It may even increase total time due to instance interruptions, even with checkpointing.

Reduce the batch size

Why wrong: Reducing batch size introduces noise in the gradient estimates, which can sometimes help generalization but is not a primary or reliable method to mitigate overfitting.

Increase the number of layers

Why wrong: Increasing the number of layers increases model capacity, which typically worsens overfitting rather than reducing it.

Increase the number of training epochs

Why wrong: Increasing the number of training epochs can lead to overfitting if the model continues to learn noise in the training data, so it does not mitigate overfitting.

Linear Learner

Why wrong: Linear Learner can model trends but not seasonality without manual feature engineering.

K-Means

Why wrong: K-Means is for clustering, not forecasting.

XGBoost

Why wrong: XGBoost does not natively handle time series seasonality; requires manual feature engineering.

Increase subsample

Why wrong: Subsample ratio doesn't directly affect tree depth or leaf weights.

Increase eta (learning rate)

Why wrong: Increasing eta makes steps larger, potentially causing more overflow.

Increase num_round

Why wrong: More rounds can increase leaf values, exacerbating overflow.

Reduce the number of features by using PCA before training.

Why wrong: Reducing features with PCA can lower memory usage but may not be enough for 100 GB data and can degrade model quality. It addresses the symptom, not the root cause.

Use SageMaker Pipe mode to stream data directly from S3 instead of downloading it.

Why wrong: Pipe mode streams data in chunks, but XGBoost still accumulates data in memory for training. Without distributed training, memory may still be exhausted.

Use SageMaker BlazingText algorithm with negative sampling.

Why wrong: BlazingText is for text data and not applicable to gradient boosting; it does not solve the memory issue for XGBoost.

Switch to SageMaker Linear Learner algorithm, which requires less memory.

Why wrong: Switching to Linear Learner does not guarantee lower memory usage; for large datasets, it may still encounter memory errors. It is not a robust fix.

F1 score (macro-averaged)

Why wrong: F1 score is based on thresholded predictions, not raw probabilities.

Accuracy

Why wrong: Accuracy does not consider probability calibration and is not sensitive to ranking.

Mean Absolute Error

Why wrong: MAE is for regression, not classification.

ROC AUC (one-vs-rest macro-averaged)

Why wrong: ROC AUC is appropriate for binary classification, not directly for multiclass probabilities.

Specificity

Why wrong: Incorrect – Specificity measures true negative rate and focuses on false positives, not false negatives; it is not appropriate when FN cost is high.

Accuracy

Why wrong: Incorrect – Accuracy can be misleading with imbalanced data and does not specifically target false negatives.

F1 score

Why wrong: Incorrect – F1 score balances precision and recall but is not solely focused on false negatives.

Precision

Why wrong: Incorrect – Precision focuses on false positives, not false negatives.

Increase the batch size to process more requests per invocation.

Why wrong: Larger batch size increases latency per request.

Switch to a compute-optimized instance like c5.18xlarge.

Why wrong: CPU instances may not accelerate deep learning inference sufficiently.

Use SageMaker Neo to compile the model for the target instance.

Why wrong: Neo optimizes for single device, not model parallelism.

Decrease the batch size.

Why wrong: Smaller batch size may help generalization but not directly stop overfitting.

Add more layers to the model.

Why wrong: More layers increase capacity and risk overfitting.

Increase the learning rate.

Why wrong: Higher learning rate may cause instability.

The role does not have permission to use KMS to decrypt the model artifacts

Why wrong: This is not the most likely reason. KMS decryption permissions are only necessary if the model artifacts are encrypted with a KMS key. The question does not indicate that encryption is used.

The policy does not include 'sagemaker:InvokeEndpoint' action

Why wrong: Incorrect. sagemaker:InvokeEndpoint is not required to create an endpoint; it is used to invoke the endpoint after it is deployed. The error occurs during creation, not invocation.

The policy uses a wildcard '*' for resources instead of specific ARNs

Why wrong: Incorrect. Using a wildcard '*' for resources is common and does not inherently cause AccessDenied errors. The issue is more likely a missing action or resource-specific permission.

Increase the image size

Why wrong: Incorrect. Increasing the image size adds more pixels per image, increasing computational load and training time.

Increase the number of training epochs

Why wrong: Incorrect. Increasing the number of training epochs adds more full passes through the dataset, directly increasing training duration.

Decrease the batch size

Why wrong: Incorrect. Decreasing the batch size typically reduces hardware utilization efficiency and may increase the number of parameter updates, often lengthening training time.

Configure a VPC for the SageMaker notebook and training job.

Why wrong: A VPC alone does not encrypt data and does not prevent access to the PII; it only provides network isolation.

Load the data into Amazon Redshift and use Redshift ML.

Why wrong: Redshift ML still exposes data during training.

Use server-side encryption with S3-managed keys (SSE-S3).

Why wrong: Encryption alone (SSE-S3) does not prevent access to the data during training; the training container can still read the data.

The policy does not allow sagemaker:InvokeEndpoint.

Why wrong: sagemaker:InvokeEndpoint is not required during endpoint creation; it is used for inference after the endpoint is active. Thus, missing this permission is not the most likely cause of the AccessDenied error.

The policy does not allow access to the S3 bucket for model artifacts.

Why wrong: While access to the S3 bucket for model artifacts is needed to create the model, this would typically cause an error at model creation time, not at endpoint creation. If the model already exists, endpoint creation does not require S3 access. So this is not the most likely cause.

The policy does not allow sagemaker:CreateModel.

Why wrong: Missing sagemaker:CreateModel would cause an error earlier when creating the model, not during endpoint creation. If the model already exists, this permission is not needed for endpoint creation.

Write a custom algorithm that calculates accuracy

Why wrong: Unnecessary; the issue is just the metric name.

Change the metric to 'validation:rmse'

Why wrong: Changing the metric to 'validation:rmse' could potentially work since it is a supported metric, but the error indicates that the metric definition is the problem. The correct action is to remove the metric definition, not to replace it. Therefore, this option is not the best step.

Use a different built-in algorithm that supports accuracy

Why wrong: XGBoost can still be used; just need to specify a supported metric.

Use a multi-variant endpoint with two variants

Why wrong: Multi-variant endpoints are for A/B testing, not for serving multiple models.

Use a serverless endpoint with provisioned concurrency

Why wrong: Serverless endpoints may have cold starts and are not optimized for sparse requests.

Use a single model endpoint with a large instance type

Why wrong: A single model endpoint may not handle sparse high-dimensional features efficiently.

Reduce the amount of training data

Why wrong: Less data typically increases overfitting risk.

Increase the maximum depth of the trees in XGBoost

Why wrong: Increasing tree depth in XGBoost adds model complexity by allowing finer-grained splits, which directly exacerbates overfitting rather than reducing it. This option tempts because deeper trees can capture intricate patterns in training data, making it a correct choice when underfitting is the problem—for example, when the model has high bias and fails to learn the training signal adequately.

Increase the number of training epochs

Why wrong: More epochs can lead to overfitting if not regularized.

Enable data augmentation.

Why wrong: Augmentation increases data size, likely increasing training time.

Use SageMaker Automatic Model Tuning.

Why wrong: Tuning explores multiple hyperparameter combinations, increasing total training time.

Use SageMaker Managed Spot Training.

Why wrong: Spot training reduces cost but may not reduce time; it can cause interruptions.

Increase the number of hidden layers.

Why wrong: Increasing layers increases model complexity and overfitting.

Decrease the learning rate.

Why wrong: Decreasing learning rate affects convergence speed, not overfitting directly.

Reduce the batch size.

Why wrong: Reducing batch size introduces noise but is not a standard method to reduce overfitting.

Use model parallelism to split the model across multiple GPUs

Why wrong: Model parallelism splits the model across GPUs, which is useful when the model is too large for one GPU. Since the model already fits on a single GPU, this adds unnecessary complexity without significant time reduction.

Use SageMaker Managed Spot Training to reduce cost

Why wrong: SageMaker Managed Spot Training reduces cost, not training time. It does not address the need for faster training.

Use SageMaker Automatic Model Tuning to find optimal hyperparameters

Why wrong: SageMaker Automatic Model Tuning optimizes hyperparameters, which may improve model performance but does not directly reduce training time.