MLS-C01 Modeling Practice Test 5 — 15 Questions

Question 1

A data scientist wants to evaluate the performance of a multiclass classification model. The model outputs probabilities for 10 classes. Which metric is most appropriate for evaluating the model's ranking performance across all classes?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Log loss (cross-entropy). Log loss measures the performance of a classification model where the prediction is a probability value, and it penalizes false classifications. Option A (F1 score macro-averaged) is not appropriate because it focuses on threshold-based metrics rather than probability ranking. Option B (Accuracy) ignores probability calibration. Option C (Mean Absolute Error) is for regression, not classification. Option E (ROC AUC one-vs-rest macro-averaged) can be used for multiclass ranking, but log loss is more appropriate when the model outputs probabilities and you want to evaluate the overall probability quality across all classes.

Answer

F1 score (macro-averaged)

Answer

Accuracy

Answer

Mean Absolute Error

Answer

ROC AUC (one-vs-rest macro-averaged)

Question 2

A company is building a recommendation system using Amazon SageMaker Factorization Machines. The dataset includes user IDs, item IDs, and implicit feedback (clicks). The data is sparse with millions of users and items. The model needs to capture interactions between users and items. Which hyperparameter tuning strategy should be used to improve model performance?

Accepted Answer

Increase the number of factors (num_factors) to capture more latent features.. (increase number of factors) is correct because increasing num_factors increases the dimensionality of the latent feature vectors, allowing the model to capture more complex interactions between users and items. Option A (L2 regularization) helps prevent overfitting but does not increase the model's capacity to capture interactions. Option B (batch size) affects training speed and stability, not the expressiveness of the model. Option C (learning rate) influences convergence but not the complexity of interactions. Option D (activation function) is not applicable since Factorization Machines are linear models and do not use activation functions like ReLU.

Answer

Increase L2 regularization to prevent overfitting.

Answer

Increase the batch size to speed up training.

Answer

Decrease the learning rate to improve convergence.

Answer

Change the activation function to ReLU.

Question 3

A data scientist is building a fraud detection model using a highly imbalanced dataset. The model uses a random forest classifier. The recall for the minority class is 0.6, and precision is 0.9. The business requires recall above 0.8. Which action should the data scientist take to improve recall?

Accepted Answer

Decrease the probability threshold for classifying a transaction as fraudulent.. Lowering the classification threshold directly increases recall by classifying more transactions as fraudulent, catching more true positives at the cost of precision. Since the business requires recall above 0.8 and the current recall is 0.6, adjusting the decision threshold is the most direct and immediate lever. This is a standard precision-recall tradeoff adjustment.

Answer

Perform feature selection to remove noisy features.

Answer

Increase the maximum depth of the trees.

Answer

Increase the class weight for the minority class in the algorithm.

Answer

Increase the number of trees in the random forest.

Question 4

A data scientist is training a gradient boosting model using SageMaker's built-in XGBoost algorithm. The dataset has missing values in several features. Which TWO actions should the data scientist take to handle missing values effectively? (Choose two.)

Accepted Answer

Impute missing values with the median of each feature using a preprocessing step.. Option A is correct because SageMaker's built-in XGBoost does not natively impute missing values in all cases, so a preprocessing step that imputes missing values with the median of each feature (a robust statistic for skewed data) ensures the model receives complete numeric input and avoids errors or bias from dropped data. Option E is correct because the SageMaker XGBoost container exposes the 'missing' hyperparameter, which lets you designate a sentinel value (e.g., 0) for missing entries; XGBoost then learns a default direction for missing values during tree construction, effectively performing learned imputation without manual preprocessing. Option B is not appropriate because one-hot encoding missingness indicators is a feature-engineering technique for categorical variables and does not directly resolve missing numeric values for XGBoost. Option C is not recommended because removing all rows with missing values discards potentially valuable data and can introduce bias, especially if missingness is not random. Option D is incorrect because PCA does not handle missing values and would fail or require imputation first, and ignoring missing values is not a valid strategy for XGBoost training.

Answer

Use one-hot encoding to create binary columns indicating missingness.

Answer

Remove all rows with missing values from the training dataset.

Answer

Apply PCA to reduce dimensionality and ignore missing values.

Question 5

A company is building a binary classifier to detect fraudulent transactions. The dataset is highly imbalanced (99% legitimate, 1% fraudulent). Which metric is most appropriate for evaluating the model?

Accepted Answer

F1-score. F1-score is the harmonic mean of precision and recall, making it the standard metric for highly imbalanced binary classification because it balances false positives and false negatives rather than being dominated by the majority class. With 99% legitimate transactions, a model predicting 'legitimate' for everything would score 99% accuracy but 0 F1 on the fraud class, so F1 exposes that failure.

Answer

Accuracy

Answer

Mean Squared Error

Answer

Area Under the ROC Curve (AUC-ROC)

Question 6

A machine learning team trains a deep learning model on SageMaker. The training job uses a single ml.p3.2xlarge instance and takes 12 hours. The team needs to reduce training time without changing the algorithm. Which approach is most effective?

Accepted Answer

Use SageMaker's distributed data parallelism across multiple instances. Using SageMaker's distributed data parallelism (e.g., with SageMaker distributed training libraries) across multiple GPUs can significantly reduce training time by splitting the mini-batches across GPUs. Increasing instance type to a single larger GPU (e.g., p3.16xlarge) helps but is less effective than multi-GPU distribution. Hyperparameter tuning doesn't directly reduce training time. Spot instances may interrupt.

Answer

Increase the learning rate

Answer

Switch to a larger instance type, such as ml.p3.16xlarge

Answer

Use managed Spot Training

Question 7

A healthcare company needs to predict patient readmission risk using clinical notes. Which AWS service can be used to preprocess the text data into numerical features for a machine learning model?

Accepted Answer

Amazon Comprehend. Amazon Comprehend is a natural language processing service that can extract entities, key phrases, sentiment, and topics from clinical text, and it provides APIs to convert unstructured text into structured features usable by ML models. For preprocessing clinical notes into numerical features, Comprehend's entity detection and topic modeling capabilities are the appropriate AWS service. It also supports custom classification and custom entity recognition trained on domain-specific data.

Answer

Amazon SageMaker Ground Truth

Answer

Amazon Translate

Answer

Amazon Rekognition

Question 8

A data scientist uses SageMaker Autopilot to automatically build a binary classification model. The dataset has 50 features and 100,000 rows. After the experiment, Autopilot provides multiple candidate models. Which candidate should the data scientist select to minimize inference latency for real-time predictions?

Accepted Answer

The model that is a linear learner. SageMaker Autopilot's linear learner candidate is a simple linear model that requires only a dot product between the input features and the learned coefficients, making it the fastest to evaluate at inference time. For a binary classification problem with 50 features, a linear model has just 51 parameters, so prediction latency is minimal compared to ensemble or deep learning candidates. The other options optimize for different objectives (memory, accuracy, loss) that do not directly minimize inference latency.

Answer

The model with the smallest memory footprint

Answer

The model with the highest validation accuracy

Answer

The model with the lowest validation loss

Question 9

A company uses SageMaker to train a time-series forecasting model using Amazon Forecast. The dataset contains historical sales data for 10,000 products over 2 years. Which data format is required for the target time series?

Accepted Answer

A CSV file with columns: timestamp, target_value, item_id. Amazon Forecast requires the target time series data to be in a CSV format with specific columns: timestamp, target_value, and item_id. This structured format allows the service to correctly identify the time series for each product and the target metric to forecast. The CSV format is the standard input for Forecast's built-in algorithms and ensures compatibility with the dataset import process.

Answer

A single JSON file with nested arrays

Answer

A text file with one value per line

Answer

A Parquet file partitioned by date

Question 10

Refer to the exhibit. An IAM policy is attached to a SageMaker execution role. A data scientist tries to create a training job that reads training data from s3://my-bucket/confidential/data.csv. What will happen?

Accepted Answer

The training job will fail with an access denied error. The training job will fail with an access denied error because the IAM policy attached to the SageMaker execution role likely contains a Deny statement for the S3 bucket or object, which overrides any Allow. Even if there is an Allow on my-bucket/*, an explicit Deny takes precedence.

Answer

The training job will succeed because there is an Allow on my-bucket/*

Answer

The training job will succeed because the Deny statement is invalid

Answer

The training job will fail because the role lacks sagemaker:CreateTrainingJob

Question 11

Refer to the exhibit. A SageMaker training job failed with the error shown. What is the most likely cause of this error?

Accepted Answer

The input data contains missing or invalid values. The most likely cause of a SageMaker training job failure with an error referencing data parsing or invalid values is that the input data contains missing, malformed, or invalid values (e.g., NaN, non-numeric strings in numeric columns). SageMaker built-in algorithms and many frameworks fail fast when they encounter such values during parsing or training. This is the most common data-related training failure.

Answer

The training algorithm is not compatible with the data type

Answer

The training instance type is not powerful enough

Answer

The training script has a syntax error

Question 12

Refer to the exhibit. A data scientist creates a SageMaker model using the configuration above. When deploying the model to an endpoint, the endpoint status remains 'Creating' for a long time and then fails. What is the most likely cause?

Accepted Answer

The IAM role lacks permission to pull the Docker image from ECR. The image URI points to an ECR repository in account 382416733822, which is not the customer's account. SageMaker expects the image to be in the same account or accessible via cross-account permissions. This URI is likely the AWS account for built-in algorithms, but if the region or repository is incorrect, it may fail. The most likely issue is that the image does not exist in that account or the role lacks permissions to pull it.

Answer

The S3 model artifact does not exist

Answer

The environment variable SAGEMAKER_REGION is incorrect

Answer

The model name is already in use

Question 13

A team is using Amazon SageMaker to train a deep learning model. The training job is taking too long, and they want to reduce training time without significant accuracy loss. They have already tried increasing the number of instances. Which technique should they consider next?

Accepted Answer

Gradient accumulation. Gradient accumulation allows training with a larger effective batch size by accumulating gradients over multiple mini-batches before updating weights, which improves GPU utilization and can reduce training time when scaling instances has plateaued. It enables larger effective batches without increasing memory per instance, often improving convergence speed. This makes it the appropriate next technique after horizontal scaling.

Answer

Increase L2 regularization

Answer

Reduce model complexity

Answer

Early stopping

Question 14

A machine learning engineer is deploying a model that predicts loan defaults. The model uses features like income, credit score, and debt-to-income ratio. After deployment, the model's performance degrades over time. Which concept best describes this phenomenon?

Accepted Answer

Model drift. Model drift is the correct answer because it is the overarching term that encompasses both data drift and concept drift, describing any degradation in model performance over time. In this scenario, the model's performance degrades after deployment, which is a classic sign of model drift. While data drift and concept drift are specific types, model drift is the general phenomenon.

Answer

Data drift

Answer

Concept drift

Answer

Overfitting

Question 15

A company uses Amazon SageMaker to train a model for detecting fraudulent transactions. The dataset is highly imbalanced (99.9% legitimate, 0.1% fraudulent). Which approach is most effective to address this imbalance?

Accepted Answer

Apply SMOTE to generate synthetic samples. SMOTE (Synthetic Minority Over-sampling Technique) generates synthetic samples for the minority class by interpolating between existing minority instances, which helps balance the dataset without simply duplicating data. This is particularly effective for highly imbalanced datasets like fraud detection, where the minority class is extremely small. By creating new, plausible minority samples, SMOTE reduces overfitting compared to random oversampling and improves model performance.

Answer

Use class weights in the loss function

Answer

Random oversampling of the minority class

Answer

Collect more data for the minority class