MLA-C01 ML Model Development — Timed 15 Questions

Question 1

A data scientist is training a binary classification model using imbalanced data where the positive class is only 1% of the dataset. The scientist wants to maximize the recall for the positive class while maintaining reasonable precision. Which evaluation metric is most appropriate to tune during model selection?

Accepted Answer

F1 score. The F1 score is the harmonic mean of precision and recall, making it ideal for imbalanced datasets where the positive class is only 1%. By tuning the F1 score, the data scientist directly balances the trade-off between maximizing recall (capturing true positives) and maintaining reasonable precision (avoiding false positives), which aligns with the stated goal.

Answer

Log loss

Answer

Area under the ROC curve (AUC)

Answer

Accuracy

Question 2

A machine learning engineer is training a deep learning model on SageMaker and notices that the training loss decreases rapidly in the first few epochs but then plateaus. The validation loss starts increasing after 10 epochs. Which action should the engineer take to improve generalization?

Accepted Answer

Use early stopping with validation loss monitoring. Early stopping is the correct action because the validation loss increasing after 10 epochs while training loss continues to decrease is a classic sign of overfitting. By monitoring validation loss and halting training when it stops improving (e.g., using a patience parameter), the engineer prevents the model from memorizing noise in the training data, thereby improving generalization. SageMaker's built-in training job features or the `EarlyStopping` callback in frameworks like TensorFlow or PyTorch can implement this directly.

Answer

Add more layers to the model

Answer

Increase the learning rate

Answer

Decrease the batch size

Question 3

A team is deploying a machine learning model for real-time fraud detection. The model must have inference latency under 10 ms and handle up to 1000 requests per second. The model is a gradient boosting model using XGBoost. Which SageMaker hosting configuration is MOST cost-effective while meeting the requirements?

Accepted Answer

Use a SageMaker Multi-Model Endpoint (MME) on an ml.c5.4xlarge instance with auto scaling. Option B is correct because a Multi-Model Endpoint (MME) on a single ml.c5.4xlarge instance allows multiple models to share the same endpoint, reducing cost while still meeting the latency (<10 ms) and throughput (1000 req/s) requirements. The ml.c5.4xlarge provides sufficient compute (16 vCPUs, 32 GB memory) for XGBoost inference, and auto scaling ensures capacity adjusts to handle peak load without over-provisioning.

Answer

Use SageMaker Batch Transform with multiple instances

Answer

Deploy on a single ml.c5.xlarge instance with a real-time endpoint

Answer

Deploy separate real-time endpoints for each model on ml.m5.large instances

Question 4

A data scientist is using Amazon SageMaker to train a linear regression model. After training, the scientist notices that the training and validation errors are both low, but the model performs poorly on new test data. What is the MOST likely cause?

Accepted Answer

There is data leakage from the validation set into the training set. Option A is correct because data leakage from the validation set into the training set would allow the model to learn patterns that are not present in truly unseen data, leading to artificially low training and validation errors but poor generalization to new test data. In SageMaker, this can occur if the dataset is not properly split before feature engineering or if preprocessing (e.g., scaling or imputation) is applied to the entire dataset before splitting, causing the validation set to influence the training process.

Answer

The features are not scaled properly

Answer

The model is overfitting the training data

Answer

The model has high bias

Question 5

A company is using SageMaker to train a neural network for image classification. The training job is taking too long. The team wants to reduce training time without sacrificing model accuracy. Which approach should they recommend?

Accepted Answer

Use a GPU-based instance such as ml.p3.2xlarge. Option B is correct because GPU-based instances like ml.p3.2xlarge are specifically designed for parallel processing of matrix operations, which are fundamental to neural network training. By offloading compute-intensive tensor operations to GPU cores, training time can be significantly reduced without altering the model architecture or data, thus preserving accuracy.

Answer

Increase the batch size to the maximum possible

Answer

Use a learning rate scheduler that reduces the learning rate over time

Answer

Add more convolutional layers to the model

Question 6

A machine learning engineer is using SageMaker Automatic Model Tuning (AMT) to optimize hyperparameters for a random forest model. The engineer notices that the tuning job is taking too long and many hyperparameter combinations are being evaluated but not improving the objective metric. Which action should the engineer take to make the tuning more efficient?

Accepted Answer

Enable early stopping for the tuning job. Option D is correct because enabling early stopping in SageMaker Automatic Model Tuning (AMT) terminates poorly performing training jobs before they complete, which reduces wasted compute time and speeds up the tuning process. This is especially effective when using Bayesian optimization, as it allows the algorithm to focus on promising hyperparameter regions and avoid evaluating combinations that are unlikely to improve the objective metric.

Answer

Switch the strategy from Bayesian to random search

Answer

Use a smaller instance type for each training job

Answer

Increase the maximum number of training jobs

Question 7

A team is developing a model to predict customer churn. The dataset has 10,000 samples with 20 features. The target variable is binary with 15% churn rate. The team wants to use logistic regression. Which data preprocessing step is MOST important to ensure proper convergence?

Accepted Answer

Standardize the features to have zero mean and unit variance. Logistic regression uses gradient descent or similar optimization algorithms that rely on the scale of the features. When features have different units or magnitudes, the cost function becomes elongated, causing slow or unstable convergence. Standardizing to zero mean and unit variance ensures that all features contribute equally to the gradient updates, leading to faster and more reliable convergence.

Answer

Remove correlated features to reduce multicollinearity

Answer

Impute missing values with the median

Answer

Apply SMOTE to balance the classes

Question 8

A data scientist is training a deep learning model using SageMaker and wants to use distributed training across multiple GPUs to reduce training time. Which TWO actions should the scientist take to configure distributed training? (Select TWO.)

Accepted Answer

Use the SageMaker distributed data parallelism library. The SageMaker distributed data parallelism library (option B) automatically partitions training data and synchronizes gradients across multiple GPUs, reducing training time without manual data splitting. Configuring the SageMaker estimator with a distribution parameter (option D) enables this library by specifying the distribution strategy (e.g., 'torch_distributed' or 'tensorflow_distributed'), which is required to activate distributed training.

Answer

Reduce the number of epochs to match the number of GPUs

Answer

Manually split the training data into shards and upload to S3

Answer

Set the instance count to 1 with a multi-GPU instance

Question 9

A machine learning engineer is deploying a custom PyTorch model to a SageMaker endpoint for real-time inference. The model requires GPU acceleration. The engineer wants to minimize latency and cost. Which THREE actions should the engineer take? (Select THREE.)

Accepted Answer

Compile the model with SageMaker Neo. SageMaker Neo compiles the PyTorch model into an optimized runtime binary that is specifically tuned for the target hardware (e.g., GPU instances like ml.p3). This reduces inference latency by applying graph-level optimizations, operator fusion, and memory layout transformations without changing the model's accuracy, while also lowering compute resource usage and cost.

Answer

Use an ml.c5.2xlarge instance with CPU only

Answer

Use SageMaker Batch Transform for inference

Question 10

A data scientist is building a text classification model using a pre-trained BERT model from the Hugging Face library on SageMaker. The scientist wants to fine-tune the model on a custom dataset. Which TWO steps are necessary to set up the fine-tuning job? (Select TWO.)

Accepted Answer

Use the HuggingFace estimator provided by SageMaker. Option A is correct because the SageMaker HuggingFace estimator is specifically designed to simplify fine-tuning of pre-trained Hugging Face models like BERT. It automatically handles the underlying infrastructure, including the correct PyTorch/TensorFlow and Transformers versions, without requiring custom Docker containers. This is the recommended approach for Hugging Face model fine-tuning on SageMaker.

Answer

Enable SageMaker Clarify for explainability during training

Answer

Build a custom Docker container with PyTorch and Transformers

Answer

Use SageMaker Processing to preprocess the data in parallel

Question 11

A data scientist is training a binary classification model using Amazon SageMaker. The dataset has a severe class imbalance (95% negative, 5% positive). The model achieves 99% accuracy but fails to identify positive cases correctly. Which action should the data scientist take to improve the model's ability to detect positive cases?

Accepted Answer

Use the F1 score as the evaluation metric and adjust the classification threshold based on the precision-recall curve.. Option D is correct because in a severely imbalanced dataset (95% negative, 5% positive), accuracy is misleading. The F1 score balances precision and recall, and adjusting the classification threshold based on the precision-recall curve allows the model to prioritize recall for the minority class, directly improving detection of positive cases. This approach is recommended in SageMaker when using built-in algorithms or custom models with imbalanced data.

Answer

Switch to a logistic regression model with balanced class weights.

Answer

Use accuracy as the evaluation metric and retrain the model.

Answer

Apply SMOTE (Synthetic Minority Over-sampling Technique) to the training data.

Question 12

A machine learning engineer is deploying a pre-trained NLP model on Amazon SageMaker for real-time inference. The model expects input sequences of variable length, and performance is critical. The engineer wants to minimize latency while handling the variable-length inputs efficiently. Which approach should the engineer choose?

Accepted Answer

Use dynamic batching with a custom inference script that groups requests by sequence length.. Option C is correct because dynamic batching with a custom inference script that groups requests by sequence length minimizes padding overhead and maximizes hardware utilization. By batching similar-length sequences together, the model avoids excessive padding to the maximum length in the batch, which reduces wasted computation and latency. This approach is particularly effective for variable-length NLP inputs on SageMaker, where the inference container can be customized to implement the grouping logic.

Answer

Reduce the model size by pruning and quantization.

Answer

Pad all input sequences to the maximum length in the batch.

Answer

Process each request individually to avoid padding overhead.

Question 13

A team wants to track and compare multiple machine learning experiments, including hyperparameters, metrics, and artifacts. They are using Amazon SageMaker. Which AWS service or feature should they use to achieve this?

Accepted Answer

Amazon SageMaker Experiments. Amazon SageMaker Experiments is the correct service because it is specifically designed to track and compare machine learning experiments, including hyperparameters, metrics, and artifacts. It provides a structured way to log, organize, and analyze multiple runs, enabling teams to identify the best-performing model configurations.

Answer

AWS CloudTrail

Answer

Amazon SageMaker Model Registry

Answer

Amazon SageMaker Studio

Question 14

A company is using Amazon SageMaker to train a large deep learning model. The training job is taking a very long time. The data scientist suspects that the GPU utilization is low due to inefficient data loading. Which action should the data scientist take to diagnose and address this issue?

Accepted Answer

Check GPU utilization using Amazon CloudWatch metrics, and if low, optimize the data loading pipeline by using Pipe mode or faster data formats.. Option B is correct because low GPU utilization during deep learning training often indicates a data loading bottleneck, where the GPU spends cycles waiting for data. Amazon CloudWatch provides GPU utilization metrics for SageMaker training jobs, and if utilization is low, optimizing the data pipeline with Pipe mode (streaming data directly from Amazon S3) or using faster data formats like RecordIO or TFRecord can reduce I/O overhead and keep the GPU busy.

Answer

Switch to a CPU-only instance to reduce overhead.

Answer

Reduce the batch size to speed up training.

Answer

Increase the number of GPUs in the training instance.

Question 15

An MLOps engineer is building an automated retraining pipeline for a fraud detection model. The model must be retrained weekly, and the new model should only be promoted to production if it meets predefined performance thresholds compared to the current model. Which combination of SageMaker capabilities should the engineer use?

Accepted Answer

Amazon SageMaker Pipelines and Amazon SageMaker Model Registry. Option D is correct because Amazon SageMaker Pipelines provides the orchestration for the automated retraining workflow (including weekly scheduling and conditional logic), while SageMaker Model Registry enables versioning, approval, and promotion of models based on performance thresholds. Together, they allow the engineer to define a pipeline that trains a new model, evaluates it against the current production model, and only registers it for deployment if it meets the predefined criteria.

Answer

Amazon SageMaker Debugger and Amazon SageMaker Clarify

Answer

Amazon SageMaker Model Monitor and Amazon SageMaker Ground Truth

Answer

Amazon SageMaker Autopilot and Amazon SageMaker Experiments