MLA-C01 ML Model Development — 20 Questions

Question 1

A data scientist is training a binary classification model using imbalanced data where the positive class is only 1% of the dataset. The scientist wants to maximize the recall for the positive class while maintaining reasonable precision. Which evaluation metric is most appropriate to tune during model selection?

Accepted Answer

F1 score. The F1 score is the harmonic mean of precision and recall, making it ideal for imbalanced datasets where the positive class is only 1%. By tuning the F1 score, the data scientist directly balances the trade-off between maximizing recall (capturing true positives) and maintaining reasonable precision (avoiding false positives), which aligns with the stated goal.

Answer

Log loss

Answer

Area under the ROC curve (AUC)

Answer

Accuracy

Question 2

A machine learning engineer is training a deep learning model on SageMaker and notices that the training loss decreases rapidly in the first few epochs but then plateaus. The validation loss starts increasing after 10 epochs. Which action should the engineer take to improve generalization?

Accepted Answer

Use early stopping with validation loss monitoring. Early stopping is the correct action because the validation loss increasing after 10 epochs while training loss continues to decrease is a classic sign of overfitting. By monitoring validation loss and halting training when it stops improving (e.g., using a patience parameter), the engineer prevents the model from memorizing noise in the training data, thereby improving generalization. SageMaker's built-in training job features or the `EarlyStopping` callback in frameworks like TensorFlow or PyTorch can implement this directly.

Answer

Add more layers to the model

Answer

Increase the learning rate

Answer

Decrease the batch size

Question 3

A team is deploying a machine learning model for real-time fraud detection. The model must have inference latency under 10 ms and handle up to 1000 requests per second. The model is a gradient boosting model using XGBoost. Which SageMaker hosting configuration is MOST cost-effective while meeting the requirements?

Accepted Answer

Use a SageMaker Multi-Model Endpoint (MME) on an ml.c5.4xlarge instance with auto scaling. Option B is correct because a Multi-Model Endpoint (MME) on a single ml.c5.4xlarge instance allows multiple models to share the same endpoint, reducing cost while still meeting the latency (<10 ms) and throughput (1000 req/s) requirements. The ml.c5.4xlarge provides sufficient compute (16 vCPUs, 32 GB memory) for XGBoost inference, and auto scaling ensures capacity adjusts to handle peak load without over-provisioning.

Answer

Use SageMaker Batch Transform with multiple instances

Answer

Deploy on a single ml.c5.xlarge instance with a real-time endpoint

Answer

Deploy separate real-time endpoints for each model on ml.m5.large instances

Question 4

A data scientist is using Amazon SageMaker to train a linear regression model. After training, the scientist notices that the training and validation errors are both low, but the model performs poorly on new test data. What is the MOST likely cause?

Accepted Answer

There is data leakage from the validation set into the training set. Option A is correct because data leakage from the validation set into the training set would allow the model to learn patterns that are not present in truly unseen data, leading to artificially low training and validation errors but poor generalization to new test data. In SageMaker, this can occur if the dataset is not properly split before feature engineering or if preprocessing (e.g., scaling or imputation) is applied to the entire dataset before splitting, causing the validation set to influence the training process.

Answer

The features are not scaled properly

Answer

The model is overfitting the training data

Answer

The model has high bias

Question 5

A company is using SageMaker to train a neural network for image classification. The training job is taking too long. The team wants to reduce training time without sacrificing model accuracy. Which approach should they recommend?

Accepted Answer

Use a GPU-based instance such as ml.p3.2xlarge. Option B is correct because GPU-based instances like ml.p3.2xlarge are specifically designed for parallel processing of matrix operations, which are fundamental to neural network training. By offloading compute-intensive tensor operations to GPU cores, training time can be significantly reduced without altering the model architecture or data, thus preserving accuracy.

Answer

Increase the batch size to the maximum possible

Answer

Use a learning rate scheduler that reduces the learning rate over time

Answer

Add more convolutional layers to the model

Question 6

A machine learning engineer is using SageMaker Automatic Model Tuning (AMT) to optimize hyperparameters for a random forest model. The engineer notices that the tuning job is taking too long and many hyperparameter combinations are being evaluated but not improving the objective metric. Which action should the engineer take to make the tuning more efficient?

Accepted Answer

Enable early stopping for the tuning job. Option D is correct because enabling early stopping in SageMaker Automatic Model Tuning (AMT) terminates poorly performing training jobs before they complete, which reduces wasted compute time and speeds up the tuning process. This is especially effective when using Bayesian optimization, as it allows the algorithm to focus on promising hyperparameter regions and avoid evaluating combinations that are unlikely to improve the objective metric.

Answer

Switch the strategy from Bayesian to random search

Answer

Use a smaller instance type for each training job

Answer

Increase the maximum number of training jobs

Question 7

A team is developing a model to predict customer churn. The dataset has 10,000 samples with 20 features. The target variable is binary with 15% churn rate. The team wants to use logistic regression. Which data preprocessing step is MOST important to ensure proper convergence?

Accepted Answer

Standardize the features to have zero mean and unit variance. Logistic regression uses gradient descent or similar optimization algorithms that rely on the scale of the features. When features have different units or magnitudes, the cost function becomes elongated, causing slow or unstable convergence. Standardizing to zero mean and unit variance ensures that all features contribute equally to the gradient updates, leading to faster and more reliable convergence.

Answer

Remove correlated features to reduce multicollinearity

Answer

Impute missing values with the median

Answer

Apply SMOTE to balance the classes

Question 8

A data scientist is training a deep learning model using SageMaker and wants to use distributed training across multiple GPUs to reduce training time. Which TWO actions should the scientist take to configure distributed training? (Select TWO.)

Accepted Answer

Use the SageMaker distributed data parallelism library. The SageMaker distributed data parallelism library (option B) automatically partitions training data and synchronizes gradients across multiple GPUs, reducing training time without manual data splitting. Configuring the SageMaker estimator with a distribution parameter (option D) enables this library by specifying the distribution strategy (e.g., 'torch_distributed' or 'tensorflow_distributed'), which is required to activate distributed training.

Answer

Reduce the number of epochs to match the number of GPUs

Answer

Manually split the training data into shards and upload to S3

Answer

Set the instance count to 1 with a multi-GPU instance

Question 9

A machine learning engineer is deploying a custom PyTorch model to a SageMaker endpoint for real-time inference. The model requires GPU acceleration. The engineer wants to minimize latency and cost. Which THREE actions should the engineer take? (Select THREE.)

Accepted Answer

Compile the model with SageMaker Neo. SageMaker Neo compiles the PyTorch model into an optimized runtime binary that is specifically tuned for the target hardware (e.g., GPU instances like ml.p3). This reduces inference latency by applying graph-level optimizations, operator fusion, and memory layout transformations without changing the model's accuracy, while also lowering compute resource usage and cost.

Answer

Use an ml.c5.2xlarge instance with CPU only

Answer

Use SageMaker Batch Transform for inference

Question 10

A data scientist is building a text classification model using a pre-trained BERT model from the Hugging Face library on SageMaker. The scientist wants to fine-tune the model on a custom dataset. Which TWO steps are necessary to set up the fine-tuning job? (Select TWO.)

Accepted Answer

Use the HuggingFace estimator provided by SageMaker. Option A is correct because the SageMaker HuggingFace estimator is specifically designed to simplify fine-tuning of pre-trained Hugging Face models like BERT. It automatically handles the underlying infrastructure, including the correct PyTorch/TensorFlow and Transformers versions, without requiring custom Docker containers. This is the recommended approach for Hugging Face model fine-tuning on SageMaker.

Answer

Enable SageMaker Clarify for explainability during training

Answer

Build a custom Docker container with PyTorch and Transformers

Answer

Use SageMaker Processing to preprocess the data in parallel

Question 11

A data scientist is training a binary classification model using Amazon SageMaker. The dataset has a severe class imbalance (95% negative, 5% positive). The model achieves 99% accuracy but fails to identify positive cases correctly. Which action should the data scientist take to improve the model's ability to detect positive cases?

Accepted Answer

Use the F1 score as the evaluation metric and adjust the classification threshold based on the precision-recall curve.. Option D is correct because in a severely imbalanced dataset (95% negative, 5% positive), accuracy is misleading. The F1 score balances precision and recall, and adjusting the classification threshold based on the precision-recall curve allows the model to prioritize recall for the minority class, directly improving detection of positive cases. This approach is recommended in SageMaker when using built-in algorithms or custom models with imbalanced data.

Answer

Switch to a logistic regression model with balanced class weights.

Answer

Use accuracy as the evaluation metric and retrain the model.

Answer

Apply SMOTE (Synthetic Minority Over-sampling Technique) to the training data.

Question 12

A machine learning engineer is deploying a pre-trained NLP model on Amazon SageMaker for real-time inference. The model expects input sequences of variable length, and performance is critical. The engineer wants to minimize latency while handling the variable-length inputs efficiently. Which approach should the engineer choose?

Accepted Answer

Use dynamic batching with a custom inference script that groups requests by sequence length.. Option C is correct because dynamic batching with a custom inference script that groups requests by sequence length minimizes padding overhead and maximizes hardware utilization. By batching similar-length sequences together, the model avoids excessive padding to the maximum length in the batch, which reduces wasted computation and latency. This approach is particularly effective for variable-length NLP inputs on SageMaker, where the inference container can be customized to implement the grouping logic.

Answer

Reduce the model size by pruning and quantization.

Answer

Pad all input sequences to the maximum length in the batch.

Answer

Process each request individually to avoid padding overhead.

Question 13

A team wants to track and compare multiple machine learning experiments, including hyperparameters, metrics, and artifacts. They are using Amazon SageMaker. Which AWS service or feature should they use to achieve this?

Accepted Answer

Amazon SageMaker Experiments. Amazon SageMaker Experiments is the correct service because it is specifically designed to track and compare machine learning experiments, including hyperparameters, metrics, and artifacts. It provides a structured way to log, organize, and analyze multiple runs, enabling teams to identify the best-performing model configurations.

Answer

AWS CloudTrail

Answer

Amazon SageMaker Model Registry

Answer

Amazon SageMaker Studio

Question 14

A company is using Amazon SageMaker to train a large deep learning model. The training job is taking a very long time. The data scientist suspects that the GPU utilization is low due to inefficient data loading. Which action should the data scientist take to diagnose and address this issue?

Accepted Answer

Check GPU utilization using Amazon CloudWatch metrics, and if low, optimize the data loading pipeline by using Pipe mode or faster data formats.. Option B is correct because low GPU utilization during deep learning training often indicates a data loading bottleneck, where the GPU spends cycles waiting for data. Amazon CloudWatch provides GPU utilization metrics for SageMaker training jobs, and if utilization is low, optimizing the data pipeline with Pipe mode (streaming data directly from Amazon S3) or using faster data formats like RecordIO or TFRecord can reduce I/O overhead and keep the GPU busy.

Answer

Switch to a CPU-only instance to reduce overhead.

Answer

Reduce the batch size to speed up training.

Answer

Increase the number of GPUs in the training instance.

Question 15

An MLOps engineer is building an automated retraining pipeline for a fraud detection model. The model must be retrained weekly, and the new model should only be promoted to production if it meets predefined performance thresholds compared to the current model. Which combination of SageMaker capabilities should the engineer use?

Accepted Answer

Amazon SageMaker Pipelines and Amazon SageMaker Model Registry. Option D is correct because Amazon SageMaker Pipelines provides the orchestration for the automated retraining workflow (including weekly scheduling and conditional logic), while SageMaker Model Registry enables versioning, approval, and promotion of models based on performance thresholds. Together, they allow the engineer to define a pipeline that trains a new model, evaluates it against the current production model, and only registers it for deployment if it meets the predefined criteria.

Answer

Amazon SageMaker Debugger and Amazon SageMaker Clarify

Answer

Amazon SageMaker Model Monitor and Amazon SageMaker Ground Truth

Answer

Amazon SageMaker Autopilot and Amazon SageMaker Experiments

Question 16

A data scientist is training a regression model in Amazon SageMaker. The dataset contains missing values in several features. The scientist wants to handle missing values as part of the training pipeline to ensure consistency between training and inference. Which approach should the scientist use?

Accepted Answer

Use a scikit-learn container in SageMaker to create a preprocessing step that imputes missing values and include it in the inference pipeline.. Option D is correct because it uses a scikit-learn container within SageMaker to create a preprocessing step that imputes missing values, then includes that step in the inference pipeline. This ensures the same imputation logic (e.g., mean, median, or custom strategy) is applied consistently during both training and inference, preventing data drift and maintaining reproducibility. SageMaker Pipelines or the built-in scikit-learn container allow the preprocessing to be serialized as part of the model artifact, so inference requests automatically undergo the same transformation.

Answer

Impute missing values in a separate Jupyter notebook and save the cleaned data.

Answer

Use SageMaker Autopilot to automatically handle missing values.

Answer

Drop all rows with missing values before training.

Question 17

A machine learning team is using Amazon SageMaker to train a model. They notice that the training job is taking longer than expected and the logs show repeated warnings about 'loss not decreasing'. Which SageMaker feature should they use to diagnose and visualize the training process?

Accepted Answer

Amazon SageMaker Debugger. Amazon SageMaker Debugger is the correct choice because it provides real-time monitoring and visualization of training metrics, including loss values, gradients, and weights. The repeated 'loss not decreasing' warnings indicate a training issue (e.g., vanishing gradients or learning rate problems), and Debugger can capture these tensors and emit alerts or trigger actions (like stopping the job) via built-in or custom rules. It also integrates with SageMaker Studio for interactive visualization of the training progress.

Answer

Amazon SageMaker Clarify

Answer

Amazon SageMaker Experiments

Answer

Amazon SageMaker Model Monitor

Question 18

A data scientist is building a text classification model using Amazon SageMaker. The dataset is stored as a CSV file in Amazon S3. The scientist wants to use the SageMaker built-in BlazingText algorithm. Which of the following steps are required to prepare the data for training? (Choose TWO.)

Accepted Answer

Convert the CSV file to the format of a single file with one instance per line.. Option C is correct because BlazingText expects input data in a single file where each line represents one training instance. This is a specific requirement of the algorithm's input format, not a general SageMaker practice. The CSV file must be converted to this line-per-instance format for BlazingText to process it correctly.

Answer

Convert the text to one-hot encoded vectors.

Answer

Tokenize and remove stop words from the text.

Answer

Upload the data to an Amazon SageMaker notebook instance.

Question 19

An MLOps team is designing a CI/CD pipeline for deploying machine learning models to production on Amazon SageMaker. They want to ensure that the deployment process is automated and that models are automatically rolled back if performance degrades. Which of the following AWS services or features should they use to achieve this? (Choose THREE.)

Accepted Answer

Amazon SageMaker Model Registry. Amazon SageMaker Model Registry is correct because it provides a centralized catalog for managing, versioning, and approving ML models. It enables automated deployment by triggering CI/CD pipelines when a model version is approved, and supports automatic rollback by allowing you to revert to a previous approved version if performance degrades, as detected by monitoring metrics.

Answer

Amazon SageMaker Ground Truth

Answer

AWS CloudTrail

Question 20

A financial services company is deploying a real-time fraud detection model using Amazon SageMaker. The model is a gradient boosting model (XGBoost) trained on historical transaction data. The inference endpoint uses an ml.m5.2xlarge instance with a single variant. Recently, the company has experienced a 3x increase in transaction volume during peak hours, causing inference latency to exceed the 200ms SLA. The data science team has already optimized the model by reducing the number of trees and feature set, but the latency remains high during spikes. The team considers using SageMaker's built-in scaling policies. They currently have a single endpoint with one production variant. The team wants to maintain low latency without over-provisioning resources. They have ruled out model changes. Which approach should the team take?

Accepted Answer

Configure an Application Auto Scaling target tracking scaling policy for the variant based on the 'SageMakerVariantInvocationsPerInstance' metric, with a target value that keeps the inference latency within the SLA.. Option A is correct because SageMaker's built-in target tracking scaling policy using the 'SageMakerVariantInvocationsPerInstance' metric allows the endpoint to automatically adjust the instance count based on real-time invocation load. By setting a target value that correlates with the 200ms SLA, the policy dynamically scales out during traffic spikes and scales in during lulls, preventing over-provisioning while maintaining low latency. This approach directly addresses the 3x peak-hour volume increase without requiring manual intervention or model changes.

Answer

Deploy the model on multiple endpoints behind an Application Load Balancer.

Answer

Use scheduled scaling to increase the instance count during known peak hours.

Answer

Manually increase the instance count during peak hours.