MLA-C01 ML Model Development Practice Test 6 — 15 Questions

Question 1

A data scientist is using SageMaker to train a custom PyTorch model for image classification. They want to use SageMaker Debugger to detect training issues. Which TWO built-in rules are most relevant for detecting common training problems? (Select TWO.)

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Overfit. ExplodingGradients detects gradients becoming too large, and Overfit detects when validation loss diverges from training loss. Both are common issues.

Answer

DataDistribution

Answer

ImageQuality

Answer

ConfusionMatrix

Question 2

A company is fine-tuning a foundation model using RLHF (Reinforcement Learning from Human Feedback) on SageMaker. They want to reduce memory usage and training time. Which THREE techniques should they consider? (Select THREE.)

Accepted Answer

Use a smaller foundation model (e.g., 7B instead of 70B parameters). LoRA/QLoRA reduces trainable parameters, PPO is the standard RLHF algorithm, and using smaller foundation models reduces memory and compute requirements.

Answer

Use SageMaker Data Parallelism with sharded data

Answer

Use full fine-tuning on a larger instance

Question 3

A data scientist wants to use SageMaker Clarify to analyze bias during training of a binary classification model. Which TWO types of bias metrics can SageMaker Clarify compute? (Select TWO.)

Accepted Answer

Post-training bias metrics (e.g., Difference in Positive Proportions, AD). SageMaker Clarify computes pre-training bias (e.g., class imbalance) and post-training bias (e.g., difference in positive proportions across groups).

Answer

Feature importance

Answer

SHAP values

Answer

Confusion matrix

Question 4

A company wants to build a customer service chatbot that answers questions about their internal policy documents. The documents are updated monthly, and the team cannot afford to retrain a model each time. Which approach is MOST appropriate?

Accepted Answer

Use Retrieval-Augmented Generation (RAG) with the policy documents indexed in a vector store. RAG (Retrieval-Augmented Generation) allows the LLM to retrieve relevant document sections at inference time, so knowledge stays current without retraining. The other options either require expensive retraining for each update or lack document grounding.

Answer

Use a larger foundation model with a longer context window and paste all documents into each prompt

Answer

Fine-tune a base LLM on the policy documents monthly

Answer

Train a custom model from scratch on the policy documents each month

Question 5

A data scientist is training an XGBoost model on a large tabular dataset using SageMaker. The training job is taking too long. The scientist wants to reduce training time while maintaining model quality. Which action should the scientist take?

Accepted Answer

Use SageMaker distributed data parallelism across multiple instances. Using SageMaker's managed spot training can significantly reduce cost, but it may cause interruptions. The best approach to reduce training time is to use distributed data parallelism with multiple instances. Increasing instance type can also speed up training, but distributed training is more scalable. Using Hyperband is for hyperparameter tuning, not for reducing training time directly. Converting to a different algorithm is not necessary.

Answer

Enable SageMaker managed spot training

Answer

Switch to Hyperband for hyperparameter tuning

Answer

Convert the XGBoost model to a Linear Learner model

Question 6

An ML engineer is fine-tuning a large language model using LoRA on SageMaker. The training is converging slowly, and GPU utilization is low. The engineer suspects the bottleneck is data loading. Which action should the engineer take to improve GPU utilization?

Accepted Answer

Use SageMaker Pipe mode to stream data from S3 directly to the training instances. Low GPU utilization during training is often due to a data pipeline bottleneck. Using SageMaker Pipe mode streams data directly from S3, reducing I/O wait times. Increasing batch size may improve utilization but can cause OOM. Using spot instances and saving checkpoints helps with interruptions but not utilization. Reducing model parallelism may help if communication is the bottleneck, but the scenario suggests data loading.

Answer

Increase the batch size to maximize GPU memory usage

Answer

Enable checkpointing and use spot instances

Answer

Reduce model parallelism to decrease communication overhead

Question 7

Which SageMaker built-in algorithm is specifically designed for time series forecasting?

Accepted Answer

DeepAR. DeepAR is a supervised learning algorithm for forecasting scalar time series using recurrent neural networks. The other algorithms are for different tasks: XGBoost for classification/regression, BlazingText for NLP, and Image Classification for computer vision.

Answer

Image Classification

Answer

BlazingText

Answer

XGBoost

Question 8

A team is training a PyTorch model using SageMaker with a custom training script. They want to track hyperparameters and metrics across multiple experiments. Which service should they use?

Accepted Answer

SageMaker Experiments. SageMaker Experiments is the native service for tracking machine learning experiments, including hyperparameters and metrics. SageMaker Debugger is for debugging training jobs. SageMaker Model Monitor is for inference monitoring. SageMaker Clarify is for bias analysis.

Answer

SageMaker Clarify

Answer

SageMaker Model Monitor

Answer

SageMaker Debugger

Question 9

A company is training a large Transformer model on SageMaker and wants to use model parallelism to fit the model into memory. The model has 10 billion parameters. Which instance type is MOST cost-effective for this task while supporting SageMaker's model parallelism?

Accepted Answer

ml.p4d.24xlarge. The ml.p4d.24xlarge instances are optimized for large-scale distributed training with high memory and support SageMaker's model parallelism. ml.trn1 instances are designed for training with AWS Trainium, but they use a different chip architecture and may require specific SDKs. ml.g4dn instances are for inference and light training. ml.c5 instances are compute-optimized but lack GPU memory for large models.

Answer

ml.trn1.32xlarge

Answer

ml.c5.18xlarge

Answer

ml.g4dn.12xlarge

Question 10

Which SageMaker built-in algorithm is best suited for detecting anomalous login attempts based on IP addresses and user behavior?

Accepted Answer

IP Insights. IP Insights is a built-in algorithm for learning IP address usage patterns and detecting anomalous behavior. The other algorithms are for different purposes: XGBoost for classification, K-Means for clustering, and PCA for dimensionality reduction.

Answer

XGBoost

Answer

PCA

Answer

K-Means

Question 11

A data scientist is using SageMaker Autopilot to automatically build a binary classification model. The dataset is imbalanced. Which action will Autopilot take by default to address class imbalance?

Accepted Answer

Use class balancing weights. Autopilot automatically applies techniques to handle imbalanced data, such as class balancing weights, when it detects imbalance. It does not require manual configuration. Ensemble selection is part of Autopilot but not specifically for imbalance. SMOTE and undersampling are not built-in defaults.

Answer

Perform random undersampling of the majority class

Answer

Ignore the imbalance and proceed with raw data

Answer

Apply SMOTE oversampling

Question 12

An ML engineer is debugging a training job that is consistently failing due to an out-of-memory error. The engineer is using SageMaker's built-in XGBoost algorithm. Which Debugger rule can help identify the issue?

Accepted Answer

Exploding gradients. The 'Exploding gradients' rule detects when gradients become too large, which is a common cause of training instability but not necessarily OOM. The 'Overfit' rule detects overfitting. The 'Dead relu' rule is for ReLU activation. None of these directly address OOM. However, Debugger does not have a specific OOM rule; instead, the engineer should monitor memory utilization via CloudWatch or adjust instance type. Among the options, 'Exploding gradients' is the most relevant because large gradients can lead to memory spikes.

Answer

Overfit

Answer

Dead relu

Answer

OOM rule

Question 13

A team is fine-tuning a Hugging Face BERT model for text classification using SageMaker. They want to use the Hugging Face estimator for convenience. Which parameter must be set to use a custom training script?

Accepted Answer

entry_point. The entry_point parameter specifies the path to the training script. The instance_type is for hardware selection. The hyperparameters dictionary passes parameters to the script. The framework_version specifies the Hugging Face version.

Answer

framework_version

Answer

instance_type

Answer

hyperparameters

Question 14

An ML team is using SageMaker Automatic Model Tuning to optimize hyperparameters for a neural network. They want to prioritize exploration of the hyperparameter space early in the tuning process. Which strategy should they choose?

Accepted Answer

Bayesian optimization. Bayesian optimization balances exploration and exploitation, but early in the process it tends to explore more. Random search explores uniformly without adaptation. Hyperband focuses on early stopping. Grid search is exhaustive. Bayesian optimization is the best choice for systematic exploration.

Answer

Grid search

Answer

Random search

Answer

Hyperband

Question 15

Which SageMaker feature automatically generates model cards, feature importance, and bias reports without requiring manual coding?

Accepted Answer

SageMaker Autopilot. SageMaker Clarify provides bias detection and feature importance, and it can generate reports. SageMaker Autopilot generates model cards and explanations. SageMaker Experiments tracks experiments. SageMaker Model Monitor is for monitoring. Autopilot is the correct answer because it automates the entire pipeline including model cards and explanations.

Answer

SageMaker Experiments

Answer

SageMaker Clarify

Answer

SageMaker Model Monitor