MLA-C01 ML Model Development Practice Test 8 — 15 Questions

Question 1

A company is training a large NLP model on SageMaker and wants to reduce costs by using Spot Instances. Which TWO configurations should they implement to handle Spot interruptions gracefully?

Accepted Answer

Set `use_spot_instances=True` and `max_wait` in the estimator. Checkpointing saves progress so training can resume from the last checkpoint. Managed Spot Training with `use_spot_instances=True` automates handling of interruptions. Using a single instance or increasing max runtime does not handle interruptions; `keep_alive_period` is for persistent notebooks, not training.

Answer

Use a single large instance to reduce interruption probability

Answer

Increase the `max_run` parameter to allow longer training

Answer

Use `keep_alive_period` to keep the instance alive after training

Question 2

A data scientist is evaluating a binary classification model for loan default prediction. Which THREE metrics should they consider to thoroughly assess model performance, especially for imbalanced classes?

Accepted Answer

Recall. For imbalanced classification, accuracy can be misleading. AUC (Area under ROC curve) is robust to imbalance, F1 balances precision and recall, and recall (true positive rate) is important to catch defaults. RMSE and R² are for regression, NDCG is for ranking.

Answer

R²

Answer

RMSE

Question 3

A company wants to use SageMaker to fine-tune a foundation model for a text generation task using RLHF (Reinforcement Learning from Human Feedback). Which THREE components are required in the RLHF pipeline?

Accepted Answer

A pre-trained base model. RLHF typically requires: a pre-trained base model to start, a reward model trained on human preferences, and a reinforcement learning algorithm (like PPO) to update the base model. A LoRA adapter is optional but not required. A classifier is not the same as a reward model.

Answer

A LoRA adapter for parameter-efficient fine-tuning

Answer

A classifier to distinguish generated text from real text

Question 4

A data scientist is using SageMaker built-in XGBoost algorithm for a regression problem. Which metric is most appropriate as the objective metric for hyperparameter tuning?

Accepted Answer

RMSE. For regression tasks, RMSE is a common objective metric. AUC is for classification, F1 is for classification, and NDCG is for ranking.

Answer

NDCG

Answer

AUC

Answer

F1

Question 5

A team is training a large language model using PyTorch on SageMaker. They need to reduce training time. The model has 10 billion parameters. Which distributed training strategy should they use?

Accepted Answer

Model parallelism with SageMaker distributed. For large models that do not fit into GPU memory, model parallelism is required. Data parallelism replicates the model on each GPU, which would cause out-of-memory errors.

Answer

Data parallelism with Horovod

Answer

Single GPU training

Answer

Use a larger instance type without parallelism

Question 6

During a SageMaker training job, the loss stops decreasing and the validation accuracy plateaus early. SageMaker Debugger rules are enabled. Which rule is MOST likely to identify this issue?

Accepted Answer

Overfit rule. The overfit rule detects when validation accuracy plateaus or decreases while training accuracy continues to improve, which is a sign of overfitting. Exploding gradients detects gradient spikes, dead relu detects dead neurons, and weight distribution checks weight distributions but not directly overfitting.

Answer

Weight distribution rule

Answer

Exploding gradients rule

Answer

Dead relu rule

Question 7

A company wants to use SageMaker Autopilot to automatically build a binary classification model. Which output does Autopilot provide to help understand model decisions?

Accepted Answer

An explainability report with feature importance. SageMaker Autopilot generates an explainability report with feature importance. It does not provide a confusion matrix by default; users must evaluate separately. It does not provide SHAP values directly but uses similar techniques. Leaderboard is for ranking trials, not explainability.

Answer

A leaderboard of models with only accuracy metrics

Answer

A confusion matrix for each candidate model

Answer

A SHAP values summary plot for each trial

Question 8

A data scientist is using SageMaker Experiments to track multiple training runs. They want to compare the F1 scores across runs. Which component should they use to log the F1 score?

Accepted Answer

Metric. In SageMaker Experiments, metrics are logged using the SageMaker SDK's log_metric method or by reporting through the training job's metric definitions. Hyperparameters are logged separately. Artifacts are for model files or datasets.

Answer

Parameter

Answer

Hyperparameter

Answer

Artifact

Question 9

Which SageMaker built-in algorithm is designed for time series forecasting?

Accepted Answer

DeepAR. DeepAR is a built-in algorithm specifically for time series forecasting. BlazingText is for text, XGBoost is for tabular data, and IP Insights is for anomaly detection in IP traffic.

Answer

BlazingText

Answer

IP Insights

Answer

XGBoost

Question 10

A team is fine-tuning a foundation model using LoRA. They want to reduce memory usage during training. Which technique should they combine LoRA with to further reduce memory?

Accepted Answer

QLoRA. QLoRA combines LoRA with quantization (e.g., 4-bit) to drastically reduce memory. Instruction tuning is a method, not a memory reduction technique. RLHF is a training process. Pruning reduces model size but is not typically combined with LoRA in this context.

Answer

Instruction tuning

Answer

Pruning

Answer

RLHF

Question 11

A data scientist is training a model using SageMaker and wants to use spot instances to reduce costs. The training job is checkpointed every 5 minutes. However, the job gets interrupted frequently and never completes. What is the MOST likely cause?

Accepted Answer

The checkpoint interval is too long relative to the interruption frequency. Spot instances can be reclaimed with little notice. If the job checkpoint interval is longer than the average interruption notice, progress may be lost. Using a smaller instance type reduces cost but not interruption frequency. Incorrect checkpoint path causes save failures. Too few max retries causes job to stop after few interruptions.

Answer

The checkpoint S3 URI is incorrect

Answer

The instance type is too small for the training job

Answer

The job is configured with too few max retries

Question 12

Which SageMaker feature allows you to automatically tune hyperparameters using Bayesian optimization?

Accepted Answer

SageMaker Automatic Model Tuning. SageMaker Automatic Model Tuning (AMT) supports Bayesian optimization, random search, and Hyperband. Debugger is for monitoring. Experiments is for tracking. Autopilot is for AutoML.

Answer

SageMaker Autopilot

Answer

SageMaker Experiments

Answer

SageMaker Debugger

Question 13

A company uses SageMaker Clarify to detect bias in their training data. They find that the model has a high disparate impact for a protected attribute. What should they do to mitigate this bias during training?

Accepted Answer

Preprocess the data using techniques like reweighing or resampling to reduce bias. SageMaker Clarify can generate bias reports, but mitigation techniques like reweighing or using bias-aware algorithms are applied separately. Adjusting the threshold does not address training bias. Removing the attribute may not eliminate indirect bias. Using a different algorithm may help but is not the direct mitigation step from Clarify.

Answer

Use SageMaker Clarify’s built-in bias mitigation algorithm during training

Answer

Remove the protected attribute from the dataset

Answer

Increase the model complexity to capture more patterns

Question 14

A data scientist is using SageMaker built-in Image Classification algorithm on a dataset with 1000 classes. The training is very slow. They want to speed it up without sacrificing accuracy. Which instance type and training configuration is MOST appropriate?

Accepted Answer

Use ml.p3.16xlarge instances with data parallelism. For image classification, GPU instances like ml.p3 or ml.g4dn are suitable. ml.p3.16xlarge provides 8 V100 GPUs. ml.m5 is CPU only. ml.c5 is CPU. ml.trn1 is for training, but for this built-in algorithm, GPU instances are standard.

Answer

Use ml.trn1.32xlarge instances with data parallelism

Answer

Use ml.m5.24xlarge instances with data parallelism

Answer

Use ml.c5.18xlarge instances with model parallelism

Question 15

A data scientist is using SageMaker Automatic Model Tuning with Hyperband. They want to stop poorly performing trials early to save resources. Which strategy does Hyperband use?

Accepted Answer

Successive Halving. Hyperband uses early stopping by allocating resources to promising configurations and stopping poorly performing ones. Bayesian optimization uses acquisition functions. Random search does not early stop. Grid search exhaustively evaluates all combinations.

Answer

Grid search

Answer

Random search

Answer

Bayesian optimization