MLA-C01 ML Model Development Practice Test 7 — 15 Questions

Question 1

A data scientist needs to train a binary classification model on a large tabular dataset stored in Amazon S3. The team wants to minimize training time and cost while using a built-in SageMaker algorithm. Which algorithm should they use?

Accepted Answer

Linear Learner. Linear Learner is a built-in SageMaker algorithm designed for binary classification and regression, and it scales efficiently on large datasets. XGBoost is better for structured data with non-linear relationships, DeepAR is for time series, and BlazingText is for text.

Answer

BlazingText

Answer

DeepAR

Answer

XGBoost

Question 2

A team is training a large deep learning model on SageMaker using a single ml.p3.16xlarge instance. Training is taking too long. They want to reduce time by distributing across multiple GPUs but are constrained by model size that does not fit in a single GPU memory. Which distributed training strategy should they use?

Accepted Answer

Model parallelism using SageMaker distributed model parallelism. Model parallelism splits the model across multiple GPUs, which is needed when the model does not fit in a single GPU. Data parallelism replicates the model on each GPU and splits data, which requires the model to fit in each GPU's memory.

Answer

Data parallelism using SageMaker distributed data parallelism

Answer

Switch to a smaller instance type and use horizontal scaling

Answer

Use multiple training jobs with hyperparameter tuning

Question 3

A company is using SageMaker Automatic Model Tuning to optimize a regression model. They want to minimize the root mean squared error (RMSE). The tuner has completed 20 jobs, and the RMSE has plateaued. Which action should the data scientist take to potentially improve the results?

Accepted Answer

Decrease the range of hyperparameters to focus on promising areas. Reducing the search space can help the tuner focus on more promising regions. Increasing parallelism or max jobs may explore the same plateau, while switching to a different algorithm altogether might not be necessary.

Answer

Increase the maximum number of training jobs

Answer

Increase the number of parallel training jobs

Answer

Switch the objective metric to mean absolute error (MAE)

Question 4

A machine learning engineer is using Amazon SageMaker Debugger to monitor a training job for a deep neural network. They receive a rule alert indicating 'exploding gradients'. Which action should they take to address this issue?

Accepted Answer

Reduce the learning rate. Exploding gradients occur when gradients become too large, causing instability. Reducing the learning rate mitigates this. Increasing batch size can also help by smoothing gradients, but reducing learning rate is a direct solution.

Answer

Use a smaller batch size

Answer

Increase the number of layers to absorb gradients

Answer

Increase the learning rate

Question 5

A data scientist is using SageMaker Autopilot to automatically build a binary classification model on a balanced dataset. They want to understand the relationship between the input features and the model predictions. Which feature in SageMaker Autopilot should they use?

Accepted Answer

Explainability reports. SageMaker Autopilot generates explainability reports, including feature importance and model insights, via the 'Explainability' feature. This provides the relationship between features and predictions.

Answer

Data visualizations

Answer

Model tuning results

Answer

Model candidate definitions

Question 6

A team wants to fine-tune a pre-trained Hugging Face transformer model for text classification using SageMaker. They have a custom training script. Which SageMaker estimator should they use?

Accepted Answer

SageMaker Hugging Face estimator. The Hugging Face estimator is the recommended way to run Hugging Face models on SageMaker, as it automatically handles the environment and dependencies.

Answer

SageMaker generic estimator with a custom container

Answer

SageMaker PyTorch estimator

Answer

SageMaker TensorFlow estimator

Question 7

A financial services company trains multiple models on SageMaker and needs to track hyperparameters, metrics, and artifacts for each experiment. Which SageMaker feature should they use to organize and compare experiments?

Accepted Answer

SageMaker Experiments. SageMaker Experiments provides experiment management, allowing users to track parameters, metrics, and artifacts, and compare runs. SageMaker Studio offers an interface but the core feature is Experiments.

Answer

SageMaker Model Registry

Answer

SageMaker Pipelines

Answer

SageMaker Debugger

Question 8

A company is fine-tuning a large language model using LoRA on SageMaker. They want to reduce GPU memory usage during training. Which configuration change would help?

Accepted Answer

Use QLoRA (quantized LoRA) with 4-bit quantization. LoRA reduces trainable parameters, and when combined with QLoRA (quantized LoRA), it further reduces memory by quantizing the base model to 4-bit or 8-bit. Increasing batch size or sequence length typically increases memory usage. Gradient accumulation also increases memory as it requires storing gradients for multiple steps. QLoRA is specifically designed for memory reduction.

Answer

Enable gradient accumulation

Answer

Increase the sequence length

Answer

Increase the batch size

Question 9

A data scientist is using SageMaker to train an XGBoost model for a regression problem. After training, they evaluate the model on a test set and get an RMSE of 10 and an R² of 0.85. Which additional metric would give the MOST insight into the model's average prediction error magnitude?

Accepted Answer

Mean Absolute Error (MAE). MAE (Mean Absolute Error) gives the average absolute prediction error, which is easy to interpret in the same units as the target. RMSE gives a similar but squared metric, and R² indicates variance explained, but MAE directly answers the average error magnitude.

Answer

AUC

Answer

Confusion matrix

Answer

F1 score

Question 10

A company wants to detect anomalies in login events from a large user base, focusing on unusual patterns that may indicate compromised accounts. Which SageMaker built-in algorithm is most suitable for this task?

Accepted Answer

IP Insights. IP Insights is designed for anomaly detection in IP address usage, learning typical login patterns and flagging unusual ones. The other algorithms are not specialized for this use case.

Answer

K-Means

Answer

DeepAR

Answer

Factorisation Machines

Question 11

A team is using SageMaker to train a distributed model with data parallelism. They notice that the training loss is not decreasing as expected and suspect a bug in the data loading pipeline. Which SageMaker Debugger feature can help them inspect the data distributions during training?

Accepted Answer

Custom rules to monitor input tensors. SageMaker Debugger can capture tensors (including inputs and outputs) during training. By saving input tensors, the team can inspect data distributions. Rules fire on issues like overfitting or dead relu, but tensor captures allow direct data inspection. SaveConfig defines which tensors to save. Using a different instance type does not help debug data issues.

Answer

SageMaker Model Monitor

Answer

Built-in rules such as overfit detection

Answer

SageMaker Processing jobs

Question 12

A data scientist needs to run a hyperparameter tuning job for a PyTorch model using SageMaker. They want to use Hyperband for efficient resource allocation. Which tuning strategy should they select in the HyperparameterTuner?

Accepted Answer

Hyperband. SageMaker Automatic Model Tuner supports Bayesian, Random, and Hyperband strategies. Hyperband is an early stopping-based method that allocates resources adaptively. The 'Hyperband' strategy should be selected explicitly.

Answer

Bayesian optimization

Answer

Random search

Answer

Grid search

Question 13

A company is training a large NLP model on SageMaker and wants to reduce costs by using Spot Instances. Which TWO configurations should they implement to handle Spot interruptions gracefully?

Accepted Answer

Set `use_spot_instances=True` and `max_wait` in the estimator. Checkpointing saves progress so training can resume from the last checkpoint. Managed Spot Training with `use_spot_instances=True` automates handling of interruptions. Using a single instance or increasing max runtime does not handle interruptions; `keep_alive_period` is for persistent notebooks, not training.

Answer

Use a single large instance to reduce interruption probability

Answer

Increase the `max_run` parameter to allow longer training

Answer

Use `keep_alive_period` to keep the instance alive after training

Question 14

A data scientist is evaluating a binary classification model for loan default prediction. Which THREE metrics should they consider to thoroughly assess model performance, especially for imbalanced classes?

Accepted Answer

Recall. For imbalanced classification, accuracy can be misleading. AUC (Area under ROC curve) is robust to imbalance, F1 balances precision and recall, and recall (true positive rate) is important to catch defaults. RMSE and R² are for regression, NDCG is for ranking.

Answer

R²

Answer

RMSE

Question 15

A company wants to use SageMaker to fine-tune a foundation model for a text generation task using RLHF (Reinforcement Learning from Human Feedback). Which THREE components are required in the RLHF pipeline?

Accepted Answer

A pre-trained base model. RLHF typically requires: a pre-trained base model to start, a reward model trained on human preferences, and a reinforcement learning algorithm (like PPO) to update the base model. A LoRA adapter is optional but not required. A classifier is not the same as a reward model.

Answer

A LoRA adapter for parameter-efficient fine-tuning

Answer

A classifier to distinguish generated text from real text