MLA-C01 ML Model Development Practice Test 8 — 15 Questions

Question 1

A team is training a PyTorch model using SageMaker and wants to use their own custom training container with a specific PyTorch version. Which approach should they use?

Accepted Answer

Use SageMaker Bring Your Own Container (BYOC) with a custom Docker image. When a team needs a custom training container with a specific PyTorch version or custom dependencies that are not available in the built-in SageMaker images, the correct approach is Bring Your Own Container (BYOC), where they build a Docker image and push it to Amazon ECR, then reference it in the SageMaker estimator. This gives full control over the framework version and environment.

Answer

Use the SageMaker built-in PyTorch estimator and set the framework_version

Answer

Use SageMaker Script Mode with a PyTorch script

Answer

Use SageMaker Autopilot to automatically select the container

Question 2

A data scientist trains a binary classification model using SageMaker and obtains an AUC of 0.95 on the test set. However, the precision-recall curve shows low precision for high recall thresholds. The business requires a model that performs well on the minority class. Which metric should the team primarily optimize during hyperparameter tuning?

Accepted Answer

F1-score on the validation set. The F1-score is the harmonic mean of precision and recall, so it directly penalizes models that achieve high recall at the cost of low precision — exactly the failure mode described. Since the business cares about the minority class, optimizing F1 during hyperparameter tuning forces the model to balance both false positives and false negatives on that class. AUC-ROC can remain deceptively high (0.95) even when precision collapses at high recall because it aggregates performance across all thresholds and is dominated by the majority class.

Answer

Accuracy

Answer

AUC (Area Under the ROC Curve)

Answer

Log loss

Question 3

Which SageMaker built-in algorithm should be used for forecasting time series data with seasonal patterns?

Accepted Answer

DeepAR. DeepAR is a SageMaker built-in algorithm specifically designed for time series forecasting using recurrent neural networks (RNNs). It excels at handling complex seasonal patterns and can incorporate additional features, making it ideal for forecasting time series data with seasonality. It is the only built-in algorithm among the options that is purpose-built for time series forecasting.

Answer

IP Insights

Answer

BlazingText

Answer

Factorization Machines

Question 4

A data scientist suspects that a deep learning model is overfitting. They enable SageMaker Debugger and want to detect overfitting automatically. Which built-in rule should they use?

Accepted Answer

Overfit. SageMaker Debugger includes a built-in rule called Overfit that monitors the gap between training and validation loss (or accuracy) and triggers when the model begins to overfit. It is the direct, purpose-built rule for automatic overfitting detection. Enabling it requires the training script to emit both training and validation metrics via the debugger hook.

Answer

ExplodingGradients

Answer

PoorWeightInitialization

Answer

DeadRelu

Question 5

A company uses SageMaker Autopilot to build a binary classification model. The generated leaderboard shows an ensemble model as the best candidate. The team needs a model that can be deployed for real-time inference with latency < 10ms. What should they do?

Accepted Answer

Select the best single model from the leaderboard (non-ensemble candidate) and deploy it. Ensemble models in SageMaker Autopilot combine multiple base models (e.g., stacking or voting), which increases inference latency because every base model must run and their outputs aggregated. For a strict <10ms real-time latency requirement, the best approach is to select the best-performing single (non-ensemble) model from the leaderboard, which has lower inference overhead. Autopilot's leaderboard explicitly lists both ensemble and individual model candidates, so the team can pick a single model that meets the latency SLA.

Answer

Use SageMaker Inference Recommender to profile the ensemble model and optimize it

Answer

Deploy the ensemble model as a SageMaker endpoint; ensemble models are optimized for low latency

Answer

Retrain the ensemble model with fewer base estimators using a custom container

Question 6

Which SageMaker feature provides AutoML capabilities, including automatic data preprocessing, model selection, and hyperparameter tuning?

Accepted Answer

SageMaker Autopilot. SageMaker Autopilot is the only SageMaker feature that provides end-to-end AutoML: it automatically inspects and preprocesses raw tabular data, explores candidate algorithms, performs hyperparameter tuning, and selects the best model. It also generates explainability reports and notebooks so users can understand and reproduce the pipeline. The other options are narrower tools that address only one part of the ML lifecycle.

Answer

SageMaker Data Wrangler

Answer

SageMaker Automatic Model Tuning

Answer

SageMaker Experiments

Question 7

A machine learning engineer is training a TensorFlow model using SageMaker with distributed training. They need to implement data parallelism across multiple GPUs. Which SageMaker feature should they use to distribute the training?

Accepted Answer

SageMaker Distributed Data Parallelism. SageMaker Distributed Data Parallelism is the correct choice because it is purpose-built to distribute training across multiple GPUs by replicating the model on each GPU and splitting the input data across them. It uses the AllReduce algorithm with custom communication optimizations (e.g., balanced fusion of gradients) to synchronize gradients efficiently, reducing communication overhead compared to standard Horovod or torch.distributed. This feature is integrated into the SageMaker training toolkit and works with TensorFlow, PyTorch, and MXNet, making it the right tool for data parallelism.

Answer

SageMaker Automatic Model Tuning

Answer

SageMaker Debugger

Answer

SageMaker Model Parallelism

Question 8

A data scientist is using SageMaker to train a custom PyTorch model for image classification. They want to use SageMaker Debugger to detect training issues. Which TWO built-in rules are most relevant for detecting common training problems? (Select TWO.)

Accepted Answer

Overfit. Option B (Overfit) is correct because SageMaker Debugger's built-in Overfit rule monitors the gap between training and validation loss across steps and raises an issue when validation loss stops improving while training loss keeps decreasing, which is the classic signature of overfitting in a PyTorch image-classification job. Option C (ExplodingGradients) is correct because the ExplodingGradients rule inspects the gradients tensor emitted by the framework and flags abnormally large gradient values or spikes, which cause unstable or diverging training and are a common problem in deep networks. Option A (DataDistribution) is not a built-in Debugger rule for detecting training problems; it relates to SageMaker Clarify/Model Monitor data and bias analysis rather than Debugger's training-issue rule set. Option D (ImageQuality) is not a SageMaker Debugger built-in rule; image-quality checks would be a custom preprocessing concern, not a Debugger rule. Option E (ConfusionMatrix) is not a Debugger training-issue rule; confusion matrices are evaluation artifacts computed after training (for example with SageMaker Clarify or custom code), not a built-in Debugger rule for detecting training problems.

Answer

DataDistribution

Answer

ImageQuality

Answer

ConfusionMatrix

Question 9

A data scientist wants to use SageMaker Clarify to analyze bias during training of a binary classification model. Which TWO types of bias metrics can SageMaker Clarify compute? (Select TWO.)

Accepted Answer

Post-training bias metrics (e.g., Difference in Positive Proportions, AD). SageMaker Clarify computes bias metrics in two distinct phases of the ML lifecycle, and both are valid answers here. Option B is correct because post-training bias metrics evaluate the model's predictions after training, using measures such as Difference in Positive Proportions in Predicted Labels (DPPL), Accuracy Difference (AD), and Disparate Impact, which quantify whether outcomes differ across groups. Option D is correct because pre-training bias metrics analyze the training data before any model is built, using measures such as Class Imbalance (CI), Difference in Proportions of Labels (DPL), and Label Imbalance to detect skew in the dataset itself. Option A is not a bias metric category — feature importance (e.g., via SHAP) explains which features drive predictions, not whether outcomes are biased. Option C is likewise an explainability technique, not a bias metric, and Clarify reports SHAP values under its explainability analysis. Option E is a standard model evaluation artifact for classification performance, not a bias metric computed by Clarify.

Answer

Feature importance

Answer

SHAP values

Answer

Confusion matrix

Question 10

An ML engineer is fine-tuning a large language model using LoRA on SageMaker. The training is converging slowly, and GPU utilization is low. The engineer suspects the bottleneck is data loading. Which action should the engineer take to improve GPU utilization?

Accepted Answer

Use SageMaker Pipe mode to stream data from S3 directly to the training instances. SageMaker Pipe mode streams training data directly from Amazon S3 to the training container over a high-throughput channel, bypassing the local disk and the download-then-read pattern of File mode. This removes the I/O bottleneck that starves the GPU when the dataset is large or when many small files cause slow reads. With faster data delivery, the GPU spends less time idle waiting for batches, raising utilization and speeding convergence.

Answer

Increase the batch size to maximize GPU memory usage

Answer

Enable checkpointing and use spot instances

Answer

Reduce model parallelism to decrease communication overhead

Question 11

Which SageMaker built-in algorithm is specifically designed for time series forecasting?

Accepted Answer

DeepAR. DeepAR is a SageMaker built-in algorithm based on recurrent neural networks (RNNs) specifically designed for time series forecasting. It learns from historical time series data and can predict future values, making it the correct choice for forecasting tasks. It supports both univariate and multivariate time series and can incorporate related time series.

Answer

Image Classification

Answer

BlazingText

Answer

XGBoost

Question 12

A team is training a PyTorch model using SageMaker with a custom training script. They want to track hyperparameters and metrics across multiple experiments. Which service should they use?

Accepted Answer

SageMaker Experiments. SageMaker Experiments is purpose-built for tracking ML experiment runs, automatically capturing hyperparameters, metrics, artifacts, and lineage across training jobs. It integrates natively with SageMaker training jobs and custom PyTorch scripts via the SageMaker SDK, letting teams compare runs in the Studio UI. This directly matches the requirement to track hyperparameters and metrics across multiple experiments.

Answer

SageMaker Clarify

Answer

SageMaker Model Monitor

Answer

SageMaker Debugger

Question 13

Which SageMaker built-in algorithm is best suited for detecting anomalous login attempts based on IP addresses and user behavior?

Accepted Answer

IP Insights. IP Insights is a SageMaker built-in algorithm purpose-built for learning the patterns of IP address usage by users and resources, making it ideal for detecting anomalous login attempts. It uses a neural network to embed IP addresses and entities (like user IDs) into a vector space, flagging unusual combinations as anomalies. XGBoost, PCA, and K-Means are general-purpose algorithms not designed for IP-entity behavioral analysis.

Answer

XGBoost

Answer

PCA

Answer

K-Means

Question 14

A team is fine-tuning a Hugging Face BERT model for text classification using SageMaker. They want to use the Hugging Face estimator for convenience. Which parameter must be set to use a custom training script?

Accepted Answer

entry_point. The entry_point parameter specifies the path to the custom training script that the Hugging Face estimator will execute inside the container. Without it, the estimator has no user-supplied Python file to run, so it cannot perform custom fine-tuning logic. framework_version, instance_type, and hyperparameters configure the environment but do not point to the training code itself.

Answer

framework_version

Answer

instance_type

Answer

hyperparameters

Question 15

An ML engineer is fine-tuning a foundation model using RLHF on SageMaker. Which THREE components are essential for this workflow? (Select THREE.)

Accepted Answer

A reward model trained on the preference data. Option A is correct because RLHF requires a reward model that has been trained on human preference data to serve as the scalar reward signal guiding policy optimization. Option C is correct because PPO (Proximal Policy Optimization) is the standard reinforcement learning algorithm used to update the policy (the language model) against the reward model while constraining updates via a KL penalty to the reference model. Option D is correct because a preference dataset containing human rankings (e.g., chosen vs. rejected responses) is the foundational input used to train the reward model in the first place. Option B is not essential to the RLHF training workflow itself; a validation set is useful for evaluation but is not a required RLHF component. Option E is not essential because PEFT methods like LoRA are an optional efficiency technique for fine-tuning, not a required element of the RLHF pipeline.

Answer

A large validation dataset for final evaluation

Answer

A PEFT technique like LoRA