MLA-C01 ML Model Development Practice Test 9 — 15 Questions

Question 1

A team wants to evaluate a binary classification model for credit risk. They need to understand the trade-off between false positives and false negatives. Which TWO metrics should they use? (Select TWO.)

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Recall. Recall (A) is correct because it measures the proportion of actual positives (e.g., true defaults) that the model correctly identifies, directly quantifying the cost of false negatives, which is critical in credit risk where missing a defaulter is costly. Precision (B) is correct because it measures the proportion of predicted positives that are actually positive, directly quantifying the cost of false positives, such as rejecting good customers; together, recall and precision expose the false-positive/false-negative trade-off. NDCG (C) is a ranking-quality metric for graded relevance in search/recommendation, not binary classification. AUC-ROC (D) summarizes ranking performance across thresholds but does not separately expose the precision/recall trade-off the team wants to evaluate. RMSE (E) is a regression error metric and is inappropriate for binary classification.

Answer

NDCG

Answer

AUC-ROC

Answer

RMSE

Question 2

A team is training a large deep learning model on SageMaker using a single ml.p3.16xlarge instance. Training is taking too long. They want to reduce time by distributing across multiple GPUs but are constrained by model size that does not fit in a single GPU memory. Which distributed training strategy should they use?

Accepted Answer

Model parallelism using SageMaker distributed model parallelism. Model parallelism splits the model across multiple GPUs, which is needed when the model does not fit in a single GPU. Data parallelism replicates the model on each GPU and splits data, which requires the model to fit in each GPU's memory.

Answer

Data parallelism using SageMaker distributed data parallelism

Answer

Switch to a smaller instance type and use horizontal scaling

Answer

Use multiple training jobs with hyperparameter tuning

Question 3

A company is using SageMaker Automatic Model Tuning to optimize a regression model. They want to minimize the root mean squared error (RMSE). The tuner has completed 20 jobs, and the RMSE has plateaued. Which action should the data scientist take to potentially improve the results?

Accepted Answer

Decrease the range of hyperparameters to focus on promising areas. When RMSE has plateaued, narrowing the hyperparameter ranges to focus on promising areas can help the tuner explore more finely around good values. This is a common technique in Bayesian optimization to refine the search. Increasing jobs or parallelism may not help if the search space is too broad.

Answer

Increase the maximum number of training jobs

Answer

Increase the number of parallel training jobs

Answer

Switch the objective metric to mean absolute error (MAE)

Question 4

A machine learning engineer is using Amazon SageMaker Debugger to monitor a training job for a deep neural network. They receive a rule alert indicating 'exploding gradients'. Which action should they take to address this issue?

Accepted Answer

Reduce the learning rate. Exploding gradients occur when large error gradients accumulate during backpropagation, causing unstable updates and divergence. Reducing the learning rate directly scales down the parameter update step (Δθ = -η∇J), preventing the weights from overshooting and stabilizing training. This is the standard first-line remedy for exploding gradients in deep networks.

Answer

Use a smaller batch size

Answer

Increase the number of layers to absorb gradients

Answer

Increase the learning rate

Question 5

A data scientist is using SageMaker Autopilot to automatically build a binary classification model on a balanced dataset. They want to understand the relationship between the input features and the model predictions. Which feature in SageMaker Autopilot should they use?

Accepted Answer

Explainability reports. SageMaker Autopilot's explainability reports provide feature importance and partial dependence plots (PDPs) that show how each input feature influences model predictions. For a binary classification model, these reports help data scientists understand the relationship between features and predictions. This is exactly what the question asks for.

Answer

Data visualizations

Answer

Model tuning results

Answer

Model candidate definitions

Question 6

A team wants to fine-tune a pre-trained Hugging Face transformer model for text classification using SageMaker. They have a custom training script. Which SageMaker estimator should they use?

Accepted Answer

SageMaker Hugging Face estimator. The SageMaker Hugging Face estimator is purpose-built for Hugging Face transformer models: it uses the official Hugging Face Deep Learning Containers, supports the transformers, datasets, and accelerate libraries, and accepts hyperparameters like epoch, learning_rate, and model_name_or_path. It is the natural choice for fine-tuning a pre-trained Hugging Face model with a custom training script.

Answer

SageMaker generic estimator with a custom container

Answer

SageMaker PyTorch estimator

Answer

SageMaker TensorFlow estimator

Question 7

A financial services company trains multiple models on SageMaker and needs to track hyperparameters, metrics, and artifacts for each experiment. Which SageMaker feature should they use to organize and compare experiments?

Accepted Answer

SageMaker Experiments. SageMaker Experiments is purpose-built for tracking, organizing, and comparing machine learning experiments — it captures hyperparameters, metrics, input datasets, and output artifacts for each trial and groups them into experiments and runs. This lets data scientists visualize and compare results across many training jobs in a single view.

Answer

SageMaker Model Registry

Answer

SageMaker Pipelines

Answer

SageMaker Debugger

Question 8

A company is fine-tuning a large language model using LoRA on SageMaker. They want to reduce GPU memory usage during training. Which configuration change would help?

Accepted Answer

Use QLoRA (quantized LoRA) with 4-bit quantization. QLoRA extends LoRA by quantizing the frozen base model weights to 4-bit (typically NF4) while keeping LoRA adapters in higher precision, dramatically reducing GPU memory required for fine-tuning. This lets you train larger models on smaller GPUs with minimal accuracy loss, directly addressing the goal of reducing memory usage. It is the standard memory-optimization technique for LoRA fine-tuning on SageMaker.

Answer

Enable gradient accumulation

Answer

Increase the sequence length

Answer

Increase the batch size

Question 9

A data scientist is using SageMaker to train an XGBoost model for a regression problem. After training, they evaluate the model on a test set and get an RMSE of 10 and an R² of 0.85. Which additional metric would give the MOST insight into the model's average prediction error magnitude?

Accepted Answer

Mean Absolute Error (MAE). MAE measures the average absolute difference between predicted and actual values in the same units as the target, so it directly answers 'how far off is the model on average?' RMSE of 10 is also in target units but is dominated by large errors due to squaring, so MAE complements it by showing typical error magnitude without outlier amplification.

Answer

AUC

Answer

Confusion matrix

Answer

F1 score

Question 10

A company wants to detect anomalies in login events from a large user base, focusing on unusual patterns that may indicate compromised accounts. Which SageMaker built-in algorithm is most suitable for this task?

Accepted Answer

IP Insights. IP Insights is a SageMaker built-in unsupervised algorithm designed to learn the relationship between user entities and IP addresses, making it ideal for detecting anomalous login events such as a user logging in from an unusual IP or a compromised account accessing from a new location. It is specifically built for the login-anomaly use case described.

Answer

K-Means

Answer

DeepAR

Answer

Factorisation Machines

Question 11

A data scientist needs to run a hyperparameter tuning job for a PyTorch model using SageMaker. They want to use Hyperband for efficient resource allocation. Which tuning strategy should they select in the HyperparameterTuner?

Accepted Answer

Hyperband. SageMaker's HyperparameterTuner supports Hyperband as a first-class strategy, which implements the multi-armed bandit early-stopping algorithm to allocate more resources to promising configurations and prune poor ones early. Selecting 'Hyperband' as the strategy directly enables this efficient resource allocation. It is distinct from Bayesian, Random, and Grid strategies, which do not perform the successive-halving resource allocation that Hyperband does.

Answer

Bayesian optimization

Answer

Random search

Answer

Grid search

Question 12

A company is training a large NLP model on SageMaker and wants to reduce costs by using Spot Instances. Which TWO configurations should they implement to handle Spot interruptions gracefully?

Accepted Answer

Set `use_spot_instances=True` and `max_wait` in the estimator. Option B is correct because in the SageMaker SDK estimator you must explicitly set use_spot_instances=True to enable managed Spot training, and max_wait defines the maximum wall-clock time SageMaker will wait for the Spot capacity (including interruptions and restarts), which is required to let training resume after an interruption. Option E is correct because enabling checkpointing (e.g., via checkpoint_s3_uri) periodically saves model state to Amazon S3, so when a Spot instance is reclaimed the job can restart from the last checkpoint instead of from scratch, which is the core mechanism for handling interruptions gracefully. Option A is not correct because using a single large instance does not meaningfully reduce interruption probability and actually increases the cost/impact of a single interruption. Option C is not correct because increasing max_run only sets the maximum training duration; it does not help the job survive or recover from a Spot interruption. Option D is not correct because keep_alive_period is used for managed warm pools to reduce cold-start latency between jobs, not to preserve training state across Spot interruptions.

Answer

Use a single large instance to reduce interruption probability

Answer

Increase the `max_run` parameter to allow longer training

Answer

Use `keep_alive_period` to keep the instance alive after training

Question 13

A company wants to use SageMaker to fine-tune a foundation model for a text generation task using RLHF (Reinforcement Learning from Human Feedback). Which THREE components are required in the RLHF pipeline?

Accepted Answer

A pre-trained base model. Option B is correct because RLHF always starts from a pre-trained foundation model (the policy) that already has broad language capabilities; this base model is then fine-tuned with human preference signals rather than trained from scratch. Option D is correct because RLHF requires a reward model trained on human preference comparisons (e.g., chosen vs. rejected responses) to serve as a differentiable proxy for human judgment during optimization. Option E is correct because the policy is optimized against that reward model using a reinforcement learning algorithm such as PPO (Proximal Policy Optimization), typically with a KL penalty to the reference model to prevent reward hacking. Option A is not required: LoRA is a parameter-efficient fine-tuning technique that can optionally be used, but RLHF does not mandate adapters. Option C is not required: distinguishing generated from real text is a GAN discriminator concept, not part of the RLHF pipeline, which relies on preference-based reward modeling instead.

Answer

A LoRA adapter for parameter-efficient fine-tuning

Answer

A classifier to distinguish generated text from real text

Question 14

A data scientist is using SageMaker built-in XGBoost algorithm for a regression problem. Which metric is most appropriate as the objective metric for hyperparameter tuning?

Accepted Answer

RMSE. RMSE (Root Mean Squared Error) is the most appropriate objective metric for hyperparameter tuning in a regression problem because it directly measures the average magnitude of prediction errors, with larger errors penalized more heavily. SageMaker's built-in XGBoost algorithm supports RMSE as an evaluation metric for regression, and it is commonly used as the objective metric for tuning jobs.

Answer

NDCG

Answer

AUC

Answer

F1

Question 15

A team is training a large language model using PyTorch on SageMaker. They need to reduce training time. The model has 10 billion parameters. Which distributed training strategy should they use?

Accepted Answer

Model parallelism with SageMaker distributed. Model parallelism with SageMaker distributed is correct because a 10-billion-parameter model cannot fit into the memory of a single GPU, so the model itself must be partitioned across multiple GPUs/devices. SageMaker's distributed model parallelism library shards model layers and parameters across instances, enabling training of models that exceed single-device memory. This directly addresses the memory bottleneck that prevents scaling with data parallelism alone.

Answer

Data parallelism with Horovod

Answer

Single GPU training

Answer

Use a larger instance type without parallelism