MLS-C01 Modeling Practice Question

A company is using Amazon SageMaker to train a large language model with billions of parameters. The training job uses multiple GPU instances in a distributed fashion. The training is converging but the loss is not decreasing as expected. The data scientist suspects that the learning rate is too high. Which technique should the data scientist use to automatically adjust the learning rate during training?

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

Implement learning rate scheduling with a cosine annealing schedule

Learning rate scheduling, such as a cosine annealing schedule, can automatically reduce the learning rate over time. This helps the model converge better. SageMaker's built-in algorithms support learning rate scheduling, or the user can implement it in custom training scripts.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✗
Use a fixed learning rate and train for more epochs
Why it's wrong here
A fixed high learning rate may cause the loss to plateau or diverge.
✗
Increase the batch size to reduce variance
Why it's wrong here
Increasing batch size may not directly address a high learning rate issue.
✓
Implement learning rate scheduling with a cosine annealing schedule
Why this is correct
Cosine annealing reduces the learning rate smoothly, helping convergence.
✗
Use gradient clipping
Why it's wrong here
Gradient clipping prevents exploding gradients but does not adjust learning rate.

Go deeper

MLS-C01 Modeling Practice Question

Why each option matters

Option-by-option breakdown

Related to this question