MLA-C01 ML Model Development Practice Question

A team is training a large language model on SageMaker using PyTorch with data parallelism. The model is too large to fit on a single GPU. Which distributed training strategy should they use to split the model across multiple GPUs?

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

Model parallelism

Model parallelism splits the model itself across devices, which is necessary when the model is too large for one GPU. SageMaker's model parallelism library supports this.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✓
Model parallelism
Why this is correct
Model parallelism partitions the model across GPUs, allowing training of models that exceed single GPU memory.
✗
Tensor parallelism
Why it's wrong here
Tensor parallelism is a specific technique within model parallelism, but the more general answer is model parallelism.
✗
Data parallelism
Why it's wrong here
Data parallelism replicates the model on each GPU and splits the data, which still requires the model to fit on each GPU.
✗
Pipeline parallelism
Why it's wrong here
Pipeline parallelism is a form of model parallelism but not the only one; the question asks for a general strategy.

Go deeper

MLA-C01 ML Model Development Practice Question

Why each option matters

Option-by-option breakdown

Related to this question