MLA-C01 ML Model Development Practice Question

A team is training a large language model and needs to split the model layers across multiple GPUs due to memory constraints. Which distributed training strategy should they use?

⚠ Common exam trap

MLA-C01 often tests the distinction between data parallelism (replicate model, split data) and model parallelism (split model, replicate data) — candidates pick data parallelism by default because it is more common, missing the memory-constraint keyword.

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

Model parallelism

Model parallelism splits the model's layers (or tensors) across multiple GPUs so that each GPU holds only a portion of the model's parameters, which is required when the model is too large to fit in a single GPU's memory. Data parallelism, by contrast, replicates the full model on every GPU and only partitions the training data, so it does not solve a memory-constraint problem.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✗
Data parallelism
Why it's wrong here
Data parallelism replicates the full model on every GPU and splits only the batch, so per-GPU memory is unchanged and the constraint persists. It suits scenarios where the model fits on one device but throughput must scale. Layer-splitting across devices is instead pipeline or tensor parallelism.
✗
Hyperparameter tuning
Why it's wrong here
Hyperparameter tuning searches configurations such as learning rate and batch size; it does not partition model layers across devices and cannot relieve GPU memory pressure. It would be the correct choice when optimising model accuracy or convergence, not when the model itself exceeds single-GPU memory.
✗
Autopilot
Why it's wrong here
Autopilot automates tabular model building and feature engineering; it cannot shard layers across GPUs. It is tempting because it handles training orchestration end to end, and would be the right choice for automatically producing a regression or classification model from tabular data without writing training code.
✓
Model parallelism
Why this is correct
Model parallelism partitions the model's layers across multiple GPUs, so each device holds only a subset of weights. This directly resolves the memory constraint described, where a single GPU cannot hold the entire large language model, unlike data parallelism which replicates the full model on every device.

Go deeper

MLA-C01 ML Model Development Practice Question

Why each option matters

Option-by-option breakdown

Related to this question