Distributed Multi-GPU Training on SageMaker with Model Parallelism and Horovod
A data science team is training a large deep learning model using Amazon SageMaker. The training job is taking a long time because the model has many layers and the dataset is large. The team wants to reduce training time by distributing the training across multiple GPUs on a single instance, as well as across multiple instances. Which TWO actions should the team take? (Choose two.)
Quick Answer
The scenario asks for two distinct forms of scaling, across multiple GPUs on one instance and across multiple instances, and Horovod and the SageMaker model parallelism library address these through two different parallelism strategies that work well together. Horovod implements data parallelism: it takes copies of the same model, spreads different batches of training data across multiple GPUs and multiple instances, and synchronizes gradient updates between them, which speeds up training primarily by processing more data in parallel rather than by changing how the model itself fits in memory. The SageMaker model parallelism (SMP) library solves a different problem: when a model has enough layers and parameters that it's too large to fit comfortably on a single GPU, SMP splits the model itself across multiple GPUs using techniques like pipeline or tensor parallelism, allowing training to proceed even though no single device holds the entire model. Because data parallelism and model parallelism attack different bottlenecks, one is about processing more data faster, the other is about fitting and training a model too large for one device, combining them lets a training job scale both within an instance and across instances simultaneously. Whenever a question describes a large, many-layered model that also needs faster training across multiple GPUs and instances, expect the correct combination to include one technique for data parallelism and another for model parallelism, rather than assuming a single technique covers both dimensions of scale.
⚠ Common exam trap
Test-takers frequently confuse data parallelism (which shards data) with model parallelism (which shards the model), and assume that simply using multiple GPUs on a single instance automatically distributes the model, when in fact explicit model parallelism libraries like SMP are required for large models that do not fit in GPU memory.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Configure the training job to use SageMaker's model parallelism (SMP) library for pipeline or tensor parallelism.
The SageMaker model parallelism (SMP) library is specifically designed to split large deep learning models across multiple GPUs using pipeline or tensor parallelism. This allows the team to train models that are too large to fit on a single GPU and to reduce training time by parallelizing computation across devices within and across instances.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Use SageMaker's distributed data parallelism (SMDDP) library to shard the model across GPUs.
Why it's wrong here
SMDDP is for data parallelism, not model parallelism. It replicates the model on each GPU and splits the data, which does not reduce memory footprint per GPU for large models.
- ✓
Configure the training job to use SageMaker's model parallelism (SMP) library for pipeline or tensor parallelism.
Why this is correct
SMP allows splitting the model across multiple GPUs and instances, reducing memory footprint per GPU and enabling training of large models that would otherwise not fit. This complements data parallelism.
- ✗
Use SageMaker's managed training with a single instance containing multiple GPUs and enable data parallelism.
Why it's wrong here
This approach uses only one instance, so it does not distribute across multiple instances. While multiple GPUs on one instance help, the question asks for actions to reduce training time by distributing across both GPUs and instances.
- ✓
Use Horovod for data parallelism across multiple instances.
Why this is correct
Horovod is a distributed training framework that supports data parallelism across multiple instances, reducing training time by processing more data in parallel.
- ✗
Set the instance type to a single GPU instance and rely on automatic model parallelism.
Why it's wrong here
A single GPU instance limits parallelism. Automatic model parallelism is not a built-in SageMaker feature; training on one GPU would be slower.
Go deeper
Related to this question
About these practice questions
This MLS-C01 question is part of Courseiva's 1,672-question bank — original exam-style content with full explanations and wrong-answer analysis, never real exam questions or exam dumps. Learn why practice questions differ from exam dumps →
Same concept, more angles
2 more ways this is tested on MLS-C01
These questions test the same concept from different angles. Work through them to make sure you can recognise it however the exam phrases it.
Variation 1. A data scientist is training a deep learning model on a large dataset using Amazon SageMaker. The training job is taking too long and the scientist wants to reduce the training time by distributing the workload across multiple GPUs. Which SageMaker feature should be used to achieve this?
easy- ✓ A.Use SageMaker's distributed training libraries
- B.Use Amazon EMR to distribute the training
- C.Use SageMaker Automatic Model Tuning
- D.Use SageMaker Hyperparameter Tuning
Why A: SageMaker's distributed training libraries provide built-in optimized implementations of data parallelism and model parallelism, enabling efficient distribution of deep learning workloads across multiple GPUs with minimal code changes. This directly addresses the need to reduce training time by leveraging multiple GPUs in a SageMaker training job.
Variation 2. A data scientist is using SageMaker to train a deep learning model. The training script uses TensorFlow and runs on a single p3.2xlarge instance. The scientist wants to reduce training time by using multiple GPUs. What should the scientist do?
medium- A.Increase the instance count to 4 without changing the script.
- ✓ B.Modify the training script to use Horovod for distributed training.
- C.Switch to PyTorch framework.
- D.Use SageMaker Managed Spot Training.
Why B: Horovod is a distributed deep learning framework that integrates with TensorFlow to enable multi-GPU training across multiple instances. By modifying the training script to use Horovod's `hvd.DistributedOptimizer` and broadcasting initial variables, the data scientist can leverage multiple GPUs on a single p3.2xlarge instance (which has 1 GPU) or scale to multiple instances, directly reducing training time through data parallelism.
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This MLS-C01 practice question is part of Courseiva's free Amazon Web Services certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the MLS-C01 exam.