How to Set Up PyTorch Distributed Data Parallelism with SageMaker
A data scientist is training a deep learning model on Amazon SageMaker using a PyTorch estimator. The training job runs on a single ml.p3.2xlarge instance but is taking too long. The scientist wants to reduce training time by using distributed data parallelism across multiple GPUs. Which change to the training script and SageMaker estimator is required?
Quick Answer
The correct answer is to modify the training script to use `torch.nn.parallel.DistributedDataParallel` and set `instance_count` to 2 in the SageMaker PyTorch estimator. This is required because PyTorch distributed data parallelism on SageMaker relies on DistributedDataParallel (DDP) to synchronize gradients across multiple nodes, while the `instance_count` parameter tells SageMaker to launch multiple training instances—each with its own GPU—enabling true multi-node parallelism rather than just multi-GPU on a single machine. On the AWS Certified Machine Learning Specialty MLS-C01 exam, this scenario tests your understanding of how SageMaker abstracts distributed training infrastructure: a common trap is confusing `DataParallel` (single-node, multi-GPU) with `DistributedDataParallel` (multi-node), or forgetting that SageMaker requires you to increase `instance_count` explicitly. Remember the mnemonic: “DDP for distributed, count up for clusters”—if you see multiple instances in the question, your script needs DDP and your estimator needs a higher instance count.
⚠ Common exam trap
The MLS-C01 exam often tests the distinction between `DataParallel` (single-node, multi-GPU) and `DistributedDataParallel` (multi-node, multi-GPU), leading candidates to incorrectly choose `DataParallel` because they overlook the requirement for multiple instances.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Modify the script to use torch.nn.parallel.DistributedDataParallel and set instance_count to 2 in the estimator.
To achieve distributed data parallelism across multiple GPUs on multiple instances with PyTorch, you must modify the training script to use `torch.nn.parallel.DistributedDataParallel` (DDP), which handles gradient synchronization across nodes. Additionally, you must set `instance_count` to 2 (or more) in the SageMaker PyTorch estimator to launch multiple instances, each with its own GPU, enabling true multi-node distributed training.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Add the SageMaker distributed data parallelism configuration in the estimator and modify the script to use the SageMaker distributed library.
Why it's wrong here
That is for SageMaker's own distributed library, not PyTorch DDP.
- ✗
Change the framework to TensorFlow and use tf.distribute.MirroredStrategy with instance_count=2.
Why it's wrong here
Changing framework is unnecessary and may introduce compatibility issues.
- ✓
Modify the script to use torch.nn.parallel.DistributedDataParallel and set instance_count to 2 in the estimator.
Why this is correct
DDP is efficient for multi-node training.
- ✗
Modify the script to use torch.nn.DataParallel and keep instance_count as 1.
Why it's wrong here
DataParallel is single-node multi-GPU; it won't help with multiple instances.
Go deeper
Related to this question
About these practice questions
Courseiva writes every MLS-C01 question from scratch — 1,672 in total, each with an explanation and a wrong-answer breakdown. None are copied from real exams or dumps. Learn why practice questions differ from exam dumps →
Same concept, more angles
1 more way this is tested on MLS-C01
These questions test the same concept from different angles. Work through them to make sure you can recognise it however the exam phrases it.
Variation 1. A data scientist is training a deep learning model on Amazon SageMaker and notices that training is taking much longer than expected. The training job uses a single GPU instance. The model is a large transformer with millions of parameters. Which change would most likely reduce training time?
hard- A.Reduce the batch size to fit in memory
- B.Use a smaller instance type
- C.Switch to a CPU instance
- ✓ D.Use SageMaker's distributed data parallelism with multiple GPU instances
Why D: Using data parallelism with multiple GPU instances can significantly reduce training time for large models by distributing the workload across multiple GPUs. Model parallelism is also possible but data parallelism is more common and easier to implement.
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This MLS-C01 practice question is part of Courseiva's free Amazon Web Services certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the MLS-C01 exam.