PMLE Scaling Prototypes into ML Models Practice Question
You are designing a distributed training job on Vertex AI for a PyTorch model using DataDistributedParallel (DDP). You have 4 nodes, each with 4 GPUs. What is the total number of workers that should be configured in the TF_CONFIG equivalent for PyTorch?
⚠ Common exam trap
PMLE often tests the worker-count calculation — candidates multiply nodes by GPUs incorrectly or forget that each GPU runs its own DDP process, leading them to pick the node count instead of the total process count.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
16
In PyTorch DDP on Vertex AI, the number of workers equals the total number of processes across all nodes, which is nodes × GPUs per node. With 4 nodes and 4 GPUs each, the total is 16 workers, so the TF_CONFIG-equivalent worker count should be 16. Each GPU runs one DDP process, and all 16 processes participate in the all-reduce synchronization.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
4
Why it's wrong here
Four workers equals the node count, leaving each process to address four GPUs, which DDP does not do; one process must own one GPU. It is tempting because node count is the natural unit when provisioning the cluster, which would be correct for a job with one GPU per node.
- ✗
8
Why it's wrong here
Eight workers covers only half the sixteen GPUs, so eight accelerators remain unused and the effective batch size is halved. It is tempting because it doubles the node count, which would be correct if each node carried two GPUs instead of four.
- ✗
1
Why it's wrong here
One worker describes a single-process job, so the remaining fifteen GPUs sit idle and DDP cannot shard gradients across them. It is tempting because one process per node is how some launchers are invoked, which would be correct only if each node exposed a single accelerator rather than four.
- ✓
16
Why this is correct
DataDistributedParallel assigns one worker process per GPU, so 4 nodes multiplied by 4 GPUs each yields 16 workers. Configuring 16 matches the total GPU count and ensures every device participates in the distributed training job.
Go deeper
Related to this question
About these practice questions
One of 775 original PMLE practice questions on Courseiva, each with a full explanation and wrong-answer analysis — not exam dumps or protected exam content. Learn why practice questions differ from exam dumps →
Same concept, more angles
1 more way this is tested on PMLE
These questions test the same concept from different angles. Work through them to make sure you can recognise it however the exam phrases it.
Variation 1. Your PyTorch training script uses DistributedDataParallel (DDP) across 4 vertices each with 4 GPUs (16 GPUs total). You submit a Vertex AI custom training job. How should you configure the worker pool spec?
medium- ✓ A.Create one worker pool with 4 replicas, each with machine type having 4 GPUs
- B.Create a chief worker pool with 1 replica (4 GPUs) and a parameter server pool with 4 replicas (no GPUs)
- C.Create 4 separate jobs, each with 1 replica and 4 GPUs
- D.Create one worker pool with 16 replicas, each with 1 GPU
Why A: For DDP across multiple machines, use MultiWorkerMirroredStrategy equivalent in PyTorch: set replicas to 4, each with machine type having 4 GPUs. The TF_CONFIG env var is not needed; Vertex AI sets necessary environment variables for distributed training.
JA
Written and reviewed by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
Last reviewed September 2026 · checked against the official Google Cloud exam blueprint
This PMLE practice question is part of Courseiva's free Google Cloud certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the PMLE exam.