PMLE Scaling Prototypes into ML Models Practice Question

• A.Data parallelism with MirroredStrategy
• B.Data parallelism with MultiWorkerMirroredStrategy
• C.Use TPU training as TPUs have more memory
• ✓ D.Model parallelism using pipeline parallelism

Why D: When a model does not fit on a single GPU, model parallelism is required because it partitions the model itself across devices. Pipeline parallelism is a specific model parallelism technique that splits the model into stages across GPUs and pipelines micro-batches to maintain utilization, making it the appropriate approach for training a very large model on multiple GPUs in one machine.

• A.tf.distribute.experimental.MultiWorkerMirroredStrategy
• B.tf.distribute.experimental.ParameterServerStrategy
• ✓ C.Manual device placement using tf.device to assign layers to specific GPUs
• D.tf.distribute.MirroredStrategy

Why C: When a single model's layers exceed one GPU's memory, the model itself must be partitioned across devices — this is model parallelism. Manual device placement with tf.device('/GPU:0'), tf.device('/GPU:1'), etc. is the TensorFlow-native way to assign specific layers or operations to specific GPUs, splitting the model across them.

• A.ParameterServerStrategy
• ✓ B.Pipeline parallelism
• ✓ C.Operator-level model parallelism
• D.Data parallelism with MirroredStrategy
• E.MultiWorkerMirroredStrategy

Why B: Pipeline parallelism (B) is correct because it splits the model's layers into sequential stages placed on different devices, so a model too large for one machine can be distributed across devices while micro-batches flow through the pipeline. Operator-level model parallelism (C) is also correct because it partitions individual operations (e.g., splitting a large matrix multiplication or a layer's weights) across devices, which directly addresses a model that does not fit on a single machine. ParameterServerStrategy (A) is a data-parallel approach that replicates the full model on each worker and only shards the parameter updates, so it does not solve the memory problem of a model too large for one device. Data parallelism with MirroredStrategy (D) likewise replicates the entire model on every device, which is infeasible when the model does not fit on one machine. MultiWorkerMirroredStrategy (E) is also a data-parallel strategy that requires the full model to fit on each worker, so it does not satisfy the requirement.

• A.Use Vertex AI Hyperparameter Tuning to find optimal model partitioning
• B.Use tf.distribute.MirroredStrategy across all GPUs
• ✓ C.Implement pipeline parallelism by manually splitting the model layers across GPUs and using a framework like PyTorch's RPC or Megatron-LM
• D.Use Vertex AI distributed training with TF_CONFIG to set up multi-worker mirrored strategy and rely on XLA to partition the model

Why C: When a model does not fit into a single GPU's memory, model parallelism is required. Pipeline parallelism splits the model layers across multiple GPUs, and frameworks like PyTorch's RPC or Megatron-LM provide the necessary primitives to implement it. This is the correct strategy for fitting a large model like LLaMA 2 across multiple GPUs.

• ✓ A.Model parallelism using pipeline parallelism
• B.Data parallelism with MultiWorkerMirroredStrategy
• C.Mixed precision training (FP16)
• D.Data parallelism with tf.distribute.MirroredStrategy

Why A: When a model is too large to fit on a single GPU, model parallelism is required to split the model's layers across multiple devices. Pipeline parallelism is a specific form of model parallelism that partitions layers into stages and pipelines micro-batches across devices, enabling training of models like a 7B-parameter LLM across multiple GPUs. This is the correct approach when memory, not throughput, is the binding constraint.

• A.Using Vertex AI Vizier to optimize the model parallelism strategy
• B.Enabling Vertex AI AutoML to automatically distribute the model
• ✓ C.Implementing pipeline parallelism manually in the training script using torch.distributed.pipeline.sync.Pipe
• ✓ D.A custom container with the distributed framework (e.g., PyTorch DDP) installed
• ✓ E.Setting the --worker-machine-count flag when submitting the job

Why C: Pipeline parallelism requires explicit implementation in the training script, such as using `torch.distributed.pipeline.sync.Pipe` in PyTorch, to split the model layers across multiple GPUs. This is necessary when the model does not fit on a single GPU, and Vertex AI does not automatically handle model parallelism—it must be coded by the user.

• ✓ A.Model parallelism using tf.distribute.experimental.PipelineMirroredStrategy
• B.Data parallelism using tf.distribute.MirroredStrategy
• C.Multi-worker mirrored strategy with a single worker per node
• D.Hyperparameter tuning with Vertex AI Vizier

Why A: PipelineMirroredStrategy combines model parallelism (splitting the transformer layers across multiple GPUs) with pipeline parallelism to handle models that exceed single GPU memory. This strategy partitions the model into stages, each placed on a different GPU, and uses micro-batching to keep all GPUs busy, which is essential for large transformer models that cannot fit into a single GPU's memory.

• A.Use Vertex AI's AutoML to automatically distribute the model.
• ✓ B.Use pipeline parallelism via a custom container with DeepSpeed and data parallelism across workers using PyTorch DDP, configured with Vertex AI distributed training.
• C.Use Vertex AI's hyperparameter tuning with multiple trials.
• D.Configure a multi-worker mirrored strategy with TensorFlow, setting TF_CONFIG to use all GPUs on each worker.

Why B: It combines pipeline parallelism (via DeepSpeed) to split the 7B-parameter model across multiple GPUs, with data parallelism (via PyTorch DDP) to replicate the model across workers for training on larger batches. Vertex AI distributed training coordinates the multi-worker setup, making this the only viable strategy for a model that exceeds single-GPU memory while requiring data parallelism.