A data scientist trained a model on a single GPU but needs to train on multiple GPUs for a larger dataset. They observe that training time does not decrease linearly with additional GPUs. Which common issue is most likely?
I/O or preprocessing bottleneck limits GPU utilization.
Why this answer
Data pipeline bottleneck. When training on multiple GPUs, the expected speedup is linear if data loading and preprocessing can keep up with the GPUs. However, if the data pipeline is slow (e.g., due to disk I/O, network latency, or insufficient CPU preprocessing), GPUs will spend time idle waiting for data, preventing linear scaling.
Option A is incorrect because overfitting affects model accuracy, not training speed. Option B is incorrect because model architecture simplicity does not directly cause non-linear speedup; simpler architectures may actually speed up training. Option C is incorrect because learning rate affects convergence behavior, not the efficiency of parallel training.