You are training a TensorFlow model on Vertex AI using a custom container with a single Tesla T4 GPU. You notice that training is slower than expected, and GPU utilization is consistently below 20%. Profiling shows that the input pipeline is the bottleneck. Which change should you make to improve GPU utilization?
The tf.data API with prefetching and parallel extraction allows data preprocessing to occur on CPU while the GPU computes on the previous batch, effectively overlapping I/O and compute. This directly addresses the input pipeline bottleneck, increasing GPU utilization and reducing training time. It is the recommended approach for optimizing input pipelines in TensorFlow.
Why this answer
When the input pipeline is the bottleneck, the GPU waits for data. Using tf.data with prefetching and parallel extraction overlaps CPU preprocessing with GPU computation, ensuring the GPU is continuously fed. This is the standard TensorFlow optimization for such scenarios and directly improves GPU utilization.
Exam trap
The trap here is assuming that a faster GPU or more memory will fix low GPU utilization, when the issue is actually data starvation from an inefficient input pipeline.