NCP-AIO Troubleshooting and Optimization Practice Question
An operations engineer is investigating a sudden drop in throughput for a multi-GPU training job on an NVIDIA DGX A100. The job uses PyTorch with DDP. Logs show that one GPU is consistently at 100% utilization while others are below 50%. Which tool and approach should be used to identify the bottleneck?
⚠ Common exam trap
The trap here is choosing a profiling tool that is too granular, like Nsight Compute, or a monitoring tool that lacks detail, like nvidia-smi, instead of the system-level profiler needed for this multi-GPU imbalance.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Use NVIDIA Nsight Systems to profile the job and examine the CUDA API and kernel timeline to identify serialization or synchronization issues.
Nsight Systems is designed for system-level profiling, capturing CPU and GPU activities across multiple devices. It can show if one GPU is performing more work or if others are idle waiting for synchronization. This makes it the right tool to identify the bottleneck in a DDP job where one GPU is overutilized. Profiling before making changes is essential.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Use nvidia-smi to monitor GPU utilization and memory, then rebalance the workload by reducing the batch size on the overutilized GPU.
Why it's wrong here
nvidia-smi provides overall utilization but does not reveal the cause of imbalance. Reducing batch size on one GPU is not possible with DDP, which requires equal batch sizes across processes. This approach does not identify the root cause and is not feasible. It also may degrade performance further.
- ✓
Use NVIDIA Nsight Systems to profile the job and examine the CUDA API and kernel timeline to identify serialization or synchronization issues.
Why this is correct
Nsight Systems provides a detailed timeline of CPU and GPU activity, showing kernel executions, memory copies, and synchronization points. It can reveal if one GPU is waiting on data or if there is an imbalance in computation. This is the correct tool to pinpoint the bottleneck in a multi-GPU job, as it captures cross-GPU interactions and can highlight stragglers.
- ✗
Use NVIDIA Nsight Compute to profile each GPU's kernels and compare execution times to find the slowest kernel.
Why it's wrong here
Nsight Compute is for deep-diving into individual kernel performance, not for analyzing overall job throughput or inter-GPU communication. While it can show kernel durations, it does not provide the system-wide view needed to identify why one GPU is overutilized relative to others. It is too granular for this scenario.
- ✗
Use nvidia-smi topo -m to check the GPU topology and then adjust the NCCL communication settings to use a different ring order.
Why it's wrong here
Checking topology is useful for understanding interconnect, but the symptom of one GPU at 100% while others are low suggests a computational or data loading imbalance, not necessarily a topology issue. Adjusting NCCL ring order is a complex step that may not address the root cause. This approach skips the diagnostic step of profiling to find the actual bottleneck.
About these practice questions
One of 309 original NCP-AIO practice questions on Courseiva, each with a full explanation and wrong-answer analysis — not exam dumps or protected exam content. Learn why practice questions differ from exam dumps →
JA
Written and reviewed by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
Last reviewed September 2026 · checked against the official NVIDIA exam blueprint
This NCP-AIO practice question is part of Courseiva's free NVIDIA certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the NCP-AIO exam.