NCP-AIO Troubleshooting and Optimization Practice Question

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

• ✓

  Verify that all nodes have identical NCCL versions and environment variables, and that InfiniBand fabric manager is running.

  Why this is correct

  Inconsistent NCCL versions or missing environment variables (e.g., NCCL_IB_HCA, NCCL_SOCKET_IFNAME) can cause suboptimal transport selection or failures. The InfiniBand fabric manager ensures proper link configuration. Ensuring uniformity across nodes is crucial for optimal collective communication performance and is a common fix for scaling inefficiencies.

• ✓

  Run NCCL_DEBUG=INFO and inspect the logs for warnings about falling back to slower transports or retries.

  Why this is correct

  NCCL debug logs reveal the transport used (e.g., InfiniBand vs. TCP) and any errors or retries. If NCCL falls back to TCP or shows excessive retransmissions, it indicates a communication bottleneck. This is a primary diagnostic step to confirm whether the interconnect is the limiting factor and to guide corrective actions such as driver or topology fixes.

• ✗

  Switch the job to use Ethernet instead of InfiniBand to simplify troubleshooting.

  Why it's wrong here

  Ethernet typically offers lower bandwidth and higher latency than InfiniBand in HPC clusters, which would likely worsen scaling efficiency. Switching to Ethernet is not a troubleshooting step but a downgrade. The goal is to identify and fix InfiniBand issues, not to avoid them by using an inferior interconnect.

• ✗

  Use nvidia-smi to monitor GPU utilization and memory bandwidth on each node during training.

  Why it's wrong here

  nvidia-smi provides GPU utilization and memory throughput but does not directly measure interconnect performance or NCCL communication efficiency. While low GPU utilization could hint at communication stalls, it is not specific enough to confirm an InfiniBand bottleneck. This action is more about overall GPU health than network diagnostics.

• ✗

  Increase the global batch size proportionally to the number of GPUs to improve compute-to-communication ratio.

  Why it's wrong here

  Increasing batch size can improve efficiency by amortizing communication overhead, but it does not diagnose the root cause and may harm convergence. In this scenario, the team first needs to confirm the bottleneck. Blindly changing hyperparameters could mask the issue without resolving underlying network problems, and it may not be appropriate for all models.