Troubleshooting Scenario Questions

The GPU driver is corrupted and requires a reinstall.

Why it fails: Corrupted drivers usually result in 'device not found' or kernel initialization failures rather than an 'invalid ordinal' error. An invalid ordinal refers to a specific integer index that is out of range, pointing to a configuration issue regarding how the application is selecting the hardware, not a driver malfunction.

The GPU memory is full.

Why it fails: Memory fullness results in OOM (Out of Memory) errors, not an 'invalid device ordinal' error. The system knows the device exists, but the allocation request fails. An invalid ordinal implies that the device selection itself is the point of failure, regardless of the memory state of any physical GPU.

The InfiniBand fabric is misconfigured.

Why it fails: InfiniBand issues affect communication latency and connectivity between nodes, not the local indexing of GPUs within the host system. CUDA device indexing is handled by the local driver and runtime, independent of the network fabric. Misconfigurations here would manifest as NCCL errors, not device ordinal errors at startup.

Verify that the GPU memory usage is below 50% on all nodes.

Why it fails: GPU memory usage is a workload-specific metric related to the model size and batch size. While relevant for OOM errors, it is unrelated to inter-node communication latency or bandwidth bottlenecks. High memory usage does not explain why communication across the network fabric would be slow between cluster nodes.

Restart the Kubernetes API server to refresh the node connection states.

Why it fails: Restarting the Kubernetes API server is a disruptive action that does not address network throughput issues. Inter-node performance is managed by the network fabric and the NCCL library, not the Kubernetes API controller. This action is unlikely to resolve performance bottlenecks and could cause cluster-wide instability during the process.

Update the NVIDIA driver to the latest gaming-optimized release.

Why it fails: Gaming drivers are not optimized for data center workloads or multi-node training stability. In fact, using gaming drivers in an enterprise AI environment is unsupported and can lead to unpredictable behavior during long-running training jobs. Performance issues should be addressed by tuning network parameters, not updating to inappropriate driver versions.

nvidia-smi

Why it fails: nvidia-smi is a management and monitoring tool, not a debugger or code analyzer. It provides hardware status and utilization metrics but cannot inspect the state of a running kernel or detect logical errors like memory leaks, race conditions, or incorrect memory accesses in the application code.

nsys (Nsight Systems)

Why it fails: Nsight Systems is designed for performance profiling and visualization of the entire application timeline, including CPU and GPU activity. While it shows when a crash happens, it is not designed to perform the low-level functional analysis needed to identify specific memory leaks or data race conditions.

nvprof

Why it fails: nvprof is a legacy performance profiling tool. It is focused on collecting telemetry data about kernel execution time and memory throughput. It does not possess the functional analysis capabilities required to detect memory corruption, leaks, or race conditions within the logic of the custom CUDA kernel code.

Switch the inference precision from FP16 to FP32 to improve numerical stability.

Why it fails: Changing precision affects numerical accuracy and may alter performance, but it does not help identify why the GPU is stalled. FP32 typically reduces throughput and increases memory usage, which is counterproductive when the goal is to find the bottleneck. This action is unrelated to the observed serialization symptoms.

Enable MPS (Multi-Process Service) to allow concurrent kernel execution from multiple processes.

Why it fails: MPS is designed for concurrent execution across multiple processes, not for diagnosing a single inference service with serialized kernels. Enabling it here would add complexity without addressing the low memory controller utilization or the stalls. Profiling should come first to confirm whether concurrency is even the issue.

Increase the batch size in the inference server configuration to improve GPU occupancy.

Why it fails: Increasing batch size may raise throughput temporarily but does not diagnose the underlying serialization or latency issue. If the GPU is already reporting 100% utilization but low memory activity, blindly increasing batch size can worsen latency without fixing the root cause. This action skips the necessary profiling step and may mask the real problem.

Reset the GPU thermal throttling threshold.

Why it fails: Thermal throttling is a hardware safety mechanism that should not be adjusted to fix network latency issues. Lowering thresholds risks hardware damage, and increasing them does not address the communication delays inherent in the network fabric. Focus on fabric diagnostics rather than modifying critical hardware safety parameters.

Update the OS kernel to the latest version.

Why it fails: Blindly updating the kernel can introduce stability regressions in driver modules like OFED or NVIDIA GPU drivers. Troubleshooting should follow a structured approach starting with diagnostics rather than system-wide updates. Unnecessary kernel changes often complicate the environment, making it harder to identify the root cause of the latency.

Increase the system swap space to 512GB.

Why it fails: Increasing swap space is not a solution for InfiniBand network latency. Training jobs should avoid swapping entirely to maintain performance. Relying on swap indicates a lack of system memory, which requires addressing the workload footprint or adding physical RAM, not expanding disk-based virtual memory which is too slow.

The GPU driver is outdated and needs a patch.

Why it fails: Outdated drivers would show compatibility warnings or failures in initialization, not hardware-specific thermal slowdown flags. The logs explicitly point to hardware-level thermal events, which are independent of the software driver version. Prioritizing driver updates would ignore the physical cooling problem clearly indicated by the diagnostic tool.

The power supply unit is malfunctioning.

Why it fails: A failing PSU usually results in system instability, unexpected shutdowns, or power-related throttling flags, not thermal slowdowns. While power issues can cause heat, the logs specifically categorize this as a thermal event. The administrator must focus on cooling infrastructure rather than testing the electrical power delivery system.

The workload exceeds the GPU memory capacity.

Why it fails: Exceeding GPU memory capacity causes 'Out of Memory' (OOM) errors, leading to immediate process termination. Memory usage does not cause hardware thermal slowdowns. The exhibit clearly displays performance throttling flags, which are unrelated to the current allocation of VRAM by the running AI training or inference tasks.

Configure the InfiniBand fabric to use RoCEv2 instead of InfiniBand native protocol to improve compatibility with GPUDirect RDMA.

Why it fails: GPUDirect RDMA works over both InfiniBand and RoCE, but switching to RoCEv2 is not necessary and could introduce additional configuration complexity. The native InfiniBand protocol is typically preferred for its low latency and high throughput. This action does not verify or enforce GDR usage and may degrade performance if not properly configured.

Set NCCL_NET_GDR_LEVEL=0 to force NCCL to use GPUDirect RDMA for all inter-node communication.

Why it fails: NCCL_NET_GDR_LEVEL controls the threshold for using GPUDirect RDMA based on the distance between GPU and NIC. Setting it to 0 does not force GDR; it actually disables GDR because the level represents the maximum allowed distance. To enable GDR, the level should be set to a value that matches or exceeds the actual topology distance, such as 3 or 4. Thus, this action would likely disable GDR, not enforce it.

Run nvidia-smi topo -m on each node to check the GPU-to-NIC affinity and ensure that GPUs and InfiniBand adapters are connected via PCIe switches.

Why it fails: nvidia-smi topo -m shows the topology between GPUs and NICs, which is important for optimal performance, but it does not verify whether NCCL is actually using GPUDirect RDMA. It only shows the physical connectivity. While useful for diagnosing potential bottlenecks, it does not confirm the active transport or enforce its use.

Reinstall the NVIDIA driver on every node to refresh the kernel module.

Why it fails: The scenario states the driver loads correctly and the GPUs are healthy, so reinstalling it addresses a problem that is not present. It also risks unnecessary node disruption. The missing nvidia.com/gpu resource points to the device plugin registration path, not to the driver.

Increase the kubelet's pod density limit so the plugin can be scheduled.

Why it fails: Pod density limits could prevent scheduling in extreme cases, but the symptom of no allocatable GPU resource across all nodes is characteristic of the device plugin not registering, not of pod capacity. Raising limits would not create the missing resource.

Enable Multi-Instance GPU mode on all GPUs so each partition advertises capacity.

Why it fails: MIG changes how a GPU is partitioned and advertised, but enabling it does not fix a device plugin that is not registering at all. If the plugin is down, MIG instances also would not be advertised, so this step does not address the root cause.

Add a node selector that allows multiple pods to bind to the same nvidia.com/gpu resource slot.

Why it fails: A node selector only filters which nodes a pod may run on; it does not change how GPU resources are counted or allocated. The scheduler still treats each nvidia.com/gpu unit as consumable by exactly one pod unless a sharing mode is configured. Node selectors therefore cannot enable multiple pods to occupy the same physical GPU slot.

Create a ResourceQuota that counts GPU requests as fractional values such as 0.5 per pod.

Why it fails: Kubernetes does not support fractional extended resources like nvidia.com/gpu; requests must be whole integers unless a sharing mechanism changes how devices are advertised. A ResourceQuota merely enforces limits on those integer requests. Defining fractional values in quota will not let two pods share one GPU and will likely be rejected or ignored.

Raise the GPU Operator's driver version so the device plugin reports extra virtual GPUs per card.

Why it fails: Driver versions do not create additional schedulable GPU resources. The number of advertised devices is governed by the device plugin's configuration, such as MIG or time-slicing settings, not by the driver release. Upgrading the driver might fix compatibility issues, but it will not by itself allow more pods to share a single physical GPU.

Increase `NCCL_BUFFSIZE` to 16 MB and set `NCCL_NTHREADS` to 8 to raise channel throughput.

Why it fails: Tuning buffer size and thread counts can improve overlap, but it does not change the transport. Since the job is constrained to the management interface, the fabric bandwidth ceiling remains. Raising these values would yield marginal gains at best and could even increase memory pressure without addressing the missing RDMA path.

Set `NCCL_P2P_DISABLE=1` so that all inter-node traffic is routed through the CPUs, avoiding PCIe switch contention.

Why it fails: Disabling peer-to-peer would force traffic through host memory, which is the slow path the scenario is already suffering from. It does not restore the compute fabric and would likely worsen latency. This setting is used for debugging topology issues, not for recovering InfiniBand bandwidth.

Bind the job to a single NUMA node using `numactl --cpunodebind=0 --membind=0` to reduce cross-socket traffic.

Why it fails: NUMA binding can help CPU-side affinity and memory locality, but it cannot fix a NIC that is not being used. The bottleneck is transport selection, not CPU locality. Binding to one socket may even restrict access to NICs attached to the other socket and lower throughput further.

Disable GPUDirect RDMA by setting NCCL_NET_GDR_LEVEL=0 across all nodes to force host-memory staging.

Why it fails: Forcing NCCL_NET_GDR_LEVEL=0 changes the data path but does not diagnose the stall. If the fabric or NCCL interface binding is broken, staging through host memory will also hang. Disabling GPUDirect RDMA is a workaround to test after the transport in use has been identified, not a first diagnostic step.

Increase NCCL_BUFFSIZE to 16 MB on every rank and rerun the job to rule out buffer exhaustion.

Why it fails: NCCL_BUFFSIZE affects per-transfer chunking performance, not initialization. A job that never reaches gradient exchange has not begun collective communication, so enlarging buffers cannot resolve a connection or topology failure. This change would only be relevant after initialization succeeds and throughput is being tuned.

Reduce the number of ranks per node so each process has a dedicated GPU, then rerun the multi-node test.

Why it fails: Reducing ranks changes the job topology but does not explain why a working single-node configuration fails the moment a second node joins. This risks masking the real inter-node fault and wastes time. The team needs evidence about transport selection and connection progress before altering rank placement.

Disable NVLink by setting NCCL_P2P_DISABLE=1 to force PCIe communication.

Why it fails: Disabling NVLink would force communication over PCIe, which has lower bandwidth and higher latency, likely degrading performance. It might mask a NVLink hardware issue but does not diagnose it and sacrifices performance. This is a workaround, not a diagnostic step, and should only be considered after confirming NVLink is faulty.

Reinstall the NVIDIA driver and CUDA toolkit on all nodes.

Why it fails: Reinstalling drivers and CUDA is a heavy-handed action that may not address the root cause and could introduce new issues. It should be a last resort after identifying a software corruption. Without diagnostic data, this is unlikely to resolve an intermittent NCCL error and may waste time.

Reduce the number of GPUs used in the job to four to see if the error disappears.

Why it fails: Reducing GPU count might avoid the error if it's related to specific GPUs or NVLink topology, but it does not diagnose the problem and reduces training throughput. It is a troubleshooting step that could provide a clue, but it is not the first action; gathering logs is more informative and less disruptive.

The errors are benign and expected during large all-reduce jobs; suppress them by setting the DCGM health check to ignore XID 74 and 79.

Why it fails: XID 74 and 79 are not benign; they indicate serious GPU or NVLink faults that can cause job failures and silent data corruption. Suppressing them in DCGM health checks would hide a real hardware problem and violate operational best practice. The correct action is to investigate and remediate the fault, not to silence the monitoring that is correctly alerting on a failing component.

The errors indicate a thermal shutdown; immediately lower the GPU clock with `nvidia-smi -lgc` and rerun the job.

Why it fails: XID 74 and 79 are not thermal shutdown codes. Thermal issues typically produce XID 61 or clock-throttle events, and lowering clocks may mask a hardware fault rather than diagnose it. The engineer should first gather the full XID context and check the GPU's NVLink and PCIe health, because these XIDs point to a GPU or link-level fault, not a simple thermal condition that clock limiting would resolve.

The errors are caused by an outdated NCCL version; upgrade NCCL to the latest release and rerun the job without further hardware checks.

Why it fails: While NCCL version mismatches can cause job failures, XID 74 and 79 are GPU-level hardware and link errors reported by the driver, not software version issues. Upgrading NCCL would not address a GPU falling off the bus or an NVLink error. The engineer must first investigate the hardware state of the suspect GPU, because ignoring these XIDs can lead to data corruption and further node instability.

Restart the NVIDIA driver with rmmod and modprobe commands.

Why it fails: Restarting the driver might temporarily clear the error but does not diagnose the root cause. If the Xid error indicates a hardware failure, a driver restart could lead to data loss or further instability. It is not a diagnostic step but a remedial action. The first step should be to identify the fault via the Xid code before attempting any remediation.

Enable NCCL_DEBUG=INFO and re-run the job to capture detailed NCCL logs.

Why it fails: Enabling NCCL_DEBUG=INFO provides detailed communication logs but does not directly address Xid errors, which are GPU hardware or driver errors. While NCCL logs might show where the job failed, the Xid errors indicate a lower-level GPU issue. The first step should be to interpret the Xid errors, as they point to the root cause. NCCL debugging is useful but secondary.

Run nvidia-smi -q to check GPU temperature and power usage.

Why it fails: Checking temperature and power can reveal thermal or power throttling, but Xid errors are not typically caused by throttling. They indicate more serious faults like ECC errors or illegal memory access. While nvidia-smi -q is a good general health check, it does not decode the Xid error. The specific Xid code must be interpreted first to understand the fault.

Enable dynamic batching in Triton to automatically convert FP16 inputs to FP32.

Why it fails: Dynamic batching groups requests but does not perform data type conversion. It cannot fix a mismatch between client and model data types. The issue is a type mismatch, not batching, so this action would not resolve the incorrect predictions.

Set the model to use FP16 precision to match the client's input data type.

Why it fails: Changing the model to FP16 would align with the client's data type but may alter the model's numerical behavior and accuracy. The model was trained for FP32, so switching to FP16 could introduce precision loss and still not guarantee correct predictions. It is a risky workaround, not a proper fix.

Increase the instance count to handle the load and reduce the chance of data corruption.

Why it fails: Instance count affects concurrency and throughput, not data type correctness. Incorrect predictions due to FP16 versus FP32 inputs are not caused by insufficient instances. Adding instances would not address the root cause and could waste resources.