Troubleshooting and Optimization practice questions

Using the same static tensor size throughout the entire training process.

Why it fails: Using static tensor sizes is actually the best practice for preventing fragmentation. If all tensors are the same size, the allocator can reuse deallocated memory slots perfectly, keeping the memory layout clean and efficient throughout the duration of the training job, thus avoiding the fragmentation problem.

Running the training job on a system with insufficient host RAM.

Why it fails: Host RAM (CPU memory) is separate from VRAM (GPU memory). While insufficient CPU RAM causes swapping and system performance degradation, it does not directly drive the fragmentation of the GPU memory space itself, as those are two distinct hardware memory address spaces handled by different allocators.

--device=/dev/nvidia0

Why it fails: While this maps a specific device, it is a manual, non-portable approach. The modern standard uses the NVIDIA runtime, which automatically detects and maps all available GPUs and their associated character devices, ensuring that the container is portable across different NVIDIA hardware configurations without requiring manual device path mapping.

--privileged

Why it fails: Using --privileged grants the container elevated permissions on the host system, which is a significant security risk. It is not the correct mechanism for GPU access and should be avoided in production AI environments where hardware resource isolation and security are critical enterprise requirements.

--volume /usr/lib/nvidia

Why it fails: Manually mounting driver libraries is error-prone and does not handle the necessary device node mapping required for GPU communication. Using the NVIDIA runtime with the --gpus flag handles all necessary mounts and configurations, ensuring driver compatibility and proper communication between the container and the GPU hardware.

Restart the system BIOS to reset GPU clock frequencies.

Why it fails: BIOS settings for clock frequencies do not impact the software driver communication. This step is unrelated to the driver-level failure and would only result in unnecessary downtime without addressing the actual cause of the communication failure between the OS and the NVIDIA driver stack.

Reinstall the CUDA toolkit to update the compiler version.

Why it fails: The CUDA toolkit contains libraries and compilers, not the kernel-mode driver that manages the GPU device itself. Reinstalling the toolkit will not fix a failure in the driver's kernel interface, which is the root cause of the error message encountered when running nvidia-smi.

Increase the size of the swap partition on the disk.

Why it fails: Swap partition size is irrelevant to the communication between the NVIDIA driver and the hardware. While disk space is important for the system overall, it has no bearing on the kernel's ability to communicate with the GPU through the installed and loaded driver modules.

Maintain all master weights in FP64 to ensure maximum precision.

Why it fails: Maintaining master weights in FP64 is unnecessary and counterproductive. Standard practice is to keep them in FP32, which provides sufficient precision for the weight updates while keeping memory usage manageable. Using FP64 adds excessive memory and compute overhead without offering a meaningful benefit to the final model convergence.

Disable the use of the cuDNN library for convolution operations.

Why it fails: cuDNN is highly optimized for NVIDIA hardware and provides the fastest implementations of convolution operations. Disabling it would force the use of suboptimal, generic implementations, which would dramatically decrease throughput and negate the performance benefits gained from using mixed precision and specialized Tensor Core hardware.

Increase the physical RAM on the host server.

Why it fails: Increasing system RAM does not affect the VRAM capacity of the GPU. Since the error is a CUDA memory issue, the problem is localized to the GPU's onboard memory. Adding host memory will not solve the shortage of high-speed memory required by the GPU for the model weights.

Reinstall the NVIDIA display drivers.

Why it fails: Reinstalling drivers is an unlikely fix for a localized application-level memory issue. CUDA OOM errors are almost always caused by the application requesting more memory than available, not by the driver itself. This step is time-consuming and does not address the actual logic error in the code.

Switch to a larger GPU model.

Why it fails: Hardware upgrades should be a last resort after confirming the software is optimized. If there is a memory leak, a larger GPU will only delay the inevitable OOM error. Properly debugging the memory usage pattern is required to ensure that the software is operating efficiently within the hardware limits.

The container's network proxy settings.

Why it fails: Network proxy settings affect the ability to pull images or reach external APIs, but they do not influence the initialization of local GPU hardware acceleration. While important for pulling images, they are not a factor in whether the container successfully initializes its connection to the host NVIDIA GPU.

The local disk storage capacity.

Why it fails: Disk capacity concerns the persistence of data and logs, not the runtime initialization of hardware-accelerated containers. While a full disk can cause application failures, it is not a direct cause of a failure to initialize GPU drivers or CUDA libraries during the initial container startup phase.

The InfiniBand cabling is loose.

Why it fails: Loose cabling usually results in 'link down' or 'CRC error' flags, not a total inability to find the HCA on the PCIe bus. If the OS cannot see the HCA hardware on the bus, the driver has not probed the device, which is a software or bus-level initialization issue.

The NCCL version is incompatible with the GPU.

Why it fails: NCCL version issues typically manifest as 'compatibility' or 'feature not supported' errors during collective calls. The specific 'NET_IB_NOT_FOUND' error points to a network device detection failure rather than a versioning conflict between the software and the GPU compute architecture. The error is isolated to the network hardware probing layer.

The GPU memory is insufficient for NCCL buffers.

Why it fails: Insufficient memory would cause OOM errors during the buffer allocation phase of NCCL initialization. It would not cause the system to fail at finding the InfiniBand HCA hardware. The error is explicitly related to network subsystem discovery, not the memory availability on the local GPU or host system.

The batch size is too large, causing memory fragmentation.

Why it fails: Excessive batch sizes typically lead to out-of-memory errors rather than hardware-level clock throttling. Memory fragmentation affects allocation success rates but does not inherently trigger the power management unit to downclock the GPU core frequency unless extreme thermal load from memory controller activity is present.

The NVLink fabric is disconnected on one GPU in the cluster.

Why it fails: NVLink connectivity issues typically manifest as decreased inter-GPU communication bandwidth or kernel launch failures. This does not cause the individual GPU core clocks to throttle, as the communication fabric is independent of the compute engine's power management and clock frequency regulation logic.

The CUDA driver version is incompatible with the installed cuDNN library.

Why it fails: Software library incompatibilities lead to runtime exceptions, incorrect output, or failure to initialize the CUDA context. These issues do not influence the hardware power management firmware, which is responsible for clock throttling based on physical parameters like temperature and power consumption metrics.

Increase the GPU power limit to provide more voltage stability.

Why it fails: Increasing the power limit is not a standard solution for XID 79 errors. XID 79 is typically software-driven or indicative of a specific kernel fault. Modifying power limits will not fix driver-level instruction processing errors and may introduce further thermal instability into the system.

Replace the PCIe riser card and re-seat the GPU hardware.

Why it fails: Physical hardware interventions like replacing risers are reserved for XID errors related to link training or PCIe bus resets, such as XID 31 or 43. XID 79 is primarily an error related to the GPU engine's internal instruction processing, not the physical PCIe interconnect.

Lower the precision to FP16 to reduce the computational complexity.

Why it fails: Changing precision is a performance optimization, not a troubleshooting step for system-level errors. If the driver is encountering an XID 79 error, changing the data format will not address the underlying software or firmware exception being reported by the GPU during the kernel execution process.

The total system CPU utilization percentage.

Why it fails: CPU utilization is secondary to GPU performance in deep learning. While high CPU usage can impact data loading, it is not a direct measure of inter-node synchronization latency and does not help in identifying a straggler node compared to GPU-specific communication and execution metrics.

The ambient server room temperature.

Why it fails: Ambient temperature affects the whole rack and is a holistic environmental factor. It does not provide the granularity required to identify a specific straggler node, as individual nodes are usually subjected to the same cooling environment within a standard managed rack or DGX pod.

nvidia-smi

Why it fails: nvidia-smi provides high-level metrics like overall VRAM usage and GPU utilization. It does not provide the granular, kernel-level visibility required to identify specific memory leaks within custom CUDA code, making it unsuitable for debugging internal memory management issues within a kernel.

NCCL tests

Why it fails: NCCL tests are used exclusively for benchmarking collective communication performance across a cluster. They have no functionality related to inspecting internal memory usage or debugging memory leaks within individual CUDA kernels, as their focus is on network-level throughput rather than device-level memory allocation.

DCGM Exporter

Why it fails: DCGM is a monitoring tool for telemetry and operational metrics in a production cluster. It is not designed for debugging or code-level profiling of custom kernels and cannot provide the instruction-level data necessary to track down memory leaks within a specific CUDA implementation.

Increase the batch size significantly to utilize all available VRAM.

Why it fails: Increasing batch size further might exacerbate the situation if the bottleneck is compute-bound, as it does not address the underlying inefficiency of the compute kernels. If memory bandwidth is already low, adding more data per batch will likely increase latency without improving the actual throughput of the processing pipeline.

Enable mixed precision training using FP16 or BF16.

Why it fails: While mixed precision typically improves throughput, it primarily targets memory bandwidth and compute throughput simultaneously. If the system is already compute-bound with low memory bandwidth, switching to FP16 might not resolve the underlying kernel inefficiency causing the imbalance between compute utilization and actual data throughput.

Upgrade the system RAM to provide faster data access to the GPU.

Why it fails: System RAM speed is rarely the primary bottleneck for GPU training once data is staged in the GPU memory. Upgrading host RAM does not improve the efficiency of compute kernels executing on the GPU. The bottleneck is likely in how the GPU processes the data currently available in its VRAM.

nvidia-smi -q

Why it fails: The nvidia-smi tool provides a snapshot of current GPU status. While useful for immediate checks on a single node, it lacks the persistent monitoring, alerting, and fleet-wide aggregation capabilities required for professional-grade AI infrastructure management compared to more robust tools like DCGM.

nv-host-monitor

Why it fails: This is not a standard NVIDIA-provided tool for GPU telemetry. Relying on non-standard or custom tools for critical infrastructure monitoring can lead to incomplete data visibility, making it difficult to perform root-cause analysis when performance issues or hardware failures occur within the AI compute environment.

cat /proc/driver/nvidia/gpus/0/information

Why it fails: Accessing driver files directly is a low-level method that does not provide real-time thermal, power, or utilization metrics. It is not suitable for fleet monitoring or troubleshooting and does not offer the depth of information required to manage complex DGX systems effectively.

Run 'nvidia-smi' on the node to verify GPU driver installation.

Why it fails: While verifying drivers is important, 'nvidia-smi' does not explain why the Kubernetes scheduler is reporting insufficient resources. The scheduling issue is specifically managed by the NVIDIA device plugin, not the underlying driver state, assuming the host-level driver is already functional and correctly reporting hardware to the kernel.

Increase the GPU memory limit in the container manifest.

Why it fails: Increasing memory limits does not resolve scheduling issues related to 'insufficient resources'. It only affects the pod's runtime configuration. If the scheduler cannot find a node with available GPU resources, changing container parameters will not force the scheduler to place the pod on an ineligible node.

Restart the kubelet service on all worker nodes.

Why it fails: Restarting the kubelet is a drastic measure that may disrupt running services. The root cause for resource scheduling failures is typically related to device plugin configuration or node labels rather than the kubelet process itself, which is rarely the source of resource reporting issues in this context.

Disable all ECC memory reporting features.

Why it fails: Disabling ECC is counterproductive as it masks memory errors. ECC is designed to detect and correct single-bit flips, and reporting them is crucial for diagnosing failing hardware. Turning it off removes valuable diagnostic data that could pinpoint if the hang is caused by physical memory degradation.

Run the training loop on a single CPU core.

Why it fails: Running a model on a single CPU core is not a valid reproduction step for a GPU hang. It eliminates the GPU from the execution pipeline, which is the exact opposite of what is required to reproduce the hardware-specific failure occurring during the training loop execution.

Dynamic Batching

Why it fails: Dynamic batching groups individual inference requests into a single batch to maximize GPU throughput. While it improves overall throughput, it can introduce latency variability as it waits to fill the batch, which is the opposite of the objective to stabilize latency spikes during inference.

Instance Grouping

Why it fails: Instance grouping allows multiple copies of a model to run on the same GPU. While this improves concurrency, it does not address the underlying issue of memory allocation overhead that causes spikes in per-request latency. It simply scales the server's capacity to handle more concurrent inference requests.

Model Ensembling

Why it fails: Model ensembling allows chaining multiple models together in a single pipeline. This is a workflow feature and does not provide mechanisms for memory optimization or latency stabilization. It actually increases the complexity of the pipeline, potentially leading to even more latency variability if not managed correctly.

Set the NCCL_IB_DISABLE environment variable to 1 to prevent NCCL from attempting to use InfiniBand and fall back to Ethernet.

Why it fails: Disabling InfiniBand forces NCCL to use Ethernet, which is the opposite of what is needed. The job is already using Ethernet and suffering performance loss. This action would cement the bottleneck and likely worsen multi-node scaling, not resolve it.

Increase the NCCL_BUFFSIZE environment variable to 64 MB to improve throughput over Ethernet.

Why it fails: NCCL_BUFFSIZE controls the size of communication buffers but does not change the underlying transport. Tuning buffer size on a slow Ethernet fabric may yield marginal gains but cannot match InfiniBand bandwidth. The root cause is transport selection, not buffer sizing, so this is a misdirected optimization.

Run `nvidia-smi nvlink -s` to verify NVLink status and then restart the job.

Why it fails: NVLink status is already confirmed healthy within each node via topology output. The issue is inter-node communication, not intra-node NVLink. Checking NVLink again does not address the Ethernet versus InfiniBand selection problem and will not change the job's network path.

Increase the GPU's memory clock using nvidia-smi -ac to improve memory bandwidth and reduce latency.

Why it fails: Overclocking memory can improve bandwidth but does not address fragmentation. It may also void warranties or cause instability. This option is a hardware tweak that does not solve the software-level issue of fragmented memory. It would not confirm the cause and could introduce new problems, making it an incorrect choice.

Enable CUDA_MODULE_LOADING=LAZY and monitor GPU memory usage with nvidia-smi.

Why it fails: Lazy module loading reduces startup time by deferring kernel loading, not memory fragmentation during steady-state inference. Monitoring with nvidia-smi shows overall memory but not fragmentation patterns. This option misidentifies the cause and does not mitigate the issue; it would not confirm or resolve the latency spikes.

Use NVIDIA Nsight Systems to capture a timeline and check for cudaMalloc/cudaFree calls during inference.

Why it fails: Nsight Systems does show API calls, but cudaMalloc/cudaFree are not the primary cause of fragmentation in long-running inference; frameworks typically use caching allocators. The tool would not directly confirm fragmentation, and the mitigation would not be clear. This option is plausible but does not address the root cause or provide a definitive confirmation method.

Set NCCL_BUFFSIZE to a larger value to increase the buffer size for communication.

Why it fails: Increasing NCCL_BUFFSIZE can sometimes help with large messages, but the scenario indicates that the all-reduce is taking longer than compute, suggesting an algorithmic or topological mismatch rather than buffer size. The ring algorithm may be suboptimal for the DGX NVLink topology. Simply increasing buffer size is unlikely to resolve the scaling issue.

Set NCCL_P2P_LEVEL=SYS to force peer-to-peer communication over the system bus.

Why it fails: NCCL_P2P_LEVEL controls the level at which peer-to-peer communication is attempted. Setting it to SYS would force communication over the slower system bus (PCIe) instead of NVLink, degrading performance. The system already has NVLink, so this would not help and would likely worsen the all-reduce time.

Set NCCL_IB_DISABLE=1 to disable InfiniBand and use Ethernet instead.

Why it fails: Disabling InfiniBand would force communication over Ethernet, which is typically slower and higher latency. The scenario describes a DGX system with NVLink, so intra-node communication should use NVLink. Disabling IB is irrelevant for intra-node all-reduce and could harm inter-node communication if present. This change would not improve scaling efficiency.

Configure Triton to use pinned memory for input tensors to speed up data transfer.

Why it fails: Pinned memory can reduce host-to-device transfer latency, but it does not reduce GPU memory usage for model execution. The out-of-memory errors are due to GPU memory exhaustion during inference, not transfer bottlenecks. This change would not alleviate memory pressure or significantly improve throughput.

Increase the instance group count to 2 to allow concurrent model execution.

Why it fails: Increasing instance count creates multiple copies of the model, each consuming additional GPU memory. With memory already above 90%, this would likely cause more out-of-memory errors. While concurrency can improve throughput, it exacerbates memory pressure and is not suitable here.

Set the model to use FP32 precision instead of FP16 to reduce memory usage.

Why it fails: FP32 precision uses more memory per parameter than FP16, so switching to FP32 would increase memory consumption, not reduce it. This would worsen the out-of-memory condition and likely decrease throughput due to higher computational load. It is the opposite of what is needed.