NCP-AIO Troubleshooting and Optimization Practice Test 6 — 15 Questions

Question 1

An AI operations team is troubleshooting a distributed training job on a cluster of NVIDIA DGX A100 systems connected via InfiniBand. The job runs but achieves only 40% of expected scaling efficiency. The team suspects communication bottlenecks. Which two actions should they take to confirm and address the issue? (Choose two.)

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Verify that all nodes have identical NCCL versions and environment variables, and that InfiniBand fabric manager is running.. Scaling inefficiency in distributed training often stems from communication overhead. NCCL debug logs directly show transport selection and errors, while ensuring consistent NCCL versions and InfiniBand fabric health addresses common misconfigurations. Together, these actions confirm and resolve bottlenecks. The other options either do not diagnose the network or would degrade performance.

Answer

Switch the job to use Ethernet instead of InfiniBand to simplify troubleshooting.

Answer

Use nvidia-smi to monitor GPU utilization and memory bandwidth on each node during training.

Answer

Increase the global batch size proportionally to the number of GPUs to improve compute-to-communication ratio.

Question 2

A team is running a multi-GPU training job on an NVIDIA DGX A100 system using NCCL for inter-GPU communication. Training throughput is much lower than expected, and the NCCL logs show repeated 'NCCL WARN Call to ibv_reg_mr failed' errors. The job uses a container with host networking. Which action should the AI operations engineer take to resolve the issue?

Accepted Answer

Verify and increase the container's locked memory limit (ulimit -l) and ensure the NVIDIA driver and OFED stack are compatible.. The NCCL warning 'ibv_reg_mr failed' points to an inability to register memory regions for InfiniBand, commonly caused by a low locked memory limit or mismatched OFED and NVIDIA drivers. In containerized environments, the default ulimit -l is often too low. Increasing the locked memory limit and ensuring driver compatibility directly fixes the registration failure, restoring proper NCCL communication and training throughput.

Answer

Reduce the batch size per GPU to lower memory pressure and avoid the need for large memory registrations.

Answer

Increase the NCCL_IB_TIMEOUT environment variable to allow more time for memory registration.

Answer

Disable InfiniBand by setting NCCL_IB_DISABLE=1 to force NCCL to use TCP sockets.

Question 3

An operations team is troubleshooting a multi-node NCCL training job on an NVIDIA DGX SuperPOD that stalls at initialization and never begins gradient exchange. Running `nccl-tests` with `all_reduce_perf` on the same nodes fails identically, but single-node `all_reduce_perf` succeeds. Which action should the team take first to isolate the fault?

Accepted Answer

Set NCCL_DEBUG=INFO and NCCL_DEBUG_SUBSYS=INIT,COLL,NET on the launcher and inspect which transport (NET/IB or P2P) is selected and where the hang occurs.. Because single-node all_reduce_perf succeeds while multi-node fails, the failure is in inter-node communication setup rather than GPU-local collectives. Enabling NCCL_DEBUG=INFO with INIT, COLL, and NET subsystems exposes which transport NCCL selects, how rings and trees are built, and where connection establishment stalls, giving the team actionable evidence before any configuration change.

Answer

Disable GPUDirect RDMA by setting NCCL_NET_GDR_LEVEL=0 across all nodes to force host-memory staging.

Answer

Increase NCCL_BUFFSIZE to 16 MB on every rank and rerun the job to rule out buffer exhaustion.

Answer

Reduce the number of ranks per node so each process has a dedicated GPU, then rerun the multi-node test.

Question 4

An AI operations engineer notices that a real-time inference service on an NVIDIA T4 GPU has highly variable latency, with occasional spikes to over 100 ms. The service uses TensorRT and runs in a Docker container. Which action should the engineer take to reduce latency variability?

Accepted Answer

Lock the GPU clocks to a fixed frequency using nvidia-smi -lgc.. Latency variability in GPU inference often results from dynamic clock adjustments as the GPU balances power and thermal constraints. Locking clocks to a stable frequency eliminates this variability, providing consistent execution times. Other options either add latency (dynamic batching), address different issues (persistence mode), or do not target GPU clock behavior.

Answer

Enable dynamic batching in the inference server configuration.

Answer

Increase the number of CPU cores allocated to the container.

Answer

Set the GPU to persistence mode using nvidia-smi -pm 1.

Question 5

An operations team runs a multi-node NCCL all-reduce training job across four DGX nodes connected via InfiniBand. Training throughput is far below the expected linear scaling, and `nvidia-smi` shows GPU utilization oscillating between 20% and 40%. The network fabric is healthy and the GPUs are not thermally throttled. Which diagnostic step is MOST appropriate to identify the bottleneck?

Accepted Answer

Run `nccl-tests` (all_reduce_perf) with varying message sizes and collect NCCL debug logs by setting NCCL_DEBUG=INFO to inspect topology and algorithm selection.. The oscillating, low GPU utilization across multiple nodes during an all-reduce strongly suggests the collective communication is stalling. Running `nccl-tests` with multiple message sizes measures achieved bandwidth and highlights which sizes are inefficient, while NCCL_DEBUG=INFO reveals the detected topology, algorithm, and channels. Together these pinpoint whether the bottleneck is the fabric, the ring/tree algorithm, or an unexpected topology detection.

Answer

Reinstall the CUDA toolkit on all nodes to ensure matching driver and runtime versions.

Answer

Increase the training batch size by 4x to raise arithmetic intensity and re-measure GPU utilization.

Answer

Enable ECC memory scrubbing on all GPUs and monitor for corrected errors during training.

Question 6

An AI operations team is troubleshooting a distributed training job on a cluster of NVIDIA DGX A100 systems connected via InfiniBand. The job runs but achieves only 40% of expected scaling efficiency. The team suspects communication bottlenecks. Which two actions should they take to confirm and address the issue? (Choose two.)

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Set NCCL_DEBUG=INFO and NCCL_DEBUG_SUBSYS=INIT,COLL to capture detailed NCCL initialization and collective logs.. Intermittent hangs in NCCL all-reduce often stem from configuration or network issues. Enabling detailed NCCL logging helps identify the exact failure point, while verifying consistent NCCL versions and network interface health addresses common root causes. These two actions together provide both diagnostic information and a path to resolution without degrading performance.

Answer

Disable the use of InfiniBand and force NCCL to use TCP sockets for communication.

Answer

Restart the training job with a smaller number of GPUs to see if the hang persists.

Answer

Increase the batch size to reduce the frequency of all-reduce operations.

Question 7

An AI operations engineer is optimizing a real-time inference pipeline on an NVIDIA T4 GPU. The pipeline uses TensorRT and receives requests with variable input sizes. Profiling shows that the engine recompiles for each new input shape, causing latency spikes. Which optimization should the engineer apply to eliminate recompilation while maintaining acceptable accuracy?

Accepted Answer

Enable dynamic shaping in TensorRT by defining an optimization profile with min, opt, and max shapes.. TensorRT dynamic shaping with an optimization profile allows one engine to handle a range of input shapes, eliminating the need to rebuild the engine for each new size. This directly removes the latency spikes from recompilation while maintaining performance across variable inputs. Other options either do not address recompilation or trade off too much efficiency or accuracy.

Answer

Pad all input tensors to a fixed maximum size before inference.

Answer

Convert the model to use INT8 precision with a calibration dataset to reduce inference time.

Answer

Increase the workspace size allocated for TensorRT to allow more kernel autotuning.

Question 8

A data scientist reports that a PyTorch training job on an NVIDIA V100 GPU is running slower than expected. The job uses a data loader with num_workers=4. Monitoring shows GPU utilization is around 50%, and CPU usage is high. Which action should an AI operations engineer recommend to improve GPU utilization?

Accepted Answer

Increase num_workers in the DataLoader to overlap data preprocessing with GPU computation.. The combination of high CPU usage and low GPU utilization points to a data loading bottleneck where the GPU is starved for data. Increasing the number of DataLoader workers enables more parallel data preprocessing, allowing the GPU to be fed continuously. Other options do not address the CPU-bound pipeline and may not improve the situation.

Answer

Pin memory in the DataLoader to speed up host-to-device transfers.

Answer

Enable mixed precision training to speed up computations.

Answer

Increase the batch size to better utilize the GPU.

Question 9

An AI operations engineer is troubleshooting a multi-node NCCL training job on an NVIDIA DGX SuperPOD. The job runs but scales poorly: inter-node bandwidth is roughly half of the expected 200 Gb/s per GPU, while intra-node NVLink traffic is at full rate. Running `nvidia-smi topo -m` shows that GPUs in each node are connected to the NICs through the PCIe switch, but the job sets `NCCL_SOCKET_IFNAME` to the management interface. Which action is the most appropriate to resolve the bottleneck?

Accepted Answer

Enable GPUDirect RDMA by allowing the container access to the host IB verbs devices and the `/dev/infiniband` tree, and set `NCCL_IB_HCA` to the compute fabric adapters.. When intra-node NVLink performs at full rate but inter-node bandwidth is roughly half, the job is usually not using the compute fabric. GPUDirect RDMA bypasses host memory copies by letting the HCA access GPU memory directly, and `NCCL_IB_HCA` ensures NCCL selects the correct adapters. Exposing `/dev/infiniband` to the container is required for RDMA verbs. This restores the expected inter-node bandwidth.

Answer

Increase `NCCL_BUFFSIZE` to 16 MB and set `NCCL_NTHREADS` to 8 to raise channel throughput.

Answer

Set `NCCL_P2P_DISABLE=1` so that all inter-node traffic is routed through the CPUs, avoiding PCIe switch contention.

Answer

Bind the job to a single NUMA node using `numactl --cpunodebind=0 --membind=0` to reduce cross-socket traffic.

Question 10

A machine learning engineer is optimizing a recommendation model for inference on an NVIDIA T4 GPU. The model uses dynamic input shapes, and profiling shows that kernel launch overhead is a significant contributor to latency. Which optimization technique should be applied to reduce this overhead?

Accepted Answer

Convert the model to TensorRT with dynamic shapes and enable CUDA graphs during inference.. CUDA graphs reduce kernel launch overhead by capturing a sequence of operations and replaying them as a single graph. TensorRT with dynamic shapes can leverage CUDA graphs to maintain flexibility while minimizing overhead. This directly addresses the profiling finding, making it the most effective optimization for the described scenario.

Answer

Use mixed precision (FP16) to reduce the number of kernels executed.

Answer

Enable NVIDIA MPS (Multi-Process Service) to allow concurrent kernel execution.

Answer

Increase the batch size to amortize kernel launch overhead across more samples.

Question 11

A production inference service on NVIDIA GPUs reports that p99 latency spikes every few minutes while p50 remains stable. Metrics show GPU memory utilization near the limit and periodic `cudaMalloc` calls in the application logs. The model and batch size are fixed. Which change is MOST likely to eliminate the latency spikes?

Accepted Answer

Pre-allocate the inference workspace and reuse CUDA memory pools instead of calling cudaMalloc during request handling.. The correlation between p99 spikes and periodic `cudaMalloc` calls, combined with stable p50, points to allocation occurring inside the request path. Synchronous allocation can serialize GPU work and introduce jitter. Pre-allocating workspaces and using a memory pool or CUDA graphs with static buffers removes the allocator from the hot path, eliminating the periodic stalls while leaving steady-state latency unchanged.

Answer

Enable CUDA lazy module loading to defer kernel module initialization until first use.

Answer

Increase the number of model instances per GPU to smooth out request distribution.

Answer

Lower the GPU clock to reduce power draw and stabilize thermals during inference.

Question 12

An AI operations engineer is troubleshooting a multi-GPU training job that fails intermittently with a NCCL error: 'unhandled system error'. The job runs on a DGX-1 with eight V100 GPUs connected via NVLink. Which step should the engineer take first to resolve the issue?

Accepted Answer

Set NCCL_DEBUG=INFO and reproduce the failure to capture detailed logs.. The first step in troubleshooting any NCCL error should be to gather detailed logs using NCCL_DEBUG=INFO. This provides visibility into the communication path and error specifics, enabling targeted fixes. Other options either mask the issue, involve unnecessary reinstallation, or reduce resources without diagnosis.

Answer

Disable NVLink by setting NCCL_P2P_DISABLE=1 to force PCIe communication.

Answer

Reinstall the NVIDIA driver and CUDA toolkit on all nodes.

Answer

Reduce the number of GPUs used in the job to four to see if the error disappears.

Question 13

A team is diagnosing a training job that intermittently stalls for several seconds at the start of each epoch. The job uses a distributed data loader and an NVIDIA DGX system with local NVMe. Monitoring shows GPU utilization dropping to near zero during the stalls while host CPU utilization spikes. Which two actions should the AI operations engineer take to identify and mitigate the stall? (Choose two.)

Accepted Answer

Use `nvidia-smi dmon` to capture per-second GPU utilization and power draw while the job runs, correlating the drop with the stall window.. The signature of host CPU spikes with idle GPUs is input pipeline starvation. Prefetching with multiple worker processes and pinned memory keeps the device fed, removing the serialized load at epoch boundaries. Concurrently, `nvidia-smi dmon` supplies the time-series evidence that ties the utilization dips to the stall windows, confirming the diagnosis before and after the fix.

Answer

Increase the batch size until the GPU memory is nearly full to amortize the data loading cost.

Answer

Set `CUDA_LAUNCH_BLOCKING=1` to serialize kernel launches and make the stall easier to observe.

Answer

Move the dataset from local NVMe to a networked file system to rule out local disk contention.

Question 14

A distributed training job on a multi-GPU node is exhibiting poor scaling efficiency: each GPU shows high utilization, but overall throughput increases by only 15% when doubling the number of GPUs. The job uses NCCL for communication. Which diagnostic step is most appropriate to identify the bottleneck?

Accepted Answer

Run `nvidia-smi topo -m` to inspect the GPU interconnect topology and verify that NCCL is using the fastest available path between GPUs.. Poor scaling efficiency in distributed training often results from communication bottlenecks. The `nvidia-smi topo -m` command provides a matrix of GPU interconnect paths, helping verify whether NCCL can use NVLink or is forced over slower PCIe/QPI links. If the topology shows suboptimal paths, adjusting NCCL environment variables or hardware configuration can improve scaling. Thus, inspecting the topology is the most direct diagnostic step.

Answer

Enable NVIDIA MPS (Multi-Process Service) to allow multiple processes to share the GPUs more efficiently.

Answer

Check the GPU clock speeds with `nvidia-smi -q -d CLOCK` to ensure they are running at maximum frequency.

Answer

Increase the batch size per GPU to reduce the frequency of gradient synchronization across GPUs.

Question 15

An AI operations engineer is investigating a training job on an NVIDIA DGX system that intermittently fails with 'uncorrectable ECC error' on a GPU. The job is using NCCL for multi-GPU communication. The engineer needs to identify the appropriate immediate actions to diagnose and mitigate the issue. (Choose two.)

Accepted Answer

Drain the affected GPU from the scheduler to prevent new jobs from being assigned.. The immediate actions should include checking ECC error counts with nvidia-smi -q to identify the affected GPU, and draining that GPU from the scheduler to prevent further job failures. These steps diagnose the issue and mitigate risk while preserving the ability to perform maintenance. Restarting, resetting, or adjusting NCCL timeouts do not address the underlying hardware error and may delay proper resolution.

Answer

Increase the NCCL timeout to allow the job to recover from the ECC error.

Answer

Restart the training job immediately to see if the error recurs.

Answer

Use nvidia-smi --gpu-reset to reset the GPU and clear the error state.