NCP-AIO Troubleshooting and Optimization Practice Test 7 — 15 Questions

Question 1

A monitoring system reports that a DGX node's GPUs are running at reduced clocks during a long training job, and `nvidia-smi -q -d PERFORMANCE` shows the throttle reason as 'SW Power Cap'. The job's power draw is at the configured limit. Which action is MOST appropriate to restore higher clocks?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Raise the GPU power limit with `nvidia-smi -pl` to a value within the GPU's supported maximum and re-run the job.. The throttle reason 'SW Power Cap' indicates the GPU is constrained by its configured power limit. Raising the power limit to a supported value within the GPU's validated maximum allows the device to sustain higher clocks, provided cooling and power delivery can support it. This directly removes the cause of the reduced clocks rather than working around it.

Answer

Reduce the batch size so the GPU consumes less power and stays below the cap.

Answer

Enable persistence mode with `nvidia-smi -pm 1` to keep the driver loaded and improve clock stability.

Answer

Lock the GPU clocks to their maximum with `nvidia-smi -lgc` to override the throttle.

Question 2

An operations engineer is investigating a sudden drop in throughput for a multi-GPU training job on an NVIDIA DGX A100. The job uses PyTorch with DDP. Logs show that one GPU is consistently at 100% utilization while others are below 50%. Which tool and approach should be used to identify the bottleneck?

Accepted Answer

Use NVIDIA Nsight Systems to profile the job and examine the CUDA API and kernel timeline to identify serialization or synchronization issues.. Nsight Systems is designed for system-level profiling, capturing CPU and GPU activities across multiple devices. It can show if one GPU is performing more work or if others are idle waiting for synchronization. This makes it the right tool to identify the bottleneck in a DDP job where one GPU is overutilized. Profiling before making changes is essential.

Answer

Use nvidia-smi to monitor GPU utilization and memory, then rebalance the workload by reducing the batch size on the overutilized GPU.

Answer

Use NVIDIA Nsight Compute to profile each GPU's kernels and compare execution times to find the slowest kernel.

Answer

Use nvidia-smi topo -m to check the GPU topology and then adjust the NCCL communication settings to use a different ring order.

Question 3

An AI operations engineer is investigating an inference service running on NVIDIA Triton Inference Server. Clients report sporadic 500 errors under peak load. The server logs show occasional 'Failed to allocate memory' messages, and `nvidia-smi` shows VRAM nearly full. The service uses dynamic batching with a maximum batch size of 64 and multiple model instances per GPU. Which change is the most appropriate first step to stabilize the service?

Accepted Answer

Reduce the number of model instances per GPU and lower the maximum dynamic batch size to match the available VRAM headroom.. Triton allocates memory per model instance and per active batch execution. When instances and the maximum dynamic batch size are sized for average load, peak concurrency can exceed VRAM and cause allocation failures. Reducing instances and capping batch size brings the peak footprint within budget, letting the scheduler queue requests instead of failing them, which stabilizes error rates under load.

Answer

Switch the service to use the CPU execution provider for overflow requests when GPU memory is exhausted.

Answer

Enable `--strict-model-config=false` so Triton can auto-generate the model configuration and allocate memory more efficiently.

Answer

Increase the GPU memory pool by setting `CUDA_MPS_ACTIVE_THREAD_PERCENTAGE` to 100.

Question 4

A production inference service running on NVIDIA T4 GPUs shows that GPU utilization is consistently below 20% while request latency is high. Profiling with Nsight Systems reveals that the model execution time is short but there are frequent gaps between kernels. Which optimization should be applied first to improve GPU utilization?

Accepted Answer

Increase the batch size in the inference server configuration.. The low GPU utilization and gaps between kernels indicate that the GPU is not receiving enough work per launch. Increasing the batch size allows more data to be processed per kernel, filling the gaps and raising utilization. This is the most direct and effective first step before considering more complex techniques like CUDA graphs or precision changes.

Answer

Switch from FP32 to FP16 precision for the model weights.

Answer

Increase the number of concurrent model instances on each GPU.

Answer

Enable CUDA graphs to capture and replay the inference sequence.

Question 5

An AI operations engineer is deploying a model on an NVIDIA GPU and notices that inference latency is higher than expected. The model uses a batch size of 1, and profiling shows that the GPU is idle between kernel launches. Which optimization technique should the engineer use to reduce latency by overlapping data transfer with computation?

Accepted Answer

Use CUDA streams to overlap host-to-device memory copies with kernel execution.. The idle time between kernel launches indicates that data transfers and kernel executions are serialized. Using CUDA streams to overlap host-to-device copies with kernel execution can hide transfer latency and reduce overall inference time. This is the correct technique for overlapping data transfer with computation, especially with small batch sizes where transfer overhead is significant.

Answer

Enable NVIDIA MPS to allow multiple processes to share the GPU.

Answer

Switch to a lower-precision data type such as FP16 to reduce memory transfer size.

Answer

Increase the batch size to improve GPU utilization and reduce per-inference latency.

Question 6

A team is profiling a distributed training job using NVIDIA NCCL for inter-GPU communication on a DGX A100 system. They observe that all-reduce operations are taking longer than expected, and the NCCL debug logs show frequent 'NVLS' (NVLink SHARP) errors. Which action should be taken to resolve the issue?

Accepted Answer

Disable NVLink SHARP by setting NCCL_NVLS_ENABLE=0 in the environment and restart the job.. NVLS errors indicate that the NVLink SHARP acceleration is failing, likely due to hardware or software incompatibility. Disabling NVLS via the NCCL_NVLS_ENABLE environment variable forces NCCL to use standard algorithms, which are robust and should eliminate the errors. Other options either do not target NVLS or are less direct and potentially disruptive.

Answer

Increase the NCCL buffer size by setting NCCL_BUFFSIZE to a larger value to reduce the number of messages.

Answer

Update the GPU driver and CUDA toolkit to the latest versions to fix NVLS bugs.

Answer

Switch the NCCL algorithm to 'Tree' by setting NCCL_ALGO=Tree to avoid NVLS usage.

Question 7

An operations team observes that a distributed training job using NVIDIA Collective Communications Library (NCCL) across eight nodes occasionally hangs during the all-reduce phase. Logs show no errors, and the hang resolves only after a node is manually restarted. Which action is MOST appropriate to diagnose the intermittent hang?

Accepted Answer

Enable NCCL debug logging with NCCL_DEBUG=INFO and NCCL_DEBUG_SUBSYS=COLL, and set a NCCL watchdog timeout to capture the stalled collective.. Intermittent NCCL hangs without errors are best diagnosed by capturing state at the moment of the stall. Enabling NCCL debug logging for the collective subsystem and configuring a watchdog timeout produces logs that show which rank and collective stopped progressing. This evidence is necessary to distinguish a slow rank, a fabric problem, or a software defect from a simple performance issue.

Answer

Disable InfiniBand and fall back to TCP sockets by setting NCCL_IB_DISABLE=1.

Answer

Increase the NCCL buffer size with NCCL_BUFFSIZE to reduce the number of messages exchanged.

Answer

Set NCCL_ALGO=RING to force a single algorithm and eliminate variability.

Question 8

A team trains a model inside an NGC PyTorch container on a DGX H100 node. Training starts, but after a few minutes the process dies and `dmesg` shows `Xid 79: GPU has fallen off the bus` on one GPU. The team needs to determine whether the fault is hardware or software before opening an RMA. Which two actions should they take to gather useful evidence? (Choose two.)

Accepted Answer

Run `nvidia-smi -q` and `nvidia-smi -q -d ECC,PAGE_RETIREMENT,ROW_REMAPPER` and capture the output.. Xid 79 indicates the GPU dropped off the PCIe bus, which can stem from a hardware fault, a degraded link, or a driver escalation after ECC or row-remap events. Before resetting anything, engineers should query ECC and row-remapper state and bundle the full bug report with the matching kernel log. These two sources together show whether the GPU had pending remaps or link errors, enabling an accurate hardware-versus-software determination and a defensible RMA decision.

Answer

Immediately reboot the node and re-run the training job to see whether it fails again.

Answer

Reset the GPU with `nvidia-smi --gpu-reset` and continue using the node if training succeeds afterward.

Answer

Delete and recreate the NGC container, then pull the image again from the registry.

Question 9

A team is deploying a large language model for inference using NVIDIA Triton Inference Server on a GPU. They observe that the first inference request has high latency compared to subsequent requests. What is the most likely cause and the appropriate optimization?

Accepted Answer

The first request triggers model loading and CUDA context initialization; enable model warmup in Triton to pre-load and initialize the model.. The first inference request often incurs overhead from loading the model into GPU memory, compiling kernels, and initializing CUDA contexts. Triton's model warmup feature performs dummy inferences at startup, effectively pre-warming the model. This shifts the initialization cost away from the first real request, resulting in consistent low latency for all requests.

Answer

The GPU is thermal throttling on the first request; set a higher power limit using nvidia-smi -pl to stabilize performance.

Answer

The model is not using TensorRT; converting it to a TensorRT engine will eliminate first-request latency.

Answer

The model uses dynamic batching; disable dynamic batching to ensure the first request is processed immediately.

Question 10

An AI operations engineer is validating a new NVIDIA-certified server before putting it into production. The job runs correctly but the team wants to confirm that the GPUs are operating at the expected clocks and not being limited by power or thermal constraints. Which command provides the most direct evidence of the current power and thermal limits and any throttling reasons?

Accepted Answer

`nvidia-smi -q -d PERFORMANCE`. The performance query reports clocks, performance state, and explicit throttle reason flags such as power cap, thermal slowdown, and hardware limit. That combination gives direct evidence of whether a GPU is constrained and by what. Other commands either report inventory or topology and cannot answer the throttling question.

Answer

`nvidia-smi --query-gpu=name,driver_version --format=csv`

Answer

`nvidia-smi topo -m`

Answer

`nvidia-smi --gpu-reset -i 0`

Question 11

An AI operations team is troubleshooting a training job that crashes with a segmentation fault after several hours. The job uses multiple GPUs and NCCL for communication. System logs show no errors, but dmesg reveals repeated 'NVRM: Xid' errors. Which action should be taken first to diagnose the issue?

Accepted Answer

Check the Xid error code in the NVIDIA documentation to identify the specific GPU fault.. Xid errors are logged by the NVIDIA driver and each code maps to a specific GPU fault. Interpreting the Xid code is the fastest way to understand whether the segmentation fault is due to a hardware issue, driver bug, or application error. This guides subsequent troubleshooting steps, such as replacing hardware or updating the driver.

Answer

Restart the NVIDIA driver with rmmod and modprobe commands.

Answer

Enable NCCL_DEBUG=INFO and re-run the job to capture detailed NCCL logs.

Answer

Run nvidia-smi -q to check GPU temperature and power usage.

Question 12

An inference service runs a 70B parameter model with TensorRT-LLM on a single H100 using in-flight batching. Operators report that time-to-first-token is acceptable, but inter-token latency degrades sharply once concurrent request count exceeds a certain point, and GPU memory utilization sits near 98 percent. Which change most directly addresses the inter-token latency degradation?

Accepted Answer

Enable FP8 quantization for the KV cache and reduce the maximum batch size in the TensorRT-LLM build configuration.. When GPU memory is nearly exhausted, the runtime cannot hold enough KV cache plus workspace to serve all admitted sequences efficiently, so each decode step stretches and inter-token latency rises with concurrency. Shrinking the KV cache with FP8 quantization and lowering the maximum batch size reduces per-step memory and compute demand, restoring shorter decode iterations. These changes target the resource constraint that actually causes the latency curve to bend upward.

Answer

Disable in-flight batching so each request is processed to completion before the next begins.

Answer

Increase the maximum batch size so more requests are processed per decode step.

Answer

Switch the service to a round-robin load balancer across two replicas of the same model on one GPU.

Question 13

An AI operations engineer notices that a training job on an NVIDIA A100 GPU is running slower than expected. Running nvidia-smi shows that the GPU is in 'P0' state but the 'SM Clock' is significantly lower than the maximum boost clock. The job is not memory-bound. Which action should the engineer take first to diagnose the issue?

Accepted Answer

Check the GPU's power consumption and whether it is hitting the power limit.. When a GPU is in P0 state but SM clock is low, power or thermal throttling is a likely cause. Checking power consumption and limits is a quick, non-invasive diagnostic step. If the GPU is hitting its power limit, the engineer can then adjust power settings or optimize the workload to reduce power draw. The other actions do not directly diagnose the throttling issue.

Answer

Reinstall the NVIDIA driver to ensure the latest version.

Answer

Increase the batch size to improve GPU utilization.

Answer

Switch the job to use mixed precision to reduce compute load.

Question 14

An AI engineer is optimizing a real-time inference pipeline on an NVIDIA A100 GPU. The model uses dynamic input shapes, and profiling shows that the GPU spends significant time on memory copies between host and device. Which optimization should be implemented to reduce this overhead?

Accepted Answer

Use pinned (page-locked) host memory for input and output buffers.. The overhead from host-device memory copies is best reduced by using pinned memory, which enables faster DMA transfers. This is especially important for real-time inference with dynamic shapes where copies are frequent. While CUDA streams can overlap transfers with compute, they are more effective when combined with pinned memory.

Answer

Enable Unified Memory to automatically manage data migration.

Answer

Increase the batch size to amortize the cost of memory copies.

Answer

Use CUDA streams to overlap memory copies with computation.

Question 15

An AI operations engineer is optimizing a PyTorch training job on an NVIDIA DGX A100 system. The job uses a data loader with multiple workers, but the engineer observes that GPU utilization fluctuates between 40% and 60%, and `nvidia-smi dmon` shows periods of zero GPU utilization. The engineer suspects that the data input pipeline is the bottleneck. Which TWO actions should the engineer take to improve GPU utilization? (Choose two.)

Accepted Answer

Increase the number of DataLoader workers to better overlap data loading with GPU computation.. The observed GPU utilization fluctuations and zero-utilization periods indicate the GPU is often waiting for data. Increasing DataLoader workers parallelizes data preprocessing, and enabling pinned memory accelerates host-to-device transfers. Together, these reduce data loading latency and improve overlap with GPU computation, raising utilization. Other options either do not address the bottleneck or harm performance.

Answer

Decrease the batch size to reduce memory usage and allow more frequent updates.

Answer

Set the CUDA_LAUNCH_BLOCKING environment variable to 1 to ensure synchronous kernel execution.

Answer

Enable mixed precision training to reduce memory footprint and increase throughput.