NCP-AIO Troubleshooting and Optimization Practice Test 2 — 15 Questions

Question 1

An AI infrastructure team is deploying NVIDIA Triton Inference Server. They notice that the latency for model inference is inconsistent, spiking periodically. Which feature should be enabled to stabilize latency by reducing the overhead of repeated memory allocations?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

CUDA Memory Pool. Memory allocation is a high-overhead operation that can lead to jitter in inference latency. By pre-allocating memory and utilizing specific caching or queuing strategies within Triton, engineers can ensure that the server is ready to handle requests without the latency penalty of dynamic allocation. Stabilizing latency is critical for meeting Service Level Agreements (SLAs) in production-grade AI deployments where consistent performance is often more important than theoretical peak throughput.

Answer

Dynamic Batching

Answer

Instance Grouping

Answer

Model Ensembling

Question 2

An AI operations engineer is troubleshooting a multi-node NCCL training job on an NVIDIA DGX SuperPOD. The job runs successfully on a single node, but when scaled to four nodes, the GPUs sit idle for long periods and throughput collapses. Running `nvidia-smi topo -m` on each node shows all GPUs connected via NVLink within the node, and `ibstat` on each node reports the InfiniBand HCA state as 'Active' with a link width of 4x. However, the engineer notices that the NCCL job logs show it is selecting the Ethernet interface instead of the InfiniBand interface. Which action should the engineer take to resolve this issue?

Accepted Answer

Set the NCCL_SOCKET_IFNAME environment variable to the InfiniBand interface name (e.g., ib0) and ensure the InfiniBand interface is up.. NCCL selects the first available network interface, which may be Ethernet even when InfiniBand is present and active. Forcing NCCL to use the InfiniBand interface via NCCL_SOCKET_IFNAME ensures multi-node communication leverages the high-bandwidth, low-latency fabric, resolving the idle GPU periods and throughput collapse.

Answer

Set the NCCL_IB_DISABLE environment variable to 1 to prevent NCCL from attempting to use InfiniBand and fall back to Ethernet.

Answer

Increase the NCCL_BUFFSIZE environment variable to 64 MB to improve throughput over Ethernet.

Answer

Run `nvidia-smi nvlink -s` to verify NVLink status and then restart the job.

Question 3

A production inference service running on an NVIDIA A100 GPU shows intermittent spikes in latency every few minutes. The operations team suspects GPU memory fragmentation from repeated allocation and deallocation of tensors. Which action should be taken to confirm and mitigate this?

Accepted Answer

Set the environment variable PYTORCH_CUDA_ALLOC_CONF to expandable_segments:True and observe if latency spikes subside.. Memory fragmentation occurs when the caching allocator cannot find a contiguous block for a new tensor, causing extra cudaMalloc calls and latency spikes. PyTorch's expandable_segments feature allows segments to grow, reducing fragmentation. This is a targeted mitigation that directly addresses the suspected cause. Confirming via environment change is efficient and low-risk compared to profiling or overclocking.

Answer

Increase the GPU's memory clock using nvidia-smi -ac to improve memory bandwidth and reduce latency.

Answer

Enable CUDA_MODULE_LOADING=LAZY and monitor GPU memory usage with nvidia-smi.

Answer

Use NVIDIA Nsight Systems to capture a timeline and check for cudaMalloc/cudaFree calls during inference.

Question 4

A data scientist reports that a multi-GPU training job using NCCL for inter-GPU communication is achieving only 30% of expected scaling efficiency. The job runs on a DGX system with NVLink. Profiling reveals that NCCL is using the ring algorithm and that the all-reduce operation is taking significantly longer than the compute phase. Which NCCL configuration change is most likely to improve performance?

Accepted Answer

Set NCCL_ALGO=Tree to enable tree-based algorithms for all-reduce.. The ring algorithm can be less efficient than tree-based algorithms for all-reduce on high-bandwidth, low-latency interconnects like NVLink, especially as GPU count grows. Switching to tree-based algorithms via NCCL_ALGO=Tree can better utilize the topology, reduce communication time, and improve scaling efficiency. The other options either degrade performance or do not address the algorithmic inefficiency.

Answer

Set NCCL_BUFFSIZE to a larger value to increase the buffer size for communication.

Answer

Set NCCL_P2P_LEVEL=SYS to force peer-to-peer communication over the system bus.

Answer

Set NCCL_IB_DISABLE=1 to disable InfiniBand and use Ethernet instead.

Question 5

A team is running a large language model inference service on NVIDIA A100 GPUs using Triton Inference Server. They observe that the GPU memory utilization is consistently above 90%, and the server occasionally returns 'CUDA out of memory' errors during peak load. The model is loaded with a fixed batch size of 1, and the team wants to increase throughput without changing the model. Which configuration change should they implement to reduce memory pressure while improving throughput?

Accepted Answer

Enable dynamic batching in Triton with a max batch size and preferred batch sizes to allow the server to combine requests.. Dynamic batching groups multiple inference requests into a single batch, allowing the GPU to process them together. This improves throughput by increasing parallelism and reduces per-request memory overhead, easing memory pressure. Other options either increase memory usage or do not address the root cause of memory exhaustion during peak load.

Answer

Configure Triton to use pinned memory for input tensors to speed up data transfer.

Answer

Increase the instance group count to 2 to allow concurrent model execution.

Answer

Set the model to use FP32 precision instead of FP16 to reduce memory usage.

Question 6

An AI operations engineer is troubleshooting a multi-node NCCL training job that intermittently hangs during the all-reduce phase. Logs show the job stalls on one node, and `nvidia-smi topo -m` reveals that GPUs on that node communicate across PCIe switches rather than NVLink. Which action most directly resolves the hang?

Accepted Answer

Set NCCL_P2P_LEVEL to SYS to allow peer-to-peer across PCIe switches.. The hang is caused by NCCL not using the correct peer-to-peer path across PCIe switches. Setting NCCL_P2P_LEVEL to SYS allows P2P communication at the system level, which is appropriate for GPUs connected via PCIe switches. This resolves the stall by enabling direct GPU-to-GPU transfers without host memory staging.

Answer

Reconfigure the job to use NCCL_SHM_DISABLE=1 and rely on network sockets.

Answer

Set NCCL_P2P_DISABLE=1 to force PCIe peer-to-peer communication.

Answer

Set NCCL_IB_DISABLE=1 to prevent InfiniBand usage and fall back to Ethernet.

Question 7

A production inference service using NVIDIA Triton Inference Server on an A100 GPU shows that GPU utilization is low while request concurrency is high. Metrics indicate that requests are waiting in the scheduler queue. Which configuration change should be applied to improve throughput?

Accepted Answer

Increase the instance group count in the model configuration to allow more concurrent model executions.. The low GPU utilization with high concurrency and queue buildup indicates that the model is not being executed in parallel enough to keep the GPU busy. Increasing the instance group count allows multiple concurrent executions of the model, which can better utilize the GPU's SMs and reduce queuing. Other options either do not increase parallelism or are invalid configurations.

Answer

Enable dynamic batching with a larger maximum batch size and longer maximum queue delay.

Answer

Set the model's 'max_batch_size' to a higher value and enable 'prefer_memory_pool' to reduce allocation overhead.

Answer

Reduce the number of concurrent requests by lowering the client-side request rate.

Question 8

An AI operations engineer notices that a Triton Inference Server deployment on a single NVIDIA GPU shows p99 latency spikes whenever batch size grows, even though GPU compute utilization stays below 40 percent. Model instances are set to one. Which configuration change most directly addresses the bottleneck?

Accepted Answer

Increase the instance_group count so multiple model instances share the GPU and overlap execution.. Low GPU utilization with rising p99 latency as batch size increases points to serialized execution rather than compute saturation. A single model instance processes requests sequentially, leaving the GPU idle between kernels. Raising the instance group count creates concurrent execution streams on the same device, overlapping work and smoothing tail latency, which is the most direct fix for this bottleneck.

Answer

Convert the model to TensorRT precision calibration to reduce kernel execution time per inference.

Answer

Enable dynamic batching with max_batch_size and preferred_batch_size tuned to the observed request arrival rate.

Answer

Increase the response cache size so repeated prompts bypass model execution entirely.

Question 9

An AI operations engineer is diagnosing a training job that intermittently reports 'NCCL timeout' errors after running for several hours. The job uses 8 GPUs on a single node connected via NVLink. The engineer checks `nvidia-smi` and sees that all GPUs are healthy and not throttling. Network logs show no packet loss. Which action should the engineer take to resolve the NCCL timeouts?

Accepted Answer

Increase the NCCL_TIMEOUT environment variable to allow more time for collective operations.. NCCL timeouts often stem from transient delays in collective operations, such as a GPU temporarily stalling due to other system activity. Increasing NCCL_TIMEOUT provides a larger window for these operations to complete, preventing spurious aborts. It is a targeted fix that maintains performance and addresses the intermittent nature of the errors.

Answer

Disable NVLink by setting NCCL_P2P_DISABLE=1 to force communication over PCIe.

Answer

Reduce the batch size to decrease communication volume.

Answer

Set NCCL_DEBUG=INFO to enable detailed logging and then rerun the job.

Question 10

A team is troubleshooting an NVIDIA Triton Inference Server deployment that is experiencing high tail latency under load. They have already verified that the model is optimized with TensorRT and that the GPU is not saturated. Which two actions should they take to reduce tail latency? (Choose two.)

Accepted Answer

Increase the instance count for the model to allow more concurrent executions.. Enabling dynamic batching groups requests to improve GPU utilization and reduce per-request overhead, while increasing instance count allows concurrent execution to handle bursts. Together, they mitigate queuing and smooth tail latency. The other options either increase computational load, risk overload, or do not directly address batching and concurrency.

Answer

Switch the model to use FP32 precision to ensure numerical stability.

Answer

Configure a larger maximum batch size in the model configuration.

Answer

Disable the Triton rate limiter to allow unlimited requests.

Question 11

An AI operations team is troubleshooting a TensorFlow model that shows inconsistent inference latency when deployed on a NVIDIA T4 GPU. The model uses dynamic input shapes, and latency spikes occur sporadically. Profiling reveals that the CUDA kernels are being recompiled frequently. What is the most effective optimization to stabilize latency?

Accepted Answer

Use the CUDA caching allocator and enable JIT compilation caching by setting `CUDA_CACHE_MAXSIZE` to a larger value.. Sporadic latency spikes with dynamic input shapes are often due to repeated JIT compilation of CUDA kernels for each new shape. The CUDA cache stores compiled kernels; if the cache is too small, kernels are evicted and must be recompiled. Increasing `CUDA_CACHE_MAXSIZE` allows more kernels to persist, reducing compilation overhead and smoothing latency. This approach maintains dynamic shape support while optimizing performance.

Answer

Enable NVIDIA CUDA Graphs to capture the kernel launches and replay them, reducing launch overhead and recompilation.

Answer

Convert the model to TensorRT with a fixed batch size and input dimensions to enable kernel caching.

Answer

Increase the GPU's clock speed via `nvidia-smi -ac` to provide more compute headroom for kernel compilation.

Question 12

A cluster administrator is diagnosing why a distributed PyTorch training job across eight nodes completes successfully but at roughly one-third of expected throughput. Profiling shows GPUs idle during data loading and the fabric is lightly loaded. Which two actions should the administrator take to address the throughput problem? (Choose two.)

Accepted Answer

Tune the DataLoader with more worker processes and pinned memory so host-to-device transfers overlap with compute.. Profiling shows GPUs idle during data loading while the fabric stays lightly loaded, which isolates the bottleneck to the input pipeline rather than compute or communication. Expanding DataLoader parallelism with pinned memory overlaps transfers with compute, and caching shards on local NVMe reduces read latency. Together these keep the GPUs fed and raise sustained throughput.

Answer

Increase NCCL_ALGO ring depth or switch collectives to the tree algorithm to reduce communication time.

Answer

Reduce the global batch size so each step finishes faster and the job completes more iterations per hour.

Answer

Enable TF32 or BF16 mixed precision for matrix operations to increase arithmetic throughput on the Tensor Cores.

Question 13

An AI operations engineer is optimizing a multi-GPU training job on an NVIDIA DGX system. Profiling shows that a large all-reduce operation takes significantly longer than the theoretical NVLink time, and the timeline reveals that the collective waits for one GPU that finishes its local backward pass late. The job uses PyTorch DDP with default settings. Which action is the most appropriate to reduce the collective wait?

Accepted Answer

Increase the number of gradient buckets in DDP so that smaller all-reduce operations can overlap with the backward pass as soon as each bucket is ready.. The profiler shows the collective idling while one GPU finishes its backward pass. DDP's overlap mechanism depends on how gradients are grouped: with more, smaller buckets, all-reduce for a bucket can begin as soon as that bucket's gradients are ready, overlapping communication with the remaining backward computation and reducing idle time at the synchronization point.

Answer

Enable `torch.backends.cudnn.benchmark=True` so that convolution kernels are selected for maximum speed and the slow GPU catches up.

Answer

Set `NCCL_ALGO=Ring` to force ring all-reduce instead of tree, which always lowers latency for large messages.

Answer

Move the all-reduce to a separate CUDA stream and synchronize it only at the optimizer step to fully hide the latency.

Question 14

An AI operations engineer is investigating a performance regression in a PyTorch training job that uses NVIDIA A100 GPUs with Multi-Instance GPU (MIG) enabled. The job previously ran faster without MIG. Which two factors should be checked to ensure MIG configuration is not causing the slowdown? (Choose two.)

Accepted Answer

Verify that the MIG instance has sufficient memory bandwidth and compute slices allocated for the workload's demands.. Performance regression after enabling MIG often stems from insufficient resources allocated to the MIG instance or incorrect device visibility. Verifying that the instance has adequate memory bandwidth and compute slices ensures the workload isn't throttled. Confirming `CUDA_VISIBLE_DEVICES` prevents the job from accidentally using a smaller instance or multiple instances. These two checks directly address common MIG misconfigurations that degrade performance.

Answer

Verify that the MIG instance is using the correct compute instance (CI) and memory instance (GI) combination for optimal performance.

Answer

Ensure that the MIG instances are not oversubscribed by multiple processes, which could cause resource contention.

Answer

Check whether the A100 GPU is running in persistence mode to avoid driver initialization overhead.

Question 15

An operations engineer is investigating a containerized inference service that runs on an NVIDIA GPU node but reports far lower throughput than the same model run directly on the host. The container starts cleanly and sees the GPU. Which check is most appropriate to explain the gap?

Accepted Answer

Verify that the container was started with the NVIDIA Container Toolkit runtime so the full driver capability set and device nodes are exposed.. A container that starts and sees the GPU but underperforms the host most often suffers from incomplete GPU exposure. Confirming the NVIDIA Container Toolkit runtime ensures the container receives the same driver libraries, device nodes, and capabilities as the host, which is the most appropriate check before investigating application-level tuning.

Answer

Increase the container's CPU limit so preprocessing threads are not starved.

Answer

Rebuild the image on a newer base layer to pick up the latest CUDA toolkit version.

Answer

Confirm that the container image uses the same Python version as the host environment.