NCP-AIO Troubleshooting and Optimization Practice Test 3 — 15 Questions

Question 1

An AI operations team is using NVIDIA Triton Inference Server to serve a model. They observe that the server's throughput is lower than expected, and the GPU utilization is consistently around 40%. The model has a fixed batch size of 1. Which configuration change is most likely to improve throughput?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Enable dynamic batching in the Triton model configuration to allow the server to combine multiple inference requests into a single batch.. Low GPU utilization with a fixed batch size of 1 indicates that the GPU is underfed. Triton's dynamic batching feature aggregates multiple inference requests into a single batch, increasing the work per execution and improving GPU utilization. Configuring `dynamic_batching` with appropriate `max_batch_size` and `preferred_batch_size` allows Triton to optimize throughput without changing the model. This is the most direct and effective optimization.

Answer

Configure the Triton server to use a larger `instance_group` count and set `max_batch_size` to 1.

Answer

Increase the number of model instances per GPU to allow more concurrent executions.

Answer

Switch the model to use FP16 precision to reduce computation time.

Question 2

An operations team runs a multi-node PyTorch training job across four DGX H100 nodes connected with NVIDIA Quantum-2 InfiniBand using NCCL. Logs show repeated NCCL 'unhandled system error' and the job stalls during the all-reduce phase. `nvidia-smi` on all nodes reports healthy GPUs, and the application code has not changed. Which action should the team take first to diagnose the NCCL communication failure?

Accepted Answer

Set `NCCL_DEBUG=INFO` and `NCCL_DEBUG_SUBSYS=INIT,NET` and rerun the job to capture detailed NCCL initialization and network logs.. The most effective first step is to enable NCCL debug logging to capture detailed initialization and network information. This directly shows which interfaces NCCL selects and where the communication error occurs, guiding further troubleshooting. Other actions either mask the problem or focus on intra-node components that are less relevant to a multi-node InfiniBand failure.

Answer

Disable peer-to-peer communication by setting `NCCL_P2P_DISABLE=1` and rerun the job.

Answer

Increase the NCCL timeout by setting `NCCL_TIMEOUT=1800` and restart the job.

Answer

Run `nvidia-smi nvlink -s` on each node to check NVLink status between GPUs.

Question 3

An AI operations engineer is investigating a sudden drop in inference throughput for a TensorRT-optimized model deployed on an NVIDIA T4 GPU. The model previously performed well. Which two actions should be taken to diagnose the performance regression? (Choose two.)

Accepted Answer

Check GPU clocks and power limits using nvidia-smi to ensure the GPU is not throttling due to power or thermal constraints.. The two primary diagnostic actions are to check for hardware throttling via nvidia-smi and to profile the model with Nsight Systems. These steps help determine whether the regression is due to power/thermal issues or software bottlenecks. Other options are either optimizations or specific checks that are part of a broader diagnostic process but not the first actions to take.

Answer

Increase the batch size in the inference requests to improve GPU utilization.

Answer

Recompile the TensorRT engine with the latest TensorRT version to ensure optimal performance.

Answer

Verify that the model is still using the same precision (e.g., FP16) and that no fallback to FP32 has occurred.

Question 4

An AI engineer observes that a model training job on an NVIDIA DGX system is underutilizing the GPU. The training loop shows frequent "CPU bottleneck" warnings in the logs. Which action should the engineer take first to optimize throughput?

Accepted Answer

Implement NVIDIA DALI to offload preprocessing tasks from the CPU to the GPU.. Identifying CPU bottlenecks is critical because data pipelines often struggle to keep up with GPU compute speed. By optimizing data preprocessing, specifically increasing the number of workers in the DataLoader or using NVIDIA DALI, the engineer ensures the GPU remains saturated with data. This optimization directly impacts total training time and infrastructure cost efficiency, ensuring that high-performance hardware is not left idling while waiting for I/O operations.

Answer

Upgrade to a higher-end GPU model to handle the processing load.

Answer

Increase the batch size significantly to fill the GPU memory.

Answer

Reduce the number of training epochs to lower CPU overhead.

Question 5

An AI researcher is debugging a multi-node training job using NCCL. Which TWO actions should they take to diagnose potential network-related performance degradation?

Accepted Answer

Set NCCL_DEBUG=INFO to monitor communication patterns and connection issues.. Debugging multi-node communication requires verifying both software configurations and physical network health. NCCL_DEBUG settings provide granular insight into connection establishment and topology detection, while NCCL tests provide baseline performance metrics. Identifying these issues early is essential for scaling models across large clusters, as network overhead can quickly become the dominant factor in training time during distributed synchronized gradient descent.

Answer

Increase the number of threads in the PyTorch DataLoader.

Answer

Switch from NCCL to MPI for all communication operations.

Answer

Lower the precision of the model to float16.

Question 6

Refer to the exhibit. The training job fails with an OOM error. Which optimization strategy will most effectively resolve this while maintaining model convergence?

Accepted Answer

Implement Gradient Accumulation to simulate a larger batch size.. Memory management is central to deep learning stability. When a model exceeds physical VRAM, gradient accumulation allows for larger effective batch sizes without increasing memory footprint. By accumulating gradients over multiple small steps and performing a single weight update, the model effectively sees a larger batch size, maintaining the convergence characteristics of the original design while fitting within the strict physical memory constraints of the GPU hardware.

Answer

Increase the learning rate to compensate for smaller batches.

Answer

Disable mixed-precision training to reduce memory overhead.

Answer

Clear the GPU cache using torch.cuda.empty_cache() after every iteration.

Question 7

Refer to the exhibit. The system has two GPUs. What is the most likely cause of the observed performance discrepancy?

Accepted Answer

Only one GPU is receiving the workload due to improper data distribution.. The exhibit shows one GPU heavily utilized while the second is idling despite similar memory consumption. This indicates a data parallelism imbalance, where one process is doing the bulk of the work. This is a common issue in multi-GPU setups where workload distribution is not correctly handled, leading to massive inefficiencies where expensive hardware is under-utilized, significantly increasing the time required for model training or inference tasks.

Answer

The GPUs have different amounts of physical memory.

Answer

The system is limited by the PCIe bus bandwidth.

Answer

The model is too small to be parallelized across two GPUs.

Question 8

A production inference service using TensorRT is showing lower than expected throughput. Profiling shows that the model is spending significant time in "host-to-device" transfers. What is the best optimization?

Accepted Answer

Use pinned memory for data transfers.. Data movement is a primary latency bottleneck in deep learning inference. By using pinned memory (page-locked memory) for host-side buffers, the system can enable faster direct memory access (DMA) transfers between the CPU and the GPU. This minimizes the time spent in data copy operations, directly reducing latency and increasing total throughput, which is essential for meeting strict Service Level Agreements (SLAs) in production AI deployments.

Answer

Increase the GPU clock speed via nvidia-smi.

Answer

Switch to FP64 precision for higher accuracy.

Answer

Implement multi-threaded data preprocessing on the GPU.

Question 9

Refer to the exhibit. The training job is showing intermittent "thermal throttling" warnings. Which configuration change is the most appropriate adjustment?

Accepted Answer

Lower the "power_limit_watts" value.. Thermal throttling occurs when the GPU reaches its maximum operating temperature and lowers its clock speed to prevent physical damage. While power limits can be adjusted to reduce heat, simply lowering the limit may negatively impact training performance. Adjusting the power limit to a value that balances thermal overhead with compute requirements is a necessary trade-off to ensure stable, consistent performance during long training runs without hardware damage.

Answer

Set "compute_mode" to "exclusive_process".

Answer

Disable ECC mode to lower the power consumption.

Answer

Set "persistence" to "disabled".

Question 10

An AI engineer needs to monitor GPU utilization across a large cluster of nodes in real-time. Which NVIDIA tool is the most appropriate for this high-level observability task?

Accepted Answer

NVIDIA DCGM (Data Center GPU Manager). Observability at scale requires tools that can aggregate metrics across multiple nodes. NVIDIA DCGM (Data Center GPU Manager) is specifically architected for this purpose, providing a comprehensive API and service to monitor GPU health, utilization, and power consumption across entire clusters. This is essential for operations teams managing large-scale AI infrastructure to proactively detect performance issues and optimize resource utilization across the fleet.

Answer

nvidia-smi

Answer

CUDA Profiler (nsys)

Answer

nvcc

Question 11

During a training job, the system reports "NCCL WARN" regarding a slow network path. What is the most likely culprit for this performance bottleneck in a multi-node InfiniBand environment?

Accepted Answer

InfiniBand link speed is negotiated at a lower rate than expected.. In high-performance clusters, the network fabric is often the bottleneck. InfiniBand performance relies on proper subnet manager configuration and accurate link-speed reporting. Identifying "slow paths" using tools like ibdiagnet helps pinpoint physical or configuration issues in the interconnect. Addressing these issues is vital because, in distributed training, the speed of the cluster is limited by the slowest link in the communication path, severely impacting the overall training efficiency.

Answer

The GPU clock speed is set too low.

Answer

The batch size is too small.

Answer

The CPU is running at maximum capacity.

Question 12

An engineer is troubleshooting a CUDA program that terminates unexpectedly. Which tool should be used to detect memory leaks and race conditions in the CUDA kernel code?

Accepted Answer

NVIDIA Compute Sanitizer. Debugging parallel code is inherently difficult due to the non-deterministic nature of race conditions and memory access patterns. The NVIDIA Compute Sanitizer is the essential diagnostic tool for identifying these issues, such as out-of-bounds memory access or race conditions, before they lead to erratic, intermittent crashes in production. Using it early in the development lifecycle saves massive amounts of time by ensuring code correctness and robustness.

Answer

nvidia-smi

Answer

nsys (Nsight Systems)

Answer

nvprof

Question 13

An AI engineer observes that a model training job on an NVIDIA DGX system is underutilizing the GPU. The monitoring logs show high CPU wait times and low GPU duty cycles. Which action should the engineer take first to resolve the bottleneck?

Accepted Answer

Optimize the data preprocessing pipeline using NVIDIA DALI.. High CPU wait times alongside low GPU utilization suggest an I/O or data preprocessing bottleneck. The CPU cannot feed data to the GPU fast enough, forcing the GPU to idle while waiting for the next batch. Optimizing the data pipeline, such as increasing prefetch buffers or using NVIDIA DALI to move image processing to the GPU, directly addresses the starvation of the compute resources, ensuring efficient utilization.

Answer

Increase the GPU clock frequency via NVML.

Answer

Enable GPUDirect Storage on the local drive.

Answer

Increase the batch size to maximize memory usage.

Question 14

An AI researcher is deploying a multi-node training job using NCCL on an InfiniBand network. The job is suffering from intermittent latency spikes. Which TWO steps should the engineer perform to troubleshoot the network configuration?

Accepted Answer

Check IB link status and error counters using 'ibstat'.. Troubleshooting high-performance networking in distributed training requires checking for physical layer stability and protocol-level misconfigurations. Verifying InfiniBand counters helps identify packet drops or link flapping, while ensuring NCCL environment variables are correctly set for the specific topology avoids suboptimal routing. These steps ensure that the inter-GPU communication remains within the high-bandwidth low-latency envelope required for scaling deep learning workloads effectively.

Answer

Reset the GPU thermal throttling threshold.

Answer

Update the OS kernel to the latest version.

Answer

Increase the system swap space to 512GB.

Question 15

Refer to the exhibit. An AI administrator investigates why a GPU node is performing significantly slower than expected. Based on the output, what is the most likely cause?

Accepted Answer

The server's cooling system is failing or airflow is restricted.. The exhibit shows 'HW Thermal Slowdown' and 'HW Slowdown' are active. This indicates the GPU hardware is actively reducing its clock frequency to prevent damage due to excessive heat. This is a critical performance issue that requires immediate attention to the data center cooling or physical airflow within the server chassis. Ensuring adequate thermal management is fundamental to maintaining consistent compute performance during heavy training loads.

Answer

The GPU driver is outdated and needs a patch.

Answer

The power supply unit is malfunctioning.

Answer

The workload exceeds the GPU memory capacity.