NCP-AIO Troubleshooting and Optimization Practice Test 5 — 15 Questions

Question 1

An administrator wants to ensure that a training process is limited to a single GPU on a multi-GPU node. Which environment variable should be set?

Accepted Answer

CUDA_VISIBLE_DEVICES=0. Controlling GPU visibility is a fundamental skill for resource management in multi-tenant environments. By using CUDA_VISIBLE_DEVICES, an administrator can restrict a process to a specific device, preventing multiple jobs from competing for the same GPU. This isolation is crucial for maintaining performance stability and ensuring that individual jobs receive consistent, predictable access to compute resources without interference from other concurrent tasks.

Answer

NCCL_DEBUG=INFO

Answer

NVIDIA_DRIVER_CAPABILITIES=compute

Answer

OMP_NUM_THREADS=1

Question 2

What is the most accurate way to verify that a training job is utilizing Tensor Cores?

Accepted Answer

Monitor the SM Tensor utilization metric via Nsight Compute.. Tensor Cores are specialized hardware units for mixed-precision matrix operations. Monitoring the SM occupancy and specific instruction usage via Nsight Compute is the definitive way to confirm their engagement. Understanding how to verify this is essential for engineers to ensure that their optimization efforts, such as mixed-precision training, are actually resulting in the intended hardware utilization and performance improvements.

Answer

Check the GPU power consumption in nvidia-smi.

Answer

Check the memory bandwidth in DCGM.

Answer

Verify the CUDA driver version is above 500.0.

Question 3

If a training job on a multi-node cluster shows a significant performance drop during checkpointing, what is the most likely bottleneck?

Accepted Answer

Shared filesystem I/O throughput.. Checkpointing involves writing large model states to non-volatile storage. If this process is not handled asynchronously or if the underlying storage system is undersized, the training process will stall. Identifying this bottleneck allows engineers to implement optimized I/O strategies, such as using distributed file systems or NVMe-based local storage, to minimize the time spent in the checkpoint phase and maximize overall cluster efficiency.

Answer

Network latency between compute nodes.

Answer

GPU memory allocation overhead.

Answer

The model weight update frequency.

Question 4

Which of the following is the primary indicator of PCIe bus saturation when profiling a training job on an NVIDIA DGX system?

Accepted Answer

Low GPU duty cycle and high PCIe throughput utilization.. PCIe bus saturation occurs when the GPU's requirement for data exceeds the transfer capacity of the bus. This leads to GPU starvation, where the compute cores idle while waiting for data. Recognizing the specific correlation between low GPU utilization and high PCIe throughput is essential for identifying I/O-bound bottlenecks and optimizing data pipelines for large-scale training jobs.

Answer

High SM occupancy but low memory bandwidth usage.

Answer

High GPU temperature and low clock speeds.

Answer

Frequent ECC errors in the GPU memory logs.

Question 5

Refer to the exhibit. During a multi-node training job, communication between nodes fails. What is the most likely cause of this error?

Accepted Answer

The NCCL_SOCKET_IFNAME environment variable is misconfigured for the cluster network.. The error indicates a network connectivity issue during the NCCL initialization phase, specifically failing to route traffic between nodes. NCCL relies on correct interface configuration for inter-node communication. Misconfigured network interfaces or firewall rules between nodes are common failure points in distributed training. Troubleshooting these network layer issues is vital for ensuring high-performance communication across GPU clusters during large-scale model training.

Answer

The GPU driver version is incompatible with the installed NCCL library.

Answer

The model weights are too large to be synchronized across the network.

Answer

The CUDA visibility is restricted to only one GPU per node.

Question 6

Which TWO of the following actions should be taken to optimize GPU memory usage when encountering Out-of-Memory (OOM) errors during model training?

Accepted Answer

Implement gradient checkpointing to trade compute for memory.. OOM errors are common in deep learning when the model size, activation memory, and batch size exceed VRAM capacity. Implementing gradient checkpointing and mixed-precision training are standard industry practices to manage memory pressure. Mastery of these techniques is essential for AI operations engineers to maintain system stability and enable the training of large-scale models without requiring immediate hardware upgrades.

Answer

Increase the number of CPU threads in the data loader.

Answer

Disable the use of NCCL to reduce inter-node memory overhead.

Answer

Switch from the Adam optimizer to Stochastic Gradient Descent (SGD).

Question 7

An AI engineer observes that a training job on an NVIDIA DGX H100 system is experiencing significant performance degradation. The GPU utilization is high, but the throughput remains low. Which tool should be used first to identify if the bottleneck is related to data loading or PCIe bandwidth saturation?

Accepted Answer

NVIDIA Nsight Systems. NVIDIA Nsight Systems is the primary tool for analyzing system-wide performance, including CPU-GPU interactions and data transfer bottlenecks. Identifying whether a bottleneck occurs in the data pipeline or within the hardware interconnects is critical for optimizing training speed. By visualizing timelines, engineers can pinpoint if the GPU is starving for data or if the PCIe bus is congested, enabling targeted remediation for large-scale distributed training clusters.

Answer

NVIDIA-SMI

Answer

NVIDIA DCGM

Answer

NVIDIA Nsight Compute

Question 8

Refer to the exhibit. The training job fails with a CUDA OOM error. Given the memory profile, which optimization strategy provides the most immediate relief while maintaining model performance?

Accepted Answer

Enable Automatic Mixed Precision (AMP).. Switching to Mixed Precision (FP16/BF16) effectively halves the memory requirement for model activations and gradients. This is a standard optimization strategy for Transformer models when GPU memory capacity is the primary constraint. By reducing the memory footprint of floating-point operations, researchers can fit larger batch sizes or deeper architectures into the same GPU memory, significantly improving training efficiency and throughput without sacrificing convergence stability.

Answer

Increase the batch size to 256.

Answer

Disable all CUDA kernels.

Answer

Replace the GPU with a higher-clocked model.

Question 9

An AI Operations engineer is managing a multi-node training job using NVIDIA NCCL. The logs indicate frequent 'NCCL WARN' messages related to 'net_ib_init' failures. What is the most likely cause of this issue?

Accepted Answer

Misconfigured InfiniBand subnet manager or mismatched firmware.. NCCL relies on InfiniBand (IB) for high-speed inter-GPU communication in multi-node setups. Failures in 'net_ib_init' suggest that the network configuration or driver stack is misaligned between nodes. Identifying and resolving these connectivity issues is critical because NCCL is the backbone of distributed training; any failure here leads to synchronized hangs, data corruption, or severe training slowdowns, undermining the benefits of cluster-level resource parallelism.

Answer

Incompatible NCCL versions across the cluster nodes.

Answer

Insufficient system RAM on the worker nodes.

Answer

Incorrect GPU driver version installed on the login node.

Question 10

Which component of the NVIDIA AI Enterprise stack is primarily responsible for ensuring the long-term stability and compatibility of drivers and libraries across heterogeneous hardware configurations?

Accepted Answer

NVIDIA AI Enterprise. NVIDIA AI Enterprise provides a validated and supported software stack that ensures consistency across different hardware generations. Stability is the foundation of AI operations; without a supported, validated stack, minor version mismatches between kernels and libraries can lead to silent failures or system crashes. This support model is essential for enterprises to avoid the 'dependency hell' that often plagues research-grade environments when scaling to production operations.

Answer

NVIDIA CUDA Toolkit

Answer

NVIDIA Triton Inference Server

Answer

NVIDIA Collective Communications Library

Question 11

A multi-tenant NVIDIA GPU-accelerated Kubernetes cluster utilizing NVIDIA AI Enterprise experiences intermittent out-of-memory errors on Triton Inference Server pods despite adequate node memory reservation. Which monitoring and troubleshooting action correctly isolates the root cause?

Accepted Answer

Inspect DCGM-Exporter metrics for DCGM_FI_DEV_FB_FREE and DCGM_FI_DEV_GPU_TEMP alongside Triton server request concurrency logs.. Inspect NVIDIA Data Center GPU Manager metrics via Prometheus and DCGM-Exporter to capture real-time device memory consumption patterns. This practice is critical because standard Kubernetes container metrics fail to track internal GPU frame buffer allocations and fragmentation specific to deep learning inference engines.

Answer

Analyze cAdvisor container memory metrics using kubectl top pods to evaluate resident set size growth over sustained inference workloads.

Answer

Review the Kubernetes cluster autoscaler logs to determine if node scaling events triggered unexpected pod evictions and restarts.

Answer

Execute systemctl status containerd on the worker node to verify container runtime stability and daemon responsiveness.

Question 12

A team runs a multi-node NCCL all-reduce training job on four DGX H100 nodes connected by an InfiniBand fabric. Scaling efficiency is poor: throughput barely improves beyond two nodes, and `nvidia-smi` shows NIC transmit counters on each GPU's assigned HCA are far below the PCIe link capacity while GPU compute utilization sits at ~55%. The fabric manager logs report all links as up with no symbol errors. Which action should the administrator take first to diagnose the interconnect bottleneck?

Accepted Answer

Run `nccl-tests` all_reduce_perf with the same process count and message sizes, then compare its reported bus bandwidth against the theoretical peak for the fabric.. When every fabric link is error-free but HCA counters sit well below link bandwidth, the bottleneck is in how the collective is scheduled or placed, not in raw link health. Reproducing the all-reduce with nccl-tests and comparing achieved bus bandwidth to the fabric's theoretical peak isolates the communication path and reveals whether the gap comes from topology detection, ring construction, or process placement before any tuning is applied.

Answer

Switch the job to use the NCCL tree algorithm via NCCL_ALGO=TREE because ring all-reduce cannot scale across four nodes.

Answer

Enable GPUDirect RDMA by exporting NCCL_NET_GDR_LEVEL=SYS and rerunning the job to confirm whether host-memory staging was the cause.

Answer

Increase the NCCL_BUFFSIZE environment variable to its maximum so each all-reduce chunk transfers more data per operation.

Question 13

A production inference service on an NVIDIA A100 GPU experiences a gradual increase in latency over several hours, eventually requiring a pod restart. GPU memory utilization climbs steadily, but the model and batch size are fixed. Which action should an AI operations engineer take first to diagnose the root cause?

Accepted Answer

Use PyTorch's torch.cuda.memory_summary() or TensorFlow's memory profiler to capture allocation snapshots and compare over time.. The steady memory increase with fixed workload strongly suggests a memory leak in the inference application or framework. Framework-specific memory profilers provide allocation-level visibility, allowing engineers to identify unreleased tensors or cached allocations. Aggregate GPU monitoring or sanitizer tools lack the necessary granularity. Increasing limits merely postpones failure.

Answer

Run nvidia-smi --query-gpu=memory.used --format=csv -l 1 to log memory usage over time and correlate with request rate.

Answer

Enable CUDA memory leak detection with compute-sanitizer --tool memcheck on the running inference process.

Answer

Increase the GPU memory limit in the Kubernetes pod spec to prevent the pod from being killed.

Question 14

A production inference service running on NVIDIA GPUs exhibits periodic latency spikes every few minutes, correlating with CPU-side stalls and low GPU utilization during those intervals. Profiling with Nsight Systems shows large gaps between kernel launches and frequent cudaMalloc/cudaFree calls. Which action best addresses the root cause?

Accepted Answer

Enable CUDA Graph capture for the inference sequence and reuse the captured graph for repeated executions.. The profile shows CPU-side stalls and frequent cudaMalloc/cudaFree, indicating launch and allocation overhead. CUDA Graphs capture the entire sequence and replay it with minimal CPU involvement, eliminating both the allocation churn and the launch gaps. This directly reduces latency spikes and improves GPU utilization, making it the most effective remedy for the described symptoms.

Answer

Switch the model to use TensorRT with FP16 precision to reduce kernel execution time.

Answer

Increase the number of worker threads on the CPU to better overlap preprocessing with GPU execution.

Answer

Enable Multi-Process Service (MPS) to allow concurrent kernel execution from multiple processes.

Question 15

An AI operations engineer is troubleshooting an inference service on an NVIDIA A100 GPU that shows intermittent stalls. The monitoring dashboard reports GPU utilization at 100%, but request throughput is far below the validated baseline. Running nvidia-smi dmon reveals the SM utilization is high while memory controller utilization is low. Which action should the engineer take first to identify the bottleneck?

Accepted Answer

Profile the inference process with Nsight Systems to inspect kernel execution and identify serialization gaps.. The high SM utilization with low memory controller utilization suggests the GPU is busy but not doing useful work, often due to kernel serialization or CPU-GPU synchronization stalls. Profiling with Nsight Systems provides the timeline needed to see gaps and serialization. Only after identifying the specific stall should configuration changes be considered, making profiling the correct first step.

Answer

Switch the inference precision from FP16 to FP32 to improve numerical stability.

Answer

Enable MPS (Multi-Process Service) to allow concurrent kernel execution from multiple processes.

Answer

Increase the batch size in the inference server configuration to improve GPU occupancy.