NCP-AIO Troubleshooting and Optimization Practice Test 1 — 15 Questions

Question 1

Which THREE factors are primary contributors to GPU memory fragmentation during long-running training jobs?

Accepted Answer

Frequent allocation and deallocation of tensors with varying sizes.. Memory fragmentation occurs when the allocator cannot find a contiguous block of memory for new tensors despite sufficient total free memory. This is common in long-running jobs with variable input sizes or frequent allocation/deallocation patterns. Monitoring and mitigating fragmentation is crucial for maintaining uptime, as it prevents sudden OOM crashes that disrupt training progress and waste significant GPU compute hours in production settings.

Answer

Using the same static tensor size throughout the entire training process.

Answer

Running the training job on a system with insufficient host RAM.

Question 2

An administrator wants to ensure that a containerized AI workload can access all available NVIDIA GPUs on a node. Which configuration flag is mandatory in the Docker runtime specification?

Accepted Answer

--gpus all. Proper GPU isolation and access control are required for containerized AI environments. Using the NVIDIA Container Toolkit ensures that the container environment correctly maps the host's GPU driver and libraries into the container namespace. Without these specifications, the container will not be able to identify or interact with the physical GPU hardware, rendering the AI training or inference tasks impossible to execute on the intended hardware.

Answer

--device=/dev/nvidia0

Answer

--privileged

Answer

--volume /usr/lib/nvidia

Question 3

When troubleshooting a job that failed with an "NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver" error, which TWO steps should be taken to verify the installation?

Accepted Answer

Check if the kernel module is loaded using lsmod | grep nvidia.. This error indicates a critical communication breakdown between the kernel-mode driver and user-mode tools. Verifying driver status is the first step in restoring functionality. Without a functional driver, the GPU is invisible to the OS, stopping all AI workflows. Resolving this promptly is essential to minimize downtime and ensure that the infrastructure is correctly reporting its health for scheduling systems and monitoring agents.

Answer

Restart the system BIOS to reset GPU clock frequencies.

Answer

Reinstall the CUDA toolkit to update the compiler version.

Answer

Increase the size of the swap partition on the disk.

Question 4

Which THREE techniques are recommended to optimize the performance of a model using mixed precision (FP16/BF16) on NVIDIA GPUs?

Accepted Answer

Use loss scaling to prevent underflow of small gradients.. Mixed precision leverages Tensor Cores to significantly boost throughput while reducing memory usage. Proper implementation requires careful consideration of numerical stability and data scaling to prevent vanishing gradients during training. These techniques are standard in modern deep learning to maximize hardware utilization, enabling the training of larger models that would otherwise be infeasible due to memory or time constraints on current hardware.

Answer

Maintain all master weights in FP64 to ensure maximum precision.

Answer

Disable the use of the cuDNN library for convolution operations.

Question 5

A deep learning model is experiencing frequent 'CUDA_ERROR_OUT_OF_MEMORY' during the training phase. The model architecture has not changed, and the batch size remains constant. What should be the first step to investigate?

Accepted Answer

Inspect the training loop for tensor accumulation.. Frequent OOM errors despite unchanged architecture and batch size often point to memory leaks in the training loop or rogue processes consuming VRAM. By checking current memory allocation per process, the administrator can identify if the training application is correctly releasing tensors after each iteration or if another application is encroaching on the GPU's memory pool. Efficient memory management is vital for stable, long-running AI training jobs.

Answer

Increase the physical RAM on the host server.

Answer

Reinstall the NVIDIA display drivers.

Answer

Switch to a larger GPU model.

Question 6

An NVIDIA NGC containerized workload is failing to initialize. Which THREE items should be verified to ensure the environment is correctly set up for GPU acceleration?

Accepted Answer

NVIDIA Container Toolkit installation.. Successful containerized GPU execution requires tight integration between the host driver, the container runtime, and the container's environment variables. Verifying these ensures that the CUDA library calls are correctly mapped to the physical hardware. Misconfigurations in any of these layers will result in the container failing to detect the GPU, rendering the training environment unusable and preventing the execution of accelerated AI tasks.

Answer

The container's network proxy settings.

Answer

The local disk storage capacity.

Question 7

Refer to the exhibit. An engineer is trying to run a multi-node training job. What is the most probable cause of this error?

Accepted Answer

The OFED drivers are not installed or loaded.. The error 'NCCL_ERROR_NET_IB_NOT_FOUND' indicates that the NCCL communication library cannot detect the required InfiniBand hardware. This occurs when the HCA (Host Channel Adapter) is not correctly initialized, the driver is missing, or the PCIe device is not visible to the OS. Without the physical hardware correctly plumbed into the PCIe bus, the NCCL collective operations for multi-node communication will fail to initialize.

Answer

The InfiniBand cabling is loose.

Answer

The NCCL version is incompatible with the GPU.

Answer

The GPU memory is insufficient for NCCL buffers.

Question 8

An AI engineer observes that a model training job on an NVIDIA DGX system is underutilizing the GPU. The monitoring logs show consistent GPU clock throttling. What is the most likely cause?

Accepted Answer

The power limit settings are set below the default factory TDP.. GPU clock throttling during heavy compute tasks often indicates thermal constraints or insufficient power delivery. By analyzing the thermal profile and power limits via nvidia-smi, engineers can identify if the cooling system is failing or if the power supply unit is failing to deliver consistent voltage. Resolving this ensures the GPU maintains its boost clock frequency for optimal training throughput and hardware longevity.

Answer

The batch size is too large, causing memory fragmentation.

Answer

The NVLink fabric is disconnected on one GPU in the cluster.

Answer

The CUDA driver version is incompatible with the installed cuDNN library.

Question 9

During a large-scale training run, an engineer notices 'XID 79' errors in the kernel logs. What is the standard troubleshooting procedure for this specific GPU error?

Accepted Answer

Check for driver version compatibility and perform a clean driver reinstallation.. XID 79 errors indicate a GPU-reported error where the GPU engine has faulted, often due to an invalid instruction or a software-induced hardware exception. These errors are critical because they often lead to system instability. Following the documented NVIDIA XID troubleshooting flow, which includes checking for driver updates and hardware diagnostics, ensures that the issue is not masking a failing physical component or an unstable kernel module.

Answer

Increase the GPU power limit to provide more voltage stability.

Answer

Replace the PCIe riser card and re-seat the GPU hardware.

Answer

Lower the precision to FP16 to reduce the computational complexity.

Question 10

Which THREE factors should be monitored to effectively detect 'Straggler' nodes in a distributed training job?

Accepted Answer

Individual GPU duty cycles across all nodes.. Straggler nodes are nodes that are slower than others, causing the entire cluster to wait and reducing overall training efficiency. Monitoring GPU utilization, communication latency, and synchronization time helps isolate these nodes. Identifying stragglers early allows engineers to rebalance workloads or investigate specific hardware issues on individual nodes, which is vital for maintaining high throughput in large-scale cluster environments.

Answer

The total system CPU utilization percentage.

Answer

The ambient server room temperature.

Question 11

Which tool is best suited for identifying memory leaks within a custom CUDA kernel during the development and optimization phase?

Accepted Answer

NVIDIA Nsight Compute. NVIDIA Nsight Compute is the industry-standard tool for profiling and debugging CUDA kernels. It provides deep visibility into memory usage, occupancy, and register pressure. Using it early in the optimization cycle helps prevent memory leaks and performance bottlenecks before the code reaches production, ensuring that AI models run efficiently and reliably on the target GPU architecture.

Answer

nvidia-smi

Answer

NCCL tests

Answer

DCGM Exporter

Question 12

An AI engineer is running a training job on an NVIDIA DGX system. The training throughput is significantly lower than baseline, and `nvidia-smi` reports high GPU utilization but low memory bandwidth usage. Which optimization step should the engineer prioritize?

Accepted Answer

Profile the application using NVIDIA Nsight Systems to identify kernel hotspots.. High GPU utilization with low memory bandwidth suggests the system is compute-bound rather than memory-bound, often indicating inefficient kernel execution or suboptimal data loading. Optimizing the data pipeline using NVIDIA DALI or refining the model architecture to better fit the GPU cache hierarchy can resolve bottlenecks. Addressing these throughput issues is critical to maximizing ROI on expensive GPU compute resources by reducing training time and energy consumption.

Answer

Increase the batch size significantly to utilize all available VRAM.

Answer

Enable mixed precision training using FP16 or BF16.

Answer

Upgrade the system RAM to provide faster data access to the GPU.

Question 13

An AI administrator needs to monitor GPU health and thermal status in real-time across a fleet of DGX nodes. Which command-line utility provides the most comprehensive persistent monitoring interface for these metrics?

Accepted Answer

dcgmi diag -r 1. NVIDIA Data Center GPU Manager (DCGM) is the industry-standard tool for fleet-level monitoring. Unlike simple one-time checks, DCGM provides persistent telemetry, policy management, and diagnostic capabilities. Effective AI operations rely on continuous monitoring tools to detect hardware degradation or thermal throttling before they result in job failures, ensuring high availability and long-term reliability of the compute infrastructure.

Answer

nvidia-smi -q

Answer

nv-host-monitor

Answer

cat /proc/driver/nvidia/gpus/0/information

Question 14

A Kubernetes cluster using the NVIDIA GPU Operator is experiencing pods stuck in a Pending state with 'Insufficient nvidia.com/gpu' errors. Which TWO steps should be performed to troubleshoot this resource allocation issue?

Accepted Answer

Verify the status of the NVIDIA GPU Operator pods using 'kubectl get pods -n gpu-operator'.. These steps are crucial for identifying discrepancies between expected and available hardware resources in a containerized environment. By verifying node labels and verifying the status of the GPU Operator components, an administrator can determine if the scheduler is failing to recognize the GPUs. This diagnostic process is essential for maintaining high availability in multi-tenant environments where resource requests must match physical capabilities to ensure successful scheduling.

Answer

Run 'nvidia-smi' on the node to verify GPU driver installation.

Answer

Increase the GPU memory limit in the container manifest.

Answer

Restart the kubelet service on all worker nodes.

Question 15

Which THREE actions are recommended when debugging an intermittent GPU hang that occurs only during specific model training iterations?

Accepted Answer

Enable CUDA kernel synchronous error checking by setting CUDA_LAUNCH_BLOCKING=1.. Intermittent hangs are notoriously difficult to debug because they do not leave clear stack traces. By combining kernel-level profiling, error logging, and workload isolation, engineers can narrow down the root cause. This methodology is essential for production environments where GPU stability is paramount, allowing teams to distinguish between application-level coding bugs, kernel-level drivers, or hardware-level thermal issues that only manifest under peak utilization.

Answer

Disable all ECC memory reporting features.

Answer

Run the training loop on a single CPU core.