NCP-AIO Practice Test 16 — 15 Questions

Question 1

An administrator is preparing a bare-metal GPU server for AI workloads and needs to verify that the NVIDIA driver and CUDA toolkit are properly installed. The server has an NVIDIA A100 GPU. Which command should the administrator run to display the GPU model, driver version, and CUDA version?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

nvidia-smi. The nvidia-smi utility is the primary interface for monitoring and managing NVIDIA GPUs. It provides a summary of each GPU's model, driver version, CUDA version, temperature, power usage, and memory utilization. For an administrator validating a new server, running nvidia-smi immediately confirms that the driver is loaded and the GPU is recognized, and it shows the CUDA version supported by the driver, which is essential for AI workload compatibility.

Answer

nvidia-debugdump --list

Answer

nvcc --version

Answer

lspci | grep -i nvidia

Question 2

A monitoring system reports that a DGX node's GPUs are running at reduced clocks during a long training job, and `nvidia-smi -q -d PERFORMANCE` shows the throttle reason as 'SW Power Cap'. The job's power draw is at the configured limit. Which action is MOST appropriate to restore higher clocks?

Accepted Answer

Raise the GPU power limit with `nvidia-smi -pl` to a value within the GPU's supported maximum and re-run the job.. The throttle reason 'SW Power Cap' indicates the GPU is constrained by its configured power limit. Raising the power limit to a supported value within the GPU's validated maximum allows the device to sustain higher clocks, provided cooling and power delivery can support it. This directly removes the cause of the reduced clocks rather than working around it.

Answer

Reduce the batch size so the GPU consumes less power and stays below the cap.

Answer

Enable persistence mode with `nvidia-smi -pm 1` to keep the driver loaded and improve clock stability.

Answer

Lock the GPU clocks to their maximum with `nvidia-smi -lgc` to override the throttle.

Question 3

An AI operations engineer is investigating an inference service running on NVIDIA Triton Inference Server. Clients report sporadic 500 errors under peak load. The server logs show occasional 'Failed to allocate memory' messages, and `nvidia-smi` shows VRAM nearly full. The service uses dynamic batching with a maximum batch size of 64 and multiple model instances per GPU. Which change is the most appropriate first step to stabilize the service?

Accepted Answer

Reduce the number of model instances per GPU and lower the maximum dynamic batch size to match the available VRAM headroom.. Triton allocates memory per model instance and per active batch execution. When instances and the maximum dynamic batch size are sized for average load, peak concurrency can exceed VRAM and cause allocation failures. Reducing instances and capping batch size brings the peak footprint within budget, letting the scheduler queue requests instead of failing them, which stabilizes error rates under load.

Answer

Switch the service to use the CPU execution provider for overflow requests when GPU memory is exhausted.

Answer

Enable `--strict-model-config=false` so Triton can auto-generate the model configuration and allocate memory more efficiently.

Answer

Increase the GPU memory pool by setting `CUDA_MPS_ACTIVE_THREAD_PERCENTAGE` to 100.

Question 4

An operations team observes that a distributed training job using NVIDIA Collective Communications Library (NCCL) across eight nodes occasionally hangs during the all-reduce phase. Logs show no errors, and the hang resolves only after a node is manually restarted. Which action is MOST appropriate to diagnose the intermittent hang?

Accepted Answer

Enable NCCL debug logging with NCCL_DEBUG=INFO and NCCL_DEBUG_SUBSYS=COLL, and set a NCCL watchdog timeout to capture the stalled collective.. Intermittent NCCL hangs without errors are best diagnosed by capturing state at the moment of the stall. Enabling NCCL debug logging for the collective subsystem and configuring a watchdog timeout produces logs that show which rank and collective stopped progressing. This evidence is necessary to distinguish a slow rank, a fabric problem, or a software defect from a simple performance issue.

Answer

Disable InfiniBand and fall back to TCP sockets by setting NCCL_IB_DISABLE=1.

Answer

Increase the NCCL buffer size with NCCL_BUFFSIZE to reduce the number of messages exchanged.

Answer

Set NCCL_ALGO=RING to force a single algorithm and eliminate variability.

Question 5

An AI operations engineer is optimizing a PyTorch training job on an NVIDIA DGX A100 system. The job uses a data loader with multiple workers, but the engineer observes that GPU utilization fluctuates between 40% and 60%, and `nvidia-smi dmon` shows periods of zero GPU utilization. The engineer suspects that the data input pipeline is the bottleneck. Which TWO actions should the engineer take to improve GPU utilization? (Choose two.)

Accepted Answer

Increase the number of DataLoader workers to better overlap data loading with GPU computation.. The observed GPU utilization fluctuations and zero-utilization periods indicate the GPU is often waiting for data. Increasing DataLoader workers parallelizes data preprocessing, and enabling pinned memory accelerates host-to-device transfers. Together, these reduce data loading latency and improve overlap with GPU computation, raising utilization. Other options either do not address the bottleneck or harm performance.

Answer

Decrease the batch size to reduce memory usage and allow more frequent updates.

Answer

Set the CUDA_LAUNCH_BLOCKING environment variable to 1 to ensure synchronous kernel execution.

Answer

Enable mixed precision training to reduce memory footprint and increase throughput.

Question 6

A distributed training job using PyTorch DDP across eight GPUs on one DGX A100 node shows GPU utilization oscillating between 20 and 40 percent, while `nvidia-smi dmon` shows low SM activity but sustained high memory-controller utilization. The data loader reads from a local NVMe RAID array and applies heavy CPU augmentation. Which action best improves GPU utilization?

Accepted Answer

Increase the number of DataLoader worker processes and enable pinned memory with non-blocking host-to-device transfers.. Oscillating SM utilization with high memory-controller traffic is the classic signature of an input pipeline that cannot keep pace with compute. Adding DataLoader workers parallelizes augmentation across CPU cores, and pinned memory plus non-blocking transfers let DMA copies overlap with kernels. This keeps the GPUs fed continuously, converting idle gaps into productive compute and raising sustained utilization without altering the model or optimizer.

Answer

Enable NCCL gradient compression to reduce inter-GPU communication volume.

Answer

Reduce the per-GPU batch size so each step completes faster and utilization appears smoother.

Answer

Switch the job from DDP to a single-process data-parallel loop that iterates over all GPUs in Python.

Question 7

An AI operations engineer manages a shared Kubernetes cluster where several teams submit GPU jobs. The engineer must prevent any single namespace from consuming all GPU capacity and must also ensure that jobs from one team cannot starve others during peak periods. Which combination of Kubernetes and NVIDIA GPU Operator features should the engineer implement?

Accepted Answer

Define a ResourceQuota on nvidia.com/gpu per namespace and use the GPU Operator's device plugin to advertise capacity so the quota can enforce limits.. ResourceQuota is the Kubernetes mechanism that caps aggregate resource consumption per namespace, and it works for GPU extended resources once the NVIDIA device plugin advertises them. By setting a quota on nvidia.com/gpu, the administrator prevents any single namespace from consuming all GPUs, ensuring other teams retain capacity. This directly addresses both the monopolization and starvation concerns.

Answer

Enable the MIG Manager and assign a fixed mig profile to every namespace through a LimitRange.

Answer

Apply a NetworkPolicy that restricts pod-to-pod traffic between namespaces and rely on it to limit GPU consumption.

Answer

Set a node taint for nvidia.com/gpu and add tolerations only to the highest-priority team's pods.

Question 8

A production inference service running on NVIDIA T4 GPUs shows that GPU utilization is consistently below 20% while request latency is high. Profiling with Nsight Systems reveals that the model execution time is short but there are frequent gaps between kernels. Which optimization should be applied first to improve GPU utilization?

Accepted Answer

Increase the batch size in the inference server configuration.. The low GPU utilization and gaps between kernels indicate that the GPU is not receiving enough work per launch. Increasing the batch size allows more data to be processed per kernel, filling the gaps and raising utilization. This is the most direct and effective first step before considering more complex techniques like CUDA graphs or precision changes.

Answer

Switch from FP32 to FP16 precision for the model weights.

Answer

Increase the number of concurrent model instances on each GPU.

Answer

Enable CUDA graphs to capture and replay the inference sequence.

Question 9

An administrator is deploying NVIDIA AI Enterprise on a vSphere cluster with multiple ESXi hosts, each containing several NVIDIA A100 GPUs. They need to enable GPU sharing across multiple virtual machines to maximize utilization. Which vSphere feature should they configure?

Accepted Answer

NVIDIA vGPU. NVIDIA vGPU is the vSphere feature that partitions a physical GPU into multiple virtual GPUs, allowing multiple VMs to share the same physical GPU. This maximizes GPU utilization and is designed for AI workloads. DirectPath I/O provides exclusive access, while DRS and SR-IOV do not enable GPU sharing.

Answer

vSphere DRS

Answer

SR-IOV

Answer

DirectPath I/O

Question 10

A site reliability engineer is troubleshooting a DGX A100 node that intermittently drops out of the cluster during large NCCL all-reduce jobs. `nvidia-smi` shows all eight A100 GPUs healthy, but DCGM reports XID errors 74 and 79 on one GPU during the failures. The engineer needs to determine the most likely cause and the correct administrative action. Which combination best describes the cause and the appropriate first step?

Accepted Answer

The errors point to a GPU falling off the bus or an NVLink error; run `nvidia-smi -q` and DCGM diagnostics on the suspect GPU, then consider reseating or replacing the GPU board.. XID 74 and 79 are driver-reported errors indicating a GPU has fallen off the bus or an NVLink error has occurred. These are hardware and link-level faults, so the right first step is to gather detailed GPU and NVLink diagnostics with `nvidia-smi -q` and DCGM, then remediate the hardware. Ignoring or masking these errors risks job failures and data corruption, while treating them as thermal or software issues misses the actual fault domain.

Answer

The errors are benign and expected during large all-reduce jobs; suppress them by setting the DCGM health check to ignore XID 74 and 79.

Answer

The errors indicate a thermal shutdown; immediately lower the GPU clock with `nvidia-smi -lgc` and rerun the job.

Answer

The errors are caused by an outdated NCCL version; upgrade NCCL to the latest release and rerun the job without further hardware checks.

Question 11

An administrator is responsible for maintaining an NVIDIA AI Enterprise cluster and needs to ensure high availability of GPU resources for critical inference workloads. Which two practices should the administrator implement? (Choose two.)

Accepted Answer

Implement pod disruption budgets to maintain a minimum number of available replicas. High availability for inference workloads requires spreading pods across multiple nodes to avoid single points of failure and using pod disruption budgets to maintain minimum replica counts during disruptions. Node anti-affinity ensures distribution, while PDBs protect against voluntary evictions. Time-slicing, MIG, and single-node concentration do not provide fault tolerance against node or GPU failures.

Answer

Enable GPU time-slicing to allow multiple pods to share a single GPU

Answer

Schedule all inference pods on a single node with multiple GPUs to reduce network latency

Answer

Use NVIDIA MIG to partition a GPU into isolated instances for each workload

Question 12

An inference service runs a 70B parameter model with TensorRT-LLM on a single H100 using in-flight batching. Operators report that time-to-first-token is acceptable, but inter-token latency degrades sharply once concurrent request count exceeds a certain point, and GPU memory utilization sits near 98 percent. Which change most directly addresses the inter-token latency degradation?

Accepted Answer

Enable FP8 quantization for the KV cache and reduce the maximum batch size in the TensorRT-LLM build configuration.. When GPU memory is nearly exhausted, the runtime cannot hold enough KV cache plus workspace to serve all admitted sequences efficiently, so each decode step stretches and inter-token latency rises with concurrency. Shrinking the KV cache with FP8 quantization and lowering the maximum batch size reduces per-step memory and compute demand, restoring shorter decode iterations. These changes target the resource constraint that actually causes the latency curve to bend upward.

Answer

Disable in-flight batching so each request is processed to completion before the next begins.

Answer

Increase the maximum batch size so more requests are processed per decode step.

Answer

Switch the service to a round-robin load balancer across two replicas of the same model on one GPU.

Question 13

An AI operations team is troubleshooting a training job that crashes with a segmentation fault after several hours. The job uses multiple GPUs and NCCL for communication. System logs show no errors, but dmesg reveals repeated 'NVRM: Xid' errors. Which action should be taken first to diagnose the issue?

Accepted Answer

Check the Xid error code in the NVIDIA documentation to identify the specific GPU fault.. Xid errors are logged by the NVIDIA driver and each code maps to a specific GPU fault. Interpreting the Xid code is the fastest way to understand whether the segmentation fault is due to a hardware issue, driver bug, or application error. This guides subsequent troubleshooting steps, such as replacing hardware or updating the driver.

Answer

Restart the NVIDIA driver with rmmod and modprobe commands.

Answer

Enable NCCL_DEBUG=INFO and re-run the job to capture detailed NCCL logs.

Answer

Run nvidia-smi -q to check GPU temperature and power usage.

Question 14

An AI operations team is deploying NVIDIA Base Command Manager to manage a cluster of DGX nodes. They want to ensure that only authorized users can submit jobs and that all job submissions are audited. Which combination of Base Command Manager features should the administrator configure to meet these requirements?

Accepted Answer

Integrate Base Command Manager with an LDAP or Active Directory identity provider and enable the audit logging feature.. Base Command Manager centralizes cluster administration, including user authentication and job scheduling. Integrating with LDAP or Active Directory ensures only authorized users can authenticate and submit jobs, while audit logging records those submissions for compliance. This combination directly addresses both the access control and auditing requirements in the scenario.

Answer

Deploy a separate Kubernetes cluster with RBAC policies and use Base Command Manager only for node provisioning.

Answer

Configure IPMI access controls on each DGX node and enable the Base Command Manager telemetry collector.

Answer

Enable SELinux enforcing mode on all nodes and configure Base Command Manager to use SSH key-based authentication only.

Question 15

An administrator is preparing a multi-node NVIDIA DGX H100 cluster for a distributed training job using NVIDIA Base Command. The cluster nodes have InfiniBand adapters, but the job's inter-node throughput is far below expectations. The administrator runs `ibstat` and sees that the ports are in the INIT state rather than ACTIVE. Which action should the administrator take first?

Accepted Answer

Verify that the InfiniBand subnet manager is running and that the IPoIB interfaces are configured correctly.. InfiniBand ports stuck in INIT have not been configured by a subnet manager, which is the authoritative service that initializes and activates fabric links. Checking the subnet manager and IPoIB configuration directly addresses why ports never reach ACTIVE. Without an active subnet manager, distributed training traffic cannot flow at expected speeds, so this is the correct first action.

Answer

Enable GPUDirect Storage on each node to bypass the CPU for inter-node communication.

Answer

Reinstall the NVIDIA GPU driver on all nodes to reset the InfiniBand firmware.

Answer

Disable ECC memory on the GPUs to increase available bandwidth for inter-node traffic.