NCP-AIO Practice Test 15 — 15 Questions

Question 1

A team is running a multi-GPU training job on an NVIDIA DGX A100 system using NCCL for inter-GPU communication. Training throughput is much lower than expected, and the NCCL logs show repeated 'NCCL WARN Call to ibv_reg_mr failed' errors. The job uses a container with host networking. Which action should the AI operations engineer take to resolve the issue?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Verify and increase the container's locked memory limit (ulimit -l) and ensure the NVIDIA driver and OFED stack are compatible.. The NCCL warning 'ibv_reg_mr failed' points to an inability to register memory regions for InfiniBand, commonly caused by a low locked memory limit or mismatched OFED and NVIDIA drivers. In containerized environments, the default ulimit -l is often too low. Increasing the locked memory limit and ensuring driver compatibility directly fixes the registration failure, restoring proper NCCL communication and training throughput.

Answer

Reduce the batch size per GPU to lower memory pressure and avoid the need for large memory registrations.

Answer

Increase the NCCL_IB_TIMEOUT environment variable to allow more time for memory registration.

Answer

Disable InfiniBand by setting NCCL_IB_DISABLE=1 to force NCCL to use TCP sockets.

Question 2

An AI operations engineer is investigating a training job on an NVIDIA DGX system that intermittently fails with 'uncorrectable ECC error' on a GPU. The job is using NCCL for multi-GPU communication. The engineer needs to identify the appropriate immediate actions to diagnose and mitigate the issue. (Choose two.)

Accepted Answer

Drain the affected GPU from the scheduler to prevent new jobs from being assigned.. The immediate actions should include checking ECC error counts with nvidia-smi -q to identify the affected GPU, and draining that GPU from the scheduler to prevent further job failures. These steps diagnose the issue and mitigate risk while preserving the ability to perform maintenance. Restarting, resetting, or adjusting NCCL timeouts do not address the underlying hardware error and may delay proper resolution.

Answer

Increase the NCCL timeout to allow the job to recover from the ECC error.

Answer

Restart the training job immediately to see if the error recurs.

Answer

Use nvidia-smi --gpu-reset to reset the GPU and clear the error state.

Question 3

An administrator is configuring NVIDIA Base Command Manager to manage a cluster of DGX nodes. They want to ensure that when a node's GPU temperature exceeds a defined threshold, the node is automatically drained and an alert is sent to the operations team. Which combination of Base Command Manager features should the administrator configure to achieve this?

Accepted Answer

Health checks with a custom script that triggers a node drain via the Base Command Manager API, and an alert rule that sends a notification.. Base Command Manager provides health checks that can execute custom scripts on nodes. A script can monitor GPU temperature and invoke the Base Command Manager API to drain the node, while an alert rule notifies the team. This leverages native features for automated remediation and alerting, unlike external monitoring stacks or Kubernetes-specific tools.

Answer

NVIDIA GPU Operator with node problem detector, and a Kubernetes taint that evicts pods.

Answer

Prometheus Alertmanager with a webhook that drains the node, and Grafana dashboards for visualization.

Answer

Base Command Manager job scheduler with a preemption policy that kills jobs when temperature is high.

Question 4

An AI operations engineer is troubleshooting a multi-node NCCL training job on an NVIDIA DGX SuperPOD. The job runs but scales poorly: inter-node bandwidth is roughly half of the expected 200 Gb/s per GPU, while intra-node NVLink traffic is at full rate. Running `nvidia-smi topo -m` shows that GPUs in each node are connected to the NICs through the PCIe switch, but the job sets `NCCL_SOCKET_IFNAME` to the management interface. Which action is the most appropriate to resolve the bottleneck?

Accepted Answer

Enable GPUDirect RDMA by allowing the container access to the host IB verbs devices and the `/dev/infiniband` tree, and set `NCCL_IB_HCA` to the compute fabric adapters.. When intra-node NVLink performs at full rate but inter-node bandwidth is roughly half, the job is usually not using the compute fabric. GPUDirect RDMA bypasses host memory copies by letting the HCA access GPU memory directly, and `NCCL_IB_HCA` ensures NCCL selects the correct adapters. Exposing `/dev/infiniband` to the container is required for RDMA verbs. This restores the expected inter-node bandwidth.

Answer

Increase `NCCL_BUFFSIZE` to 16 MB and set `NCCL_NTHREADS` to 8 to raise channel throughput.

Answer

Set `NCCL_P2P_DISABLE=1` so that all inter-node traffic is routed through the CPUs, avoiding PCIe switch contention.

Answer

Bind the job to a single NUMA node using `numactl --cpunodebind=0 --membind=0` to reduce cross-socket traffic.

Question 5

An operations team is troubleshooting a multi-node NCCL training job on an NVIDIA DGX SuperPOD that stalls at initialization and never begins gradient exchange. Running `nccl-tests` with `all_reduce_perf` on the same nodes fails identically, but single-node `all_reduce_perf` succeeds. Which action should the team take first to isolate the fault?

Accepted Answer

Set NCCL_DEBUG=INFO and NCCL_DEBUG_SUBSYS=INIT,COLL,NET on the launcher and inspect which transport (NET/IB or P2P) is selected and where the hang occurs.. Because single-node all_reduce_perf succeeds while multi-node fails, the failure is in inter-node communication setup rather than GPU-local collectives. Enabling NCCL_DEBUG=INFO with INIT, COLL, and NET subsystems exposes which transport NCCL selects, how rings and trees are built, and where connection establishment stalls, giving the team actionable evidence before any configuration change.

Answer

Disable GPUDirect RDMA by setting NCCL_NET_GDR_LEVEL=0 across all nodes to force host-memory staging.

Answer

Increase NCCL_BUFFSIZE to 16 MB on every rank and rerun the job to rule out buffer exhaustion.

Answer

Reduce the number of ranks per node so each process has a dedicated GPU, then rerun the multi-node test.

Question 6

A data science team submits a PyTorch distributed training job to a Kubernetes cluster with the NVIDIA GPU Operator installed. The job's pods repeatedly fail with a CUDA initialization error, while a simple `nvidia-smi` check inside an interactive pod on the same node succeeds. The administrator confirms the node's driver is healthy and the device plugin is advertising GPUs. Which configuration should the administrator verify first?

Accepted Answer

That each training pod includes a GPU resource request or limit so the device plugin injects the driver libraries and device nodes.. In an operator-managed cluster, GPU access is granted during admission and kubelet setup only when a pod requests an extended GPU resource; the device plugin and admission controller then inject the driver libraries, CUDA binaries, and device nodes. A pod without that request runs with no visible device, which is precisely why CUDA initialization fails while nvidia-smi succeeds in a different pod on the same node. The other options concern version skew, a specific capability, or a legacy runtime setting that would not produce this selective symptom.

Answer

That the node's `/etc/docker/daemon.json` still lists `nvidia` as the default runtime for all containers.

Answer

That the pods' securityContext drops the `IPC_LOCK` capability required for pinned host memory.

Answer

That the training pods' containers were built with a CUDA toolkit version newer than the node driver supports.

Question 7

An administrator supports a multi-tenant cluster where several teams share GPUs. Leadership requires that each team's batch jobs receive a fair share of GPU time and that one team cannot monopolize devices by submitting thousands of low-priority pods. Jobs are submitted through a Kubernetes-native batch scheduler that supports queueing. Which approach best enforces fair-share GPU allocation across teams?

Accepted Answer

Define per-team queues with weighted fair-share policies and GPU-aware scheduling in the batch scheduler. Weighted fair-share queues in a GPU-aware batch scheduler allocate device time proportionally among teams and enforce queue-level limits, preventing a single tenant from monopolizing GPUs by volume. The default scheduler, equal priorities, or static node partitioning cannot provide dynamic proportional sharing or gang scheduling for distributed jobs.

Answer

Use node taints and tolerations to dedicate specific GPU nodes to each team permanently

Answer

Assign each team a distinct Kubernetes namespace and rely on the default kube-scheduler for GPU placement

Answer

Apply identical PriorityClass values to all team pods so the scheduler treats them equally

Question 8

An AI operations engineer notices that a real-time inference service on an NVIDIA T4 GPU has highly variable latency, with occasional spikes to over 100 ms. The service uses TensorRT and runs in a Docker container. Which action should the engineer take to reduce latency variability?

Accepted Answer

Lock the GPU clocks to a fixed frequency using nvidia-smi -lgc.. Latency variability in GPU inference often results from dynamic clock adjustments as the GPU balances power and thermal constraints. Locking clocks to a stable frequency eliminates this variability, providing consistent execution times. Other options either add latency (dynamic batching), address different issues (persistence mode), or do not target GPU clock behavior.

Answer

Enable dynamic batching in the inference server configuration.

Answer

Increase the number of CPU cores allocated to the container.

Answer

Set the GPU to persistence mode using nvidia-smi -pm 1.

Question 9

An administrator manages an NVIDIA AI Enterprise cluster using NVIDIA Run:ai. A data science team complains that their submitted training job has been stuck in a Pending state for over an hour, even though the Run:ai scheduler shows free GPUs in the cluster. The administrator verifies that the job requests 2 GPUs and the node pool has 4 idle GPUs. Which Run:ai administrative configuration is the most likely cause of the job remaining Pending?

Accepted Answer

The project's GPU quota is exhausted by other running workloads, so the scheduler cannot allocate the requested GPUs.. Run:ai uses projects to enforce GPU quotas. When a project's quota is fully consumed, new jobs remain Pending even if the cluster has idle GPUs. The administrator should check the project's quota usage and either increase the quota or free resources. Node affinity, missing GPU Operator, or missing Container Toolkit would produce different symptoms.

Answer

The job's container image does not include the NVIDIA Container Toolkit, so the scheduler cannot start the job.

Answer

The NVIDIA GPU Operator is not installed on the cluster, so the scheduler cannot see GPU resources.

Answer

The node pool is configured with a node affinity that excludes the nodes with idle GPUs, so the scheduler cannot place the job.

Question 10

A research team runs a multi-node distributed training job spanning eight GPU nodes. Jobs frequently begin execution before all worker pods are running, and the collective initialization hangs until the operator manually scales the job down and up. The administrator wants the scheduler to admit the job only when all of its pods can be placed together. Which mechanism should be used?

Accepted Answer

Configure gang scheduling through a scheduler plugin so the job's pods are placed atomically only when the full group can be accommodated.. Collective initialization requires all ranks to be present before computation proceeds, so partial startup leads to hangs. Gang scheduling solves this by treating the job group as a single scheduling unit and admitting it only when every member fits. Scheduler plugins that implement coscheduling or gang semantics provide this all-or-nothing placement, which is why they are standard in large distributed training environments.

Answer

Increase the kube-scheduler's default backoff period so pods that fail to schedule retry less aggressively and wait for peers to become ready.

Answer

Set a high PriorityClass on every worker pod so the scheduler treats the group as important and places members as soon as resources appear.

Answer

Add a pod anti-affinity rule requiring each worker to run on a distinct node so that placement spreads evenly across the eight GPU nodes.

Question 11

An administrator is tuning a Kubernetes cluster that runs many small inference pods on NVIDIA GPUs. Utilization is low because each pod reserves a full GPU while using only a fraction of its memory and compute. The administrator wants to share GPUs across pods while preserving memory-level isolation between processes. Which TWO configurations achieve this? (Choose two.)

Accepted Answer

Enable the NVIDIA MPS control daemon through the GPU Operator and configure pods to share a GPU with memory limits. MPS and MIG are the two NVIDIA mechanisms that allow several pods to share a physical GPU while maintaining isolation. MPS enforces per-client memory limits and concurrent execution under a control daemon, while MIG provides hardware-level partitioning with dedicated memory and compute slices advertised as distinct resources. Time-slicing shares a GPU without memory isolation, and environment variables or LimitRanges cannot partition GPU resources.

Answer

Set the CUDA_VISIBLE_DEVICES environment variable in each pod to a different index of the same physical GPU

Answer

Enable time-slicing in the device plugin so multiple pods receive the same GPU replica with no memory limit

Answer

Apply a LimitRange that sets a memory limit on the nvidia.com/gpu resource in the namespace

Question 12

An administrator is tuning a Kubernetes cluster that runs GPU Operator. Users report that GPU jobs are sometimes scheduled onto nodes whose drivers are older than the CUDA version the container needs, causing runtime failures. The administrator wants to prevent incompatible placements before pods are bound. (Choose two.)

Accepted Answer

Advertise the driver and CUDA versions as node labels through the GPU Operator's node feature discovery, then use nodeAffinity on the pods to require a compatible version.. Preventing incompatible placement requires exposing driver and CUDA versions as schedulable node attributes and then constraining pods against them. Node feature discovery under the GPU Operator publishes version labels, and nodeAffinity lets pods require a compatible generation. A validating admission policy adds a second layer by rejecting pods whose declared CUDA needs exceed the target node's advertised driver, catching mistakes before binding and avoiding the runtime failures users report.

Answer

Rely on the device plugin to compare the container's CUDA version against the node driver and refuse to allocate the GPU when they mismatch.

Answer

Enable time-slicing with a replica count high enough that every node advertises spare GPU capacity, so incompatible nodes are simply bypassed.

Answer

Set the container image's CUDA version to match the newest driver in the cluster and rely on the container runtime to downgrade the driver automatically.

Question 13

A team is diagnosing a training job that intermittently stalls for several seconds at the start of each epoch. The job uses a distributed data loader and an NVIDIA DGX system with local NVMe. Monitoring shows GPU utilization dropping to near zero during the stalls while host CPU utilization spikes. Which two actions should the AI operations engineer take to identify and mitigate the stall? (Choose two.)

Accepted Answer

Use `nvidia-smi dmon` to capture per-second GPU utilization and power draw while the job runs, correlating the drop with the stall window.. The signature of host CPU spikes with idle GPUs is input pipeline starvation. Prefetching with multiple worker processes and pinned memory keeps the device fed, removing the serialized load at epoch boundaries. Concurrently, `nvidia-smi dmon` supplies the time-series evidence that ties the utilization dips to the stall windows, confirming the diagnosis before and after the fix.

Answer

Increase the batch size until the GPU memory is nearly full to amortize the data loading cost.

Answer

Set `CUDA_LAUNCH_BLOCKING=1` to serialize kernel launches and make the stall easier to observe.

Answer

Move the dataset from local NVMe to a networked file system to rule out local disk contention.

Question 14

An AI operations engineer is troubleshooting a multi-GPU training job that fails intermittently with a NCCL error: 'unhandled system error'. The job runs on a DGX-1 with eight V100 GPUs connected via NVLink. Which step should the engineer take first to resolve the issue?

Accepted Answer

Set NCCL_DEBUG=INFO and reproduce the failure to capture detailed logs.. The first step in troubleshooting any NCCL error should be to gather detailed logs using NCCL_DEBUG=INFO. This provides visibility into the communication path and error specifics, enabling targeted fixes. Other options either mask the issue, involve unnecessary reinstallation, or reduce resources without diagnosis.

Answer

Disable NVLink by setting NCCL_P2P_DISABLE=1 to force PCIe communication.

Answer

Reinstall the NVIDIA driver and CUDA toolkit on all nodes.

Answer

Reduce the number of GPUs used in the job to four to see if the error disappears.

Question 15

An administrator notices that GPU utilization on a training cluster hovers around 25 percent even though many jobs are queued. Investigation shows that each job requests a full GPU, but the models are small and alternate between short data-loading phases and brief compute bursts. The administrator wants to increase effective GPU utilization without changing model code. Which action should be taken first?

Accepted Answer

Enable GPU time-slicing or MIG so multiple small jobs can share a device concurrently. When small, bursty jobs each hold a full GPU exclusively, the device idles during data-loading windows. Enabling time-slicing or MIG allows multiple jobs to share the device concurrently, filling those idle gaps and raising effective utilization without touching model code or adding hardware.

Answer

Move data loading to CPU-only nodes to eliminate the idle phases

Answer

Add more GPU nodes to the cluster so queued jobs start sooner

Answer

Increase the batch size of each job so compute bursts last longer