NCP-AIO Administration Practice Test 4 — 15 Questions

Question 1

An administrator manages an NVIDIA AI Enterprise cluster running multiple Kubernetes nodes, each with several A100 GPUs. After upgrading the NVIDIA GPU Operator to a newer version, the administrator notices that pods requesting GPUs remain in a Pending state, and the node's allocatable GPU count is reported as zero. Which command should the administrator run first to diagnose the issue?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

kubectl describe node <node-name>. The node's allocatable GPU count is reported as zero, indicating that the kubelet is not receiving GPU resource advertisements from the NVIDIA device plugin. Describing the node reveals capacity, allocatable resources, and relevant events, such as device plugin registration failures. This directly identifies whether the GPU Operator's device plugin is functioning, making it the correct first step.

Answer

kubectl get pods --all-namespaces -o wide

Answer

nvidia-smi -q

Answer

kubectl logs -n gpu-operator <gpu-operator-pod>

Question 2

An administrator manages an NVIDIA AI Enterprise cluster and needs to enforce GPU resource quotas across multiple Kubernetes namespaces. Which NVIDIA component should be configured to enforce these quotas?

Accepted Answer

Kubernetes ResourceQuota with extended resources. Kubernetes ResourceQuota is the native mechanism to limit aggregate resource consumption per namespace, including extended resources like nvidia.com/gpu. When the NVIDIA device plugin advertises GPUs, administrators can define a ResourceQuota specifying a maximum for nvidia.com/gpu, thereby enforcing GPU quotas. Other NVIDIA tools focus on deployment, management, or monitoring and do not provide quota enforcement.

Answer

NVIDIA DCGM Exporter

Answer

NVIDIA Base Command Manager

Answer

NVIDIA GPU Operator

Question 3

An administrator is deploying NVIDIA AI Enterprise on a Kubernetes cluster and wants to verify that the GPU Operator has successfully installed all required components. Which command should the administrator use to check the status of the GPU Operator pods?

Accepted Answer

kubectl get pods -n gpu-operator. The GPU Operator installs its components as pods in the gpu-operator namespace. Checking pod status with kubectl get pods -n gpu-operator is the correct way to verify that all required components are running. Other commands provide GPU-level or node-level information but do not show the status of the operator's pods.

Answer

nvidia-smi -q

Answer

kubectl get nodes -o wide

Answer

dcgmi discovery -l

Question 4

An administrator manages an NVIDIA AI Enterprise cluster running multiple Kubernetes nodes, each with several A100 GPUs. After upgrading the NVIDIA GPU Operator to a newer version, the administrator notices that pods requesting GPUs remain in a Pending state, and the node's allocatable GPU count is reported as zero. Which command should the administrator run first to diagnose the issue?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

The project's GPU quota is exhausted by other running workloads, so the scheduler cannot allocate the requested GPUs.. Run:ai uses projects to enforce GPU quotas. When a project's quota is fully consumed, new jobs remain Pending even if the cluster has idle GPUs. The administrator should check the project's quota usage and either increase the quota or free resources. Node affinity, missing GPU Operator, or missing Container Toolkit would produce different symptoms.

Answer

The job's container image does not include the NVIDIA Container Toolkit, so the scheduler cannot start the job.

Answer

The NVIDIA GPU Operator is not installed on the cluster, so the scheduler cannot see GPU resources.

Answer

The node pool is configured with a node affinity that excludes the nodes with idle GPUs, so the scheduler cannot place the job.

Question 5

An administrator is configuring NVIDIA Base Command Manager to manage a cluster of DGX nodes. They want to ensure that when a node's GPU temperature exceeds a defined threshold, the node is automatically drained and an alert is sent to the operations team. Which combination of Base Command Manager features should the administrator configure to achieve this?

Accepted Answer

Health checks with a custom script that triggers a node drain via the Base Command Manager API, and an alert rule that sends a notification.. Base Command Manager provides health checks that can execute custom scripts on nodes. A script can monitor GPU temperature and invoke the Base Command Manager API to drain the node, while an alert rule notifies the team. This leverages native features for automated remediation and alerting, unlike external monitoring stacks or Kubernetes-specific tools.

Answer

NVIDIA GPU Operator with node problem detector, and a Kubernetes taint that evicts pods.

Answer

Prometheus Alertmanager with a webhook that drains the node, and Grafana dashboards for visualization.

Answer

Base Command Manager job scheduler with a preemption policy that kills jobs when temperature is high.

Question 6

An administrator is preparing a bare-metal GPU server for AI workloads and needs to verify that the NVIDIA driver and CUDA toolkit are properly installed. The server has an NVIDIA A100 GPU. Which command should the administrator run to display the GPU model, driver version, and CUDA version?

Accepted Answer

nvidia-smi. The nvidia-smi utility is the primary interface for monitoring and managing NVIDIA GPUs. It provides a summary of each GPU's model, driver version, CUDA version, temperature, power usage, and memory utilization. For an administrator validating a new server, running nvidia-smi immediately confirms that the driver is loaded and the GPU is recognized, and it shows the CUDA version supported by the driver, which is essential for AI workload compatibility.

Answer

nvidia-debugdump --list

Answer

nvcc --version

Answer

lspci | grep -i nvidia

Question 7

An administrator is responsible for maintaining an NVIDIA AI Enterprise cluster and needs to ensure high availability of GPU resources for critical inference workloads. Which two practices should the administrator implement? (Choose two.)

Accepted Answer

Implement pod disruption budgets to maintain a minimum number of available replicas. High availability for inference workloads requires spreading pods across multiple nodes to avoid single points of failure and using pod disruption budgets to maintain minimum replica counts during disruptions. Node anti-affinity ensures distribution, while PDBs protect against voluntary evictions. Time-slicing, MIG, and single-node concentration do not provide fault tolerance against node or GPU failures.

Answer

Enable GPU time-slicing to allow multiple pods to share a single GPU

Answer

Schedule all inference pods on a single node with multiple GPUs to reduce network latency

Answer

Use NVIDIA MIG to partition a GPU into isolated instances for each workload

Question 8

An administrator is deploying NVIDIA AI Enterprise on a vSphere cluster with multiple ESXi hosts, each containing several NVIDIA A100 GPUs. They need to enable GPU sharing across multiple virtual machines to maximize utilization. Which vSphere feature should they configure?

Accepted Answer

NVIDIA vGPU. NVIDIA vGPU is the vSphere feature that partitions a physical GPU into multiple virtual GPUs, allowing multiple VMs to share the same physical GPU. This maximizes GPU utilization and is designed for AI workloads. DirectPath I/O provides exclusive access, while DRS and SR-IOV do not enable GPU sharing.

Answer

vSphere DRS

Answer

SR-IOV

Answer

DirectPath I/O

Question 9

An administrator is preparing a multi-node NVIDIA DGX H100 cluster for a distributed training job using NVIDIA Base Command. The cluster nodes have InfiniBand adapters, but the job's inter-node throughput is far below expectations. The administrator runs `ibstat` and sees that the ports are in the INIT state rather than ACTIVE. Which action should the administrator take first?

Accepted Answer

Verify that the InfiniBand subnet manager is running and that the IPoIB interfaces are configured correctly.. InfiniBand ports stuck in INIT have not been configured by a subnet manager, which is the authoritative service that initializes and activates fabric links. Checking the subnet manager and IPoIB configuration directly addresses why ports never reach ACTIVE. Without an active subnet manager, distributed training traffic cannot flow at expected speeds, so this is the correct first action.

Answer

Enable GPUDirect Storage on each node to bypass the CPU for inter-node communication.

Answer

Reinstall the NVIDIA GPU driver on all nodes to reset the InfiniBand firmware.

Answer

Disable ECC memory on the GPUs to increase available bandwidth for inter-node traffic.

Question 10

An administrator needs to collect GPU telemetry from an NVIDIA AI Enterprise cluster and store it in a time-series database for long-term analysis. Which component should be deployed to export GPU metrics in Prometheus format?

Accepted Answer

NVIDIA DCGM Exporter. DCGM Exporter is the component that collects GPU metrics via DCGM and exposes them as Prometheus metrics. It is typically deployed as a daemonset or pod and can be scraped by Prometheus for long-term storage and analysis. Other NVIDIA components manage deployment or container runtime integration but do not provide metric export functionality.

Answer

NVIDIA Container Toolkit

Answer

NVIDIA Base Command Manager

Answer

NVIDIA GPU Operator

Question 11

An administrator manages an NVIDIA AI Enterprise cluster running multiple Kubernetes nodes, each with several A100 GPUs. After upgrading the NVIDIA GPU Operator to a newer version, the administrator notices that pods requesting GPUs remain in a Pending state, and the node's allocatable GPU count is reported as zero. Which command should the administrator run first to diagnose the issue?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

The DCGM Exporter ServiceMonitor or PodMonitor selector labels match the Prometheus operator's serviceMonitorSelector.. In Prometheus Operator deployments, scrape targets are selected by label matching between the ServiceMonitor or PodMonitor and the Prometheus custom resource. If the DCGM Exporter's monitor labels do not align with the operator's selector, Prometheus never scrapes it, producing exactly the symptom of no GPU metrics. Verifying label alignment is therefore the correct first check.

Answer

The NVIDIA GPU Operator's device plugin is configured to advertise GPUs with the correct resource name.

Answer

The GPU driver version installed on the nodes matches the version bundled in the DCGM Exporter container.

Answer

The cluster's CNI plugin supports multicast so that Prometheus can discover the exporter.

Question 12

An administrator supports a shared Kubernetes cluster running NVIDIA GPU Operator. Data scientists report that their inference pods remain in Pending state, yet the GPU Operator pods and node feature discovery pods are healthy, and the GPU nodes show no hardware alarms. The administrator confirms that the cluster has a mixture of MIG-capable A100 nodes and non-MIG T4 nodes. Which immediate administrative action is most appropriate to diagnose the scheduling failure?

Accepted Answer

Inspect the pod events with kubectl describe pod and verify the MIG strategy configured in the device plugin, because a mismatch between the node's MIG configuration and the requested resource can prevent GPU resource advertisement.. The pod events are the fastest way to see whether the scheduler cannot find a suitable node due to resource requests. In a mixed MIG and non-MIG environment, the device plugin advertises different resource names, and a mismatch between the requested GPU resource and what the node exposes leaves pods Pending. Verifying the MIG strategy and node labels directly identifies that mismatch.

Answer

Restart the kubelet on all worker nodes to force the device plugin to re-register GPU resources, because a stale kubelet cache is the most frequent cause of Pending workloads.

Answer

Delete the pending pods and recreate them with a higher priority class, because the scheduler may be preempting them in favor of system pods.

Answer

Scale the GPU Operator controller manager to zero replicas and back to one, because the operator may have lost track of node labels after a recent upgrade.

Question 13

An administrator is setting up an NVIDIA AI Enterprise cluster and wants to verify that the NVIDIA GPU Operator has successfully deployed all required components on a worker node. Which command should the administrator use to list the GPU Operator pods running on that node?

Accepted Answer

kubectl get pods -n gpu-operator --field-selector spec.nodeName=<node-name>. The NVIDIA GPU Operator runs its components as pods in the gpu-operator namespace. To confirm deployment on a specific worker node, filtering pods by spec.nodeName in that namespace lists exactly the operator pods scheduled there. This provides direct evidence that the driver, container toolkit, device plugin, and related daemonsets are running on the node.

Answer

nvidia-smi -q -d COMPUTE

Answer

kubectl describe node <node-name> | grep nvidia

Answer

helm list -n gpu-operator

Question 14

An AI operations engineer is preparing a DGX H100 system for a multi-node training workload. The engineer runs `nvidia-smi topo -m` and notices that GPU4 and GPU5 report a connection type of SYS, while all other GPU pairs show NV18. What is the most likely cause of this topology anomaly?

Accepted Answer

GPU4 and GPU5 are connected through the PCIe host bridge rather than the NVSwitch fabric, possibly due to a degraded or missing NVLink connection.. A healthy DGX H100 shows NV18 (NVLink 4.0, 18 links) for every GPU pair because all GPUs are connected through NVSwitch. A SYS entry for a specific pair means traffic between those GPUs falls back to the PCIe/CPU path, which severely reduces bandwidth and increases latency. This typically signals a hardware or cabling fault on the NVLink side that should be resolved before running distributed training.

Answer

The NVSwitch fabric is operating in a fallback mode that only activates when peer-to-peer traffic exceeds a threshold.

Answer

GPU4 and GPU5 are configured in MIG mode, which forces inter-GPU communication over the PCIe bus.

Answer

GPU4 and GPU5 are reserved for display output, so the driver automatically disables their NVLink interfaces.

Question 15

An administrator is setting up NVIDIA Base Command Manager to provision and manage a new AI cluster. They need to ensure that the cluster can automatically discover and configure new GPU nodes. Which component is responsible for node discovery and initial configuration?

Accepted Answer

NVIDIA Base Command Manager head node. In NVIDIA Base Command Manager, the head node is the central management server that performs node discovery and initial configuration. It provides the provisioning services that allow new GPU nodes to be automatically detected and configured. The GPU Operator, Fleet Command, and Morpheus serve different purposes and are not involved in the initial provisioning of a Base Command Manager cluster.

Answer

NVIDIA Fleet Command

Answer

NVIDIA GPU Operator

Answer

NVIDIA Morpheus