NCP-AIO Administration Practice Test 2 — 15 Questions

Question 1

An administrator is troubleshooting a DGX system where a training job intermittently fails with XID 48 errors in the system logs. The administrator wants to determine whether the error is related to a specific GPU. Which action should the administrator take?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Correlate the XID 48 timestamps in dmesg or the system journal with the GPU PCI bus IDs reported by nvidia-smi.. XID 48 is a double-bit ECC error logged by the NVIDIA driver with the PCI bus ID of the affected GPU. Correlating the error's timestamp and bus ID in dmesg or journalctl with nvidia-smi output pinpoints the specific GPU. This device-level attribution is essential before draining or replacing hardware, and it is the correct diagnostic action.

Answer

Restart the nvidia-persistenced service on all nodes to clear the error state.

Answer

Run nvidia-smi -q and check the ECC error counters and retired pages for each GPU.

Answer

Increase the GPU memory clock offset using nvidia-smi -lgc to stabilize the failing GPU.

Question 2

An administrator is using NVIDIA AI Enterprise with Run:ai for cluster management. They need to implement a policy that allows only members of the 'data-science' group to submit jobs that request more than 4 GPUs, while other users are limited to 4 GPUs. Which Run:ai feature should they use?

Accepted Answer

Run:ai quotas and priorities. Run:ai quotas and priorities allow administrators to set limits on GPU resources for different user groups. By configuring a higher quota for the 'data-science' group, they can allow requests exceeding 4 GPUs, while others are capped at 4. This enforces the policy automatically. Other features do not provide this level of group-based resource control.

Answer

Run:ai node pools

Answer

Run:ai projects

Answer

Run:ai CLI commands

Question 3

An administrator manages a Kubernetes cluster running NVIDIA GPU Operator. A data science team reports that their PyTorch training pods are stuck in Pending state with the event '0/8 nodes are available: 8 Insufficient nvidia.com/gpu.' The administrator verifies that the nodes have GPUs and that the NVIDIA device plugin pods are running. Which command should the administrator run first to diagnose why the GPU resource is not being advertised?

Accepted Answer

kubectl logs -n gpu-operator <nvidia-device-plugin-pod>. The NVIDIA device plugin exposes nvidia.com/gpu resources to the Kubernetes scheduler. When pods report insufficient GPU resources despite physical GPUs being present, the most direct diagnostic is to inspect the device plugin's logs for initialization errors, such as NVML failures or missing device files. This identifies why the resource count is zero and guides corrective action.

Answer

kubectl get pods -n gpu-operator -o wide

Answer

kubectl exec -n gpu-operator <nvidia-device-plugin-pod> -- nvidia-smi

Answer

kubectl describe node <node-name> | grep -A 10 'Allocatable'

Question 4

An administrator is responsible for a Kubernetes cluster running multiple inference workloads that share GPU nodes. The workloads belong to different teams and have varying performance requirements, but the administrator wants to prevent a single low-priority job from monopolizing a GPU and starving other pods. Which NVIDIA feature should be configured to enforce fair sharing of a single physical GPU among multiple containers?

Accepted Answer

NVIDIA GPU Operator with the device plugin and compute preemption enabled. The NVIDIA GPU Operator with the device plugin and compute preemption allows multiple containers to share a single physical GPU through time-slicing, with preemption ensuring that a low-priority job cannot starve others. This provides the dynamic, fair sharing required in a multi-tenant Kubernetes environment, unlike hard partitioning or virtualization approaches.

Answer

NVIDIA Multi-Instance GPU (MIG)

Answer

NVIDIA GPUDirect Storage

Answer

NVIDIA vGPU with time-sliced scheduling

Question 5

An administrator manages an NVIDIA AI Enterprise deployment on Kubernetes. A data science team reports that their training pods are stuck in Pending state with the event '0/8 nodes are available: 8 Insufficient nvidia.com/gpu'. The administrator confirms that GPUs are present and healthy on all nodes, and the NVIDIA GPU Operator is installed. Which action should the administrator take first to resolve the scheduling failure?

Accepted Answer

Verify that the GPU Operator's device plugin DaemonSet is running and that node labels include nvidia.com/gpu.present=true.. The NVIDIA GPU Operator deploys a device plugin that registers nvidia.com/gpu as an extended resource on each node. When that DaemonSet is unhealthy or nodes are not labeled, the scheduler sees zero GPU capacity and emits Insufficient nvidia.com/gpu. Confirming the plugin's health and node labeling is the correct first step before investigating deeper driver or container runtime issues.

Answer

Add a nodeSelector to the training pods that matches the GPU model label, such as nvidia.com/gpu.product=NVIDIA-H100-80GB-HBM3.

Answer

Increase the cluster's GPU resource quota by editing the nvidia.com/gpu ResourceQuota object in the kube-system namespace.

Answer

Restart the kube-scheduler pod to force it to re-read node capacity and update its cache.

Question 6

An administrator manages a Kubernetes cluster running AI workloads with NVIDIA GPUs. They need to enforce that only pods with a specific label can access GPU resources, and that GPU usage is tracked per namespace. Which combination of Kubernetes and NVIDIA components should they use?

Accepted Answer

NVIDIA GPU Operator with the device plugin, plus Kubernetes admission controllers and custom metrics for GPU usage.. The NVIDIA GPU Operator deploys the device plugin that advertises GPUs to Kubernetes. To enforce label-based GPU access, an admission controller can intercept pod creation and validate labels before allowing GPU resource requests. For per-namespace GPU usage tracking, custom metrics from DCGM or the device plugin can be exposed and aggregated. This combination provides both access control and monitoring.

Answer

NVIDIA Base Command Manager with Kubernetes integration, plus NVIDIA GPU Operator and Prometheus for monitoring.

Answer

NVIDIA GPU Operator with the device plugin and MIG manager, plus Kubernetes ResourceQuota and NetworkPolicy.

Answer

NVIDIA vGPU manager with vGPU profiles, plus Kubernetes PodSecurityPolicy and ResourceQuota.

Question 7

An AI operations engineer is preparing a bare-metal DGX H100 node for a new multi-node training cluster. The node must be consistently provisioned with the same NVIDIA driver, CUDA, and container runtime stack as the rest of the fleet, and the team wants to avoid manual per-node drift. Which NVIDIA-provided approach should the engineer use to install and manage the full software stack on this node?

Accepted Answer

Use the NVIDIA GPU Operator with a Kubernetes cluster and node feature discovery to manage driver, container toolkit, and device plugin lifecycle.. The GPU Operator is the NVIDIA-recommended way to install and manage the GPU software stack on Kubernetes nodes. It reconciles the driver, container toolkit, device plugin, and monitoring components so that every node converges to the same declared state. This eliminates manual per-node installation and drift, which is precisely what the engineer needs for a consistent multi-node AI training cluster.

Answer

Deploy the NVIDIA Base Command Manager to the node and use its built-in image management to push the driver and CUDA stack.

Answer

Use the NVIDIA DGX OS image and the NVIDIA DGX Software Stack (DGX OS) installation media, then validate with `dcgm` and `nvidia-smi`.

Answer

Run the NVIDIA driver .run installer on each node and rely on `nvidia-smi` to confirm the version matches across the fleet.

Question 8

An administrator is configuring a Kubernetes cluster with NVIDIA GPU Operator to support multi-instance GPU (MIG) on A100 GPUs. They need to ensure that MIG instances are properly advertised and can be requested by pods. Which two configurations are required to enable MIG support in this environment? (Choose two.)

Accepted Answer

Enable MIG mode on the GPUs using nvidia-smi mig -cgi or through the GPU Operator's MIG manager.. To enable MIG support in a Kubernetes cluster with NVIDIA GPU Operator, two key steps are required: enabling MIG mode on the GPUs and configuring the NVIDIA Device Plugin with the correct MIG strategy. Enabling MIG mode can be done manually or via the GPU Operator's MIG manager. The device plugin then advertises MIG instances as schedulable resources when the MIG strategy is set appropriately. These two actions ensure that MIG instances are available for pods to request.

Answer

Create a custom resource definition (CRD) for MIG instances and deploy a controller to manage them.

Answer

Set the 'nvidia.com/gpu' resource limit to the number of MIG instances in the pod specification.

Answer

Install the NVIDIA Container Toolkit on all nodes and configure the Docker runtime to use the NVIDIA runtime.

Question 9

An administrator manages a Kubernetes cluster running inference workloads on NVIDIA GPUs. The cluster uses NVIDIA GPU Operator with MIG-enabled A100 GPUs. A data science team reports that their pod is stuck in Pending state with the event '0/8 nodes are available: 8 Insufficient nvidia.com/mig-1g.5gb.' The administrator verifies that each node has seven MIG instances of 1g.5gb available and no other pods are using them. Which action should the administrator take to resolve the pending pod?

Accepted Answer

Verify that the pod's resource request includes the correct MIG profile and that the node's MIG configuration matches the profile name exactly.. The scheduler error indicates that the pod requests a MIG resource that the nodes do not advertise. The most likely cause is a mismatch between the MIG profile name in the pod specification and the profile advertised by the nodes. The administrator should verify the exact resource name (e.g., nvidia.com/mig-1g.5gb) on the nodes and ensure the pod requests that same name. Once aligned, the pod can be scheduled.

Answer

Restart the NVIDIA GPU Operator daemonset to reinitialize MIG configuration on all nodes.

Answer

Patch the pod specification to request nvidia.com/mig-1g.5gb with a quantity that matches an available MIG instance.

Answer

Disable MIG mode on all nodes and let the pod use a full GPU instead.

Question 10

An administrator is deploying NVIDIA AI Enterprise on a vSphere cluster with multiple ESXi hosts, each containing several A100 GPUs. The administrator needs to enable GPU sharing among multiple virtual machines (VMs) on the same host to maximize utilization for inference workloads. Which NVIDIA vGPU mode should be configured?

Accepted Answer

NVIDIA vGPU with time-sliced scheduling. Time-sliced vGPU is designed to share a physical GPU among multiple VMs by scheduling GPU time slices, which maximizes utilization for inference workloads that do not need full GPU resources. MIG would isolate resources, and passthrough would dedicate the GPU to one VM, neither of which achieves the desired sharing.

Answer

NVIDIA Multi-Instance GPU (MIG) mode

Answer

NVIDIA GPUDirect RDMA

Answer

NVIDIA vComputeServer with bare-metal passthrough

Question 11

An administrator is troubleshooting a multi-node NVIDIA GPU cluster where NCCL-based training jobs intermittently hang. They suspect network issues. Which diagnostic step should they perform first to isolate the problem?

Accepted Answer

Check the NCCL debug logs by setting NCCL_DEBUG=INFO and NCCL_DEBUG_SUBSYS=INIT,COLL to identify the failing collective operation.. Intermittent hangs in multi-node NCCL jobs often stem from network or collective communication issues. Enabling NCCL debug logging with NCCL_DEBUG=INFO and NCCL_DEBUG_SUBSYS=INIT,COLL provides granular details about initialization and collective operations, helping identify timeouts or misconfigurations. This is the most efficient first step to isolate the problem before diving into lower-level network diagnostics.

Answer

Use ibstat and ibping to verify InfiniBand link status and connectivity between nodes.

Answer

Run a bandwidth test with nvidia-smi nvlink -s to check NVLink throughput between GPUs.

Answer

Run nvidia-smi topo -m on each node to verify GPU topology and NVLink connections.

Question 12

An administrator is setting up a new NVIDIA AI Enterprise environment and needs to ensure that the NVIDIA GPU Operator is installed and functioning correctly on a Kubernetes cluster. After installation, which command should the administrator use to verify that all GPU Operator components are healthy and that the GPU is available as a schedulable resource?

Accepted Answer

kubectl describe node <node-name> | grep nvidia.com/gpu. To confirm that the GPU Operator is functioning and the GPU is schedulable, the administrator should check that the node advertises the nvidia.com/gpu resource. This indicates the device plugin has registered the GPU. While checking operator pods is also important, the presence of the extended resource on the node is the definitive sign of schedulability.

Answer

kubectl exec -n gpu-operator <nvidia-driver-daemonset-pod> -- nvidia-smi

Answer

kubectl get pods -n gpu-operator

Answer

kubectl get clusterpolicy -n gpu-operator

Question 13

A platform administrator is configuring NVIDIA MIG (Multi-Instance GPU) on an A100 node that will host multiple small inference tenants. The administrator wants to ensure that each tenant gets an isolated GPU instance with dedicated memory and compute slices, and that the configuration survives a reboot. Which two actions are required? (Choose two.)

Accepted Answer

Configure the NVIDIA Container Toolkit to pass the MIG device UUIDs to containers instead of the full GPU device.. MIG requires enabling MIG mode on the physical GPU and creating GPU and compute instances with `nvidia-smi mig`. For containerized tenants, the NVIDIA Container Toolkit must then be pointed at the specific MIG device UUID so each tenant sees only its isolated slice. Together these actions provide dedicated memory and compute isolation and make the configuration usable by the intended workloads.

Answer

Disable ECC memory on the GPU so that MIG instances can be created with the desired memory sizes.

Answer

Install the NVIDIA vGPU software and create a vGPU profile for each tenant on the A100 node.

Answer

Set the GPU persistence mode to disabled so that MIG configuration is retained across reboots.

Question 14

An administrator is configuring NVIDIA MIG (Multi-Instance GPU) on an A100 GPU to support multiple isolated tenants on a single physical GPU. The administrator wants to ensure that each tenant receives a dedicated slice of memory and compute resources with strict isolation. Which two actions are required to correctly set up MIG for this purpose? (Choose two.)

Accepted Answer

Enable MIG mode on the GPU using nvidia-smi mig -cgi or the equivalent configuration command.. To set up MIG for isolated tenants, the administrator must first enable MIG mode on the physical GPU, then create GPU instances and compute instances with appropriate profiles. These two actions establish the partitioned resources that provide dedicated memory and compute isolation for each tenant. Other options involve unrelated technologies.

Answer

Install the NVIDIA vGPU software to manage MIG instances as virtual GPUs.

Answer

Configure SR-IOV virtual functions on the GPU to expose each MIG instance as a separate PCIe device.

Answer

Disable ECC memory on the GPU to allow MIG instances to share memory more efficiently.

Question 15

An administrator is responsible for managing an NVIDIA DGX SuperPOD used for large-scale AI training. The administrator needs to perform routine health checks and gather telemetry from all DGX nodes, including GPU temperature, power usage, and ECC error counts. Which NVIDIA tool provides a centralized, cluster-wide view of this telemetry and health status?

Accepted Answer

NVIDIA Base Command Manager. Base Command Manager is designed for centralized management and monitoring of NVIDIA DGX SuperPOD clusters. It aggregates telemetry from all nodes, including GPU temperature, power, and ECC errors, and provides health checks and alerts. While DCGM and nvidia-smi are useful for per-node data, they lack the cluster-wide aggregation and dashboard capabilities that Base Command Manager offers.

Answer

NVIDIA Container Toolkit

Answer

NVIDIA System Management Interface (nvidia-smi)

Answer

NVIDIA Data Center GPU Manager (DCGM)