NCP-AIO Practice Test 10 — 10 Questions

Question 1

When installing the NVIDIA GPU Operator, which namespace is typically used to ensure proper isolation and role-based access control?

Accepted Answer

gpu-operator. Using a dedicated namespace like 'gpu-operator' is a best practice in Kubernetes. It isolates the operator's resources, permissions, and lifecycle from other cluster services. This separation allows for granular security policies, making it easier to manage access and ensuring that only authorized personnel can modify the operator's configuration, which is vital for maintaining the security and integrity of the GPU-enabled infrastructure in a production environment.

Answer

default

Answer

kube-system

Answer

public-apps

Question 2

When deploying large-scale distributed training jobs, why is it recommended to use the NVIDIA Network Operator in conjunction with the GPU Operator?

Accepted Answer

To optimize RDMA throughput for distributed training.. The Network Operator automates the configuration of high-speed interconnects like InfiniBand or RoCE, which are crucial for distributed training across multiple nodes. By aligning network resources with GPU placement, the operator minimizes latency and maximizes throughput. This ensures that the GPU compute capacity is not bottlenecked by slow network communication, which is essential for scaling complex AI model training across large clusters effectively.

Answer

To increase the number of supported GPU containers.

Answer

To provide automatic GPU driver updates.

Answer

To enable multi-tenancy on GPU nodes.

Question 3

An administrator manages an NVIDIA AI Enterprise cluster running multiple Kubernetes nodes, each with several A100 GPUs. After upgrading the NVIDIA GPU Operator to a newer version, the administrator notices that pods requesting GPUs remain in a Pending state, and the node's allocatable GPU count is reported as zero. Which command should the administrator run first to diagnose the issue?

Accepted Answer

kubectl describe node <node-name>. The node's allocatable GPU count is reported as zero, indicating that the kubelet is not receiving GPU resource advertisements from the NVIDIA device plugin. Describing the node reveals capacity, allocatable resources, and relevant events, such as device plugin registration failures. This directly identifies whether the GPU Operator's device plugin is functioning, making it the correct first step.

Answer

kubectl get pods --all-namespaces -o wide

Answer

nvidia-smi -q

Answer

kubectl logs -n gpu-operator <gpu-operator-pod>

Question 4

An AI platform engineer is preparing a fleet of NVIDIA DGX H100 systems for production workloads using the NVIDIA Base Command Manager (BCM). During initial bare-metal provisioning via PXE boot, the provisioning server successfully hands out IP addresses, but nodes consistently fail during the OS image deployment phase, throwing a kernel panic related to missing storage drivers. Which deployment step must be verified or corrected to ensure successful hardware-specific image deployment?

Accepted Answer

Verify that the target operating system image profile assigned in Base Command Manager includes the necessary storage controller modules and hardware support packages for the specific server model.. NVIDIA Base Command Manager relies heavily on matching the hardware profile of complex servers like DGX H100 with the correct custom software image and specialized storage drivers. If the deployment image lacks the required kernel modules for modern high-performance NVMe controllers, PXE booting fails at the kernel load stage. Ensuring the software image profile incorporates the exact hardware-specific drivers resolves this mismatch, allowing enterprise-grade bare-metal automation to complete seamlessly across the GPU cluster.

Answer

Reconfigure the DHCP server scope options to extend lease times and include custom vendor-class identifiers for the provisioning daemon.

Answer

Update the system BIOS Unified Extensible Firmware Interface boot order to prioritize internal redundant array of independent disks storage over network interfaces.

Answer

Modify the dynamic port forwarding settings on the top-of-rack management switches to allow uninterrupted trivial file transfer protocol block size extensions.

Question 5

An AI Operations engineer is managing a multi-node training job using NVIDIA NCCL. The logs indicate frequent 'NCCL WARN' messages related to 'net_ib_init' failures. What is the most likely cause of this issue?

Accepted Answer

Misconfigured InfiniBand subnet manager or mismatched firmware.. NCCL relies on InfiniBand (IB) for high-speed inter-GPU communication in multi-node setups. Failures in 'net_ib_init' suggest that the network configuration or driver stack is misaligned between nodes. Identifying and resolving these connectivity issues is critical because NCCL is the backbone of distributed training; any failure here leads to synchronized hangs, data corruption, or severe training slowdowns, undermining the benefits of cluster-level resource parallelism.

Answer

Incompatible NCCL versions across the cluster nodes.

Answer

Insufficient system RAM on the worker nodes.

Answer

Incorrect GPU driver version installed on the login node.

Question 6

An administrator is troubleshooting a performance degradation in a multi-node NVIDIA NCCL-based training job. The job spans four DGX nodes connected via InfiniBand. The administrator suspects that NCCL is not using the optimal network path. Which action should the administrator take to verify and enforce the use of GPUDirect RDMA for inter-node communication?

Accepted Answer

Set the environment variable NCCL_DEBUG=INFO and inspect the logs for messages indicating that GPUDirect RDMA is enabled.. To verify and enforce GPUDirect RDMA usage, the administrator should enable NCCL debug logging with NCCL_DEBUG=INFO and examine the logs for transport information. This will show if GDR is active and if any fallbacks occur. Checking topology with nvidia-smi topo -m is useful but does not confirm active GDR. Setting NCCL_NET_GDR_LEVEL=0 would disable GDR, and switching to RoCEv2 is irrelevant.

Answer

Configure the InfiniBand fabric to use RoCEv2 instead of InfiniBand native protocol to improve compatibility with GPUDirect RDMA.

Answer

Set NCCL_NET_GDR_LEVEL=0 to force NCCL to use GPUDirect RDMA for all inter-node communication.

Answer

Run nvidia-smi topo -m on each node to check the GPU-to-NIC affinity and ensure that GPUs and InfiniBand adapters are connected via PCIe switches.

Question 7

An AI operations team is validating a new Kubernetes cluster before installing the NVIDIA GPU Operator with the driver managed by the Operator itself. The nodes run a supported Linux distribution with GPUs physically installed. Which two conditions must be satisfied for the Operator's driver container to build and load the kernel module successfully? (Choose two.)

Accepted Answer

Secure Boot must be disabled or the module must be signed with an enrolled key.. For the Operator to manage drivers, the driver container must compile the NVIDIA kernel module against the exact running kernel, which requires matching kernel headers and build dependencies on the node. In addition, UEFI Secure Boot blocks unsigned modules, so it must be disabled or the module signed with an enrolled key. These two conditions gate whether the module can be built and inserted; without them the node cannot advertise GPU capacity to the scheduler.

Answer

Each node must have at least one Multi-Instance GPU profile pre-created.

Answer

The kubelet must be configured with the NVIDIA device plugin endpoint flag.

Answer

The cluster must have the NVIDIA Network Operator installed first.

Question 8

A cloud operations engineer is deploying the NVIDIA GPU Operator on a managed Kubernetes service where the worker nodes already have the NVIDIA data center driver installed by the cloud provider. The team wants the Operator to manage only the device plugin, container toolkit, and monitoring components. Which Helm value should the engineer set during installation?

Accepted Answer

--set driver.enabled=false. The GPU Operator chart exposes driver.enabled to control whether the driver container is deployed. On managed Kubernetes services the node image already contains a validated NVIDIA driver, so disabling the Operator's driver avoids conflicts and duplicate work while still letting the Operator manage the device plugin, container toolkit, DCGM exporter, and related components. The other values either do not exist or disable components the team needs.

Answer

--set operator.driver.install=false

Answer

--set mig.strategy=none

Answer

--set toolkit.enabled=false

Question 9

A financial services company is deploying NVIDIA AI Enterprise in an air-gapped data center. They need to install the NVIDIA GPU Operator on their Kubernetes cluster without internet access. Which additional step must they take to ensure a successful installation?

Accepted Answer

Configure the GPU Operator to use a private container registry with mirrored images.. For an air-gapped installation, the critical step is to make all required container images available within the isolated network. This is done by mirroring the images to a private registry and configuring the GPU Operator to use that registry. The operator's Helm chart provides values to specify the registry and image repository. Without this, the operator cannot pull the necessary components. Other options either do not address the air-gap limitation or propose unnecessary manual steps.

Answer

Disable the GPU Operator's driver container and manually install the driver on each node.

Answer

Use a Kubernetes cluster that supports dynamic volume provisioning for GPU drivers.

Answer

Enable the GPU Operator's built-in proxy to fetch images from the internet.

Question 10

An administrator is preparing a Kubernetes cluster for AI workloads and needs to ensure that the NVIDIA GPU Operator can be installed. The cluster nodes have NVIDIA GPUs, and the administrator wants to verify that the nodes are ready. Which command should the administrator run to check if the NVIDIA driver is already loaded on a node?

Accepted Answer

nvidia-smi. To verify that the NVIDIA driver is loaded on a node, the administrator should run nvidia-smi. This command queries the driver and displays GPU details such as driver version, GPU utilization, and memory usage. It is the definitive check for driver functionality. Other commands may show GPU hardware or Kubernetes resources but do not confirm driver operation.

Answer

lspci | grep -i nvidia

Answer

kubectl get nodes -o wide

Answer

kubectl describe node <node-name>