NCP-AIO Practice Test 2 — 10 Questions

Question 1

An AI researcher is debugging a multi-node training job using NCCL. Which TWO actions should they take to diagnose potential network-related performance degradation?

Accepted Answer

Set NCCL_DEBUG=INFO to monitor communication patterns and connection issues.. Debugging multi-node communication requires verifying both software configurations and physical network health. NCCL_DEBUG settings provide granular insight into connection establishment and topology detection, while NCCL tests provide baseline performance metrics. Identifying these issues early is essential for scaling models across large clusters, as network overhead can quickly become the dominant factor in training time during distributed synchronized gradient descent.

Answer

Increase the number of threads in the PyTorch DataLoader.

Answer

Switch from NCCL to MPI for all communication operations.

Answer

Lower the precision of the model to float16.

Question 2

Which utility is primarily used to monitor and manage NVIDIA GPU power, temperature, and usage statistics in real-time on a Linux-based deployment?

Accepted Answer

nvidia-smi. nvidia-smi (NVIDIA System Management Interface) is the command-line utility used to interact with the NVIDIA driver for monitoring and managing GPU devices. It is the fundamental tool for AI ops teams to verify that GPUs are properly detected and performing within expected thermal and power envelopes during model training or inference tasks on DGX or workstation systems.

Answer

top

Answer

htop

Answer

dmesg

Question 3

Refer to the exhibit. An administrator attempts to deploy a GPU-based pod, but it remains in the 'Pending' state. What is the most likely cause based on the error log?

Accepted Answer

The NVIDIA device plugin is not correctly advertising GPU resources.. The error 'Insufficient nvidia.com/gpu' indicates that the Kubernetes scheduler cannot find a node with available GPU resources that match the pod's request. This typically happens when the device plugin is not correctly reporting resource availability to the scheduler, or the cluster is over-provisioned. Resolving this requires ensuring the GPU Operator is running and that the device plugin has successfully registered the GPUs with the Kubelet on the worker nodes.

Answer

The pod is requesting more CPU cores than the node can provide.

Answer

The node is in a 'NotReady' state due to high disk latency.

Answer

The pod security policy prohibits the use of GPU-accelerated containers.

Question 4

An administrator wants to prevent unauthorized users from accessing sensitive model weights stored in GPU memory. Which security feature should be implemented to ensure hardware-level isolation of the memory space?

Accepted Answer

Implement NVIDIA Confidential Computing.. Hardware-level memory isolation via Confidential Computing technologies, such as NVIDIA Confidential Computing (CC), protects data in use by encrypting memory contents. For administrators handling sensitive IP, this provides a root-of-trust that persists even if the OS or hypervisor is compromised. This is a critical administrative control for ensuring compliance and data sovereignty in multi-tenant cloud environments where infrastructure is shared among different departments or organizations.

Answer

Enable standard Linux filesystem permissions.

Answer

Use a simple SSH firewall rule.

Answer

Update the NVIDIA CUDA shared library path.

Question 5

When debugging a workload that consistently crashes with 'Out of Memory' (OOM) errors despite sufficient GPU VRAM, what is the most likely cause related to workload management?

Accepted Answer

The container memory limit is lower than the data processing requirements.. OOM errors can occur due to host-side system memory exhaustion if the workload manages large datasets in system RAM before loading them into the GPU. If the container memory limit is set too low for the data processing pipeline, the entire container will be terminated. This highlights the need to correctly balance both GPU VRAM and system memory limits in the container specification.

Answer

The CUDA driver version is incompatible with the installed GPU.

Answer

The GPU device plugin is not correctly reporting free memory.

Answer

The training job is using mixed-precision training (FP16).

Question 6

Refer to the exhibit. A cluster administrator notices that GPU jobs with this PriorityClass are failing to start even when empty GPUs are available. What is the most likely cause?

Accepted Answer

Insufficient nodes satisfy the 'minAvailable' requirement for the pod group.. When using the Volcano scheduler with NVIDIA GPUs, priority classes alone are insufficient if the Gang Scheduling policy is not met. If the job requests multiple pods that cannot be satisfied simultaneously, Volcano will leave the pods in a pending state to avoid resource fragmentation. This ensures that massive training jobs do not consume isolated resources that would otherwise result in deadlocks during collective communication synchronization.

Answer

The PriorityClass value is too low to trigger preemption.

Answer

The NVIDIA Device Plugin requires a restart to recognize the PriorityClass.

Answer

The PriorityClass name violates Kubernetes naming conventions.

Question 7

Which TWO of the following steps are essential when deploying the NVIDIA GPU Operator on a Kubernetes cluster to ensure that GPU resources are discoverable by the scheduler?

Accepted Answer

Enable the node feature discovery (NFD) service in the operator configuration.. The NVIDIA GPU Operator automates the lifecycle of NVIDIA software components, including the driver, toolkit, and device plugin. By configuring the operator to deploy the GPU device plugin and the Node Feature Discovery (NFD) service, the cluster gains the ability to identify GPU hardware and advertise it as an allocatable resource. Without these, the Kubernetes scheduler cannot place pods requiring GPU resources, leading to 'Pending' status for those workloads.

Answer

Manually install the NVIDIA Container Toolkit on every worker node prior to operator deployment.

Answer

Modify the Kubernetes API server manifest to include the NVIDIA-specific admission controller.

Answer

Disable the default Kubernetes scheduler to allow the NVIDIA scheduler plugin to take over.

Question 8

Which administrative practice ensures that a cluster is prepared for the arrival of new NVIDIA GPU hardware with minimal downtime?

Accepted Answer

Update the OS and NVIDIA drivers across the cluster.. Maintaining an up-to-date driver and software stack via a robust package management system is the key to minimizing downtime during hardware upgrades. Administrators must ensure that the kernel and driver versions are compatible with the new hardware before it arrives. This proactive preparation is vital for maximizing cluster uptime and ensuring that researchers can immediately begin using the new hardware for their AI experiments without configuration delays.

Answer

Wait for the hardware to arrive before checking driver support.

Answer

Manually install every CUDA library on all nodes.

Answer

Modify the BIOS settings to ignore PCIe errors.

Question 9

Which THREE factors should be considered when estimating GPU memory requirements for a Large Language Model (LLM) fine-tuning job?

Accepted Answer

The memory size of the model parameters.. When sizing LLM workloads, you must account for the model weights, optimizer states, and gradient buffers. Additionally, activations consume significant memory during the forward and backward passes. Understanding these components is essential for AI Ops, as incorrect sizing leads to OOM crashes early in the training process, wasting significant compute time and delaying model delivery in production environments.

Answer

The total number of CPU threads used.

Answer

The network latency between nodes.

Question 10

An administrator is planning to monitor GPU utilization across a large cluster. Which component should be deployed to collect metrics that are compatible with Prometheus?

Accepted Answer

NVIDIA DCGM Exporter. NVIDIA DCGM (Data Center GPU Manager) is the industry-standard tool for collecting GPU health and telemetry data. By deploying the DCGM Exporter, the administrator enables the conversion of hardware telemetry into a Prometheus-friendly format. This integration is vital for observability, allowing teams to set up alerts and dashboards to track GPU usage, power consumption, and memory allocation across the entire fleet of accelerated computing nodes.

Answer

NVIDIA GPU Operator

Answer

NVIDIA Triton Inference Server

Answer

NVIDIA CUDA Toolkit