NCP-AIO Practice Test 20 — 15 Questions

Question 1

An administrator is deploying NVIDIA AI Enterprise on a bare-metal cluster. Which component is mandatory to provide the necessary abstraction layer for containerized GPU resources?

Accepted Answer

NVIDIA GPU Operator. The NVIDIA GPU Operator is essential for automating the management of all NVIDIA software components needed to provision Kubernetes with GPU support. It manages the lifecycle of drivers, container runtimes, and monitoring tools. By leveraging the Operator, administrators ensure consistent configuration across the cluster, preventing drift and ensuring that CUDA workloads have the correct dependencies to execute reliably on bare-metal infrastructure.

Answer

NVIDIA Triton Inference Server

Answer

NVIDIA Base Command

Answer

NVIDIA CUDA Toolkit

Question 2

Which mechanism does the NVIDIA GPU Operator use to ensure that the driver installed on a worker node matches the specific architecture of the installed GPU hardware?

Accepted Answer

Node Feature Discovery (NFD).. The GPU Operator uses Node Feature Discovery (NFD) to identify the specific GPU hardware present on each node. By labeling nodes with these hardware characteristics, the operator can ensure that the correct driver and kernel modules are selected and deployed. This mapping is vital in heterogeneous clusters, where different nodes might require different driver builds, preventing installation errors and ensuring optimal performance across the entire fleet.

Answer

NVIDIA System Management Interface (nvidia-smi).

Answer

The container runtime's default configuration.

Answer

Hardcoding the driver version in the deployment manifest.

Question 3

An administrator is configuring a new cluster and wants to ensure that telemetry data from GPUs is collected in a centralized manner. Which tool is best suited for this requirement?

Accepted Answer

NVIDIA DCGM Exporter. NVIDIA DCGM Exporter is the industry-standard tool for collecting GPU telemetry data in Kubernetes environments. It aggregates metrics from the Data Center GPU Manager and exposes them in a format compatible with Prometheus, allowing for centralized monitoring, visualization, and alerting. This visibility is essential for understanding cluster performance, identifying anomalies, and optimizing resource utilization in large-scale AI infrastructure deployments, making it the preferred choice for enterprise monitoring solutions.

Answer

nvidia-smi log file rotation

Answer

Manual polling of /proc/driver/nvidia

Answer

NVIDIA Nsight Systems

Question 4

Which TWO methods are effective for enforcing GPU resource isolation in a multi-tenant NVIDIA Kubernetes environment?

Accepted Answer

Enabling NVIDIA Multi-Instance GPU (MIG) for hardware partitioning.. Resource isolation is paramount in multi-tenant environments to prevent noisy neighbor effects where one workload consumes disproportionate GPU cycles. NVIDIA Multi-Instance GPU (MIG) provides hardware-level isolation for partitioning, while Kubernetes-native device plugins with affinity and tolerations provide software-level scheduling control. These mechanisms combined ensure predictable performance and security, preventing cross-tenant interference during intensive training or inference cycles in shared GPU infrastructure clusters.

Answer

Configuring standard Docker cgroups for GPU memory limits.

Answer

Implementing standard OS-level priority queuing via 'nice'.

Answer

Setting a global environment variable for GPU frequency scaling.

Question 5

An AI engineer observes that a training job on an NVIDIA DGX H100 system is experiencing significant performance degradation. The GPU utilization is high, but the throughput remains low. Which tool should be used first to identify if the bottleneck is related to data loading or PCIe bandwidth saturation?

Accepted Answer

NVIDIA Nsight Systems. NVIDIA Nsight Systems is the primary tool for analyzing system-wide performance, including CPU-GPU interactions and data transfer bottlenecks. Identifying whether a bottleneck occurs in the data pipeline or within the hardware interconnects is critical for optimizing training speed. By visualizing timelines, engineers can pinpoint if the GPU is starving for data or if the PCIe bus is congested, enabling targeted remediation for large-scale distributed training clusters.

Answer

NVIDIA-SMI

Answer

NVIDIA DCGM

Answer

NVIDIA Nsight Compute

Question 6

Which mechanism does the NVIDIA Device Plugin use to communicate GPU availability to the Kubernetes Kubelet?

Accepted Answer

It uses a gRPC-based device plugin API.. The NVIDIA Device Plugin functions as a gRPC service that registers itself with the Kubelet upon startup. It performs active monitoring of the GPU nodes and reports discovered devices, including their count and health, to the Kubelet. This information is subsequently relayed to the Kubernetes API server, allowing the scheduler to make placement decisions based on real-time GPU availability and ensuring efficient workload distribution in AI clusters.

Answer

It writes to a configuration file that the Kubelet watches.

Answer

It queries the API server directly via REST calls.

Answer

It relies on the container runtime to inspect hardware.

Question 7

An operations engineer is investigating a sudden drop in throughput for a multi-GPU training job on an NVIDIA DGX A100. The job uses PyTorch with DDP. Logs show that one GPU is consistently at 100% utilization while others are below 50%. Which tool and approach should be used to identify the bottleneck?

Accepted Answer

Use NVIDIA Nsight Systems to profile the job and examine the CUDA API and kernel timeline to identify serialization or synchronization issues.. Nsight Systems is designed for system-level profiling, capturing CPU and GPU activities across multiple devices. It can show if one GPU is performing more work or if others are idle waiting for synchronization. This makes it the right tool to identify the bottleneck in a DDP job where one GPU is overutilized. Profiling before making changes is essential.

Answer

Use nvidia-smi to monitor GPU utilization and memory, then rebalance the workload by reducing the batch size on the overutilized GPU.

Answer

Use NVIDIA Nsight Compute to profile each GPU's kernels and compare execution times to find the slowest kernel.

Answer

Use nvidia-smi topo -m to check the GPU topology and then adjust the NCCL communication settings to use a different ring order.

Question 8

When designing a workload management strategy for multi-tenant AI training, what is the most effective way to ensure isolation between different tenants using the same physical GPU nodes?

Accepted Answer

Implement NVIDIA MIG to partition hardware at the compute and memory level.. Using NVIDIA Multi-Instance GPU (MIG) is the most robust method for hardware-level isolation. MIG partitions a single physical GPU into multiple independent instances, each with its own memory and compute resources. This allows multiple tenants to run workloads simultaneously on the same hardware without interfering with each other's performance, ensuring predictable and secure resource allocation for multi-tenant AI environments.

Answer

Set strict Kubernetes memory limits on every pod to prevent memory leakage.

Answer

Use Kubernetes namespaces to logically separate the tenants.

Answer

Enable GPU sharing through the NVIDIA Container Toolkit's time-slicing configuration.

Question 9

Which THREE factors must be considered when sizing a persistent storage solution for multi-node distributed training checkpoints?

Accepted Answer

Aggregate write bandwidth to avoid stalling the training loop.. Checkpointing large models requires significant I/O throughput to avoid blocking the training loop, sufficient capacity for multiple historical versions, and high availability to ensure data integrity. By addressing these three factors—throughput, capacity, and reliability—organizations can ensure that the training process remains robust against hardware failures without incurring unnecessary performance penalties during the frequent write cycles required for large-scale model training.

Answer

The number of CPU cores on the storage controller.

Answer

The number of users accessing the storage simultaneously.

Question 10

Which component in the NVIDIA AI ecosystem is responsible for monitoring and reporting GPU telemetry data, such as power usage, temperature, and utilization, to Prometheus?

Accepted Answer

The NVIDIA DCGM Exporter.. The NVIDIA DCGM Exporter is the standard component designed to collect GPU metrics via the Data Center GPU Manager (DCGM) and export them into a format that Prometheus can scrape. This is vital for workload management because it provides the real-time observability required to trigger auto-scaling or identify jobs that are failing to utilize allocated GPU resources effectively across the cluster.

Answer

The NVIDIA Device Plugin.

Answer

The Kubernetes Kubelet.

Answer

The NVIDIA Container Runtime.

Question 11

A team is profiling a distributed training job using NVIDIA NCCL for inter-GPU communication on a DGX A100 system. They observe that all-reduce operations are taking longer than expected, and the NCCL debug logs show frequent 'NVLS' (NVLink SHARP) errors. Which action should be taken to resolve the issue?

Accepted Answer

Disable NVLink SHARP by setting NCCL_NVLS_ENABLE=0 in the environment and restart the job.. NVLS errors indicate that the NVLink SHARP acceleration is failing, likely due to hardware or software incompatibility. Disabling NVLS via the NCCL_NVLS_ENABLE environment variable forces NCCL to use standard algorithms, which are robust and should eliminate the errors. Other options either do not target NVLS or are less direct and potentially disruptive.

Answer

Increase the NCCL buffer size by setting NCCL_BUFFSIZE to a larger value to reduce the number of messages.

Answer

Update the GPU driver and CUDA toolkit to the latest versions to fix NVLS bugs.

Answer

Switch the NCCL algorithm to 'Tree' by setting NCCL_ALGO=Tree to avoid NVLS usage.

Question 12

An AI ops engineer notices that a specific training workload is experiencing high 'wait' times for GPU resources despite the cluster having available idle GPUs. What is the most likely cause?

Accepted Answer

The pod has unmet affinity or toleration requirements.. High wait times despite idle resources often point to scheduling constraints, such as mismatched node affinity, taints, or tolerations. The scheduler may be unable to place the pod because the available GPUs do not meet specific requirements (e.g., specific memory requirements, architecture, or interconnect features). Troubleshooting requires examining the Kubernetes scheduler logs or pod event descriptions to identify why the pending pod cannot be bound to the available nodes.

Answer

The GPU driver is too new for the current Kubernetes version.

Answer

The system clock is unsynchronized between nodes.

Answer

The container image is too large to pull efficiently.

Question 13

An administrator observes that despite the GPU Operator being installed, the pods cannot access the GPU. What is the most likely cause if the NVIDIA container runtime is properly configured?

Accepted Answer

The host NVIDIA driver is not loaded.. If the runtime is configured, the most common remaining failure is the lack of the correct NVIDIA drivers on the host node. If the kernel modules are not loaded or the driver version is incompatible with the installed GPU, the runtime will be unable to successfully inject the device nodes. This is a common installation oversight where the operator might be deployed but the driver installation task failed or was skipped.

Answer

The Kubernetes API server is down.

Answer

The pod has too little memory requested.

Answer

The container image is missing the CUDA library.

Question 14

An AI researcher is running a training job using mixed precision (FP16/BF16). The loss function is diverging unexpectedly. What is the most likely culprit?

Accepted Answer

The lack of loss scaling for FP16.. Mixed precision training uses lower precision formats to speed up math and reduce memory usage, but this can lead to numerical instability. Divergence in loss is a classic symptom of 'underflow' or 'overflow' issues occurring in FP16, where small gradients or large weight updates exceed the representable range. Implementing loss scaling is the standard industry technique to preserve the precision of gradients and ensure the training process remains stable.

Answer

The GPU driver is too old for FP16.

Answer

The learning rate is too high for mixed precision.

Answer

The batch size is too large.

Question 15

An administrator wants to ensure that a training process is limited to a single GPU on a multi-GPU node. Which environment variable should be set?

Accepted Answer

CUDA_VISIBLE_DEVICES=0. Controlling GPU visibility is a fundamental skill for resource management in multi-tenant environments. By using CUDA_VISIBLE_DEVICES, an administrator can restrict a process to a specific device, preventing multiple jobs from competing for the same GPU. This isolation is crucial for maintaining performance stability and ensuring that individual jobs receive consistent, predictable access to compute resources without interference from other concurrent tasks.

Answer

NCCL_DEBUG=INFO

Answer

NVIDIA_DRIVER_CAPABILITIES=compute

Answer

OMP_NUM_THREADS=1