NCP-AIO Practice Questions 1 — 50 Questions

Question 1

Refer to the exhibit. An administrator notices that 'user_a' is consistently hitting resource limits despite having sufficient total system GPU memory. Based on the policy JSON, what is the cause?

Accepted Answer

The user is exceeding the 16GB VRAM limit.. The policy specifies a hard resource quota of '16GB' for the user. Even if the total system memory is higher, the scheduler enforces this limit per user. When the user's workload attempts to allocate more than 16GB of VRAM, the job will fail or throttle. This is a common method for preventing single users from starving the entire cluster of shared resources.

Answer

The system has reached the max_concurrent_jobs limit.

Answer

The NVIDIA driver is blocking the user's access.

Answer

The GPU memory is fragmented across the nodes.

Question 2

Refer to the exhibit. An administrator applies this security policy to a container runtime environment. What is the immediate effect on containerized AI applications within this scope?

Accepted Answer

The container is unable to detect or communicate with the NVIDIA GPU. The policy explicitly denies access to the character device files associated with the NVIDIA GPU. Without access to /dev/nvidia0, /dev/nvidiactl, and /dev/nvidia-uvm, the CUDA runtime cannot interact with the GPU hardware. Consequently, any attempt to initialize a CUDA device will fail, causing the application to crash. This policy is a common restrictive measure in high-security environments where GPU access must be strictly managed or audited.

Answer

The container gains elevated privileges to access the host GPU

Answer

The container can use the GPU but cannot perform memory mapping

Answer

The container experiences increased latency for GPU operations

Question 3

Which NVIDIA technology enables a GPU to be shared among multiple virtual machines or containers while maintaining strict hardware isolation?

Accepted Answer

NVIDIA Multi-Instance GPU (MIG). NVIDIA Multi-Instance GPU (MIG) is the core technology that enables hardware-level partitioning on supported GPUs. By creating dedicated instances that have their own memory and compute resources, MIG ensures that workloads remain isolated, providing predictable quality of service and security in multi-tenant environments. This is a foundational technology for maximizing GPU utilization in cloud and enterprise data center environments where mixed workloads are common.

Answer

NVIDIA CUDA streams

Answer

NVIDIA Collective Communications Library (NCCL)

Answer

NVIDIA GPUDirect RDMA

Question 4

An administrator is optimizing multi-GPU utilization. Which TWO of the following configurations allow multiple containers to share a single physical GPU on a supported NVIDIA architecture?

Accepted Answer

Enabling NVIDIA Time-Slicing in the GPU Operator configuration.. Multi-Instance GPU (MIG) and Time-Slicing are the primary methods for sharing physical GPU resources. MIG provides hardware-level isolation, while Time-Slicing provides software-based multiplexing. Understanding these options is essential for AI operations, as it allows administrators to maximize hardware ROI by supporting smaller workloads that do not require an entire A100 or H100 GPU, effectively increasing the density of the training or inference environment.

Answer

Increasing the CUDA_VISIBLE_DEVICES environment variable.

Answer

Setting a higher GPU limit in the Kubernetes manifest.

Answer

Deploying the NVIDIA Network Operator.

Question 5

An administrator is optimizing a cluster for AI model training using NVIDIA Base Command. Which TWO tasks are critical for ensuring consistent performance across the training nodes?

Accepted Answer

Standardizing NVIDIA driver and NCCL library versions across all nodes. Consistency in high-performance AI clusters depends on hardware synchronization and resource availability. Ensuring that all nodes run identical driver and firmware versions prevents subtle performance regressions during multi-node training. Furthermore, configuring GPUDirect RDMA is essential for reducing latency in inter-GPU communication, which is a major bottleneck in distributed training jobs. These steps are fundamental to maintaining high utilization and predictable training times across large-scale NVIDIA accelerated infrastructure.

Answer

Increasing the size of the swap partition on each node

Answer

Disabling the NVIDIA persistence mode to save power

Answer

Reducing the number of CUDA streams per training job

Question 6

An AI researcher is running a large-scale training job on an NVIDIA DGX system using Kubernetes. They observe that GPU utilization is consistently low despite high CPU load. Which workload management configuration is most likely to resolve this bottleneck by optimizing data pipeline throughput?

Accepted Answer

Optimize the data loader by increasing worker threads and implementing buffered prefetching.. Low GPU utilization often stems from data starvation where the GPU waits for the CPU to preprocess or fetch data from storage. Implementing data prefetching and increasing the number of workers in the data loader ensures the GPU remains fed with batches. This is critical in AI operations to maximize compute ROI and reduce total training time for expensive cluster resources.

Answer

Increase the GPU memory limit in the pod specification.

Answer

Enable NVIDIA Multi-Instance GPU (MIG) for this specific job.

Answer

Lower the batch size to reduce the memory footprint on the GPU.

Question 7

Which THREE components are required for a container to successfully leverage NVIDIA GPUs on a Kubernetes cluster?

Accepted Answer

NVIDIA Container Toolkit. Successful GPU integration in Kubernetes relies on the interaction between the container engine, the NVIDIA driver, and the device plugin. The driver provides the kernel-level interface, the container runtime handles the low-level injection, and the device plugin advertises the hardware to the scheduler. These three parts must be correctly configured and compatible to ensure that containers can schedule and utilize GPUs effectively without manual configuration or persistent connectivity issues.

Answer

NVIDIA Data Center GPU Manager (DCGM)

Answer

Pre-installed CUDA Toolkit inside the container

Question 8

A system administrator is troubleshooting a 'CUDA error: invalid device ordinal' when launching a job on a multi-GPU system. What is the most likely cause?

Accepted Answer

CUDA_VISIBLE_DEVICES is set to an out-of-range index.. The 'invalid device ordinal' error typically indicates that the application is attempting to access a GPU index (e.g., GPU 4) that does not exist or is not visible to the process. This is often caused by environment variables like CUDA_VISIBLE_DEVICES being configured incorrectly, mapping the application to non-existent hardware. Ensuring the logical-to-physical GPU mapping is accurate is vital for correct job execution on shared infrastructure.

Answer

The GPU driver is corrupted and requires a reinstall.

Answer

The GPU memory is full.

Answer

The InfiniBand fabric is misconfigured.

Question 9

During the deployment of an AI model training workload on a multi-node cluster, the administrator notices that inter-node communication is significantly slower than expected. Which deployment aspect should be investigated first?

Accepted Answer

Check the NCCL_DEBUG and network interface configuration for the training job.. NCCL (NVIDIA Collective Communications Library) is the primary engine for inter-node communication in distributed training. Misconfiguration of the network interface or the underlying fabric provider (e.g., InfiniBand or RoCE) will lead to significant performance bottlenecks. Investigating the NCCL configuration and the network topology ensures that the GPUs are utilizing the highest bandwidth available, which is vital for preventing training jobs from stalling during gradient synchronization across multiple nodes.

Answer

Verify that the GPU memory usage is below 50% on all nodes.

Answer

Restart the Kubernetes API server to refresh the node connection states.

Answer

Update the NVIDIA driver to the latest gaming-optimized release.

Question 10

A platform team is deploying the NVIDIA GPU Operator on a Kubernetes cluster to manage GPU nodes. They want the Operator to automatically install the NVIDIA driver, the container toolkit, and the device plugin on each GPU node. Which component of the GPU Operator is responsible for installing the NVIDIA driver on the host?

Accepted Answer

NVIDIA Driver Container. The GPU Operator automates the deployment of all NVIDIA software components on Kubernetes. The Driver Container is specifically designed to install the NVIDIA driver on the host. It runs as a DaemonSet and installs the driver into the host's filesystem, eliminating the need for manual driver installation. The other components handle container runtime integration, resource advertisement, and validation.

Answer

NVIDIA Container Toolkit

Answer

NVIDIA Device Plugin

Answer

NVIDIA GPU Operator Validator

Question 11

An AI researcher is debugging a multi-node training job using NCCL. Which TWO actions should they take to diagnose potential network-related performance degradation?

Accepted Answer

Set NCCL_DEBUG=INFO to monitor communication patterns and connection issues.. Debugging multi-node communication requires verifying both software configurations and physical network health. NCCL_DEBUG settings provide granular insight into connection establishment and topology detection, while NCCL tests provide baseline performance metrics. Identifying these issues early is essential for scaling models across large clusters, as network overhead can quickly become the dominant factor in training time during distributed synchronized gradient descent.

Answer

Increase the number of threads in the PyTorch DataLoader.

Answer

Switch from NCCL to MPI for all communication operations.

Answer

Lower the precision of the model to float16.

Question 12

Which utility is primarily used to monitor and manage NVIDIA GPU power, temperature, and usage statistics in real-time on a Linux-based deployment?

Accepted Answer

nvidia-smi. nvidia-smi (NVIDIA System Management Interface) is the command-line utility used to interact with the NVIDIA driver for monitoring and managing GPU devices. It is the fundamental tool for AI ops teams to verify that GPUs are properly detected and performing within expected thermal and power envelopes during model training or inference tasks on DGX or workstation systems.

Answer

top

Answer

htop

Answer

dmesg

Question 13

Refer to the exhibit. An administrator attempts to deploy a GPU-based pod, but it remains in the 'Pending' state. What is the most likely cause based on the error log?

Accepted Answer

The NVIDIA device plugin is not correctly advertising GPU resources.. The error 'Insufficient nvidia.com/gpu' indicates that the Kubernetes scheduler cannot find a node with available GPU resources that match the pod's request. This typically happens when the device plugin is not correctly reporting resource availability to the scheduler, or the cluster is over-provisioned. Resolving this requires ensuring the GPU Operator is running and that the device plugin has successfully registered the GPUs with the Kubelet on the worker nodes.

Answer

The pod is requesting more CPU cores than the node can provide.

Answer

The node is in a 'NotReady' state due to high disk latency.

Answer

The pod security policy prohibits the use of GPU-accelerated containers.

Question 14

An administrator wants to prevent unauthorized users from accessing sensitive model weights stored in GPU memory. Which security feature should be implemented to ensure hardware-level isolation of the memory space?

Accepted Answer

Implement NVIDIA Confidential Computing.. Hardware-level memory isolation via Confidential Computing technologies, such as NVIDIA Confidential Computing (CC), protects data in use by encrypting memory contents. For administrators handling sensitive IP, this provides a root-of-trust that persists even if the OS or hypervisor is compromised. This is a critical administrative control for ensuring compliance and data sovereignty in multi-tenant cloud environments where infrastructure is shared among different departments or organizations.

Answer

Enable standard Linux filesystem permissions.

Answer

Use a simple SSH firewall rule.

Answer

Update the NVIDIA CUDA shared library path.

Question 15

When debugging a workload that consistently crashes with 'Out of Memory' (OOM) errors despite sufficient GPU VRAM, what is the most likely cause related to workload management?

Accepted Answer

The container memory limit is lower than the data processing requirements.. OOM errors can occur due to host-side system memory exhaustion if the workload manages large datasets in system RAM before loading them into the GPU. If the container memory limit is set too low for the data processing pipeline, the entire container will be terminated. This highlights the need to correctly balance both GPU VRAM and system memory limits in the container specification.

Answer

The CUDA driver version is incompatible with the installed GPU.

Answer

The GPU device plugin is not correctly reporting free memory.

Answer

The training job is using mixed-precision training (FP16).

Question 16

Refer to the exhibit. A cluster administrator notices that GPU jobs with this PriorityClass are failing to start even when empty GPUs are available. What is the most likely cause?

Accepted Answer

Insufficient nodes satisfy the 'minAvailable' requirement for the pod group.. When using the Volcano scheduler with NVIDIA GPUs, priority classes alone are insufficient if the Gang Scheduling policy is not met. If the job requests multiple pods that cannot be satisfied simultaneously, Volcano will leave the pods in a pending state to avoid resource fragmentation. This ensures that massive training jobs do not consume isolated resources that would otherwise result in deadlocks during collective communication synchronization.

Answer

The PriorityClass value is too low to trigger preemption.

Answer

The NVIDIA Device Plugin requires a restart to recognize the PriorityClass.

Answer

The PriorityClass name violates Kubernetes naming conventions.

Question 17

Which TWO of the following steps are essential when deploying the NVIDIA GPU Operator on a Kubernetes cluster to ensure that GPU resources are discoverable by the scheduler?

Accepted Answer

Enable the node feature discovery (NFD) service in the operator configuration.. The NVIDIA GPU Operator automates the lifecycle of NVIDIA software components, including the driver, toolkit, and device plugin. By configuring the operator to deploy the GPU device plugin and the Node Feature Discovery (NFD) service, the cluster gains the ability to identify GPU hardware and advertise it as an allocatable resource. Without these, the Kubernetes scheduler cannot place pods requiring GPU resources, leading to 'Pending' status for those workloads.

Answer

Manually install the NVIDIA Container Toolkit on every worker node prior to operator deployment.

Answer

Modify the Kubernetes API server manifest to include the NVIDIA-specific admission controller.

Answer

Disable the default Kubernetes scheduler to allow the NVIDIA scheduler plugin to take over.

Question 18

Which administrative practice ensures that a cluster is prepared for the arrival of new NVIDIA GPU hardware with minimal downtime?

Accepted Answer

Update the OS and NVIDIA drivers across the cluster.. Maintaining an up-to-date driver and software stack via a robust package management system is the key to minimizing downtime during hardware upgrades. Administrators must ensure that the kernel and driver versions are compatible with the new hardware before it arrives. This proactive preparation is vital for maximizing cluster uptime and ensuring that researchers can immediately begin using the new hardware for their AI experiments without configuration delays.

Answer

Wait for the hardware to arrive before checking driver support.

Answer

Manually install every CUDA library on all nodes.

Answer

Modify the BIOS settings to ignore PCIe errors.

Question 19

Which THREE factors should be considered when estimating GPU memory requirements for a Large Language Model (LLM) fine-tuning job?

Accepted Answer

The memory size of the model parameters.. When sizing LLM workloads, you must account for the model weights, optimizer states, and gradient buffers. Additionally, activations consume significant memory during the forward and backward passes. Understanding these components is essential for AI Ops, as incorrect sizing leads to OOM crashes early in the training process, wasting significant compute time and delaying model delivery in production environments.

Answer

The total number of CPU threads used.

Answer

The network latency between nodes.

Question 20

An administrator is planning to monitor GPU utilization across a large cluster. Which component should be deployed to collect metrics that are compatible with Prometheus?

Accepted Answer

NVIDIA DCGM Exporter. NVIDIA DCGM (Data Center GPU Manager) is the industry-standard tool for collecting GPU health and telemetry data. By deploying the DCGM Exporter, the administrator enables the conversion of hardware telemetry into a Prometheus-friendly format. This integration is vital for observability, allowing teams to set up alerts and dashboards to track GPU usage, power consumption, and memory allocation across the entire fleet of accelerated computing nodes.

Answer

NVIDIA GPU Operator

Answer

NVIDIA Triton Inference Server

Answer

NVIDIA CUDA Toolkit