NCP-AIO Practice Questions 2 — 50 Questions

Question 1

An AI operations team manages a shared Kubernetes cluster where a nightly batch training workload requests nvidia.com/gpu resources and occasionally consumes all GPU memory on a node, causing a co-located interactive notebook pod to fail with CUDA out-of-memory errors. The team wants the interactive notebook to be isolated from the batch workload's memory usage without adding new hardware. Which action best achieves this on supported data center GPUs?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Configure Multi-Instance GPU profiles so the notebook and batch workloads run on separate MIG instances with dedicated memory partitions on the same physical GPU.. The conflict is shared GPU memory, not scheduling order. MIG is the only listed mechanism that gives each workload a dedicated, hardware-isolated memory partition on the same physical device, letting the notebook and batch job coexist without the batch job starving the notebook's framebuffer, and it requires no extra hardware.

Answer

Enable time-slicing with a replica count of four so the notebook and batch workloads alternate on the same GPU in round-robin fashion.

Answer

Set a memory limit on the batch container using the standard Kubernetes resources.limits.memory field to cap its GPU framebuffer usage.

Answer

Add a higher PriorityClass to the notebook pod so the kube-scheduler preempts the batch workload whenever memory pressure occurs.

Question 2

A research team submits a multi-node training job using a `Job` with eight pods, each requesting one GPU. The cluster has eight GPU nodes, each with one A100. The administrator observes that all eight pods are spread one per node and the job runs, but throughput is far below expectations and NCCL logs show repeated fallback from GPUDirect RDMA to socket transport. Which action most directly addresses the root cause?

Accepted Answer

Deploy the NVIDIA Network Operator to install and configure the RDMA stack and GPUDirect RDMA on the GPU nodes.. The symptom points to NCCL abandoning RDMA and using the slower socket transport for inter-node collectives. That path depends on the high-speed fabric drivers, RDMA devices, and GPUDirect RDMA support being installed and configured on every GPU node, which is precisely what the NVIDIA Network Operator provisions in tandem with the GPU Operator. Changing rank counts, disabling peer-to-peer, or collapsing topology onto one node does not restore the missing RDMA capability.

Answer

Set `NCCL_P2P_DISABLE=1` in the job's environment to force peer-to-peer transfers over PCIe.

Answer

Add a node affinity rule forcing all eight pods onto a single node to enable NVLink communication.

Answer

Increase the number of replicas in the Job so more GPUs participate in the collective.

Question 3

In an NVIDIA-accelerated Kubernetes environment, why is it critical to configure a 'RuntimeClass' for GPU-enabled pods?

Accepted Answer

To ensure the container runtime correctly injects GPU drivers and libraries into the pod.. The RuntimeClass ensures that the container is started with the correct NVIDIA container runtime (e.g., nvidia-container-runtime). This runtime is responsible for performing the necessary hardware mapping, injecting the CUDA libraries, and setting the environment variables required for the container to actually utilize the GPU. Without this configuration, the pod might be scheduled successfully but will fail at runtime because it cannot communicate with the NVIDIA driver.

Answer

To allow the Kubernetes scheduler to place pods on nodes based on GPU availability.

Answer

To define the specific NVIDIA driver version required by the application.

Answer

To increase the priority of GPU workloads over standard CPU-only workloads.

Question 4

During a multi-GPU training job, you notice that one GPU consistently reports lower utilization and longer communication times compared to others. What is the most likely reason for this performance imbalance?

Accepted Answer

The GPU is connected via a slower interconnect.. In multi-GPU systems, uneven load distribution often stems from mismatched NVLink topologies or PCIe lane configurations. If one GPU is connected via a slower PCIe link rather than a high-speed NVLink interconnect, it becomes the bottleneck in collective communication operations like AllReduce. Ensuring symmetric connectivity across all GPUs is essential for predictable performance and preventing the 'straggler' effect in distributed deep learning training.

Answer

The GPU is defective and needs replacement.

Answer

The training framework is not using CUDA streams.

Answer

The ambient server temperature is too high.

Question 5

An administrator must run a batch inference job that requires exactly two NVIDIA GPUs on a Kubernetes cluster managed by the NVIDIA GPU Operator. Which pod specification field should be used to request those GPUs?

Accepted Answer

resources.limits with the key nvidia.com/gpu set to 2. GPUs exposed by the NVIDIA device plugin appear to Kubernetes as the extended resource nvidia.com/gpu. Extended resources must be declared in resources.limits, and setting that limit to 2 makes the scheduler reserve two GPUs on a suitable node and pass the devices into the container.

Answer

resources.requests with the key nvidia.com/gpu set to 2 and no limit

Answer

annotations with the key nvidia.com/gpu.count set to 2

Answer

nodeSelector with the key nvidia.com/gpu.count set to 2

Question 6

An AI operations engineer is preparing a Kubernetes cluster to run GPU-accelerated workloads using the NVIDIA GPU Operator. The cluster nodes already have NVIDIA data center GPUs installed and the NVIDIA driver is pre-installed on the host. The engineer wants to use the GPU Operator to manage the container toolkit, device plugin, and monitoring components but must avoid the Operator managing or upgrading the driver. Which configuration should be applied to the GPU Operator deployment?

Accepted Answer

Set the driver.enabled parameter to false in the GPU Operator's Helm chart values.. When the NVIDIA driver is already present on host nodes and should remain externally managed, the GPU Operator must be configured to not deploy its own driver container. The driver.enabled=false Helm value achieves this by skipping the driver daemonset while still deploying other components like the container toolkit, device plugin, and DCGM exporter. This preserves the existing driver and avoids conflicts.

Answer

Use the NVIDIA GPU Operator with the --set toolkit.enabled=false option to prevent driver installation.

Answer

Install the GPU Operator with the --set operator.driverVersion=latest flag to pin the driver version.

Answer

Deploy the GPU Operator but remove the nvidia-driver-daemonset after installation.

Question 7

A cloud architect is deploying NVIDIA AI Enterprise on a vSphere cluster with multiple ESXi hosts, each containing NVIDIA A100 GPUs. The architect must enable GPU virtualization using NVIDIA vGPU. Which two components are required to support vGPU on the ESXi hosts? (Choose two.)

Accepted Answer

NVIDIA vGPU guest driver installed in each VM. To enable NVIDIA vGPU on VMware ESXi, the host must have the NVIDIA vGPU Manager VIB installed, which allows the hypervisor to partition the physical GPU. Each VM that uses a vGPU must also have the NVIDIA vGPU guest driver installed. These two components are essential; other tools like the GPU Operator or Container Toolkit are not used on ESXi.

Answer

NVIDIA GPU Operator for Kubernetes

Answer

NVIDIA CUDA Toolkit installed on the ESXi host

Answer

NVIDIA Container Toolkit installed on the ESXi host

Question 8

Refer to the exhibit. An AI engineer observes that a model training job is running slower than expected. Based on the output, what is the primary cause of the performance degradation?

Accepted Answer

An administrator has set a software power limit that is lower than the GPU's maximum performance threshold.. The 'SW Power Cap' throttle reason indicates that the power limit is configured below the card's maximum design capacity, causing the GPU to downclock to P12 state to stay within the power envelope. This is crucial for AI operations because it signals that the hardware is being artificially throttled, necessitating a review of power policies to ensure optimal training performance in high-compute scenarios.

Answer

The GPU is overheating and the thermal management system has engaged to prevent hardware damage.

Answer

The GPU is currently idle and the driver has shifted the card to an energy-saving state.

Answer

The GPU is experiencing a PCIe bus error, forcing the system to reduce clock speeds to maintain stability.

Question 9

Which action must be performed after updating the NVIDIA driver on a Linux host to ensure that all active GPU containers recognize the new driver version?

Accepted Answer

Restart the containerized applications.. Containers utilize the host-level NVIDIA drivers injected via the container toolkit. When the host driver is updated, active containers will still be referencing the old driver libraries in their environment until they are restarted. Restarting the containers is the mandatory step to force them to bind against the newly installed libraries, ensuring compatibility and preventing runtime errors associated with outdated driver interfaces.

Answer

Rebuild all container images with the latest CUDA toolkit.

Answer

Upgrade the Docker daemon version.

Answer

Re-run the nvidia-container-toolkit installation script.

Question 10

What is the primary benefit of using an NVIDIA-Certified System for AI Enterprise deployments?

Accepted Answer

Validated compatibility and performance. NVIDIA-Certified Systems have passed rigorous testing to ensure optimal performance and compatibility with the NVIDIA AI Enterprise software stack. This certification minimizes the risk of hardware-software conflicts, performance bottlenecks, and stability issues during production. For AI Ops professionals, this translates to faster deployment times, lower maintenance costs, and high confidence in the reliability of the underlying infrastructure for mission-critical deep learning and inference workloads.

Answer

Guaranteed 100% network uptime

Answer

Automatic software license renewal

Answer

Free access to all AI models

Question 11

Refer to the exhibit. An administrator observes this in the logs during a multi-GPU training job. What is the performance implication of this setting?

Accepted Answer

GPU-to-GPU data transfer performance is significantly reduced. Disabling Peer-to-Peer (P2P) communication forces NCCL to use system memory (often via the network or PCIe bus) to facilitate data movement between GPUs. This bypasses the high-bandwidth NVLink interconnect, resulting in significantly increased latency and lower throughput for collective operations. This configuration is typically used only for debugging or when hardware incompatibility prevents direct P2P access, as it severely hinders the performance of multi-GPU, multi-node AI training workloads.

Answer

The training job will have lower memory consumption

Answer

The job will be more stable across nodes

Answer

The system will utilize less power during training

Question 12

When managing GPU resources in a shared cluster, which configuration best prevents 'noisy neighbor' scenarios where one GPU task consumes all available memory bandwidth?

Accepted Answer

Implementing NVIDIA MIG and strictly defining resource limits.. Using a combination of NVIDIA MIG and Kubernetes resource quotas provides the strongest isolation. By enforcing hardware-level partitioning via MIG, you ensure that memory bandwidth is strictly bounded for each slice. This is vital for AI Ops because it prevents high-throughput training jobs from starving small inference requests, ensuring stable latency across the entire cluster environment and improving total system reliability.

Answer

Increasing the CUDA_VISIBLE_DEVICES environment variable.

Answer

Setting a higher priority class for inference pods.

Answer

Configuring the NVIDIA DCGM Exporter alerts.

Question 13

An AI researcher is deploying a multi-node training job using NCCL on an InfiniBand network. The job is suffering from intermittent latency spikes. Which TWO steps should the engineer perform to troubleshoot the network configuration?

Accepted Answer

Check IB link status and error counters using 'ibstat'.. Troubleshooting high-performance networking in distributed training requires checking for physical layer stability and protocol-level misconfigurations. Verifying InfiniBand counters helps identify packet drops or link flapping, while ensuring NCCL environment variables are correctly set for the specific topology avoids suboptimal routing. These steps ensure that the inter-GPU communication remains within the high-bandwidth low-latency envelope required for scaling deep learning workloads effectively.

Answer

Reset the GPU thermal throttling threshold.

Answer

Update the OS kernel to the latest version.

Answer

Increase the system swap space to 512GB.

Question 14

Refer to the exhibit. An administrator is attempting to deploy a job to a namespace with a ResourceQuota defined. What is the cause of this error?

Accepted Answer

The pod manifest is missing the required GPU resource limits.. The error indicates that the namespace has a ResourceQuota enforcing that all pods must specify GPU limits, but the submitted pod manifest lacks a 'resources.limits.nvidia.com/gpu' entry. In multi-tenant environments, ResourceQuotas are essential for preventing a single user from consuming the entire GPU capacity. The manifest must include a valid GPU limit to satisfy the namespace policy, ensuring the cluster remains balanced across different organizational teams.

Answer

The GPU driver version is incompatible with the quota controller.

Answer

The cluster is out of available GPU capacity.

Answer

The NVIDIA Device Plugin is not running in the namespace.

Question 15

An administrator is tasked with deploying a multi-node training job using the NVIDIA GPU Operator. Which configuration must be present to ensure that pods are scheduled on nodes with identical GPU architectures to prevent performance degradation?

Accepted Answer

Use node affinity labels based on NFD-provided hardware information.. To maintain high-performance, synchronized training, all participating nodes should share the same GPU architecture and interconnect type (e.g., NVLink or InfiniBand). Using Kubernetes node affinity or anti-affinity rules combined with the hardware labels automatically generated by the Node Feature Discovery (NFD) service ensures that the scheduler places the job exclusively on compatible nodes. This prevents the training from falling back to slower, sub-optimal communication paths or heterogeneous hardware modes.

Answer

Enable the 'auto-scaling' feature in the NVIDIA GPU Operator.

Answer

Increase the timeout values for the NCCL collective communication operations.

Answer

Set the 'nvidia.com/gpu' resource limit to zero on all nodes except the master node.

Question 16

Which component in the NVIDIA AI Enterprise stack is responsible for providing the necessary user-space libraries and binaries to run GPU-accelerated applications inside containers?

Accepted Answer

NVIDIA Container Toolkit. The NVIDIA Container Toolkit is essential because it allows the container runtime to interact with the host's NVIDIA drivers. It provides the necessary libraries and the container runtime wrapper to expose GPUs inside the container environment. Without this toolkit, applications inside a container cannot access the GPU hardware, even if the host has the correct drivers installed, making it a critical deployment component.

Answer

NVIDIA GPU Operator

Answer

NVIDIA License System

Answer

NVIDIA Unified Fabric Manager

Question 17

Refer to the exhibit. An administrator notices poor performance in an AI training job. What is the most likely cause based on the CLI output?

Accepted Answer

The power limit is set too low for the current workload.. The output indicates that the GPU is currently throttling due to 'Sw Power Cap'. This means the software-defined power limit is restricting the GPU performance to stay within a specific wattage budget. This is common in densely packed servers or cloud environments where power infrastructure is shared. Monitoring power usage is vital because it directly impacts clock speeds, which in turn bottleneck training throughput and lengthen the time required for model convergence.

Answer

The GPU is overheating due to a fan failure.

Answer

The GPU driver is corrupted or out of date.

Answer

The workload is waiting for CPU memory allocation.

Question 18

Refer to the exhibit. The administrator has deployed the GPU Operator, but the node does not show GPU resources. What is the most likely cause?

Accepted Answer

The NFD service is not installed or configured correctly.. The lack of GPU resources in the node description indicates that the Node Feature Discovery (NFD) or the GPU Operator has not correctly identified or labeled the GPU hardware. This is a common deployment issue where the hardware-to-software handshake fails due to missing labels. Identifying this early is key to ensuring that the Kubernetes scheduler can correctly place AI workloads on nodes equipped with the necessary compute resources.

Answer

The node is not part of a valid Kubernetes namespace.

Answer

The GPU firmware is out of date.

Answer

The cluster is using an unsupported CPU architecture.

Question 19

An administrator is deploying NVIDIA AI Enterprise on a bare-metal cluster. The workload requires full GPU isolation with minimal latency. Which configuration should the administrator select to achieve this goal?

Accepted Answer

Implement bare-metal installation with NVIDIA GPUDirect RDMA enabled.. GPU Direct and bare-metal deployments are essential for latency-sensitive workloads. By avoiding hypervisor overhead, the administrator ensures direct path access to the GPU memory and interconnects. This configuration is critical in high-performance computing environments where jitter and interrupt latency can degrade model training performance. Selecting the right deployment mode is a foundational step in AI Operations to ensure optimal hardware utilization and predictable execution times for deep learning models.

Answer

Deploy using NVIDIA vGPU on a KVM hypervisor.

Answer

Configure the system using NVIDIA License System (NLS) in disconnected mode.

Answer

Utilize containerized GPU passthrough with a standard Docker runtime.

Question 20

When deploying NVIDIA AI Enterprise, why is the selection of the correct CUDA version in the container image critical during the installation phase?

Accepted Answer

The CUDA version must be compatible with the host driver version.. The CUDA version dictates which APIs and features are available to the AI application. Because the driver on the host must support the CUDA version used by the container (backward compatibility), mismatching these leads to runtime failures. This is a crucial AI Ops consideration as it directly affects the stability of the entire stack, ensuring that the software environment aligns with the underlying hardware capabilities for maximum performance and reliability.

Answer

It determines the speed of the GPU's memory bus.

Answer

It enables the use of the NVIDIA License System.

Answer

It is required for the installation of the GPU Operator.