NCP-AIO Practice Test 13 — 15 Questions

Question 1

An AI operations team is deploying NVIDIA AI Enterprise on a Kubernetes cluster using the NVIDIA GPU Operator. They need to ensure that GPU metrics such as utilization, memory usage, and temperature are collected and exposed to Prometheus for monitoring. Which component of the GPU Operator is responsible for this?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

NVIDIA DCGM Exporter. The NVIDIA DCGM Exporter is the component of the GPU Operator that gathers GPU metrics via DCGM and exposes them in Prometheus format. It is specifically designed for monitoring GPU health and performance in Kubernetes. Other components like the Container Toolkit, Device Plugin, and Validator serve different purposes and do not provide metrics collection for Prometheus.

Answer

NVIDIA GPU Operator Validator

Answer

NVIDIA GPU Device Plugin

Answer

NVIDIA Container Toolkit

Question 2

An administrator is deploying the NVIDIA GPU Operator on a Kubernetes cluster using Helm. The cluster nodes have NVIDIA GPUs and the administrator wants to ensure that the GPU Operator can automatically label nodes with GPU properties and install the device plugin. Which prerequisite must be met on the cluster nodes before installing the GPU Operator?

Accepted Answer

The nodes must have a supported container runtime such as Docker or containerd installed and configured.. Before installing the NVIDIA GPU Operator, the Kubernetes cluster must have a container runtime such as Docker or containerd installed and configured on all nodes. This is a basic Kubernetes requirement because the Operator deploys its components as containers. Other components like the NVIDIA driver, container toolkit, and DCGM exporter are managed by the Operator itself and do not need to be pre-installed.

Answer

The NVIDIA driver must be pre-installed on the nodes.

Answer

The nodes must have the NVIDIA Container Toolkit installed.

Answer

The nodes must have the NVIDIA DCGM exporter pre-installed for monitoring.

Question 3

A platform team operates a Kubernetes cluster where several teams submit GPU training jobs. The administrator needs to enforce per-namespace limits on the number of GPUs that can be consumed and prevent a single namespace from monopolizing all GPU capacity. Which TWO Kubernetes resources should be configured to achieve this? (Choose two.)

Accepted Answer

A LimitRange that sets a default and maximum nvidia.com/gpu value for containers in the namespace.. ResourceQuota enforces an aggregate ceiling on nvidia.com/gpu per namespace, while LimitRange constrains and defaults the per-container GPU request. Together they bound total namespace consumption and prevent any single container from grabbing an outsized share, which is exactly the governance the platform team needs.

Answer

A PriorityClass that assigns a low priority value to all training pods in the namespace.

Answer

A PodSecurityPolicy that denies privileged containers in the namespace.

Answer

A NetworkPolicy that restricts traffic between pods in different namespaces.

Question 4

A research group submits a distributed PyTorch training job spanning eight GPUs across two nodes. The job completes but produces a model with accuracy far below the single-node baseline, and logs show that several ranks started training before their peers had initialized the process group. The administrator must ensure that all ranks are launched together and that a failed rank terminates the whole job. Which combination of Kubernetes mechanisms should be used?

Accepted Answer

Use a gang-scheduling mechanism such as Volcano or the Kubeflow Training Operator with PyTorchJob and configure the rendezvous endpoint so all worker replicas are admitted together.. Distributed training needs collective admission and coordinated rank metadata. A gang scheduler plus the Kubeflow Training Operator's PyTorchJob admits all replicas together and injects rendezvous details, eliminating the race where early ranks initialize before their peers, while the controller fails the whole job if any replica cannot run.

Answer

Schedule the job on a single node with eight GPUs and set the CUDA_VISIBLE_DEVICES variable per container to isolate each rank.

Answer

Create eight separate Deployments, one per rank, and pass the rank index through an environment variable in each Deployment manifest.

Answer

Deploy the job as a Kubernetes Job with parallelism set to eight and completions set to one, relying on the default pod startup ordering.

Question 5

An administrator manages a cluster where inference services and batch training share the same GPU nodes. During business hours, inference pods must be scheduled promptly, while training jobs can wait. The administrator wants preemption so that a pending high-priority inference pod can evict a lower-priority training pod when no GPU is free, with evicted training resuming later. Which configuration achieves this?

Accepted Answer

Create a PriorityClass with a high value for inference and a lower one for training, reference them in the pod specs, and enable preemption in the scheduler so lower-priority pods are evicted when needed.. Priority and preemption are the native scheduling features designed for exactly this pattern: a higher-priority pending pod triggers eviction of lower-priority running pods to free capacity. Defining distinct PriorityClasses for inference and training and referencing them in pod specs enables the scheduler to make that decision. Because the training workload is managed by a group-aware controller, the evicted job can be requeued and restarted from its last checkpoint.

Answer

Set the training pods' terminationGracePeriodSeconds to zero and add a PodDisruptionBudget so the scheduler can remove them immediately when capacity is needed.

Answer

Apply node affinity rules that pin inference pods to dedicated nodes and training pods to separate nodes, then let each workload schedule only within its own pool.

Answer

Define a ResourceQuota on the training namespace capping GPU requests so that free capacity always remains available for inference pods.

Question 6

An administrator manages a shared NVIDIA cluster where several teams run inference services. One team's pods are being evicted repeatedly, and DCGM metrics show the node's GPUs are healthy but memory on the devices is nearly exhausted. The team insists their model fits. Which action should the administrator take FIRST to identify the cause?

Accepted Answer

Inspect the pod specs and running processes for GPU memory held by other containers on the same node.. Healthy GPUs with near-exhausted device memory in a shared cluster usually indicate that another container on the same node is holding memory on the same device. The fastest way to confirm this is to inspect pod specifications and running processes to see which workloads share the GPU. Replacing hardware, changing host memory limits, or tuning telemetry do not address the allocation conflict and delay the correct diagnosis.

Answer

Increase the pod's memory limit in the deployment manifest and restart it.

Answer

Replace the affected GPUs and re-run the job on fresh hardware.

Answer

Lower the DCGM sampling interval so metrics capture the spike more precisely.

Question 7

An AI platform team runs GPU-accelerated inference pods on a Kubernetes cluster with the NVIDIA GPU Operator. During peak load, high-priority latency-sensitive inference pods are frequently preempted by large batch training jobs that were submitted later. The team wants the scheduler to guarantee that inference pods always win placement and preemption decisions against training pods without manually cordoning nodes. Which mechanism should the administrator configure?

Accepted Answer

Assign the inference pods a higher PriorityClass value and reference it in their pod spec so the kube-scheduler preempts lower-priority training pods when resources are scarce.. Kubernetes resolves GPU contention through the scheduler's priority and preemption logic, not through device sharing or quota limits. Giving inference pods a higher PriorityClass value makes the kube-scheduler place them first and evict lower-priority training pods when capacity is exhausted, which delivers the guaranteed preference the platform team requires.

Answer

Enable the GPU Operator's time-slicing feature with a replica count greater than one so inference and training pods share each physical GPU concurrently.

Answer

Create a separate namespace with a ResourceQuota that limits training pods to a small number of GPUs and place all inference pods in the default namespace.

Answer

Add a nodeSelector to the inference pods that pins them to nodes labeled with the NVIDIA GPU product name, so they only run on nodes not used by training.

Question 8

A platform team is preparing a Kubernetes cluster for AI workloads and wants the GPU device plugin, driver containers, and monitoring components deployed and kept in sync automatically on every GPU node. Which component should be installed to achieve this?

Accepted Answer

NVIDIA GPU Operator. The NVIDIA GPU Operator is purpose-built to manage the entire GPU software stack on Kubernetes nodes through automated reconciliation. It deploys the driver, container toolkit configuration, device plugin, and DCGM-based monitoring, so GPU resources are advertised and maintained without manual per-node work. The Network Operator addresses networking, while DCGM and the Container Toolkit are individual pieces the Operator already orchestrates.

Answer

NVIDIA Network Operator

Answer

NVIDIA DCGM standalone on each node

Answer

NVIDIA Container Toolkit installed manually on each node

Question 9

A platform team runs mixed training and inference workloads on a Kubernetes cluster with the NVIDIA GPU Operator. Inference pods are latency-sensitive and must not be preempted, while training pods can be interrupted and restarted. The team wants training jobs to yield GPUs to inference jobs when capacity is scarce, without manual intervention. Which Kubernetes mechanism should the team configure to achieve this behavior?

Accepted Answer

PriorityClass with preemptionPolicy set to PreemptLowerPriority on the inference pods.. Priority and preemption are the scheduler features designed for exactly this pattern. Assigning inference pods a higher-priority class with preemption enabled lets the scheduler evict lower-priority training pods when a node lacks free GPUs. Because training pods are restartable, the disruption is acceptable, and inference keeps its latency guarantees without manual intervention.

Answer

A PodDisruptionBudget on the training pods that guarantees a minimum number of running replicas.

Answer

ResourceQuota on the training namespace limiting total GPU requests below cluster capacity.

Answer

A node affinity rule on the inference pods that targets nodes with the highest available GPU memory.

Question 10

An AI operations engineer manages a Kubernetes cluster running the NVIDIA GPU Operator. A team wants its long-running inference deployment to be automatically rescheduled if the GPU on a node develops an uncorrectable error that the device plugin or health checks detect. The team also wants the node to stop accepting new GPU pods until the issue is resolved. Which combination of behaviors should the engineer rely on to meet these requirements?

Accepted Answer

The GPU Operator's health checks mark the GPU unhealthy, the device plugin stops advertising it, and Kubernetes taints or cordons the affected node so pods are rescheduled and no new GPU pods land there.. Meeting both requirements needs health-driven device withdrawal plus node-level scheduling exclusion. The GPU Operator's health checks detect uncorrectable errors, the device plugin stops advertising the faulty GPU, and the node is tainted or cordoned. Controllers then recreate pods on healthy nodes, and the taint keeps new GPU pods away until the fault is cleared.

Answer

A Horizontal Pod Autoscaler scales up replicas so healthy copies absorb traffic while the faulty node remains in service.

Answer

A liveness probe on the inference container restarts the pod on the same node when the GPU error causes a request failure.

Answer

A PodDisruptionBudget on the inference deployment forces the scheduler to migrate pods off the node when the GPU fails.

Question 11

A platform team runs a Kubernetes cluster where the NVIDIA GPU Operator is installed and time-slicing is configured with a ConfigMap that advertises four replicas per physical GPU. A data scientist submits a PyTorch training pod requesting nvidia.com/gpu: 1. The pod stays Pending indefinitely, and the scheduler event reads 'Insufficient nvidia.com/gpu'. The node's GPUs are otherwise idle and healthy. Which action most directly resolves the pending state?

Accepted Answer

Verify the time-slicing ConfigMap is labeled for the device plugin and that the node's GPU capacity now reports the multiplied replica count, then requeue the pod.. Time-slicing is enabled by a ConfigMap that the NVIDIA device plugin watches; without the expected label the plugin ignores it and keeps advertising one GPU per device. Confirming the ConfigMap is labeled and that the node's allocatable nvidia.com/gpu equals replicas multiplied by physical GPUs verifies the change propagated. Once capacity is correctly republished, the pending pod can be scheduled without altering its resource request.

Answer

Restart the nvidia-device-plugin pod so it re-reads the time-slicing ConfigMap and republishes the replicated resource capacity to the kubelet.

Answer

Add a nodeSelector that pins the pod to a specific GPU node, because the scheduler cannot match GPU requests without an explicit node affinity rule.

Answer

Change the pod's resource request to nvidia.com/gpu.shared: 1 so it matches the resource name that time-slicing exposes.

Question 12

An administrator is using NVIDIA Base Command Manager to manage a cluster with a mix of GPU and CPU nodes. They need to ensure that a newly added GPU node is correctly recognized and that jobs can be scheduled on it. Which TWO actions must be performed to integrate the new node into the Base Command Manager cluster? (Choose two.)

Accepted Answer

Install the Base Command Manager agent on the new node and register it with the head node.. Integrating a new node into a Base Command Manager cluster requires installing and registering the Base Command Manager agent so the head node can manage it, and defining the node in the cluster configuration with an appropriate node group assignment so it inherits policies and becomes schedulable. These two actions together ensure the node is recognized, provisioned correctly, and available for job scheduling.

Answer

Add the node's hostname and IP address to the /etc/hosts file on all cluster nodes.

Answer

Configure a static IP address for the node and add it to the cluster's DNS server.

Answer

Manually install the NVIDIA GPU driver on the node without using Base Command Manager's provisioning tools.

Question 13

A team trains a model inside an NGC PyTorch container on a DGX H100 node. Training starts, but after a few minutes the process dies and `dmesg` shows `Xid 79: GPU has fallen off the bus` on one GPU. The team needs to determine whether the fault is hardware or software before opening an RMA. Which two actions should they take to gather useful evidence? (Choose two.)

Accepted Answer

Run `nvidia-smi -q` and `nvidia-smi -q -d ECC,PAGE_RETIREMENT,ROW_REMAPPER` and capture the output.. Xid 79 indicates the GPU dropped off the PCIe bus, which can stem from a hardware fault, a degraded link, or a driver escalation after ECC or row-remap events. Before resetting anything, engineers should query ECC and row-remapper state and bundle the full bug report with the matching kernel log. These two sources together show whether the GPU had pending remaps or link errors, enabling an accurate hardware-versus-software determination and a defensible RMA decision.

Answer

Immediately reboot the node and re-run the training job to see whether it fails again.

Answer

Reset the GPU with `nvidia-smi --gpu-reset` and continue using the node if training succeeds afterward.

Answer

Delete and recreate the NGC container, then pull the image again from the registry.

Question 14

An operations team runs a multi-node NCCL all-reduce training job across four DGX nodes connected via InfiniBand. Training throughput is far below the expected linear scaling, and `nvidia-smi` shows GPU utilization oscillating between 20% and 40%. The network fabric is healthy and the GPUs are not thermally throttled. Which diagnostic step is MOST appropriate to identify the bottleneck?

Accepted Answer

Run `nccl-tests` (all_reduce_perf) with varying message sizes and collect NCCL debug logs by setting NCCL_DEBUG=INFO to inspect topology and algorithm selection.. The oscillating, low GPU utilization across multiple nodes during an all-reduce strongly suggests the collective communication is stalling. Running `nccl-tests` with multiple message sizes measures achieved bandwidth and highlights which sizes are inefficient, while NCCL_DEBUG=INFO reveals the detected topology, algorithm, and channels. Together these pinpoint whether the bottleneck is the fabric, the ring/tree algorithm, or an unexpected topology detection.

Answer

Reinstall the CUDA toolkit on all nodes to ensure matching driver and runtime versions.

Answer

Increase the training batch size by 4x to raise arithmetic intensity and re-measure GPU utilization.

Answer

Enable ECC memory scrubbing on all GPUs and monitor for corrected errors during training.

Question 15

A platform engineer manages an NVIDIA-accelerated Kubernetes cluster running the NVIDIA GPU Operator on nodes with A100 GPUs. Several data-science teams submit training jobs, and the engineer must ensure each team's pods receive a full physical GPU exclusively, with no two pods sharing the same device. Which scheduling configuration should the engineer apply to the pod specification to guarantee exclusive whole-GPU allocation?

Accepted Answer

Set resources.limits to nvidia.com/gpu: 1 and resources.requests to nvidia.com/gpu: 1, letting the NVIDIA device plugin allocate an exclusive GPU.. Declaring both a request and a limit for nvidia.com/gpu equal to 1 is the canonical way to obtain an exclusive whole GPU in Kubernetes. The NVIDIA device plugin advertises GPUs as integer extended resources, so the scheduler reserves one entire device for the pod. Because extended resources cannot be fractional, no co-scheduling or sharing occurs, which precisely meets the exclusive allocation requirement.

Answer

Enable MIG by setting the nvidia.com/mig.config label on the node to all-balanced and request nvidia.com/mig-1g.5gb in the pod.

Answer

Configure the pod to mount the hostPath /dev/nvidia0 and set privileged: true so the container can access the first GPU directly.

Answer

Set a nodeSelector for nvidia.com/gpu.product=A100 and rely on the scheduler to place one pod per node automatically.