NCP-AIO Practice Test 12 — 15 Questions

Question 1

An AI infrastructure team is preparing an air-gapped data center to install the NVIDIA GPU Operator. They have mirrored all required container images into a private registry. During installation, the Operator's pods fail with ImagePullBackOff because the components still reference images under nvcr.io. Which configuration is required to make the Operator and its managed components pull from the private registry?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Set the chart values for the image repository and tag for each component, and provide an image pull secret in the Operator namespace.. Air-gapped installs require pre-mirroring every image and rewriting all image references in the Helm values to the private registry, including the Operator and each managed component. Authentication to the private registry is supplied through an image pull secret placed in the Operator namespace so the kubelet can pull. DNS tricks, pull policy changes, or imagined auto-mirroring do not redirect image references and leave the components failing.

Answer

Enable the Operator's auto-mirroring feature so it copies images from nvcr.io at runtime.

Answer

Configure a DNS override so nvcr.io resolves to the private registry's IP address.

Answer

Add nvcr.io to the cluster's imagePullPolicy as Never.

Question 2

A platform team runs an on-premises Kubernetes cluster for AI inference. Several teams submit pods that request the same GPU device, and the scheduler places more pods onto a node than there are available GPUs, causing OOM errors on the device. The administrator wants the Kubernetes scheduler itself to prevent overcommitting GPUs without any custom admission controller. Which action should the administrator take?

Accepted Answer

Install the NVIDIA device plugin so GPUs are advertised as schedulable extended resources and add a matching resource request to each pod spec.. Advertising GPUs as extended resources through the NVIDIA device plugin is what lets the default scheduler treat each device as countable, finite node capacity. Pods that declare a GPU request are then placed only where an unallocated device exists, and no custom admission controller is required because the scheduler performs the arithmetic itself. Quotas, taints, and autoscalers influence eligibility or CPU and memory sizing, not device-level accounting.

Answer

Set a node taint on each GPU node and add a toleration to pods, so only pods that explicitly opt in are placed there.

Answer

Enable the Kubernetes Vertical Pod Autoscaler in recommendation mode on the GPU namespaces to detect device pressure and reschedule pods.

Answer

Create a ResourceQuota in each team namespace limiting the count of pods that may run, so fewer pods land on GPU nodes.

Question 3

An AI operations team is installing the NVIDIA GPU Operator on a Kubernetes cluster that uses a custom containerd configuration. They need to ensure that the GPU Operator can properly manage the container runtime. Which action should they take before installing the GPU Operator?

Accepted Answer

Ensure that the containerd configuration does not already include conflicting NVIDIA runtime settings.. Before installing the GPU Operator, any pre-existing NVIDIA runtime configuration in containerd should be removed to avoid conflicts. The operator manages the runtime configuration itself, so manual settings can interfere. Other options like changing the default runtime or disabling containerd are incorrect because the operator integrates with the existing runtime rather than replacing or requiring manual overrides.

Answer

Label the nodes with the appropriate container runtime version.

Answer

Set the default runtime in containerd to nvidia-container-runtime.

Answer

Disable the containerd systemd service and let the GPU Operator start its own runtime.

Question 4

An AI operations engineer manages a shared Kubernetes cluster running NVIDIA GPU Operator. Several teams report that their inference pods remain in a Pending state with the event '0/8 nodes are available: 8 Insufficient nvidia.com/gpu.' The administrator verifies that nvidia-smi on all nodes shows idle GPUs. Which action should the administrator take first to resolve the scheduling failure?

Accepted Answer

Inspect the node allocatable GPU count and any taints or labels that prevent scheduling on the idle nodes.. The Pending event shows the scheduler believes no node has a free nvidia.com/gpu resource, even though nvidia-smi reports idle devices. That gap is almost always caused by the device plugin not advertising GPUs, or by taints, labels, or affinity rules excluding the nodes. Inspecting allocatable counts and scheduling constraints identifies the actual blocker before any disruptive remediation.

Answer

Reinstall the NVIDIA GPU Operator and restart the kubelet on all worker nodes.

Answer

Add a nodeSelector for kubernetes.io/os=linux to the pod template.

Answer

Increase the GPU memory limit in the pod specification so the scheduler can fit the workload.

Question 5

An AI operations engineer manages a Kubernetes cluster where the NVIDIA GPU Operator's device plugin exposes GPUs as schedulable resources. A data science team submits a batch inference job that requests one GPU but does not specify a node selector or tolerations. The job stays in Pending while other GPU nodes remain idle because they carry a taint the GPU Operator applied to reserve them for a specific workload class. Which approach is the most appropriate for the engineer to make the job schedulable without disrupting the reserved nodes?

Accepted Answer

Add an appropriate toleration and node selector to the job so it can target the reserved GPU nodes.. A node taint repels pods unless they carry a matching toleration. The reserved GPU nodes are intentionally tainted, so a job that neither tolerates the taint nor selects an untainted node cannot schedule. Adding the correct toleration plus a node selector places the job on the reserved nodes while preserving the reservation for other classes of work.

Answer

Delete and recreate the device plugin daemonset so GPUs are re-advertised to the scheduler.

Answer

Increase the cluster's GPU resource quota in the namespace so the scheduler can allocate a card.

Answer

Remove the node taint from all GPU nodes so the job can be scheduled anywhere.

Question 6

A media company is deploying an inference service on a Kubernetes cluster with the NVIDIA GPU Operator installed. The service pods remain in Pending with the message that no nodes have the requested nvidia.com/gpu resource, even though the GPUs are healthy and the driver loads correctly on every node. Which troubleshooting step should the engineer perform first?

Accepted Answer

Check whether the NVIDIA device plugin pods are running and have registered the GPUs with kubelet.. Kubernetes learns about GPUs through the device plugin framework: the NVIDIA device plugin advertises nvidia.com/gpu for each visible GPU by registering with kubelet. When the driver is healthy but pods report no such resource, the plugin is the missing link. Checking its pod status and logs quickly reveals scheduling failures, crashes, or socket registration errors before deeper investigation.

Answer

Reinstall the NVIDIA driver on every node to refresh the kernel module.

Answer

Increase the kubelet's pod density limit so the plugin can be scheduled.

Answer

Enable Multi-Instance GPU mode on all GPUs so each partition advertises capacity.

Question 7

A team runs a multi-node NCCL all-reduce training job on four DGX H100 nodes connected by an InfiniBand fabric. Scaling efficiency is poor: throughput barely improves beyond two nodes, and `nvidia-smi` shows NIC transmit counters on each GPU's assigned HCA are far below the PCIe link capacity while GPU compute utilization sits at ~55%. The fabric manager logs report all links as up with no symbol errors. Which action should the administrator take first to diagnose the interconnect bottleneck?

Accepted Answer

Run `nccl-tests` all_reduce_perf with the same process count and message sizes, then compare its reported bus bandwidth against the theoretical peak for the fabric.. When every fabric link is error-free but HCA counters sit well below link bandwidth, the bottleneck is in how the collective is scheduled or placed, not in raw link health. Reproducing the all-reduce with nccl-tests and comparing achieved bus bandwidth to the fabric's theoretical peak isolates the communication path and reveals whether the gap comes from topology detection, ring construction, or process placement before any tuning is applied.

Answer

Switch the job to use the NCCL tree algorithm via NCCL_ALGO=TREE because ring all-reduce cannot scale across four nodes.

Answer

Enable GPUDirect RDMA by exporting NCCL_NET_GDR_LEVEL=SYS and rerunning the job to confirm whether host-memory staging was the cause.

Answer

Increase the NCCL_BUFFSIZE environment variable to its maximum so each all-reduce chunk transfers more data per operation.

Question 8

A research organization runs an NVIDIA DGX SuperPOD with a Kubernetes cluster managed by the NVIDIA GPU Operator and Network Operator. A distributed training job using PyTorch DDP across 32 nodes stalls at initialization, and the administrator suspects the collective communication library is not selecting the high-speed fabric. Which configuration should the administrator verify first to ensure NCCL uses the correct network interface and topology?

Accepted Answer

Confirm that NCCL_IB_DISABLE is set to 0, NCCL_SOCKET_IFNAME matches the high-speed fabric interface, and NCCL_TOPO_FILE or the topology-aware plugin is loaded on each node.. NCCL chooses transports and interfaces based on environment variables and detected topology. When NCCL_IB_DISABLE is set incorrectly or NCCL_SOCKET_IFNAME points to the wrong interface, collectives fall back to TCP over the management network or fail to connect, causing distributed jobs to hang at initialization. Verifying these variables and the topology file on every node is the correct first diagnostic step.

Answer

Increase the pod's nvidia.com/gpu limit to 8 so each node exposes all GPUs to the training process.

Answer

Set the CUDA_VISIBLE_DEVICES variable to list all GPUs and restart the training job.

Answer

Enable the NVIDIA MIG feature on all nodes so each rank gets an isolated GPU slice for communication.

Question 9

A financial services company is deploying NVIDIA AI Enterprise on a VMware vSphere cluster with NVIDIA A100 GPUs. The security team requires that GPU workloads be isolated at the hardware level, with separate memory and fault domains, to meet regulatory compliance. The company also wants to maximize GPU utilization by running multiple workloads concurrently. Which NVIDIA feature should be enabled to meet these requirements?

Accepted Answer

NVIDIA Multi-Instance GPU (MIG). NVIDIA Multi-Instance GPU (MIG) is the only feature that provides hardware-level partitioning of an A100 GPU into isolated instances with dedicated memory, cache, and compute resources. This meets the need for fault domain separation and regulatory compliance while enabling concurrent execution of multiple workloads, thereby maximizing utilization. Other options either share resources without isolation or address different concerns like data transfer or inter-GPU communication.

Answer

NVIDIA NVLink with SHARP

Answer

NVIDIA vGPU with time-sliced scheduling

Answer

NVIDIA GPUDirect Storage

Question 10

An MLOps engineer needs to guarantee that a latency-sensitive inference Deployment always has GPU capacity available, even when a large training Job is submitted to the same namespace. The cluster uses the NVIDIA GPU Operator and nodes have four A100 GPUs each. Which approach reliably reserves GPU capacity for the inference Deployment?

Accepted Answer

Create a separate node pool and use nodeSelector or nodeAffinity to pin the inference Deployment to nodes tainted for inference only. Only a dedicated, tainted node pool combined with nodeSelector or nodeAffinity creates a hard capacity reservation that training workloads cannot violate. Priority-based preemption and MPS improve scheduling behavior or sharing but do not prevent a co-located training Job from consuming the GPUs first, and a namespace ResourceQuota limits aggregate requests without reserving specific capacity for the inference Deployment.

Answer

Set a higher PriorityClass on the inference Deployment so it preempts training pods when needed

Answer

Enable the NVIDIA MPS control daemon and configure each inference pod with a shared memory fraction

Answer

Apply a ResourceQuota on the namespace that limits nvidia.com/gpu to the number of inference replicas

Question 11

A platform team is installing the NVIDIA GPU Operator on a Kubernetes cluster that runs a mix of GPU and non-GPU nodes. They want the operator to manage the driver lifecycle only on nodes that actually have NVIDIA GPUs, without requiring manual taints on non-GPU nodes. Which configuration should they apply to the ClusterPolicy to achieve this?

Accepted Answer

Set `driver.nodeSelector` in the ClusterPolicy to match a label such as `nvidia.com/gpu.present: "true"`.. The GPU Operator uses node selectors to determine where components are deployed. Configuring `driver.nodeSelector` with a label that identifies GPU-equipped nodes ensures driver management is scoped to those nodes only. This avoids the need for manual taints on non-GPU nodes and aligns with the operator's declarative model for heterogeneous clusters.

Answer

Set `driver.enabled: false` and rely on preinstalled drivers on all nodes.

Answer

Set `driver.rdma.enabled: true` to limit driver operations to GPU nodes.

Answer

Set `driver.enabled: true` and `driver.useNvidiaDriverRoot: true` in the ClusterPolicy.

Question 12

An administrator runs a Kubernetes cluster with the NVIDIA GPU Operator. A data science team wants to run several small inference containers that each use only a fraction of a GPU's compute and memory, but the cluster currently assigns whole GPUs per pod. Which approach allows multiple containers to share a single physical GPU with memory isolation?

Accepted Answer

Enable Multi-Instance GPU mode on supported GPUs and configure the device plugin to advertise MIG instances as schedulable resources.. MIG is the only mechanism here that divides a physical GPU into hardware-isolated instances with separate memory and compute slices. By enabling MIG and having the device plugin advertise each instance, the scheduler can place multiple inference containers on one GPU while keeping their memory and faults isolated.

Answer

Reduce the container's nvidia.com/gpu request to a fractional value such as 0.5.

Answer

Set the device plugin's time-slicing configuration so that each GPU is advertised multiple times.

Answer

Configure the MPS control daemon to allow concurrent kernel execution across containers.

Question 13

During an air-gapped installation of the NVIDIA GPU Operator, the administrator must make all required images available to the cluster. Which component is responsible for pulling the Operator's operand images from the private registry?

Accepted Answer

The GPU Operator's ClusterPolicy must reference the private registry so operand pods use images from that registry.. Air-gapped deployments require the ClusterPolicy to specify the private registry and repository paths for each operand image. This directs the Operator to deploy pods that pull driver, toolkit, device plugin, and monitoring images from the internal registry. Kubelet mirrors, toolkit configuration, and image pull secrets address adjacent concerns but do not select the operand image sources.

Answer

The NVIDIA Container Toolkit must be pointed at the private registry through its config.toml file.

Answer

The Operator's Helm chart values must include an imagePullSecret for the private registry on every namespace.

Answer

The kubelet on each node must be configured with a mirror registry that rewrites all container image pulls.

Question 14

A Kubernetes cluster administrator is installing the NVIDIA GPU Operator and wants to ensure that GPU workloads are scheduled only on nodes with healthy GPUs. The administrator plans to use the operator's built-in health checks. Which component is responsible for monitoring GPU health and marking nodes as unschedulable when a GPU fails?

Accepted Answer

NVIDIA GPU Operator's health check component (part of the operator's node validation). The GPU Operator's health check component continuously monitors GPU status and taints nodes with unhealthy GPUs. This prevents Kubernetes from scheduling new GPU workloads on those nodes, improving reliability. The device plugin handles resource advertisement, while DCGM Exporter provides metrics; neither automatically taints nodes based on health.

Answer

NVIDIA DCGM Exporter

Answer

NVIDIA GPU Feature Discovery

Answer

NVIDIA Device Plugin

Question 15

A healthcare company is deploying NVIDIA AI Enterprise on a Kubernetes cluster to run medical imaging AI models. The cluster administrator needs to verify that the NVIDIA GPU Operator is installed and functioning correctly. Which command should the administrator use to check the status of the GPU Operator pods?

Accepted Answer

kubectl get pods -n gpu-operator. The NVIDIA GPU Operator deploys its components into the gpu-operator namespace. To verify that the Operator is installed and functioning, the administrator should list the pods in that namespace. This provides a quick overview of all related pods, such as the driver, container toolkit, and device plugin, and their current status.

Answer

kubectl describe daemonset nvidia-device-plugin -n kube-system

Answer

kubectl logs -n gpu-operator nvidia-driver-daemonset

Answer

kubectl get nodes -o wide