NCP-AIO Practice Test 5 — 25 Questions

Question 1

A platform engineer is preparing a Kubernetes cluster to run AI training jobs that require GPU access. The cluster nodes have NVIDIA GPUs, and the engineer wants the GPU Operator to manage the driver lifecycle. Which component must be installed on the host nodes to allow the GPU Operator to load kernel modules and create device nodes?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

NVIDIA GPU Operator Driver Container. The GPU Operator uses a driver container to manage the NVIDIA driver on host nodes. This container loads kernel modules, creates device nodes, and ensures the driver version matches the operator's requirements. The NVIDIA Container Toolkit and Device Plugin are also deployed by the operator but serve different purposes: the toolkit enables container GPU access, and the device plugin advertises GPUs to Kubernetes. The driver container is specifically responsible for the driver lifecycle, making it the correct component.

Answer

NVIDIA Kubernetes Device Plugin

Answer

NVIDIA Container Toolkit

Answer

NVIDIA GPU Driver

Question 2

A financial services company is deploying NVIDIA AI Enterprise on a Kubernetes cluster with strict security policies. They need to ensure that GPU workloads are isolated and that the NVIDIA GPU Operator components are deployed with least privilege. Which feature of the NVIDIA GPU Operator allows administrators to define granular permissions for its components?

Accepted Answer

Role-Based Access Control (RBAC). RBAC is the Kubernetes mechanism for defining granular permissions. The NVIDIA GPU Operator deploys components with specific ServiceAccounts and RBAC roles that grant only the permissions needed to perform their functions. This aligns with least-privilege principles and is critical for security-sensitive deployments. Other options are either deprecated, network-focused, or platform-specific.

Answer

Security Context Constraints

Answer

PodSecurityPolicy

Answer

Network Policies

Question 3

A company is deploying NVIDIA AI Enterprise on a vSphere cluster with multiple ESXi hosts, each containing NVIDIA A100 GPUs. They need to enable vGPU functionality for virtual machines running AI workloads. Which configuration step is required on the ESXi host to allow vGPU assignment to VMs?

Accepted Answer

Install the NVIDIA vGPU Manager on the ESXi host and set the graphics type to shared or direct.. Enabling vGPU on ESXi requires the NVIDIA vGPU Manager VIB and setting the GPU graphics type appropriately. This allows the hypervisor to present virtual GPUs to VMs. Alternatives like the Container Toolkit or GPU Operator are not applicable to ESXi host-level configuration, and SR-IOV is not the mechanism used for vGPU in this context.

Answer

Install the NVIDIA GPU Operator on the ESXi host and create a custom resource for vGPU.

Answer

Enable SR-IOV on the ESXi host and configure virtual functions for each VM.

Answer

Install the NVIDIA Container Toolkit on the ESXi host and configure the Docker daemon.

Question 4

An AI operations engineer is preparing a Kubernetes cluster to run GPU-accelerated inference workloads using the NVIDIA GPU Operator. The cluster nodes already have NVIDIA data center GPUs installed, and the engineer wants to avoid installing the driver manually on each node. Which component of the GPU Operator is responsible for automatically deploying the NVIDIA driver on worker nodes?

Accepted Answer

NVIDIA GPU Operator's driver container. The driver container within the GPU Operator automates the deployment and management of NVIDIA drivers on Kubernetes nodes. It runs as a DaemonSet and ensures the correct driver version is loaded without manual intervention. This simplifies operations and maintains consistency across the cluster, which is essential for AI workloads that depend on specific driver capabilities.

Answer

NVIDIA Device Plugin

Answer

NVIDIA DCGM Exporter

Answer

NVIDIA Container Toolkit

Question 5

A platform engineer is preparing a bare-metal Ubuntu 22.04 server with four A100 GPUs for an NVIDIA AI Enterprise deployment. The GPUs are not yet visible to the operating system tooling. Which command should the engineer run to confirm the driver loaded successfully and that all four GPUs are enumerated with their current driver version?

Accepted Answer

nvidia-smi. The NVIDIA kernel driver is the foundation of every AI Enterprise workload, and nvidia-smi is the standard utility that both triggers a driver query and renders the full GPU inventory with driver and CUDA versions. Administrative toolkits such as DCGM or the container toolkit sit above the driver and only report meaningful data once the driver has bound successfully to all adapters.

Answer

dcgmi discovery -l

Answer

nvidia-container-cli --info

Answer

lspci -d 10de:

Question 6

A platform engineer is preparing a bare-metal Kubernetes cluster to run GPU-accelerated AI workloads using the NVIDIA GPU Operator. The cluster nodes have NVIDIA Ampere GPUs and run Ubuntu 22.04 with containerd as the container runtime. The engineer wants to avoid installing any NVIDIA drivers or CUDA components directly on the host. Which GPU Operator configuration should be used to achieve this?

Accepted Answer

Deploy the GPU Operator with the default configuration, which includes the driver container.. The NVIDIA GPU Operator is designed to manage the full stack of GPU software, including the driver, by running a driver container on each node. In a default deployment, the driver container builds and loads the kernel modules, so no manual host driver installation is needed. This aligns with the requirement to keep the host clean and let the Operator handle everything.

Answer

Set driver.enabled=false and rely on pre-installed host drivers.

Answer

Use the operator's 'driver' Helm chart with 'driver.enabled=true' but set 'driver.usePrecompiled=true'.

Answer

Install the NVIDIA Container Toolkit manually and disable the operator's driver management.

Question 7

A platform engineer is preparing an Ubuntu 22.04 server that will host GPU-accelerated inference containers managed by containerd (not Docker). The team wants the NVIDIA Container Toolkit to expose GPUs to those containers. After installing the toolkit packages, which action must the engineer take so that containerd actually invokes the NVIDIA runtime for GPU workloads?

Accepted Answer

Run nvidia-ctk runtime configure --runtime=containerd and restart the containerd service.. Registering the NVIDIA runtime with the container engine is the essential post-install step. The nvidia-ctk runtime configure command writes the runtime entry and default-runtime setting into containerd's config.toml, and containerd must be restarted to apply it. Merely installing packages or setting container environment variables does not change which runtime containerd uses, so GPU devices remain invisible to containers until the engine configuration is updated and reloaded.

Answer

Install the nvidia-container-runtime package and symlink it as /usr/bin/runc on the host.

Answer

Add the user to the video group and grant read-write access to /dev/nvidiactl.

Answer

Set the environment variable NVIDIA_VISIBLE_DEVICES=all in the host shell profile and reboot the node.

Question 8

An administrator is preparing to install the NVIDIA GPU Operator on a new Kubernetes cluster. The cluster uses containerd as the container runtime. Which prerequisite must be satisfied on each GPU node before the operator can successfully deploy the driver container?

Accepted Answer

The Linux kernel headers for the running kernel must be available on the node.. The driver container builds NVIDIA kernel modules on the host, which requires kernel headers and a compiler toolchain matching the running kernel. Without these, the container cannot compile the modules and the driver fails to load. This prerequisite is essential for any node where the operator manages the driver, and it is often overlooked during cluster preparation.

Answer

The node must have the `nvidia.com/gpu.present` label applied manually by the administrator.

Answer

The NVIDIA Container Toolkit must be installed manually on each node.

Answer

The node must have the NVIDIA driver preinstalled at the exact version specified by the operator.

Question 9

An AI infrastructure team is deploying NVIDIA AI Enterprise on a Kubernetes cluster using the NVIDIA GPU Operator. They need to ensure that the GPU Operator can successfully manage GPUs and that workloads can consume GPU resources. Which two components does the GPU Operator deploy to enable GPU scheduling and container GPU access? (Choose two.)

Accepted Answer

NVIDIA Device Plugin. The NVIDIA GPU Operator deploys the NVIDIA Device Plugin to enable Kubernetes to schedule pods with GPU resources, and the NVIDIA Container Toolkit to allow containers to access GPUs. The device plugin registers GPUs with the Kubernetes API, while the container toolkit configures the container runtime. Together, they provide the necessary integration for GPU-accelerated workloads. Other components like the driver and MIG manager are important but not the specific enablers for scheduling and container access.

Answer

NVIDIA Persistence Daemon

Answer

NVIDIA MIG Manager

Answer

NVIDIA GPU Driver

Question 10

An AI operations team runs mixed training and inference workloads on a Kubernetes cluster managed with the NVIDIA GPU Operator. Inference pods frequently arrive in bursts and must start within seconds, while long-running training jobs occupy most MIG-capable A100 GPUs for days. Administrators want burst inference pods to obtain GPU capacity immediately without preempting or restarting the training jobs, and they want the cluster to reclaim those resources automatically when the burst ends. Which approach best satisfies these requirements?

Accepted Answer

Create a node pool reserved for inference, and configure a cluster autoscaler with scale-to-zero so burst pods trigger new GPU nodes that are removed when idle.. The scenario requires two properties simultaneously: burst inference capacity that appears without disturbing running training jobs, and automatic reclamation when the burst ends. Only a dedicated autoscaled node pool satisfies both, because new GPU nodes are provisioned on demand for inference pods while training pods remain scheduled and running on their existing nodes. When demand drops, scale-to-zero removes the extra nodes, freeing GPU resources without any preemption or job restarts.

Answer

Configure the inference Deployment with a topologySpreadConstraint across all GPU nodes so its replicas distribute evenly and find free capacity.

Answer

Enable the GPU Operator's time-slicing configuration so inference and training containers share the same physical GPUs through a shared device plugin.

Answer

Deploy inference pods with a higher PriorityClass and enable preemption so the scheduler evicts lower-priority training pods when capacity is exhausted.

Question 11

An AI operations team is deploying the NVIDIA GPU Operator on a Kubernetes cluster that uses containerd as the container runtime. The cluster nodes have NVIDIA GPUs, and the team wants to ensure that GPU workloads can request GPU resources. After installing the operator, they notice that pods requesting 'nvidia.com/gpu' remain in Pending state. Which component of the GPU Operator is most likely misconfigured or missing?

Accepted Answer

The NVIDIA device plugin, which advertises GPU resources to the Kubernetes API server.. The NVIDIA device plugin is a DaemonSet that runs on each node and registers GPUs as extended resources. When it is missing or misconfigured, the Kubernetes API server has no knowledge of available GPUs, so any pod requesting 'nvidia.com/gpu' cannot be scheduled. Ensuring the device plugin is healthy and running is essential for GPU scheduling.

Answer

The Kubernetes scheduler configuration, which may not be aware of GPU resources.

Answer

The NVIDIA Container Toolkit, which enables containers to access GPUs.

Answer

The GPU Operator's driver container, which loads the NVIDIA kernel modules.

Question 12

A system administrator is installing the NVIDIA Container Toolkit on a standalone Ubuntu server to run GPU-accelerated containers. After installation, they want to verify that the toolkit is correctly configured. Which command should they run to test GPU access from a container?

Accepted Answer

docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi. To verify that the NVIDIA Container Toolkit is correctly configured, the administrator should run a container with GPU access and execute nvidia-smi inside it. The command docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi uses the --gpus all flag to expose all GPUs, and the nvidia-smi output confirms that the container can see and use the GPUs. This tests the entire stack from Docker to the toolkit. Other commands only check installation or host-level GPU status.

Answer

nvidia-smi

Answer

systemctl status nvidia-container-runtime

Answer

nvidia-container-cli --version

Question 13

A DevOps engineer is deploying the NVIDIA GPU Operator on a Kubernetes cluster that uses containerd as the container runtime. The engineer notices that the Operator's validation pod fails with an error indicating that the NVIDIA container runtime is not configured. Which action should the engineer take to resolve this?

Accepted Answer

Ensure that the GPU Operator's container-toolkit daemonset is enabled and has the necessary permissions to modify the container runtime configuration.. The GPU Operator uses a container-toolkit daemonset to install and configure the NVIDIA Container Toolkit on each node, including modifying the containerd configuration. If this daemonset is disabled or lacks privileges, the runtime won't be set up, causing validation failures. Ensuring the daemonset is enabled and has proper permissions resolves the issue.

Answer

Manually install the NVIDIA Container Toolkit on all nodes and set the default runtime to nvidia.

Answer

Switch the cluster to use Docker as the container runtime, as the GPU Operator only supports Docker.

Answer

Disable the validation pod in the GPU Operator's Helm chart to suppress the error.

Question 14

An AI operations engineer is deploying a multi-node Kubernetes cluster with NVIDIA A100 GPUs for distributed training. The engineer wants to ensure that GPUs are correctly discovered and that workloads can request GPU resources. After installing the NVIDIA GPU Operator, the engineer notices that the GPU nodes are not advertising any 'nvidia.com/gpu' resources. Which component should the engineer verify first to resolve this issue?

Accepted Answer

NVIDIA Device Plugin. When GPU resources are not advertised, the NVIDIA Device Plugin is the primary component to investigate. It runs as a DaemonSet on GPU nodes and registers GPUs with the kubelet. Issues such as pod failures, missing driver libraries, or misconfigured kubelet settings can prevent resource advertisement. Checking its logs and status will reveal the root cause.

Answer

NVIDIA DCGM Exporter

Answer

NVIDIA GPU Operator's driver container

Answer

NVIDIA Container Toolkit

Question 15

An administrator is deploying the NVIDIA GPU Operator into an existing Kubernetes cluster where the NVIDIA driver is already installed and maintained by the node image. The team wants the Operator to manage only the container runtime, device plugin, and monitoring components. Which configuration should be applied to the GPU Operator's ClusterPolicy?

Accepted Answer

Set driver.enabled to false so the Operator does not deploy the driver container.. To use the GPU Operator with drivers already managed by the node image, the ClusterPolicy must disable driver management so the Operator skips the driver container. The container toolkit, device plugin, and DCGM exporter remain enabled so the Operator still configures runtime integration, advertises GPU resources, and exposes telemetry as the team intends.

Answer

Set devicePlugin.enabled to false so the Operator does not advertise GPU resources.

Answer

Set dcgmExporter.enabled to false so the Operator does not conflict with the existing driver.

Answer

Set toolkit.enabled to false so the Operator does not modify the container runtime configuration.

Question 16

An AI infrastructure team is preparing an air-gapped data center to install the NVIDIA GPU Operator. They have mirrored all required container images into a private registry. During installation, the Operator's pods fail with ImagePullBackOff because the components still reference images under nvcr.io. Which configuration is required to make the Operator and its managed components pull from the private registry?

Accepted Answer

Set the chart values for the image repository and tag for each component, and provide an image pull secret in the Operator namespace.. Air-gapped installs require pre-mirroring every image and rewriting all image references in the Helm values to the private registry, including the Operator and each managed component. Authentication to the private registry is supplied through an image pull secret placed in the Operator namespace so the kubelet can pull. DNS tricks, pull policy changes, or imagined auto-mirroring do not redirect image references and leave the components failing.

Answer

Enable the Operator's auto-mirroring feature so it copies images from nvcr.io at runtime.

Answer

Configure a DNS override so nvcr.io resolves to the private registry's IP address.

Answer

Add nvcr.io to the cluster's imagePullPolicy as Never.

Question 17

A platform team runs an on-premises Kubernetes cluster for AI inference. Several teams submit pods that request the same GPU device, and the scheduler places more pods onto a node than there are available GPUs, causing OOM errors on the device. The administrator wants the Kubernetes scheduler itself to prevent overcommitting GPUs without any custom admission controller. Which action should the administrator take?

Accepted Answer

Install the NVIDIA device plugin so GPUs are advertised as schedulable extended resources and add a matching resource request to each pod spec.. Advertising GPUs as extended resources through the NVIDIA device plugin is what lets the default scheduler treat each device as countable, finite node capacity. Pods that declare a GPU request are then placed only where an unallocated device exists, and no custom admission controller is required because the scheduler performs the arithmetic itself. Quotas, taints, and autoscalers influence eligibility or CPU and memory sizing, not device-level accounting.

Answer

Set a node taint on each GPU node and add a toleration to pods, so only pods that explicitly opt in are placed there.

Answer

Enable the Kubernetes Vertical Pod Autoscaler in recommendation mode on the GPU namespaces to detect device pressure and reschedule pods.

Answer

Create a ResourceQuota in each team namespace limiting the count of pods that may run, so fewer pods land on GPU nodes.

Question 18

An AI operations team is installing the NVIDIA GPU Operator on a Kubernetes cluster that uses a custom containerd configuration. They need to ensure that the GPU Operator can properly manage the container runtime. Which action should they take before installing the GPU Operator?

Accepted Answer

Ensure that the containerd configuration does not already include conflicting NVIDIA runtime settings.. Before installing the GPU Operator, any pre-existing NVIDIA runtime configuration in containerd should be removed to avoid conflicts. The operator manages the runtime configuration itself, so manual settings can interfere. Other options like changing the default runtime or disabling containerd are incorrect because the operator integrates with the existing runtime rather than replacing or requiring manual overrides.

Answer

Label the nodes with the appropriate container runtime version.

Answer

Set the default runtime in containerd to nvidia-container-runtime.

Answer

Disable the containerd systemd service and let the GPU Operator start its own runtime.

Question 19

An AI operations engineer manages a shared Kubernetes cluster running NVIDIA GPU Operator. Several teams report that their inference pods remain in a Pending state with the event '0/8 nodes are available: 8 Insufficient nvidia.com/gpu.' The administrator verifies that nvidia-smi on all nodes shows idle GPUs. Which action should the administrator take first to resolve the scheduling failure?

Accepted Answer

Inspect the node allocatable GPU count and any taints or labels that prevent scheduling on the idle nodes.. The Pending event shows the scheduler believes no node has a free nvidia.com/gpu resource, even though nvidia-smi reports idle devices. That gap is almost always caused by the device plugin not advertising GPUs, or by taints, labels, or affinity rules excluding the nodes. Inspecting allocatable counts and scheduling constraints identifies the actual blocker before any disruptive remediation.

Answer

Reinstall the NVIDIA GPU Operator and restart the kubelet on all worker nodes.

Answer

Add a nodeSelector for kubernetes.io/os=linux to the pod template.

Answer

Increase the GPU memory limit in the pod specification so the scheduler can fit the workload.

Question 20

An AI operations engineer manages a Kubernetes cluster where the NVIDIA GPU Operator's device plugin exposes GPUs as schedulable resources. A data science team submits a batch inference job that requests one GPU but does not specify a node selector or tolerations. The job stays in Pending while other GPU nodes remain idle because they carry a taint the GPU Operator applied to reserve them for a specific workload class. Which approach is the most appropriate for the engineer to make the job schedulable without disrupting the reserved nodes?

Accepted Answer

Add an appropriate toleration and node selector to the job so it can target the reserved GPU nodes.. A node taint repels pods unless they carry a matching toleration. The reserved GPU nodes are intentionally tainted, so a job that neither tolerates the taint nor selects an untainted node cannot schedule. Adding the correct toleration plus a node selector places the job on the reserved nodes while preserving the reservation for other classes of work.

Answer

Delete and recreate the device plugin daemonset so GPUs are re-advertised to the scheduler.

Answer

Increase the cluster's GPU resource quota in the namespace so the scheduler can allocate a card.

Answer

Remove the node taint from all GPU nodes so the job can be scheduled anywhere.