NCP-AIO Practice Exam 3 — 20 Questions

Question 1

When deploying NVIDIA containers using the NVIDIA Container Toolkit, what is the primary function of the 'nvidia-container-runtime'?

Accepted Answer

It exposes the host's NVIDIA GPUs and driver libraries to the container environment.. The nvidia-container-runtime is a crucial component that allows Docker containers to interface with host GPUs. By modifying the container runtime specification, it ensures that the necessary device nodes and NVIDIA driver libraries are injected into the container namespace at startup. This enables seamless hardware acceleration for AI applications without requiring users to manually manage drivers or complex device path configurations inside their container images.

Answer

It automatically recompiles the application code for the specific GPU architecture found on the host.

Answer

It manages the lifecycle of the GPU driver installation on the host operating system.

Answer

It monitors the temperature and power consumption of the GPUs during container execution.

Question 2

An administrator is deploying NVIDIA AI Enterprise on a bare-metal cluster. Which component must be installed first to ensure proper communication between the Kubernetes scheduler and the underlying GPU hardware?

Accepted Answer

NVIDIA GPU Operator. The NVIDIA GPU Operator is essential for automating the management of all NVIDIA software components in Kubernetes. By installing it first, the administrator ensures that the device plugin, monitoring tools, and drivers are correctly configured. This foundation is critical for scheduling GPU-accelerated pods, as the Kubernetes scheduler requires the device plugin to advertise available GPU resources to the cluster's API server, enabling seamless workload orchestration across the infrastructure.

Answer

NVIDIA Triton Inference Server

Answer

NVIDIA NeMo Framework

Answer

NVIDIA Base Command Manager

Question 3

An operations engineer notices that an inference container on an A100 is intermittently returning stale predictions after a model update. The container mounts the model directory from a host path, and the update process replaces files in place. Which change most reliably prevents the stale predictions?

Accepted Answer

Use a versioned, immutable model artifact and restart or roll the inference container so it loads the new version atomically.. When model files are overwritten in place, a running inference process keeps using the weights it already loaded, so clients see stale results until the process restarts. Treating each model as an immutable, versioned artifact and performing a container restart or rolling deployment guarantees an atomic switch to a consistent set of weights. This removes the timing race between the update and the load, which is the actual source of the stale predictions.

Answer

Increase the container's shared memory size so the model can be cached entirely in RAM.

Answer

Reduce the inference batch size so each prediction reads the model from disk again.

Answer

Enable read-only mounts for the model directory to block writes from the container.

Question 4

Which component is strictly necessary for managing NVIDIA AI Enterprise licensing across a distributed cluster of nodes?

Accepted Answer

NVIDIA License System (NLS) Instance.. The NVIDIA License System (NLS) is the centralized authority for managing product entitlements. In a distributed environment, nodes must reach this server to validate their software keys. This component is crucial because it decouples the license management from the individual worker nodes, allowing for flexible scaling and centralized compliance reporting, which are essential in enterprise AI infrastructure where hardware resources are frequently provisioned and decommissioned.

Answer

NVIDIA CUDA Toolkit.

Answer

NVIDIA Container Toolkit.

Answer

NVIDIA Deep Learning GPU Training System (DIGITS).

Question 5

An administrator manages a Kubernetes cluster where a training job repeatedly fails with an OutOfMemory error on the GPU even though the pod requests one nvidia.com/gpu. DCGM metrics show that another pod on the same node is consuming GPU memory concurrently. GPU sharing via time-slicing is enabled cluster-wide. Which action should the administrator take to prevent this cross-pod interference while preserving the ability to share GPUs among trusted inference workloads?

Accepted Answer

Disable time-slicing on the node pool hosting training jobs and use MIG or dedicated GPUs for those workloads. Time-slicing multiplexes a GPU across pods without memory partitioning, so one workload can starve another and trigger OutOfMemory errors. The reliable fix is to separate untrusted or memory-intensive training workloads onto dedicated GPUs or MIG instances, while leaving time-slicing for inference workloads where memory contention is acceptable and controlled.

Answer

Apply a ResourceQuota limiting nvidia.com/gpu to one per namespace

Answer

Set the CUDA_VISIBLE_DEVICES environment variable manually in the training pod spec

Answer

Increase the pod's nvidia.com/gpu request to two GPUs

Question 6

A platform team runs an NVIDIA GPU Operator-managed Kubernetes cluster shared by two research groups. Group A's pods request `nvidia.com/gpu: 1` and are scheduled correctly, but Group B's pods that omit any GPU resource request are also landing on GPU nodes and consuming host memory and CPU, degrading Group A's jobs. The team wants Group B's non-GPU pods to stop consuming capacity on the GPU node pool without changing Group B's manifests. Which action should the administrator take?

Accepted Answer

Apply a taint such as `nvidia.com/gpu=present:NoSchedule` to the GPU nodes and add a matching toleration to Group A's pod templates.. Dedicating a node pool to GPU consumers is done at the scheduler layer: a taint on the GPU nodes repels every pod that does not carry a matching toleration. Because Group B's manifests cannot be changed, repelling rather than attracting is the only workable direction, and adding tolerations to Group A's templates is a one-time change under the platform team's control. Quotas, driver flags, and plugin options all operate after or outside scheduling and cannot keep non-GPU pods off those nodes.

Answer

Set `NVIDIA_DRIVER_CAPABILITIES=compute,utility` on the GPU Operator DaemonSet so non-GPU pods cannot attach to the device.

Answer

Create a `ResourceQuota` in each namespace that caps `requests.nvidia.com/gpu` at zero for Group B.

Answer

Enable the `nvidia` device plugin's `--fail-on-init-error=false` flag so unresolvable GPU requests fall back to CPU scheduling.

Question 7

A research lab is deploying NVIDIA AI Enterprise on an air-gapped Kubernetes cluster. The cluster has no internet access, and all software must be installed from a local registry. The administrator plans to use the NVIDIA GPU Operator. Which two actions must be performed to ensure a successful deployment in this environment? (Choose two.)

Accepted Answer

Configure the GPU Operator to use the private registry by setting the appropriate Helm chart values.. In an air-gapped environment, the GPU Operator cannot reach public registries to pull container images. Therefore, all necessary images must be mirrored to a private registry, and the GPU Operator must be configured to use that registry via Helm values. These two actions ensure that the Operator can deploy its components without internet access. Other options are either not required or invalid for this scenario.

Answer

Install the NVIDIA GPU Operator using the --offline flag.

Answer

Disable the GPU Operator's driver management and install the driver manually on each node.

Answer

Ensure the cluster nodes have direct access to the NVIDIA licensing server for vGPU.

Question 8

A research team is submitting many short-lived experiment jobs to an NVIDIA-accelerated Kubernetes cluster. The operations team wants to reduce GPU idle time and improve overall utilization without modifying the training code. Which TWO approaches should the operations team implement? (Choose two.)

Accepted Answer

Use a gang-scheduling or batch scheduler with backfill so queued jobs start as soon as GPUs free up. Short-lived experiment jobs create idle GPU gaps. Time-slicing allows multiple such pods to share a device, and a batch scheduler with backfill keeps the queue moving so GPUs are assigned as soon as capacity frees. Both measures raise utilization without touching training code, which is exactly what the operations team needs.

Answer

Pin each experiment job to a dedicated physical GPU using node affinity

Answer

Disable the NVIDIA device plugin and mount GPUs directly via hostPath

Answer

Increase the pod's nvidia.com/gpu request to reserve more GPU memory

Question 9

An AI operations team is using NVIDIA DCGM to monitor a cluster of GPUs. They want to set up alerts for when GPUs are running at high temperatures for extended periods. Which DCGM feature should they use?

Accepted Answer

DCGM health checks with custom thresholds and alerting via DCGM exporter to Prometheus.. DCGM health checks allow defining thresholds for GPU metrics, including temperature. When combined with the DCGM exporter and Prometheus, teams can create alerting rules that trigger on sustained high temperatures. This is the recommended method for cluster-wide monitoring. Other options either misuse DCGM diagnostics, assume non-existent features, or use less scalable methods.

Answer

DCGM's built-in email alerting system configured via dcgm.conf.

Answer

DCGM diagnostics with the '-r' flag to run a comprehensive test and report temperature violations.

Answer

NVIDIA-smi's '--query-gpu=temperature.gpu' with a cron job to check and send alerts.

Question 10

An administrator is deploying NVIDIA AI Enterprise on a Kubernetes cluster and wants to verify that the GPU Operator has successfully installed all required components. Which command should the administrator use to check the status of the GPU Operator pods?

Accepted Answer

kubectl get pods -n gpu-operator. The GPU Operator installs its components as pods in the gpu-operator namespace. Checking pod status with kubectl get pods -n gpu-operator is the correct way to verify that all required components are running. Other commands provide GPU-level or node-level information but do not show the status of the operator's pods.

Answer

nvidia-smi -q

Answer

kubectl get nodes -o wide

Answer

dcgmi discovery -l

Question 11

An AI operations team manages a shared Kubernetes cluster where a nightly batch training workload requests nvidia.com/gpu resources and occasionally consumes all GPU memory on a node, causing a co-located interactive notebook pod to fail with CUDA out-of-memory errors. The team wants the interactive notebook to be isolated from the batch workload's memory usage without adding new hardware. Which action best achieves this on supported data center GPUs?

Accepted Answer

Configure Multi-Instance GPU profiles so the notebook and batch workloads run on separate MIG instances with dedicated memory partitions on the same physical GPU.. The conflict is shared GPU memory, not scheduling order. MIG is the only listed mechanism that gives each workload a dedicated, hardware-isolated memory partition on the same physical device, letting the notebook and batch job coexist without the batch job starving the notebook's framebuffer, and it requires no extra hardware.

Answer

Enable time-slicing with a replica count of four so the notebook and batch workloads alternate on the same GPU in round-robin fashion.

Answer

Set a memory limit on the batch container using the standard Kubernetes resources.limits.memory field to cap its GPU framebuffer usage.

Answer

Add a higher PriorityClass to the notebook pod so the kube-scheduler preempts the batch workload whenever memory pressure occurs.

Question 12

A research team submits a multi-node training job using a `Job` with eight pods, each requesting one GPU. The cluster has eight GPU nodes, each with one A100. The administrator observes that all eight pods are spread one per node and the job runs, but throughput is far below expectations and NCCL logs show repeated fallback from GPUDirect RDMA to socket transport. Which action most directly addresses the root cause?

Accepted Answer

Deploy the NVIDIA Network Operator to install and configure the RDMA stack and GPUDirect RDMA on the GPU nodes.. The symptom points to NCCL abandoning RDMA and using the slower socket transport for inter-node collectives. That path depends on the high-speed fabric drivers, RDMA devices, and GPUDirect RDMA support being installed and configured on every GPU node, which is precisely what the NVIDIA Network Operator provisions in tandem with the GPU Operator. Changing rank counts, disabling peer-to-peer, or collapsing topology onto one node does not restore the missing RDMA capability.

Answer

Set `NCCL_P2P_DISABLE=1` in the job's environment to force peer-to-peer transfers over PCIe.

Answer

Add a node affinity rule forcing all eight pods onto a single node to enable NVLink communication.

Answer

Increase the number of replicas in the Job so more GPUs participate in the collective.

Question 13

In an NVIDIA-accelerated Kubernetes environment, why is it critical to configure a 'RuntimeClass' for GPU-enabled pods?

Accepted Answer

To ensure the container runtime correctly injects GPU drivers and libraries into the pod.. The RuntimeClass ensures that the container is started with the correct NVIDIA container runtime (e.g., nvidia-container-runtime). This runtime is responsible for performing the necessary hardware mapping, injecting the CUDA libraries, and setting the environment variables required for the container to actually utilize the GPU. Without this configuration, the pod might be scheduled successfully but will fail at runtime because it cannot communicate with the NVIDIA driver.

Answer

To allow the Kubernetes scheduler to place pods on nodes based on GPU availability.

Answer

To define the specific NVIDIA driver version required by the application.

Answer

To increase the priority of GPU workloads over standard CPU-only workloads.

Question 14

During a multi-GPU training job, you notice that one GPU consistently reports lower utilization and longer communication times compared to others. What is the most likely reason for this performance imbalance?

Accepted Answer

The GPU is connected via a slower interconnect.. In multi-GPU systems, uneven load distribution often stems from mismatched NVLink topologies or PCIe lane configurations. If one GPU is connected via a slower PCIe link rather than a high-speed NVLink interconnect, it becomes the bottleneck in collective communication operations like AllReduce. Ensuring symmetric connectivity across all GPUs is essential for predictable performance and preventing the 'straggler' effect in distributed deep learning training.

Answer

The GPU is defective and needs replacement.

Answer

The training framework is not using CUDA streams.

Answer

The ambient server temperature is too high.

Question 15

An administrator must run a batch inference job that requires exactly two NVIDIA GPUs on a Kubernetes cluster managed by the NVIDIA GPU Operator. Which pod specification field should be used to request those GPUs?

Accepted Answer

resources.limits with the key nvidia.com/gpu set to 2. GPUs exposed by the NVIDIA device plugin appear to Kubernetes as the extended resource nvidia.com/gpu. Extended resources must be declared in resources.limits, and setting that limit to 2 makes the scheduler reserve two GPUs on a suitable node and pass the devices into the container.

Answer

resources.requests with the key nvidia.com/gpu set to 2 and no limit

Answer

annotations with the key nvidia.com/gpu.count set to 2

Answer

nodeSelector with the key nvidia.com/gpu.count set to 2

Question 16

An AI operations engineer is preparing a Kubernetes cluster to run GPU-accelerated workloads using the NVIDIA GPU Operator. The cluster nodes already have NVIDIA data center GPUs installed and the NVIDIA driver is pre-installed on the host. The engineer wants to use the GPU Operator to manage the container toolkit, device plugin, and monitoring components but must avoid the Operator managing or upgrading the driver. Which configuration should be applied to the GPU Operator deployment?

Accepted Answer

Set the driver.enabled parameter to false in the GPU Operator's Helm chart values.. When the NVIDIA driver is already present on host nodes and should remain externally managed, the GPU Operator must be configured to not deploy its own driver container. The driver.enabled=false Helm value achieves this by skipping the driver daemonset while still deploying other components like the container toolkit, device plugin, and DCGM exporter. This preserves the existing driver and avoids conflicts.

Answer

Use the NVIDIA GPU Operator with the --set toolkit.enabled=false option to prevent driver installation.

Answer

Install the GPU Operator with the --set operator.driverVersion=latest flag to pin the driver version.

Answer

Deploy the GPU Operator but remove the nvidia-driver-daemonset after installation.

Question 17

A cloud architect is deploying NVIDIA AI Enterprise on a vSphere cluster with multiple ESXi hosts, each containing NVIDIA A100 GPUs. The architect must enable GPU virtualization using NVIDIA vGPU. Which two components are required to support vGPU on the ESXi hosts? (Choose two.)

Accepted Answer

NVIDIA vGPU guest driver installed in each VM. To enable NVIDIA vGPU on VMware ESXi, the host must have the NVIDIA vGPU Manager VIB installed, which allows the hypervisor to partition the physical GPU. Each VM that uses a vGPU must also have the NVIDIA vGPU guest driver installed. These two components are essential; other tools like the GPU Operator or Container Toolkit are not used on ESXi.

Answer

NVIDIA GPU Operator for Kubernetes

Answer

NVIDIA CUDA Toolkit installed on the ESXi host

Answer

NVIDIA Container Toolkit installed on the ESXi host

Question 18

Refer to the exhibit. An AI engineer observes that a model training job is running slower than expected. Based on the output, what is the primary cause of the performance degradation?

Accepted Answer

An administrator has set a software power limit that is lower than the GPU's maximum performance threshold.. The 'SW Power Cap' throttle reason indicates that the power limit is configured below the card's maximum design capacity, causing the GPU to downclock to P12 state to stay within the power envelope. This is crucial for AI operations because it signals that the hardware is being artificially throttled, necessitating a review of power policies to ensure optimal training performance in high-compute scenarios.

Answer

The GPU is overheating and the thermal management system has engaged to prevent hardware damage.

Answer

The GPU is currently idle and the driver has shifted the card to an energy-saving state.

Answer

The GPU is experiencing a PCIe bus error, forcing the system to reduce clock speeds to maintain stability.

Question 19

Which action must be performed after updating the NVIDIA driver on a Linux host to ensure that all active GPU containers recognize the new driver version?

Accepted Answer

Restart the containerized applications.. Containers utilize the host-level NVIDIA drivers injected via the container toolkit. When the host driver is updated, active containers will still be referencing the old driver libraries in their environment until they are restarted. Restarting the containers is the mandatory step to force them to bind against the newly installed libraries, ensuring compatibility and preventing runtime errors associated with outdated driver interfaces.

Answer

Rebuild all container images with the latest CUDA toolkit.

Answer

Upgrade the Docker daemon version.

Answer

Re-run the nvidia-container-toolkit installation script.

Question 20

What is the primary benefit of using an NVIDIA-Certified System for AI Enterprise deployments?

Accepted Answer

Validated compatibility and performance. NVIDIA-Certified Systems have passed rigorous testing to ensure optimal performance and compatibility with the NVIDIA AI Enterprise software stack. This certification minimizes the risk of hardware-software conflicts, performance bottlenecks, and stability issues during production. For AI Ops professionals, this translates to faster deployment times, lower maintenance costs, and high confidence in the reliability of the underlying infrastructure for mission-critical deep learning and inference workloads.

Answer

Guaranteed 100% network uptime

Answer

Automatic software license renewal

Answer

Free access to all AI models