NCP-AIO Administration Practice Test 1 — 15 Questions

Question 1

Refer to the exhibit. An administrator encounters this error when attempting to run 'nvidia-smi' inside a privileged container. What is the most likely cause for this failure?

Accepted Answer

The container lacks the CAP_SYS_ADMIN capability. This error indicates that the container process lacks the necessary Linux capabilities to interface with the NVIDIA kernel driver. Even in privileged mode, if the host device nodes are not properly mapped or if the NVIDIA driver is not correctly exposed to the container namespace, communication via the NVIDIA Management Library (NVML) fails. Resolving this requires verifying device injection and ensuring the driver versions match the host kernel environment.

Answer

The NVIDIA container toolkit is missing from the container image

Answer

The NVIDIA driver version is incompatible with the CUDA runtime

Answer

The GPU is currently in persistence mode

Question 2

An administrator is managing a cluster where training jobs are failing due to Out-of-Memory (OOM) errors. What is the most effective approach to troubleshoot the memory distribution across multiple GPUs?

Accepted Answer

Use NVIDIA DCGM to monitor memory utilization per process. Using NVIDIA DCGM (Data Center GPU Manager) allows for fine-grained monitoring of memory allocation across the entire cluster. By analyzing memory usage trends over time, administrators can identify if a specific job has a memory leak or if the model footprint exceeds available hardware limits. This visibility is key to optimizing job placement and resource allocation, ensuring that large-scale training jobs are efficiently distributed across the available fleet without exceeding physical memory capacity.

Answer

Increase the virtual memory limit on the host kernel

Answer

Disable the CUDA memory allocator in the training script

Answer

Restart the NVIDIA Persistence Daemon on the training node

Question 3

Which THREE actions are essential when performing a clean upgrade of NVIDIA drivers in a production AI environment to minimize downtime?

Accepted Answer

Stop all containers and services relying on the NVIDIA runtime. Driver upgrades in production environments require careful orchestration to prevent system instability. First, stopping all dependent services ensures no open handles remain. Second, removing the current driver prevents binary conflicts. Finally, verifying the new installation ensures the kernel modules are loaded correctly. These steps, when combined with a controlled reboot, ensure that the transition to the new driver stack is reliable and that all GPU resources remain available for production workloads after the upgrade process.

Answer

Immediately run the installer without unloading existing modules

Answer

Automatically reboot the host after every driver update

Question 4

An administrator needs to implement a policy where only specific authorized users can access GPU resources on a shared cluster. What is the recommended approach?

Accepted Answer

Use Kubernetes RBAC to control access to GPU-capable namespaces. Using Kubernetes RBAC (Role-Based Access Control) in conjunction with NVIDIA Device Plugins is the industry-standard way to manage access to hardware accelerators. By defining cluster roles that permit access to specific GPU-enabled pods, administrators can control who has the ability to schedule jobs on GPU nodes. This approach provides a robust security layer that integrates with existing authentication providers, ensuring that GPU resources are allocated only to authorized users in a multi-tenant environment.

Answer

Restrict access via file permissions on /dev/nvidia0

Answer

Create individual user accounts on the host GPU servers

Answer

Hard-code the user ID in the Dockerfile of the training image

Question 5

An administrator needs to optimize GPU utilization across a multi-tenant NVIDIA DGX cluster. Which action ensures equitable resource allocation while preventing a single container from saturating the NVLink fabric?

Accepted Answer

Configure NVIDIA Multi-Instance GPU (MIG) profiles.. Implementing NVIDIA Multi-Instance GPU (MIG) allows the administrator to partition a single A100 or H100 GPU into isolated instances. By assigning specific compute and memory slices to individual workloads, the administrator ensures predictable performance and prevents resource contention. This practice is essential in multi-tenant environments where shared infrastructure requires strict isolation and quality-of-service guarantees to maintain high-efficiency throughput across heterogeneous AI workloads.

Answer

Enable exclusive process mode on all GPUs globally.

Answer

Increase the oversubscription ratio in the cluster scheduler.

Answer

Disable ECC memory to boost clock speeds.

Question 6

Which TWO steps are required to ensure the NVIDIA Container Toolkit correctly exposes GPUs to a Docker container on a Linux host?

Accepted Answer

Set the default runtime to 'nvidia' in /etc/docker/daemon.json.. The NVIDIA Container Toolkit is essential for mapping host GPU resources into containerized environments. By configuring the default runtime and ensuring the driver matches the container base image, the administrator guarantees that CUDA applications can access the underlying hardware. These steps are fundamental for AI operations, as mismatched drivers or missing runtimes result in 'driver not found' errors during container startup, stalling model development workflows.

Answer

Install the NVIDIA driver on the container image.

Answer

Run 'nvidia-smi' inside the container at startup.

Answer

Enable GPU passthrough in the host BIOS.

Question 7

An administrator is monitoring GPU memory bandwidth utilization. Which tool provides real-time, per-process visibility into GPU memory usage for troubleshooting memory-bound AI workloads?

Accepted Answer

NVIDIA DCGM (Data Center GPU Manager). DCGM (Data Center GPU Manager) provides deep telemetry into GPU performance, including granular memory bandwidth metrics and per-process memory consumption. This level of detail is vital for AI operations because memory-bound workloads often cause bottlenecks that simple utilization metrics fail to capture. Using DCGM allows for data-driven optimization of model architectures, ensuring that the GPU memory hierarchy is utilized efficiently during large-scale training and inference cycles.

Answer

nvidia-smi (System Management Interface)

Answer

top or htop command

Answer

dmesg kernel logs

Question 8

When deploying NVIDIA AI Enterprise software, which component serves as the core base for managing and monitoring the infrastructure across a heterogeneous cluster?

Accepted Answer

NVIDIA GPU Operator. NVIDIA Base Command or similar management suites are designed to orchestrate AI workloads across a cluster. Understanding the management layer is crucial for administrators to maintain cluster health, schedule jobs efficiently, and ensure that AI models have access to the necessary compute resources. Proper infrastructure management prevents configuration drift and allows for standardized deployment workflows across the entire enterprise AI environment.

Answer

NVIDIA CUDA Toolkit

Answer

NVIDIA TensorRT

Answer

NVIDIA cuDNN

Question 9

An administrator observes high 'ECC error' counts in the system logs for a specific GPU. What is the most appropriate next step for administrative action?

Accepted Answer

Schedule the node for hardware inspection and potential replacement.. Monitoring and responding to ECC errors is critical for maintaining data integrity in AI training. ECC (Error Correction Code) memory detects and fixes single-bit flips, but a spike in errors indicates potential hardware degradation. By logging these and proactively replacing the hardware, administrators prevent silent data corruption, which is expensive to detect and recover from in long-running training jobs, ensuring the reliability of the entire cluster's output.

Answer

Ignore the logs if the training job does not crash.

Answer

Lower the GPU clock frequency via software.

Answer

Reboot the server to reset the error registers.

Question 10

An administrator is troubleshooting a GPU-accelerated pod that fails to start with the error 'no NVIDIA devices found'. The GPU Operator is installed and the node has GPUs. Which action should the administrator take first to diagnose the issue?

Accepted Answer

Check the logs of the NVIDIA device plugin pod. The device plugin is responsible for discovering and advertising GPUs to Kubernetes. If it fails, pods cannot request GPUs, resulting in 'no NVIDIA devices found'. Checking its logs provides direct insight into discovery failures, such as missing device files or permission errors. Other actions like restarting kubelet or reinstalling the operator are remediation steps that should follow diagnosis.

Answer

Restart the kubelet service on the node

Answer

Run nvidia-smi on the host to verify GPU visibility

Answer

Reinstall the NVIDIA GPU Operator

Question 11

An administrator is managing a Kubernetes cluster with NVIDIA GPU Operator installed. A new node with NVIDIA GPUs is added, but pods requesting GPUs remain in Pending state. The administrator runs 'kubectl describe node <node-name>' and sees that the node has no 'nvidia.com/gpu' resource. Which component of the GPU Operator is most likely failing to label the node with GPU resources?

Accepted Answer

NVIDIA GPU Feature Discovery. In a Kubernetes cluster with NVIDIA GPU Operator, GPU Feature Discovery (GFD) runs as a DaemonSet that detects GPU hardware and applies labels to each node. These labels include the GPU product name, count, and other attributes. The NVIDIA Device Plugin then uses these labels to register the 'nvidia.com/gpu' resource with the kubelet. If GFD is not functioning, the node lacks the labels, and the device plugin cannot advertise GPU resources, leading to pods stuck in Pending state when requesting GPUs.

Answer

NVIDIA Container Toolkit

Answer

NVIDIA Device Plugin

Answer

NVIDIA Driver DaemonSet

Question 12

An administrator is troubleshooting a multi-node NVIDIA AI training job that uses NCCL for inter-GPU communication. The job runs on a cluster with 8 GPUs per node connected via NVLink, and nodes connected via InfiniBand. The administrator observes that NCCL is not utilizing the InfiniBand fabric, and communication falls back to Ethernet. Which action should they take to ensure NCCL uses InfiniBand?

Accepted Answer

Set the NCCL_IB_DISABLE environment variable to 0.. NCCL_IB_DISABLE set to 0 enables InfiniBand usage. When set to 1, NCCL disables InfiniBand and falls back to other transports like Ethernet. The administrator should ensure this variable is not set to 1 and is explicitly set to 0 to enable InfiniBand. Other options do not enable InfiniBand.

Answer

Set the NCCL_DEBUG environment variable to INFO to force InfiniBand usage.

Answer

Install and configure the NVIDIA Fabric Manager on all nodes.

Answer

Set the NCCL_SOCKET_IFNAME environment variable to the Ethernet interface.

Question 13

An AI operations engineer needs to isolate GPU-accelerated workloads on a Kubernetes node so that only pods from a specific namespace can consume GPU resources. The cluster uses the NVIDIA GPU Operator and MIG is enabled on the A100 GPUs. Which Kubernetes resource should be configured to enforce this isolation?

Accepted Answer

Admission controller (e.g., OPA Gatekeeper) that validates pod GPU requests against namespace labels.. Namespace-level GPU isolation requires a policy engine that can evaluate pod metadata and enforce rules before scheduling. An admission controller like OPA Gatekeeper can inspect the namespace of a pod and deny GPU resource requests from unauthorized namespaces. This ensures that only pods in the designated namespace can consume GPU resources, regardless of other configurations like quotas or taints. It is the most direct and enforceable method for the stated requirement.

Answer

ResourceQuota on the namespace to limit nvidia.com/gpu requests.

Answer

Node taints and tolerations on the GPU node, with tolerations only in the target namespace's pods.

Answer

MIG (Multi-Instance GPU) configuration on the GPU Operator to create separate GPU instances per namespace.

Question 14

An administrator is deploying a multi-node NVIDIA Base Command Platform cluster. They need to ensure that the cluster's shared storage is properly configured for AI training workloads. Which storage solution is specifically optimized and recommended by NVIDIA for Base Command Platform to provide high-throughput, low-latency access for GPU-accelerated workloads?

Accepted Answer

NVIDIA Magnum IO GPUDirect Storage. NVIDIA Base Command Platform is designed to streamline AI workload management, and storage performance is critical for training jobs that frequently read large datasets. NVIDIA Magnum IO GPUDirect Storage provides a direct data path between GPU memory and storage devices, such as NVMe or network-attached storage, eliminating CPU bottlenecks. This results in higher throughput and lower latency, which are essential for keeping GPUs fully utilized during training.

Answer

NFS over TCP

Answer

Ceph RADOS Block Device

Answer

iSCSI block storage

Question 15

A platform engineer manages an NVIDIA AI Enterprise cluster with multiple DGX nodes. The security team requires that GPU telemetry and management traffic be isolated from tenant workload traffic. The cluster uses NVIDIA Base Command Manager for provisioning and monitoring. Which configuration should the administrator implement to meet this requirement while preserving full remote management and telemetry collection?

Accepted Answer

Configure a dedicated management network for BMC and telemetry, bind the NVIDIA Data Center GPU Manager (DCGM) exporter to the management interface, and restrict tenant workloads to a separate workload network with firewall rules.. Isolating management and telemetry traffic requires network segmentation, not just encryption or GPU partitioning. A dedicated management network for BMC and telemetry, with DCGM exporter bound to that interface and firewall rules separating tenant workloads, ensures that management traffic cannot be reached from tenant networks while preserving remote management and metrics collection.

Answer

Deploy a service mesh across all nodes to encrypt traffic between workloads, and rely on mutual TLS to prevent tenants from accessing management endpoints.

Answer

Enable a dedicated out-of-band management VLAN on the BMC interfaces and a separate in-band VLAN for GPU telemetry, then configure Base Command Manager to use the in-band VLAN for metrics collection.

Answer

Use NVIDIA Multi-Instance GPU (MIG) to partition each GPU and assign separate MIG instances to management and tenant workloads, ensuring physical isolation of telemetry paths.