Administration practice questions

The NVIDIA container toolkit is missing from the container image

Why it fails: The NVIDIA container toolkit is a host-level component that bridges the container runtime with the NVIDIA driver. Its absence would prevent the GPU from appearing inside the container entirely, but it would typically result in a 'device not found' error rather than a permissions-based NVML initialization failure.

The NVIDIA driver version is incompatible with the CUDA runtime

Why it fails: Driver and runtime compatibility issues usually manifest during the application execution phase when attempting to launch a CUDA kernel. These mismatches result in 'CUDA driver version is insufficient for CUDA runtime version' errors, whereas NVML errors are strictly related to driver management and low-level diagnostic communication.

The GPU is currently in persistence mode

Why it fails: Persistence mode is a driver setting designed to keep the GPU initialized even when no clients are connected. It facilitates faster start times and is considered a best practice for production AI environments, as it prevents the driver from unloading and reloading, which would cause significant latency spikes.

Increase the virtual memory limit on the host kernel

Why it fails: Increasing host swap or virtual memory does not resolve GPU-specific memory limits. OOM errors in AI training typically occur when the model parameters or activation buffers exceed the physical VRAM capacity of the GPU, which is independent of the host operating system's virtual memory management settings.

Disable the CUDA memory allocator in the training script

Why it fails: Disabling the CUDA memory allocator would force the system to request memory directly from the driver for every small allocation, significantly increasing fragmentation and performance overhead. This would not prevent OOM errors; rather, it would likely degrade training speed and lead to more frequent memory allocation failures.

Restart the NVIDIA Persistence Daemon on the training node

Why it fails: The persistence daemon is responsible for maintaining the GPU state when no applications are active. Restarting it may clean up stray device handles, but it does not address the fundamental issue of a model's memory footprint exceeding the available hardware capacity, which is the root cause of OOM errors.

Immediately run the installer without unloading existing modules

Why it fails: Attempting to install drivers while modules are in use often results in corruption or partial updates. The system may enter an inconsistent state where the new driver binary is present but the old kernel module is still active, causing kernel panics or failure to communicate with the hardware.

Automatically reboot the host after every driver update

Why it fails: While a reboot is often necessary, doing so automatically without verifying the installation can lead to downtime if the driver configuration is invalid. Administrators should verify the driver status using diagnostic tools before confirming that the system is safe to return to the active production pool.

Restrict access via file permissions on /dev/nvidia0

Why it fails: Manipulating file permissions on device nodes is volatile and easily bypassed by containers with elevated privileges. It is not a scalable or secure method for managing user access in a multi-tenant cluster environment where pods are dynamic and deployed frequently by various users across different development teams.

Create individual user accounts on the host GPU servers

Why it fails: Managing local user accounts on physical nodes is an outdated practice that does not scale in modern containerized environments. It ignores the abstraction layer provided by orchestrators like Kubernetes and creates significant administrative overhead, as individual credentials must be synchronized across every node in the entire training cluster.

Hard-code the user ID in the Dockerfile of the training image

Why it fails: Hard-coding user IDs is a security anti-pattern and does not provide actual access control. Any user capable of running the container image can override the entry point or user ID at runtime. It fails to address the requirement for cluster-wide, centralized authorization for accessing shared, high-value hardware resources.

Enable exclusive process mode on all GPUs globally.

Why it fails: Exclusive process mode restricts a GPU to a single compute context. While this provides isolation, it severely underutilizes the massive parallelism of modern NVIDIA GPUs and prevents multiple containerized workloads from sharing the same physical hardware, which contradicts the goal of maximizing cluster-wide resource utilization efficiency.

Increase the oversubscription ratio in the cluster scheduler.

Why it fails: Increasing oversubscription leads to resource contention and context switching overhead. This approach does not physically isolate hardware resources, meaning a single aggressive workload can still monopolize the NVLink fabric or memory bandwidth, ultimately degrading the performance of other concurrent tasks sharing the same physical GPU hardware.

Disable ECC memory to boost clock speeds.

Why it fails: Disabling Error Correction Code (ECC) memory might provide a negligible increase in raw performance, but it introduces significant risk of silent data corruption in large-scale AI training. Furthermore, it does nothing to address resource allocation or multi-tenancy requirements, making it an inappropriate solution for cluster management.

Install the NVIDIA driver on the container image.

Why it fails: The NVIDIA host driver must reside on the host system to interact with the kernel. Installing the driver inside the container image is not recommended and is unnecessary, as the container utilizes the host's driver through volume mounts and the NVIDIA runtime, maintaining image portability and size efficiency.

Run 'nvidia-smi' inside the container at startup.

Why it fails: While 'nvidia-smi' is useful for verification, it is not a configuration step for enabling GPU support. It is a diagnostic tool used to confirm functionality after the toolkit has been properly installed and the container runtime has been successfully configured to expose the device resources.

Enable GPU passthrough in the host BIOS.

Why it fails: GPU passthrough is primarily a concern for virtualized environments, such as KVM or VMware, to expose physical GPUs to guest VMs. For containerized applications running directly on the Linux host, GPU passthrough settings in the BIOS are not required, as the host kernel manages the devices directly.

nvidia-smi (System Management Interface)

Why it fails: While 'nvidia-smi' provides high-level memory statistics, it is a static snapshot tool and lacks the granular per-process performance counters and long-term telemetry features needed for deep analysis of memory-bound bottlenecks. It is suitable for basic health checks but insufficient for advanced performance tuning of complex AI pipelines.

top or htop command

Why it fails: Standard Linux utilities like 'top' and 'htop' monitor CPU and system RAM usage but remain oblivious to GPU internal memory architecture or device-specific bandwidth metrics. Using them to debug GPU-accelerated workloads will provide misleading data, as they cannot report on the high-bandwidth memory (HBM) performance of the GPU.

dmesg kernel logs

Why it fails: The 'dmesg' utility displays kernel ring buffer messages, which are useful for identifying driver crashes or hardware errors. It does not provide real-time performance telemetry or per-process memory metrics. Relying on 'dmesg' for bandwidth monitoring is impossible as it does not track active application-level performance data or GPU-specific memory metrics.

NVIDIA CUDA Toolkit

Why it fails: The CUDA Toolkit is a development environment for creating GPU-accelerated applications. It contains compilers, libraries, and debuggers. It is not an infrastructure management or monitoring platform for a cluster; it is a software development kit used by developers to write the code that eventually runs on the GPU hardware.

NVIDIA TensorRT

Why it fails: TensorRT is a high-performance deep learning inference optimizer and runtime. Its function is to optimize trained models for faster deployment on NVIDIA hardware. It has no role in managing cluster infrastructure, node-level configuration, or monitoring hardware health, as it operates at the model application level, not the system administration level.

NVIDIA cuDNN

Why it fails: cuDNN is a GPU-accelerated library of primitives for deep neural networks. It provides the building blocks for frameworks like PyTorch and TensorFlow. Like other libraries, it does not provide administrative tools for cluster management or system monitoring. It is strictly a compute library for executing mathematical operations on the GPU.

Ignore the logs if the training job does not crash.

Why it fails: Ignoring ECC error warnings is dangerous because ECC only corrects single-bit errors. If errors continue to increase, they eventually escalate into uncorrectable multi-bit errors, leading to silent data corruption or sudden job crashes. Proactive replacement is required to maintain the accuracy of trained models and the reliability of cluster operations.

Lower the GPU clock frequency via software.

Why it fails: Lowering GPU clock frequency may temporarily hide stability issues, but it does not fix the underlying physical degradation of the memory hardware. This approach is a temporary workaround that leaves the root cause unaddressed, making it a poor administrative practice that risks intermittent failures during future, possibly more critical, jobs.

Reboot the server to reset the error registers.

Why it fails: Rebooting the server only clears the transient error registers in software; it does not repair physical memory defects. The ECC error logs reflect actual hardware events occurring on the physical silicon. Resetting the registers will not stop the underlying hardware from producing errors, making this an ineffective response to persistent faults.

Restart the kubelet service on the node

Why it fails: Restarting kubelet may temporarily resolve some issues, but it is not a diagnostic step. It could disrupt other workloads and does not provide information about why GPUs are not detected. Without understanding the root cause, a restart may not fix the problem and could mask underlying issues. Diagnosis should precede remediation.

Run nvidia-smi on the host to verify GPU visibility

Why it fails: Running nvidia-smi on the host confirms that the driver and GPUs are functional at the host level. However, the error indicates that the pod cannot see the devices, which could be due to issues in the device plugin or container runtime configuration. While host-level verification is useful, it does not directly diagnose the Kubernetes integration problem. Checking device plugin logs is more targeted.

Reinstall the NVIDIA GPU Operator

Why it fails: Reinstalling the GPU Operator is a drastic measure that could cause downtime and may not address the root cause. It is not a diagnostic step and should be considered only after other troubleshooting fails. The error suggests a specific failure in device advertisement, so inspecting the device plugin logs is a more appropriate first action.

NVIDIA Container Toolkit

Why it fails: The NVIDIA Container Toolkit enables containers to access GPUs by configuring the container runtime, but it does not label nodes or advertise GPU resources to Kubernetes. It is a prerequisite for running GPU workloads, but its failure would typically manifest as container runtime errors, not as missing node resources or Pending pods.

NVIDIA Device Plugin

Why it fails: The NVIDIA Device Plugin is responsible for advertising GPU resources to the Kubernetes API server, but it relies on the node being labeled by the GPU Feature Discovery component first. Without proper labeling, the device plugin may not schedule correctly, but the root cause of missing GPU resource advertisement is typically the failure of the GPU Feature Discovery pod to label the node.

NVIDIA Driver DaemonSet

Why it fails: The NVIDIA Driver DaemonSet installs the GPU driver on nodes, but it does not directly label nodes with GPU resource information. If the driver fails to install, the GPU would be unusable, but the specific symptom of missing 'nvidia.com/gpu' resource is more directly tied to the labeling done by GPU Feature Discovery.

Set the NCCL_DEBUG environment variable to INFO to force InfiniBand usage.

Why it fails: NCCL_DEBUG only controls logging verbosity; it does not influence the transport selection. Setting it to INFO will provide more logs but will not enable InfiniBand if it is disabled. It is a diagnostic tool, not a configuration for transport.

Install and configure the NVIDIA Fabric Manager on all nodes.

Why it fails: Fabric Manager is required for NVLink/NVSwitch management within a node, not for InfiniBand communication between nodes. While it is important for intra-node GPU communication, it does not affect NCCL's choice to use InfiniBand for inter-node communication.

Set the NCCL_SOCKET_IFNAME environment variable to the Ethernet interface.

Why it fails: NCCL_SOCKET_IFNAME specifies the network interface for socket communication, typically used for bootstrap or when InfiniBand is not available. Setting it to Ethernet would force NCCL to use Ethernet for communication, which is the opposite of the desired outcome.

ResourceQuota on the namespace to limit nvidia.com/gpu requests.

Why it fails: A ResourceQuota limits the total number of GPU resources that can be requested within a namespace, but it does not restrict other namespaces from consuming GPUs. It enforces aggregate usage per namespace, not exclusive access. To isolate a specific namespace, you need a mechanism that ties GPU access to namespace identity, such as node affinity or taints and tolerations combined with MIG partitioning. ResourceQuota alone cannot prevent other namespaces from scheduling GPU pods on the same node.

Node taints and tolerations on the GPU node, with tolerations only in the target namespace's pods.

Why it fails: Taints and tolerations can repel pods that do not tolerate the taint, but they operate at the node level, not namespace level. Any pod in any namespace with the appropriate toleration could still schedule. While taints can restrict access to GPU nodes, they do not enforce namespace-specific isolation because tolerations are not namespace-scoped. This approach may reduce accidental scheduling but does not guarantee exclusive access for a specific namespace.

MIG (Multi-Instance GPU) configuration on the GPU Operator to create separate GPU instances per namespace.

Why it fails: MIG partitions a physical GPU into isolated instances with dedicated memory and compute, but it does not inherently associate those instances with Kubernetes namespaces. MIG profiles are assigned to pods via resource requests, but any pod with the correct resource request can use a MIG instance regardless of namespace. Without additional admission control or scheduling constraints, MIG alone does not enforce namespace-based isolation.

NFS over TCP

Why it fails: NFS over TCP is a general-purpose network file system that can be used for shared storage, but it is not specifically optimized for the high-throughput, low-latency demands of GPU-accelerated AI training. It may introduce bottlenecks for large-scale training jobs that require fast data loading, and NVIDIA does not recommend it as the primary storage solution for Base Command Platform.

Ceph RADOS Block Device

Why it fails: Ceph RBD is a distributed block storage system that can scale well, but it does not natively support direct GPU memory access. Without GPUDirect Storage, data must traverse the CPU and system memory, adding latency and reducing throughput. It is not the NVIDIA-recommended storage solution for maximizing AI training performance on Base Command Platform.

iSCSI block storage

Why it fails: iSCSI provides block-level storage over TCP/IP, but it does not offer the direct GPU-to-storage data path that GPUDirect Storage provides. While it can be used for shared storage, it is not specifically optimized for the high-bandwidth, low-latency requirements of AI training workloads on Base Command Platform, and would typically require additional configuration to achieve acceptable performance.

Deploy a service mesh across all nodes to encrypt traffic between workloads, and rely on mutual TLS to prevent tenants from accessing management endpoints.

Why it fails: A service mesh encrypts and authenticates workload-to-workload traffic, but it does not separate management and telemetry traffic from tenant networks. Mutual TLS controls access between services but does not prevent a tenant pod from reaching a management endpoint if network routes exist. This does not meet the isolation requirement.

Enable a dedicated out-of-band management VLAN on the BMC interfaces and a separate in-band VLAN for GPU telemetry, then configure Base Command Manager to use the in-band VLAN for metrics collection.

Why it fails: Separating BMC and telemetry VLANs is good practice, but requiring Base Command Manager to collect metrics over the in-band VLAN does not isolate management traffic from tenant workloads if that VLAN is shared. The requirement explicitly asks for isolation from tenant workload traffic, so telemetry should traverse a dedicated management network, not an in-band path that may carry tenant data.

Use NVIDIA Multi-Instance GPU (MIG) to partition each GPU and assign separate MIG instances to management and tenant workloads, ensuring physical isolation of telemetry paths.

Why it fails: MIG partitions GPU compute and memory resources but does not create network isolation for telemetry or management traffic. Telemetry from DCGM still traverses the host network interface. MIG is unrelated to VLAN separation and would not prevent tenants from accessing management endpoints over the network.

Restart the nvidia-persistenced service on all nodes to clear the error state.

Why it fails: nvidia-persistenced maintains driver state when no clients are attached, but restarting it does not clear XID errors or identify the faulty GPU. XID 48 is a hardware-level double-bit ECC event that persists in logs and may recur. Restarting the service across nodes is disruptive and does not provide the device-level attribution the administrator needs, so it fails to address the diagnostic goal.

Run nvidia-smi -q and check the ECC error counters and retired pages for each GPU.

Why it fails: ECC counters and retired pages relate to memory errors such as XID 48's sibling double-bit errors, but XID 48 specifically indicates a double-bit ECC error that has been contained. Checking counters is useful for context, yet it does not directly attribute the XID 48 to a specific GPU or confirm the failing unit. The administrator needs log correlation rather than aggregate counters to pinpoint the affected GPU.

Increase the GPU memory clock offset using nvidia-smi -lgc to stabilize the failing GPU.

Why it fails: Adjusting GPU clocks does not resolve double-bit ECC errors and could worsen instability. XID 48 indicates a memory error that the driver has contained, often pointing to a degrading GPU or memory module. Applying clock changes masks the symptom without identifying the device, and it may void support or cause further errors, making it an inappropriate response for this diagnostic scenario.

Run:ai node pools

Why it fails: Node pools are used to group nodes with similar characteristics, such as GPU type, and to schedule workloads to specific pools. They do not enforce per-user or per-group GPU request limits. While they can isolate workloads, they do not provide the granular quota control needed here.

Run:ai projects

Why it fails: Projects in Run:ai are used to organize workloads and manage access, but they do not directly enforce GPU request limits per user or group. Quotas are set at the project or department level, but the specific policy of limiting GPU requests based on group membership requires quotas, not just project membership.

Run:ai CLI commands

Why it fails: The Run:ai CLI allows administrators to manage resources, but it is not a feature that enforces policies. Policies are enforced through the platform's quota and access control mechanisms. Using CLI commands manually would not automatically enforce the limit for all users.

kubectl get pods -n gpu-operator -o wide

Why it fails: Listing pods with wide output shows pod status and node placement but does not expose internal errors within the device plugin container. A pod can appear Running while the plugin fails to register GPUs due to runtime issues. This command provides a high-level view but lacks the detailed error information needed to diagnose the missing GPU resource advertisement.

kubectl exec -n gpu-operator <nvidia-device-plugin-pod> -- nvidia-smi

Why it fails: Running nvidia-smi inside the device plugin pod can confirm GPU visibility from the container, but if the plugin is already failing to advertise resources, this command may not reveal the underlying configuration or driver mismatch. It is also more intrusive and less direct than reading the plugin's logs, which typically contain the specific error causing the registration failure.

kubectl describe node <node-name> | grep -A 10 'Allocatable'

Why it fails: This command shows the allocatable resources on a node, but if nvidia.com/gpu is missing from Allocatable, it only confirms the symptom already reported by the scheduler event. It does not reveal why the device plugin is not registering the GPU, such as missing driver libraries or a failed plugin pod, so it is not the most direct first diagnostic step.

NVIDIA Multi-Instance GPU (MIG)

Why it fails: MIG partitions a GPU into isolated instances with dedicated compute, memory, and bandwidth, but each instance is assigned exclusively to one process or container. It does not provide dynamic, time-sliced fair sharing among multiple containers on the same GPU instance; instead, it enforces hard partitioning, which is not the goal of preventing one job from monopolizing a shared GPU.

NVIDIA GPUDirect Storage

Why it fails: GPUDirect Storage is a technology that enables direct memory access between GPU memory and storage devices, bypassing the CPU and system memory. It is used to accelerate data loading for AI workloads, not to manage or enforce fair sharing of GPU compute resources among multiple containers. It has no role in scheduling or resource arbitration.

NVIDIA vGPU with time-sliced scheduling

Why it fails: NVIDIA vGPU is designed for virtualized environments, not for Kubernetes pods on bare metal. While it supports time-sliced sharing, it requires a hypervisor and vGPU manager, adding complexity and overhead. In a Kubernetes cluster with containerized workloads, vGPU is not the appropriate mechanism to enforce fair sharing among pods.

Add a nodeSelector to the training pods that matches the GPU model label, such as nvidia.com/gpu.product=NVIDIA-H100-80GB-HBM3.

Why it fails: Adding a nodeSelector can help target specific GPU models, but it does not resolve a scheduling failure caused by missing GPU resource advertisement. If the device plugin is not running, no node will report nvidia.com/gpu capacity regardless of labels. This action would not make the pods schedulable and could further restrict placement.

Increase the cluster's GPU resource quota by editing the nvidia.com/gpu ResourceQuota object in the kube-system namespace.

Why it fails: ResourceQuota objects limit namespace-level consumption, but they do not cause the scheduler to report Insufficient nvidia.com/gpu across all nodes. The error indicates the scheduler sees no allocatable GPU resources, not that a quota is blocking allocation. Editing a quota would not address missing device plugin advertisement.

Restart the kube-scheduler pod to force it to re-read node capacity and update its cache.

Why it fails: The Kubernetes scheduler continuously watches node status and does not require a restart to detect GPU capacity changes. If the device plugin is not advertising resources, restarting the scheduler will not create allocatable GPUs. This action does not address the root cause and may disrupt scheduling for other workloads.