Refer to the Exhibit Practice Questions

The system has reached the max_concurrent_jobs limit.

Why it fails: While the policy limits concurrent jobs to two, the stem specifically mentions hitting 'resource limits' related to memory. The memory constraint is an independent limit defined in the policy. The error reported is clearly tied to the resource quota configuration, not the job count limit defined in the JSON.

The NVIDIA driver is blocking the user's access.

Why it fails: The NVIDIA driver does not enforce per-user memory quotas; this is handled by a higher-level orchestration or scheduling layer. The JSON policy indicates an management-level configuration file, not a driver-level limitation. The error is due to the policy quota, not a restriction imposed by the underlying low-level hardware driver.

The GPU memory is fragmented across the nodes.

Why it fails: Fragmentation can cause issues, but in this context, the specific JSON quota is the bottleneck being applied. The administrator is looking at a policy-driven limit. Fragmentation is a low-level memory management issue that would manifest differently than a hard limit defined by an administrative resource policy configuration file.

The container gains elevated privileges to access the host GPU

Why it fails: Access to device files does not confer elevated privileges; it simply provides the required interface for the driver library to communicate with the hardware. Restricting these files prevents the container from seeing or using the GPU at all, which is the opposite of providing elevated access or permissions.

The container can use the GPU but cannot perform memory mapping

Why it fails: GPU communication is binary; the driver either successfully initializes the device or it does not. There is no partial state where a container can use the GPU for logic but fail at memory mapping. If the device nodes are blocked, the entire interface is rendered completely unusable.

The container experiences increased latency for GPU operations

Why it fails: Latency is a measure of performance degradation, not a binary access control outcome. If the policy denies access to the hardware devices, the application will not experience latency; it will encounter a catastrophic failure or segmentation fault immediately upon attempting to query the GPU device drivers.

The pod is requesting more CPU cores than the node can provide.

Why it fails: The error specifically mentions 'Insufficient nvidia.com/gpu', not CPU or memory. Kubernetes provides specific error messages for each resource type. Therefore, focusing on CPU availability is a distraction from the root cause, which is related to the GPU resource tracking mechanism and its communication with the Kubernetes scheduler.

The node is in a 'NotReady' state due to high disk latency.

Why it fails: If a node were in a 'NotReady' state, the scheduler would ignore it entirely, but the error would be different, likely indicating that there are no nodes matching the scheduler's criteria. The 'Insufficient' error implies that the scheduler evaluated the nodes but found that the GPU resource capacity was already fully utilized.

The pod security policy prohibits the use of GPU-accelerated containers.

Why it fails: Pod security policies (or Pod Security Admissions) control permissions and capabilities, not resource allocation. If a policy violation occurred, the pod would fail with a 'Forbidden' or 'Admission denied' error, not an 'Insufficient resource' error, which is strictly related to the accounting of requested versus available cluster capacity.

The PriorityClass value is too low to trigger preemption.

Why it fails: A value of one million is significantly higher than the standard Kubernetes system defaults, which typically range from 1,000 to 10,000. Increasing the value further would not resolve the issue if the underlying issue is related to Gang Scheduling constraints or resource placement logic, rather than actual preemption.

The NVIDIA Device Plugin requires a restart to recognize the PriorityClass.

Why it fails: The NVIDIA Device Plugin does not require a restart to acknowledge new PriorityClass objects. The scheduling logic is handled by the control plane and the scheduler component; the plugin simply advertises the GPU resources available to the scheduler, which makes decisions based on existing cluster state information.

The PriorityClass name violates Kubernetes naming conventions.

Why it fails: The name 'high-priority-gpu' is a valid alphanumeric string following standard Kubernetes resource naming conventions. The issue is logically rooted in how the scheduler interprets resource requirements versus cluster availability, not a syntax error within the manifest file itself, which would have been rejected during initial creation.

Add a preferredDuringSchedulingIgnoredDuringExecution nodeAffinity rule that favors the Frankfurt region with a high weight.

Why it fails: Preferred affinity is only a soft hint; if Frankfurt capacity is unavailable, the scheduler will happily place the pod on a node in another region. That behavior directly violates the requirement that the job must never run outside Frankfurt, so a weighted preference cannot satisfy the audit control.

Set the pod's restartPolicy to Never and add a toleration for the region-specific taint on Frankfurt nodes.

Why it fails: A toleration merely allows a pod to be placed on a tainted node; it does not require that placement, so the scheduler could still choose an untainted node in another region. Changing restartPolicy affects failure handling only and provides no geographic guarantee, leaving the audit requirement unmet.

Create a PodDisruptionBudget for the training job that limits voluntary evictions to zero during the run.

Why it fails: A PodDisruptionBudget controls voluntary disruptions such as node drains; it has no influence on where the scheduler initially places a pod. The training job could still be scheduled in another region at admission time, so this mechanism does nothing to enforce the geographic restriction the auditors require.

The GPUs have different amounts of physical memory.

Why it fails: If the GPUs had different physical memory, the system would likely throw an error or fail to initialize the distributed environment. The memory usage is similar, which suggests the hardware is physically capable, but the software is failing to schedule compute tasks across both devices effectively.

The system is limited by the PCIe bus bandwidth.

Why it fails: PCIe bandwidth issues would cause both GPUs to show low utilization while waiting for data. In this scenario, one GPU is performing at near-maximum capacity, which proves the bus can handle the data load; the bottleneck is specifically the application logic distributing tasks to only one device.

The model is too small to be parallelized across two GPUs.

Why it fails: Even small models can be parallelized using techniques like batch parallelism. The issue here is not the model size, but rather the execution environment. The presence of high memory usage on both GPUs suggests the model is loaded on both, but the compute is not being distributed.

Set "compute_mode" to "exclusive_process".

Why it fails: Changing the compute mode to exclusive_process prevents multiple processes from sharing the GPU, but it does not address the thermal output of the hardware. The GPU will still generate the same amount of heat, meaning thermal throttling will continue to occur regardless of the process mode.

Disable ECC mode to lower the power consumption.

Why it fails: Disabling ECC mode can lead to data corruption in large-scale training jobs, which is unacceptable for model reliability. Furthermore, the power saving from disabling ECC is negligible compared to the total thermal output of the GPU, making this an ineffective strategy for preventing thermal throttling.

Set "persistence" to "disabled".

Why it fails: Disabling persistence mode would result in the driver re-initializing the GPU between tasks, adding overhead and increasing latency. It does not reduce the heat generated by the GPU under load and will not prevent thermal throttling during active training operations.

The GPU is overheating and the thermal management system has engaged to prevent hardware damage.

Why it fails: Thermal throttling would be indicated by 'HW Thermal Slowdown' in the throttle reasons field. The presence of 'SW Power Cap' specifically points to a configuration constraint set by the software, not a cooling system failure or excessive ambient temperatures affecting the GPU's operational state.

The GPU is currently idle and the driver has shifted the card to an energy-saving state.

Why it fails: While P12 is a low-power state, the explicit mention of 'SW Power Cap' in the throttle reasons confirms that the driver is actively restricting performance due to policy settings. This is distinct from a normal idle state transition, which would not typically trigger a throttle reason alert.

The GPU is experiencing a PCIe bus error, forcing the system to reduce clock speeds to maintain stability.

Why it fails: PCIe bus errors are typically logged in the kernel ring buffer or via specialized diagnostic tools, not as a power cap throttle. This message indicates a power management policy enforcement, not a hardware communication fault or an interrupt-driven slowdown originating from the system bus interface.

The training job will have lower memory consumption

Why it fails: Disabling P2P does not reduce memory usage; in some cases, it may actually increase it due to the need for additional staging buffers in system RAM. The primary impact is on communication latency and bandwidth, not on the total memory footprint of the application or the model parameters.

The job will be more stable across nodes

Why it fails: Stability is not improved by disabling P2P; in fact, relying on network-based communication increases the susceptibility to network congestion and latency jitter. This setting is a performance workaround, not a reliability feature, and it does not enhance the robustness of the training job against network or node-level failures.

The system will utilize less power during training

Why it fails: While bypassing NVLink might theoretically reduce the power consumed by the interconnect, the performance penalty is so severe that the total energy efficiency of the training job decreases significantly. The job takes longer to complete, leading to higher overall power consumption for the same amount of computation.

The GPU driver version is incompatible with the quota controller.

Why it fails: ResourceQuotas are handled by the Kubernetes API server and controller manager, not the GPU driver. The error is purely related to admission control policies in Kubernetes. The driver version does not influence how quotas are evaluated; the error is clearly a violation of a policy defined for that namespace.

The cluster is out of available GPU capacity.

Why it fails: If the cluster were out of capacity, the error would manifest as a scheduling failure (Pending status) rather than a rejection at the API server level. Admission control errors regarding 'forbidden' actions indicate policy violations rather than physical hardware availability issues within the cluster nodes themselves.

The NVIDIA Device Plugin is not running in the namespace.

Why it fails: The device plugin's presence does not impact the API server's ability to enforce resource quotas. A quota error is a configuration issue within the namespace settings and the submitted YAML manifest. The device plugin is only relevant after the pod passes the admission controller and begins the scheduling phase.

The GPU is overheating due to a fan failure.

Why it fails: If the GPU were overheating, the 'HW Slowdown' or thermal throttling flags would be active. The current logs explicitly state that the hardware slowdown is not active, pointing the root cause toward a configured software limit rather than a physical cooling or hardware-level environmental issue in the chassis.

The GPU driver is corrupted or out of date.

Why it fails: A corrupted driver usually results in failed device initialization, kernel panic, or complete lack of visibility into the GPU status via nvidia-smi. Since the device is successfully reporting performance states and throttle reasons, the driver is operational and communicating with the hardware correctly for power management.

The workload is waiting for CPU memory allocation.

Why it fails: While CPU memory bottlenecks can occur, they do not manifest as GPU power throttling events. This indicator is specifically related to the GPU's internal power management logic. A CPU bottleneck would be identified through system-level monitoring tools like top, htop, or iostat rather than GPU-specific power management metrics.

The node is not part of a valid Kubernetes namespace.

Why it fails: Kubernetes nodes are cluster-scoped resources and are not assigned to specific namespaces. A node's availability for scheduling is determined by labels and taints, not by namespace membership. This is a misunderstanding of core Kubernetes architecture, as namespaces are for organizing logical resources like pods and services.

The GPU firmware is out of date.

Why it fails: While firmware updates are important for performance and stability, they rarely prevent the OS from seeing the GPU hardware. Even with outdated firmware, the kernel should still identify the PCIe device, allowing NFD to report it. The issue here is clearly a failure of the orchestration layer's detection.

The cluster is using an unsupported CPU architecture.

Why it fails: Most AI Enterprise deployments support standard x86_64 architectures. While specific hardware mismatches can occur, it is highly unlikely that a working cluster would be completely unable to detect a GPU solely due to the CPU architecture. The issue is almost certainly related to the software-defined detection logic.

The GPU driver is outdated and needs a patch.

Why it fails: Outdated drivers would show compatibility warnings or failures in initialization, not hardware-specific thermal slowdown flags. The logs explicitly point to hardware-level thermal events, which are independent of the software driver version. Prioritizing driver updates would ignore the physical cooling problem clearly indicated by the diagnostic tool.

The power supply unit is malfunctioning.

Why it fails: A failing PSU usually results in system instability, unexpected shutdowns, or power-related throttling flags, not thermal slowdowns. While power issues can cause heat, the logs specifically categorize this as a thermal event. The administrator must focus on cooling infrastructure rather than testing the electrical power delivery system.

The workload exceeds the GPU memory capacity.

Why it fails: Exceeding GPU memory capacity causes 'Out of Memory' (OOM) errors, leading to immediate process termination. Memory usage does not cause hardware thermal slowdowns. The exhibit clearly displays performance throttling flags, which are unrelated to the current allocation of VRAM by the running AI training or inference tasks.

The GPU driver version is incompatible with the installed NCCL library.

Why it fails: Incompatible driver versions typically cause CUDA initialization failures or segmentation faults at the kernel level rather than network socket errors. A connection refused error explicitly indicates that the network stack is rejecting the connection attempt, pointing toward networking configuration rather than driver-level compatibility issues.

The model weights are too large to be synchronized across the network.

Why it fails: Model weight size does not trigger connection refused errors during initial setup. Large weights might lead to timeouts during gradient synchronization if the network is saturated, but a failure at the connect() call indicates the nodes are unable to establish the initial network handshake regardless of traffic volume.

The CUDA visibility is restricted to only one GPU per node.

Why it fails: CUDA visibility settings, such as CUDA_VISIBLE_DEVICES, limit the GPUs available to a process but do not affect the network connectivity between nodes. Even if visibility were restricted, the network socket logic would still attempt to initialize, provided the network interface is correctly configured for the node.

Increase the pod's nvidia.com/gpu limit to 8 so each node exposes all GPUs to the training process.

Why it fails: GPU count per pod affects local parallelism but does not influence which network fabric NCCL selects for inter-node communication. If the interface variables are wrong, adding more GPUs per pod will not fix the collective hang and may even increase the number of stalled ranks. The network path selection is governed by NCCL environment variables, not GPU limits.

Set the CUDA_VISIBLE_DEVICES variable to list all GPUs and restart the training job.

Why it fails: CUDA_VISIBLE_DEVICES controls which GPUs a process can see locally; it has no effect on inter-node communication or fabric selection. A hang during distributed initialization is typically caused by NCCL failing to establish connections over the intended interface, not by GPU visibility. Adjusting this variable will not resolve a collective communication stall.

Enable the NVIDIA MIG feature on all nodes so each rank gets an isolated GPU slice for communication.

Why it fails: MIG isolates GPU resources within a single device but does not alter how NCCL selects network interfaces across nodes. Enabling MIG could actually reduce available bandwidth per rank and complicate topology discovery. The stall at initialization points to fabric selection, so MIG configuration is irrelevant to resolving the communication failure.