Workload Management practice questions

Increase the number of worker nodes to distribute the memory load.

Why it fails: Scaling out horizontally does not solve local GPU memory fragmentation issues. Adding more nodes only distributes the workload, but each individual process will still encounter the same underlying memory management issues if the allocation patterns within the container remain unchanged, leading to identical OOM events across all nodes.

Switch to a larger instance type with more total VRAM capacity.

Why it fails: While increasing total VRAM provides more headroom, it does not address the root cause of fragmentation. Fragmentation occurs when free memory is split into non-contiguous blocks, making large allocations impossible even if total capacity is sufficient. Simply throwing more hardware at the problem ignores efficient memory management practices.

Increase the Kubernetes resource limit for system memory.

Why it fails: System memory limits controlled by Kubernetes are distinct from GPU VRAM usage. Adjusting system memory parameters will not influence how CUDA handles memory buffers on the GPU device. This configuration change would have no impact on preventing fragmentation-related OOM errors originating from within the GPU's memory space.

The preferred color scheme of the container logs.

Why it fails: Logging styles and formatting are application-level concerns that do not influence the cluster scheduler or the hardware allocation process. Including such parameters in a resource definition would be ignored by the Kubernetes scheduler and does not contribute to the effective management of hardware resources or workload placement.

The current market price of cloud GPU instances.

Why it fails: Pricing is a business or financial concern and is irrelevant to the technical requirements of the Kubernetes scheduler. The scheduler only evaluates hardware resource availability and node affinity policies when determining where to place a workload, regardless of the cost associated with the underlying infrastructure.

Disable the GPU Operator to allow raw Kubernetes to manage resource requests.

Why it fails: The GPU Operator is required to manage NVIDIA device plugins and driver lifecycles. Without it, the Kubernetes scheduler cannot properly identify or manage GPU resources, leading to failed pod deployments or unauthorized access. Disabling it removes the necessary infrastructure for observability and scheduling of NVIDIA-specific hardware resources.

Define Tolerations to permit pods to run on all nodes regardless of GPU type.

Why it fails: Tolerations allow pods to be scheduled on tainted nodes but do not guarantee that the node has the necessary GPU architecture. Relying on tolerations for heterogeneous clusters ignores the specific hardware requirements of the workload, leading to performance issues or application crashes when incompatible hardware is allocated.

Use Resource Quotas to prevent any single workload from consuming all available memory.

Why it fails: Resource quotas manage overall cluster capacity limits but do not solve the scheduling logic required for selecting the correct GPU hardware type. While useful for administrative fairness, they are insufficient for ensuring that specific workloads receive the optimized hardware they need for high-performance training or inference tasks.

Terminate all other jobs on the node to provide more memory to the failing job.

Why it fails: Terminating other jobs is an extreme and disruptive measure that impacts cluster-wide utilization and SLA compliance. It is not an appropriate response to a single job's failure, as the root cause is likely an incorrect memory specification or batch size, not a lack of node-level memory.

Disable GPU memory limits to allow the pod to consume all available hardware memory.

Why it fails: Disabling limits is dangerous as it allows a runaway process to consume all available memory, potentially crashing other pods and the underlying node OS. Proper workload management should enforce limits while allowing for intelligent retries with adjusted parameters, rather than removing safety guardrails entirely from the environment.

Move the job to a CPU-only node to bypass the GPU memory constraint.

Why it fails: Moving an AI training job to a CPU-only node is counterproductive, as the performance would be orders of magnitude slower, effectively rendering the job unusable. The goal is to successfully run the workload on the intended GPU hardware, not to shift the bottleneck to the CPU.

Allowing pods to preempt each other based on CPU usage only.

Why it fails: Preempting based on CPU usage is irrelevant for GPU workloads. The focus should be on the GPU's state and availability. Using CPU usage as the primary metric for preemption would lead to incorrect decisions, as many GPU-bound jobs have very low CPU usage but are mission-critical for the business.

Increasing the number of GPU-less nodes in the cluster.

Why it fails: Adding GPU-less nodes does not help manage the preemption of GPU workloads. It only provides more capacity for standard tasks, which does not solve the contention for GPU hardware. The focus for GPU preemption policy should be solely on the management of the GPU-enabled compute resources available.

The Pod lacks a 'tolerations' section for GPU-tainted nodes.

Why it fails: Tolerations are only required if nodes have specific taints applied to them. While common in production, the absence of a toleration does not inherently cause a scheduling failure unless the cluster policy specifically mandates tainting for all GPU nodes, which is not implied by the provided manifest.

The container image 'nvidia/cuda:11.0-base' is incompatible with A100 GPUs.

Why it fails: The image compatibility relates to runtime execution, not the scheduling process. Even if the CUDA version is suboptimal for the specific GPU architecture, the Kubernetes scheduler would still place the pod on the node; the pod would simply fail during the initialization phase when attempting to execute.

The resource limit 'nvidia.com/gpu: 2' is invalid for Kubernetes.

Why it fails: The resource key 'nvidia.com/gpu' is the standard naming convention for the NVIDIA device plugin in Kubernetes. Requesting two GPUs is a valid request, provided the cluster nodes are configured to support multi-GPU allocations. The syntax itself is correct and would not be the cause of failure.

Synchronous checkpointing triggered by a cron job.

Why it fails: Synchronous checkpointing forces the GPU to pause until the entire model state is written to persistent storage. This creates significant overhead and idle time, especially with massive LLM checkpoints, leading to reduced training efficiency and increased total time to convergence, which is suboptimal for high-performance AI operations.

Reducing the frequency of checkpoints to once per day.

Why it fails: Reducing checkpoint frequency increases the amount of work lost in the event of a failure. Given the volatility of large-scale distributed systems, low checkpoint frequency is a significant operational risk that can lead to weeks of wasted training progress, making it an unacceptable strategy for robust workload management.

Increasing the GPU batch size to compensate for I/O waits.

Why it fails: Increasing batch size changes the mathematical properties of the training process, potentially affecting model convergence quality. Furthermore, it does not resolve the blocking nature of synchronous I/O. If I/O is the bottleneck, increasing batch size will simply cause the GPU to wait for I/O more frequently, not less.

The high-priority job will automatically inherit the low-priority resource limit.

Why it fails: High-priority jobs are expected to have distinct resource policies compared to low-priority experimental tasks. Automatically applying a restrictive 10% limit to a high-priority job would be counterproductive, as these workloads typically require significant hardware acceleration to meet performance SLAs, defeating the purpose of assigning a higher priority level.

The policy will be ignored, and the high-priority job will fail to submit.

Why it fails: Workload policies are designed to be enforced by the scheduler. A well-configured system would not ignore a policy but rather evaluate the job against global resource constraints. If the job is valid, the scheduler will attempt to allocate the necessary GPUs, rather than failing the submission outright.

The policy requires the low-priority job to run on CPU only.

Why it fails: The policy limits usage to 10% but does not mandate a switch to CPU-only execution. Limiting GPU usage is typically implemented via partitioning or rate limiting. Forcing an AI workload to CPU only would lead to catastrophic performance drops, making the job effectively useless for deep learning training.

NVIDIA Container Toolkit

Why it fails: The Container Toolkit focuses on injecting the necessary NVIDIA drivers and libraries into the container runtime. It does not possess the scheduling logic or cluster-wide visibility required to make preemption decisions for pods across multiple nodes in a distributed training scenario.

NVIDIA DCGM Exporter

Why it fails: The DCGM Exporter is strictly a telemetry gathering tool designed to collect GPU metrics such as temperature, power, and utilization. It does not participate in the decision-making process for pod scheduling or the logic required to evict lower-priority tasks to free up hardware resources.

Kubelet Registry

Why it fails: The Kubelet is responsible for managing container lifecycles on a specific node based on instructions from the control plane. While it executes eviction commands received from the scheduler, it does not define the policy or the logic for preempting specific AI workloads based on job priority.

Limiting pod memory to the minimum allowed by the OS.

Why it fails: Severely limiting pod memory often leads to application crashes or performance degradation, especially in AI workloads that require substantial datasets to be loaded into memory. This action does not improve GPU utilization and is more likely to cause job failures that decrease the overall efficiency of the cluster.

Disabling GPU monitoring to reduce overhead.

Why it fails: Monitoring is essential for identifying underutilized GPUs. Disabling it makes it impossible to make data-driven decisions about cluster scaling or workload placement. Effective AI Ops requires deep observability to identify bottlenecks and ensure that hardware resources are being allocated in a way that optimizes for throughput and latency.

MIG_STRATEGY

Why it fails: MIG_STRATEGY is used to configure how MIG instances are created and assigned to nodes, specifically for architectures that support hardware-level partitioning. It is distinct from time-slicing, which is a software-based approach to GPU sharing and does not require the hardware support that MIG mandates on the GPU silicon.

REPLICA_COUNT

Why it fails: While intuitive, REPLICA_COUNT is not the specific key used in the NVIDIA Device Plugin configuration to enable time-slicing. The configuration utilizes a list of reconfigurables where you define the 'replicas' key under a specific 'timeSlicing' block to instruct the plugin to advertise more devices than exist physically.

GPU_OVERCOMMIT

Why it fails: GPU_OVERCOMMIT is not a standard configuration parameter within the NVIDIA Device Plugin. While the concept of overcommitting resources exists in Kubernetes, the specific implementation for NVIDIA GPUs is handled through the explicitly defined 'timeSlicing' configuration, which provides the necessary hooks for the scheduler to see multiple virtualized devices.

To immediately free up GPU memory.

Why it fails: While draining does free up memory, the primary goal is to ensure graceful termination of workloads. Simply freeing memory is a side effect. If the goal were just to kill jobs for memory, one could use other methods, but 'drain' is specifically designed to safely migrate running application processes.

To force a hardware reset of the GPU.

Why it fails: Draining a node is a software-level orchestration task that affects the Kubernetes pod lifecycle. It does not initiate a hardware reset or a power cycle of the physical GPU cards. Hardware resets are handled by separate administrative procedures involving the node's power management or the NVIDIA driver interface.

To update the NVIDIA driver without a reboot.

Why it fails: Updating the NVIDIA driver typically requires a reboot or a complete restart of the GPU-related services, regardless of the drain status. Draining just ensures that no containers are actively using the driver during this process, but it does not enable live-patching of the driver kernel modules themselves.

Configure node affinity for all high-priority pods.

Why it fails: Node affinity restricts where pods can be scheduled based on labels, but it does not provide a mechanism for preemption. If all nodes are full, affinity will simply cause the high-priority pod to remain in a pending state until resources become available, failing the requirement for immediate preemption.

Increase the GPU memory limit in the pod specification.

Why it fails: Modifying memory limits does not control scheduling behavior or resource preemption. It only restricts the maximum amount of GPU memory a container can consume. Increasing these limits would likely cause more pods to fail or hit OOM errors rather than enabling the preemption of existing lower-priority workloads.

Apply a Taint to the GPU-enabled nodes.

Why it fails: Taints and tolerations are used to repel pods from specific nodes. While they can prevent certain pods from landing on GPU nodes, they do not facilitate the eviction of already running pods to clear capacity. They serve as a gatekeeping mechanism rather than an active resource management or preemption tool.

The pod requires a toleration for the GPU taint.

Why it fails: While some GPU-enabled nodes use taints, the primary issue shown here is the incorrect nodeSelector. Even if a taint were present, the scheduler would report a scheduling failure specifically citing the taint. The provided manifest lacks a matching label, which is a more fundamental error in cluster resource targeting.

The image path is incorrect for local container registries.

Why it fails: The image path points to the NVIDIA container registry, which is publicly accessible. Unless there is a network restriction or DNS issue, the image pull should succeed. Scheduling failures of this type are almost always due to resource constraints or metadata mismatches, not the validity of the container image itself.

The GPU limit is set too high for the node.

Why it fails: Requesting 1 GPU is a valid configuration for almost any modern NVIDIA GPU-accelerated node. Unless the cluster is completely devoid of GPUs, this limit would not cause a permanent 'Pending' state. The issue lies in the node selection metadata, not in the resource request quantity of a single unit.

The version of the installed NVIDIA GPU driver.

Why it fails: Modern NVIDIA drivers support both Time-Slicing and exclusive modes. While some advanced features might require specific driver versions, the choice between shared and exclusive modes is primarily an architectural and configuration decision based on workload needs, not a constraint imposed by the basic version of the installed driver.

The brand of the underlying server motherboard.

Why it fails: Server hardware architecture is largely abstracted by the GPU Operator and the Kubernetes device plugin. The mode selection is a logical decision managed within the software stack. The underlying motherboard manufacturer does not influence how GPU resource multiplexing or exclusive access is configured within the Kubernetes environment.

Ensure the 'State' field in the node configuration is set to 'IDLE' for all nodes.

Why it fails: Manually setting the State field to IDLE is a static configuration that does not reflect dynamic node health. Slurm's node state is managed by the slurmctld daemon and can be changed at runtime. Forcing IDLE would bypass the scheduler's health checks and could lead to jobs being placed on unhealthy nodes, worsening the problem rather than solving it.

Set the 'MaxJobCount' parameter to limit the number of jobs that can run concurrently on drained nodes.

Why it fails: MaxJobCount limits the total number of jobs that can be submitted to the cluster, not the placement of jobs on specific nodes. It does not prevent the scheduler from assigning jobs to drained nodes. This parameter is unrelated to node state management and would not address the issue of jobs being allocated to nodes that are not ready for service.

Configure the 'SelectType' parameter to use 'select/linear' to ensure only idle nodes are selected.

Why it fails: SelectType determines the node selection plugin used by Slurm, such as select/linear or select/cons_res. Changing it to select/linear does not inherently prevent the scheduler from selecting drained nodes; it only changes the allocation algorithm. The issue is about node state, not the selection plugin, so this change would not reliably prevent jobs from being placed on drained nodes.

Topology Manager with the single-numa-node policy

Why it fails: Topology Manager aligns CPU, memory, and device allocations to a single NUMA node for latency-sensitive workloads. It does not control whether two pods can share a GPU; it only ensures resource affinity within a node. Enabling it would not cause inference pods to be packed together or training pods to be isolated, so it does not meet the stated scheduling goal.

NVIDIA device plugin with the shareable resource flag and node affinity rules

Why it fails: The NVIDIA device plugin can advertise shareable GPU resources, but simply enabling sharing does not differentiate between inference and training pods or automatically pack inference pods. Node affinity controls which nodes pods land on, not GPU-level co-location. This combination lacks the policy granularity needed to separate inference packing from training isolation.

MIG (Multi-Instance GPU) profiles via the NVIDIA GPU Operator

Why it fails: MIG partitions a single physical GPU into isolated instances with dedicated memory and compute slices, which is the opposite of packing multiple inference pods onto the same GPU slice. While MIG does provide hardware-level isolation, it does not let the scheduler co-locate inference pods on already-occupied GPU resources, so it fails to deliver the desired packing behavior for this scenario.

PodDisruptionBudget set to minAvailable: 1 on the interactive notebook deployments

Why it fails: PodDisruptionBudget only constrains voluntary disruptions such as node drains, not scheduler-driven preemption. Configuring minAvailable on the interactive deployments would actually protect those pods from being evicted, making the starvation problem worse rather than enabling batch jobs to reclaim GPUs.

LimitRange enforcing a maximum nvidia.com/gpu value per pod

Why it fails: LimitRange sets default or maximum resource values for individual pods in a namespace. It has no mechanism to reclaim GPUs from already-running interactive pods, so idle notebook sessions would continue to hold memory and batch jobs would still be starved.

ResourceQuota limiting nvidia.com/gpu requests per namespace

Why it fails: ResourceQuota caps the total number of GPU requests a namespace can make, but it does not evict or preempt running pods. Once the quota is filled, new batch pods would simply be rejected rather than allowed to take GPUs from idle interactive sessions, so it cannot satisfy the preemption requirement.

Use the 'gang scheduling' feature to group the inference job with training jobs.

Why it fails: Gang scheduling ensures that a set of pods are scheduled together or not at all, which is useful for distributed training. However, it does not provide preemption capabilities. Grouping the inference job with training jobs would not allow the inference job to preempt; instead, it might cause the inference job to wait for all training pods to be scheduled, which is counterproductive.

Configure a ResourceQuota for the training jobs to limit their GPU usage.

Why it fails: ResourceQuotas limit the total resources a namespace can consume, but they do not trigger preemption. They are enforced at admission time and prevent new pods from exceeding quotas, but they do not evict running pods. In this scenario, the training jobs are already running and occupying GPUs; a quota would not cause them to be preempted for the inference job.

Set the 'Node Affinity' for the inference job to target GPU nodes exclusively.

Why it fails: Node affinity controls which nodes a pod can be scheduled on based on labels. While it can influence placement, it does not enable preemption. Even if the inference job is targeted to GPU nodes, if those nodes are occupied by lower-priority jobs, the scheduler will not evict them without preemption enabled. Thus, this setting alone would not resolve the issue.

Configure the device plugin to use time-slicing with a replica count of 4 for all GPU resources.

Why it fails: Time-slicing allows multiple containers to share a GPU by interleaving execution, but it provides no memory or fault isolation between tenants. A noisy inference workload could be preempted by training bursts, violating the isolation requirement. Time-slicing is useful for improving utilization of best-effort workloads, but it is the wrong mechanism here because the scenario explicitly demands tenant isolation for latency-sensitive inference.

Set the CUDA_VISIBLE_DEVICES environment variable per pod to pin each container to a specific GPU index.

Why it fails: Setting CUDA_VISIBLE_DEVICES manually conflicts with Kubernetes device allocation and is not a supported isolation mechanism in a GPU Operator deployment. The scheduler and device plugin already control device assignment; overriding it can cause two pods to target the same physical device. This option neither improves utilization safely nor provides the hardware partitioning required for tenant isolation.

Apply node taints and tolerations so training pods only run on nodes without MIG-enabled GPUs.

Why it fails: Taints and tolerations are scheduling controls that separate workloads onto different nodes; they do not create isolation within a GPU. This approach would reduce the benefit of MIG by splitting the fleet rather than partitioning devices. It also adds operational overhead without addressing utilization of the MIG-capable GPUs, so it fails to satisfy the stated goals of higher utilization and in-device tenant isolation.