NCP-AIO Workload Management — 40 Questions

Question 1

An AI researcher is running a multi-node training job on a DGX SuperPOD using Kubernetes. The job frequently fails due to GPU memory fragmentation. Which workload management strategy best mitigates this issue?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Implement a CUDA memory pool using CUB or explicit memory allocators.. Memory fragmentation often occurs when long-running processes allocate and deallocate tensors of varying sizes without sufficient defragmentation. Implementing a fixed memory pooling strategy or utilizing NVIDIA Triton Inference Server's dynamic batching with strictly defined memory budgets ensures consistent allocation patterns. This approach is critical for high-uptime AI operations where unpredictable memory spikes can lead to OOM kills, effectively stabilizing resource utilization across the cluster's GPU nodes.

Answer

Increase the number of worker nodes to distribute the memory load.

Answer

Switch to a larger instance type with more total VRAM capacity.

Answer

Increase the Kubernetes resource limit for system memory.

Question 2

Which THREE factors should be considered when defining GPU resource requests for a containerized AI workload to prevent scheduling failures?

Accepted Answer

The required GPU memory capacity for the model weights.. Proper resource definition is critical for cluster stability. Defining memory, compute, and hardware requirements ensures that the scheduler has accurate information to place workloads. Inadequate definitions lead to scheduling failures or resource starvation, while over-provisioning leads to under-utilization. By accurately representing the workload's needs, operators ensure efficient bin-packing and prevent contention in high-demand environments, maintaining the performance SLAs required for production-grade artificial intelligence applications.

Answer

The preferred color scheme of the container logs.

Answer

The current market price of cloud GPU instances.

Question 3

An administrator is optimizing a Kubernetes cluster using NVIDIA GPU Operator. Which TWO configurations must be correctly implemented to ensure that GPU-bound workloads are scheduled efficiently across nodes with heterogeneous GPU types?

Accepted Answer

Configure Node Affinity using the nvidia.com/gpu.family label.. Proper workload management in heterogeneous GPU environments relies on precise node labeling and scheduling constraints. Using node affinity ensures that pods are only placed on nodes equipped with the required GPU architecture, while GPU resource requests allow the scheduler to account for specific memory and compute requirements, preventing scheduling failures or performance degradation due to mismatched hardware capabilities.

Answer

Disable the GPU Operator to allow raw Kubernetes to manage resource requests.

Answer

Define Tolerations to permit pods to run on all nodes regardless of GPU type.

Answer

Use Resource Quotas to prevent any single workload from consuming all available memory.

Question 4

When a job fails due to an OOM (Out of Memory) error on the GPU, which action should a workload manager take to best improve job success rates for subsequent attempts?

Accepted Answer

Automatically restart the job with an increased GPU memory request if the system supports dynamic resizing.. Automatically triggering a profile-aware retry with increased memory requests or a smaller batch size is the most effective approach. By analyzing the failure metadata, the scheduler can adjust the job parameters for the next iteration. This ensures that the workload eventually succeeds without requiring manual intervention, improving the overall reliability of the automated training pipeline in a busy production environment.

Answer

Terminate all other jobs on the node to provide more memory to the failing job.

Answer

Disable GPU memory limits to allow the pod to consume all available hardware memory.

Answer

Move the job to a CPU-only node to bypass the GPU memory constraint.

Question 5

Which THREE factors are essential when planning a cluster-wide policy for GPU workload preemption?

Accepted Answer

Setting appropriate PriorityClasses for different workload types.. Effective preemption policies must consider job priorities, grace periods, and the impact on distributed training state. High-priority jobs (like production inference or urgent training) must be able to displace lower-priority ones. Grace periods ensure that jobs can checkpoint their state before termination, preventing data loss. Finally, avoiding frequent preemption of distributed jobs is crucial because they are sensitive to latency and synchronization overhead, which can cause cascading performance failures.

Answer

Allowing pods to preempt each other based on CPU usage only.

Answer

Increasing the number of GPU-less nodes in the cluster.

Question 6

Refer to the exhibit. The pod is failing to schedule in a cluster with available A100 GPUs. Which factor is most likely preventing the scheduler from placing the pod?

Accepted Answer

The node selector 'gpu-type: tesla-a100' does not match the node labels.. The pod requires two GPUs on a single node, but the current configuration does not specify topology awareness or node affinity requirements beyond a basic selector. If the available GPUs are fragmented or the node lacks the necessary interconnects for a multi-GPU job, the scheduler will fail. This highlights the importance of matching workload requirements to physical cluster topology for successful deployment.

Answer

The Pod lacks a 'tolerations' section for GPU-tainted nodes.

Answer

The container image 'nvidia/cuda:11.0-base' is incompatible with A100 GPUs.

Answer

The resource limit 'nvidia.com/gpu: 2' is invalid for Kubernetes.

Question 7

When implementing a job queue for LLM training that requires periodic checkpointing to distributed storage, which strategy best minimizes the impact of checkpointing on training throughput?

Accepted Answer

Asynchronous checkpointing to local node storage.. Asynchronous checkpointing allows the training process to continue while the state is offloaded to storage in the background. By decoupling the checkpointing operation from the main compute loop, this strategy hides I/O latency, ensuring the GPUs are not idle during slow disk writes. This is critical for long-running LLM training where frequent state saving is required to mitigate the risk of hardware failures.

Answer

Synchronous checkpointing triggered by a cron job.

Answer

Reducing the frequency of checkpoints to once per day.

Answer

Increasing the GPU batch size to compensate for I/O waits.

Question 8

Refer to the exhibit. If a high-priority training job is submitted, what is the expected outcome of this policy in a restricted workload management system?

Accepted Answer

The scheduler will attempt to preempt low-priority jobs to free up GPUs for the new job.. The provided policy restricts low-priority jobs to 10% of cluster resources. When a high-priority job arrives, the scheduler must evaluate existing resource availability. If the low-priority job is consuming its allowed quota, the scheduler will prioritize the new high-priority request. Depending on the system, this could trigger pod preemption or simply block the submission if the high-priority resource requirements exceed current capacity.

Answer

The high-priority job will automatically inherit the low-priority resource limit.

Answer

The policy will be ignored, and the high-priority job will fail to submit.

Answer

The policy requires the low-priority job to run on CPU only.

Question 9

An AI researcher reports that a multi-node training job is frequently preempted on a Kubernetes cluster using the NVIDIA GPU Operator. Which component is responsible for orchestrating the preemption logic based on pod priority and GPU resource availability?

Accepted Answer

Volcano Scheduler. The NVIDIA K8s Device Plugin, in coordination with the Volcano scheduler, manages gang scheduling and resource preemption for AI workloads. Understanding this integration is critical for AI Ops professionals, as standard Kubernetes schedulers lack the awareness of NVLink topology or GPU memory constraints, often leading to suboptimal placement or frequent resource eviction when high-priority distributed training jobs request affinity-based scheduling in a multi-tenant environment.

Answer

NVIDIA Container Toolkit

Answer

NVIDIA DCGM Exporter

Answer

Kubelet Registry

Question 10

An AI Ops engineer is optimizing a large-scale training cluster. Which TWO actions would effectively improve GPU utilization in a multi-tenant environment?

Accepted Answer

Implementing MIG to partition GPUs.. Implementing MIG and sharing GPU resources through time-slicing are the primary ways to maximize utilization. MIG provides hard isolation for small tasks, while time-slicing allows multiple containers to share a single GPU by multiplexing access to the compute cores. Both strategies are crucial for preventing resource fragmentation and ensuring that the expensive hardware is consistently busy rather than sitting idle during smaller development or inference tasks.

Answer

Limiting pod memory to the minimum allowed by the OS.

Answer

Disabling GPU monitoring to reduce overhead.

Question 11

Which parameter in the NVIDIA Device Plugin configuration is used to enable the creation of multiple GPU instances via time-slicing?

Accepted Answer

timeSlicing. The 'timeSlicing' configuration section within the NVIDIA Device Plugin config map allows administrators to define multiple replicas for a single GPU. This technique is widely used in AI Ops to increase the number of available 'virtual' GPUs for lightweight inference or development tasks, allowing the scheduler to place more pods than there are physical devices, thereby maximizing the return on investment for high-end hardware.

Answer

MIG_STRATEGY

Answer

REPLICA_COUNT

Answer

GPU_OVERCOMMIT

Question 12

Why would an AI Ops engineer choose to use 'drain' mode on a GPU node before performing maintenance?

Accepted Answer

To prevent job termination and data loss.. Draining a node ensures that active training jobs are gracefully terminated or rescheduled, preventing data corruption or loss of progress. In AI workloads, where training can run for days, losing a checkpoint or crashing a job due to abrupt node removal is unacceptable. Proper drainage allows the orchestrator to migrate workloads, maintaining the integrity of long-running training processes and ensuring cluster stability during necessary updates.

Answer

To immediately free up GPU memory.

Answer

To force a hardware reset of the GPU.

Answer

To update the NVIDIA driver without a reboot.

Question 13

An organization is running multi-tenant AI training workloads on a Kubernetes cluster with NVIDIA GPU Operator. Which mechanism should an administrator use to ensure that a high-priority training job preempts low-priority development pods when GPU resources are exhausted?

Accepted Answer

Implement Kubernetes PriorityClass for pod scheduling.. PriorityClass in Kubernetes, when integrated with the NVIDIA GPU Operator and the device plugin, allows the scheduler to evict lower-priority pods to make room for higher-priority training tasks. This is critical for managing expensive GPU resources effectively, ensuring that mission-critical model training processes obtain the necessary compute hardware immediately, rather than waiting in a queue behind non-urgent development tasks that can be safely rescheduled later.

Answer

Configure node affinity for all high-priority pods.

Answer

Increase the GPU memory limit in the pod specification.

Answer

Apply a Taint to the GPU-enabled nodes.

Question 14

Refer to the exhibit. The pod remains in a 'Pending' state. After verifying that the GPU Operator is installed and nodes have available capacity, what is the most likely cause of this scheduling failure?

Accepted Answer

The nodeSelector key does not match existing node labels.. The pod uses a nodeSelector key that is not standard for the NVIDIA GPU Operator. By default, the operator labels nodes with 'nvidia.com/gpu.deploy.device-plugin'. Using 'nvidia.com/gpu.present' is non-standard and likely does not match any labels on the worker nodes. This is a common configuration error in AI infrastructure where incorrect node labels prevent the scheduler from finding an appropriate node, resulting in the pod being stuck in Pending status.

Answer

The pod requires a toleration for the GPU taint.

Answer

The image path is incorrect for local container registries.

Answer

The GPU limit is set too high for the node.

Question 15

Which THREE factors are primary drivers for choosing between 'exclusive' GPU mode and 'shared' GPU mode using Time-Slicing in a production AI environment?

Accepted Answer

Workload sensitivity to performance jitter and context switching.. Selecting between exclusive and shared modes depends on workload isolation, performance consistency, and resource efficiency requirements. Exclusive mode offers maximum performance and zero interference, ideal for large training jobs. Shared mode via Time-Slicing improves utilization for smaller tasks but introduces context-switching overhead and potential performance jitter. Balancing these factors is a core responsibility for AI platform engineers to optimize cost versus performance for diverse end-user workloads.

Answer

The version of the installed NVIDIA GPU driver.

Answer

The brand of the underlying server motherboard.

Question 16

A platform engineer is deploying a multi-node large language model training job on a Slurm cluster managed by NVIDIA Base Command Manager. The job repeatedly fails with a 'node not responding' error, and the scheduler log shows that the job was allocated nodes that had been drained for maintenance but were not yet returned to service. Which Slurm configuration should the engineer verify to ensure the scheduler does not assign jobs to nodes in a drained state?

Accepted Answer

Verify that the 'ReturnToService' parameter in slurm.conf is set to 1 to automatically return drained nodes to service.. The ReturnToService parameter in Slurm's configuration controls whether drained nodes are automatically returned to service after maintenance or reboot. When set to 1, nodes that were drained can be reintegrated into the scheduling pool once they are healthy, preventing the scheduler from allocating jobs to nodes that are still in a drained state. This directly addresses the scenario where jobs fail because they are assigned to nodes not yet returned to service.

Answer

Ensure the 'State' field in the node configuration is set to 'IDLE' for all nodes.

Answer

Set the 'MaxJobCount' parameter to limit the number of jobs that can run concurrently on drained nodes.

Answer

Configure the 'SelectType' parameter to use 'select/linear' to ensure only idle nodes are selected.

Question 17

An AI operations team runs mixed inference and training pods on a shared NVIDIA-accelerated Kubernetes cluster. They want the cluster scheduler to pack inference pods onto GPUs that already host other inference pods, while keeping training pods isolated on dedicated GPUs. Which Kubernetes mechanism should the administrator configure to achieve this placement behavior?

Accepted Answer

Extended resource scheduling with GPU sharing enabled through time-slicing or MPS and node taints/tolerations. The goal is to co-locate inference pods on shared GPUs while keeping training pods on dedicated hardware. GPU sharing via time-slicing or MPS lets multiple inference pods occupy one GPU, and taints/tolerations steer training pods to separate nodes. Together these mechanisms give the scheduler the signals it needs to pack and isolate workloads according to the stated policy.

Answer

Topology Manager with the single-numa-node policy

Answer

NVIDIA device plugin with the shareable resource flag and node affinity rules

Answer

MIG (Multi-Instance GPU) profiles via the NVIDIA GPU Operator

Question 18

A research team runs a mixture of long-running interactive Jupyter notebook sessions and short-lived batch inference jobs on the same Kubernetes cluster managed by the NVIDIA GPU Operator. Interactive sessions frequently hold GPU memory for hours while idle, starving the batch jobs. The administrator must let batch jobs preempt idle interactive pods without killing sessions that are actively computing. Which Kubernetes capability should be configured?

Accepted Answer

PriorityClass with preemptionPolicy: PreemptLowerPriority assigned to the batch jobs. PriorityClass with the PreemptLowerPriority policy lets the scheduler evict lower-priority pods when a higher-priority pod cannot be placed, which is exactly the mechanism needed to let batch jobs reclaim GPUs from idle interactive sessions. Quotas, LimitRanges, and PodDisruptionBudgets control admission or voluntary disruption but cannot trigger preemption based on pending demand.

Answer

PodDisruptionBudget set to minAvailable: 1 on the interactive notebook deployments

Answer

LimitRange enforcing a maximum nvidia.com/gpu value per pod

Answer

ResourceQuota limiting nvidia.com/gpu requests per namespace

Question 19

An AI operations team is using NVIDIA Run:ai to manage a shared GPU cluster. They want to ensure that a high-priority inference job can preempt lower-priority training jobs when GPU resources are scarce. The team has defined priorities in the Run:ai UI but observes that preemption is not occurring. Which Run:ai feature must be enabled to allow preemption based on priority?

Accepted Answer

Enable the 'Preemption' toggle in the Run:ai cluster settings and set the scheduler to 'Priority'.. Run:ai's preemption mechanism is not active by default; it must be explicitly enabled in the cluster settings. Setting the scheduler to prioritize based on defined priorities ensures that when resources are scarce, higher-priority jobs can evict lower-priority ones. This combination allows the inference job to preempt training jobs, aligning with the team's goal of ensuring high-priority workloads get resources when needed.

Answer

Use the 'gang scheduling' feature to group the inference job with training jobs.

Answer

Configure a ResourceQuota for the training jobs to limit their GPU usage.

Answer

Set the 'Node Affinity' for the inference job to target GPU nodes exclusively.

Question 20

A platform team runs mixed training and inference workloads on a Kubernetes cluster with NVIDIA GPU Operator and MIG-enabled A100 GPUs. They want to improve GPU utilization while keeping tenant isolation for latency-sensitive inference. Which TWO configurations should they implement? (Choose two.)

Accepted Answer

Deploy the NVIDIA GPU Operator with the MIG Manager enabled and label nodes with the desired MIG configuration.. MIG provides hardware-partitioned GPU instances with dedicated memory and compute, which is the only listed mechanism that simultaneously boosts utilization of a single A100 and preserves isolation for latency-sensitive inference. The GPU Operator's MIG Manager makes those partitions reproducible and label-driven. Together they let tenants request right-sized MIG resources such as nvidia.com/mig-1g.5gb without manual nvidia-smi tuning.

Answer

Configure the device plugin to use time-slicing with a replica count of 4 for all GPU resources.

Answer

Set the CUDA_VISIBLE_DEVICES environment variable per pod to pin each container to a specific GPU index.

Answer

Apply node taints and tolerations so training pods only run on nodes without MIG-enabled GPUs.