NCP-AIO Workload Management Practice Test 1 — 15 Questions

Question 1

An AI researcher is running a multi-node training job on a DGX SuperPOD using Kubernetes. The job frequently fails due to GPU memory fragmentation. Which workload management strategy best mitigates this issue?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Implement a CUDA memory pool using CUB or explicit memory allocators.. Memory fragmentation often occurs when long-running processes allocate and deallocate tensors of varying sizes without sufficient defragmentation. Implementing a fixed memory pooling strategy or utilizing NVIDIA Triton Inference Server's dynamic batching with strictly defined memory budgets ensures consistent allocation patterns. This approach is critical for high-uptime AI operations where unpredictable memory spikes can lead to OOM kills, effectively stabilizing resource utilization across the cluster's GPU nodes.

Answer

Increase the number of worker nodes to distribute the memory load.

Answer

Switch to a larger instance type with more total VRAM capacity.

Answer

Increase the Kubernetes resource limit for system memory.

Question 2

Which THREE factors should be considered when defining GPU resource requests for a containerized AI workload to prevent scheduling failures?

Accepted Answer

The required GPU memory capacity for the model weights.. Proper resource definition is critical for cluster stability. Defining memory, compute, and hardware requirements ensures that the scheduler has accurate information to place workloads. Inadequate definitions lead to scheduling failures or resource starvation, while over-provisioning leads to under-utilization. By accurately representing the workload's needs, operators ensure efficient bin-packing and prevent contention in high-demand environments, maintaining the performance SLAs required for production-grade artificial intelligence applications.

Answer

The preferred color scheme of the container logs.

Answer

The current market price of cloud GPU instances.

Question 3

An administrator is optimizing a Kubernetes cluster using NVIDIA GPU Operator. Which TWO configurations must be correctly implemented to ensure that GPU-bound workloads are scheduled efficiently across nodes with heterogeneous GPU types?

Accepted Answer

Configure Node Affinity using the nvidia.com/gpu.family label.. Proper workload management in heterogeneous GPU environments relies on precise node labeling and scheduling constraints. Using node affinity ensures that pods are only placed on nodes equipped with the required GPU architecture, while GPU resource requests allow the scheduler to account for specific memory and compute requirements, preventing scheduling failures or performance degradation due to mismatched hardware capabilities.

Answer

Disable the GPU Operator to allow raw Kubernetes to manage resource requests.

Answer

Define Tolerations to permit pods to run on all nodes regardless of GPU type.

Answer

Use Resource Quotas to prevent any single workload from consuming all available memory.

Question 4

When a job fails due to an OOM (Out of Memory) error on the GPU, which action should a workload manager take to best improve job success rates for subsequent attempts?

Accepted Answer

Automatically restart the job with an increased GPU memory request if the system supports dynamic resizing.. Automatically triggering a profile-aware retry with increased memory requests or a smaller batch size is the most effective approach. By analyzing the failure metadata, the scheduler can adjust the job parameters for the next iteration. This ensures that the workload eventually succeeds without requiring manual intervention, improving the overall reliability of the automated training pipeline in a busy production environment.

Answer

Terminate all other jobs on the node to provide more memory to the failing job.

Answer

Disable GPU memory limits to allow the pod to consume all available hardware memory.

Answer

Move the job to a CPU-only node to bypass the GPU memory constraint.

Question 5

Which THREE factors are essential when planning a cluster-wide policy for GPU workload preemption?

Accepted Answer

Setting appropriate PriorityClasses for different workload types.. Effective preemption policies must consider job priorities, grace periods, and the impact on distributed training state. High-priority jobs (like production inference or urgent training) must be able to displace lower-priority ones. Grace periods ensure that jobs can checkpoint their state before termination, preventing data loss. Finally, avoiding frequent preemption of distributed jobs is crucial because they are sensitive to latency and synchronization overhead, which can cause cascading performance failures.

Answer

Allowing pods to preempt each other based on CPU usage only.

Answer

Increasing the number of GPU-less nodes in the cluster.

Question 6

Refer to the exhibit. The pod is failing to schedule in a cluster with available A100 GPUs. Which factor is most likely preventing the scheduler from placing the pod?

Accepted Answer

The node selector 'gpu-type: tesla-a100' does not match the node labels.. The pod requires two GPUs on a single node, but the current configuration does not specify topology awareness or node affinity requirements beyond a basic selector. If the available GPUs are fragmented or the node lacks the necessary interconnects for a multi-GPU job, the scheduler will fail. This highlights the importance of matching workload requirements to physical cluster topology for successful deployment.

Answer

The Pod lacks a 'tolerations' section for GPU-tainted nodes.

Answer

The container image 'nvidia/cuda:11.0-base' is incompatible with A100 GPUs.

Answer

The resource limit 'nvidia.com/gpu: 2' is invalid for Kubernetes.

Question 7

When implementing a job queue for LLM training that requires periodic checkpointing to distributed storage, which strategy best minimizes the impact of checkpointing on training throughput?

Accepted Answer

Asynchronous checkpointing to local node storage.. Asynchronous checkpointing allows the training process to continue while the state is offloaded to storage in the background. By decoupling the checkpointing operation from the main compute loop, this strategy hides I/O latency, ensuring the GPUs are not idle during slow disk writes. This is critical for long-running LLM training where frequent state saving is required to mitigate the risk of hardware failures.

Answer

Synchronous checkpointing triggered by a cron job.

Answer

Reducing the frequency of checkpoints to once per day.

Answer

Increasing the GPU batch size to compensate for I/O waits.

Question 8

Refer to the exhibit. If a high-priority training job is submitted, what is the expected outcome of this policy in a restricted workload management system?

Accepted Answer

The scheduler will attempt to preempt low-priority jobs to free up GPUs for the new job.. The provided policy restricts low-priority jobs to 10% of cluster resources. When a high-priority job arrives, the scheduler must evaluate existing resource availability. If the low-priority job is consuming its allowed quota, the scheduler will prioritize the new high-priority request. Depending on the system, this could trigger pod preemption or simply block the submission if the high-priority resource requirements exceed current capacity.

Answer

The high-priority job will automatically inherit the low-priority resource limit.

Answer

The policy will be ignored, and the high-priority job will fail to submit.

Answer

The policy requires the low-priority job to run on CPU only.

Question 9

An AI researcher reports that a multi-node training job is frequently preempted on a Kubernetes cluster using the NVIDIA GPU Operator. Which component is responsible for orchestrating the preemption logic based on pod priority and GPU resource availability?

Accepted Answer

Volcano Scheduler. The NVIDIA K8s Device Plugin, in coordination with the Volcano scheduler, manages gang scheduling and resource preemption for AI workloads. Understanding this integration is critical for AI Ops professionals, as standard Kubernetes schedulers lack the awareness of NVLink topology or GPU memory constraints, often leading to suboptimal placement or frequent resource eviction when high-priority distributed training jobs request affinity-based scheduling in a multi-tenant environment.

Answer

NVIDIA Container Toolkit

Answer

NVIDIA DCGM Exporter

Answer

Kubelet Registry

Question 10

An AI Ops engineer is optimizing a large-scale training cluster. Which TWO actions would effectively improve GPU utilization in a multi-tenant environment?

Accepted Answer

Implementing MIG to partition GPUs.. Implementing MIG and sharing GPU resources through time-slicing are the primary ways to maximize utilization. MIG provides hard isolation for small tasks, while time-slicing allows multiple containers to share a single GPU by multiplexing access to the compute cores. Both strategies are crucial for preventing resource fragmentation and ensuring that the expensive hardware is consistently busy rather than sitting idle during smaller development or inference tasks.

Answer

Limiting pod memory to the minimum allowed by the OS.

Answer

Disabling GPU monitoring to reduce overhead.

Question 11

Which parameter in the NVIDIA Device Plugin configuration is used to enable the creation of multiple GPU instances via time-slicing?

Accepted Answer

timeSlicing. The 'timeSlicing' configuration section within the NVIDIA Device Plugin config map allows administrators to define multiple replicas for a single GPU. This technique is widely used in AI Ops to increase the number of available 'virtual' GPUs for lightweight inference or development tasks, allowing the scheduler to place more pods than there are physical devices, thereby maximizing the return on investment for high-end hardware.

Answer

MIG_STRATEGY

Answer

REPLICA_COUNT

Answer

GPU_OVERCOMMIT

Question 12

Why would an AI Ops engineer choose to use 'drain' mode on a GPU node before performing maintenance?

Accepted Answer

To prevent job termination and data loss.. Draining a node ensures that active training jobs are gracefully terminated or rescheduled, preventing data corruption or loss of progress. In AI workloads, where training can run for days, losing a checkpoint or crashing a job due to abrupt node removal is unacceptable. Proper drainage allows the orchestrator to migrate workloads, maintaining the integrity of long-running training processes and ensuring cluster stability during necessary updates.

Answer

To immediately free up GPU memory.

Answer

To force a hardware reset of the GPU.

Answer

To update the NVIDIA driver without a reboot.

Question 13

An organization is running multi-tenant AI training workloads on a Kubernetes cluster with NVIDIA GPU Operator. Which mechanism should an administrator use to ensure that a high-priority training job preempts low-priority development pods when GPU resources are exhausted?

Accepted Answer

Implement Kubernetes PriorityClass for pod scheduling.. PriorityClass in Kubernetes, when integrated with the NVIDIA GPU Operator and the device plugin, allows the scheduler to evict lower-priority pods to make room for higher-priority training tasks. This is critical for managing expensive GPU resources effectively, ensuring that mission-critical model training processes obtain the necessary compute hardware immediately, rather than waiting in a queue behind non-urgent development tasks that can be safely rescheduled later.

Answer

Configure node affinity for all high-priority pods.

Answer

Increase the GPU memory limit in the pod specification.

Answer

Apply a Taint to the GPU-enabled nodes.

Question 14

Refer to the exhibit. The pod remains in a 'Pending' state. After verifying that the GPU Operator is installed and nodes have available capacity, what is the most likely cause of this scheduling failure?

Accepted Answer

The nodeSelector key does not match existing node labels.. The pod uses a nodeSelector key that is not standard for the NVIDIA GPU Operator. By default, the operator labels nodes with 'nvidia.com/gpu.deploy.device-plugin'. Using 'nvidia.com/gpu.present' is non-standard and likely does not match any labels on the worker nodes. This is a common configuration error in AI infrastructure where incorrect node labels prevent the scheduler from finding an appropriate node, resulting in the pod being stuck in Pending status.

Answer

The pod requires a toleration for the GPU taint.

Answer

The image path is incorrect for local container registries.

Answer

The GPU limit is set too high for the node.

Question 15

Which THREE factors are primary drivers for choosing between 'exclusive' GPU mode and 'shared' GPU mode using Time-Slicing in a production AI environment?

Accepted Answer

Workload sensitivity to performance jitter and context switching.. Selecting between exclusive and shared modes depends on workload isolation, performance consistency, and resource efficiency requirements. Exclusive mode offers maximum performance and zero interference, ideal for large training jobs. Shared mode via Time-Slicing improves utilization for smaller tasks but introduces context-switching overhead and potential performance jitter. Balancing these factors is a core responsibility for AI platform engineers to optimize cost versus performance for diverse end-user workloads.

Answer

The version of the installed NVIDIA GPU driver.

Answer

The brand of the underlying server motherboard.