NCP-AIO Workload Management Practice Test 2 — 15 Questions

Question 1

A platform engineer is deploying a multi-node large language model training job on a Slurm cluster managed by NVIDIA Base Command Manager. The job repeatedly fails with a 'node not responding' error, and the scheduler log shows that the job was allocated nodes that had been drained for maintenance but were not yet returned to service. Which Slurm configuration should the engineer verify to ensure the scheduler does not assign jobs to nodes in a drained state?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Verify that the 'ReturnToService' parameter in slurm.conf is set to 1 to automatically return drained nodes to service.. The ReturnToService parameter in Slurm's configuration controls whether drained nodes are automatically returned to service after maintenance or reboot. When set to 1, nodes that were drained can be reintegrated into the scheduling pool once they are healthy, preventing the scheduler from allocating jobs to nodes that are still in a drained state. This directly addresses the scenario where jobs fail because they are assigned to nodes not yet returned to service.

Answer

Ensure the 'State' field in the node configuration is set to 'IDLE' for all nodes.

Answer

Set the 'MaxJobCount' parameter to limit the number of jobs that can run concurrently on drained nodes.

Answer

Configure the 'SelectType' parameter to use 'select/linear' to ensure only idle nodes are selected.

Question 2

An AI operations team runs mixed inference and training pods on a shared NVIDIA-accelerated Kubernetes cluster. They want the cluster scheduler to pack inference pods onto GPUs that already host other inference pods, while keeping training pods isolated on dedicated GPUs. Which Kubernetes mechanism should the administrator configure to achieve this placement behavior?

Accepted Answer

Extended resource scheduling with GPU sharing enabled through time-slicing or MPS and node taints/tolerations. The goal is to co-locate inference pods on shared GPUs while keeping training pods on dedicated hardware. GPU sharing via time-slicing or MPS lets multiple inference pods occupy one GPU, and taints/tolerations steer training pods to separate nodes. Together these mechanisms give the scheduler the signals it needs to pack and isolate workloads according to the stated policy.

Answer

Topology Manager with the single-numa-node policy

Answer

NVIDIA device plugin with the shareable resource flag and node affinity rules

Answer

MIG (Multi-Instance GPU) profiles via the NVIDIA GPU Operator

Question 3

A research team runs a mixture of long-running interactive Jupyter notebook sessions and short-lived batch inference jobs on the same Kubernetes cluster managed by the NVIDIA GPU Operator. Interactive sessions frequently hold GPU memory for hours while idle, starving the batch jobs. The administrator must let batch jobs preempt idle interactive pods without killing sessions that are actively computing. Which Kubernetes capability should be configured?

Accepted Answer

PriorityClass with preemptionPolicy: PreemptLowerPriority assigned to the batch jobs. PriorityClass with the PreemptLowerPriority policy lets the scheduler evict lower-priority pods when a higher-priority pod cannot be placed, which is exactly the mechanism needed to let batch jobs reclaim GPUs from idle interactive sessions. Quotas, LimitRanges, and PodDisruptionBudgets control admission or voluntary disruption but cannot trigger preemption based on pending demand.

Answer

PodDisruptionBudget set to minAvailable: 1 on the interactive notebook deployments

Answer

LimitRange enforcing a maximum nvidia.com/gpu value per pod

Answer

ResourceQuota limiting nvidia.com/gpu requests per namespace

Question 4

An AI operations team is using NVIDIA Run:ai to manage a shared GPU cluster. They want to ensure that a high-priority inference job can preempt lower-priority training jobs when GPU resources are scarce. The team has defined priorities in the Run:ai UI but observes that preemption is not occurring. Which Run:ai feature must be enabled to allow preemption based on priority?

Accepted Answer

Enable the 'Preemption' toggle in the Run:ai cluster settings and set the scheduler to 'Priority'.. Run:ai's preemption mechanism is not active by default; it must be explicitly enabled in the cluster settings. Setting the scheduler to prioritize based on defined priorities ensures that when resources are scarce, higher-priority jobs can evict lower-priority ones. This combination allows the inference job to preempt training jobs, aligning with the team's goal of ensuring high-priority workloads get resources when needed.

Answer

Use the 'gang scheduling' feature to group the inference job with training jobs.

Answer

Configure a ResourceQuota for the training jobs to limit their GPU usage.

Answer

Set the 'Node Affinity' for the inference job to target GPU nodes exclusively.

Question 5

A platform team runs mixed training and inference workloads on a Kubernetes cluster with NVIDIA GPU Operator and MIG-enabled A100 GPUs. They want to improve GPU utilization while keeping tenant isolation for latency-sensitive inference. Which TWO configurations should they implement? (Choose two.)

Accepted Answer

Deploy the NVIDIA GPU Operator with the MIG Manager enabled and label nodes with the desired MIG configuration.. MIG provides hardware-partitioned GPU instances with dedicated memory and compute, which is the only listed mechanism that simultaneously boosts utilization of a single A100 and preserves isolation for latency-sensitive inference. The GPU Operator's MIG Manager makes those partitions reproducible and label-driven. Together they let tenants request right-sized MIG resources such as nvidia.com/mig-1g.5gb without manual nvidia-smi tuning.

Answer

Configure the device plugin to use time-slicing with a replica count of 4 for all GPU resources.

Answer

Set the CUDA_VISIBLE_DEVICES environment variable per pod to pin each container to a specific GPU index.

Answer

Apply node taints and tolerations so training pods only run on nodes without MIG-enabled GPUs.

Question 6

A platform team runs mixed AI inference and batch training workloads on a Kubernetes cluster managed by the NVIDIA GPU Operator. Job owners report that high-priority inference pods are frequently preempted by long-running training pods that were admitted earlier. The team wants to guarantee that inference workloads always receive GPU capacity first without manually cordoning nodes. Which Kubernetes scheduling mechanism should the administrator configure to achieve this?

Accepted Answer

PriorityClass assigned to inference pods combined with preemption enabled on the training pods. Assigning a higher PriorityClass to inference pods and allowing preemption on lower-priority training pods makes the scheduler admit inference first and evict training when GPU capacity is scarce. This is the native Kubernetes mechanism for workload prioritization on GPU nodes managed by the NVIDIA GPU Operator, and it avoids manual intervention such as cordoning or draining nodes.

Answer

ResourceQuota objects scoped to the training namespace that cap the number of GPUs training may consume

Answer

Node affinity rules that pin inference pods to dedicated GPU nodes labeled by the GPU Operator

Answer

PodDisruptionBudget applied to inference workloads to protect them from voluntary eviction

Question 7

An administrator runs several short inference services on a single NVIDIA A100 in a Kubernetes cluster managed by the NVIDIA GPU Operator. Each pod requests `nvidia.com/gpu: 1`, so the device plugin advertises only one allocatable GPU and every additional replica stays Pending. The services are latency-tolerant and each needs only a fraction of the GPU's memory and compute. Which approach best allows multiple pods to share the physical GPU?

Accepted Answer

Deploy the NVIDIA MIG Manager to create memory-partitioned GPU instances and advertise each instance as a separate schedulable resource.. To let several pods each occupy a piece of one physical GPU, the device must be partitioned into separately schedulable units. On A100 hardware, MIG does this with memory and compute isolation, and the GPU Operator's MIG Manager plus device plugin expose each GPU instance as an allocatable resource, so every replica can be granted its own slice instead of competing for a single nvidia.com/gpu unit. Fractional requests are invalid, and the remaining options address failure handling or host memory rather than device partitioning.

Answer

Configure the device plugin with `--fail-on-init-error=false` and redeploy the services with lower memory limits.

Answer

Set `nvidia.com/gpu: 0.5` on each pod's resource requests so the device plugin can subdivide the GPU among replicas.

Answer

Enable time-slicing in the device plugin's ConfigMap so pods share the GPU, and set `replicas` equal to the number of services.

Question 8

A media company runs many small, bursty GPU inference services on a shared Kubernetes cluster managed by the NVIDIA GPU Operator. Administrators observe that most GPUs sit idle between requests, and each service currently reserves an entire GPU exclusively. They want to raise average utilization by allowing several containers to share one physical GPU while still capping the compute each container receives. Which TWO mechanisms should the administrator implement? (Choose two.)

Accepted Answer

Enable Multi-Instance GPU mode on supported data center GPUs and assign each container its own MIG instance with dedicated memory and compute partitions.. GPU sharing on Kubernetes is delivered through the device plugin's time-slicing replicas, which let many containers interleave on one device, and through MIG on supported hardware, which carves a GPU into isolated instances with bounded memory and compute. Together they lift average utilization and give each container a defined share, matching the media company's goal.

Answer

Apply a ResourceQuota to the namespace that limits the number of nvidia.com/gpu requests to less than the number of physical GPUs in the cluster.

Answer

Create a PriorityClass with a very high value for every inference service so the kube-scheduler can pack them onto the fewest possible GPUs.

Answer

Set the CUDA_VISIBLE_DEVICES environment variable to a fraction such as 0.5 inside each inference container manifest.

Question 9

A data science team submits a PyTorch training Job that requests four GPUs on a single node. The cluster's GPU Operator is healthy, but the Job remains Pending and the scheduler reports insufficient nvidia.com/gpu. Each node has four A100 GPUs, and no other workloads are running. Which configuration issue is the most likely cause?

Accepted Answer

The pod requests a MIG profile that does not exist on the nodes, so no allocatable GPU resources match. The insufficient nvidia.com/gpu event means the scheduler could not find a node advertising enough of the exact extended resource the pod requested. A MIG profile name that is not exposed on the nodes is the classic cause, because those resource names are distinct from nvidia.com/gpu and exist only when MIG is configured and the device plugin advertises them.

Answer

The node's kubelet has not been restarted after the device plugin was installed

Answer

The Job's parallelism is set higher than the number of requested GPUs

Answer

The pod omits a RuntimeClass, so the kubelet refuses to admit it

Question 10

A research team submits many short-lived GPU jobs to a Kubernetes cluster. Jobs frequently sit in Pending even though the cluster has free GPUs, and the scheduler log shows no fit errors. The administrator suspects that gang scheduling is missing, so that pods of the same job are not being placed together. Which Kubernetes-native feature should be enabled to allow a job's pods to be scheduled atomically as a group?

Accepted Answer

Kubernetes PodGroup with a scheduling gate, using the PodGroup and podgroup scheduling primitives to hold pods until the group can be placed together.. Gang scheduling requires a group-level construct that declares the combined resource need and holds members until all can be placed. The PodGroup API paired with a scheduling gate provides exactly that: pods carry a gate that prevents binding until the group's requirements are satisfiable, and the scheduler evaluates the group as a unit. This eliminates the partial-placement deadlock where a subset of pods occupies GPUs and the remainder waits indefinitely.

Answer

ResourceQuota with a high nvidia.com/gpu ceiling on the namespace, so the scheduler reserves capacity for the entire job before binding any pod.

Answer

Pod topology spread constraints, so pods of a job are distributed evenly across GPU nodes and never compete for the same device.

Answer

PriorityClass with preemption enabled, so higher-priority job pods evict lower-priority pods and free GPUs for the group.

Question 11

An AI infrastructure team is using NVIDIA Fleet Command to manage edge AI workloads across multiple remote sites. They need to ensure that a critical inference application continues to run even if the central management plane becomes unreachable. Which Fleet Command feature should they configure to meet this requirement?

Accepted Answer

Configure 'Autonomous Mode' on the edge nodes to allow local operation without the management plane.. Fleet Command's Autonomous Mode allows edge nodes to operate independently when the central management plane is unreachable. The edge node retains the last known application configuration and continues to run workloads, ensuring business continuity for critical inference applications. This feature is specifically designed for scenarios where network connectivity to the central management plane is intermittent or lost, providing resilience at the edge.

Answer

Set up a VPN tunnel between the edge nodes and the Fleet Command server.

Answer

Deploy a local instance of the Fleet Command server at each edge site.

Answer

Enable 'High Availability' mode on the Fleet Command server.

Question 12

An administrator manages a Kubernetes cluster where several teams share a node pool of NVIDIA GPUs. The platform must guarantee that a high-priority training job can preempt lower-priority inference pods when GPU capacity is exhausted, and that a runaway inference pod cannot starve the training job of GPU memory once both are running on the same device. Which TWO configurations are required to meet these goals? (Choose two.)

Accepted Answer

Assign PriorityClass objects so the training job has a higher value than the inference pods and enable preemption in the scheduler.. The two goals map to two distinct mechanisms. Preemption, driven by PriorityClass values, lets a queued high-priority training job evict lower-priority inference pods when the GPU pool is full. Memory isolation between co-resident workloads requires hardware partitioning, which MIG provides by assigning each workload a GPU instance with a fixed memory slice. Time-slicing adds concurrency without isolation, while quotas and limit ranges constrain requested quantities or host memory rather than device memory.

Answer

Set a `ResourceQuota` on the training namespace limiting `requests.nvidia.com/gpu` to one device.

Answer

Enable time-slicing in the device plugin so both workloads share the same GPU with round-robin execution.

Answer

Apply a `LimitRange` that sets a default GPU memory limit for containers in the inference namespace.

Question 13

A research team runs interactive Jupyter notebooks and batch inference jobs on the same Kubernetes cluster with NVIDIA GPU Operator. They want notebooks to keep a GPU for the duration of a session while allowing batch jobs to use remaining capacity. Which resource management approach should the administrator implement?

Accepted Answer

Use MIG or time-slicing configurations to expose fractional GPU resources, and schedule notebooks and batch jobs onto appropriately sized slices.. Allowing a notebook to hold a GPU while batch jobs use leftover capacity requires that a single physical GPU be divisible into schedulable units. MIG provides hardware-isolated slices, while time-slicing offers lighter-weight sharing without strict isolation. Either exposes fractional resources the scheduler can allocate, so interactive and batch workloads coexist on the same device pool instead of competing for whole GPUs.

Answer

Assign notebooks a long-running pod with a full GPU request, and give batch jobs a lower priority so they are preempted when needed.

Answer

Configure pod affinity so notebooks and batch jobs are placed on the same node but different GPUs.

Answer

Set a ResourceQuota on the namespace to limit the total number of GPU requests across all pods.

Question 14

A data platform team runs multi-tenant AI workloads on a Kubernetes cluster with the NVIDIA GPU Operator. Tenant A reports that its training pods are preempted repeatedly, while Tenant B's long-running inference pods hold GPUs for weeks. The administrator must guarantee that Tenant A's critical training jobs can reclaim GPU capacity from lower-priority tenants without evicting Tenant B's protected inference pods. Which configuration meets this requirement?

Accepted Answer

Give Tenant A's training pods a high-priority class and set preemptionPolicy: Never on Tenant B's inference pods so they cannot be victims of preemption.. Reclaiming capacity while shielding specific workloads requires two coordinated settings. Tenant A's training pods need a high-priority class with preemption enabled so they can evict lower-priority occupants when GPUs are scarce. Tenant B's inference pods need preemptionPolicy: Never, which excludes them from the victim pool so they keep their devices despite lower priority. This pairing delivers both reclaim and protection without affecting the protected pods.

Answer

Create a ResourceQuota per tenant that caps GPU usage, and rely on the quota controller to reclaim GPUs from Tenant B when Tenant A's jobs arrive.

Answer

Assign Tenant A's training pods a high-priority PriorityClass with preemptionPolicy: PreemptLowerPriority, and give Tenant B's inference pods a lower-priority class.

Answer

Enable time-slicing on all GPUs so Tenant A's jobs share devices with Tenant B's inference pods, avoiding preemption entirely.

Question 15

An AI operations team is deploying an NVIDIA GPU Operator-managed Kubernetes cluster to support both latency-sensitive inference and throughput-oriented batch training on the same fleet of A100 GPUs. The team wants to reduce resource contention between these workload classes without physically partitioning the hardware. Which TWO mechanisms should the administrator implement to achieve workload prioritization and fair GPU sharing? (Choose two.)

Accepted Answer

Define Kubernetes PriorityClass objects and assign higher priority to inference pods so the scheduler preempts lower-priority training pods when GPUs are scarce.. PriorityClass plus preemption lets the scheduler protect latency-sensitive inference by displacing batch training when GPUs are scarce, while time-slicing lets multiple pods share each physical A100, reducing contention without partitioning hardware. Together they provide both prioritization and fair sharing across the mixed workload classes on the same fleet.

Answer

Enable MIG on each A100 and create fixed 1g.5gb instances dedicated to inference while training uses the remaining slices.

Answer

Attach a node affinity rule that pins inference pods to a subset of GPU nodes and training pods to the remaining nodes.

Answer

Configure MPS with a percentage-based active thread limit per container and apply pod-level resource quotas for nvidia.com/gpu.