NCP-AIO Workload Management Practice Test 6 — 15 Questions

Question 1

An administrator manages a shared NVIDIA cluster where several teams run inference services. One team's pods are being evicted repeatedly, and DCGM metrics show the node's GPUs are healthy but memory on the devices is nearly exhausted. The team insists their model fits. Which action should the administrator take FIRST to identify the cause?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Inspect the pod specs and running processes for GPU memory held by other containers on the same node.. Healthy GPUs with near-exhausted device memory in a shared cluster usually indicate that another container on the same node is holding memory on the same device. The fastest way to confirm this is to inspect pod specifications and running processes to see which workloads share the GPU. Replacing hardware, changing host memory limits, or tuning telemetry do not address the allocation conflict and delay the correct diagnosis.

Answer

Increase the pod's memory limit in the deployment manifest and restart it.

Answer

Replace the affected GPUs and re-run the job on fresh hardware.

Answer

Lower the DCGM sampling interval so metrics capture the spike more precisely.

Question 2

An AI operations engineer manages a shared Kubernetes cluster running NVIDIA GPU Operator. Several teams report that their inference pods remain in a Pending state with the event '0/8 nodes are available: 8 Insufficient nvidia.com/gpu.' The administrator verifies that nvidia-smi on all nodes shows idle GPUs. Which action should the administrator take first to resolve the scheduling failure?

Accepted Answer

Inspect the node allocatable GPU count and any taints or labels that prevent scheduling on the idle nodes.. The Pending event shows the scheduler believes no node has a free nvidia.com/gpu resource, even though nvidia-smi reports idle devices. That gap is almost always caused by the device plugin not advertising GPUs, or by taints, labels, or affinity rules excluding the nodes. Inspecting allocatable counts and scheduling constraints identifies the actual blocker before any disruptive remediation.

Answer

Reinstall the NVIDIA GPU Operator and restart the kubelet on all worker nodes.

Answer

Add a nodeSelector for kubernetes.io/os=linux to the pod template.

Answer

Increase the GPU memory limit in the pod specification so the scheduler can fit the workload.

Question 3

A platform engineer is preparing an NVIDIA-accelerated Kubernetes cluster for a new team that will submit PyTorch training jobs. The team wants jobs to request GPUs without hardcoding device indices. Which Kubernetes resource should the engineer ensure is installed and healthy so pods can request nvidia.com/gpu resources?

Accepted Answer

NVIDIA GPU Operator with the device plugin component enabled. Kubernetes learns about specialized hardware through device plugins that advertise extended resources. The NVIDIA device plugin, managed by the GPU Operator, publishes nvidia.com/gpu counts so the scheduler can allocate GPUs to pods. With it healthy, PyTorch jobs can declare a GPU request and receive an assigned device without the user specifying a physical index.

Answer

NVIDIA Container Toolkit installed on each worker node

Answer

NVIDIA DCGM Exporter deployed as a DaemonSet

Answer

NVIDIA Network Operator with RDMA shared device plugin

Question 4

A data engineering team runs nightly batch inference on a Kubernetes cluster with NVIDIA GPUs. Jobs sometimes fail because two pods are scheduled onto the same physical GPU and one exhausts framebuffer memory. The team wants each pod to receive an isolated slice of a GPU with dedicated memory. Which NVIDIA feature should they enable?

Accepted Answer

Multi-Instance GPU (MIG). The team needs hardware-level isolation with dedicated memory per workload. Multi-Instance GPU partitions a physical GPU into independent instances, each with its own memory and compute, and the device plugin can expose those instances as discrete resources. GPUDirect Storage, NCCL, and the Container Toolkit each address I/O paths, communication, or containerization rather than partitioning a GPU among tenants.

Answer

NVIDIA Collective Communications Library (NCCL)

Answer

GPUDirect Storage

Answer

NVIDIA Container Toolkit

Question 5

An AI platform team runs GPU-accelerated inference pods on a Kubernetes cluster with the NVIDIA GPU Operator. During peak load, high-priority latency-sensitive inference pods are frequently preempted by large batch training jobs that were submitted later. The team wants the scheduler to guarantee that inference pods always win placement and preemption decisions against training pods without manually cordoning nodes. Which mechanism should the administrator configure?

Accepted Answer

Assign the inference pods a higher PriorityClass value and reference it in their pod spec so the kube-scheduler preempts lower-priority training pods when resources are scarce.. Kubernetes resolves GPU contention through the scheduler's priority and preemption logic, not through device sharing or quota limits. Giving inference pods a higher PriorityClass value makes the kube-scheduler place them first and evict lower-priority training pods when capacity is exhausted, which delivers the guaranteed preference the platform team requires.

Answer

Enable the GPU Operator's time-slicing feature with a replica count greater than one so inference and training pods share each physical GPU concurrently.

Answer

Create a separate namespace with a ResourceQuota that limits training pods to a small number of GPUs and place all inference pods in the default namespace.

Answer

Add a nodeSelector to the inference pods that pins them to nodes labeled with the NVIDIA GPU product name, so they only run on nodes not used by training.

Question 6

An AI operations engineer manages a Kubernetes cluster where the NVIDIA GPU Operator's device plugin exposes GPUs as schedulable resources. A data science team submits a batch inference job that requests one GPU but does not specify a node selector or tolerations. The job stays in Pending while other GPU nodes remain idle because they carry a taint the GPU Operator applied to reserve them for a specific workload class. Which approach is the most appropriate for the engineer to make the job schedulable without disrupting the reserved nodes?

Accepted Answer

Add an appropriate toleration and node selector to the job so it can target the reserved GPU nodes.. A node taint repels pods unless they carry a matching toleration. The reserved GPU nodes are intentionally tainted, so a job that neither tolerates the taint nor selects an untainted node cannot schedule. Adding the correct toleration plus a node selector places the job on the reserved nodes while preserving the reservation for other classes of work.

Answer

Delete and recreate the device plugin daemonset so GPUs are re-advertised to the scheduler.

Answer

Increase the cluster's GPU resource quota in the namespace so the scheduler can allocate a card.

Answer

Remove the node taint from all GPU nodes so the job can be scheduled anywhere.

Question 7

An MLOps engineer needs to guarantee that a latency-sensitive inference Deployment always has GPU capacity available, even when a large training Job is submitted to the same namespace. The cluster uses the NVIDIA GPU Operator and nodes have four A100 GPUs each. Which approach reliably reserves GPU capacity for the inference Deployment?

Accepted Answer

Create a separate node pool and use nodeSelector or nodeAffinity to pin the inference Deployment to nodes tainted for inference only. Only a dedicated, tainted node pool combined with nodeSelector or nodeAffinity creates a hard capacity reservation that training workloads cannot violate. Priority-based preemption and MPS improve scheduling behavior or sharing but do not prevent a co-located training Job from consuming the GPUs first, and a namespace ResourceQuota limits aggregate requests without reserving specific capacity for the inference Deployment.

Answer

Set a higher PriorityClass on the inference Deployment so it preempts training pods when needed

Answer

Enable the NVIDIA MPS control daemon and configure each inference pod with a shared memory fraction

Answer

Apply a ResourceQuota on the namespace that limits nvidia.com/gpu to the number of inference replicas

Question 8

An administrator manages a Kubernetes cluster where a training job repeatedly fails with an OutOfMemory error on the GPU even though the pod requests one nvidia.com/gpu. DCGM metrics show that another pod on the same node is consuming GPU memory concurrently. GPU sharing via time-slicing is enabled cluster-wide. Which action should the administrator take to prevent this cross-pod interference while preserving the ability to share GPUs among trusted inference workloads?

Accepted Answer

Disable time-slicing on the node pool hosting training jobs and use MIG or dedicated GPUs for those workloads. Time-slicing multiplexes a GPU across pods without memory partitioning, so one workload can starve another and trigger OutOfMemory errors. The reliable fix is to separate untrusted or memory-intensive training workloads onto dedicated GPUs or MIG instances, while leaving time-slicing for inference workloads where memory contention is acceptable and controlled.

Answer

Apply a ResourceQuota limiting nvidia.com/gpu to one per namespace

Answer

Set the CUDA_VISIBLE_DEVICES environment variable manually in the training pod spec

Answer

Increase the pod's nvidia.com/gpu request to two GPUs

Question 9

A platform team runs an NVIDIA GPU Operator-managed Kubernetes cluster shared by two research groups. Group A's pods request `nvidia.com/gpu: 1` and are scheduled correctly, but Group B's pods that omit any GPU resource request are also landing on GPU nodes and consuming host memory and CPU, degrading Group A's jobs. The team wants Group B's non-GPU pods to stop consuming capacity on the GPU node pool without changing Group B's manifests. Which action should the administrator take?

Accepted Answer

Apply a taint such as `nvidia.com/gpu=present:NoSchedule` to the GPU nodes and add a matching toleration to Group A's pod templates.. Dedicating a node pool to GPU consumers is done at the scheduler layer: a taint on the GPU nodes repels every pod that does not carry a matching toleration. Because Group B's manifests cannot be changed, repelling rather than attracting is the only workable direction, and adding tolerations to Group A's templates is a one-time change under the platform team's control. Quotas, driver flags, and plugin options all operate after or outside scheduling and cannot keep non-GPU pods off those nodes.

Answer

Set `NVIDIA_DRIVER_CAPABILITIES=compute,utility` on the GPU Operator DaemonSet so non-GPU pods cannot attach to the device.

Answer

Create a `ResourceQuota` in each namespace that caps `requests.nvidia.com/gpu` at zero for Group B.

Answer

Enable the `nvidia` device plugin's `--fail-on-init-error=false` flag so unresolvable GPU requests fall back to CPU scheduling.

Question 10

A platform team runs a Kubernetes cluster where the NVIDIA GPU Operator is installed and time-slicing is configured with a ConfigMap that advertises four replicas per physical GPU. A data scientist submits a PyTorch training pod requesting nvidia.com/gpu: 1. The pod stays Pending indefinitely, and the scheduler event reads 'Insufficient nvidia.com/gpu'. The node's GPUs are otherwise idle and healthy. Which action most directly resolves the pending state?

Accepted Answer

Verify the time-slicing ConfigMap is labeled for the device plugin and that the node's GPU capacity now reports the multiplied replica count, then requeue the pod.. Time-slicing is enabled by a ConfigMap that the NVIDIA device plugin watches; without the expected label the plugin ignores it and keeps advertising one GPU per device. Confirming the ConfigMap is labeled and that the node's allocatable nvidia.com/gpu equals replicas multiplied by physical GPUs verifies the change propagated. Once capacity is correctly republished, the pending pod can be scheduled without altering its resource request.

Answer

Restart the nvidia-device-plugin pod so it re-reads the time-slicing ConfigMap and republishes the replicated resource capacity to the kubelet.

Answer

Add a nodeSelector that pins the pod to a specific GPU node, because the scheduler cannot match GPU requests without an explicit node affinity rule.

Answer

Change the pod's resource request to nvidia.com/gpu.shared: 1 so it matches the resource name that time-slicing exposes.

Question 11

A cluster runs mixed workloads: latency-sensitive inference services and best-effort batch jobs. Administrators observe that batch jobs occasionally occupy all GPUs, causing inference requests to queue and breach service level objectives. They want inference pods to be admitted immediately while allowing batch work to use remaining capacity and be preempted when needed. Which approach should they implement?

Accepted Answer

Define priority classes and a preemption policy so inference pods have higher priority than batch jobs.. The requirement is immediate admission for inference and preemptible use of leftover capacity by batch work. Priority classes with preemption give inference pods the ability to displace lower-priority batch pods when GPUs are scarce, while batch jobs still consume idle capacity during quiet periods. Affinity, quotas, and time-slicing each constrain or share resources but cannot prioritize or preempt running workloads.

Answer

Enable time-slicing on all GPUs so batch and inference pods share devices.

Answer

Set a ResourceQuota on the batch namespace limiting total GPU requests.

Answer

Apply node affinity so inference pods only run on a dedicated subset of nodes.

Question 12

A platform team runs mixed training and inference workloads on a Kubernetes cluster with the NVIDIA GPU Operator. Inference pods are latency-sensitive and must not be preempted, while training pods can be interrupted and restarted. The team wants training jobs to yield GPUs to inference jobs when capacity is scarce, without manual intervention. Which Kubernetes mechanism should the team configure to achieve this behavior?

Accepted Answer

PriorityClass with preemptionPolicy set to PreemptLowerPriority on the inference pods.. Priority and preemption are the scheduler features designed for exactly this pattern. Assigning inference pods a higher-priority class with preemption enabled lets the scheduler evict lower-priority training pods when a node lacks free GPUs. Because training pods are restartable, the disruption is acceptable, and inference keeps its latency guarantees without manual intervention.

Answer

A PodDisruptionBudget on the training pods that guarantees a minimum number of running replicas.

Answer

ResourceQuota on the training namespace limiting total GPU requests below cluster capacity.

Answer

A node affinity rule on the inference pods that targets nodes with the highest available GPU memory.

Question 13

An administrator is using NVIDIA Base Command Manager to manage a cluster with a mix of GPU and CPU nodes. They need to ensure that a newly added GPU node is correctly recognized and that jobs can be scheduled on it. Which TWO actions must be performed to integrate the new node into the Base Command Manager cluster? (Choose two.)

Accepted Answer

Install the Base Command Manager agent on the new node and register it with the head node.. Integrating a new node into a Base Command Manager cluster requires installing and registering the Base Command Manager agent so the head node can manage it, and defining the node in the cluster configuration with an appropriate node group assignment so it inherits policies and becomes schedulable. These two actions together ensure the node is recognized, provisioned correctly, and available for job scheduling.

Answer

Add the node's hostname and IP address to the /etc/hosts file on all cluster nodes.

Answer

Configure a static IP address for the node and add it to the cluster's DNS server.

Answer

Manually install the NVIDIA GPU driver on the node without using Base Command Manager's provisioning tools.

Question 14

A platform team runs an on-premises Kubernetes cluster for AI inference. Several teams submit pods that request the same GPU device, and the scheduler places more pods onto a node than there are available GPUs, causing OOM errors on the device. The administrator wants the Kubernetes scheduler itself to prevent overcommitting GPUs without any custom admission controller. Which action should the administrator take?

Accepted Answer

Install the NVIDIA device plugin so GPUs are advertised as schedulable extended resources and add a matching resource request to each pod spec.. Advertising GPUs as extended resources through the NVIDIA device plugin is what lets the default scheduler treat each device as countable, finite node capacity. Pods that declare a GPU request are then placed only where an unallocated device exists, and no custom admission controller is required because the scheduler performs the arithmetic itself. Quotas, taints, and autoscalers influence eligibility or CPU and memory sizing, not device-level accounting.

Answer

Set a node taint on each GPU node and add a toleration to pods, so only pods that explicitly opt in are placed there.

Answer

Enable the Kubernetes Vertical Pod Autoscaler in recommendation mode on the GPU namespaces to detect device pressure and reschedule pods.

Answer

Create a ResourceQuota in each team namespace limiting the count of pods that may run, so fewer pods land on GPU nodes.

Question 15

A research group submits a distributed PyTorch training job spanning eight GPUs across two nodes. The job completes but produces a model with accuracy far below the single-node baseline, and logs show that several ranks started training before their peers had initialized the process group. The administrator must ensure that all ranks are launched together and that a failed rank terminates the whole job. Which combination of Kubernetes mechanisms should be used?

Accepted Answer

Use a gang-scheduling mechanism such as Volcano or the Kubeflow Training Operator with PyTorchJob and configure the rendezvous endpoint so all worker replicas are admitted together.. Distributed training needs collective admission and coordinated rank metadata. A gang scheduler plus the Kubeflow Training Operator's PyTorchJob admits all replicas together and injects rendezvous details, eliminating the race where early ranks initialize before their peers, while the controller fails the whole job if any replica cannot run.

Answer

Schedule the job on a single node with eight GPUs and set the CUDA_VISIBLE_DEVICES variable per container to isolate each rank.

Answer

Create eight separate Deployments, one per rank, and pass the rank index through an environment variable in each Deployment manifest.

Answer

Deploy the job as a Kubernetes Job with parallelism set to eight and completions set to one, relying on the default pod startup ordering.