NCP-AIO Workload Management Practice Test 7 — 15 Questions

Question 1

A platform team is preparing a Kubernetes cluster for AI workloads and wants the GPU device plugin, driver containers, and monitoring components deployed and kept in sync automatically on every GPU node. Which component should be installed to achieve this?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

NVIDIA GPU Operator. The NVIDIA GPU Operator is purpose-built to manage the entire GPU software stack on Kubernetes nodes through automated reconciliation. It deploys the driver, container toolkit configuration, device plugin, and DCGM-based monitoring, so GPU resources are advertised and maintained without manual per-node work. The Network Operator addresses networking, while DCGM and the Container Toolkit are individual pieces the Operator already orchestrates.

Answer

NVIDIA Network Operator

Answer

NVIDIA DCGM standalone on each node

Answer

NVIDIA Container Toolkit installed manually on each node

Question 2

An MLOps engineer manages a Kubernetes cluster where the NVIDIA GPU Operator runs the MIG manager. Several inference pods must each receive an isolated, fixed slice of a single A100, and the team wants the slices to survive node reboots without manual reconfiguration. Which combination of settings should the engineer apply?

Accepted Answer

Configure a MIG config labeled for the MIG manager with named profiles and set the device plugin strategy to 'mixed', so whole GPUs and MIG instances are both advertised.. Persistent MIG slices require the MIG manager to read a labeled configuration that defines the desired geometry, which it reapplies after reboots and driver reloads. Setting the device plugin strategy to mixed lets both whole GPUs and the carved MIG instances be advertised under distinct resource names. Pods can then request a specific profile such as nvidia.com/mig-1g.5gb, obtaining fixed, hardware-isolated slices that meet the isolation and persistence requirements.

Answer

Set the MIG manager's config to 'all-disabled' and rely on the device plugin to carve profiles dynamically per pod request.

Answer

Disable the MIG manager entirely and create MIG instances manually with nvidia-smi on each node, then label the nodes so pods schedule there.

Answer

Enable time-slicing with four replicas and set the MIG manager strategy to 'single', because replicas provide the same isolation as MIG instances.

Question 3

A research team is submitting many short-lived experiment jobs to an NVIDIA-accelerated Kubernetes cluster. The operations team wants to reduce GPU idle time and improve overall utilization without modifying the training code. Which TWO approaches should the operations team implement? (Choose two.)

Accepted Answer

Use a gang-scheduling or batch scheduler with backfill so queued jobs start as soon as GPUs free up. Short-lived experiment jobs create idle GPU gaps. Time-slicing allows multiple such pods to share a device, and a batch scheduler with backfill keeps the queue moving so GPUs are assigned as soon as capacity frees. Both measures raise utilization without touching training code, which is exactly what the operations team needs.

Answer

Pin each experiment job to a dedicated physical GPU using node affinity

Answer

Disable the NVIDIA device plugin and mount GPUs directly via hostPath

Answer

Increase the pod's nvidia.com/gpu request to reserve more GPU memory

Question 4

A cloud operations team is using NVIDIA AI Enterprise with Kubernetes to deploy inference workloads. They want to ensure that GPU resources are allocated to pods only when explicitly requested, and that pods without GPU requests do not consume GPU resources. Which Kubernetes feature should they use to enforce this behavior?

Accepted Answer

Define resource requests and limits for 'nvidia.com/gpu' in the pod specification.. Kubernetes requires pods to explicitly request GPU resources using the 'nvidia.com/gpu' resource name in their resource specifications. The scheduler then allocates GPUs only to those pods, ensuring that pods without such requests do not consume GPU resources even if they run on GPU nodes. This mechanism enforces the desired explicit allocation policy and is the standard way to manage GPU resources in Kubernetes.

Answer

Use node selectors to schedule pods only on GPU nodes.

Answer

Enable the GPU Operator's 'device plugin' to automatically inject GPU requests into all pods.

Answer

Use a mutating admission webhook to add GPU requests to pods based on their namespace.

Question 5

An AI operations engineer is troubleshooting a Kubernetes cluster where several GPU training pods fail to start with a device plugin allocation error, even though the nodes report healthy GPUs. The engineer suspects the pods are requesting more GPU resources than a single physical card can provide without a sharing mechanism. Which TWO configurations would legitimately allow multiple pods to consume a single physical GPU on these nodes? (Choose two.)

Accepted Answer

Enable NVIDIA Multi-Instance GPU (MIG) on supported GPUs and expose the MIG instances as schedulable resources.. Both MIG and time-slicing change how the device plugin advertises and allocates a physical GPU, enabling concurrent consumption by more than one pod. MIG offers hardware-partitioned, isolated instances, while time-slicing interleaves workloads on the whole card. Each is a supported sharing mode configured through the GPU Operator, unlike driver upgrades, node selectors, or quota edits, which do not alter device advertisement.

Answer

Add a node selector that allows multiple pods to bind to the same nvidia.com/gpu resource slot.

Answer

Create a ResourceQuota that counts GPU requests as fractional values such as 0.5 per pod.

Answer

Raise the GPU Operator's driver version so the device plugin reports extra virtual GPUs per card.

Question 6

An administrator supports a shared inference cluster where a single A100 GPU must serve several small models concurrently. They configure the NVIDIA device plugin with a time-slicing configuration that advertises multiple replicas of the same physical device. After deployment, users report that one noisy model starves the others and latency spikes unpredictably. Which statement best explains the observed behaviour?

Accepted Answer

Time-slicing interleaves work on one physical GPU with no memory or fault isolation, so a heavy workload can dominate device time and memory bandwidth.. Time-slicing creates multiple schedulable replicas of one physical GPU, which improves packing density but provides no hardware-level isolation. All replicas contend for the same memory, caches, and execution bandwidth, so a dominant workload can starve lighter ones and produce erratic tail latency. Where strict isolation or predictable performance is required, hardware partitioning such as Multi-Instance GPU is the appropriate mechanism instead of replica multiplexing.

Answer

Multi-Instance GPU mode must be enabled alongside time-slicing, because the two modes are designed to be combined on the same device.

Answer

The device plugin failed to register replicas, so the scheduler placed all pods on one logical device instead of distributing them.

Answer

The pods lack a RuntimeClass reference, so the NVIDIA container runtime never applies the configured replica strategy at container start.

Question 7

A platform engineer manages an NVIDIA-accelerated Kubernetes cluster running the NVIDIA GPU Operator on nodes with A100 GPUs. Several data-science teams submit training jobs, and the engineer must ensure each team's pods receive a full physical GPU exclusively, with no two pods sharing the same device. Which scheduling configuration should the engineer apply to the pod specification to guarantee exclusive whole-GPU allocation?

Accepted Answer

Set resources.limits to nvidia.com/gpu: 1 and resources.requests to nvidia.com/gpu: 1, letting the NVIDIA device plugin allocate an exclusive GPU.. Declaring both a request and a limit for nvidia.com/gpu equal to 1 is the canonical way to obtain an exclusive whole GPU in Kubernetes. The NVIDIA device plugin advertises GPUs as integer extended resources, so the scheduler reserves one entire device for the pod. Because extended resources cannot be fractional, no co-scheduling or sharing occurs, which precisely meets the exclusive allocation requirement.

Answer

Enable MIG by setting the nvidia.com/mig.config label on the node to all-balanced and request nvidia.com/mig-1g.5gb in the pod.

Answer

Configure the pod to mount the hostPath /dev/nvidia0 and set privileged: true so the container can access the first GPU directly.

Answer

Set a nodeSelector for nvidia.com/gpu.product=A100 and rely on the scheduler to place one pod per node automatically.

Question 8

An administrator is configuring a Kubernetes cluster where some nodes have A100 GPUs and others have H100 GPUs. A training job requires specific GPU memory capacity and CUDA compute capability. Which mechanism should the administrator use to ensure the job is only scheduled onto nodes with the correct GPU model?

Accepted Answer

Set a nodeSelector matching the GPU model label applied by the GPU Operator's node feature discovery. Node Feature Discovery, part of the GPU Operator stack, labels nodes with GPU product details such as nvidia.com/gpu.product. A nodeSelector or affinity rule referencing that label restricts scheduling to nodes with the required GPU model, which is the precise way to satisfy memory and compute capability requirements without manual node naming.

Answer

Apply a taint to nodes without the required GPU and rely on the default scheduler

Answer

Increase the nvidia.com/gpu resource request to match the GPU memory size

Answer

Use a ResourceQuota that limits GPU requests per namespace

Question 9

An administrator manages a Kubernetes cluster where the NVIDIA GPU Operator has deployed the device plugin and MIG Manager. A tenant wants to run several small inference services that each need only a fraction of a GPU, isolated from other tenants' memory and fault domains. The administrator decides to use Multi-Instance GPU mode. Which TWO actions must be performed to make MIG-backed GPU resources schedulable to those pods? (Choose two.)

Accepted Answer

Enable MIG mode on the physical GPU and define a MIG profile/geometry via the MIG Manager configuration. MIG-backed scheduling requires two coordinated steps: enabling MIG mode and applying a geometry through the MIG Manager so instances exist, and having pods request the specific profile resource name that the device plugin advertises, such as nvidia.com/mig-1g.5gb. Together these produce isolated GPU slices with dedicated memory and fault domains for each tenant's inference service.

Answer

Install the NVIDIA Network Operator and enable RDMA device advertisement on the node

Answer

Create a RuntimeClass that points to the nvidia-container-runtime and reference it from each inference pod

Answer

Apply a ResourceQuota that limits nvidia.com/gpu to zero in the tenant namespace

Question 10

An AI operations team runs a shared Kubernetes cluster with the NVIDIA GPU Operator and several namespaces owned by different groups. A group reports that its training pods are stuck Pending with an event indicating insufficient nvidia.com/gpu, yet cluster-wide dashboards show many GPUs idle. Investigation reveals the idle GPUs belong to nodes labeled for another group, and the affected namespace has a node affinity rule pinning its pods to a specific GPU generation that is fully consumed. Which action best resolves the Pending pods while respecting multi-tenant boundaries?

Accepted Answer

Provision additional GPUs of the required generation for that tenant or rebalance existing capacity to that node pool.. The pods are constrained by a node affinity rule targeting a specific GPU generation whose nodes are fully allocated. Idle GPUs on other nodes cannot satisfy that rule. The correct fix is to add capacity of the required generation or rebalance matching cards into that pool, which resolves the shortage while preserving the tenant isolation the affinity rule enforces.

Answer

Disable the device plugin on the other tenants' nodes so their GPUs become available to the affected namespace.

Answer

Lower the GPU request in the pod spec so the scheduler accepts a partial device from an idle node.

Answer

Remove the node affinity rule so the pods can schedule onto any idle GPU node in the cluster.

Question 11

A data science team submits a PyTorch training job to a Kubernetes cluster managed by Run:ai. The job requests two GPUs but only one is allocated, and the second worker hangs waiting for a peer. Which Run:ai capability should the administrator verify is configured so the distributed job receives all requested GPUs atomically?

Accepted Answer

Gang scheduling, which ensures all pods in a distributed job are scheduled together or not at all.. Distributed training depends on every rank being present before collective operations begin; a single missing worker causes the rest to block. Run:ai's gang scheduling is the mechanism that admits the entire workload as a unit, so all requested GPUs are granted together. Confirming gang scheduling is enabled and applied to the job resolves the partial allocation that produced the hang.

Answer

A higher priority class assigned to the training job so it preempts other workloads.

Answer

Node affinity rules that pin each worker to a specific GPU node by hostname.

Answer

Enabling time-slicing on the GPU device plugin to increase the apparent GPU count.

Question 12

An administrator manages a cluster where inference services and batch training share the same GPU nodes. During business hours, inference pods must be scheduled promptly, while training jobs can wait. The administrator wants preemption so that a pending high-priority inference pod can evict a lower-priority training pod when no GPU is free, with evicted training resuming later. Which configuration achieves this?

Accepted Answer

Create a PriorityClass with a high value for inference and a lower one for training, reference them in the pod specs, and enable preemption in the scheduler so lower-priority pods are evicted when needed.. Priority and preemption are the native scheduling features designed for exactly this pattern: a higher-priority pending pod triggers eviction of lower-priority running pods to free capacity. Defining distinct PriorityClasses for inference and training and referencing them in pod specs enables the scheduler to make that decision. Because the training workload is managed by a group-aware controller, the evicted job can be requeued and restarted from its last checkpoint.

Answer

Set the training pods' terminationGracePeriodSeconds to zero and add a PodDisruptionBudget so the scheduler can remove them immediately when capacity is needed.

Answer

Apply node affinity rules that pin inference pods to dedicated nodes and training pods to separate nodes, then let each workload schedule only within its own pool.

Answer

Define a ResourceQuota on the training namespace capping GPU requests so that free capacity always remains available for inference pods.

Question 13

An ML platform team runs an NVIDIA GPU Operator-managed cluster and wants to allow multiple pods to share a single A100 GPU so that small inference services can co-reside without each consuming a whole device. The team needs a time-slicing configuration that applies to all GPU nodes in the cluster. Which approach should the administrator take?

Accepted Answer

Create a ConfigMap containing the time-slicing configuration and reference it in the ClusterPolicy so the GPU Operator propagates the device plugin config across nodes.. Time-slicing in the NVIDIA GPU Operator is driven by a device-plugin configuration that the operator distributes to every GPU node through the ClusterPolicy. The referenced ConfigMap declares a replica count, and the device plugin then advertises that many virtual GPUs per physical device, allowing the scheduler to place multiple pods on one GPU. This is the supported, cluster-wide mechanism for enabling time-sliced sharing.

Answer

Add a tolerations entry for nvidia.com/gpu to each pod and increase the kubelet's --max-pods flag on GPU nodes.

Answer

Set the environment variable NVIDIA_VISIBLE_DEVICES=all on each pod and let the runtime divide GPU time among containers.

Answer

Install the NVIDIA MIG Manager and set the nvidia.com/mig.config label to all-1g.5gb on each node to subdivide the GPUs.

Question 14

A data science team submits a PyTorch training job to a Kubernetes cluster where the NVIDIA GPU Operator is installed. The pod stays in Pending state, and kubectl describe shows the message '0/6 nodes are available: 6 Insufficient nvidia.com/gpu.' The administrator confirms the nodes have healthy GPUs and the device plugin pods are Running. What is the most likely cause?

Accepted Answer

The pod requested more nvidia.com/gpu resources than any single node can advertise. Extended resources like nvidia.com/gpu are advertised per node and cannot be aggregated across nodes, so a request exceeding any single node's advertised count leaves the pod unschedulable with an insufficient-resource message. Verifying the per-node GPU count and lowering the pod's request to fit within one node restores scheduling.

Answer

The container image does not include the NVIDIA CUDA base layer

Answer

The pod lacks a nodeSelector matching the GPU node label nvidia.com/gpu.present

Answer

The GPU Operator's driver container has not yet built the kernel module on the nodes

Question 15

An AI operations engineer manages a Kubernetes cluster running the NVIDIA GPU Operator. A team wants its long-running inference deployment to be automatically rescheduled if the GPU on a node develops an uncorrectable error that the device plugin or health checks detect. The team also wants the node to stop accepting new GPU pods until the issue is resolved. Which combination of behaviors should the engineer rely on to meet these requirements?

Accepted Answer

The GPU Operator's health checks mark the GPU unhealthy, the device plugin stops advertising it, and Kubernetes taints or cordons the affected node so pods are rescheduled and no new GPU pods land there.. Meeting both requirements needs health-driven device withdrawal plus node-level scheduling exclusion. The GPU Operator's health checks detect uncorrectable errors, the device plugin stops advertising the faulty GPU, and the node is tainted or cordoned. Controllers then recreate pods on healthy nodes, and the taint keeps new GPU pods away until the fault is cleared.

Answer

A Horizontal Pod Autoscaler scales up replicas so healthy copies absorb traffic while the faulty node remains in service.

Answer

A liveness probe on the inference container restarts the pod on the same node when the GPU error causes a request failure.

Answer

A PodDisruptionBudget on the inference deployment forces the scheduler to migrate pods off the node when the GPU fails.