NCP-AIO Workload Management Practice Test 5 — 15 Questions

Question 1

When managing GPU resources in a shared cluster, which configuration best prevents 'noisy neighbor' scenarios where one GPU task consumes all available memory bandwidth?

Accepted Answer

Implementing NVIDIA MIG and strictly defining resource limits.. Using a combination of NVIDIA MIG and Kubernetes resource quotas provides the strongest isolation. By enforcing hardware-level partitioning via MIG, you ensure that memory bandwidth is strictly bounded for each slice. This is vital for AI Ops because it prevents high-throughput training jobs from starving small inference requests, ensuring stable latency across the entire cluster environment and improving total system reliability.

Answer

Increasing the CUDA_VISIBLE_DEVICES environment variable.

Answer

Setting a higher priority class for inference pods.

Answer

Configuring the NVIDIA DCGM Exporter alerts.

Question 2

Refer to the exhibit. The cluster administrator observes near-capacity memory utilization across three GPUs. What is the most likely consequence if an additional pod is scheduled to these GPUs without memory partitioning?

Accepted Answer

The job will experience OOM errors and fail.. Scheduling an additional pod onto these GPUs will lead to Out-of-Memory (OOM) errors. Because the GPUs are already at near-maximum capacity, the driver will likely kill the incoming process or potentially destabilize existing tasks. This highlights the importance of workload monitoring and resource planning, as AI Ops must ensure that requests match actual hardware capacity to prevent catastrophic job failures in production environments.

Answer

The scheduler will automatically enable MIG.

Answer

The scheduler will use CPU swap memory.

Answer

The job will be throttled but complete successfully.

Question 3

Which strategy is most effective for managing heterogeneous GPU clusters containing both older architectures (e.g., V100) and newer architectures (e.g., H100)?

Accepted Answer

Using Node Selectors and Taints/Tolerations.. Using Kubernetes Node Selectors and Taints/Tolerations allows administrators to route specific workloads to compatible hardware. This is essential because newer GPUs support features like MIG or specific precision formats that older hardware lacks. By properly tagging nodes based on architecture, AI Ops prevents 'Incompatible Device' errors and ensures that high-performance training jobs are scheduled on hardware that meets their specific computational requirements, optimizing cluster-wide performance and efficiency.

Answer

Enabling uniform scheduling across all nodes.

Answer

Configuring the NVIDIA Device Plugin to hide older GPUs.

Answer

Deploying a single unified container image for all jobs.

Question 4

Which THREE factors should be considered when estimating GPU memory requirements for a Large Language Model (LLM) fine-tuning job?

Accepted Answer

The memory size of the model parameters.. When sizing LLM workloads, you must account for the model weights, optimizer states, and gradient buffers. Additionally, activations consume significant memory during the forward and backward passes. Understanding these components is essential for AI Ops, as incorrect sizing leads to OOM crashes early in the training process, wasting significant compute time and delaying model delivery in production environments.

Answer

The total number of CPU threads used.

Answer

The network latency between nodes.

Question 5

An administrator is optimizing multi-GPU utilization. Which TWO of the following configurations allow multiple containers to share a single physical GPU on a supported NVIDIA architecture?

Accepted Answer

Enabling NVIDIA Time-Slicing in the GPU Operator configuration.. Multi-Instance GPU (MIG) and Time-Slicing are the primary methods for sharing physical GPU resources. MIG provides hardware-level isolation, while Time-Slicing provides software-based multiplexing. Understanding these options is essential for AI operations, as it allows administrators to maximize hardware ROI by supporting smaller workloads that do not require an entire A100 or H100 GPU, effectively increasing the density of the training or inference environment.

Answer

Increasing the CUDA_VISIBLE_DEVICES environment variable.

Answer

Setting a higher GPU limit in the Kubernetes manifest.

Answer

Deploying the NVIDIA Network Operator.

Question 6

Which component of the NVIDIA GPU Operator is responsible for monitoring GPU health and reporting telemetry data to the Kubernetes control plane?

Accepted Answer

NVIDIA DCGM Exporter.. The DCGM Exporter is the primary tool within the NVIDIA GPU Operator ecosystem for collecting metrics. It interacts with the Data Center GPU Manager (DCGM) to gather telemetry such as utilization, temperature, and memory health. This is vital for AI Operations because it provides the observability necessary to trigger auto-scaling, identify failing hardware, and ensure that training jobs are performing optimally within the cluster environment.

Answer

NVIDIA Device Plugin.

Answer

NVIDIA Container Toolkit.

Answer

NVIDIA Node Feature Discovery.

Question 7

Refer to the exhibit. An administrator is attempting to deploy a job to a namespace with a ResourceQuota defined. What is the cause of this error?

Accepted Answer

The pod manifest is missing the required GPU resource limits.. The error indicates that the namespace has a ResourceQuota enforcing that all pods must specify GPU limits, but the submitted pod manifest lacks a 'resources.limits.nvidia.com/gpu' entry. In multi-tenant environments, ResourceQuotas are essential for preventing a single user from consuming the entire GPU capacity. The manifest must include a valid GPU limit to satisfy the namespace policy, ensuring the cluster remains balanced across different organizational teams.

Answer

The GPU driver version is incompatible with the quota controller.

Answer

The cluster is out of available GPU capacity.

Answer

The NVIDIA Device Plugin is not running in the namespace.

Question 8

When deploying large-scale distributed training jobs, why is it recommended to use the NVIDIA Network Operator in conjunction with the GPU Operator?

Accepted Answer

To optimize RDMA throughput for distributed training.. The Network Operator automates the configuration of high-speed interconnects like InfiniBand or RoCE, which are crucial for distributed training across multiple nodes. By aligning network resources with GPU placement, the operator minimizes latency and maximizes throughput. This ensures that the GPU compute capacity is not bottlenecked by slow network communication, which is essential for scaling complex AI model training across large clusters effectively.

Answer

To increase the number of supported GPU containers.

Answer

To provide automatic GPU driver updates.

Answer

To enable multi-tenancy on GPU nodes.

Question 9

An AI operations team runs mixed training and inference workloads on a Kubernetes cluster managed with the NVIDIA GPU Operator. Inference pods frequently arrive in bursts and must start within seconds, while long-running training jobs occupy most MIG-capable A100 GPUs for days. Administrators want burst inference pods to obtain GPU capacity immediately without preempting or restarting the training jobs, and they want the cluster to reclaim those resources automatically when the burst ends. Which approach best satisfies these requirements?

Accepted Answer

Create a node pool reserved for inference, and configure a cluster autoscaler with scale-to-zero so burst pods trigger new GPU nodes that are removed when idle.. The scenario requires two properties simultaneously: burst inference capacity that appears without disturbing running training jobs, and automatic reclamation when the burst ends. Only a dedicated autoscaled node pool satisfies both, because new GPU nodes are provisioned on demand for inference pods while training pods remain scheduled and running on their existing nodes. When demand drops, scale-to-zero removes the extra nodes, freeing GPU resources without any preemption or job restarts.

Answer

Configure the inference Deployment with a topologySpreadConstraint across all GPU nodes so its replicas distribute evenly and find free capacity.

Answer

Enable the GPU Operator's time-slicing configuration so inference and training containers share the same physical GPUs through a shared device plugin.

Answer

Deploy inference pods with a higher PriorityClass and enable preemption so the scheduler evicts lower-priority training pods when capacity is exhausted.

Question 10

A Kubernetes cluster running the NVIDIA GPU Operator is shared by an inference team and a research team. The research team's training pods repeatedly evict the inference pods from GPUs, causing latency spikes in production. The administrator wants to guarantee that inference pods always get GPU access first. Which Kubernetes scheduling mechanism should be configured?

Accepted Answer

Assign a higher PriorityClass to the inference pods and enable preemption on the scheduler.. Priority and preemption are the native Kubernetes controls that decide which pods win when GPU resources are contended. Giving inference pods a higher PriorityClass lets the scheduler admit them first and evict lower-priority training pods when necessary, which directly satisfies the requirement that production inference always obtains GPU access.

Answer

Enable the NVIDIA MIG feature on all GPUs and dedicate a MIG instance to each inference pod.

Answer

Configure a taint on the GPU nodes and add the corresponding toleration only to the inference pods.

Answer

Create a PodDisruptionBudget for the inference deployment and set maxUnavailable to zero.

Question 11

An administrator must run a batch inference job that requires exactly two NVIDIA GPUs on a Kubernetes cluster managed by the NVIDIA GPU Operator. Which pod specification field should be used to request those GPUs?

Accepted Answer

resources.limits with the key nvidia.com/gpu set to 2. GPUs exposed by the NVIDIA device plugin appear to Kubernetes as the extended resource nvidia.com/gpu. Extended resources must be declared in resources.limits, and setting that limit to 2 makes the scheduler reserve two GPUs on a suitable node and pass the devices into the container.

Answer

resources.requests with the key nvidia.com/gpu set to 2 and no limit

Answer

annotations with the key nvidia.com/gpu.count set to 2

Answer

nodeSelector with the key nvidia.com/gpu.count set to 2

Question 12

A platform team operates a Kubernetes cluster where several teams submit GPU training jobs. The administrator needs to enforce per-namespace limits on the number of GPUs that can be consumed and prevent a single namespace from monopolizing all GPU capacity. Which TWO Kubernetes resources should be configured to achieve this? (Choose two.)

Accepted Answer

A LimitRange that sets a default and maximum nvidia.com/gpu value for containers in the namespace.. ResourceQuota enforces an aggregate ceiling on nvidia.com/gpu per namespace, while LimitRange constrains and defaults the per-container GPU request. Together they bound total namespace consumption and prevent any single container from grabbing an outsized share, which is exactly the governance the platform team needs.

Answer

A PriorityClass that assigns a low priority value to all training pods in the namespace.

Answer

A PodSecurityPolicy that denies privileged containers in the namespace.

Answer

A NetworkPolicy that restricts traffic between pods in different namespaces.

Question 13

An administrator runs a Kubernetes cluster with the NVIDIA GPU Operator. A data science team wants to run several small inference containers that each use only a fraction of a GPU's compute and memory, but the cluster currently assigns whole GPUs per pod. Which approach allows multiple containers to share a single physical GPU with memory isolation?

Accepted Answer

Enable Multi-Instance GPU mode on supported GPUs and configure the device plugin to advertise MIG instances as schedulable resources.. MIG is the only mechanism here that divides a physical GPU into hardware-isolated instances with separate memory and compute slices. By enabling MIG and having the device plugin advertise each instance, the scheduler can place multiple inference containers on one GPU while keeping their memory and faults isolated.

Answer

Reduce the container's nvidia.com/gpu request to a fractional value such as 0.5.

Answer

Set the device plugin's time-slicing configuration so that each GPU is advertised multiple times.

Answer

Configure the MPS control daemon to allow concurrent kernel execution across containers.

Question 14

A platform team runs an NVIDIA AI cluster with the GPU Operator deployed. Users submit jobs directly with kubectl and frequently request whole GPUs even when their notebooks only need a fraction of one. The team wants Kubernetes itself to admit and queue jobs based on GPU demand without users changing their manifests. Which component should the team deploy to meet this requirement?

Accepted Answer

NVIDIA KAI Scheduler. The requirement is for Kubernetes to make admission and queueing decisions based on GPU demand while users submit normal manifests. KAI Scheduler is the NVIDIA component that provides queue-based, gang-aware scheduling for AI workloads and understands fractional GPU requests, so jobs wait rather than fail. The GPU Operator, MIG Manager, and DCGM Exporter each address installation, partitioning, or observability rather than scheduling admission.

Answer

NVIDIA MIG Manager configured for every GPU

Answer

NVIDIA GPU Operator with the device plugin enabled

Answer

NVIDIA DCGM Exporter with Prometheus alerting

Question 15

A researcher submits a distributed training job that spans four pods, each needing one GPU, and the pods must start together or not at all. The administrator wants Kubernetes to schedule all four pods only when four GPUs are simultaneously available. Which workload management construct should be used?

Accepted Answer

A PodGroup managed by a scheduler that supports gang scheduling, such as Volcano or the scheduler-plugins coscheduling plugin.. Distributed training needs gang scheduling so that either every worker pod is placed or none is. A PodGroup interpreted by a gang-aware scheduler like Volcano or the coscheduling plugin holds the pods until the full set of GPUs is available, avoiding deadlocks and wasted accelerator time.

Answer

A Kubernetes Job with parallelism set to 4 and completions set to 4.

Answer

A StatefulSet with podManagementPolicy set to Parallel.

Answer

A DaemonSet that places one training pod on each GPU node in the cluster.