NCP-AIO Practice Test 14 — 15 Questions

Question 1

An MLOps engineer manages a Kubernetes cluster where the NVIDIA GPU Operator runs the MIG manager. Several inference pods must each receive an isolated, fixed slice of a single A100, and the team wants the slices to survive node reboots without manual reconfiguration. Which combination of settings should the engineer apply?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Configure a MIG config labeled for the MIG manager with named profiles and set the device plugin strategy to 'mixed', so whole GPUs and MIG instances are both advertised.. Persistent MIG slices require the MIG manager to read a labeled configuration that defines the desired geometry, which it reapplies after reboots and driver reloads. Setting the device plugin strategy to mixed lets both whole GPUs and the carved MIG instances be advertised under distinct resource names. Pods can then request a specific profile such as nvidia.com/mig-1g.5gb, obtaining fixed, hardware-isolated slices that meet the isolation and persistence requirements.

Answer

Set the MIG manager's config to 'all-disabled' and rely on the device plugin to carve profiles dynamically per pod request.

Answer

Disable the MIG manager entirely and create MIG instances manually with nvidia-smi on each node, then label the nodes so pods schedule there.

Answer

Enable time-slicing with four replicas and set the MIG manager strategy to 'single', because replicas provide the same isolation as MIG instances.

Question 2

An AI operations engineer is troubleshooting a Kubernetes cluster where several GPU training pods fail to start with a device plugin allocation error, even though the nodes report healthy GPUs. The engineer suspects the pods are requesting more GPU resources than a single physical card can provide without a sharing mechanism. Which TWO configurations would legitimately allow multiple pods to consume a single physical GPU on these nodes? (Choose two.)

Accepted Answer

Enable NVIDIA Multi-Instance GPU (MIG) on supported GPUs and expose the MIG instances as schedulable resources.. Both MIG and time-slicing change how the device plugin advertises and allocates a physical GPU, enabling concurrent consumption by more than one pod. MIG offers hardware-partitioned, isolated instances, while time-slicing interleaves workloads on the whole card. Each is a supported sharing mode configured through the GPU Operator, unlike driver upgrades, node selectors, or quota edits, which do not alter device advertisement.

Answer

Add a node selector that allows multiple pods to bind to the same nvidia.com/gpu resource slot.

Answer

Create a ResourceQuota that counts GPU requests as fractional values such as 0.5 per pod.

Answer

Raise the GPU Operator's driver version so the device plugin reports extra virtual GPUs per card.

Question 3

A data science team submits a PyTorch training job to a Kubernetes cluster where the NVIDIA GPU Operator is installed. The pod stays in Pending state, and kubectl describe shows the message '0/6 nodes are available: 6 Insufficient nvidia.com/gpu.' The administrator confirms the nodes have healthy GPUs and the device plugin pods are Running. What is the most likely cause?

Accepted Answer

The pod requested more nvidia.com/gpu resources than any single node can advertise. Extended resources like nvidia.com/gpu are advertised per node and cannot be aggregated across nodes, so a request exceeding any single node's advertised count leaves the pod unschedulable with an insufficient-resource message. Verifying the per-node GPU count and lowering the pod's request to fit within one node restores scheduling.

Answer

The container image does not include the NVIDIA CUDA base layer

Answer

The pod lacks a nodeSelector matching the GPU node label nvidia.com/gpu.present

Answer

The GPU Operator's driver container has not yet built the kernel module on the nodes

Question 4

An administrator manages an NVIDIA AI Enterprise cluster and needs to enforce GPU resource quotas across multiple Kubernetes namespaces. Which NVIDIA component should be configured to enforce these quotas?

Accepted Answer

Kubernetes ResourceQuota with extended resources. Kubernetes ResourceQuota is the native mechanism to limit aggregate resource consumption per namespace, including extended resources like nvidia.com/gpu. When the NVIDIA device plugin advertises GPUs, administrators can define a ResourceQuota specifying a maximum for nvidia.com/gpu, thereby enforcing GPU quotas. Other NVIDIA tools focus on deployment, management, or monitoring and do not provide quota enforcement.

Answer

NVIDIA DCGM Exporter

Answer

NVIDIA Base Command Manager

Answer

NVIDIA GPU Operator

Question 5

A cloud operations team is using NVIDIA AI Enterprise with Kubernetes to deploy inference workloads. They want to ensure that GPU resources are allocated to pods only when explicitly requested, and that pods without GPU requests do not consume GPU resources. Which Kubernetes feature should they use to enforce this behavior?

Accepted Answer

Define resource requests and limits for 'nvidia.com/gpu' in the pod specification.. Kubernetes requires pods to explicitly request GPU resources using the 'nvidia.com/gpu' resource name in their resource specifications. The scheduler then allocates GPUs only to those pods, ensuring that pods without such requests do not consume GPU resources even if they run on GPU nodes. This mechanism enforces the desired explicit allocation policy and is the standard way to manage GPU resources in Kubernetes.

Answer

Use node selectors to schedule pods only on GPU nodes.

Answer

Enable the GPU Operator's 'device plugin' to automatically inject GPU requests into all pods.

Answer

Use a mutating admission webhook to add GPU requests to pods based on their namespace.

Question 6

A production inference service running on NVIDIA GPUs exhibits periodic latency spikes every few minutes, correlating with CPU-side stalls and low GPU utilization during those intervals. Profiling with Nsight Systems shows large gaps between kernel launches and frequent cudaMalloc/cudaFree calls. Which action best addresses the root cause?

Accepted Answer

Enable CUDA Graph capture for the inference sequence and reuse the captured graph for repeated executions.. The profile shows CPU-side stalls and frequent cudaMalloc/cudaFree, indicating launch and allocation overhead. CUDA Graphs capture the entire sequence and replay it with minimal CPU involvement, eliminating both the allocation churn and the launch gaps. This directly reduces latency spikes and improves GPU utilization, making it the most effective remedy for the described symptoms.

Answer

Switch the model to use TensorRT with FP16 precision to reduce kernel execution time.

Answer

Increase the number of worker threads on the CPU to better overlap preprocessing with GPU execution.

Answer

Enable Multi-Process Service (MPS) to allow concurrent kernel execution from multiple processes.

Question 7

An administrator supports a shared inference cluster where a single A100 GPU must serve several small models concurrently. They configure the NVIDIA device plugin with a time-slicing configuration that advertises multiple replicas of the same physical device. After deployment, users report that one noisy model starves the others and latency spikes unpredictably. Which statement best explains the observed behaviour?

Accepted Answer

Time-slicing interleaves work on one physical GPU with no memory or fault isolation, so a heavy workload can dominate device time and memory bandwidth.. Time-slicing creates multiple schedulable replicas of one physical GPU, which improves packing density but provides no hardware-level isolation. All replicas contend for the same memory, caches, and execution bandwidth, so a dominant workload can starve lighter ones and produce erratic tail latency. Where strict isolation or predictable performance is required, hardware partitioning such as Multi-Instance GPU is the appropriate mechanism instead of replica multiplexing.

Answer

Multi-Instance GPU mode must be enabled alongside time-slicing, because the two modes are designed to be combined on the same device.

Answer

The device plugin failed to register replicas, so the scheduler placed all pods on one logical device instead of distributing them.

Answer

The pods lack a RuntimeClass reference, so the NVIDIA container runtime never applies the configured replica strategy at container start.

Question 8

An AI operations team runs long-running training jobs on a Kubernetes cluster with NVIDIA GPU Operator. They observe that after a node is rebooted for maintenance, some pods resume but report CUDA 'unknown error' and the device plugin shows unhealthy GPUs. Which configuration should the administrator review to ensure the driver and device plugin recover cleanly after reboot?

Accepted Answer

The GPU Operator's driver validation and node reboot handling settings, including the driver DaemonSet and readiness gates.. A reboot invalidates the previously loaded driver and device plugin state, so the GPU Operator must reload the driver and re-register devices before pods can use them. Driver validation and readiness gates hold workloads until health checks pass, preventing CUDA errors from reaching applications. Reviewing these settings ensures clean recovery and avoids the unhealthy plugin state observed after maintenance.

Answer

The pod's restartPolicy set to Always so containers restart automatically after node recovery.

Answer

The cluster autoscaler settings so replacement nodes are provisioned immediately after reboot.

Answer

The container image's CUDA version to ensure it matches the host driver version.

Question 9

An administrator manages a Kubernetes cluster where the NVIDIA GPU Operator has deployed the device plugin and MIG Manager. A tenant wants to run several small inference services that each need only a fraction of a GPU, isolated from other tenants' memory and fault domains. The administrator decides to use Multi-Instance GPU mode. Which TWO actions must be performed to make MIG-backed GPU resources schedulable to those pods? (Choose two.)

Accepted Answer

Enable MIG mode on the physical GPU and define a MIG profile/geometry via the MIG Manager configuration. MIG-backed scheduling requires two coordinated steps: enabling MIG mode and applying a geometry through the MIG Manager so instances exist, and having pods request the specific profile resource name that the device plugin advertises, such as nvidia.com/mig-1g.5gb. Together these produce isolated GPU slices with dedicated memory and fault domains for each tenant's inference service.

Answer

Install the NVIDIA Network Operator and enable RDMA device advertisement on the node

Answer

Create a RuntimeClass that points to the nvidia-container-runtime and reference it from each inference pod

Answer

Apply a ResourceQuota that limits nvidia.com/gpu to zero in the tenant namespace

Question 10

An administrator is configuring a Kubernetes cluster where some nodes have A100 GPUs and others have H100 GPUs. A training job requires specific GPU memory capacity and CUDA compute capability. Which mechanism should the administrator use to ensure the job is only scheduled onto nodes with the correct GPU model?

Accepted Answer

Set a nodeSelector matching the GPU model label applied by the GPU Operator's node feature discovery. Node Feature Discovery, part of the GPU Operator stack, labels nodes with GPU product details such as nvidia.com/gpu.product. A nodeSelector or affinity rule referencing that label restricts scheduling to nodes with the required GPU model, which is the precise way to satisfy memory and compute capability requirements without manual node naming.

Answer

Apply a taint to nodes without the required GPU and rely on the default scheduler

Answer

Increase the nvidia.com/gpu resource request to match the GPU memory size

Answer

Use a ResourceQuota that limits GPU requests per namespace

Question 11

An AI operations team runs a shared Kubernetes cluster with the NVIDIA GPU Operator and several namespaces owned by different groups. A group reports that its training pods are stuck Pending with an event indicating insufficient nvidia.com/gpu, yet cluster-wide dashboards show many GPUs idle. Investigation reveals the idle GPUs belong to nodes labeled for another group, and the affected namespace has a node affinity rule pinning its pods to a specific GPU generation that is fully consumed. Which action best resolves the Pending pods while respecting multi-tenant boundaries?

Accepted Answer

Provision additional GPUs of the required generation for that tenant or rebalance existing capacity to that node pool.. The pods are constrained by a node affinity rule targeting a specific GPU generation whose nodes are fully allocated. Idle GPUs on other nodes cannot satisfy that rule. The correct fix is to add capacity of the required generation or rebalance matching cards into that pool, which resolves the shortage while preserving the tenant isolation the affinity rule enforces.

Answer

Disable the device plugin on the other tenants' nodes so their GPUs become available to the affected namespace.

Answer

Lower the GPU request in the pod spec so the scheduler accepts a partial device from an idle node.

Answer

Remove the node affinity rule so the pods can schedule onto any idle GPU node in the cluster.

Question 12

An ML platform team runs an NVIDIA GPU Operator-managed cluster and wants to allow multiple pods to share a single A100 GPU so that small inference services can co-reside without each consuming a whole device. The team needs a time-slicing configuration that applies to all GPU nodes in the cluster. Which approach should the administrator take?

Accepted Answer

Create a ConfigMap containing the time-slicing configuration and reference it in the ClusterPolicy so the GPU Operator propagates the device plugin config across nodes.. Time-slicing in the NVIDIA GPU Operator is driven by a device-plugin configuration that the operator distributes to every GPU node through the ClusterPolicy. The referenced ConfigMap declares a replica count, and the device plugin then advertises that many virtual GPUs per physical device, allowing the scheduler to place multiple pods on one GPU. This is the supported, cluster-wide mechanism for enabling time-sliced sharing.

Answer

Add a tolerations entry for nvidia.com/gpu to each pod and increase the kubelet's --max-pods flag on GPU nodes.

Answer

Set the environment variable NVIDIA_VISIBLE_DEVICES=all on each pod and let the runtime divide GPU time among containers.

Answer

Install the NVIDIA MIG Manager and set the nvidia.com/mig.config label to all-1g.5gb on each node to subdivide the GPUs.

Question 13

A data science team submits a PyTorch training job to a Kubernetes cluster managed by Run:ai. The job requests two GPUs but only one is allocated, and the second worker hangs waiting for a peer. Which Run:ai capability should the administrator verify is configured so the distributed job receives all requested GPUs atomically?

Accepted Answer

Gang scheduling, which ensures all pods in a distributed job are scheduled together or not at all.. Distributed training depends on every rank being present before collective operations begin; a single missing worker causes the rest to block. Run:ai's gang scheduling is the mechanism that admits the entire workload as a unit, so all requested GPUs are granted together. Confirming gang scheduling is enabled and applied to the job resolves the partial allocation that produced the hang.

Answer

A higher priority class assigned to the training job so it preempts other workloads.

Answer

Node affinity rules that pin each worker to a specific GPU node by hostname.

Answer

Enabling time-slicing on the GPU device plugin to increase the apparent GPU count.

Question 14

An AI operations engineer is optimizing a real-time inference pipeline on an NVIDIA T4 GPU. The pipeline uses TensorRT and receives requests with variable input sizes. Profiling shows that the engine recompiles for each new input shape, causing latency spikes. Which optimization should the engineer apply to eliminate recompilation while maintaining acceptable accuracy?

Accepted Answer

Enable dynamic shaping in TensorRT by defining an optimization profile with min, opt, and max shapes.. TensorRT dynamic shaping with an optimization profile allows one engine to handle a range of input shapes, eliminating the need to rebuild the engine for each new size. This directly removes the latency spikes from recompilation while maintaining performance across variable inputs. Other options either do not address recompilation or trade off too much efficiency or accuracy.

Answer

Pad all input tensors to a fixed maximum size before inference.

Answer

Convert the model to use INT8 precision with a calibration dataset to reduce inference time.

Answer

Increase the workspace size allocated for TensorRT to allow more kernel autotuning.

Question 15

A machine learning engineer is optimizing a recommendation model for inference on an NVIDIA T4 GPU. The model uses dynamic input shapes, and profiling shows that kernel launch overhead is a significant contributor to latency. Which optimization technique should be applied to reduce this overhead?

Accepted Answer

Convert the model to TensorRT with dynamic shapes and enable CUDA graphs during inference.. CUDA graphs reduce kernel launch overhead by capturing a sequence of operations and replaying them as a single graph. TensorRT with dynamic shapes can leverage CUDA graphs to maintain flexibility while minimizing overhead. This directly addresses the profiling finding, making it the most effective optimization for the described scenario.

Answer

Use mixed precision (FP16) to reduce the number of kernels executed.

Answer

Enable NVIDIA MPS (Multi-Process Service) to allow concurrent kernel execution.

Answer

Increase the batch size to amortize kernel launch overhead across more samples.