NVIDIA Certified Professional: AI Operations (NCP-AIO) — Questions 151–225

A 'SW Power Cap' throttle reason means the GPU is hitting its configured power limit, so clocks are reduced to stay within budget. If the hardware and cooling support it, raising the power limit to a validated value within the GPU's supported range allows higher sustained clocks, directly addressing the throttle cause. This is the correct, targeted remediation for the reported condition.

The GPU Operator supports time-slicing by reading a ConfigMap referenced in the ClusterPolicy's device plugin configuration. When applied, the operator propagates the config to all GPU nodes and restarts the device plugin, causing each physical GPU to advertise a multiplied replica count so multiple pods can share one device. This is the supported cluster-wide method.

Maintaining a unified orchestration layer allows for uniform resource management and scheduling logic across environments. By standardizing the GPU Operator and the Kubernetes API, teams can move workloads seamlessly without rewriting manifests or changing operational workflows, which is the primary challenge in managing hybrid GPU infrastructure.

Periodic `cudaMalloc` calls during request handling force synchronous memory allocation and can trigger driver-level operations that stall the GPU pipeline, producing p99 spikes while p50 stays flat. Pre-allocating workspaces and using a memory pool (or CUDA graphs with static allocations) removes allocation from the hot path, which directly addresses the observed pattern and is the correct remediation here.

Gang scheduling ensures that all pods in a PodGroup are scheduled together or none are, preventing partial starts that waste GPUs and stall distributed training. This directly satisfies the requirement that the four GPU pods begin only when four GPUs are simultaneously available.

NCCL falling back to socket transport indicates the RDMA path is unavailable, typically because the high-speed fabric drivers, RDMA devices, and GPUDirect RDMA support are not provisioned on the nodes. The Network Operator deploys and configures those components alongside the GPU Operator, restoring the RDMA transport that NCCL prefers for multi-node collectives.

RBAC in Kubernetes allows administrators to define roles and role bindings that specify which actions are permitted on which resources. The GPU Operator uses RBAC to grant its components the minimum necessary permissions. This enables least-privilege access and is essential for strict security environments.

When Prometheus is managed by the Prometheus Operator, it only scrapes targets whose ServiceMonitor or PodMonitor labels match the operator's serviceMonitorSelector or podMonitorSelector. If the DCGM Exporter's monitor labels do not match, Prometheus silently ignores the endpoint even though the exporter is running. Verifying selector label alignment is the correct first step because it is the most common cause of missing metrics in operator-managed Prometheus.

NVIDIA Confidential Computing utilizes hardware-based Trusted Execution Environments (TEEs) to encrypt data while it resides in GPU memory. This prevents unauthorized access from other processes, the kernel, or the hypervisor, ensuring that sensitive model weights remain secure even if the software environment is considered untrusted or potentially compromised.

The symptom of full intra-node NVLink but degraded inter-node throughput points to the job falling back to TCP over the management interface. GPUDirect RDMA lets the NIC read/write GPU memory directly over the InfiniBand/RoCE compute fabric, bypassing host memory copies. Exposing `/dev/infiniband` to the container and pinning `NCCL_IB_HCA` restores the intended high-speed path.

Nsight Systems provides a system-wide view of CPU and GPU activities, allowing engineers to correlate data transfer operations with kernel execution. This visibility is essential for identifying stalls, data starvation, or PCIe bus congestion, which are the most common causes of low throughput despite high GPU utilization.

DCGM policy management allows administrators to define policies that trigger actions when health violations occur, such as resetting a GPU or cordoning a node. This enables automated mitigation, reducing downtime. It works in conjunction with health checks. Configuring policies ensures that detected issues are handled promptly, which is essential for proactive health management in a large GPU cluster.

The GPU Operator's device plugin exposes GPUs as schedulable resources (nvidia.com/gpu). Enforcing a policy that requires pods to request this resource ensures that only pods explicitly asking for GPUs receive device nodes. Forbidding privileged mode enforces least privilege. Together, these configurations ensure GPU device nodes are not exposed to pods that do not request them and that pods run with minimal privileges.

The Kubernetes Device Plugin framework is built on gRPC. The NVIDIA plugin implements this API to provide the Kubelet with the necessary information about GPU resources. This standard interface allows Kubernetes to treat different hardware devices consistently while maintaining the flexibility needed for NVIDIA-specific hardware management.

GPUDirect RDMA allows direct memory access between the GPU and third-party devices such as NICs, effectively bypassing the host CPU. This architecture eliminates unnecessary data copies and context switches, providing the lowest possible latency for high-speed AI data pipelines and distributed training environments on physical infrastructure.

When the NVIDIA driver is updated and the system reboots, any existing processes that had already initialized CUDA hold references to the old driver. The Jupyter kernel is such a process. Restarting the kernel terminates the old process and starts a new one that loads the updated driver, allowing CUDA calls to succeed and GPU memory to be allocated.

The 'Sw Power Cap' status confirms that the GPU is limited by the current software configuration. To improve performance, the administrator should evaluate the power policy settings to determine if the wattage limit can be safely increased to allow the GPU to reach its maximum boost clock frequency.

NCCL_DEBUG=INFO with NCCL_DEBUG_SUBSYS=INIT,COLL,NET surfaces transport selection, ring/tree topology construction, and per-rank connection progress. Because single-node all_reduce_perf passes but multi-node fails, the fault is almost certainly in the inter-node path, and this logging pinpoints whether NCCL fell back to a broken interface or is stuck establishing connections.

GDS relies on RDMA and peer-to-peer (P2P) communication to transfer data directly between storage and GPU memory without CPU involvement. The storage devices (NVMe over Fabrics) and network adapters (InfiniBand) must support RDMA, and the system must be configured to allow P2P DMA. This includes enabling PCIe Access Control Services (ACS) and IOMMU settings that permit direct peer-to-peer transfers. Without this, GDS cannot achieve direct data paths.

NCCL debug logs provide detailed information about the communication setup, including which transports are used, any fallbacks, and specific error codes. This is the most direct way to diagnose the cause of an 'unhandled system error', which can stem from hardware, driver, or configuration issues. Capturing logs during failure is the essential first step before attempting fixes.

MIG provides spatial and temporal hardware partitioning, ensuring that individual workloads have their own dedicated hardware paths. This prevents performance degradation caused by noisy neighbors and provides security by isolating memory and compute resources, which is critical for multi-tenant clusters that require predictable performance and isolation.

Nsight Systems provides a detailed timeline of CPU and GPU activity, showing kernel executions, memory copies, and synchronization points. It can reveal if one GPU is waiting on data or if there is an imbalance in computation. This is the correct tool to pinpoint the bottleneck in a multi-GPU job, as it captures cross-GPU interactions and can highlight stragglers.

The Container Toolkit provides the necessary hooks and libraries to map GPU resources into the container namespace. This allows the application running inside the container to make calls to the GPU hardware, which would otherwise be inaccessible due to the isolation boundaries of the container runtime environment.

The 'SW Power Cap' indicator explicitly confirms that an administrative policy or software command has capped the power usage of the GPU. This forces the GPU to maintain lower clock speeds, directly impacting the throughput of the training job by limiting the available computational resources.

Xid errors are reported by the NVIDIA driver and indicate GPU hardware or driver issues. Each Xid code corresponds to a specific fault, such as a corrupted memory access or a fallen off the bus error. Looking up the code in NVIDIA's documentation provides the exact cause and recommended actions. This is the most direct first step to diagnose the segmentation fault linked to GPU errors.

CUDA graphs capture a sequence of kernel launches and replay them with a single launch, drastically reducing launch overhead. TensorRT supports dynamic shapes and can build engines that use CUDA graphs. This is ideal for models with variable input shapes where launch overhead is high. The combination reduces CPU overhead and improves latency.

This runtime transparently mounts the host GPU devices and user-mode NVIDIA libraries into the container. It modifies the container's OCI specification during the execution phase, ensuring that the containerized process can communicate with the physical GPU drivers installed on the host operating system.

Workloads often require substantial system memory for preprocessing before moving data to the GPU. If the container memory limit is exceeded, the orchestrator will terminate the pod with an OOM error, regardless of how much GPU VRAM is available. This is a common oversight when configuring resource limits.

The GPU Operator manages the NVIDIA Container Toolkit and runtime configuration. If containerd already has manually added NVIDIA runtime settings, these can conflict with the operator's configuration, leading to failures. It is best practice to remove any existing NVIDIA runtime configuration before installation so the operator can set it up cleanly.

Framework-level memory profilers show exactly which tensors or operations are allocating memory and whether those allocations are freed. By taking snapshots at intervals, an engineer can see if memory is retained across inference calls, indicating a leak in the application code or framework caching allocator. This directly identifies the source without disrupting the service.

Kubernetes priority and preemption let a pending pod with a higher PriorityClass displace running pods of lower priority when resources are scarce. Assigning a high class to inference and a lower class to training makes the scheduler evict a training pod to admit the inference pod. Combined with a group-aware training controller, the evicted job can be requeued and resume from checkpoint.

Triton validates input data types against the model's configuration. If the model expects FP32 but receives FP16, it may either reject the request or misinterpret the data, leading to incorrect predictions. Setting the correct data type in the model configuration and aligning the client ensures proper preprocessing and accurate inference.

Node feature discovery running under the GPU Operator publishes labels such as nvidia.com/cuda.driver.major and minor versions. Pods can then express a nodeAffinity requirement matching the needed driver generation, so the scheduler only binds them to nodes whose advertised driver satisfies the container's CUDA expectation. This moves incompatibility detection to scheduling time, preventing the runtime failures users currently see.

Time-slicing enables a physical GPU to be oversubscribed by allowing multiple pods to execute in turns on the same hardware. This increases the utilization of the GPU in scenarios where the individual pods do not require the full dedicated throughput of the entire device at all times.

A ValidatingAdmissionWebhook can intercept pod creation requests and evaluate custom logic, such as checking for a specific label before allowing the pod to request GPU resources. This provides the flexibility to enforce label-based policies on extended resources. By deploying a webhook that inspects pod labels and resource requests, the administrator can deny non-compliant pods, meeting the requirement.

If an InfiniBand link negotiates at a lower rate (e.g., SDR instead of HDR), the network throughput will be severely throttled. This mismatch is a classic cause of "slow path" NCCL warnings, as the collective communication operations are unable to achieve the expected bandwidth required for efficient multi-node training.

PriorityClass assigns a numeric priority, and when a high-priority pod cannot schedule, the scheduler may evict lower-priority pods on a node to make room. Setting the inference pods to a higher priority with preemption enabled lets them displace training pods, which can restart. This directly implements automatic yielding of GPUs to latency-sensitive inference workloads without manual operator action.

The Base Command Manager agent is required on each node to communicate with the head node, report status, and receive commands. Installing and registering the agent allows the head node to recognize the new node, manage its resources, and include it in the cluster's scheduling pool. Without the agent, the node remains unmanaged and cannot be utilized for jobs.

In an air-gapped environment, the ClusterPolicy is configured with the private registry path and image repository settings for each operand, such as driver, toolkit, device plugin, and DCGM exporter. This ensures the Operator deploys pods that pull from the internal registry rather than the public NVIDIA registry, which is unreachable.

The DCGM Exporter is purpose-built to interface with the underlying NVIDIA driver and DCGM to gather detailed telemetry. It exposes these metrics as Prometheus-formatted data, which is essential for monitoring the health and performance of GPU workloads within a cluster and making data-driven infrastructure management decisions.

Poor scaling with high GPU utilization often indicates communication overhead. `nvidia-smi topo -m` reveals whether GPUs are connected via NVLink or PCIe and whether data traverses slower paths like QPI/UPI. If NCCL cannot leverage NVLink, all-reduce operations become the bottleneck. This command is the standard first step to diagnose interconnect issues and ensure NCCL selects the optimal topology-aware route, directly addressing the scaling inefficiency.

Pinned memory allows the GPU to directly access host memory via DMA, avoiding the overhead of staging through pageable memory. This reduces the time spent on host-to-device and device-to-host copies, which is critical for real-time inference with dynamic shapes where copies are frequent. It is a standard optimization to improve transfer efficiency and lower latency.

A kernel panic citing missing storage drivers means the assigned OS image profile lacks the storage controller modules and hardware support packages required by that server model. Correcting the image profile in Base Command Manager lets the deployed kernel detect the disks.

Distributed training generates massive amounts of state data simultaneously. If the storage system cannot handle the aggregate write load, the compute nodes will stall, waiting for I/O completion. Sufficient write bandwidth is therefore non-negotiable to maintain the high performance required for large-scale GPU training clusters.

DCGM provides comprehensive GPU health monitoring, and its integration with Slurm via the health check plugin allows automatic detection of unhealthy GPUs. When DCGM identifies a failed GPU, the plugin can drain the node from Slurm, preventing new jobs from being scheduled on it. This directly addresses the requirement to schedule only on healthy GPUs and automatically remove failed ones.

The allocation failures under peak load with near-full VRAM indicate that the configured instances and maximum batch size exceed the memory budget when concurrent requests spike. Reducing instances and capping the batch size lowers the peak memory footprint per inference, restoring headroom so the allocator can satisfy every request instead of failing.

CUDA Graphs capture a sequence of kernel launches and memory operations into a single graph, then replay it with minimal CPU overhead. This eliminates repeated cudaMalloc/cudaFree and launch gaps, smoothing latency spikes. It directly targets the CPU-side stalls and low GPU utilization observed, making it the appropriate optimization for this scenario.

The NLS instance acts as the centralized point for distributing and validating licenses to nodes in the cluster. It ensures that the enterprise software features are appropriately licensed, providing the necessary reporting and validation required by the AI Enterprise software subscription models for large-scale distributed computing environments.

Node Feature Discovery is responsible for scanning the hardware and applying labels such as 'nvidia.com/gpu.present'. Without these labels, the GPU Operator will not know which nodes to target for driver installation, causing the node to remain without detected GPU resources in the Kubernetes API server.

A private registry often requires authentication. Creating an image pull secret in the operator's namespace and referencing it in the ClusterPolicy or service accounts allows nodes to pull images securely. This is a standard requirement for air-gapped installations where the internal registry is not publicly accessible, ensuring components can authenticate and retrieve images.

The nvidia-smi tool is specifically designed to provide a comprehensive view of all NVIDIA GPUs installed in the system. It displays critical health metrics like power usage, temperature, memory usage, and compute utilization in a real-time format, serving as the primary diagnostic tool for AI infrastructure administrators.

NVLS (NVLink SHARP) is an in-network reduction feature that can accelerate all-reduce operations, but it requires compatible hardware and software. If errors occur, disabling it forces NCCL to fall back to standard ring or tree algorithms, which are reliable. This resolves the immediate errors and restores communication performance, though it may not achieve peak efficiency. It is a valid troubleshooting step when NVLS is unstable.

Time-slicing provides no memory isolation, so concurrent pods can exhaust GPU memory and cause OutOfMemory failures. Removing training jobs from time-sliced nodes and giving them dedicated GPUs or MIG instances ensures exclusive memory access while still allowing time-slicing to remain available for trusted inference workloads on separate node pools.

Base Command Manager health checks can run custom scripts periodically. A script can query GPU temperature and, if over threshold, call the Base Command Manager API to drain the node. An alert rule then sends a notification. This provides automated remediation and alerting as required.

Even with the correct runtime, the container requires the underlying host driver to communicate with the hardware. If the driver is not installed or the kernel module is not loaded, the runtime will fail to map the GPU device, leading to a situation where the GPU is inaccessible.

High communication time can be caused by network congestion or errors on the InfiniBand fabric. Monitoring counters such as port errors, link down events, or congestion can reveal underlying issues. If the fabric is congested, all-reduce operations will slow down, impacting scaling. This is a fundamental check before tuning NCCL parameters. It directly addresses potential network-level bottlenecks.

If a pod requires a specific node label or has a taint that the available nodes do not satisfy, it will remain in a pending state. This scenario is a common cause for pods failing to schedule, even when the underlying hardware resources appear idle to the cluster administrator.

The GPU Operator runs health checks that can detect uncorrectable GPU errors and signal the device plugin to stop advertising the faulty device. The operator can also apply taints or cordon the node. Existing pods become unschedulable and are recreated elsewhere by their controller, while new GPU pods are kept off the node, matching both stated requirements.

A private registry acts as a local source of truth for container images. In production environments, it is essential for security and stability, allowing administrators to control exactly which software versions are deployed to the cluster while reducing reliance on external registries that may have latency or availability issues.

A taint on a node only repels pods that do not tolerate it. Adding the matching toleration and a node selector that identifies the reserved GPU nodes lets this job schedule there while every other untolerated workload stays off those nodes. The reservation remains intact and the job becomes runnable, which directly resolves the Pending state.

Batch schedulers with backfill can start smaller jobs ahead of larger queued jobs when resources are available, keeping GPUs busy between experiment runs. This reduces idle gaps and improves throughput for many short-lived jobs, complementing GPU sharing without requiring modifications to the training scripts.

Checkpointing requires writing large amounts of data to disk. If multiple nodes attempt to write to a shared filesystem simultaneously, the aggregate I/O demand can exceed the filesystem's bandwidth, causing the training process to hang while waiting for the write operation to complete.

NVIDIA AI Enterprise is a cloud-native software suite that provides validated, secure, and supported software. It is specifically designed to ensure that the entire stack, from drivers to containers, is compatible and stable, allowing organizations to maintain production consistency across diverse data center hardware environments over long periods.

FP16 has a limited dynamic range. During backpropagation, many small gradient values can be rounded to zero (underflow). Loss scaling multiplies the loss before backpropagation, pushing the gradients into a representable range in FP16, then unscaling them before applying weight updates. This is critical for preventing divergence in mixed precision.

In high-performance computing, Volcano requires that all pods within a group start simultaneously. If the requested number of GPUs cannot be satisfied across available nodes due to locality or capacity constraints, the job remains pending, even if individual nodes have free GPUs, to prevent incomplete distributed training runs.

Intermittent hangs without errors require visibility into which collective and rank stalled. NCCL_DEBUG=INFO with NCCL_DEBUG_SUBSYS=COLL prints per-collective progress and rank participation, while a watchdog timeout can trigger a dump when progress stops. This combination captures the state at the moment of the hang, making it the correct diagnostic action for this scenario.

After a reboot, the GPU Operator must reload the driver and reinitialize the device plugin before workloads can use GPUs safely. Driver validation and readiness gates prevent pods from starting until the driver and plugin report healthy. Reviewing these settings addresses the CUDA error and unhealthy device plugin state observed after maintenance reboots.

DCGM provides health checks that can be configured with thresholds for temperature and other metrics. The DCGM exporter can expose these metrics to Prometheus, which supports alerting rules based on sustained conditions. This combination allows for proactive monitoring and alerting on high temperatures over time. It is the standard approach for cluster-wide GPU monitoring.

MIG and fractional sharing allow multiple inference workloads to run concurrently on a single GPU. This effectively increases the number of available 'virtual' GPUs, maximizing utilization and ensuring that small, bursty requests can be handled with minimal latency, which is essential for cost-effective, high-scale inference services.

The GPU Operator deploys its components into the gpu-operator namespace by default. Running kubectl get pods -n gpu-operator lists all pods in that namespace, allowing the administrator to see if the operator, driver, container toolkit, device plugin, and other components are running. This is the standard way to verify the operator's deployment status.

Low SM clock despite P0 state often indicates power or thermal throttling. Checking power consumption against the configured limit helps determine if the GPU is throttling due to power constraints. This is a common cause of reduced clocks and is a logical first diagnostic step before changing application settings.

SYS indicates that communication between those two GPUs traverses the system's PCIe and CPU interconnect instead of the high-speed NVLink/NVSwitch fabric. On a DGX H100, all GPU pairs should normally show NV18 via NVSwitch, so this points to a degraded or missing NVLink path between GPU4 and GPU5 that must be investigated.

Running nccl-tests all_reduce_perf reproduces the collective in isolation and reports algorithmic and bus bandwidth, so a gap versus theoretical fabric peak localizes the loss to the communication path rather than the model code. Because every link is error-free and HCA counters are low, the fault is likely in topology, ring/tree selection, or placement, and this microbenchmark exposes exactly that before any configuration is changed.

MIG allows a physical GPU to be partitioned into multiple isolated instances at the hardware level. Each instance has its own dedicated memory and compute resources, providing the isolation necessary for multi-tenant environments where security and performance guarantees are required for concurrent, independent workloads on a single piece of hardware.

Running nccl-tests with debug logging reproduces the collective communication pattern outside the training job. It reveals which transport (InfiniBand, RoCE, or sockets) is used, whether topology detection fails, and where timeouts occur. This isolates NCCL configuration issues from fabric problems before deeper network diagnostics.