NCP-AIO Practice Test 23 — 15 Questions

Question 1

Which of the following is the primary indicator of PCIe bus saturation when profiling a training job on an NVIDIA DGX system?

Accepted Answer

Low GPU duty cycle and high PCIe throughput utilization.. PCIe bus saturation occurs when the GPU's requirement for data exceeds the transfer capacity of the bus. This leads to GPU starvation, where the compute cores idle while waiting for data. Recognizing the specific correlation between low GPU utilization and high PCIe throughput is essential for identifying I/O-bound bottlenecks and optimizing data pipelines for large-scale training jobs.

Answer

High SM occupancy but low memory bandwidth usage.

Answer

High GPU temperature and low clock speeds.

Answer

Frequent ECC errors in the GPU memory logs.

Question 2

A data scientist reports that a PyTorch training job on an NVIDIA V100 GPU is running slower than expected. The job uses a data loader with num_workers=4. Monitoring shows GPU utilization is around 50%, and CPU usage is high. Which action should an AI operations engineer recommend to improve GPU utilization?

Accepted Answer

Increase num_workers in the DataLoader to overlap data preprocessing with GPU computation.. The combination of high CPU usage and low GPU utilization points to a data loading bottleneck where the GPU is starved for data. Increasing the number of DataLoader workers enables more parallel data preprocessing, allowing the GPU to be fed continuously. Other options do not address the CPU-bound pipeline and may not improve the situation.

Answer

Pin memory in the DataLoader to speed up host-to-device transfers.

Answer

Enable mixed precision training to speed up computations.

Answer

Increase the batch size to better utilize the GPU.

Question 3

Refer to the exhibit. What is the most effective way to resolve this specific throttling condition?

Accepted Answer

Increase the power limit for the GPU using nvidia-smi.. The exhibit indicates that the GPU is hitting a software-defined power cap. This is a common situation when a node is configured with a restricted power limit to manage rack-level power consumption. Adjusting the power limit using nvidia-smi ensures the GPU can access the necessary power to reach higher clock frequencies, which is essential for maximizing performance in compute-intensive deep learning tasks.

Answer

Update the GPU firmware to the latest release.

Answer

Reinstall the NVIDIA display driver.

Answer

Replace the power distribution unit (PDU) in the rack.

Question 4

An AI operations team is troubleshooting a distributed training job on a cluster of NVIDIA DGX A100 systems connected via InfiniBand. The job runs but achieves only 40% of expected scaling efficiency. The team suspects communication bottlenecks. Which two actions should they take to confirm and address the issue? (Choose two.)

Accepted Answer

Verify that all nodes have identical NCCL versions and environment variables, and that InfiniBand fabric manager is running.. Scaling inefficiency in distributed training often stems from communication overhead. NCCL debug logs directly show transport selection and errors, while ensuring consistent NCCL versions and InfiniBand fabric health addresses common misconfigurations. Together, these actions confirm and resolve bottlenecks. The other options either do not diagnose the network or would degrade performance.

Answer

Switch the job to use Ethernet instead of InfiniBand to simplify troubleshooting.

Answer

Use nvidia-smi to monitor GPU utilization and memory bandwidth on each node during training.

Answer

Increase the global batch size proportionally to the number of GPUs to improve compute-to-communication ratio.

Question 5

Refer to the exhibit. The training job fails with an OOM error. Which optimization strategy will most effectively resolve this while maintaining model convergence?

Accepted Answer

Implement Gradient Accumulation to simulate a larger batch size.. Memory management is central to deep learning stability. When a model exceeds physical VRAM, gradient accumulation allows for larger effective batch sizes without increasing memory footprint. By accumulating gradients over multiple small steps and performing a single weight update, the model effectively sees a larger batch size, maintaining the convergence characteristics of the original design while fitting within the strict physical memory constraints of the GPU hardware.

Answer

Increase the learning rate to compensate for smaller batches.

Answer

Disable mixed-precision training to reduce memory overhead.

Answer

Clear the GPU cache using torch.cuda.empty_cache() after every iteration.

Question 6

A platform engineer must validate a new NVIDIA GPU Operator deployment on a Kubernetes cluster before handing it to data scientists. Which two checks confirm that the Operator has correctly exposed GPU resources to the cluster scheduler? (Choose two.)

Accepted Answer

Run a CUDA-enabled pod that requests nvidia.com/gpu and verify it reaches Running state and reports the expected device.. GPU exposure is confirmed by two complementary signals: the node advertises the nvidia.com/gpu extended resource with an allocatable count matching the physical GPUs, and a test pod requesting that resource schedules successfully and can access the device. Together they validate device plugin registration and end-to-end runtime injection. Driver image tags, node runtime swaps, and DaemonSet placement on CPU-only nodes are irrelevant to this validation.

Answer

Confirm that the GPU Operator's driver DaemonSet pods are scheduled on all nodes, including CPU-only nodes.

Answer

Check that the NVIDIA driver container image tag matches the CUDA toolkit version installed in the workload image.

Answer

Verify that the container runtime on each node has been switched from containerd to Docker with the nvidia runtime as default.

Question 7

Which component of the NVIDIA GPU Operator is responsible for monitoring GPU health and reporting telemetry data to the Kubernetes control plane?

Accepted Answer

NVIDIA DCGM Exporter.. The DCGM Exporter is the primary tool within the NVIDIA GPU Operator ecosystem for collecting metrics. It interacts with the Data Center GPU Manager (DCGM) to gather telemetry such as utilization, temperature, and memory health. This is vital for AI Operations because it provides the observability necessary to trigger auto-scaling, identify failing hardware, and ensure that training jobs are performing optimally within the cluster environment.

Answer

NVIDIA Device Plugin.

Answer

NVIDIA Container Toolkit.

Answer

NVIDIA Node Feature Discovery.

Question 8

When deploying the NVIDIA GPU Operator in a restricted-access environment (air-gapped), which THREE requirements must be addressed to ensure a successful installation?

Accepted Answer

Mirror all required images to a private registry.. In air-gapped environments, the inability to reach public registries is the primary failure point. Administrators must mirror all required container images to a local private registry, configure the operator to point to these local locations, and ensure that all necessary kernel headers are available locally for the driver build process. These steps ensure that the GPU Operator can complete its setup without external internet dependencies.

Answer

Upgrade the host BIOS to the latest version.

Answer

Install the NVIDIA Triton Inference Server first.

Question 9

Which NVIDIA technology enables the partitioning of a single physical GPU into multiple independent instances for use by different virtual machines or containers?

Accepted Answer

NVIDIA Multi-Instance GPU (MIG).. NVIDIA Multi-Instance GPU (MIG) technology is the correct answer. It allows a single GPU to be securely partitioned at the hardware level, providing guaranteed QoS and isolation for different workloads. This is essential for maximizing GPU utilization in enterprise AI, as it enables the co-location of small inference tasks alongside larger training workloads on a single piece of high-end hardware.

Answer

NVIDIA GPUDirect Storage.

Answer

NVIDIA NVLink.

Answer

NVIDIA vGPU Profiles.

Question 10

Which NVIDIA software component is responsible for providing the necessary CUDA libraries to containerized applications?

Accepted Answer

NVIDIA Container Toolkit. The NVIDIA Container Toolkit is the critical bridge. It provides the necessary libraries and container runtime hooks that allow processes inside a container to access the host's GPU and CUDA environment. This setup is fundamental for AI Enterprise, as it enables portability of AI applications while maintaining high-performance access to physical GPU hardware, regardless of the underlying host OS distribution or container runtime used.

Answer

NVIDIA Driver

Answer

NVIDIA vGPU Manager

Answer

NVIDIA Triton

Question 11

An AI operations team is deploying NVIDIA AI Enterprise on a bare-metal Kubernetes cluster with DGX A100 systems. They need to enable GPUDirect Storage to accelerate data loading from a local NVMe array. Which component must be installed and configured on the DGX nodes to support GPUDirect Storage?

Accepted Answer

NVIDIA Magnum IO GPUDirect Storage kernel module and user-space libraries. GPUDirect Storage is enabled by installing the NVIDIA Magnum IO GPUDirect Storage components, which include the nvidia-fs kernel module and CUDA libraries. These allow direct DMA transfers between NVMe storage and GPU memory, bypassing the CPU. On DGX systems, these components are typically part of the DGX software stack and must be properly configured.

Answer

NVIDIA Peer-to-Peer (P2P) over PCIe with IOMMU disabled

Answer

NVIDIA GPU Operator with the RDMA shared device plugin enabled

Answer

NVIDIA Container Toolkit with the 'nvidia-container-runtime' configured for privileged mode

Question 12

An administrator is deploying NVIDIA AI Enterprise on a Kubernetes cluster and must decide how GPU workloads should request accelerators. The environment has a mix of full-GPU training jobs and inference services that share a single A100. Which approach correctly allows a pod to consume a specific MIG-backed slice rather than the whole device?

Accepted Answer

Request the specific extended resource, such as nvidia.com/mig-2g.10gb, in the pod's resource limits.. In Kubernetes, GPU allocation is expressed exclusively through extended resource requests handled by the NVIDIA device plugin. When MIG is enabled, each configured profile is advertised under a distinct resource name, so a pod requesting a specific MIG profile is bound to a matching instance. This is what enables a single A100 to serve both whole-device training and sliced inference workloads.

Answer

Request nvidia.com/gpu: 1 and add the annotation nvidia.com/mig-profile=2g.10gb to the pod metadata.

Answer

Set the environment variable NVIDIA_MIG_PROFILE=2g.10gb in the container spec and request nvidia.com/gpu: 1.

Answer

Deploy a separate RuntimeClass named mig-2g.10gb and reference it from the pod's spec.runtimeClassName.

Question 13

Which component is responsible for exposing the GPU as a schedulable resource in a Kubernetes cluster?

Accepted Answer

NVIDIA Device Plugin. The Kubernetes device plugin is the interface that allows the kubelet to communicate with the GPU. It advertises the number of available GPUs on each node to the Kubernetes API server. When a pod requests a GPU, the scheduler uses this information to place the workload on the correct node. Without this plugin, Kubernetes is unaware of the GPU hardware, making it impossible to manage and allocate GPU resources for containerized workloads.

Answer

NVIDIA Container Toolkit

Answer

NVIDIA DCGM Exporter

Answer

The Kubernetes Scheduler.

Question 14

A cluster runs mixed workloads: latency-sensitive inference services and best-effort batch jobs. Administrators observe that batch jobs occasionally occupy all GPUs, causing inference requests to queue and breach service level objectives. They want inference pods to be admitted immediately while allowing batch work to use remaining capacity and be preempted when needed. Which approach should they implement?

Accepted Answer

Define priority classes and a preemption policy so inference pods have higher priority than batch jobs.. The requirement is immediate admission for inference and preemptible use of leftover capacity by batch work. Priority classes with preemption give inference pods the ability to displace lower-priority batch pods when GPUs are scarce, while batch jobs still consume idle capacity during quiet periods. Affinity, quotas, and time-slicing each constrain or share resources but cannot prioritize or preempt running workloads.

Answer

Enable time-slicing on all GPUs so batch and inference pods share devices.

Answer

Set a ResourceQuota on the batch namespace limiting total GPU requests.

Answer

Apply node affinity so inference pods only run on a dedicated subset of nodes.

Question 15

A system administrator is installing NVIDIA AI Enterprise on a Kubernetes cluster that will use Multi-Instance GPU (MIG) on A100 GPUs. The administrator wants to ensure that MIG instances are properly exposed as schedulable resources. Which action must be taken after enabling MIG mode on the GPUs?

Accepted Answer

Install the NVIDIA GPU Operator with the MIG strategy set to 'mixed' and configure the device plugin to advertise MIG resources.. To expose MIG instances as schedulable resources in Kubernetes, the NVIDIA GPU Operator must be installed with the MIG strategy configured (e.g., 'mixed' or 'single'). The Operator's device plugin then discovers and advertises each MIG instance as a resource. This allows pods to request specific MIG profiles via resource limits, enabling efficient scheduling.

Answer

Manually create Kubernetes custom resources for each MIG instance using the NVIDIA MIG Manager.

Answer

Set the environment variable NVIDIA_MIG_CONFIG_DEVICES to 'all' on the kubelet and restart the kubelet service.

Answer

Deploy a separate device plugin for each MIG instance using a DaemonSet with node affinity to the specific GPU.