NCP-AIO Practice Test 19 — 15 Questions

Question 1

An operations engineer is troubleshooting a distributed training job that uses NVIDIA Magnum IO GPUDirect Storage to read training data directly from a local NVMe SSD into GPU memory. The job reports lower than expected I/O bandwidth. `nvidia-smi` shows normal GPU utilization, and the NVMe drive's throughput is well below its peak. Which factor is most likely limiting GPUDirect Storage performance in this scenario?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

The GPU is not connected to the NVMe drive through a supported PCIe topology.. GPUDirect Storage achieves maximum bandwidth only when the GPU and NVMe drive can communicate directly over PCIe with peer-to-peer support. In many servers, the GPU and drive are on different PCIe root complexes or NUMA nodes, forcing data through host memory and halving effective bandwidth. Checking the PCIe topology with tools like `nvidia-smi topo -m` or `lspci` reveals whether a direct path exists. Other factors like CPU threads or file system are less likely to cause the specific bandwidth shortfall.

Answer

The training job is using too many CPU threads for data preprocessing.

Answer

The GPU's ECC memory is enabled, reducing available bandwidth.

Answer

The NVMe drive is formatted with a file system that does not support GPUDirect Storage.

Question 2

Which scheduling strategy is recommended to maximize the efficiency of long-running training jobs on preemptible instances?

Accepted Answer

Implementing granular checkpointing and automated job resumption.. Long-running jobs on preemptible (or spot) instances require frequent, efficient checkpointing to survive node reclamation. By combining a robust checkpointing schedule with smart job resubmission logic that monitors for preemption signals, organizations can take advantage of low-cost instances while minimizing the loss of progress. This approach allows for significant cost savings in non-critical training without sacrificing the overall reliability of the research project.

Answer

Avoiding preemptible instances for all training jobs.

Answer

Locking the job to a specific node using node affinity.

Answer

Increasing the priority of the job to the maximum level.

Question 3

You are troubleshooting a node where the GPU is detected, but the application fails to utilize it. Which log source would provide the most relevant information?

Accepted Answer

The NVIDIA container runtime logs.. The NVIDIA container runtime logs and the application-level logs are the most important sources. If the GPU is visible to the system but not the application, the issue is likely a driver/runtime mismatch or a library path configuration. Checking these logs allows an administrator to isolate whether the fault is in the container orchestration layer or the application's software environment, which is vital for rapid resolution in production environments.

Answer

The BIOS system event log.

Answer

The cluster's physical network switch logs.

Answer

The local NTP synchronization logs.

Question 4

In an air-gapped environment, what must an administrator do to ensure the GPU Operator correctly installs the necessary software components?

Accepted Answer

Configure the GPU Operator to use a local image registry.. In an air-gapped environment, the cluster cannot reach external repositories (e.g., NGC or Docker Hub). The administrator must pre-populate a local container registry with all required images and update the Operator configuration to point to this local registry. This is a common enterprise task for secure environments that prevents deployment failures caused by connection timeouts or authentication issues when accessing public NVIDIA software mirrors.

Answer

Enable the 'offline-mode' flag in the Kubernetes API.

Answer

Manually copy the VIB files to every node.

Answer

Disable the validation of container image signatures.

Question 5

A production inference service experiences intermittent latency spikes. The service is deployed on shared infrastructure. Which tool would best help an AI Ops engineer identify if GPU resource contention is the cause?

Accepted Answer

NVIDIA DCGM Exporter with Prometheus/Grafana.. NVIDIA DCGM (Data Center GPU Manager) provides detailed telemetry data, including GPU utilization, memory bandwidth, and power usage per process. By correlating latency spikes with DCGM metrics, engineers can pinpoint if another process on the same GPU is competing for compute resources or memory bandwidth. This visibility is essential for performance tuning and workload placement, allowing engineers to verify if resource isolation policies are functioning as expected.

Answer

Standard Kubernetes 'kubectl top' command.

Answer

The Linux 'top' command on the worker node.

Answer

A simple network latency test tool like 'ping'.

Question 6

When troubleshooting an NVIDIA GPU Operator installation, which TWO locations should an administrator check to identify why the driver installation pod is failing?

Accepted Answer

The output of 'kubectl describe pod <driver-pod-name>'.. Checking the pod's logs and describe output is the standard diagnostic path. The 'describe' output shows events like ImagePullBackOff or scheduling failures, while the logs provide the specific error message from the driver installation script itself, such as kernel header mismatches or network timeouts. These two sources provide the necessary visibility to pinpoint whether the failure is infrastructure-related, configuration-based, or due to a missing environmental dependency.

Answer

The contents of the /etc/kubernetes/manifests folder.

Answer

The system-wide /var/log/syslog file on the control plane.

Answer

The NVIDIA license server status page.

Question 7

When troubleshooting a NCCL collective communication timeout in a distributed training environment, which component should be the primary focus of initial investigation?

Accepted Answer

The network interface card (NIC) topology and interconnect health.. NCCL timeouts are frequently caused by network congestion, MTU mismatches, or faulty interconnect cables between nodes. By verifying the physical and logical network path, engineers can isolate whether the issue is at the application layer or the fabric layer. This focus is critical because distributed training is highly sensitive to latency and packet loss across the GPU cluster interconnect.

Answer

The GPU driver version on the master node.

Answer

The model weight initialization strategy.

Answer

The local disk I/O latency for checkpoint saving.

Question 8

Which approach is most effective for scaling an inference workload that experiences sudden, unpredictable spikes in request volume?

Accepted Answer

Implementing HPA with custom metrics provided by DCGM Exporter.. Predictive or reactive autoscaling based on custom GPU metrics (like utilization) allows the cluster to adjust replica counts in real-time. By monitoring the GPU load and scaling the inference pods accordingly, the system maintains low latency during spikes while keeping costs low during idle periods. This responsiveness is vital for production AI services where performance SLAs are tied directly to user experience.

Answer

Setting a fixed number of replicas to match peak expected load.

Answer

Using a load balancer to redirect traffic to an idle cluster.

Answer

Increasing the memory allocation for each inference container.

Question 9

An administrator is optimizing a large model training job to reduce checkpointing time to storage. Which strategy is most effective for minimizing the impact on training throughput?

Accepted Answer

Implement asynchronous checkpointing.. Checkpointing large models involves writing gigabytes of data to storage, which can pause training for significant periods. Using asynchronous checkpointing or offloading the save process to a background thread allows the GPU to continue training while the data is written to persistent storage. This eliminates the I/O wait time, ensuring that the heavy computational resources are not wasted during the periodic saving of model states.

Answer

Use faster NVMe drives for checkpoint storage.

Answer

Increase the checkpoint frequency.

Answer

Compress the model weights before saving.

Question 10

A multi-tenant NVIDIA GPU-accelerated Kubernetes cluster utilizing NVIDIA AI Enterprise experiences intermittent out-of-memory errors on Triton Inference Server pods despite adequate node memory reservation. Which monitoring and troubleshooting action correctly isolates the root cause?

Accepted Answer

Inspect DCGM-Exporter metrics for DCGM_FI_DEV_FB_FREE and DCGM_FI_DEV_GPU_TEMP alongside Triton server request concurrency logs.. Inspect NVIDIA Data Center GPU Manager metrics via Prometheus and DCGM-Exporter to capture real-time device memory consumption patterns. This practice is critical because standard Kubernetes container metrics fail to track internal GPU frame buffer allocations and fragmentation specific to deep learning inference engines.

Answer

Analyze cAdvisor container memory metrics using kubectl top pods to evaluate resident set size growth over sustained inference workloads.

Answer

Review the Kubernetes cluster autoscaler logs to determine if node scaling events triggered unexpected pod evictions and restarts.

Answer

Execute systemctl status containerd on the worker node to verify container runtime stability and daemon responsiveness.

Question 11

An inference model running on Triton Inference Server is reporting high latency for requests. The model uses a fixed-size batching strategy. What is the most effective way to optimize throughput while maintaining latency targets?

Accepted Answer

Configure dynamic batching with a maximum delay.. Dynamic batching is a powerful feature in Triton Inference Server that groups individual requests together to saturate the GPU's compute capability. By configuring the 'max_queue_delay_microseconds', the system waits briefly to aggregate requests, significantly increasing throughput. This optimization is crucial for balancing the trade-off between individual request latency and overall system efficiency, ensuring that the GPU is not performing trivial computations for tiny batches.

Answer

Increase the number of instances for the model.

Answer

Disable all logging to reduce CPU overhead.

Answer

Force the model to run on the CPU.

Question 12

Which of the following describes the purpose of the NVIDIA GPU Operator's 'Driver Container'?

Accepted Answer

To provide a platform-agnostic way to deploy NVIDIA drivers.. The Driver Container is a critical component that builds or pulls the correct driver for the specific host OS and kernel. It automates the complex process of driver installation, ensuring compatibility across heterogeneous node environments. By containerizing the driver, the Operator simplifies maintenance and upgrades, reducing the risk of configuration drift and ensuring that nodes always have a functional driver compatible with the latest AI software releases.

Answer

To store the persistent state of trained neural networks.

Answer

To manage the licensing of the AI Enterprise suite.

Answer

To act as a gateway for remote GPU access.

Question 13

Which TWO strategies should an administrator implement to ensure fair resource scheduling in a multi-tenant NVIDIA cluster using Kubernetes and the NVIDIA device plugin?

Accepted Answer

Configure Kubernetes ResourceQuotas to limit the total number of GPUs per namespace.. Fair scheduling in multi-tenant environments requires both hard limits to prevent resource monopolization and priority-based mechanisms to ensure critical jobs progress. By combining ResourceQuotas for capacity management and PriorityClasses for job scheduling, admins can prevent a single user from starving the cluster while maintaining high performance for latency-sensitive inference or urgent training tasks during peak usage windows.

Answer

Implement static partitioning for all GPUs in the cluster.

Answer

Use the NVIDIA Triton Inference Server to manage all batch processing.

Answer

Disable the NVIDIA device plugin to allow direct node access.

Question 14

An AI engineer needs to monitor GPU utilization across a large cluster of nodes in real-time. Which NVIDIA tool is the most appropriate for this high-level observability task?

Accepted Answer

NVIDIA DCGM (Data Center GPU Manager). Observability at scale requires tools that can aggregate metrics across multiple nodes. NVIDIA DCGM (Data Center GPU Manager) is specifically architected for this purpose, providing a comprehensive API and service to monitor GPU health, utilization, and power consumption across entire clusters. This is essential for operations teams managing large-scale AI infrastructure to proactively detect performance issues and optimize resource utilization across the fleet.

Answer

nvidia-smi

Answer

CUDA Profiler (nsys)

Answer

nvcc

Question 15

Which THREE of the following are benefits of using Multi-Instance GPU (MIG) technology in a Kubernetes environment?

Accepted Answer

Hardware-level isolation between GPU workloads.. MIG allows for hardware-level isolation, ensuring that one workload's memory usage or compute spikes do not impact others on the same physical GPU. This provides deterministic Quality of Service (QoS), which is critical for multi-tenant AI environments. By partitioning resources, administrators can optimize GPU allocation, allowing smaller, less intensive tasks to run on a fraction of the GPU, thereby significantly increasing overall cluster efficiency.

Answer

Increased total GPU count available to the OS kernel.

Answer

Automatic translation of CUDA code to work on different GPU architectures.