NCP-AIO Practice Test 18 — 15 Questions

Question 1

An AI operations engineer is investigating intermittent failures in a long-running distributed training job. The job occasionally aborts with a collective timeout error, but no GPU errors, ECC events, or fabric link flaps appear in logs. Which action should the engineer take first to identify the root cause?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Enable per-rank NCCL debug logging and correlate the last completed collective across all ranks to find the straggler.. A collective timeout with no hardware errors indicates one rank is arriving late or not at all. Enabling per-rank NCCL debug logging and comparing the last completed collective across ranks isolates the straggler and the operation it was executing, providing direct evidence of the fault without altering the job configuration or losing diagnostic information.

Answer

Restart the job with a reduced global batch size to decrease the time each collective step requires.

Answer

Lower the NCCL timeout value so failures surface faster and can be correlated with system events.

Answer

Switch the collective algorithm from ring to tree to reduce sensitivity to a single slow rank.

Question 2

An AI operations engineer is preparing a DGX H100 system for a multi-node training workload. The engineer runs `nvidia-smi topo -m` and notices that GPU4 and GPU5 report a connection type of SYS, while all other GPU pairs show NV18. What is the most likely cause of this topology anomaly?

Accepted Answer

GPU4 and GPU5 are connected through the PCIe host bridge rather than the NVSwitch fabric, possibly due to a degraded or missing NVLink connection.. A healthy DGX H100 shows NV18 (NVLink 4.0, 18 links) for every GPU pair because all GPUs are connected through NVSwitch. A SYS entry for a specific pair means traffic between those GPUs falls back to the PCIe/CPU path, which severely reduces bandwidth and increases latency. This typically signals a hardware or cabling fault on the NVLink side that should be resolved before running distributed training.

Answer

The NVSwitch fabric is operating in a fallback mode that only activates when peer-to-peer traffic exceeds a threshold.

Answer

GPU4 and GPU5 are configured in MIG mode, which forces inter-GPU communication over the PCIe bus.

Answer

GPU4 and GPU5 are reserved for display output, so the driver automatically disables their NVLink interfaces.

Question 3

An administrator is responsible for a large NVIDIA DGX SuperPOD used for multi-node training. They need to ensure that GPU telemetry and health metrics are collected centrally and can trigger alerts when GPUs exceed temperature thresholds. Which component of NVIDIA Base Command Manager (BCM) should they configure to achieve this?

Accepted Answer

NVIDIA Data Center GPU Manager (DCGM) integrated with BCM. NVIDIA DCGM is the standard tool for centralized GPU telemetry and health monitoring in data centers. When integrated with Base Command Manager, it collects metrics from all nodes and can be configured with alerting rules for temperature and other thresholds. This provides the required centralized visibility and automated alerts for the DGX SuperPOD.

Answer

NVIDIA Base Command Manager's built-in 'cluster monitoring' dashboard

Answer

NVIDIA Management Library (NVML) on each node

Answer

NVIDIA Container Toolkit on each compute node

Question 4

An administrator is responsible for monitoring a large-scale AI cluster with hundreds of NVIDIA GPUs. They need to collect telemetry data such as GPU utilization, temperature, and power consumption from all nodes and store it centrally for analysis and alerting. Which NVIDIA tool should they use to collect and export GPU metrics to a monitoring system like Prometheus?

Accepted Answer

NVIDIA DCGM Exporter. For centralized monitoring of GPU metrics in a large cluster, NVIDIA DCGM Exporter is the appropriate tool. It leverages NVIDIA Data Center GPU Manager (DCGM) to collect a comprehensive set of metrics from each GPU and exposes them via an HTTP endpoint that Prometheus can scrape. This allows administrators to aggregate metrics across all nodes, create dashboards in Grafana, and set up alerts based on thresholds.

Answer

NVIDIA Nsight Systems

Answer

nvidia-smi

Answer

NVIDIA Base Command Manager

Question 5

An administrator is troubleshooting a multi-node NVIDIA GPU training job that intermittently hangs during the all-reduce phase. The job uses NCCL over InfiniBand. Logs show that some ranks time out while others complete. The administrator suspects a network fabric issue. Which action should the administrator take first to isolate whether the problem is in the InfiniBand fabric or in the NCCL configuration?

Accepted Answer

Run the NCCL tests (nccl-tests) with the same topology and environment variables, and enable NCCL debug logging to capture detailed transport and topology information.. Using nccl-tests with debug logging is a controlled way to reproduce the collective communication and observe transport selection, topology detection, and timeouts. It helps determine whether the issue is in NCCL configuration or the InfiniBand fabric. Replacing hardware, forcing TCP, or increasing timeouts are premature and do not isolate the root cause.

Answer

Disable InfiniBand and force NCCL to use TCP sockets instead, because TCP is more reliable for collective operations.

Answer

Immediately replace all InfiniBand cables and transceivers on the affected nodes, because intermittent hangs during all-reduce almost always indicate physical link errors.

Answer

Increase the NCCL timeout value in the training script and restart the job, because the hangs are likely due to slow network convergence.

Question 6

An administrator is responsible for maintaining a fleet of NVIDIA-certified servers running AI workloads. They need to quickly identify which servers have GPUs that are overheating and may throttle performance. Which NVIDIA tool should the administrator use to monitor GPU temperature across the fleet in real time?

Accepted Answer

NVIDIA Data Center GPU Manager (DCGM). DCGM is the NVIDIA tool purpose-built for data center GPU monitoring. It exposes temperature, power, utilization, and health metrics through APIs and integrations with popular monitoring platforms. For an administrator needing real-time temperature visibility across many servers, DCGM is the correct choice.

Answer

NVIDIA Nsight Systems

Answer

NVIDIA TensorRT

Answer

NVIDIA CUDA Toolkit

Question 7

An administrator is configuring NVIDIA GPUDirect Storage (GDS) on a cluster to accelerate data loading for AI training jobs. The cluster uses Mellanox InfiniBand adapters and NVMe storage. Which two actions are required to enable GDS and ensure optimal performance? (Choose two.)

Accepted Answer

Ensure that the storage devices and network adapters support RDMA and are properly configured for peer-to-peer communication.. Enabling GPUDirect Storage requires installing the GDS kernel module and user-space libraries on GPU nodes, and ensuring that storage and network hardware support RDMA and peer-to-peer communication. These two actions create the necessary software and hardware foundation for direct data transfers between storage and GPU memory. Other options are either incorrect or not specific to GDS.

Answer

Set the environment variable NVIDIA_GDS_ENABLE=1 on all nodes.

Answer

Configure the GPU nodes to use the NVIDIA Container Toolkit for all training containers.

Answer

Disable IOMMU in the BIOS to allow direct memory access between devices.

Question 8

An administrator is managing a Kubernetes cluster with NVIDIA GPU Operator. A new policy requires that all GPU workloads run with specific environment variables set, such as NVIDIA_VISIBLE_DEVICES and NVIDIA_DRIVER_CAPABILITIES. The administrator wants to enforce these variables automatically for any pod that requests a GPU, without modifying each pod specification manually. Which approach should the administrator use?

Accepted Answer

Create a MutatingAdmissionWebhook that injects the required environment variables into pods that request nvidia.com/gpu resources.. A MutatingAdmissionWebhook is the correct Kubernetes-native way to automatically inject environment variables into pods based on their resource requests. It can inspect incoming pod specs and add the required variables when a GPU resource is requested, ensuring consistent configuration without manual intervention. Validating webhooks only reject, and runtime configuration is not pod-specific.

Answer

Use a ValidatingAdmissionWebhook to reject pods that do not include the required environment variables.

Answer

Configure the NVIDIA GPU Operator to set default environment variables in the device plugin daemonset.

Answer

Set the environment variables in the NVIDIA Container Runtime configuration on each node.

Question 9

An administrator is using NVIDIA Base Command Manager to provision a new GPU cluster. They need to ensure that the compute nodes are configured with the correct GPU driver and CUDA toolkit versions. Which Base Command Manager feature should they use to automate this?

Accepted Answer

Node groups with software images. Base Command Manager uses node groups to categorize nodes with similar roles and software requirements. By assigning a software image to a node group, administrators define the exact GPU driver and CUDA toolkit versions. During provisioning, Base Command Manager applies the image, ensuring consistency across compute nodes. This is the intended feature for automating software configuration in a Base Command Manager cluster.

Answer

Ansible playbooks executed manually

Answer

PXE boot with custom kickstart scripts

Answer

Cron-based package updates

Question 10

A data scientist reports that a Jupyter notebook running on a GPU-enabled server is extremely slow when training a small neural network, even though nvidia-smi shows the GPU is idle. The notebook uses TensorFlow. Which is the most likely cause?

Accepted Answer

TensorFlow is not configured to use the GPU; it is running on the CPU.. The idle GPU during training strongly suggests TensorFlow is not using it. Common causes include missing GPU support in TensorFlow, incorrect CUDA/cuDNN versions, or the process not having access to the GPU. Verifying TensorFlow's device configuration is the first step. Other options are less likely given the evidence.

Answer

The GPU is being used by another process, causing contention.

Answer

The Jupyter notebook kernel needs to be restarted to detect the GPU.

Answer

The neural network is too small to benefit from GPU acceleration, so TensorFlow automatically uses the CPU.

Question 11

An AI operations team needs to monitor GPU health and utilization across a fleet of DGX nodes from a single dashboard. They want per-GPU metrics such as power, temperature, utilization, and ECC errors, and they want to retain historical data for capacity planning. Which NVIDIA tool is purpose-built to collect and expose these GPU telemetry metrics for centralized monitoring?

Accepted Answer

NVIDIA DCGM (Data Center GPU Manager) with the DCGM exporter for Prometheus.. DCGM is NVIDIA's purpose-built data center GPU monitoring and management tool. It gathers health and performance metrics including power, temperature, utilization, and ECC errors, and the DCGM exporter makes them available to Prometheus for centralized dashboards and historical retention. This aligns exactly with the team's need for fleet-wide GPU telemetry and capacity planning data.

Answer

NVIDIA Base Command Manager's job scheduler logs.

Answer

NVIDIA Nsight Systems for profiling GPU kernels and collecting timeline traces.

Answer

NVIDIA CUDA Toolkit's `nvidia-smi` command run manually on each node.

Question 12

A data scientist reports that a Jupyter notebook running on a DGX station cannot allocate GPU memory, even though other users' jobs are running fine. The notebook kernel was started before a system administrator updated the NVIDIA driver and rebooted the node. Which action should the data scientist take to resolve the issue?

Accepted Answer

Restart the Jupyter kernel to reinitialize CUDA and pick up the new driver.. After a driver update and reboot, any process that started before the update holds an outdated CUDA context. The Jupyter kernel is such a process, so it cannot use the new driver. Restarting the kernel creates a fresh process that loads the updated driver and can allocate GPU memory normally, resolving the issue without affecting other users.

Answer

Run `nvidia-smi --gpu-reset` to reset the GPU and clear any stale contexts.

Answer

Set the environment variable `CUDA_VISIBLE_DEVICES=0` to force the notebook to use a specific GPU.

Answer

Reinstall the NVIDIA driver using the `.run` installer without rebooting.

Question 13

An administrator is configuring an NVIDIA AI Enterprise cluster to run multi-tenant inference workloads on Kubernetes. The administrator must ensure that GPU resources are isolated and that tenants cannot access each other's GPU memory. Which two actions should the administrator take? (Choose two.)

Accepted Answer

Deploy the NVIDIA GPU Operator with the device plugin configured to advertise MIG resources.. Hardware-level isolation for multi-tenant inference on Kubernetes is achieved by enabling MIG on supported GPUs and having the GPU Operator's device plugin advertise MIG instances as schedulable resources. Together these actions partition the GPU into isolated slices and make those slices available to tenants without memory sharing, satisfying the isolation requirement.

Answer

Use Kubernetes ResourceQuota and LimitRange objects to restrict GPU memory per namespace.

Answer

Configure NVIDIA vGPU with different vGPU profiles for each tenant.

Answer

Enable time-slicing of GPUs so that multiple tenants share the same GPU context.

Question 14

An AI operations engineer is tuning a real-time inference service on NVIDIA A100 GPUs. Profiling with Nsight Systems shows that the GPU is idle for long periods while waiting for input data, and that host-to-device memory copies are frequent and small. The service uses a fixed batch size of 1 and a custom data loader. Which two changes are most likely to improve GPU utilization and reduce inference latency? (Choose two.)

Accepted Answer

Increase the inference batch size and implement dynamic batching to group requests.. The GPU idles because it waits on small, frequent data transfers and processes one sample at a time. Increasing batch size with dynamic batching and optimizing the model with TensorRT in FP16 reduce per-inference overhead and better utilize Tensor Cores. Together they increase work per transfer and speed up computation, directly improving utilization and latency.

Answer

Increase the number of CUDA streams used for memory copies and inference kernels.

Answer

Use `cudaMemcpy` instead of `cudaMemcpyAsync` for all host-to-device transfers.

Answer

Pin the inference process to a single CPU core to reduce context switching.

Question 15

An AI operations team is running a large language model inference service on NVIDIA H100 GPUs using NVIDIA Triton Inference Server. They observe that the first inference request after a period of inactivity takes significantly longer than subsequent requests. The model is loaded and ready, but the GPU shows low utilization during the first request. Which optimization should the team implement to reduce this latency spike?

Accepted Answer

Configure Triton's model warmup to run dummy inference requests during model loading.. The first inference after inactivity is slow because CUDA kernels and memory allocations are not yet initialized on the GPU. Triton's model warmup runs dummy requests at load time to trigger this initialization, so the first real request executes at normal speed. Other options target throughput or memory pooling but do not eliminate the cold-start penalty.

Answer

Enable Triton's dynamic batching with a large maximum batch size.

Answer

Set the Triton `--pinned-memory-pool-byte-size` to a larger value.

Answer

Increase the number of model instances per GPU to allow more concurrent executions.