NCP-AIO Practice Test 17 — 15 Questions

Question 1

An administrator supports a shared Kubernetes cluster running NVIDIA GPU Operator. Data scientists report that their inference pods remain in Pending state, yet the GPU Operator pods and node feature discovery pods are healthy, and the GPU nodes show no hardware alarms. The administrator confirms that the cluster has a mixture of MIG-capable A100 nodes and non-MIG T4 nodes. Which immediate administrative action is most appropriate to diagnose the scheduling failure?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Inspect the pod events with kubectl describe pod and verify the MIG strategy configured in the device plugin, because a mismatch between the node's MIG configuration and the requested resource can prevent GPU resource advertisement.. The pod events are the fastest way to see whether the scheduler cannot find a suitable node due to resource requests. In a mixed MIG and non-MIG environment, the device plugin advertises different resource names, and a mismatch between the requested GPU resource and what the node exposes leaves pods Pending. Verifying the MIG strategy and node labels directly identifies that mismatch.

Answer

Restart the kubelet on all worker nodes to force the device plugin to re-register GPU resources, because a stale kubelet cache is the most frequent cause of Pending workloads.

Answer

Delete the pending pods and recreate them with a higher priority class, because the scheduler may be preempting them in favor of system pods.

Answer

Scale the GPU Operator controller manager to zero replicas and back to one, because the operator may have lost track of node labels after a recent upgrade.

Question 2

An administrator needs to collect GPU telemetry from an NVIDIA AI Enterprise cluster and store it in a time-series database for long-term analysis. Which component should be deployed to export GPU metrics in Prometheus format?

Accepted Answer

NVIDIA DCGM Exporter. DCGM Exporter is the component that collects GPU metrics via DCGM and exposes them as Prometheus metrics. It is typically deployed as a daemonset or pod and can be scraped by Prometheus for long-term storage and analysis. Other NVIDIA components manage deployment or container runtime integration but do not provide metric export functionality.

Answer

NVIDIA Container Toolkit

Answer

NVIDIA Base Command Manager

Answer

NVIDIA GPU Operator

Question 3

An AI operations engineer notices that a training job on an NVIDIA A100 GPU is running slower than expected. Running nvidia-smi shows that the GPU is in 'P0' state but the 'SM Clock' is significantly lower than the maximum boost clock. The job is not memory-bound. Which action should the engineer take first to diagnose the issue?

Accepted Answer

Check the GPU's power consumption and whether it is hitting the power limit.. When a GPU is in P0 state but SM clock is low, power or thermal throttling is a likely cause. Checking power consumption and limits is a quick, non-invasive diagnostic step. If the GPU is hitting its power limit, the engineer can then adjust power settings or optimize the workload to reduce power draw. The other actions do not directly diagnose the throttling issue.

Answer

Reinstall the NVIDIA driver to ensure the latest version.

Answer

Increase the batch size to improve GPU utilization.

Answer

Switch the job to use mixed precision to reduce compute load.

Question 4

An administrator is responsible for an NVIDIA AI Enterprise deployment on Kubernetes. The security team requires that all GPU-accelerated pods run with the least privilege necessary and that GPU device nodes are not exposed to pods that do not request them. Which combination of configurations should the administrator implement to meet these requirements?

Accepted Answer

Deploy the NVIDIA GPU Operator with the device plugin and configure a PodSecurityPolicy or OPA Gatekeeper policy that requires pods to request nvidia.com/gpu and forbids privileged mode.. Using the GPU Operator with the device plugin, combined with a policy that mandates explicit nvidia.com/gpu requests and prohibits privileged containers, ensures that GPU device nodes are only injected into pods that request them and that pods run with minimal privileges. This satisfies both the least privilege and isolation requirements.

Answer

Use the NVIDIA GPU Operator with the device plugin and configure pod security contexts to drop all capabilities and add only the NVIDIA_VISIBLE_DEVICES environment variable.

Answer

Install the NVIDIA k8s-device-plugin standalone and set the --pass-device-specs flag to true, then allow all pods to run as root.

Answer

Enable the NVIDIA device plugin with the --fail-on-init-error=false flag and set privileged: true in the pod security context.

Question 5

An administrator manages an NVIDIA AI Enterprise deployment on Kubernetes and needs to ensure that GPU telemetry is exported to an existing Prometheus instance. The administrator deploys the NVIDIA DCGM Exporter but sees no GPU metrics in Prometheus. Which configuration should the administrator verify first?

Accepted Answer

The DCGM Exporter ServiceMonitor or PodMonitor selector labels match the Prometheus operator's serviceMonitorSelector.. In Prometheus Operator deployments, scrape targets are selected by label matching between the ServiceMonitor or PodMonitor and the Prometheus custom resource. If the DCGM Exporter's monitor labels do not align with the operator's selector, Prometheus never scrapes it, producing exactly the symptom of no GPU metrics. Verifying label alignment is therefore the correct first check.

Answer

The NVIDIA GPU Operator's device plugin is configured to advertise GPUs with the correct resource name.

Answer

The GPU driver version installed on the nodes matches the version bundled in the DCGM Exporter container.

Answer

The cluster's CNI plugin supports multicast so that Prometheus can discover the exporter.

Question 6

An administrator is preparing a cluster for a new large language model training job that will use NVIDIA Magnum IO GPUDirect Storage to stream training data directly from a parallel file system to GPU memory. The administrator must verify that the environment supports GPUDirect Storage before the job starts. (Choose two.)

Accepted Answer

Ensure that the parallel file system client supports the cuFile API and that the file system is mounted with options that allow direct I/O and peer-to-peer memory access.. GPUDirect Storage requires compatible drivers, the nvidia-fs kernel module, and a file system client that supports the cuFile API with suitable mount options. These two checks confirm that the software stack and storage client can perform direct memory transfers. MIG, exclusive RoCE, and disabling IOMMU are not prerequisites and may be counterproductive.

Answer

Disable IOMMU in the BIOS on all GPU nodes to allow direct memory access between storage devices and GPUs.

Answer

Configure the cluster to use RDMA over Converged Ethernet (RoCE) exclusively for all storage traffic, because GPUDirect Storage requires RDMA.

Answer

Enable NVIDIA Multi-Instance GPU (MIG) on every GPU to provide isolated memory partitions for storage transfers.

Question 7

An AI operations team is using NVIDIA DCGM (Data Center GPU Manager) to monitor a cluster of A100 GPUs. They want to set up proactive health checks to detect and mitigate GPU issues before they cause job failures. Which two DCGM features should they configure? (Choose two.)

Accepted Answer

DCGM policy management for automated remediation. DCGM health checks with periodic diagnostics and DCGM policy management for automated remediation are the two features that directly enable proactive health monitoring and mitigation. Health checks detect issues, and policies define actions to take when issues are found. Together, they form a proactive health management system. Other options are either monitoring metrics or management features that do not directly address proactive health.

Answer

DCGM profiling metrics for real-time utilization

Answer

DCGM group configuration for multi-node synchronization

Answer

DCGM API integration with Prometheus for alerting

Question 8

An AI engineer is optimizing a real-time inference pipeline on an NVIDIA A100 GPU. The model uses dynamic input shapes, and profiling shows that the GPU spends significant time on memory copies between host and device. Which optimization should be implemented to reduce this overhead?

Accepted Answer

Use pinned (page-locked) host memory for input and output buffers.. The overhead from host-device memory copies is best reduced by using pinned memory, which enables faster DMA transfers. This is especially important for real-time inference with dynamic shapes where copies are frequent. While CUDA streams can overlap transfers with compute, they are more effective when combined with pinned memory.

Answer

Enable Unified Memory to automatically manage data migration.

Answer

Increase the batch size to amortize the cost of memory copies.

Answer

Use CUDA streams to overlap memory copies with computation.

Question 9

A team is deploying a large language model for inference using NVIDIA TensorRT-LLM on an H100 GPU. They observe that the first inference request takes several seconds, while subsequent requests are fast. They want to reduce this initial latency. Which technique should they implement?

Accepted Answer

Precompile the TensorRT engine and load it at server startup, then perform a warm-up inference.. The first inference request incurs one-time costs such as TensorRT engine deserialization, CUDA context setup, and kernel compilation/loading. By precompiling the engine and loading it at startup, and then running a warm-up inference, these costs are moved to server initialization. Subsequent requests then benefit from a fully initialized environment, reducing the observed initial latency.

Answer

Increase the GPU's power limit to boost clock speeds during the first request.

Answer

Use TensorRT-LLM's built-in paged KV cache and enable continuous batching.

Answer

Reduce the model's precision to INT4 to decrease computation time.

Question 10

A team runs multi-node training with NCCL over InfiniBand on a cluster of DGX systems. Jobs scale well to four nodes but throughput drops sharply at eight nodes, and `nccl-tests` all-reduce bandwidth falls well below line rate at that size. The fabric uses a fat-tree topology with adaptive routing enabled. Which investigation is most likely to reveal the cause?

Accepted Answer

Check whether NCCL is selecting the correct InfiniBand HCAs and whether the ring or tree algorithm choice matches the fabric's oversubscription ratio.. NCCL chooses rings and trees based on detected topology, and on multi-node jobs this can cross oversubscribed uplinks or use a suboptimal adapter when several are present. When scaling stops at a specific node count, the cause is usually that the communication pattern no longer matches the fabric's capacity. Confirming HCA selection and steering the algorithm or topology to respect the oversubscription ratio restores bandwidth and explains why smaller jobs appeared fine.

Answer

Move the collective operations from NCCL to a TCP socket backend over the management network.

Answer

Increase the batch size per GPU so that communication is amortized over more compute.

Answer

Disable adaptive routing on the InfiniBand fabric to force deterministic paths.

Question 11

An administrator is managing an NVIDIA DGX SuperPOD used for large-scale AI training. The cluster uses a Slurm workload manager. The administrator needs to ensure that jobs are scheduled only on nodes with healthy GPUs and that failed GPUs are automatically drained from the pool. Which integration should be configured to achieve this?

Accepted Answer

NVIDIA Data Center GPU Manager (DCGM) with the Slurm health check plugin. DCGM with the Slurm health check plugin is the correct integration because it continuously monitors GPU health and can automatically drain nodes with failed GPUs from Slurm, ensuring jobs run only on healthy hardware. Other options lack the necessary health monitoring and automatic remediation capabilities.

Answer

NVIDIA Base Command Manager with Slurm integration

Answer

NVIDIA Fleet Command with Slurm edge scheduling

Answer

NVIDIA Container Toolkit with Slurm's --gres flag

Question 12

An AI operations engineer is investigating a training job that exhibits poor scaling efficiency when moving from 8 to 32 GPUs on a DGX SuperPOD. Profiling indicates that the communication time in NCCL all-reduce operations is disproportionately high. Which two actions should be taken to improve scaling efficiency? (Choose two.)

Accepted Answer

Check that the InfiniBand fabric is not experiencing congestion or errors by monitoring performance counters.. Poor scaling in NCCL all-reduce often stems from network misconfiguration or fabric issues. Verifying that NCCL uses the correct high-bandwidth HCAs and checking the InfiniBand fabric for congestion or errors are essential first steps. These actions ensure that communication occurs over the fastest available path and that the network is healthy, directly improving scaling efficiency.

Answer

Ensure that the job is using the correct NCCL topology by setting NCCL_TOPO_FILE to a custom topology XML.

Answer

Increase the NCCL buffer size by setting NCCL_BUFFSIZE to a larger value.

Answer

Enable NCCL tree algorithm for all-reduce by setting NCCL_ALGO=Tree.

Question 13

An administrator is setting up an NVIDIA AI Enterprise cluster and wants to verify that the NVIDIA GPU Operator has successfully deployed all required components on a worker node. Which command should the administrator use to list the GPU Operator pods running on that node?

Accepted Answer

kubectl get pods -n gpu-operator --field-selector spec.nodeName=<node-name>. The NVIDIA GPU Operator runs its components as pods in the gpu-operator namespace. To confirm deployment on a specific worker node, filtering pods by spec.nodeName in that namespace lists exactly the operator pods scheduled there. This provides direct evidence that the driver, container toolkit, device plugin, and related daemonsets are running on the node.

Answer

nvidia-smi -q -d COMPUTE

Answer

kubectl describe node <node-name> | grep nvidia

Answer

helm list -n gpu-operator

Question 14

An AI operations engineer is troubleshooting a model inference service deployed with NVIDIA Triton Inference Server on a GPU. The service occasionally returns incorrect predictions, and the engineer suspects that the input data is not being preprocessed correctly. The model expects input tensors in FP32 format, but the client is sending FP16 data. Which action should the engineer take to resolve the issue?

Accepted Answer

Configure the Triton model's input to specify the correct data type (FP32) and ensure the client sends matching data.. The root cause is a mismatch between the client's data type (FP16) and the model's expected input type (FP32). Triton requires that input tensors match the model's configuration. By setting the correct data type in the model configuration and ensuring the client sends FP32 data, the engineer ensures proper data handling and restores prediction accuracy.

Answer

Enable dynamic batching in Triton to automatically convert FP16 inputs to FP32.

Answer

Set the model to use FP16 precision to match the client's input data type.

Answer

Increase the instance count to handle the load and reduce the chance of data corruption.

Question 15

A data scientist reports that a Jupyter notebook running on a NVIDIA GPU server is extremely slow when executing a deep learning model, even though `nvidia-smi` shows the GPU is idle. The notebook uses TensorFlow. Which action should be taken first to diagnose the issue?

Accepted Answer

Check that TensorFlow is configured to use the GPU by running `tf.config.list_physical_devices('GPU')`.. An idle GPU during a slow TensorFlow workload strongly suggests that TensorFlow is not using the GPU. The quickest way to confirm is to check if TensorFlow detects the GPU via `tf.config.list_physical_devices('GPU')`. If the list is empty, the issue is likely missing CUDA/cuDNN libraries or a CPU-only TensorFlow installation. This diagnostic step is non-disruptive and directly targets the root cause.

Answer

Increase the notebook's memory allocation by setting `TF_GPU_ALLOCATOR=cuda_malloc_async`.

Answer

Restart the Jupyter kernel and clear the GPU memory with `nvidia-smi --gpu-reset`.

Answer

Run `nvidia-smi -q -d PERFORMANCE` to check if the GPU is throttled due to thermal or power issues.