NCP-AIO Practice Exam 2 — 20 Questions

Question 1

A production inference service using TensorRT is showing lower than expected throughput. Profiling shows that the model is spending significant time in "host-to-device" transfers. What is the best optimization?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use pinned memory for data transfers.. Data movement is a primary latency bottleneck in deep learning inference. By using pinned memory (page-locked memory) for host-side buffers, the system can enable faster direct memory access (DMA) transfers between the CPU and the GPU. This minimizes the time spent in data copy operations, directly reducing latency and increasing total throughput, which is essential for meeting strict Service Level Agreements (SLAs) in production AI deployments.

Answer

Increase the GPU clock speed via nvidia-smi.

Answer

Switch to FP64 precision for higher accuracy.

Answer

Implement multi-threaded data preprocessing on the GPU.

Question 2

Which TWO of the following are benefits of using containerized GPU workloads compared to bare-metal deployment?

Accepted Answer

Easier management of heterogeneous software dependencies.. Containerization provides portability, version control, and dependency isolation, which are essential for managing complex AI stacks. By packaging the environment, developers ensure that models run consistently across development, testing, and production clusters. This significantly reduces 'environment drift' and simplifies the management of different CUDA, cuDNN, and framework versions required by various research projects within the same shared infrastructure cluster.

Answer

Significant increase in raw GPU compute performance.

Answer

Automatic elimination of GPU driver compatibility issues.

Answer

Direct access to the GPU firmware for kernel customization.

Question 3

When managing large-scale model training jobs, what is the primary purpose of using a Job Scheduler like Slurm or Kubernetes Batch?

Accepted Answer

To automate resource allocation, job queuing, and throughput optimization.. Job schedulers serve as the orchestration layer to queue, prioritize, and allocate computational resources based on policy. In AI operations, they ensure that high-priority training runs receive the necessary GPU throughput while lower-priority jobs wait. By automating the allocation process, schedulers prevent resource idleness and manage contention, allowing data scientists to focus on model development rather than manual infrastructure management or resource conflict resolution during heavy cluster usage.

Answer

To increase the clock speed of individual GPU cores.

Answer

To provide a direct IDE interface for writing model code.

Answer

To replace the need for containerization technologies.

Question 4

An engineer is troubleshooting a CUDA program that terminates unexpectedly. Which tool should be used to detect memory leaks and race conditions in the CUDA kernel code?

Accepted Answer

NVIDIA Compute Sanitizer. Debugging parallel code is inherently difficult due to the non-deterministic nature of race conditions and memory access patterns. The NVIDIA Compute Sanitizer is the essential diagnostic tool for identifying these issues, such as out-of-bounds memory access or race conditions, before they lead to erratic, intermittent crashes in production. Using it early in the development lifecycle saves massive amounts of time by ensuring code correctness and robustness.

Answer

nvidia-smi

Answer

nsys (Nsight Systems)

Answer

nvprof

Question 5

Which security configuration is necessary when deploying NVIDIA GPUs in a multi-tenant environment to prevent unauthorized access between containers?

Accepted Answer

Enable MIG (Multi-Instance GPU).. In multi-tenant environments, using MIG (Multi-Instance GPU) is the standard method for hardware-level isolation. MIG allows a single physical GPU to be partitioned into multiple isolated instances, each with its own dedicated compute and memory. This ensures that one tenant's workload cannot interfere with or access the resources of another, providing a robust security boundary that software-only isolation cannot reliably guarantee in accelerated computing environments.

Answer

Increase the Kubernetes pod memory limit.

Answer

Use a privileged container.

Answer

Install the latest version of CUDA.

Question 6

What is the primary function of an 'InitContainer' in an NVIDIA GPU-enabled pod deployment?

Accepted Answer

To ensure prerequisites are satisfied before the main application starts.. InitContainers are often used to ensure that environment-specific requirements—such as configuring GPU drivers, validating libraries, or mounting persistent volumes—are met before the main training application starts. In AI workloads, this is critical for ensuring that dependencies are correctly loaded or data is staged, preventing runtime failures that would occur if the main training process attempted to execute in an incomplete environment. This pattern enhances the robustness of automated AI workflows.

Answer

To serve as a secondary GPU compute thread during training.

Answer

To permanently store the model weights after training.

Answer

To bypass GPU scheduling constraints and force-load the job.

Question 7

An AI operations engineer is troubleshooting an inference service on an NVIDIA A100 GPU that shows intermittent stalls. The monitoring dashboard reports GPU utilization at 100%, but request throughput is far below the validated baseline. Running nvidia-smi dmon reveals the SM utilization is high while memory controller utilization is low. Which action should the engineer take first to identify the bottleneck?

Accepted Answer

Profile the inference process with Nsight Systems to inspect kernel execution and identify serialization gaps.. The high SM utilization with low memory controller utilization suggests the GPU is busy but not doing useful work, often due to kernel serialization or CPU-GPU synchronization stalls. Profiling with Nsight Systems provides the timeline needed to see gaps and serialization. Only after identifying the specific stall should configuration changes be considered, making profiling the correct first step.

Answer

Switch the inference precision from FP16 to FP32 to improve numerical stability.

Answer

Enable MPS (Multi-Process Service) to allow concurrent kernel execution from multiple processes.

Answer

Increase the batch size in the inference server configuration to improve GPU occupancy.

Question 8

What is the primary function of the NVIDIA Persistence Daemon in an AI deployment?

Accepted Answer

It keeps the GPU driver initialized to reduce latency during application startup.. The Persistence Daemon ensures that the NVIDIA driver remains loaded even when no applications are using the GPU. This prevents the driver from unloading and then reloading when a job starts, which significantly reduces the startup latency of AI models. It is a critical configuration for high-performance environments where frequent job scheduling would otherwise incur unnecessary overhead from repeated driver and device initialization.

Answer

It automatically updates the NVIDIA driver when a new version is released on the web.

Answer

It monitors GPU health and automatically initiates a reboot if an error is detected.

Answer

It manages the network traffic between the GPU and the storage backend.

Question 9

A financial services firm must prove to auditors that an AI training job ran on hardware located only in its Frankfurt data center and that no pod could ever be scheduled onto GPUs in other regions. The cluster spans three regions with nodes labeled topology.kubernetes.io/region. Which approach most directly enforces this placement requirement?

Accepted Answer

Define a nodeAffinity rule in the pod spec requiring topology.kubernetes.io/region to equal eu-central-1, and mark it requiredDuringSchedulingIgnoredDuringExecution.. Hard placement guarantees come from required node affinity, which makes matching labels a precondition for scheduling. Because the scheduler will not place the pod on any node whose region label differs, the job cannot leave the Frankfurt data center, giving the firm an enforceable, auditable control over GPU location.

Answer

Add a preferredDuringSchedulingIgnoredDuringExecution nodeAffinity rule that favors the Frankfurt region with a high weight.

Answer

Set the pod's restartPolicy to Never and add a toleration for the region-specific taint on Frankfurt nodes.

Answer

Create a PodDisruptionBudget for the training job that limits voluntary evictions to zero during the run.

Question 10

An administrator needs to ensure that all GPU drivers are updated across a heterogeneous cluster without causing downtime. What is the best strategy?

Accepted Answer

Perform rolling updates by draining nodes one by one. Implementing a rolling update strategy allows the cluster to be updated one node at a time while the remaining nodes continue to process training jobs. By draining the target node of all active jobs, upgrading the drivers, and then re-adding the node to the production pool, the administrator maintains service availability. This is the standard practice in production AI operations to minimize disruption and ensure consistent driver versions across the entire fleet.

Answer

Simultaneously update all nodes during a maintenance window

Answer

Use a containerized driver approach to avoid host updates

Answer

Only update the driver on the master node

Question 11

An administrator is deploying the NVIDIA GPU Operator on a Kubernetes cluster in an air-gapped environment. The cluster nodes have no internet access, and all container images must be pulled from a private registry. Which two actions are required to ensure a successful deployment? (Choose two.)

Accepted Answer

Configure the GPU Operator to use the private registry by setting the image repository in the Helm chart values.. Air-gapped deployments require that all container images are available in a private registry, and the GPU Operator must be configured to pull from that registry. Mirroring the images ensures availability, and setting the image repository in the Helm values directs the Operator to the correct location. These two actions are essential for a successful deployment without internet access.

Answer

Ensure that the private registry supports manifest lists for multi-architecture images.

Answer

Disable the driver container and install drivers manually on each node.

Answer

Set the environment variable 'AIRGAP=true' in the operator's deployment.

Question 12

When configuring a node for NVIDIA AI Enterprise in a Kubernetes environment, what is the primary function of the NVIDIA Container Toolkit?

Accepted Answer

Exposing GPUs to containers. The NVIDIA Container Toolkit allows container engines, such as Docker or containerd, to interface with physical GPUs. It enables the exposure of GPUs inside containers, ensuring that applications can access CUDA libraries and hardware acceleration. Without this toolkit, the container runtime cannot bridge the gap between the host's GPU hardware and the containerized process, effectively rendering the GPU unusable for AI workloads.

Answer

Provisioning virtual machines

Answer

Optimizing neural network layers

Answer

Managing Kubernetes network policies

Question 13

A platform team runs an NVIDIA AI cluster with the GPU Operator deployed. Users submit jobs directly with kubectl and frequently request whole GPUs even when their notebooks only need a fraction of one. The team wants Kubernetes itself to admit and queue jobs based on GPU demand without users changing their manifests. Which component should the team deploy to meet this requirement?

Accepted Answer

NVIDIA KAI Scheduler. The requirement is for Kubernetes to make admission and queueing decisions based on GPU demand while users submit normal manifests. KAI Scheduler is the NVIDIA component that provides queue-based, gang-aware scheduling for AI workloads and understands fractional GPU requests, so jobs wait rather than fail. The GPU Operator, MIG Manager, and DCGM Exporter each address installation, partitioning, or observability rather than scheduling admission.

Answer

NVIDIA MIG Manager configured for every GPU

Answer

NVIDIA GPU Operator with the device plugin enabled

Answer

NVIDIA DCGM Exporter with Prometheus alerting

Question 14

Which strategy is most effective for managing heterogeneous GPU clusters containing both older architectures (e.g., V100) and newer architectures (e.g., H100)?

Accepted Answer

Using Node Selectors and Taints/Tolerations.. Using Kubernetes Node Selectors and Taints/Tolerations allows administrators to route specific workloads to compatible hardware. This is essential because newer GPUs support features like MIG or specific precision formats that older hardware lacks. By properly tagging nodes based on architecture, AI Ops prevents 'Incompatible Device' errors and ensures that high-performance training jobs are scheduled on hardware that meets their specific computational requirements, optimizing cluster-wide performance and efficiency.

Answer

Enabling uniform scheduling across all nodes.

Answer

Configuring the NVIDIA Device Plugin to hide older GPUs.

Answer

Deploying a single unified container image for all jobs.

Question 15

A platform team is preparing a bare-metal Kubernetes cluster to run AI workloads. They want the GPU Operator to install and manage the NVIDIA driver automatically on each node. Which prerequisite must be satisfied on every worker node before the GPU Operator can succeed?

Accepted Answer

The node must have a supported Linux kernel with matching kernel headers available for the driver container to build against.. The driver container builds the NVIDIA kernel module at runtime against the host kernel, so matching kernel headers or development packages must be present on each node. The GPU Operator manages the container toolkit, DCGM, and device plugin itself, so those do not need to be preinstalled. vGPU Manager applies to virtualized deployments, not bare-metal driver automation.

Answer

The node must have the NVIDIA Container Toolkit preinstalled and configured.

Answer

The node must be running the NVIDIA Data Center GPU Manager (DCGM) as a systemd service before joining the cluster.

Answer

The node must have the NVIDIA vGPU Manager installed so the physical GPU can be partitioned before driver installation.

Question 16

A DevOps engineer needs to monitor GPU health metrics in real-time for workload management. Which tool provides the most granular visibility into GPU utilization and power consumption for individual containers?

Accepted Answer

NVIDIA DCGM Exporter.. The NVIDIA DCGM (Data Center GPU Manager) Exporter is the industry-standard tool for collecting fine-grained GPU telemetry. By integrating with Prometheus and Grafana, it allows administrators to track metrics like power usage, temperature, and utilization at the container level. This granular data is vital for proactive workload management, allowing for autoscaling based on actual hardware demand rather than generic CPU metrics.

Answer

Kubernetes Horizontal Pod Autoscaler (HPA) with CPU metrics.

Answer

Standard Linux 'top' command.

Answer

Docker stats command.

Question 17

When configuring a multi-tenant environment on an NVIDIA DGX system, how are MIG (Multi-Instance GPU) instances best provisioned?

Accepted Answer

Using the nvidia-smi command to define the desired MIG profiles.. MIG allows a single GPU to be partitioned into multiple isolated instances, each with dedicated compute and memory resources. Configuring MIG via the nvidia-smi tool at the OS level ensures that these partitions are visible to the container runtime. This is crucial for AI operations, as it enables safe, secure multi-tenancy where different users can run jobs simultaneously without resource contention or cross-tenant interference.

Answer

Through the BIOS settings, creating physical partitions on the GPU die.

Answer

By editing the /etc/nvidia/mig-config.json file directly.

Answer

Via the OS kernel boot parameters in the GRUB configuration.

Question 18

A Kubernetes cluster running the NVIDIA GPU Operator is shared by an inference team and a research team. The research team's training pods repeatedly evict the inference pods from GPUs, causing latency spikes in production. The administrator wants to guarantee that inference pods always get GPU access first. Which Kubernetes scheduling mechanism should be configured?

Accepted Answer

Assign a higher PriorityClass to the inference pods and enable preemption on the scheduler.. Priority and preemption are the native Kubernetes controls that decide which pods win when GPU resources are contended. Giving inference pods a higher PriorityClass lets the scheduler admit them first and evict lower-priority training pods when necessary, which directly satisfies the requirement that production inference always obtains GPU access.

Answer

Enable the NVIDIA MIG feature on all GPUs and dedicate a MIG instance to each inference pod.

Answer

Configure a taint on the GPU nodes and add the corresponding toleration only to the inference pods.

Answer

Create a PodDisruptionBudget for the inference deployment and set maxUnavailable to zero.

Question 19

Refer to the exhibit. The system has two GPUs. What is the most likely cause of the observed performance discrepancy?

Accepted Answer

Only one GPU is receiving the workload due to improper data distribution.. The exhibit shows one GPU heavily utilized while the second is idling despite similar memory consumption. This indicates a data parallelism imbalance, where one process is doing the bulk of the work. This is a common issue in multi-GPU setups where workload distribution is not correctly handled, leading to massive inefficiencies where expensive hardware is under-utilized, significantly increasing the time required for model training or inference tasks.

Answer

The GPUs have different amounts of physical memory.

Answer

The system is limited by the PCIe bus bandwidth.

Answer

The model is too small to be parallelized across two GPUs.

Question 20

Refer to the exhibit. The training job is showing intermittent "thermal throttling" warnings. Which configuration change is the most appropriate adjustment?

Accepted Answer

Lower the "power_limit_watts" value.. Thermal throttling occurs when the GPU reaches its maximum operating temperature and lowers its clock speed to prevent physical damage. While power limits can be adjusted to reduce heat, simply lowering the limit may negatively impact training performance. Adjusting the power limit to a value that balances thermal overhead with compute requirements is a necessary trade-off to ensure stable, consistent performance during long training runs without hardware damage.

Answer

Set "compute_mode" to "exclusive_process".

Answer

Disable ECC mode to lower the power consumption.

Answer

Set "persistence" to "disabled".