NCP-AIO Workload Management Practice Test 4 — 15 Questions

Question 1

Which of the following is the most appropriate workload management technique for a bursty AI inference workload that requires low latency but does not need full GPU power for every request?

Accepted Answer

Using MIG or fractional GPU sharing to maximize hardware throughput.. NVIDIA Multi-Instance GPU (MIG) combined with time-slicing or fractional GPU sharing is ideal for inference workloads. By partitioning GPUs, the system can handle many smaller, latency-sensitive requests efficiently. This approach balances the need for high-performance hardware with the requirement for multi-tenancy, ensuring that no single inference request monopolizes the entire GPU while maintaining the fast response times required by the application.

Answer

Assigning one full physical GPU to every single inference request.

Answer

Configuring the GPU to run in 'Maximum Performance' mode at all times.

Answer

Implementing a strict FIFO queue for all incoming inference requests.

Question 2

In an NVIDIA-accelerated Kubernetes environment, why is it critical to configure a 'RuntimeClass' for GPU-enabled pods?

Accepted Answer

To ensure the container runtime correctly injects GPU drivers and libraries into the pod.. The RuntimeClass ensures that the container is started with the correct NVIDIA container runtime (e.g., nvidia-container-runtime). This runtime is responsible for performing the necessary hardware mapping, injecting the CUDA libraries, and setting the environment variables required for the container to actually utilize the GPU. Without this configuration, the pod might be scheduled successfully but will fail at runtime because it cannot communicate with the NVIDIA driver.

Answer

To allow the Kubernetes scheduler to place pods on nodes based on GPU availability.

Answer

To define the specific NVIDIA driver version required by the application.

Answer

To increase the priority of GPU workloads over standard CPU-only workloads.

Question 3

An organization is migrating their on-premises AI training to a hybrid cloud environment. Which component is most important to maintain consistent workload management across both the on-premises DGX systems and cloud-based GPU nodes?

Accepted Answer

Using a unified Kubernetes orchestration layer with consistent GPU Operator versions.. A unified Kubernetes control plane, managed by an orchestrator like NVIDIA Base Command or a managed Kubernetes service (e.g., GKE or EKS with NVIDIA GPU support), provides a consistent API. This allows developers to use the same manifest files and CI/CD pipelines regardless of whether the physical hardware is in a local datacenter or in the cloud. It ensures that GPU scheduling, resource requests, and monitoring tools remain identical, simplifying the operations lifecycle.

Answer

Deploying identical physical GPU hardware across all sites.

Answer

Hard-coding all GPU resource requests to match the smallest cloud GPU instance.

Answer

Implementing separate workload managers for cloud and on-premises sites.

Question 4

An AI researcher is running a large-scale training job on an NVIDIA DGX system using Kubernetes. They observe that GPU utilization is consistently low despite high CPU load. Which workload management configuration is most likely to resolve this bottleneck by optimizing data pipeline throughput?

Accepted Answer

Optimize the data loader by increasing worker threads and implementing buffered prefetching.. Low GPU utilization often stems from data starvation where the GPU waits for the CPU to preprocess or fetch data from storage. Implementing data prefetching and increasing the number of workers in the data loader ensures the GPU remains fed with batches. This is critical in AI operations to maximize compute ROI and reduce total training time for expensive cluster resources.

Answer

Increase the GPU memory limit in the pod specification.

Answer

Enable NVIDIA Multi-Instance GPU (MIG) for this specific job.

Answer

Lower the batch size to reduce the memory footprint on the GPU.

Question 5

Which TWO strategies should an administrator implement to ensure fair resource scheduling in a multi-tenant NVIDIA cluster using Kubernetes and the NVIDIA device plugin?

Accepted Answer

Configure Kubernetes ResourceQuotas to limit the total number of GPUs per namespace.. Fair scheduling in multi-tenant environments requires both hard limits to prevent resource monopolization and priority-based mechanisms to ensure critical jobs progress. By combining ResourceQuotas for capacity management and PriorityClasses for job scheduling, admins can prevent a single user from starving the cluster while maintaining high performance for latency-sensitive inference or urgent training tasks during peak usage windows.

Answer

Implement static partitioning for all GPUs in the cluster.

Answer

Use the NVIDIA Triton Inference Server to manage all batch processing.

Answer

Disable the NVIDIA device plugin to allow direct node access.

Question 6

A DevOps engineer needs to monitor GPU health metrics in real-time for workload management. Which tool provides the most granular visibility into GPU utilization and power consumption for individual containers?

Accepted Answer

NVIDIA DCGM Exporter.. The NVIDIA DCGM (Data Center GPU Manager) Exporter is the industry-standard tool for collecting fine-grained GPU telemetry. By integrating with Prometheus and Grafana, it allows administrators to track metrics like power usage, temperature, and utilization at the container level. This granular data is vital for proactive workload management, allowing for autoscaling based on actual hardware demand rather than generic CPU metrics.

Answer

Kubernetes Horizontal Pod Autoscaler (HPA) with CPU metrics.

Answer

Standard Linux 'top' command.

Answer

Docker stats command.

Question 7

An organization is migrating AI workloads to a private cloud. Which feature is essential for ensuring that GPU resources are dynamically reclaimed and reallocated to different departments without manual intervention?

Accepted Answer

Automated cluster autoscaling and priority-based scheduling.. Dynamic resource scheduling, often implemented via Kubernetes schedulers or custom job orchestrators, is essential for multi-tenant environments. By utilizing features like auto-scaling, preemption, and resource quotas, the system can automatically reclaim idle GPUs from one department and reallocate them to another. This automation maximizes hardware utility and ensures that expensive GPU infrastructure is never left sitting idle due to administrative delays.

Answer

Manual pod scheduling with affinity rules.

Answer

Static GPU reservation per user.

Answer

Hard-coding node IP addresses in the configuration files.

Question 8

Which THREE factors must be considered when sizing a persistent storage solution for multi-node distributed training checkpoints?

Accepted Answer

Aggregate write bandwidth to avoid stalling the training loop.. Checkpointing large models requires significant I/O throughput to avoid blocking the training loop, sufficient capacity for multiple historical versions, and high availability to ensure data integrity. By addressing these three factors—throughput, capacity, and reliability—organizations can ensure that the training process remains robust against hardware failures without incurring unnecessary performance penalties during the frequent write cycles required for large-scale model training.

Answer

The number of CPU cores on the storage controller.

Answer

The number of users accessing the storage simultaneously.

Question 9

What is the primary benefit of using NVIDIA GPU Operator in a Kubernetes cluster for workload management?

Accepted Answer

It provides a declarative way to manage GPU drivers and software components.. The NVIDIA GPU Operator automates the lifecycle management of NVIDIA software components, including drivers, the container toolkit, and device plugins. By standardizing these installations across all cluster nodes, it ensures a consistent and stable environment for AI workloads, reducing configuration drift and manual operational overhead that often plagues large-scale distributed infrastructure deployments.

Answer

It enables automatic model fine-tuning for specific frameworks.

Answer

It optimizes the neural network architecture automatically.

Answer

It eliminates the need for containerization in AI workflows.

Question 10

When running multi-instance GPU (MIG) workloads, what is the main advantage of assigning specific MIG profiles to different Kubernetes namespaces?

Accepted Answer

It enhances GPU memory security by isolating workload address spaces.. MIG profiles allow for fine-grained hardware isolation, ensuring that different workloads—such as small inference tasks and large training runs—do not interfere with each other. By mapping profiles to specific namespaces, administrators can enforce strict hardware separation, preventing 'noisy neighbor' issues where one workload consumes memory bandwidth or compute cycles required by another, thus improving overall cluster stability and predictability.

Answer

It allows the GPU to switch between different CUDA versions per instance.

Answer

It automatically compresses the model weights for faster loading.

Answer

It allows the cluster to bypass standard Kubernetes scheduler policies.

Question 11

Which approach is most effective for scaling an inference workload that experiences sudden, unpredictable spikes in request volume?

Accepted Answer

Implementing HPA with custom metrics provided by DCGM Exporter.. Predictive or reactive autoscaling based on custom GPU metrics (like utilization) allows the cluster to adjust replica counts in real-time. By monitoring the GPU load and scaling the inference pods accordingly, the system maintains low latency during spikes while keeping costs low during idle periods. This responsiveness is vital for production AI services where performance SLAs are tied directly to user experience.

Answer

Setting a fixed number of replicas to match peak expected load.

Answer

Using a load balancer to redirect traffic to an idle cluster.

Answer

Increasing the memory allocation for each inference container.

Question 12

When debugging a workload that consistently crashes with 'Out of Memory' (OOM) errors despite sufficient GPU VRAM, what is the most likely cause related to workload management?

Accepted Answer

The container memory limit is lower than the data processing requirements.. OOM errors can occur due to host-side system memory exhaustion if the workload manages large datasets in system RAM before loading them into the GPU. If the container memory limit is set too low for the data processing pipeline, the entire container will be terminated. This highlights the need to correctly balance both GPU VRAM and system memory limits in the container specification.

Answer

The CUDA driver version is incompatible with the installed GPU.

Answer

The GPU device plugin is not correctly reporting free memory.

Answer

The training job is using mixed-precision training (FP16).

Question 13

Which scheduling strategy is recommended to maximize the efficiency of long-running training jobs on preemptible instances?

Accepted Answer

Implementing granular checkpointing and automated job resumption.. Long-running jobs on preemptible (or spot) instances require frequent, efficient checkpointing to survive node reclamation. By combining a robust checkpointing schedule with smart job resubmission logic that monitors for preemption signals, organizations can take advantage of low-cost instances while minimizing the loss of progress. This approach allows for significant cost savings in non-critical training without sacrificing the overall reliability of the research project.

Answer

Avoiding preemptible instances for all training jobs.

Answer

Locking the job to a specific node using node affinity.

Answer

Increasing the priority of the job to the maximum level.

Question 14

Refer to the exhibit. A cluster administrator notices that GPU jobs with this PriorityClass are failing to start even when empty GPUs are available. What is the most likely cause?

Accepted Answer

Insufficient nodes satisfy the 'minAvailable' requirement for the pod group.. When using the Volcano scheduler with NVIDIA GPUs, priority classes alone are insufficient if the Gang Scheduling policy is not met. If the job requests multiple pods that cannot be satisfied simultaneously, Volcano will leave the pods in a pending state to avoid resource fragmentation. This ensures that massive training jobs do not consume isolated resources that would otherwise result in deadlocks during collective communication synchronization.

Answer

The PriorityClass value is too low to trigger preemption.

Answer

The NVIDIA Device Plugin requires a restart to recognize the PriorityClass.

Answer

The PriorityClass name violates Kubernetes naming conventions.

Question 15

Which NVIDIA technology allows for partitioning a single physical GPU into multiple independent instances, each with dedicated compute and memory resources for smaller workloads?

Accepted Answer

Multi-Instance GPU (MIG). Multi-Instance GPU (MIG) is a feature in NVIDIA A100 and H100 architectures that enables the partitioning of a physical GPU into up to seven separate instances. This is essential for AI Ops to maximize resource utilization by running multiple inference tasks on a single GPU without interference, ensuring Quality of Service for each instance while preventing a single process from consuming all hardware resources.

Answer

CUDA Streams

Answer

NVIDIA Collective Communications Library (NCCL)

Answer

NVIDIA Container Runtime