Scenario practice questions

A research lab is deploying NVIDIA AI Enterprise on an air-gapped Kubernetes cluster. The cluster has no internet access, and all software must be installed from a local registry. The administrator plans to use the NVIDIA GPU Operator. Which two actions must be performed to ensure a successful deployment in this environment? (Choose two.)

Trap 1: Install the NVIDIA GPU Operator using the --offline flag.

There is no --offline flag for the NVIDIA GPU Operator Helm chart or installation. The Operator does not have a built-in offline mode. Air-gapped deployments are handled by mirroring images and configuring the registry endpoints. Relying on a non-existent flag would cause installation failure. Therefore, this is not a valid action for the scenario.

Trap 2: Disable the GPU Operator's driver management and install the driver…

While it is possible to disable driver management, it is not a required action for air-gapped deployment. The GPU Operator can manage the driver in an air-gapped environment if the driver container image is mirrored to the private registry. Disabling driver management would require manual driver installation, which is more complex and not necessary. Therefore, this is not a required action for the scenario.

Trap 3: Ensure the cluster nodes have direct access to the NVIDIA licensing…

This scenario does not mention vGPU; it is about GPU Operator on an air-gapped cluster. Even if vGPU were used, the licensing server would need to be accessible or a licensing proxy set up, but that is not a requirement for the GPU Operator itself. The question focuses on deploying the GPU Operator, which does not require licensing server access. Thus, this is not a required action.

A
Install the NVIDIA GPU Operator using the --offline flag.
Why it fails: There is no --offline flag for the NVIDIA GPU Operator Helm chart or installation. The Operator does not have a built-in offline mode. Air-gapped deployments are handled by mirroring images and configuring the registry endpoints. Relying on a non-existent flag would cause installation failure. Therefore, this is not a valid action for the scenario.
B
Disable the GPU Operator's driver management and install the driver manually on each node.
Why it fails: While it is possible to disable driver management, it is not a required action for air-gapped deployment. The GPU Operator can manage the driver in an air-gapped environment if the driver container image is mirrored to the private registry. Disabling driver management would require manual driver installation, which is more complex and not necessary. Therefore, this is not a required action for the scenario.
C
Configure the GPU Operator to use the private registry by setting the appropriate Helm chart values.
The GPU Operator must be configured to pull images from the private registry instead of the default public registries. This is done by setting Helm values such as operator.repository, driver.repository, toolkit.repository, and others to point to the private registry. Without this configuration, the Operator would attempt to pull from nvcr.io and fail due to no internet access. Thus, this is a required action for air-gapped deployment.
D
Mirror all required NVIDIA container images to a private registry accessible by the cluster.
In an air-gapped environment, the GPU Operator cannot pull images from public registries like nvcr.io. All required images, including the driver container, container toolkit, device plugin, DCGM exporter, and GPU Operator itself, must be mirrored to a private registry. The Operator must be configured to use this registry via Helm values such as operator.repository and driver.repository. This ensures that the cluster can pull images without internet access.
E
Ensure the cluster nodes have direct access to the NVIDIA licensing server for vGPU.
Why it fails: This scenario does not mention vGPU; it is about GPU Operator on an air-gapped cluster. Even if vGPU were used, the licensing server would need to be accessible or a licensing proxy set up, but that is not a requirement for the GPU Operator itself. The question focuses on deploying the GPU Operator, which does not require licensing server access. Thus, this is not a required action.

Question 2mediummultiple choice

Refer to the exhibit. An AI engineer observes that a model training job is running slower than expected. Based on the output, what is the primary cause of the performance degradation?

Exhibit

nvidia-smi -q -d PERFORMANCE

Performance State : P12
Clocks Throttle Reasons : SW Power Cap

Trap 1: The GPU is overheating and the thermal management system has…

Thermal throttling would be indicated by 'HW Thermal Slowdown' in the throttle reasons field. The presence of 'SW Power Cap' specifically points to a configuration constraint set by the software, not a cooling system failure or excessive ambient temperatures affecting the GPU's operational state.

Trap 2: The GPU is currently idle and the driver has shifted the card to an…

While P12 is a low-power state, the explicit mention of 'SW Power Cap' in the throttle reasons confirms that the driver is actively restricting performance due to policy settings. This is distinct from a normal idle state transition, which would not typically trigger a throttle reason alert.

Trap 3: The GPU is experiencing a PCIe bus error, forcing the system to…

PCIe bus errors are typically logged in the kernel ring buffer or via specialized diagnostic tools, not as a power cap throttle. This message indicates a power management policy enforcement, not a hardware communication fault or an interrupt-driven slowdown originating from the system bus interface.

A
The GPU is overheating and the thermal management system has engaged to prevent hardware damage.
Why it fails: Thermal throttling would be indicated by 'HW Thermal Slowdown' in the throttle reasons field. The presence of 'SW Power Cap' specifically points to a configuration constraint set by the software, not a cooling system failure or excessive ambient temperatures affecting the GPU's operational state.
B
The GPU is currently idle and the driver has shifted the card to an energy-saving state.
Why it fails: While P12 is a low-power state, the explicit mention of 'SW Power Cap' in the throttle reasons confirms that the driver is actively restricting performance due to policy settings. This is distinct from a normal idle state transition, which would not typically trigger a throttle reason alert.
C
An administrator has set a software power limit that is lower than the GPU's maximum performance threshold.
The 'SW Power Cap' indicator explicitly confirms that an administrative policy or software command has capped the power usage of the GPU. This forces the GPU to maintain lower clock speeds, directly impacting the throughput of the training job by limiting the available computational resources.
D
The GPU is experiencing a PCIe bus error, forcing the system to reduce clock speeds to maintain stability.
Why it fails: PCIe bus errors are typically logged in the kernel ring buffer or via specialized diagnostic tools, not as a power cap throttle. This message indicates a power management policy enforcement, not a hardware communication fault or an interrupt-driven slowdown originating from the system bus interface.

Question 3mediummultiple choice

When managing GPU resources in a shared cluster, which configuration best prevents 'noisy neighbor' scenarios where one GPU task consumes all available memory bandwidth?

Trap 1: Increasing the CUDA_VISIBLE_DEVICES environment variable.

CUDA_VISIBLE_DEVICES only restricts which physical GPUs a process can 'see' and access. It does not impose limits on memory bandwidth or throughput consumption on the assigned GPU, meaning a process can still saturate the GPU's memory interface and degrade performance for other tasks sharing that specific hardware.

Trap 2: Setting a higher priority class for inference pods.

Priority classes govern pod scheduling and preemption, not the runtime resource consumption of a process. A high-priority pod can still be a noisy neighbor if it is poorly written or overly aggressive, as priority does not regulate the amount of memory bandwidth or cache it uses while running.

Trap 3: Configuring the NVIDIA DCGM Exporter alerts.

DCGM Exporter alerts are purely diagnostic and reactive; they notify administrators of existing issues. They do not proactively prevent noisy neighbor scenarios, as they lack the enforcement mechanisms necessary to throttle or isolate resource consumption during the execution of a process on the GPU hardware.

A
Increasing the CUDA_VISIBLE_DEVICES environment variable.
Why it fails: CUDA_VISIBLE_DEVICES only restricts which physical GPUs a process can 'see' and access. It does not impose limits on memory bandwidth or throughput consumption on the assigned GPU, meaning a process can still saturate the GPU's memory interface and degrade performance for other tasks sharing that specific hardware.
B
Implementing NVIDIA MIG and strictly defining resource limits.
MIG partitions the GPU at the hardware level, providing dedicated memory and compute paths. When combined with Kubernetes resource limits, it ensures that workloads are physically constrained to their slice, preventing a single process from monopolizing memory bandwidth or compute cycles, thus effectively eliminating the noisy neighbor problem.
C
Setting a higher priority class for inference pods.
Why it fails: Priority classes govern pod scheduling and preemption, not the runtime resource consumption of a process. A high-priority pod can still be a noisy neighbor if it is poorly written or overly aggressive, as priority does not regulate the amount of memory bandwidth or cache it uses while running.
D
Configuring the NVIDIA DCGM Exporter alerts.
Why it fails: DCGM Exporter alerts are purely diagnostic and reactive; they notify administrators of existing issues. They do not proactively prevent noisy neighbor scenarios, as they lack the enforcement mechanisms necessary to throttle or isolate resource consumption during the execution of a process on the GPU hardware.

Question 4mediummultiple choice

In a multi-node training scenario, what is the significance of the NVIDIA Collective Communications Library (NCCL) in workload management?

Trap 1: It handles the automatic scaling of pods in the cluster.

NCCL is a communication library for GPU-to-GPU data transfers; it does not manage cluster scaling or pod lifecycles. Kubernetes handles pod orchestration and auto-scaling, while NCCL operates within the application layer to synchronize data during parallel training tasks across the allocated GPU devices.

Trap 2: It replaces the need for high-speed network cabling.

NCCL relies heavily on the underlying network performance to function effectively. It does not mitigate the need for high-speed cabling like InfiniBand; rather, it takes full advantage of such high-performance hardware. Without a high-speed network, NCCL's efficiency gains would be significantly limited by the physical link speed.

Trap 3: It monitors the temperature of the GPUs during training.

NCCL is strictly for data communication and synchronization. Hardware monitoring, including temperature tracking, is managed by tools like DCGM or system-level drivers. Including thermal management in the communication library would be outside its scope and would provide no benefit to the data synchronization process itself.

A
It handles the automatic scaling of pods in the cluster.
Why it fails: NCCL is a communication library for GPU-to-GPU data transfers; it does not manage cluster scaling or pod lifecycles. Kubernetes handles pod orchestration and auto-scaling, while NCCL operates within the application layer to synchronize data during parallel training tasks across the allocated GPU devices.
B
It provides a mechanism to optimize inter-node data exchange.
NCCL is purpose-built to accelerate collective operations across distributed GPUs. It intelligently utilizes hardware interconnects such as NVLink and InfiniBand to reduce communication latency, which is the primary bottleneck in distributed training, ensuring that gradient synchronization does not throttle the overall training throughput of the job.
C
It replaces the need for high-speed network cabling.
Why it fails: NCCL relies heavily on the underlying network performance to function effectively. It does not mitigate the need for high-speed cabling like InfiniBand; rather, it takes full advantage of such high-performance hardware. Without a high-speed network, NCCL's efficiency gains would be significantly limited by the physical link speed.
D
It monitors the temperature of the GPUs during training.
Why it fails: NCCL is strictly for data communication and synchronization. Hardware monitoring, including temperature tracking, is managed by tools like DCGM or system-level drivers. Including thermal management in the communication library would be outside its scope and would provide no benefit to the data synchronization process itself.

Question 5hardmultiple choice

An AI operations team runs mixed training and inference workloads on a Kubernetes cluster managed with the NVIDIA GPU Operator. Inference pods frequently arrive in bursts and must start within seconds, while long-running training jobs occupy most MIG-capable A100 GPUs for days. Administrators want burst inference pods to obtain GPU capacity immediately without preempting or restarting the training jobs, and they want the cluster to reclaim those resources automatically when the burst ends. Which approach best satisfies these requirements?

Trap 1: Configure the inference Deployment with a topologySpreadConstraint…

Topology spread constraints only influence where pods are placed among nodes that already have allocatable GPU capacity; they do not create capacity. With training jobs occupying nearly all A100 GPUs, spreading inference replicas simply leaves them Pending, because no node has a free device. This adds scheduling complexity without addressing the real constraint, which is the absence of unallocated GPU hardware during bursts.

Trap 2: Enable the GPU Operator's time-slicing configuration so inference…

Time-slicing lets multiple containers share a GPU by interleaving execution, but it does not guarantee inference pods start within seconds, since they still compete for the same device cycles as training kernels. Sharing a device with a heavy training workload introduces latency jitter and can slow both workloads. It also does not provide the automatic reclamation behavior requested, since the shared GPUs remain allocated to the node's training pods.

Trap 3: Deploy inference pods with a higher PriorityClass and enable…

Priority-based preemption does free capacity quickly, but it precisely violates the stated requirement that training jobs must not be preempted or restarted. Evicted training pods lose their running state and must restart from a checkpoint, which for multi-day jobs can waste hours of GPU time. This solves the latency problem by sacrificing the long-running workloads the team explicitly wants protected, so it is the wrong design here.

A
Create a node pool reserved for inference, and configure a cluster autoscaler with scale-to-zero so burst pods trigger new GPU nodes that are removed when idle.
A dedicated inference node pool with scale-to-zero autoscaling lets burst pods trigger immediate node provisioning while training jobs keep running untouched on their own nodes. When the burst subsides, the autoscaler drains and removes the idle inference nodes, reclaiming GPU capacity and cost automatically. This directly matches both requirements: no preemption of training and automatic reclamation after the burst.
B
Configure the inference Deployment with a topologySpreadConstraint across all GPU nodes so its replicas distribute evenly and find free capacity.
Why it fails: Topology spread constraints only influence where pods are placed among nodes that already have allocatable GPU capacity; they do not create capacity. With training jobs occupying nearly all A100 GPUs, spreading inference replicas simply leaves them Pending, because no node has a free device. This adds scheduling complexity without addressing the real constraint, which is the absence of unallocated GPU hardware during bursts.
C
Enable the GPU Operator's time-slicing configuration so inference and training containers share the same physical GPUs through a shared device plugin.
Why it fails: Time-slicing lets multiple containers share a GPU by interleaving execution, but it does not guarantee inference pods start within seconds, since they still compete for the same device cycles as training kernels. Sharing a device with a heavy training workload introduces latency jitter and can slow both workloads. It also does not provide the automatic reclamation behavior requested, since the shared GPUs remain allocated to the node's training pods.
D
Deploy inference pods with a higher PriorityClass and enable preemption so the scheduler evicts lower-priority training pods when capacity is exhausted.
Why it fails: Priority-based preemption does free capacity quickly, but it precisely violates the stated requirement that training jobs must not be preempted or restarted. Evicted training pods lose their running state and must restart from a checkpoint, which for multi-day jobs can waste hours of GPU time. This solves the latency problem by sacrificing the long-running workloads the team explicitly wants protected, so it is the wrong design here.

Question 6mediummultiple choice

A machine learning engineer is optimizing a recommendation model for inference on an NVIDIA T4 GPU. The model uses dynamic input shapes, and profiling shows that kernel launch overhead is a significant contributor to latency. Which optimization technique should be applied to reduce this overhead?

Trap 1: Use mixed precision (FP16) to reduce the number of kernels executed.

Mixed precision reduces memory bandwidth and can speed up computations, but it does not reduce the number of kernel launches. Kernel launch overhead is related to the CPU-GPU interaction, not the precision. While FP16 may improve performance, it does not directly target the overhead described.

Trap 2: Enable NVIDIA MPS (Multi-Process Service) to allow concurrent…

MPS allows multiple processes to share the GPU and can improve utilization in multi-process scenarios, but it does not reduce launch overhead for a single process. The overhead is per-kernel launch from the CPU, and MPS does not change that. This option is irrelevant to the specific problem of launch overhead in a single model inference.

Trap 3: Increase the batch size to amortize kernel launch overhead across…

Increasing batch size can improve throughput but does not reduce the per-kernel launch overhead; it merely spreads it over more samples. For latency-sensitive inference with dynamic shapes, this may increase latency. The issue is the overhead itself, not the number of samples, so this does not address the root cause.

A
Convert the model to TensorRT with dynamic shapes and enable CUDA graphs during inference.
CUDA graphs capture a sequence of kernel launches and replay them with a single launch, drastically reducing launch overhead. TensorRT supports dynamic shapes and can build engines that use CUDA graphs. This is ideal for models with variable input shapes where launch overhead is high. The combination reduces CPU overhead and improves latency.
B
Use mixed precision (FP16) to reduce the number of kernels executed.
Why it fails: Mixed precision reduces memory bandwidth and can speed up computations, but it does not reduce the number of kernel launches. Kernel launch overhead is related to the CPU-GPU interaction, not the precision. While FP16 may improve performance, it does not directly target the overhead described.
C
Enable NVIDIA MPS (Multi-Process Service) to allow concurrent kernel execution.
Why it fails: MPS allows multiple processes to share the GPU and can improve utilization in multi-process scenarios, but it does not reduce launch overhead for a single process. The overhead is per-kernel launch from the CPU, and MPS does not change that. This option is irrelevant to the specific problem of launch overhead in a single model inference.
D
Increase the batch size to amortize kernel launch overhead across more samples.
Why it fails: Increasing batch size can improve throughput but does not reduce the per-kernel launch overhead; it merely spreads it over more samples. For latency-sensitive inference with dynamic shapes, this may increase latency. The issue is the overhead itself, not the number of samples, so this does not address the root cause.

Question 7mediummultiple choice

An AI operations team is deploying NVIDIA Base Command Manager to manage a cluster of DGX nodes. They want to ensure that only authorized users can submit jobs and that all job submissions are audited. Which combination of Base Command Manager features should the administrator configure to meet these requirements?

Trap 1: Deploy a separate Kubernetes cluster with RBAC policies and use…

Base Command Manager itself provides job scheduling and user management; offloading authorization to a separate Kubernetes cluster would not integrate with its native job submission flow. This approach adds complexity and does not directly satisfy the requirement to audit job submissions within Base Command Manager. It also ignores the built-in identity and audit capabilities.

Trap 2: Configure IPMI access controls on each DGX node and enable the Base…

IPMI controls out-of-band hardware management and does not govern user job submission or authentication. The telemetry collector gathers performance and health metrics, not audit logs of job submissions. Neither feature addresses the requirement to restrict job submission to authorized users or to audit those submissions.

Trap 3: Enable SELinux enforcing mode on all nodes and configure Base…

SELinux and SSH key authentication improve system security but do not provide centralized user authorization or audit logging for job submissions. They do not integrate with an identity provider to restrict which users can submit jobs, nor do they generate the required audit records. These measures are complementary but insufficient for the stated requirements.

A
Deploy a separate Kubernetes cluster with RBAC policies and use Base Command Manager only for node provisioning.
Why it fails: Base Command Manager itself provides job scheduling and user management; offloading authorization to a separate Kubernetes cluster would not integrate with its native job submission flow. This approach adds complexity and does not directly satisfy the requirement to audit job submissions within Base Command Manager. It also ignores the built-in identity and audit capabilities.
B
Configure IPMI access controls on each DGX node and enable the Base Command Manager telemetry collector.
Why it fails: IPMI controls out-of-band hardware management and does not govern user job submission or authentication. The telemetry collector gathers performance and health metrics, not audit logs of job submissions. Neither feature addresses the requirement to restrict job submission to authorized users or to audit those submissions.
C
Enable SELinux enforcing mode on all nodes and configure Base Command Manager to use SSH key-based authentication only.
Why it fails: SELinux and SSH key authentication improve system security but do not provide centralized user authorization or audit logging for job submissions. They do not integrate with an identity provider to restrict which users can submit jobs, nor do they generate the required audit records. These measures are complementary but insufficient for the stated requirements.
D
Integrate Base Command Manager with an LDAP or Active Directory identity provider and enable the audit logging feature.
Base Command Manager supports integration with enterprise identity providers such as LDAP or Active Directory for authentication and authorization. Enabling audit logging captures job submission events and user actions. Together, these features enforce authorized access and provide the required audit trail for job submissions in this scenario.

Question 8mediummultiple choice

An operations engineer is troubleshooting a distributed training job that uses NVIDIA Magnum IO GPUDirect Storage to read training data directly from a local NVMe SSD into GPU memory. The job reports lower than expected I/O bandwidth. `nvidia-smi` shows normal GPU utilization, and the NVMe drive's throughput is well below its peak. Which factor is most likely limiting GPUDirect Storage performance in this scenario?

Trap 1: The training job is using too many CPU threads for data…

Excessive CPU threads can cause contention, but GPUDirect Storage bypasses the CPU for data movement. The bottleneck is more likely in the data path between storage and GPU. CPU thread count would affect preprocessing, not the direct memory transfer, so this is not the primary limiter for the reported I/O bandwidth issue.

Trap 2: The GPU's ECC memory is enabled, reducing available bandwidth.

ECC on HBM memory has a negligible impact on I/O bandwidth and is not related to the storage-to-GPU data path. ECC affects memory error correction within the GPU, not the PCIe transfer rate. This is not a plausible cause for low GPUDirect Storage bandwidth in this scenario.

Trap 3: The NVMe drive is formatted with a file system that does not…

GPUDirect Storage works with common file systems like ext4 and XFS through the cuFile API, and also supports block devices. While some network file systems may lack support, a local NVMe SSD with a standard file system is typically compatible. The file system type is less likely to be the limiting factor compared to PCIe topology.

A
The training job is using too many CPU threads for data preprocessing.
Why it fails: Excessive CPU threads can cause contention, but GPUDirect Storage bypasses the CPU for data movement. The bottleneck is more likely in the data path between storage and GPU. CPU thread count would affect preprocessing, not the direct memory transfer, so this is not the primary limiter for the reported I/O bandwidth issue.
B
The GPU's ECC memory is enabled, reducing available bandwidth.
Why it fails: ECC on HBM memory has a negligible impact on I/O bandwidth and is not related to the storage-to-GPU data path. ECC affects memory error correction within the GPU, not the PCIe transfer rate. This is not a plausible cause for low GPUDirect Storage bandwidth in this scenario.
C
The GPU is not connected to the NVMe drive through a supported PCIe topology.
GPUDirect Storage requires a direct data path between the storage device and the GPU, typically over PCIe with peer-to-peer support. If the NVMe drive and GPU are behind different PCIe switches or root complexes without proper peer-to-peer capabilities, data must bounce through host memory, reducing bandwidth. This is a common limitation in multi-socket servers where the GPU and NVMe are on different NUMA nodes.
D
The NVMe drive is formatted with a file system that does not support GPUDirect Storage.
Why it fails: GPUDirect Storage works with common file systems like ext4 and XFS through the cuFile API, and also supports block devices. While some network file systems may lack support, a local NVMe SSD with a standard file system is typically compatible. The file system type is less likely to be the limiting factor compared to PCIe topology.

Question 9mediummultiple choice

When configuring the NVIDIA Device Plugin for Kubernetes, what is the purpose of the 'time-slicing' configuration?

Trap 1: To schedule tasks based on the specific time of day for load…

Time-slicing in the context of the NVIDIA Device Plugin has nothing to do with wall-clock scheduling. It refers strictly to the GPU's internal compute sharing mechanism, allowing multiple processes to interleave their work on the GPU hardware, rather than being a tool for job-level scheduling.

Trap 2: To increase the GPU's clock frequency during high-demand periods.

Time-slicing does not manipulate the physical clock frequency of the GPU hardware. It is a logical partitioning method designed to maximize throughput by sharing the GPU among multiple tasks, whereas clock frequency management is controlled by power management policies and thermal monitoring systems.

Trap 3: To restrict access to the GPU based on container process priority.

Time-slicing is not a Quality of Service (QoS) or priority enforcement mechanism. It is a scheduling capability designed to provide concurrent access for multiple containers. While it uses round-robin or similar scheduling, it does not prioritize one container over another based on external process settings.

A
To schedule tasks based on the specific time of day for load balancing.
Why it fails: Time-slicing in the context of the NVIDIA Device Plugin has nothing to do with wall-clock scheduling. It refers strictly to the GPU's internal compute sharing mechanism, allowing multiple processes to interleave their work on the GPU hardware, rather than being a tool for job-level scheduling.
B
To increase the GPU's clock frequency during high-demand periods.
Why it fails: Time-slicing does not manipulate the physical clock frequency of the GPU hardware. It is a logical partitioning method designed to maximize throughput by sharing the GPU among multiple tasks, whereas clock frequency management is controlled by power management policies and thermal monitoring systems.
C
To allow multiple pods to share a single physical GPU through context switching.
Time-slicing enables a physical GPU to be oversubscribed by allowing multiple pods to execute in turns on the same hardware. This increases the utilization of the GPU in scenarios where the individual pods do not require the full dedicated throughput of the entire device at all times.
D
To restrict access to the GPU based on container process priority.
Why it fails: Time-slicing is not a Quality of Service (QoS) or priority enforcement mechanism. It is a scheduling capability designed to provide concurrent access for multiple containers. While it uses round-robin or similar scheduling, it does not prioritize one container over another based on external process settings.