NCP-AIO Administration Practice Test 5 — 15 Questions

Question 1

An administrator is preparing a cluster for a new large language model training job that will use NVIDIA Magnum IO GPUDirect Storage to stream training data directly from a parallel file system to GPU memory. The administrator must verify that the environment supports GPUDirect Storage before the job starts. (Choose two.)

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Ensure that the parallel file system client supports the cuFile API and that the file system is mounted with options that allow direct I/O and peer-to-peer memory access.. GPUDirect Storage requires compatible drivers, the nvidia-fs kernel module, and a file system client that supports the cuFile API with suitable mount options. These two checks confirm that the software stack and storage client can perform direct memory transfers. MIG, exclusive RoCE, and disabling IOMMU are not prerequisites and may be counterproductive.

Answer

Disable IOMMU in the BIOS on all GPU nodes to allow direct memory access between storage devices and GPUs.

Answer

Configure the cluster to use RDMA over Converged Ethernet (RoCE) exclusively for all storage traffic, because GPUDirect Storage requires RDMA.

Answer

Enable NVIDIA Multi-Instance GPU (MIG) on every GPU to provide isolated memory partitions for storage transfers.

Question 2

An administrator is responsible for monitoring a large-scale AI cluster with hundreds of NVIDIA GPUs. They need to collect telemetry data such as GPU utilization, temperature, and power consumption from all nodes and store it centrally for analysis and alerting. Which NVIDIA tool should they use to collect and export GPU metrics to a monitoring system like Prometheus?

Accepted Answer

NVIDIA DCGM Exporter. For centralized monitoring of GPU metrics in a large cluster, NVIDIA DCGM Exporter is the appropriate tool. It leverages NVIDIA Data Center GPU Manager (DCGM) to collect a comprehensive set of metrics from each GPU and exposes them via an HTTP endpoint that Prometheus can scrape. This allows administrators to aggregate metrics across all nodes, create dashboards in Grafana, and set up alerts based on thresholds.

Answer

NVIDIA Nsight Systems

Answer

nvidia-smi

Answer

NVIDIA Base Command Manager

Question 3

An AI operations team is using NVIDIA DCGM (Data Center GPU Manager) to monitor a cluster of A100 GPUs. They want to set up proactive health checks to detect and mitigate GPU issues before they cause job failures. Which two DCGM features should they configure? (Choose two.)

Accepted Answer

DCGM policy management for automated remediation. DCGM health checks with periodic diagnostics and DCGM policy management for automated remediation are the two features that directly enable proactive health monitoring and mitigation. Health checks detect issues, and policies define actions to take when issues are found. Together, they form a proactive health management system. Other options are either monitoring metrics or management features that do not directly address proactive health.

Answer

DCGM profiling metrics for real-time utilization

Answer

DCGM group configuration for multi-node synchronization

Answer

DCGM API integration with Prometheus for alerting

Question 4

An administrator is troubleshooting a multi-node NVIDIA GPU training job that intermittently hangs during the all-reduce phase. The job uses NCCL over InfiniBand. Logs show that some ranks time out while others complete. The administrator suspects a network fabric issue. Which action should the administrator take first to isolate whether the problem is in the InfiniBand fabric or in the NCCL configuration?

Accepted Answer

Run the NCCL tests (nccl-tests) with the same topology and environment variables, and enable NCCL debug logging to capture detailed transport and topology information.. Using nccl-tests with debug logging is a controlled way to reproduce the collective communication and observe transport selection, topology detection, and timeouts. It helps determine whether the issue is in NCCL configuration or the InfiniBand fabric. Replacing hardware, forcing TCP, or increasing timeouts are premature and do not isolate the root cause.

Answer

Disable InfiniBand and force NCCL to use TCP sockets instead, because TCP is more reliable for collective operations.

Answer

Immediately replace all InfiniBand cables and transceivers on the affected nodes, because intermittent hangs during all-reduce almost always indicate physical link errors.

Answer

Increase the NCCL timeout value in the training script and restart the job, because the hangs are likely due to slow network convergence.

Question 5

An administrator is responsible for a large NVIDIA DGX SuperPOD used for multi-node training. They need to ensure that GPU telemetry and health metrics are collected centrally and can trigger alerts when GPUs exceed temperature thresholds. Which component of NVIDIA Base Command Manager (BCM) should they configure to achieve this?

Accepted Answer

NVIDIA Data Center GPU Manager (DCGM) integrated with BCM. NVIDIA DCGM is the standard tool for centralized GPU telemetry and health monitoring in data centers. When integrated with Base Command Manager, it collects metrics from all nodes and can be configured with alerting rules for temperature and other thresholds. This provides the required centralized visibility and automated alerts for the DGX SuperPOD.

Answer

NVIDIA Base Command Manager's built-in 'cluster monitoring' dashboard

Answer

NVIDIA Management Library (NVML) on each node

Answer

NVIDIA Container Toolkit on each compute node

Question 6

An AI operations team is deploying NVIDIA Base Command Manager to manage a cluster of DGX nodes. They want to ensure that only authorized users can submit jobs and that all job submissions are audited. Which combination of Base Command Manager features should the administrator configure to meet these requirements?

Accepted Answer

Integrate Base Command Manager with an LDAP or Active Directory identity provider and enable the audit logging feature.. Base Command Manager centralizes cluster administration, including user authentication and job scheduling. Integrating with LDAP or Active Directory ensures only authorized users can authenticate and submit jobs, while audit logging records those submissions for compliance. This combination directly addresses both the access control and auditing requirements in the scenario.

Answer

Deploy a separate Kubernetes cluster with RBAC policies and use Base Command Manager only for node provisioning.

Answer

Configure IPMI access controls on each DGX node and enable the Base Command Manager telemetry collector.

Answer

Enable SELinux enforcing mode on all nodes and configure Base Command Manager to use SSH key-based authentication only.

Question 7

A company runs multiple AI workloads on a shared Kubernetes cluster with NVIDIA GPUs. The administrator needs to enforce that only pods with a specific label can consume GPU resources, while other pods are denied. Which Kubernetes admission control mechanism should be used to implement this policy?

Accepted Answer

ValidatingAdmissionWebhook. A ValidatingAdmissionWebhook allows custom admission logic to be applied to pod creation. By configuring a webhook that checks for a specific label before permitting GPU resource requests, the administrator can enforce label-based access control. This is the correct mechanism because it can inspect pod metadata and resource requests and reject non-compliant pods.

Answer

PodSecurityPolicy

Answer

ResourceQuota

Answer

LimitRange

Question 8

A site reliability engineer is troubleshooting a DGX A100 node that intermittently drops out of the cluster during large NCCL all-reduce jobs. `nvidia-smi` shows all eight A100 GPUs healthy, but DCGM reports XID errors 74 and 79 on one GPU during the failures. The engineer needs to determine the most likely cause and the correct administrative action. Which combination best describes the cause and the appropriate first step?

Accepted Answer

The errors point to a GPU falling off the bus or an NVLink error; run `nvidia-smi -q` and DCGM diagnostics on the suspect GPU, then consider reseating or replacing the GPU board.. XID 74 and 79 are driver-reported errors indicating a GPU has fallen off the bus or an NVLink error has occurred. These are hardware and link-level faults, so the right first step is to gather detailed GPU and NVLink diagnostics with `nvidia-smi -q` and DCGM, then remediate the hardware. Ignoring or masking these errors risks job failures and data corruption, while treating them as thermal or software issues misses the actual fault domain.

Answer

The errors are benign and expected during large all-reduce jobs; suppress them by setting the DCGM health check to ignore XID 74 and 79.

Answer

The errors indicate a thermal shutdown; immediately lower the GPU clock with `nvidia-smi -lgc` and rerun the job.

Answer

The errors are caused by an outdated NCCL version; upgrade NCCL to the latest release and rerun the job without further hardware checks.

Question 9

An administrator is responsible for maintaining a fleet of NVIDIA-certified servers running AI workloads. They need to quickly identify which servers have GPUs that are overheating and may throttle performance. Which NVIDIA tool should the administrator use to monitor GPU temperature across the fleet in real time?

Accepted Answer

NVIDIA Data Center GPU Manager (DCGM). DCGM is the NVIDIA tool purpose-built for data center GPU monitoring. It exposes temperature, power, utilization, and health metrics through APIs and integrations with popular monitoring platforms. For an administrator needing real-time temperature visibility across many servers, DCGM is the correct choice.

Answer

NVIDIA Nsight Systems

Answer

NVIDIA TensorRT

Answer

NVIDIA CUDA Toolkit

Question 10

An administrator is managing an NVIDIA DGX SuperPOD used for large-scale AI training. The cluster uses a Slurm workload manager. The administrator needs to ensure that jobs are scheduled only on nodes with healthy GPUs and that failed GPUs are automatically drained from the pool. Which integration should be configured to achieve this?

Accepted Answer

NVIDIA Data Center GPU Manager (DCGM) with the Slurm health check plugin. DCGM with the Slurm health check plugin is the correct integration because it continuously monitors GPU health and can automatically drain nodes with failed GPUs from Slurm, ensuring jobs run only on healthy hardware. Other options lack the necessary health monitoring and automatic remediation capabilities.

Answer

NVIDIA Base Command Manager with Slurm integration

Answer

NVIDIA Fleet Command with Slurm edge scheduling

Answer

NVIDIA Container Toolkit with Slurm's --gres flag

Question 11

An administrator is using NVIDIA Base Command Manager to provision a new GPU cluster. They need to ensure that the compute nodes are configured with the correct GPU driver and CUDA toolkit versions. Which Base Command Manager feature should they use to automate this?

Accepted Answer

Node groups with software images. Base Command Manager uses node groups to categorize nodes with similar roles and software requirements. By assigning a software image to a node group, administrators define the exact GPU driver and CUDA toolkit versions. During provisioning, Base Command Manager applies the image, ensuring consistency across compute nodes. This is the intended feature for automating software configuration in a Base Command Manager cluster.

Answer

Ansible playbooks executed manually

Answer

PXE boot with custom kickstart scripts

Answer

Cron-based package updates

Question 12

An administrator is responsible for an NVIDIA AI Enterprise deployment on Kubernetes. The security team requires that all GPU-accelerated pods run with the least privilege necessary and that GPU device nodes are not exposed to pods that do not request them. Which combination of configurations should the administrator implement to meet these requirements?

Accepted Answer

Deploy the NVIDIA GPU Operator with the device plugin and configure a PodSecurityPolicy or OPA Gatekeeper policy that requires pods to request nvidia.com/gpu and forbids privileged mode.. Using the GPU Operator with the device plugin, combined with a policy that mandates explicit nvidia.com/gpu requests and prohibits privileged containers, ensures that GPU device nodes are only injected into pods that request them and that pods run with minimal privileges. This satisfies both the least privilege and isolation requirements.

Answer

Use the NVIDIA GPU Operator with the device plugin and configure pod security contexts to drop all capabilities and add only the NVIDIA_VISIBLE_DEVICES environment variable.

Answer

Install the NVIDIA k8s-device-plugin standalone and set the --pass-device-specs flag to true, then allow all pods to run as root.

Answer

Enable the NVIDIA device plugin with the --fail-on-init-error=false flag and set privileged: true in the pod security context.

Question 13

An AI operations team needs to monitor GPU health and utilization across a fleet of DGX nodes from a single dashboard. They want per-GPU metrics such as power, temperature, utilization, and ECC errors, and they want to retain historical data for capacity planning. Which NVIDIA tool is purpose-built to collect and expose these GPU telemetry metrics for centralized monitoring?

Accepted Answer

NVIDIA DCGM (Data Center GPU Manager) with the DCGM exporter for Prometheus.. DCGM is NVIDIA's purpose-built data center GPU monitoring and management tool. It gathers health and performance metrics including power, temperature, utilization, and ECC errors, and the DCGM exporter makes them available to Prometheus for centralized dashboards and historical retention. This aligns exactly with the team's need for fleet-wide GPU telemetry and capacity planning data.

Answer

NVIDIA Base Command Manager's job scheduler logs.

Answer

NVIDIA Nsight Systems for profiling GPU kernels and collecting timeline traces.

Answer

NVIDIA CUDA Toolkit's `nvidia-smi` command run manually on each node.

Question 14

An administrator is configuring NVIDIA GPUDirect Storage (GDS) on a cluster to accelerate data loading for AI training jobs. The cluster uses Mellanox InfiniBand adapters and NVMe storage. Which two actions are required to enable GDS and ensure optimal performance? (Choose two.)

Accepted Answer

Ensure that the storage devices and network adapters support RDMA and are properly configured for peer-to-peer communication.. Enabling GPUDirect Storage requires installing the GDS kernel module and user-space libraries on GPU nodes, and ensuring that storage and network hardware support RDMA and peer-to-peer communication. These two actions create the necessary software and hardware foundation for direct data transfers between storage and GPU memory. Other options are either incorrect or not specific to GDS.

Answer

Set the environment variable NVIDIA_GDS_ENABLE=1 on all nodes.

Answer

Configure the GPU nodes to use the NVIDIA Container Toolkit for all training containers.

Answer

Disable IOMMU in the BIOS to allow direct memory access between devices.

Question 15

An administrator is managing a Kubernetes cluster with NVIDIA GPU Operator. A new policy requires that all GPU workloads run with specific environment variables set, such as NVIDIA_VISIBLE_DEVICES and NVIDIA_DRIVER_CAPABILITIES. The administrator wants to enforce these variables automatically for any pod that requests a GPU, without modifying each pod specification manually. Which approach should the administrator use?

Accepted Answer

Create a MutatingAdmissionWebhook that injects the required environment variables into pods that request nvidia.com/gpu resources.. A MutatingAdmissionWebhook is the correct Kubernetes-native way to automatically inject environment variables into pods based on their resource requests. It can inspect incoming pod specs and add the required variables when a GPU resource is requested, ensuring consistent configuration without manual intervention. Validating webhooks only reject, and runtime configuration is not pod-specific.

Answer

Use a ValidatingAdmissionWebhook to reject pods that do not include the required environment variables.

Answer

Configure the NVIDIA GPU Operator to set default environment variables in the device plugin daemonset.

Answer

Set the environment variables in the NVIDIA Container Runtime configuration on each node.