NCP-AIO Workload Management Practice Question

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

• ✗

  Set `NCCL_P2P_DISABLE=1` in the job's environment to force peer-to-peer transfers over PCIe.

  Why it's wrong here

  Disabling peer-to-peer access removes an optimization for intra-node GPU communication; it does nothing to repair the inter-node fabric path that is failing here. Since the pods are spread one per node, the relevant path is network, not PCIe, so this variable addresses a different layer and would not restore RDMA transport.

• ✓

  Deploy the NVIDIA Network Operator to install and configure the RDMA stack and GPUDirect RDMA on the GPU nodes.

  Why this is correct

  NCCL falling back to socket transport indicates the RDMA path is unavailable, typically because the high-speed fabric drivers, RDMA devices, and GPUDirect RDMA support are not provisioned on the nodes. The Network Operator deploys and configures those components alongside the GPU Operator, restoring the RDMA transport that NCCL prefers for multi-node collectives.

• ✗

  Add a node affinity rule forcing all eight pods onto a single node to enable NVLink communication.

  Why it's wrong here

  A single A100 per node cannot host eight GPU-requesting pods, so this affinity rule would leave most pods Pending and the job would never start. Even on an eight-GPU node, consolidating ranks changes the topology but does not provision the missing RDMA stack that the inter-node collectives require.

• ✗

  Increase the number of replicas in the Job so more GPUs participate in the collective.

  Why it's wrong here

  Adding replicas does not fix a transport fallback; it would simply add more participants communicating over the already-degraded socket path. The bottleneck is the interconnect path NCCL is using, not the number of ranks, so scaling out would likely worsen the symptom while consuming additional GPUs.