NCP-AIO Troubleshooting and Optimization — All Questions With Answers

Error: NCCL_ERROR_NET_IB_NOT_FOUND
Stack trace: NCCL call failed during broadcast
System Info: IB fabric configured, but no HCA found on PCIe bus.

Error: CUDA error: out of memory. Total allocated memory: 31.8 GB. Reserved memory: 32.0 GB. Current batch size: 128.

JSON Policy Configuration:
{
  "persistence": "enabled",
  "ecc_mode": "enabled",
  "compute_mode": "default",
  "power_limit_watts": 250
}

nvidia-smi -q -d PERFORMANCE
Performance State : P0
Clocks Throttle Reasons : Active
  Applications Clocks Setting : None
  SW Power Cap : Active
  HW Slowdown : Active
  HW Thermal Slowdown : Active

{
  "policy": "restrict_gpu_access",
  "targets": ["user_a"],
  "max_concurrent_jobs": 2,
  "resource_quota": {
    "gpu_memory": "16GB"
  }
}

nvidia-smi -q -d PERFORMANCE
Performance State : P0
Clocks Throttle Reasons : Active
Clocks Throttle Reason Sw Power Cap : Active

Error Log: [NCCL WARN] NET/Socket : Connection refused. [NCCL WARN] Call to connect() failed. Rank 0: GPU 0: Peer 1 is unreachable via NCCL_SOCKET_IFNAME.

Error: CUDA_ERROR_OUT_OF_MEMORY
GPU Memory Usage: 31.8GB / 32.0GB
Batch Size: 128
Precision: FP32
Model Architecture: Transformer-based