NCCL Troubleshooting: Following a Slow Training Job End to End
NCCL troubleshooting pathMEASURE EVERY BOUNDARY · CHANGE ONE VARIABLEX4 NETWORKSNCCL troubleshootingONE ITERATION · 42.6 ms GPU kernel3.2 msNCCL collective7.8 msPCIe / NIC1.4 msFabric path28.7 msRemote NIC1.3 msGPU0.2 ms TIMING WATERFALLFabric…