Weekly articles on AI networking, NVIDIA InfiniBand and GPU fabric.
InfiniBand or Ethernet?DECIDE FROM WORKLOAD + OPERATING MODELX4 NETWORKSDecision dimensionInfiniBandEthernet / RoCEWorkload fitLargest tightly coupled training clustersAI plus cloud-native and IP-integrated workloadsCongestion controlCredit-based lossless fabricPFC, ECN/CNP, adaptive routingOperationsUFM, subnet management,…
Spectrum-X / RoCE control loopPFC HEADROOM · ECN/CNP · ADAPTIVE ROUTINGX4 NETWORKSRoCE control loopPFC HEADROOMPause state: clearPer-priority buffers healthyWatch sustained pause, not any pause ECN → CNP RESPONSEMark early ·…
NCCL troubleshooting pathMEASURE EVERY BOUNDARY · CHANGE ONE VARIABLEX4 NETWORKSNCCL troubleshootingONE ITERATION · 42.6 ms GPU kernel3.2 msNCCL collective7.8 msPCIe / NIC1.4 msFabric path28.7 msRemote NIC1.3 msGPU0.2 ms TIMING WATERFALLFabric…
GPUDirect RDMA localityTHE NETWORK PATH STARTS INSIDE THE SERVERX4 NETWORKSGPUDirect and PCIe localityNUMA SOCKET 0PCIe ROOT COMPLEX 0GPU 0NVLinkGPU 1NVLinkNIC 0400GLOCAL PATHPHB NUMA SOCKET 1PCIe ROOT COMPLEX 1GPU 2NVLinkGPU 3NVLinkNIC…
Congestion diagnosisXMItWAIT IS A SYMPTOM · FIND THE DOWNSTREAM CONSTRAINTX4 NETWORKSCongestion hotspot and XmitWaitLEAF–SPINE HOTSPOT MAP Spine 1Spine 2Leaf 1Leaf 2Leaf 3Leaf 4 ! GPU AGPU BGPU CGPU D Constraint:…
UFM fabric operationsTOPOLOGY · TELEMETRY · HEALTH · EVENTSX4 NETWORKSUFM dashboardUFMOverviewTopologyPortsTelemetryEventsHealth FABRIC TOPOLOGY Spine 1Spine 2 Leaf 1Leaf 2Leaf 3L4 1 congested link · 0 down SYSTEM HEALTHHealthy16 nodes ·…
Rail-optimized topologyGPU LOCALITY CHANGES THE DIAGRAMX4 NETWORKSRail locality mapGPU NODEGPU 0–1NIC 0 · Rail 1GPU 2–3NIC 1 · Rail 2GPU 4–5NIC 2 · Rail 3GPU 6–7NIC 3 · Rail 4…
AI fabric architectureWORKLOAD → SYSTEM PATH → FABRIC → ACCEPTANCEX4 NETWORKSArchitecture chain from workload to validated rail fabricWorkload profileCollectives · ranks · messages GPU + NIC topologyNUMA · PCIe ·…
Most network engineers walk into an InfiniBand fabric with twenty years of Ethernet instincts, and those instincts will actively work against them. I've spent my career in Cisco and Arista…