Weekly articles on AI networking, NVIDIA InfiniBand and GPU fabric.
GPUDirect RDMA localityTHE NETWORK PATH STARTS INSIDE THE SERVERX4 NETWORKSGPUDirect and PCIe localityNUMA SOCKET 0PCIe ROOT COMPLEX 0GPU 0NVLinkGPU 1NVLinkNIC 0400GLOCAL PATHPHB NUMA SOCKET 1PCIe ROOT COMPLEX 1GPU 2NVLinkGPU 3NVLinkNIC…
Congestion diagnosisXMItWAIT IS A SYMPTOM · FIND THE DOWNSTREAM CONSTRAINTX4 NETWORKSCongestion hotspot and XmitWaitLEAF–SPINE HOTSPOT MAP Spine 1Spine 2Leaf 1Leaf 2Leaf 3Leaf 4 ! GPU AGPU BGPU CGPU D Constraint:…
UFM fabric operationsTOPOLOGY · TELEMETRY · HEALTH · EVENTSX4 NETWORKSUFM dashboardUFMOverviewTopologyPortsTelemetryEventsHealth FABRIC TOPOLOGY Spine 1Spine 2 Leaf 1Leaf 2Leaf 3L4 1 congested link · 0 down SYSTEM HEALTHHealthy16 nodes ·…
Rail-optimized topologyGPU LOCALITY CHANGES THE DIAGRAMX4 NETWORKSRail locality mapGPU NODEGPU 0–1NIC 0 · Rail 1GPU 2–3NIC 1 · Rail 2GPU 4–5NIC 2 · Rail 3GPU 6–7NIC 3 · Rail 4…
AI fabric architectureWORKLOAD → SYSTEM PATH → FABRIC → ACCEPTANCEX4 NETWORKSArchitecture chain from workload to validated rail fabricWorkload profileCollectives · ranks · messages GPU + NIC topologyNUMA · PCIe ·…
Most network engineers walk into an InfiniBand fabric with twenty years of Ethernet instincts, and those instincts will actively work against them. I've spent my career in Cisco and Arista…