Weekly articles on AI networking, NVIDIA InfiniBand and GPU fabric.
AI network operations runbookPRE-CHECK · BASELINE · MONITOR · RESPOND · HANDOFFX4 NETWORKSAI network runbookDAILY OPERATING BOARDSTATUS: HEALTHY · 09:15 PRE-CHECKfabric healthinventory driftcompleteclearBASELINErail symmetryknown-good snapshotcompleteclearMONITORevents + telemetryjob correlationcompleteclearRESPONDowner + scopeevidence…
Hybrid AI network pathSEPARATE DATA, CHECKPOINT, MANAGEMENT, AND INFERENCE FLOWSX4 NETWORKSHybrid AI networkON-PREM AI FABRICGPU clustertrainingHigh-speed storagedata + checkpointsManagementscheduler + telemetryEdge router CLOUDObject storagemodel artifactsBurst computetrainingInference endpointsservingCloud edge PRIVATE INTERCONNECTPrimary…
Capacity beyond port countsSCALABLE UNITS · HEADROOM · EXPANSION TRIGGERSX4 NETWORKSCapacity planningREPEATABLE SCALE UNITLeaf pairrail block8 GPU racks64 GPUsLeaf pairrail block SPINE PORT BUDGETUsed 184 · Reserved 88 · Next…
Failure domains and recoveryDESIGN THE DEGRADED STATE · TEST THE RETURN TO BASELINEX4 NETWORKSResilient fabricSpine 1FAILEDSpine 2HEALTHY Leaf pair ALeaf pair BLeaf pair CLeaf pair D GPU racks 1–2GPU racks…
Shared AI security architectureCONTROL THE BOUNDARIES · KEEP THE RDMA DATA PATH DIRECTX4 NETWORKSSecurity zones for shared AI infrastructureUsers / APIauth · rateManagementIAM · policySchedulerjob identityStorage / dataencryptionGPU fabricRDMA zones…
GPU-time observabilityCORRELATE FABRIC SIGNALS WITH THE JOB TIMELINEX4 NETWORKSCorrelated observability timeline12:44:0012:47:00 GPU utilizationNCCL phaseXmitWaitLink errorsIteration tail AllReducefabric waitcompute Congestion event → GPU idle → tail latencyGPU-TIME SLO: ON TARGETX4 Networks…
Fabric bring-up planPROVE EACH LAYER BEFORE THE FIRST TRAINING RUNX4 NETWORKSFabric bring-up sequence1Inventoryports · optics2Link statespeed · FEC3RDMApeer matrix4Rail symmetry±3% target5Collectivescale test6Failuredegraded run7Burn-inzero critical8Applicationiteration SLO PASS: rail bandwidth symmetric ·…
InfiniBand or Ethernet?DECIDE FROM WORKLOAD + OPERATING MODELX4 NETWORKSDecision dimensionInfiniBandEthernet / RoCEWorkload fitLargest tightly coupled training clustersAI plus cloud-native and IP-integrated workloadsCongestion controlCredit-based lossless fabricPFC, ECN/CNP, adaptive routingOperationsUFM, subnet management,…
Spectrum-X / RoCE control loopPFC HEADROOM · ECN/CNP · ADAPTIVE ROUTINGX4 NETWORKSRoCE control loopPFC HEADROOMPause state: clearPer-priority buffers healthyWatch sustained pause, not any pause ECN → CNP RESPONSEMark early ·…
NCCL troubleshooting pathMEASURE EVERY BOUNDARY · CHANGE ONE VARIABLEX4 NETWORKSNCCL troubleshootingONE ITERATION · 42.6 ms GPU kernel3.2 msNCCL collective7.8 msPCIe / NIC1.4 msFabric path28.7 msRemote NIC1.3 msGPU0.2 ms TIMING WATERFALLFabric…