The Fabric Is Clean. Prove the GPU-to-HCA Path.
Why a healthy InfiniBand fabric can still deliver poor NCCL performance, and how to isolate the PCIe and GPUDirect boundary without guessing. A clean InfiniBand fabric is not the same…
You Didn’t Fix the Fabric. You Just Made NCCL Wait Longer.
Why bigger InfiniBand timeout and retry values can turn a fast, diagnosable failure into a long, expensive silence. An RDMA timeout has a way of making engineers reach for a…
The Fabric Is Healthy. One Rail Is Not.
Why multi-rail AI clusters can hide one asymmetric path—and how to prove it without tuning by instinct. Multi-rail InfiniBand creates an uncomfortable troubleshooting problem: the cluster can have plenty of…
The AI Network Operations Runbook: What Mature Teams Do Every Day
AI network operations runbookPRE-CHECK · BASELINE · MONITOR · RESPOND · HANDOFFX4 NETWORKSAI network runbookDAILY OPERATING BOARDSTATUS: HEALTHY · 09:15 PRE-CHECKfabric healthinventory driftcompleteclearBASELINErail symmetryknown-good snapshotcompleteclearMONITORevents + telemetryjob correlationcompleteclearRESPONDowner + scopeevidence…
Hybrid AI Networking: Designing the Path Between Data Center and Cloud
Hybrid AI network pathSEPARATE DATA, CHECKPOINT, MANAGEMENT, AND INFERENCE FLOWSX4 NETWORKSHybrid AI networkON-PREM AI FABRICGPU clustertrainingHigh-speed storagedata + checkpointsManagementscheduler + telemetryEdge router CLOUDObject storagemodel artifactsBurst computetrainingInference endpointsservingCloud edge PRIVATE INTERCONNECTPrimary…
Scaling the AI Network: Capacity Planning Beyond Port Counts
Capacity beyond port countsSCALABLE UNITS · HEADROOM · EXPANSION TRIGGERSX4 NETWORKSCapacity planningREPEATABLE SCALE UNITLeaf pairrail block8 GPU racks64 GPUsLeaf pairrail block SPINE PORT BUDGETUsed 184 · Reserved 88 · Next…
Resilient AI Fabrics: Change Windows, Failure Domains, and Recovery
Failure domains and recoveryDESIGN THE DEGRADED STATE · TEST THE RETURN TO BASELINEX4 NETWORKSResilient fabricSpine 1FAILEDSpine 2HEALTHY Leaf pair ALeaf pair BLeaf pair CLeaf pair D GPU racks 1–2GPU racks…
Securing Shared AI Infrastructure Without Breaking Performance
Shared AI security architectureCONTROL THE BOUNDARIES · KEEP THE RDMA DATA PATH DIRECTX4 NETWORKSSecurity zones for shared AI infrastructureUsers / APIauth · rateManagementIAM · policySchedulerjob identityStorage / dataencryptionGPU fabricRDMA zones…
Observability for AI Networks: From Counters to GPU-Time SLOs
GPU-time observabilityCORRELATE FABRIC SIGNALS WITH THE JOB TIMELINEX4 NETWORKSCorrelated observability timeline12:44:0012:47:00 GPU utilizationNCCL phaseXmitWaitLink errorsIteration tail AllReducefabric waitcompute Congestion event → GPU idle → tail latencyGPU-TIME SLO: ON TARGETX4 Networks…