Most network engineers walk into an InfiniBand fabric with twenty years of Ethernet instincts, and those instincts will actively work against them.
I’ve spent my career in Cisco and Arista environments — BGP, SD-WAN, EVPN/VXLAN, the full stack of routing and switching problems that define enterprise networking. That background matters here, but only if you know which parts of it to keep and which parts to throw out. Applying Ethernet logic to an NVIDIA InfiniBand fabric supporting an AI cluster isn’t just imprecise. It’s the fastest way to misdiagnose the problem in front of you.
Here’s the principle I lead with every time: in an AI cluster, the network isn’t infrastructure sitting next to the compute. It’s part of the compute.
A Green Dashboard Is Not a Health Report
In a traditional data center, I think in terms of endpoints talking to applications — routing, packet loss, interface utilization, QoS, whether traffic is taking the path I expect. Those are the right questions for that environment.
They’re the wrong starting questions for an InfiniBand fabric.
When GPUs are exchanging gradients across a collective operation like AllReduce, a network that is technically up and passing traffic can still be quietly costing you enormous compute time. The GPUs don’t care that your five-minute utilization average looked fine. They care whether the data they needed arrived on schedule. If it didn’t, they sat idle — and idle GPU-hours on an AI cluster are not a rounding error.
A successful ping tells you nothing. An active port tells you nothing. The only question that matters is whether the fabric is delivering data at the rate and consistency the workload demands, and that is a much higher bar than “up.”
XmitWait Is a Symptom, Not a Diagnosis
If you’re managing an InfiniBand fabric and you’re not watching XmitWait correlated against workload timing, you’re flying blind.
An increasing XmitWait counter tells you a port had traffic ready to move and couldn’t, because it didn’t have the credits. That’s real signal. What it is not is a pointer to the broken component. The actual constraint is frequently sitting downstream — multiple GPU nodes converging on the same destination, a degraded link that pushed traffic onto fewer remaining paths, or a spine layer that isn’t distributing load the way the design assumes it should.
Replace the cable on the port showing high XmitWait and you will, more often than not, replace nothing that mattered. You have to follow the traffic to where the constraint actually originates — the same discipline that separates a real troubleshooter from someone chasing symptoms on Ethernet output drops.
Physical failure is a different animal, and I treat it differently. Rising symbol errors, link-recovery events, or bit-error-rate warnings during an active workload point straight at the physical path. I don’t care how large the historical counter is — I care whether it’s moving right now, under load. A port that accumulated errors during a cable install five months ago and has been silent since is not your problem today. A port that adds errors every time the cluster runs a job is.
Why the Spine Layer Is Bigger Than You’d Expect
Enterprise design habits will tell you that a large spine layer is over-engineered. Enterprise traffic patterns tolerate oversubscription because not every host transmits at line rate simultaneously.
AI workloads break that assumption completely. Hundreds of GPUs can need to move substantial data volumes at nearly the same instant. That east-west intensity will expose a design that would look perfectly reasonable in a conventional data center. The spine capacity in an AI fabric isn’t there primarily for redundancy — it exists to guarantee the paths and bandwidth required to keep the network from becoming the bottleneck in a system where the network’s job is performance, not just connectivity.
Stop Drawing the Boundary at the Switch Port
This is where most troubleshooting efforts fail, and it’s the single biggest mindset shift required to run AI infrastructure well: the network doesn’t end at the switch.
The real data path runs from GPU memory, through the server’s PCIe topology, across the HCA, through the fabric, and into the destination system. A failure anywhere on that path presents to the application team as “the network is slow” — even when every InfiniBand link is clean.
I use low-level testing to draw that line precisely. If ib_write_bw is degraded between two nodes, there is no reason to start tuning NCCL — the underlying RDMA path already has a problem, and that’s where the work belongs. If the low-level test is clean and NCCL still underperforms, the investigation moves immediately to HCA selection, GPU-to-NIC affinity, PCIe width and speed, GPU Direct RDMA, memory-locking limits, and container topology visibility.
A working node in the same cluster is worth more here than any piece of design documentation. Documentation tells you what should be true. A healthy node tells you what is true, right now, in your environment — and the differences are frequently small: a wrong HCA routing traffic across an inefficient NUMA path, a container missing the device visibility the host has, NCCL selecting an interface that’s usable but not the one intended for the workload. None of that requires a single failed switch port. All of it will quietly wreck your training throughput.
The Discipline That Actually Scales
Every team involved will report that their piece looks healthy. The application team sees a slow job. The GPU team sees available GPUs. The server team sees a reachable host. The network team sees active ports. Every dashboard is green, and the system is still underperforming.
That’s not a failure of any one team’s tools. It’s a failure to treat the AI cluster as a single system with a single data path, rather than four separate domains each reporting on their own slice of it.
The habits that have carried me through years of enterprise troubleshooting still apply: establish a baseline, isolate scope, compare known-good against suspected systems, change one variable at a time. What’s changed is where those habits have to reach. In this environment, understanding the fabric isn’t enough. You have to be fluent enough in HCA behavior, PCIe topology, and NCCL communication to follow the problem past the point where the traditional network boundary used to stop you.
An active link proves connectivity. It proves nothing about performance. If you’re operating AI infrastructure and you haven’t made that distinction the center of your troubleshooting approach, your dashboards will keep telling you everything is fine right up until the training run tells you otherwise.