Why a healthy InfiniBand fabric can still deliver poor NCCL performance, and how to isolate the PCIe and GPUDirect boundary without guessing.
A clean InfiniBand fabric is not the same thing as a clean communication path. UFM can show healthy links, stable port counters and no meaningful congestion while one node continues to underperform. That is not a contradiction. The fabric begins at the HCA port; an NCCL transfer begins in GPU memory. Between those points sits a PCIe topology, a NUMA boundary, a driver path and a memory-registration mechanism. Any one of them can make a network problem appear where the network has done nothing wrong.
This is one of the easiest places to lose hours in an AI cluster. The slow job creates pressure to inspect switches, routes and cables because the symptom is collective bandwidth. But if host-memory RDMA runs clean and GPU-memory RDMA does not, the most important evidence is already inside the server. Replacing a cable or changing adaptive-routing policy will not repair a broken GPUDirect path.
The HCA link is only the last gate
GPUDirect RDMA exists to let a network device access GPU memory directly over PCI Express, avoiding a staging copy through host memory. NVIDIA’s current GPUDirect documentation also states an important topology constraint: the peer devices must share the same upstream PCIe root complex.1 That does not mean every same-root path performs identically. A GPU and HCA separated by extra PCIe switches, a host bridge or a NUMA interconnect can still carry traffic, but the locality and contention profile are different.
I start with nvidia-smi topo -m or the newer focused GPU-to-NIC topology view. The legend matters. PIX indicates a path through a single PCIe switch; PXB crosses multiple PCIe switches; PHB crosses a PCIe host bridge; SYS adds the inter-socket fabric between NUMA nodes. NVIDIA documents these distinctions because they describe the actual path, not cosmetic inventory.2 If the affected rank is using a GPU-to-HCA pairing marked SYS while a healthy rank uses PIX, I do not call that proof of root cause. I call it a testable asymmetry.
Figure 1 shows the boundary clearly. On the direct path, GPU memory reaches a local HCA through the intended PCIe hierarchy, then enters the InfiniBand fabric. On the detour path, traffic crosses a host bridge or NUMA boundary, or falls back through CPU memory. The switches can be perfect in both cases. The performance tax is paid before the first fabric port.
Use memory type as the fault-domain switch
The most decisive test is not another full training run. It is a controlled A/B comparison using the same nodes, HCA ports, fabric path and message sizes while changing only the memory source. Run RDMA bandwidth with host memory, then repeat with GPU memory using the supported GPUDirect mechanism for the platform. NVIDIA’s NCCL networking guidance recommends this comparison specifically because it separates basic RDMA transport from the GPU-memory path.3
If both tests are poor, stay on the network path: confirm port state, negotiated width and speed, routing, congestion and physical counters. If host memory is healthy but GPU memory collapses, move inward. Verify that the intended GPU and HCA are paired, that the relevant peer-memory or DMA-BUF path is available, and that the container exposes the topology and devices NCCL needs. NCCL relies on /sys to discover PCI topology; incomplete container visibility can produce a bad transport decision even though every physical component is healthy.3
The comparison should be symmetrical. Test the suspect GPU-HCA pair against a known-good pair on the same server, then test the same HCA with another GPU where the topology allows it. Keep CPU affinity and message size constant. The goal is to make the defect follow one boundary: a GPU, an HCA, a PCIe branch, a NUMA crossing or the fabric. A result that follows the switch port is a network lead. A result that follows the GPU-to-NIC pairing is a host lead.
Do not overlook ACS and IOMMU behavior
A topology diagram can look correct while PCIe policy silently redirects peer-to-peer traffic. NVIDIA’s current NCCL troubleshooting guide warns that, on Linux bare-metal systems, translated IOMMU mode is not supported for CUDA PCIe peer-to-peer transfers. It also calls out PCI Access Control Services because ACS can redirect PCIe point-to-point traffic toward the CPU root complex, producing major performance loss or a hang.4
That is why a positive peer-access matrix is useful but not sufficient. I check the active kernel IOMMU mode and inspect ACS control on the relevant bridges. I do not disable platform security features casually, and I do not turn a diagnostic command into a permanent configuration without matching it to the server design and vendor guidance. The operational point is simpler: if the path is being redirected, NCCL is reporting the consequence. The fabric is not the cause.
Let UFM clear the fabric precisely
UFM still plays a critical role, but I use it to answer a bounded question: did the fabric contribute during the test window? Correlate the exact HCA-facing switch ports with bandwidth, PortXmitWaitExtended, receive errors, symbol errors, link recovery and link-down counters. NVIDIA defines PortXmitWaitExtended as time an egress port had data to send but could not transmit because it lacked credits or arbitration opportunity.5 Rising physical or congestion counters keep the fabric in scope. Flat error counters, expected bandwidth on the wire and a clean equivalent path make a host-side explanation more credible.
The mistake is to use a green dashboard as a universal acquittal or a single nonzero counter as a conviction. Historical totals are not incident evidence. Counter deltas aligned to the failed test are. UFM tells you what happened from the HCA port outward; the host tests tell you what happened before that boundary. Root cause appears when both timelines agree.
The practical verdict
When NCCL is slow, I want one sentence that names the first constrained segment: GPU memory to PCIe, PCIe to HCA, HCA to first switch, or fabric path beyond the edge. Until that sentence is supported, changing NCCL variables, moving routes or replacing optics is activity, not diagnosis.
The strong takeaway is this: prove the GPU-to-HCA path before blaming the InfiniBand fabric. A healthy host-memory RDMA test does not validate GPUDirect. A healthy switch port does not validate PCIe locality. And a successful collective does not prove that NCCL chose the efficient path. Treat the server and fabric as one communication system, then use controlled comparisons to identify the exact boundary where direct became indirect.
Technical references
Current NVIDIA guidance verified October 5, 2026.