Why multi-rail AI clusters can hide one asymmetric path—and how to prove it without tuning by instinct.
Multi-rail InfiniBand creates an uncomfortable troubleshooting problem: the cluster can have plenty of total bandwidth while one part of the data plane is quietly unhealthy. The job still runs. UFM still sees an operational fabric. A low-level test may even return an excellent number. Yet NCCL performance moves unpredictably because the workload is not using “the network” as one object. It is using specific GPUs, HCAs, ports, rails, and paths.
This is where aggregate health becomes a liability. If Rail 0 is clean and Rail 1 is losing efficiency, the healthy rail can make the cluster-wide view look respectable. It cannot make the collective finish on time. The slower rail still controls the part of the operation assigned to it, and synchronized GPU work has no way to average that delay away.
My rule for multi-rail incidents is straightforward: never accept a passing test until I know which HCA and which rail actually passed.
The average is lying to you
NCCL understands that multiple NICs may represent separate network rails. Its NCCL_CROSS_NIC behavior is topology-dependent: NCCL normally tries to keep the same ring or tree on corresponding NICs between nodes so traffic remains aligned with the rail design. The default value attempts that alignment but can allow different NICs when NCCL predicts better performance. In a rail-optimized fabric, that choice matters because the inter-rail path may not have the same capacity or contention characteristics as the intended straight-through path.[1]
The diagram shows the failure pattern I care about. A collective uses two apparently equivalent rails. Rail 0 delivers normally; Rail 1 is degraded by a host, HCA, link, switch-port, routing, or congestion condition. Testing both rails together tells me only that the pair underperformed. Testing Rail 0 alone produces a clean result that can falsely clear the network. The decisive test is the one that isolates Rail 1 using the exact HCA and port selected by the workload.
That isolation sequence is more informative than another round of broad cluster benchmarks. It converts a vague performance complaint into an asymmetry: both rails slow, Rail 0 clean, Rail 1 degraded. Once that pattern repeats, the investigation has a physical direction.
Multi-Rail Isolation: Name the Path
| Test | Rail 0 · HCA 0 | Rail 1 · HCA 1 | Conclusion |
|---|---|---|---|
| Both rails | Normal | Degraded | Aggregate result hides the source |
| Rail 0 only | Pass | Isolated | Clean test proves only Rail 0 |
| Rail 1 only | Isolated | Fail | Fault localized to Rail 1 |
The clean test that proves nothing
ib_write_bw is useful, but it answers only the experiment you actually ran. NVIDIA’s current NCCL troubleshooting guide recommends specifying the device with -d <device> and checking each port for active state, expected link layer, rate, and width. It also recommends comparing host-memory and GPU-memory paths when GPUDirect RDMA is involved.[2] If an engineer runs a default test, records a good result, and never confirms the selected HCA, the test may have exercised the healthy rail from start to finish.
I want a controlled rail matrix. The same node pair, message range, GPU placement, and runtime conditions are held constant while the permitted HCA changes. Rail 0 runs alone. Rail 1 runs alone. Then both run together. If the degradation follows one device, the search narrows immediately. If each rail is healthy alone but performance collapses when combined, I examine traffic placement, merging behavior, PCIe locality, and whether the fabric is forcing unexpected cross-rail movement.
Latency belongs in that matrix as well. Bandwidth can look normal while ib_write_lat reveals an unstable tail. RDMA statistics can add transport evidence: growing sequence errors, retry-related errors, or local acknowledgment timeouts during the isolated test point toward loss, retransmission, or an impaired path. mlxlink, perfquery, and ibdiagnet then move the investigation from symptom to link and topology evidence.[2]
Follow the rail from rank to port
Before interpreting UFM, I map the software path. NCCL logging at INFO level with the INIT, NET, and GRAPH subsystems exposes topology detection, network-device selection, and the communication graph. Separate files per host and process keep that evidence attributable to the correct rank.[3] The output is not decorative debug text; it tells me which GPU and network device the job believed it was using.
From there, the map continues through NUMA placement, PCIe, HCA, HCA port, switch-facing port, rail, and expected spine path. UFM telemetry is evaluated against that map—not against the fabric in general. Bandwidth imbalance across peer ports, rising physical errors, latency changes, or XmitWait concentrated on one rail are meaningful only when they align with the job window and the path NCCL selected.[4]
A weak rail does not always announce itself with a down port. One HCA may be negotiating the wrong rate. A dual-port adapter may expose asymmetric PCIe locality. A leaf uplink may be carrying more traffic because routing or a failed path reduced diversity. A cable can remain linked while error recovery consumes enough time to damage collective performance. The common feature is not the counter name. It is repeatable asymmetry under the same controlled load.
Use NCCL controls as diagnostic probes
NCCL provides powerful controls, but they should not become folklore pasted into every launch script. NCCL_IB_HCA can constrain the devices and ports used for an experiment. Exact matching matters: NVIDIA recommends the = prefix because a token such as mlx5_1 can also match mlx5_10 through mlx5_19. The variable now also supports explicit rail and plane identities.[1] That precision is exactly what a rail-isolation test needs.
The same discipline applies to merging. NCCL_IB_MERGE_NICS allows dual-port NICs to be combined into one logical device. In NCCL 2.31, NCCL_NET_MERGE_POLICY=RAIL can restrict eligible merges to devices assigned to the same rail—useful in rail-optimized systems where equal GPU distance could otherwise make different rails look mergeable.[1] These controls can prove how behavior changes when device grouping or rail crossing changes. They are not substitutes for repairing the underlying asymmetry.
Once the fault is understood, temporary diagnostic settings come back out. NVIDIA explicitly warns that many debugging parameters can create suboptimal behavior, crashes, or hangs when retained as permanent configuration.[1] A production fix should restore the intended topology, link health, affinity, or routing—not simply hide a sick rail from NCCL forever.
The takeaway
A multi-rail fabric is not healthy because its average is healthy. It is healthy when corresponding rails behave symmetrically under the workload they were designed to carry.
Prove each rail independently. Confirm the exact HCA and port. Recombine them only after the isolated paths are clean, then verify that NCCL’s topology and merging decisions match the physical design. One bad rail can hide inside a fast cluster for a long time—but it cannot hide from a test that names the path.
Technical references
Current NVIDIA guidance verified September 21, 2026.