The external fabric can be clean while training performance remains poor because the effective network begins inside the server. GPUDirect RDMA is designed to let GPUs exchange data directly with network adapters, but performance depends on hardware topology, drivers, kernel behavior, and what the workload can actually see.
When a fabric benchmark is clean but GPU communication is not, I move inside the node. The important question becomes whether the GPU and network adapter can exchange memory directly over the intended PCIe path, with the driver and runtime exposing the same topology the hardware actually implements.
SYSTEM PATH · the operating path
Map the PCIe path first
Document every GPU, NIC or HCA, PCIe switch, CPU socket, and NUMA boundary. Two adapters with the same link speed can deliver very different results if one path crosses a host interconnect.
Use a known-good node as the reference and compare topology output, link width, link speed, firmware, and device placement. Small differences can explain large performance gaps.
Confirm the direct path is active
Compatible hardware alone does not prove GPUDirect RDMA is in use. Driver mode, DMA-BUF or peer-memory support, container device access, and runtime configuration all matter.
Validate from inside the job environment, not only on the host. A container can expose a different device view or an incomplete system topology.
Separate fabric tests from collective tests
IOMMU translation and PCI Access Control Services can redirect peer-to-peer traffic through the CPU root complex. The result may look like a network problem or an application stall.
Begin with low-level RDMA bandwidth and latency between specific adapters. If that path is degraded, fix it before tuning NCCL. Layered testing gives every boundary a clear pass-or-fail result.
Establish the physical topology
Capture nvidia-smi topo -m, lspci topology, PCIe link width and speed, NUMA association, HCA names, firmware, and driver versions. Mark whether each GPU-to-NIC path is PIX, PXB, PHB, or SYS distance where applicable.
Compare every suspect node with a node that performs correctly in the same rack and job. A device on the wrong root complex, a down-trained PCIe link, or one different BIOS policy can explain a large bandwidth gap without producing a failed interface.
Verify the GPUDirect mechanism
Current environments may use nvidia-peermem or DMA-BUF depending on kernel, driver, and platform support. Confirm the selected mechanism is available after boot and inside the workload environment. Do not infer it from installed packages.
For perftest builds that support it, compare host-memory ib_write_bw with GPU-memory mode and, where applicable, DMA-BUF mode. A healthy host-memory result with a degraded GPU-memory result sharply narrows the investigation to the GPU-to-NIC path.
Treat IOMMU and ACS as architecture decisions
On Linux bare metal, translated IOMMU mode and PCIe Access Control Services can redirect or break peer-to-peer transfers. Inspect kernel boot mode and ACS control on the actual bridges in the GPU-to-NIC path. The remedy is platform-specific and must account for security and virtualization requirements.
Virtual machines require isolation behavior that bare-metal guidance may disable. In that case, validate ATS and the supported device-assignment model instead of copying a bare-metal setting into a virtual platform.
Prove topology visibility inside containers
NCCL uses /sys to understand PCI topology. Mount and device exposure errors can present a simplified topology, hide an HCA, or change interface selection. Run the topology and device checks inside the same container image, namespace, and security context as the training job.
This is also where memory-locking limits, device permissions, and runtime hooks surface. The host can be perfect while the job environment quietly removes the direct path.
Build the baseline before the incident
The production baseline for gpudirect rdma and pcie topology must be captured while the system is healthy and while a representative workload is running. I record Host vs GPU ib_write_bw, nvidia-smi topo -m, IOMMU / ACS state, and Container /sys and devices against the same time window, then annotate node count, rank placement, message-size mix, software versions, routing state, and any active maintenance condition. Without that context, a counter value is merely a number.
I keep distributions, not just averages. A healthy mean can hide one weak rail, one peer path with poor locality, or a short congestion episode that dictates the iteration tail. The comparison set should include equivalent hosts and paths, p50/p95/p99 behavior, and the variance allowed by the acceptance plan. That makes the baseline useful during an escalation instead of decorative after the fact.
Exercise the exception path
A design is not operationally complete until the team has tested the state it expects to survive. We stage one bounded failure at a time, preserve workload and placement controls, and observe the chain from GPU memory through RDMA fabric. The objective is not simply to prove that traffic still passes; it is to quantify convergence time, residual capacity, error behavior, and the GPU-time penalty.
The drill ends with a recovery check. Links, counters, routing, host affinity, and application performance must all return to the approved envelope. If a component recovers but the workload does not, the runbook needs the additional reset, drain, or revalidation step explicitly documented.
Leave an engineering record another team can use
For each release or material change, I preserve the topology revision, cable and port map, host inventory, firmware and driver set, validation commands, representative outputs, and the approved baseline. The record also identifies who owns the server, fabric, scheduler, security, and application layers. That ownership map shortens the first ten minutes of an incident more than another dashboard does.
The final handoff should answer three questions without tribal knowledge: what changed, what evidence proves it is healthy, and what condition requires rollback or escalation. For system path, that evidence includes the diagnostic matrix below as well as the application-level result. A green interface alone is never the acceptance criterion.
Field diagnostic matrix
| Signal | What it tells us | Engineering response |
|---|---|---|
| Host vs GPU ib_write_bw | Separates fabric performance from GPU-memory path | Investigate GPUDirect when only GPU mode is slow |
| nvidia-smi topo -m | GPU, NIC, PCIe, and NUMA relationship | Compare with the validated node layout |
| IOMMU / ACS state | Possible redirection of peer-to-peer traffic | Review the exact platform and virtualization model |
| Container /sys and devices | Topology and HCA visibility seen by NCCL | Validate from the production runtime |
Technical reference: NVIDIA NCCL guidance for GPUDirect, PCIe, IOMMU, and ACS.
X4 Networks’ CCIE-led team works across architecture, deployment, monitoring, and escalation—24 × 7.