Blog

A slow distributed training job is not automatically an NCCL problem, a GPU problem, or a fabric problem. It is a system-path problem until the evidence narrows it. The fastest investigations move through layers in a fixed order and compare suspected systems with a known-good peer.

A slow NCCL job is not a reason to start setting environment variables. Tuning before isolation often hides the original fault and leaves the cluster harder to operate. I move through a fixed stack: job scope, physical health, RDMA transport, GPU-to-NIC path, NCCL selection, then algorithm and protocol behavior.

Start with a precise symptom

Capture the job ID, nodes, GPUs, framework, message size, collective pattern, start time, and the exact performance regression. Vague reports such as the network feels slow create vague investigations.

Compare against the same model and scale when possible. A larger job or different placement can change traffic behavior enough to invalidate the baseline.

Prove the physical and RDMA layers

Check link state, speed, error deltas, and low-level RDMA performance between the actual nodes involved. If ib_write_bw or an equivalent test is degraded, the collective layer is not the first place to tune.

Test both directions and each intended rail. A single clean pair does not validate the cluster.

Inspect the chosen topology

NCCL discovers GPU, NIC, and PCI topology and selects communication paths. Confirm the interfaces and HCAs it chose match the design. Containerized environments must expose an accurate system topology.

Once the failing layer is isolated, change one variable and repeat the same test. Save logs, topology captures, and before-and-after numbers so the runbook improves with every incident.

Capture a reproducible job signature

Record node list, rank count, GPUs per node, container image, framework and NCCL versions, model, batch size, collective, message range, job placement, and the precise metric that regressed. Preserve timestamps with timezone so UFM and host logs align.

Run the same job on a known-good node set if capacity allows. If the problem follows the nodes, investigate hardware and local configuration. If it follows placement or scale, the topology and traffic pattern become stronger suspects.

Test the transport before the library

Use ibstat or the appropriate device view to confirm state and rate, then run ib_write_bw between the actual endpoints and HCA devices. Test both directions, each rail, a range of message sizes, and enough duration to expose intermittent behavior.

Where supported, compare host-memory and GPU-memory tests. If low-level RDMA is slow, NCCL cannot repair it. If RDMA is healthy, move up with a clean boundary.

Read NCCL’s evidence

Enable NCCL_DEBUG=INFO for a controlled reproduction and use subsystem filtering only as needed so the log remains reviewable. Confirm the network plugin, HCA, interface, GPU Direct state, channel construction, and topology decisions match the design.

NCCL_SOCKET_IFNAME or HCA-selection settings can isolate an interface problem, but they should be diagnostic controls before they become permanent configuration. A forced setting that fixes one node may encode the wrong topology for the rest of the fleet.

Use nccl-tests as a controlled middle layer

Run all_reduce_perf or the relevant nccl-tests workload across the same nodes and GPUs, sweeping message sizes. Capture algorithm bandwidth, bus bandwidth, and out-of-bounds or timeout behavior.

Compare single-node, two-node, and full-scale results. The point at which scaling breaks is evidence: inside-node transport, one network pair, a rail group, or the larger fabric. Only after that boundary is known should protocol or channel tuning begin.

Build the baseline before the incident

The production baseline for nccl troubleshooting must be captured while the system is healthy and while a representative workload is running. I record ib_write_bw, NCCL INFO log, nccl-tests scale sweep, and Job timeline + UFM against the same time window, then annotate node count, rank placement, message-size mix, software versions, routing state, and any active maintenance condition. Without that context, a counter value is merely a number.

I keep distributions, not just averages. A healthy mean can hide one weak rail, one peer path with poor locality, or a short congestion episode that dictates the iteration tail. The comparison set should include equivalent hosts and paths, p50/p95/p99 behavior, and the variance allowed by the acceptance plan. That makes the baseline useful during an escalation instead of decorative after the fact.

Exercise the exception path

A design is not operationally complete until the team has tested the state it expects to survive. We stage one bounded failure at a time, preserve workload and placement controls, and observe the chain from Job symptom through Measured fix. The objective is not simply to prove that traffic still passes; it is to quantify convergence time, residual capacity, error behavior, and the GPU-time penalty.

The drill ends with a recovery check. Links, counters, routing, host affinity, and application performance must all return to the approved envelope. If a component recovers but the workload does not, the runbook needs the additional reset, drain, or revalidation step explicitly documented.

Leave an engineering record another team can use

For each release or material change, I preserve the topology revision, cable and port map, host inventory, firmware and driver set, validation commands, representative outputs, and the approved baseline. The record also identifies who owns the server, fabric, scheduler, security, and application layers. That ownership map shortens the first ten minutes of an incident more than another dashboard does.

The final handoff should answer three questions without tribal knowledge: what changed, what evidence proves it is healthy, and what condition requires rollback or escalation. For troubleshooting, that evidence includes the diagnostic matrix below as well as the application-level result. A green interface alone is never the acceptance criterion.

Technical reference: NVIDIA NCCL troubleshooting guidance.