Why bigger InfiniBand timeout and retry values can turn a fast, diagnosable failure into a long, expensive silence.
An RDMA timeout has a way of making engineers reach for a larger timeout. The instinct is understandable: the job failed, the log named a timeout, and NVIDIA exposes a setting called NCCL_IB_TIMEOUT. Increase the number, rerun the workload, and perhaps the error disappears.
That can be a valid adjustment on a genuinely large InfiniBand fabric. It can also hide the only useful symptom you had. A longer timer does not restore a degraded lane, return missing credits, correct a bad route, move a GPU closer to its HCA or create bandwidth. It changes how long NCCL lets the transport wait before declaring that forward progress has stopped.
I treat timeout tuning as a failure-detection decision, not a performance fix. Before changing it, I want to know whether the path is slow but progressing or whether it is retrying the same unsuccessful exchange. Those are different conditions. Giving the first one more time may be appropriate. Giving the second one more time simply makes the incident quieter and more expensive.
A timeout is a clock, not a repair
The current NCCL documentation defines the InfiniBand Verbs timeout as 4.096 microseconds multiplied by 2 to the power of NCCL_IB_TIMEOUT. Since NCCL 2.23, the default value has been 20, which works out to roughly 4.3 seconds per retry. NCCL_IB_RETRY_CNT defaults to 7. Taken together, NVIDIA’s own example table describes about 30 seconds of total waiting at the defaults.[1]
Move the timeout value from 20 to 23 while leaving the retry count alone and the documented example grows to roughly 34.4 seconds per retry and about four minutes in total. The faulty path has not improved. The application has merely agreed to wait much longer for the same completion that may never arrive.
Figure 1 shows the distinction. The healthy lane makes progress and receives an acknowledgement inside the normal budget. The two fault lanes represent the same broken or non-progressing path. Raising the timeout pushes the error marker to the right; it does not move the acknowledgement closer. Only the separate scale case at the bottom represents the legitimate reason to enlarge the window: a clean path whose valid response arrives later because the fabric’s size and round-trip behavior demand it.
Same Fault, Longer Silence
| Path condition | Evidence | Timer budget | Outcome |
|---|---|---|---|
| Healthy | Bytes and operations advance; ACK returns | Default | Work completes |
| Fault | No progress; retries climb | Timeout 20 ≈ 30 sec total | Fails fast enough to diagnose |
| Same fault | Still no progress | Timeout 23 ≈ 4 min total | Same failure, later |
| Clean fabric at scale | Slow but measurable progress | Larger proven budget | Adjustment justified |
The first question is whether anything is moving
When the job is stalled, elapsed time is not the metric I trust. Progress is. I look for advancing NCCL operations, changing byte counters and completed work on the affected ranks. A slow collective that continues to retire operations is not the same incident as a rank whose traffic and operation count have frozen. The timer value cannot tell you which case you have; the surrounding evidence can.
NCCL logs help establish that boundary. INFO-level output filtered to INIT, NET and GRAPH can show the selected interfaces, transport and topology decisions, while a per-process log file keeps a large job readable.[4] If the failure follows one rank, I map that rank through its GPU, PCIe path, HCA and switch port. A timeout on the application side is the end of a causal chain, not a location.
Below NCCL, rdma statistic is particularly useful during a controlled reproduction. Counters such as local_ack_timeout_err, packet_seq_err, implied_nak_seq_err and rnr_nak_retry_err describe transport behavior that the application log cannot localize. NVIDIA’s networking guidance notes that growing values often point to loss, retries or other transport problems.[2] The word growing matters. A historical total is background; a counter that advances in the failure window is evidence.
This is also where an apparently successful rerun can mislead. If a larger timeout allows one job to finish, I still ask what consumed the extra budget. Without that answer, the next workload may cross the new limit, or worse, spend minutes occupying GPUs before producing the same failure.
Put the timer beside the fabric
The timeout event becomes useful when it is placed on the same timeline as UFM and port diagnostics. UFM Enterprise telemetry provides port-level bandwidth, congestion, error and latency information, including XmitWait-based congestion measurement. Its current documentation lists a 30-second primary sample interval by default, with historical primary data stored every five minutes.[3] A brief credit stall or link recovery can therefore be obvious to a collective and diluted in a coarse historical view. For a reproducible case, I preserve the higher-resolution window around the failure.
Then I follow the exact path. ibstat or ibstatus confirms that the chosen HCA port is active and operating at the expected rate. mlxlink verifies width, speed and physical health. perfquery -x exposes symbol, link-recovery and link-downed counters, and ibdiagnet checks fabric topology, routing and errors. If a port is recovering, a rail is downshifted or a physical counter rises under load, a larger timeout is an admission that the fault remains in service.
Congestion requires the same discipline. High XmitWait tells me that transmission was blocked waiting for credits; it does not identify the defective component. I trace the pressure downstream and compare equivalent paths. Timeout tuning cannot solve oversubscription, traffic imbalance or a capacity loss caused by a failed link. At best it stretches the period during which backpressure is tolerated.
When a larger value is justified
NVIDIA explicitly notes that increasing NCCL_IB_TIMEOUT can help on very large networks, including cases that surface ibv_poll_cq error 12. That is a scale accommodation, not a blanket remedy.[1] I make the change only after the selected path is clean, the physical links are at the intended width and speed, retry counters are not climbing abnormally, and the evidence points to a legitimate round-trip budget rather than loss of progress.
The test is controlled: one variable changes, the workload shape stays fixed, and I compare completion latency, counter deltas and failure behavior before and after. I do not casually persist NCCL debugging or workaround variables in global configuration. The documentation itself cautions that many of these settings are intended for experiments and troubleshooting and may cause suboptimal behavior when left in place.[1] Values of 0 or 32 and above produce an infinite Verbs timeout; operationally, that can turn a useful failure into an indefinite wait, so it demands exceptional justification.
The strong takeaway is simple: a timeout is an observability boundary. If increasing it makes a job appear healthy, the next question is not whether the ticket can be closed. The next question is what latency, retry or path condition has just been hidden. Prove that the fabric needed a larger window before you teach the application to wait.
Technical references
Current NVIDIA guidance verified September 28, 2026.