Blog

A fabric is not ready because every link is green. Production readiness requires a staged validation plan that begins with inventory and physical health, proves every intended path, exercises the system under load, and creates the baseline operations will use after handoff.

Cluster bring-up is where design assumptions become evidence. I want every test to establish a boundary: physical layer, fabric health, RDMA transport, GPU memory path, collective behavior, or workload. If a test fails, the last clean boundary should tell the team where to work next.

Validate what was built

Reconcile serial numbers, firmware, device names, port maps, cable labels, speeds, and topology against the approved design. Resolve every unexpected device and down port before performance testing.

Capture rack and cable evidence while the installation team is still available. Physical ambiguity becomes expensive after production starts.

Prove health before performance

Run fabric health, diagnostic, and validation checks. Clear stale counters or record their origin, then watch for new errors during controlled link tests.

A clean baseline makes later deltas meaningful. Without it, operators cannot tell whether a counter is new, old, or workload-related.

Test paths in layers

Start with link and port checks, then point-to-point RDMA, rail-by-rail tests, multi-node communication, and finally representative collectives. Each layer should have explicit pass criteria.

Handoff topology captures, health reports, performance baselines, alarm policy, escalation paths, spares, rollback steps, and the exact test commands used.

Freeze inventory before testing

Reconcile GUIDs, serial numbers, firmware, device names, rack positions, cables, optics, port speed and width, rail membership, and management reachability with the approved implementation package.

Resolve unexpected devices, duplicate identifiers, and down ports before performance work. A test result cannot validate a topology the team has not accurately inventoried.

Run health tools with clean counter context

Use UFM Fabric Health and Fabric Validation, and run IBDiagnet according to the supported procedure. Record warnings, excluded or unhealthy ports, routing checks, cable and BER information where available, and the exact tool versions.

Clear counters only when the operating plan allows it; otherwise record the initial values and calculate deltas. The goal is to know which errors occurred during this test window, not to produce an artificially empty report.

Build the test ladder

Test link state and speed, point-to-point latency and bandwidth, host-memory and GPU-memory RDMA, each rail, multiple peers, nccl-tests, then the representative application. Run both directions and sweep message sizes.

Add controlled failure tests after the healthy baseline: one uplink, leaf path, or other approved fault at a time. Confirm alarms, routing change, residual capacity, and recovery. Restore the baseline before the next fault.

Deliver a reproducible evidence package

The handoff should include raw outputs, summarized pass criteria, topology captures, UFM reports, test commands, node lists, timestamps, container images, firmware and driver inventory, and known limitations.

Operators should rerun the tests themselves before acceptance. A bring-up owned only by the installation team leaves production with documentation but no practiced capability.

Build the baseline before the incident

The production baseline for ai fabric bring-up must be captured while the system is healthy and while a representative workload is running. I record IBDiagnet / Fabric Health, RDMA message sweep, GPU-memory path, and Controlled failure against the same time window, then annotate node count, rank placement, message-size mix, software versions, routing state, and any active maintenance condition. Without that context, a counter value is merely a number.

I keep distributions, not just averages. A healthy mean can hide one weak rail, one peer path with poor locality, or a short congestion episode that dictates the iteration tail. The comparison set should include equivalent hosts and paths, p50/p95/p99 behavior, and the variance allowed by the acceptance plan. That makes the baseline useful during an escalation instead of decorative after the fact.

Exercise the exception path

A design is not operationally complete until the team has tested the state it expects to survive. We stage one bounded failure at a time, preserve workload and placement controls, and observe the chain from Inventory + cabling through Operations handoff. The objective is not simply to prove that traffic still passes; it is to quantify convergence time, residual capacity, error behavior, and the GPU-time penalty.

The drill ends with a recovery check. Links, counters, routing, host affinity, and application performance must all return to the approved envelope. If a component recovers but the workload does not, the runbook needs the additional reset, drain, or revalidation step explicitly documented.

Leave an engineering record another team can use

For each release or material change, I preserve the topology revision, cable and port map, host inventory, firmware and driver set, validation commands, representative outputs, and the approved baseline. The record also identifies who owns the server, fabric, scheduler, security, and application layers. That ownership map shortens the first ten minutes of an incident more than another dashboard does.

The final handoff should answer three questions without tribal knowledge: what changed, what evidence proves it is healthy, and what condition requires rollback or escalation. For deployment, that evidence includes the diagnostic matrix below as well as the application-level result. A green interface alone is never the acceptance criterion.

Technical reference: NVIDIA UFM fabric-health and validation procedure.