Blog

Redundancy is a design property; resilience is an operating result. An AI fabric is resilient when the team understands failure domains, knows the expected degraded behavior, can make controlled changes, and can restore performance without improvising under pressure.

I separate redundancy from resilience. Redundancy tells me there is another component. Resilience tells me the workload still has an acceptable path, the team detects the failure, and recovery occurs through a practiced change with evidence and rollback.

Name the failure domains

Map power, racks, leaves, spines, rails, management services, software versions, and external dependencies. Components that appear redundant may still share a hidden failure domain.

For every domain, record the blast radius and the capacity that remains after failure.

Define degraded performance

A surviving path does not guarantee acceptable job completion. Model and test the workload with a failed link, leaf, or spine so operators know the difference between functional and healthy.

Use the result to set job-placement rules and maintenance limits.

Make changes reversible and practice recovery

Every change needs a hypothesis, prechecks, success measures, rollback trigger, and named decision owner. Avoid combining unrelated firmware, cabling, routing, and host changes in one window.

Run controlled failure exercises and verify traffic redistribution, alarms, health reports, application impact, and the communications path between network, platform, server, and application teams.

Model residual capacity for every failure

For each link, leaf, spine, rail, management instance, power domain, and rack, calculate the paths and bandwidth that remain when it is unavailable. Show shared dependencies such as optics, patch panels, power feeds, upstream line cards, and software versions.

A nonblocking fabric can become oversubscribed after one failure. Document the expected collective and iteration-time impact so operators know whether the degraded system can keep running or should drain jobs.

Test routing convergence with the workload active

A topology may reconverge correctly while synchronized traffic experiences a long tail or timeout. During a controlled job, remove one approved path and collect route state, per-link utilization, XmitWait, errors, NCCL behavior, and application completion.

The important interval is from fault onset to stable useful bandwidth. Alert arrival alone does not measure recovery.

Build a change around decision points

Every maintenance plan needs a pre-change baseline, exact scope, dependency map, success measure, abort condition, rollback owner, and the last safe time to reverse. Separate firmware, topology, cabling, and host-policy changes whenever possible.

If the intended change requires a temporary loss of redundancy, freeze unrelated work and define the jobs that may run. Capacity protection is part of change control.

Practice cross-team recovery

Run exercises that force handoffs among network, platform, server, scheduler, storage, security, and application owners. Use the real telemetry, ticketing, escalation, and communications paths.

Afterward, update the topology, thresholds, command examples, contact path, and decision tree. A runbook that does not change after a drill is probably recording procedure rather than learning.

Build the baseline before the incident

The production baseline for resilient ai fabrics must be captured while the system is healthy and while a representative workload is running. I record Residual bisection capacity, Time to stable throughput, Abort threshold, and Runbook drift against the same time window, then annotate node count, rank placement, message-size mix, software versions, routing state, and any active maintenance condition. Without that context, a counter value is merely a number.

I keep distributions, not just averages. A healthy mean can hide one weak rail, one peer path with poor locality, or a short congestion episode that dictates the iteration tail. The comparison set should include equivalent hosts and paths, p50/p95/p99 behavior, and the variance allowed by the acceptance plan. That makes the baseline useful during an escalation instead of decorative after the fact.

Exercise the exception path

A design is not operationally complete until the team has tested the state it expects to survive. We stage one bounded failure at a time, preserve workload and placement controls, and observe the chain from Failure domain through Proven recovery. The objective is not simply to prove that traffic still passes; it is to quantify convergence time, residual capacity, error behavior, and the GPU-time penalty.

The drill ends with a recovery check. Links, counters, routing, host affinity, and application performance must all return to the approved envelope. If a component recovers but the workload does not, the runbook needs the additional reset, drain, or revalidation step explicitly documented.

Leave an engineering record another team can use

For each release or material change, I preserve the topology revision, cable and port map, host inventory, firmware and driver set, validation commands, representative outputs, and the approved baseline. The record also identifies who owns the server, fabric, scheduler, security, and application layers. That ownership map shortens the first ten minutes of an incident more than another dashboard does.

The final handoff should answer three questions without tribal knowledge: what changed, what evidence proves it is healthy, and what condition requires rollback or escalation. For resilience, that evidence includes the diagnostic matrix below as well as the application-level result. A green interface alone is never the acceptance criterion.

Technical reference: NVIDIA UFM health and validation workflow.