Blog

AI-network observability should answer a business question: is the fabric helping expensive accelerators stay productive? Interface counters remain essential, but the useful signal appears when they are correlated with job timing, topology, GPU utilization, and expected service levels.

The observability objective is not more charts. It is faster, defensible decisions about GPU productivity. I design telemetry around a time-synchronized chain: job phase, GPU state, NCCL event, host path, fabric ports, topology, and recent change. Missing one of those links turns correlation into guesswork.

Define service around the workload

Traditional uptime is too weak. A fabric can be reachable and still add enough latency or congestion to waste substantial GPU time. Define indicators such as collective completion time, effective bandwidth, iteration-time variance, and rail imbalance.

Set targets by workload class and scale. A small inference service and a thousand-GPU training job should not share a single generic threshold.

Preserve topology context

Every metric should map to a device, port, rail, node, job, and time window. Topology-aware dashboards let operators move from a slow job to the paths that carried it.

Record topology changes as events. A chart without change context often produces confident but wrong explanations.

Make the dashboard drive a decision

Collect high-frequency counters on critical paths and during representative workloads, but retain summarized data for long-term trends. The goal is enough resolution to connect an application symptom with a fabric event.

Every panel should answer a defined question and lead to an action. Review dashboards after incidents and remove panels that did not help.

Define workload-facing service indicators

Use collective duration, effective bandwidth, bus bandwidth, iteration-time tail, GPU communication stall time, rail variance, and failed-job rate alongside conventional availability. Targets should vary by workload class, cluster size, and approved degraded state.

A port can be up while the service objective is failing. Alerting should begin from the workload effect and pivot into the fabric evidence, not assume reachability is the outcome that matters.

Align time across every layer

Synchronize UFM, switches, hosts, containers, job scheduler, NCCL logs, and GPU telemetry. Preserve job ID, node list, rank placement, and topology revision with each incident window.

Even a small clock offset can make a congestion spike appear to precede or follow the collective that caused it. Time integrity is an observability dependency, not housekeeping.

Use tiered sampling and retention

Collect broad low-frequency history for capacity and drift, then high-frequency telemetry for critical paths, scheduled benchmarks, and active incidents. UFM storage grows with fabric size, port count, retention, and sample interval, so model it explicitly.

Keep raw high-resolution data long enough to cover investigation and summarized data long enough to show seasonality and growth. Mark maintenance and topology changes on the same timeline.

Create diagnostic views, not status walls

A rail-comparison view should show normalized transmit and receive rate, XmitWait, errors, and latency for peer rails. A physical-health view should show active error deltas, peer identity, cable and optic details, and recent changes.

Review which panels were used during every major incident. Remove ornamental panels, add missing context, and turn repeated manual pivots into direct links or automated queries.

Build the baseline before the incident

The production baseline for observability for ai networks must be captured while the system is healthy and while a representative workload is running. I record Collective duration p95/p99, Rail-normalized rate, High-frequency XmitWait, and Topology revision against the same time window, then annotate node count, rank placement, message-size mix, software versions, routing state, and any active maintenance condition. Without that context, a counter value is merely a number.

I keep distributions, not just averages. A healthy mean can hide one weak rail, one peer path with poor locality, or a short congestion episode that dictates the iteration tail. The comparison set should include equivalent hosts and paths, p50/p95/p99 behavior, and the variance allowed by the acceptance plan. That makes the baseline useful during an escalation instead of decorative after the fact.

Exercise the exception path

A design is not operationally complete until the team has tested the state it expects to survive. We stage one bounded failure at a time, preserve workload and placement controls, and observe the chain from Job + topology context through Operator decision. The objective is not simply to prove that traffic still passes; it is to quantify convergence time, residual capacity, error behavior, and the GPU-time penalty.

The drill ends with a recovery check. Links, counters, routing, host affinity, and application performance must all return to the approved envelope. If a component recovers but the workload does not, the runbook needs the additional reset, drain, or revalidation step explicitly documented.

Leave an engineering record another team can use

For each release or material change, I preserve the topology revision, cable and port map, host inventory, firmware and driver set, validation commands, representative outputs, and the approved baseline. The record also identifies who owns the server, fabric, scheduler, security, and application layers. That ownership map shortens the first ten minutes of an incident more than another dashboard does.

The final handoff should answer three questions without tribal knowledge: what changed, what evidence proves it is healthy, and what condition requires rollback or escalation. For observability, that evidence includes the diagnostic matrix below as well as the application-level result. A green interface alone is never the acceptance criterion.

Technical reference: NVIDIA UFM telemetry capabilities and monitored attributes.