Gray Failure Detection in Adaptive Routing

A live link can still deliver poor service, so operators need to compare expected traffic distributions with observations along individual paths.

The blind spot in health checks

A gray failure is a defect that appears differently to different observers of the same system. A device can report normal operation while some requests suffer loss or long delays. Microsoft researchers describe this difference as "differential observability."

For example, request processing can stop while the function sending health signals continues to run. The detector receives a healthy signal, but the user's request never completes. The choice of observation determines whether the system appears healthy or faulty.

Connectivity alone also cannot describe delivery quality in a distributed training network. Synchronous communication among multiple participants waits for delayed participants. Observations need to connect loss on particular paths with application waiting time.

Expected packet counts per path

Adaptive routing changes forwarding paths in response to network conditions. Packet spraying spreads the packets of a single flow across multiple paths. Knowing how many packets should traverse each path enables a comparison with arrivals.

A leaf is an edge switch connected to servers, while a spine provides paths between leaves. In a network with several spines, a receiving leaf can distinguish which spine delivered a packet. This information reveals differences between paths that an aggregate receive count can hide.

The following numbers are illustrative assumptions, not measurements. Assume a sender distributes 10,000 packets equally over four paths, giving an expectation of 2,500 per path. If the final path has delivered only 2,000 when observation ends, it has a deficit of 500 packets.

The expectation must come from the sending volume and distribution policy. Dividing the 9,500 actual arrivals into four gives 2,375 per path and understates the deficit. Deriving the baseline only from received traffic mixes the effect of loss into the healthy expectation.

Normal variation and observation windows

Random path selection does not produce identical counts even in a healthy network. Declaring a failure whenever one path receives fewer packets would mistake normal variation for a defect. The decision needs sufficient observations and a distribution measured under healthy conditions.

Late packets also need to be distinguished from lost packets. Ending observation too early counts packets still in transit as missing. Extending every observation window indefinitely delays failure detection instead.

The assumption of equal distribution also needs verification. A higher weight on one path or the removal of some paths changes the expected healthy counts. Operational comparisons should align the path set, distribution policy, and observation window.

Detection and diagnosis

A packet deficit identifies a suspicious path without immediately identifying faulty equipment. If path D in the example contains several links and switches, locating the loss requires more evidence. Other flows sharing the same segments can provide another observation.

This table proposes an operational process. Separating an anomaly signal from permission to remove a path prevents one false alarm from immediately causing a broad routing change. Both loss and application waiting time should be observed after rerouting to establish whether the intervention worked.

The SprayCheck example

SprayCheck is a research system that compares packet counts by spine at the destination leaf. It gives measurement flows high priority to reduce interference from competing traffic. That condition and calibration against a healthy distribution prevent it from being a rule that operators can attach to arbitrary receive counters.

As checked in September 2026, the paper reports detecting a 1.5% packet-loss rate on a single link in a 64-spine simulation. Detection completed within one training iteration for a specified Llama-3 70B configuration. This result depends on that communication volume and network setup; it is not a detection-time guarantee for another service.

Criteria for operational use

First establish whether the required observations are available. Without sending volume or ingress-path identification, the preceding comparison cannot be built as described. Giving measurement traffic priority also requires checking whether it increases latency for other work.

Workloads with many small flows or frequent path changes may need different calibration. Combining observations from different conditions can hide a fault or make a normal change look faulty. Recording change times and routing policies preserves the context needed to interpret observations.

Observation using live traffic covers only paths traversed by that traffic. An unused path must not be classified as healthy merely because no fault was observed. In a network where normal congestion also drops packets, not every deficit indicates defective equipment.

Operational evaluation should go beyond detection speed. Record false alarms in healthy conditions, missed detections during fault injection, and recovery after rerouting separately. These distinguish missing observations, incorrect baselines, and ineffective mitigation.

Summary

Gray failures are easy to miss when healthy device responses diverge from the delivery quality experienced by applications. A network that distributes packets across paths can compare arrivals with expectations derived from sending volume and routing policy. Valid comparisons must account for normal variation, late arrivals, and routing changes. Operationally useful observation also connects detection to diagnosis and verification that rerouting helped.