Judging Multi-Agent Adoption on Reliability

Whether to adopt a multi-agent system is a question about reliability, not about average scores. Without a design that handles how errors spread and what reversal costs, attaching a verifier to a single agent is the better choice.

Whether to add more agents is not a decision that benchmark averages can settle. An average score does not show the cost you actually pay after adoption. The reliability research published in 2026 moves the axis of the decision elsewhere. It moves to three things: consistency across repeated runs, the path an error travels, and the cost of undoing a mistake.

Capability Scores Decoupled from Reliability

Compressing evaluation into a single success rate hides operational flaws. Rabanser and colleagues take up that problem in "Towards a Science of AI Agent Reliability" (arXiv:2602.16666, 2026-02). The work decomposes reliability into twelve metrics across four dimensions: consistency, robustness, predictability, and safety. Measuring 15 models on two benchmarks, the authors found that recent capability gains produced only small improvements in reliability.

The same divergence grows sharper on long tasks. "Beyond pass@1" by Khanal and colleagues (arXiv:2603.29231, 2026-03) measured 10 models over 396 tasks and 23,392 episodes. The Graceful Degradation Score, which captures how gently performance falls off, behaved differently by domain. Software engineering dropped from 0.90 to 0.44 as tasks grew longer, while document processing stayed nearly flat at 0.74 to 0.71.

The rest of that paper bears more directly on the adoption decision. Capability rankings and reliability rankings diverged, with several rank inversions at long horizons. Frontier models had the highest meltdown rates, up to 19%, because they attempt ambitious multi-step strategies that sometimes spiral. Memory scaffolds hurt long-horizon performance across all 10 models.

The measurement infrastructure sends the same signal. The Holistic Agent Leaderboard (arXiv:2510.11977, 2025-10) ran 21,730 rollouts across 9 models and 9 benchmarks at a cost of roughly $40,000. In that data, raising reasoning effort reduced accuracy in the majority of runs. The definitions of these metrics and the five evaluation axes appear in Multi-Agent System Evaluation and Orchestration Topology.

Error Propagation Along the Topology

What separates a multi-agent setup from a single one is that an error enters someone else's context. "From Spark to Fire" by Xie and colleagues (arXiv:2603.04474, 2026-03) models this propagation by treating collaboration as a directed dependency graph. Measuring six frameworks across three topologies, the authors identified three vulnerability classes: cascade amplification, topological sensitivity, and consensus inertia. The chain group held LangChain and MetaGPT, the mesh group AutoGen and CAMEL, and the star group CrewAI and LangGraph.

Topology decides the shape of the spread. In a chain, the error advances one stage at a time and produces a stepwise rise. In a star, the moment the hub adopts the error it broadcasts the falsehood to every worker at once, producing a sharp jump.

Where the error enters decides the blast radius. Planting the same error seed at a hub node drove infection to 100%, while planting it at a leaf node reached 15.9% on CrewAI and 9.7% on LangGraph. The ratio between those values, the impact factor, is 6.29 and 10.31 respectively. Hub placement is therefore a contamination question, not a performance question.

The Rising Cost of Late Intervention

Consensus inertia is an asymmetry along the time axis. Pushing one falsehood in is cheap, but reversing it grows more expensive as work proceeds. In artifact-centric workflows, intermediate outputs become the state of the project. An early error hardens into constraints such as sources, assumptions, code skeletons, and evaluation criteria, and later steps build on top of them.

A late correction therefore does not argue with a single sentence. It collides with an entire dependency chain that has already made itself internally consistent. The same paper counted the volume of polluted context accumulated before intervention as polluted rounds.

Delaying by two steps nearly triples the contamination that must be undone. In debate structures this inertia compounds with majority voting. The herding that appears when in-progress votes are revealed in real time is covered in Multi-Agent Debate.

The Price of Reliability

The same work inserted a genealogy-graph-based governance layer at the message layer to suppress the spread. The metric is the Benign Infection Control Rate, the complement of attack success rate, measuring how often the final artifact avoids infection. The numbers below come from a setting where error seeds were deliberately injected, so they should not be read as ordinary hallucination rates.

There are two ways to read the table. Self-reflection alone stays at 0.32, which is hard to call a defense, while the governance layer lifts it to 0.89 and above. The move from 0.32 to 0.94 costs 4.5 times the tokens and 2.4 times the latency.

The ablation says something more important. The variant that detected errors but removed blocking and rollback spent 34,991 tokens and still reached only 3.1% BICR. That is effectively the same as the 2.2% of doing nothing at all. A verifier that only issues judgments pays the full cost and fails to stop propagation.

The Single Agent as the First Verifier Target

Putting the evidence together produces an order of adoption. Anthropic, in "Building Effective Agents," recommends finding the simplest solution possible and increasing complexity only when needed. The same document states that the autonomous nature of agents means higher costs and the potential for compounding errors. My own judgment is that if the verification contract does not hold for a single agent, adding agents will not make it hold.

The condition for a verifier is that it lives outside the model. A model saying it finished is not evidence of success. Judgment has to come from independent signals such as test results, schema conformance, actual tool responses, database state, and human approval. As the ablation above shows, that verifier earns its cost only when it can quarantine and roll back, not merely rule.

Debugging itself gets harder once you move to multiple agents. Who&When (arXiv:2505.00212, 2025-05) measured automated failure attribution over failure logs from 127 multi-agent systems. The best method identified the responsible agent 53.5% of the time and the decisive step only 14.2% of the time. Adding agents while you can localize a break barely half the time raises the operational burden alongside it. The loop and harness side of this design is covered in Agent Harness and Loop Engineering.

Where This Argument Does Not Hold

Three situations shake this criterion. The first is a task whose essence is parallel search. When subtasks are genuinely independent, the gain survives the coordination cost. The same evaluation post covers that boundary with numbers.

The second is a domain where verification terminates deterministically. When a compiler, a static analyzer, or a schema validator settles the truth of an intermediate result, the channel for contamination to reach downstream narrows. The risk of error cascades drops sharply, and the arithmetic of multi-agent adoption is worth redoing.

The third is the maturing of defenses. HalluProp (arXiv:2607.26836, 2026-07) infers failure risk before an agent speaks. It measures intrinsic risk from semantic misalignment between role and query, then combines that with propagation risk. It reports an average AUROC of 84.6% for localizing faulty agents, at over 65 times the speed of post-hoc methods. This is a single paper, and I could not confirm independent replication.

The limits deserve a plain statement. The BICR figures above come from an adversarial setting over six frameworks and a small set of datasets. The reliability studies cited here also measured single agents for the most part.

The repeated-execution studies I checked as of 2026-09 share that limit. ReliabilityBench (arXiv:2601.06112) measures single ReAct and Reflexion agents, and the behavioral reproducibility study (arXiv:2605.28840) measures single tool-calling agents. No public work measures pass^k curves for multi-agent systems at the same scale, so the final link between the two axes is an estimate.

Questions to Answer Before Adopting

The decision compresses into five questions. Does the system run repeatedly without a human? Is there a single-agent baseline? Is the entry point for errors a hub?

Does the verifier quarantine and roll back, and is there budget for the price of reliability? If the second question has no answer, the rest have nothing to compare against.

Organized by condition, the same decision reads as follows.

The last row is the one that comes up most often in practice. Performance usually falls short because of the verification contract and the context, not because of the number of agents.

Summary

Adopting a multi-agent system is a decision about reliability rather than average scores. Capability and reliability move separately, and errors spread through a hub more than six times as widely as through a leaf. Intervening two steps late nearly triples the contamination to undo. Buying reliability costs 4.5 times the tokens and 2.4 times the latency, and even at that price a verifier that only rules cannot stop propagation.

The default is therefore a single agent with a verifier that can quarantine and roll back. Multi-agent is reserved for tasks built on parallel search, or for those where intermediate results settle deterministically. Building the single-agent baseline first is where the decision starts.