Evaluating Multi-Agent Systems and Choosing an Orchestration Topology
Multi-agent systems add a new dimension: coordination cost. Five evaluation axes, a failure taxonomy, and topology selection in one pass.
Contents
Bolting on more agents does not make a system better. The judgment only holds up once you measure coordination cost, failure propagation, and topology together.
Coordination cost as a new dimension
A multi-agent system (MAS) is a structure in which several agents divide roles to solve one task. Carrying over single-agent benchmark scores misses the central cost of that structure. Collaboration adds new dimensions: coordination cost, message quality, inter-agent misalignment, and error propagation.
Cost comes first, and it is large. Anthropic reported that a MAS consumes roughly 15 times the tokens of ordinary chat, and a single agent roughly 4 times.
More tokens might be expected to buy accuracy, but that is not necessarily so. One scaling study (arXiv:2512.08296) measured the average MAS improvement across the board at -3.5%. On tasks where single-agent accuracy already exceeded about 45%, adding agents produced a negative return.
Errors amplify as well. The same study reported error amplification of 4.4x for centralized designs and 17.2x for independent ones, with coordination overhead in the 58-515% range. A MAS therefore pays off only on tasks where a single agent clearly stalls. Anthropic's research system, where parallel exploration is intrinsic to the task, leading a single agent by 90.2% marks that boundary.
Splitting evaluation into five axes
Collapsing MAS evaluation into one score loses information. Each axis measures something different and the axes trade off against each other, so keeping them separate is more accurate.
| Axis | What it measures | Representative metrics |
|---|---|---|
| Outcome | Final task success | Task Score, % Resolved, accuracy |
| Coordination and cost | Collaboration efficiency and resources | Coordination Score, token and dollar cost, coordination overhead |
| Process | Correctness of intermediate steps | Trajectory match, tool selection accuracy |
| Failure and reliability | Consistency and collapse | pass^k, failure mode taxonomy, failure attribution |
| Safety | Adversarial robustness | Attack Success Rate (ASR), Effective Robustness Score |
The reliability axis deserves separate emphasis. pass^k, introduced by tau-bench, measures the probability that all k attempts succeed, whereas pass@k measures the probability that at least one of k attempts succeeds. The two metrics present the same success rate very differently.
Single-attempt success rate 70%
pass@3 = 1 - (1 - 0.7)^3 = 0.973
pass^3 = 0.7^3 = 0.343The same 70% success rate looks like 97% under pass@3 and 34% under pass^3. When a human intervenes on every run, pass@k is the honest metric; for an unattended automation pipeline, pass^k is. In practice GPT-4o scores about 61% on pass^1 in tau-bench retail but drops below 25% on pass^8.
Three categories failures scatter across
Reducing failures starts with knowing where things break. MAST (arXiv:2503.13657) is a failure taxonomy built specifically for multi-agent systems. It annotates 1,642 execution traces from 7 frameworks and defines 14 failure modes across 3 categories.
| Category | Share | Description |
|---|---|---|
| Specification and design flaws | about 44% | Task and role specification violations, failure to recognize termination conditions |
| Inter-agent misalignment | about 32-37% | Coordination and communication failures, mismatch between reasoning and action |
| Weak verification | about 21-24% | Verification missing or incorrect |
The distribution says something clear. Failures are spread fairly evenly across three categories, so fixing one place does not solve the problem. Within inter-agent misalignment, the mode where reasoning and action diverge was the single largest at about 13%.
Targeted interventions had modest effects too. In the MAST analysis, restructuring orchestration gave ChatDev +9.4% and adding a verification step +15.6%. That is evidence that redesigning the structure matters more than patching parts of it. In the same analysis, 7 state-of-the-art frameworks failed on 41% to 86.7% of tasks.
Topology decides cost and performance
Topology is the connection structure through which agents exchange messages. It falls broadly into sequential, parallel, hierarchical, and hybrid forms. A central supervisor distributing work and a swarm where agents hand off directly to each other sit at the two extremes.
- Sequential pipeline: suits tasks with fixed stages, from retrieval to summarization to verification. Easy to debug, but with no parallelism.
- Supervisor-centralized (star or tree): the supervisor divides work and merges results. Accountability is clear, but the supervisor becomes the bottleneck.
- Swarm (graph or mesh): agents communicate directly with one another. Flexible, but messages explode and debugging is hard.
The topology choice decides cost and performance at the same time. In MultiAgentBench (arXiv:2503.01935), the tree topology consumed the most tokens and performed the worst, while graph and star did well. That is a signal that message volume and performance are not proportional.
AgentPrune tested this intuition. Pruning redundant messages reproduced the best topology's performance at roughly 5.6 dollars per correct answer. Reaching the same performance without pruning cost about 43.7 dollars, they reported. In other words, a substantial fraction of the messages flying around are surplus.
Matching topology to the task
There is no need to solve every task with one fixed topology. Adaptive orchestration, choosing the structure to fit the task, is the recent direction.
AdaptOrch (arXiv:2602.16873) argues that in the regime where model performance converges, topology selection makes a larger difference than the model does. A router analyzes the task decomposition graph and picks a suitable topology. The paper reports 12-23% gains over single-topology baselines on coding, reasoning, and Retrieval-Augmented Generation (RAG).
DAAO (arXiv:2509.11079) adjusts workflow complexity according to query difficulty. Easy queries are handled by a single agent in one pass, and only hard queries get a multi-stage reasoning chain. Difficulty is estimated with a variational autoencoder, and the estimate is updated with success and failure feedback.
Collaboration structures also come with traps. 'Beyond the Strongest LLM' (arXiv:2509.23537) points out one risk: revealing in-progress vote tallies in real time produces herding, pushing agents toward premature consensus. The timing and scope of information sharing has to be decided at design time.
Evaluating the process, not just the outcome
An outcome score alone cannot see collaboration quality. Determining who did what and whether the intermediate steps were correct requires scoring the trajectory.
Agent-as-a-Judge (arXiv:2410.10934) evaluates the entire execution process. On DevAI tasks it agreed with human consensus 90.4% of the time (black-box) and 92.1% (gray-box), ahead of a single LLM judge at 60.4% and 70.8%. It reportedly cut time and cost by about 97% relative to human evaluation.
Using an LLM judge as is introduces position, verbosity, and self-preference bias. Scoring only the final output misses coordination failures and contribution attribution. That is why process-aware evaluation or a calibrated panel of judges is necessary.
Bringing cost into the picture sharpens the value of a topology. In one financial document processing benchmark (arXiv:2603.22651), a hierarchical supervisor-worker structure reached F1 0.921 at 1.4x cost. That put it in the lead on the cost-performance frontier. A more expensive reflective structure produced F1 0.943 at 2.3x cost.
Summary
A multi-agent system uses roughly 15 times the tokens of a single agent and amplifies errors, so it pays off only on tasks where a single agent clearly stalls. Evaluation should be split into five axes, outcome, coordination cost, process, reliability, and safety, and unattended pipelines should track consistency with pass^k. Failures scatter across specification, misalignment, and verification, so fixing one place is not enough and topology selection has to go hand in hand with structural redesign. The more model performance converges, the more the choice of topology and the method of scoring decide whether the system succeeds.