Bolting on more agents does not make a system better. Evaluation requires measuring coordination cost, failure propagation, and topology together.
Coordination cost as a new dimension
A multi-agent system (MAS) is a structure in which several agents divide roles to solve one task. Carrying over single-agent benchmark scores misses the central cost of that structure. Collaboration adds new dimensions: coordination cost, message quality, inter-agent misalignment, and error propagation.
Team cost includes assignment, communication, and integration as well as each agent's inference. Compare against a single agent with the same tools, tasks, and budget to isolate these effects. Do not generalize one configuration's token multiplier or improvement to all multi-agent systems.
Splitting evaluation into five axes
Collapsing MAS evaluation into one score loses information. Each axis measures something different and the axes trade off against each other, so this post proposes keeping these five axes separate.
| Axis | What it measures | Representative metrics |
|---|---|---|
| Outcome | Final task success | Task Score, % Resolved, accuracy |
| Coordination and cost | Collaboration efficiency and resources | Coordination Score, token and dollar cost, coordination overhead |
| Process | Correctness of intermediate steps | Trajectory match, tool selection accuracy |
| Failure and reliability | Consistency and collapse | pass^k, failure mode taxonomy, failure attribution |
| Safety | Adversarial robustness | Attack Success Rate (ASR), Effective Robustness Score |
The reliability axis deserves separate emphasis. Introduced by tau-bench, pass^k measures the probability that all k attempts succeed. By contrast, pass@k measures whether at least one of k attempts succeeds. The two metrics present the same success rate very differently.
Single-attempt success rate 70%
pass@3 = 1 - (1 - 0.7)^3 = 0.973
pass^3 = 0.7^3 = 0.343This calculation assumes that every attempt has the same 70% success probability and that attempts are independent. Different per-task probabilities or dependence between runs invalidate applying this formula to an overall mean. Calculate operational metrics directly from repeated runs and inspect failed tasks and policy violations separately.
Classifying failure records
The following is this post's proposed debugging classification. It does not report measured category frequencies or repair effects.
| Category | What to check |
|---|---|
| Specification and assignment | Were goals, roles, and completion conditions communicated? |
| Communication and state | Was required information missing or stale? |
| Execution and verification | Were tool results and final state checked? |
Topology decides cost and performance
Topology is the connection structure through which agents exchange messages. It falls broadly into sequential, parallel, hierarchical, and hybrid forms. A central supervisor distributing work and a swarm where agents hand off directly to each other sit at the two extremes.
flowchart TD
subgraph SEQ[Sequential pipeline]
direction LR
A1[Retrieve<br/>collect passages] --> A2[Summarize<br/>extract key points] --> A3[Verify<br/>check against evidence]
end
subgraph SUP[Supervisor-centralized]
S0[Supervisor<br/>distribute and integrate] --> S1[Worker A]
S0 --> S2[Worker B]
S0 --> S3[Worker C]
end
subgraph SWA[Swarm mesh]
W1[Agent 1] <--> W2[Agent 2]
W2 <--> W3[Agent 3]
W1 <--> W3
end- Sequential pipeline: suits tasks with fixed stages, from retrieval to summarization to verification. Easy to debug, but with no parallelism.
- Supervisor-centralized (star or tree): the supervisor divides work and merges results. Accountability is clear, but the supervisor becomes the bottleneck.
- Swarm (graph or mesh): agents communicate directly with one another. Flexible, but messages explode and debugging is hard.
Comparing structures for a task
Topology changes message paths and opportunities for parallel work. The number of connections alone does not establish a performance advantage. Compare messages, total cost, completion time, and success on the same tasks.
If task difficulty selects a different structure, include that selection rule in the evaluation. Record its tokens and classification errors, then validate on tasks not used to choose the rule.
Evaluating the process, not just the outcome
An outcome score alone cannot see collaboration quality. Determining who did what and whether the intermediate steps were correct requires scoring the trajectory.
The MT-Bench and Chatbot Arena study of LLM judges analyzes position, verbosity, and self-preference biases. It evaluates conversational responses, so its judge accuracy cannot be transferred to entire multi-agent executions. For execution evaluation, this post proposes checking actual tool responses, tests, and goal state together.
Inspect cases where judge scores and final outcomes disagree. In addition to accuracy, check actions that should have been denied and reads outside the permitted scope.
Summary
Multi-agent evaluation must include both final outcomes and coordination cost. Separating outcome, cost, process, reliability, and safety helps expose failures hidden by an average. Compare topologies through repeated runs and matched-budget baselines, then check automatic judgments against execution evidence.