Multi-Agent Debate

Several agents rebut and verify each other, converging on an answer better than any single model produces.

Concept

Multi-Agent Debate is a technique in which several AI agents debate each other to solve a problem. Diverse perspectives and mutual verification lead to a better answer.

Structure

Agent A: "The answer is X, because..."

Agent B: "No, you overlooked... The answer is Y."

Agent A: "Good point. In that case, how do you read...?"

After several rounds, the final answer is decided by a vote or a mediator.

Why it works

Performance

These figures come from tables 1 and 2 of Du et al. 2023, 'Improving Factuality and Reasoning in Language Models through Multiagent Debate' (arXiv:2305.14325), comparing a single agent against three agents debating for two rounds. The same paper reports gains on Biographies (66.0% to 73.8%) and arithmetic (67.0% to 81.8%). Actual gains vary with the model, the benchmark, and the number of rounds.

Debate examples

A math problem

Problem: "Three dice are rolled. What is the probability that the sum is 10?"

Agent A (round 1): "Each die has 6 outcomes, so the total number of outcomes is $6^3 = 216$. The combinations summing to 10 are (1,3,6), (1,4,5), (2,2,6), (2,3,5), (2,4,4), (3,3,4), which is 6, so the probability is $6/216 = 1/36$."

Agent B (rebuttal): "Agent A counted only unordered combinations. The denominator 216 counts ordered outcomes, so the numerator has to be expanded too. (1,3,6), (1,4,5), and (2,3,5) each give $3! = 6$ arrangements, while (2,2,6), (2,4,4), and (3,3,4) have a repeated face and give $3!/2! = 3$ each."

Agent A (correction): "Correct. Expanded, that is $6 \times 3 + 3 \times 3 = 27$, so the probability is $27/216 = 1/8$."

Final answer: $1/8$ ✓

A logic problem

Problem: "All birds can fly. A penguin is a bird. Therefore a penguin can fly. Is this argument valid?"

Agent A: "It is valid in logical form. The conclusion follows from the major and minor premises."

Agent B (rebuttal): "The logical form is valid, but the major premise is false. 'All birds can fly' is not true. Penguins, ostriches, and kiwis are birds that cannot fly."

Agent A (correction): "Correct. Formal validity and the truth value of the premises have to be separated. This argument is formally valid but not sound, because the major premise is false."

Final answer: formally valid but not sound ✓

Organizational structures

Multi-agent systems can be organized in several ways:

Role-based debate

Assign each agent a different role:

Positive critic: "The strengths of this approach are..."

Negative critic: "But there are these problems..."

Synthesizer: "Combining the two perspectives..."

Example roles

The algorithm

The code below collects initial responses, iterates through rounds, checks for consensus, and makes a final decision. max_rounds caps the number of debate rounds and responses is a dictionary holding each agent's latest response. check_consensus ends the loop early when it detects agreement, and the final vote_or_mediate picks the answer using one of the strategies from the consensus table below.

def multi_agent_debate(question, agents, max_rounds=3):
    responses = {}
 
    # collect initial responses
    for agent in agents:
        responses[agent.id] = agent.generate(question)
 
    # debate rounds
    for round in range(max_rounds):
        for agent in agents:
            # look at the other agents' responses
            others_responses = {
                k: v for k, v in responses.items()
                if k != agent.id
            }
 
            # rebut or revise
            responses[agent.id] = agent.generate(
                f"Question: {question}\n"
                f"Other agents' responses: {others_responses}\n"
                f"Your previous response: {responses[agent.id]}\n"
                f"Rebut or revise."
            )
 
        # check for consensus
        if check_consensus(responses):
            break
 
    # decide the final answer (vote or mediation)
    return vote_or_mediate(responses)

Consensus methods

Limitations

1. Cost: multiple agent calls raise the bill
2. Time: several rounds are required
3. Failure to converge: consensus may never be reached
4. Homogeneity: identical models can share the same biases

MADKE

MADKE extends Multi-Agent Debate with a shared knowledge pool. Because every agent reads the same background knowledge, debate quality improves and information mismatches drop.

Summary

Multi-Agent Debate reduces the errors and hallucinations of a single model through a process where agents rebut and verify each other's answers. Diverse perspectives and mutual verification are what make it work, but the extra calls and rounds raise cost and latency, and identical models bring overlapping biases. It therefore fits tasks where verification is hard and the cost is worth paying, such as fact verification or complex reasoning. Extensions that reinforce the debate structure are worth evaluating alongside it to improve the quality of the consensus. MADKE, which aligns the factual base through a shared knowledge pool, is one such extension.

Related concepts

• MADKE: knowledge-enhanced debate
• LLM Agent Survey: collaboration structures
• LATS: debate plus tree search