Multi-Agent Debate

Several agents revise their answers after consulting each other's responses.

Concept

Multi-Agent Debate is a technique in which several AI agents debate each other to solve a problem. Agents revise their answers after consulting other responses.

flowchart LR
    Q["Problem"] --> A["Agent A answer"]
    Q --> B["Agent B answer"]
    A --> X["Mutual critique and evidence check"]
    B --> X
    X --> J["Vote or mediation"]
    J --> F["Final answer"]

Structure

Agent A: "The answer is X, because..."

Agent B: "No, you overlooked... The answer is Y."

Agent A: "Good point. In that case, how do you read...?"

After several rounds, the final answer is decided by a vote or a mediator.

Intended effects and conditions

Other responses provide opportunities to identify omissions and errors. However, the same model can make similar mistakes, and a correct answer can become wrong during debate. Agreement rate alone does not measure answer accuracy.

Performance

The ICML 2024 paper by Du et al. (arXiv:2305.14325) evaluates debate on mathematical and strategic reasoning and factuality tasks. Results apply to the evaluated models, tasks, and debate budgets. To isolate the value of debate, also compare against generating multiple responses without debate and voting at the same cost.

The conversations below illustrate the procedure; they are not execution records from the paper.

Debate examples

A math problem

Problem: "Three dice are rolled. What is the probability that the sum is 10?"

Agent A (round 1): "Each die has 6 outcomes, so the total number of outcomes is (6^3 = 216). The combinations summing to 10 are (1,3,6), (1,4,5), (2,2,6), (2,3,5), (2,4,4), (3,3,4), which is 6, so the probability is (6/216 = 1/36)."

Agent B (rebuttal): "Agent A counted only unordered combinations. The denominator 216 counts ordered outcomes, so the numerator has to be expanded too. (1,3,6), (1,4,5), and (2,3,5) each give (3! = 6) arrangements, while (2,2,6), (2,4,4), and (3,3,4) have a repeated face and give (3!/2! = 3) each."

Agent A (correction): "Correct. Expanded, that is (6 \times 3 + 3 \times 3 = 27), so the probability is (27/216 = 1/8)."

Final answer: (1/8) ✓

A logic problem

Problem: "All birds can fly. A penguin is a bird. Therefore a penguin can fly. Is this argument valid?"

Agent A: "It is valid in logical form. The conclusion follows from the major and minor premises."

Agent B (rebuttal): "The logical form is valid, but the major premise is false. 'All birds can fly' is not true. Penguins, ostriches, and kiwis are birds that cannot fly."

Agent A (correction): "Correct. Formal validity and the truth value of the premises have to be separated. This argument is formally valid but not sound, because the major premise is false."

Final answer: formally valid but not sound ✓

Organizational structures

Multi-agent systems can be organized in several ways:

Role-based debate

Assign each agent a different role:

Positive critic: "The strengths of this approach are..."

Negative critic: "But there are these problems..."

Synthesizer: "Combining the two perspectives..."

Example roles

The algorithm

The following is unexecuted pseudocode, not a reproduction of the paper. It collects initial responses, iterates through rounds, checks for consensus, and makes a final decision. max_rounds caps the number of debate rounds and responses is a dictionary holding each agent's latest response. check_consensus ends the loop early when it detects agreement. The final vote_or_mediate selects an answer using a strategy from the consensus table below.

def multi_agent_debate(question, agents, max_rounds=3):
    responses = {}
 
    # collect initial responses
    for agent in agents:
        responses[agent.id] = agent.generate(question)
 
    # debate rounds
    for round in range(max_rounds):
        for agent in agents:
            # look at the other agents' responses
            others_responses = {
                k: v for k, v in responses.items()
                if k != agent.id
            }
 
            # rebut or revise
            responses[agent.id] = agent.generate(
                f"Question: {question}\n"
                f"Other agents' responses: {others_responses}\n"
                f"Your previous response: {responses[agent.id]}\n"
                f"Rebut or revise."
            )
 
        # check for consensus
        if check_consensus(responses):
            break
 
    # decide the final answer (vote or mediation)
    return vote_or_mediate(responses)

Consensus methods

Limitations

1. Cost: multiple agent calls raise the bill
2. Time: several rounds are required
3. Failure to converge: consensus may never be reached
4. Homogeneity: identical models can share the same biases

Sharing retrieved evidence

A design can provide debating agents with documents that include sources. If that evidence is wrong, several agents can repeat the same error, so agreement and fact verification must be distinguished. This is a design proposal in this article, not a performance claim for a specific extension paper.

Summary

Multi-Agent Debate generates several responses and revises them through discussion. The original paper reports improvements on specific tasks, but agreement does not guarantee accuracy. Compare against a non-debate baseline with the same compute budget and measure both correct-to-wrong changes and cost.