CRITIC: Verification Through External Tools

CRITIC verifies and corrects answers with external tools such as search, calculators, and code execution instead of relying on the model's own judgment.

The idea

CRITIC stands for Large Language Models Can Self-Correct with Tool-Interactive Critiquing. The technique verifies and improves an answer using external tools (code execution, search, calculators, and so on) rather than the model's own judgment.

How it works

1. Generate an initial answer -- the model produces a first pass at the question
2. Verify with an external tool -- search, a calculator, code execution, and similar confirm the answer
3. If correct, the answer is finalized; if wrong, the error is analyzed and the answer improved

Example

Question: "What was the population of South Korea in 2023?"

First attempt -- Model: "About 52 million."

CRITIC verification -- Action: [Search] "2023 South Korea population statistics" / Result: Statistics Korea figure of 51.45 million

Verification result: Wrong (off by 550,000)

Improvement -- Model: "According to Statistics Korea, about 51.45 million."

Math verification example

Question: "What is the square root of 15 to two decimal places?"

First attempt -- Model: "About 3.87."

CRITIC verification -- Action: [Calculate] sqrt(15) / Result: 3.872983...

Verification result: Correct (3.87 when rounded)

Final answer: "The square root of 15 is about 3.87."

How this differs from Self-Refine

Tools available for verification

Code verification example

# [Question] "Please write a factorial function"
 
# [First attempt]
def factorial(n):
    if n == 0:
        return 1
    return n * factorial(n)  # bug: n instead of n-1
 
# [CRITIC verification]
# Action: [Execute] factorial(5)
# Result: RecursionError (infinite recursion)
 
# [Verification result] Failed
 
# [Improvement]
def factorial(n):
    if n == 0:
        return 1
    return n * factorial(n - 1)  # fixed
 
# [Re-verification]
# Action: [Execute] factorial(5)
# Result: 120

Performance

The original paper (Gou et al. 2023, arXiv:2305.11738) reported the following gains on a ChatGPT base. F1 is the harmonic mean of precision and recall, and GSM8k is a grade-school-level math benchmark.

• +7.7 F1 points on average for free-form question answering (QA)
• +5.7 percentage points on math (GSM8k)
• Most effective on fact-based questions

The algorithm

critic_loop repeats verification and improvement up to max_iterations times. In each round the model decides how to verify. The code branches on the [Search], [Calculate], and [Execute] tags to run the tool, and the loop stops once the result agrees with the answer.

def critic_loop(question, max_iterations=3):
    # generate the initial answer
    answer = llm.generate(question)
 
    for i in range(max_iterations):
        # decide how to verify
        verification = llm.generate(
            f"Which tool should be used to verify the following answer?\n"
            f"Question: {question}\n"
            f"Answer: {answer}"
        )
 
        # run the tool
        if "[Search]" in verification:
            result = search(verification.query)
        elif "[Calculate]" in verification:
            result = calculate(verification.expression)
        elif "[Execute]" in verification:
            result = execute_code(verification.code)
        else:
            break
 
        # analyze the verification result
        is_correct = llm.generate(
            f"Does the verification result agree with the answer?\n"
            f"Answer: {answer}\n"
            f"Verification result: {result}"
        )
 
        if "agree" in is_correct:
            break
 
        # improve
        answer = llm.generate(
            f"Revise the answer based on the verification result.\n"
            f"Previous answer: {answer}\n"
            f"Verification result: {result}"
        )
 
    return answer

When to use CRITIC

Limitations

1. Tool dependency: unusable without an appropriate verification tool
2. Verification cost: external API calls cost money
3. Verification coverage: verifying every claim is hard

Combining with other techniques

Self-Refine + CRITIC

1. Self-Refine improves quality and readability
2. CRITIC verifies the facts with external tools
3. The final result is produced

Reflexion + CRITIC

1. Attempt and fail (a test fails)
2. CRITIC analyzes where the failure occurred
3. Reflexion reflects on why it failed
4. Retry

Related concepts

• Reflexion: learning from failure
• Self-Refine: improvement from self-feedback
• ReAct: a tool-use framework
• SelfCheckGPT (arXiv 2303.08896): a verification-family method that detects hallucination through the consistency of N samples

Summary

CRITIC catches errors on objective grounds by verifying answers with external tools instead of the model's own judgment. Because results are confirmed through search, calculators, and code execution, it works best where the correct answer is externally verifiable. Fact checking, math and computation, and writing code fit that condition. In areas with no suitable verification tool, such as creative writing or subjective opinion, a self-feedback method like Self-Refine is the better fit. Its limits are that a suitable verification tool must exist, external calls cost money, and verifying every claim one by one is difficult.