CRITIC

CRITIC uses search, calculator, and code execution results to verify and correct answers, without relying only on the model's own judgment.

The idea

CRITIC stands for Large Language Models Can Self-Correct with Tool-Interactive Critiquing. The model interprets search, code execution, and calculator results to verify and improve its answer.

flowchart LR
    G["Initial answer"] --> P["Select claim to verify"]
    P --> T["Run external tool"]
    T --> D{"Matches evidence"}
    D -->|yes| F["Final answer"]
    D -->|no| C["Correct error"]
    C --> P

How it works

1. Generate an initial answer: the model produces a first pass at the question
2. Verify with an external tool: search, a calculator, code execution, and similar confirm the answer
3. If correct, the answer is finalized; if wrong, the error is analyzed and the answer improved

The examples below illustrate the procedure; they are not evaluation results or execution records from the paper.

Math verification example

Question: "What is the square root of 15 to two decimal places?"

First attempt. Model: "About 3.87."

CRITIC verification. Action: [Calculate] sqrt(15) / Result: 3.872983...

Verification result: Correct (3.87 when rounded)

Final answer: "The square root of 15 is about 3.87."

How this differs from Self-Refine

Tools available for verification

Code verification example

# [Question] "Please write a factorial function"
 
# [First attempt]
def factorial(n):
    if n == 0:
        return 1
    return n * factorial(n)  # bug: n instead of n-1
 
# [CRITIC verification]
# Action: [Execute] factorial(5)
# Result: RecursionError (infinite recursion)
 
# [Verification result] Failed
 
# [Improvement]
def factorial(n):
    if n == 0:
        return 1
    return n * factorial(n - 1)  # fixed
 
# [Re-verification]
# Action: [Execute] factorial(5)
# Result: 120

Performance

The ICLR 2024 paper by Gou et al. (arXiv:2305.11738) evaluates tool-assisted revision on question answering, math, and toxic-output mitigation. Improvements must be checked for the model, tool, and task being evaluated.

The algorithm

The following code is unexecuted illustrative pseudocode, not a reproduction of the paper. A real implementation needs structured verification results and tool-error handling.

critic_loop repeats verification and improvement up to max_iterations times. In each round the model decides how to verify. The code uses the [Search], [Calculate], and [Execute] tags to select and run a tool. The loop stops once the result agrees with the answer.

def critic_loop(question, max_iterations=3):
    # generate the initial answer
    answer = llm.generate(question)
 
    for i in range(max_iterations):
        # decide how to verify
        verification = llm.generate(
            f"Which tool should be used to verify the following answer?\n"
            f"Question: {question}\n"
            f"Answer: {answer}"
        )
 
        # run the tool
        if "[Search]" in verification:
            result = search(verification.query)
        elif "[Calculate]" in verification:
            result = calculate(verification.expression)
        elif "[Execute]" in verification:
            result = execute_code(verification.code)
        else:
            break
 
        # analyze the verification result
        is_correct = llm.generate(
            f"Does the verification result agree with the answer?\n"
            f"Answer: {answer}\n"
            f"Verification result: {result}"
        )
 
        if "agree" in is_correct:
            break
 
        # improve
        answer = llm.generate(
            f"Revise the answer based on the verification result.\n"
            f"Previous answer: {answer}\n"
            f"Verification result: {result}"
        )
 
    return answer

When to use CRITIC

Limitations

1. Tool dependency: unusable without an appropriate verification tool
2. Verification cost: external API calls cost money
3. Verification coverage: verifying every claim is hard

Combining with other techniques

Self-Refine + CRITIC

1. Self-Refine improves quality and readability
2. CRITIC verifies the facts with external tools
3. The final result is produced

Reflexion + CRITIC

1. Attempt and fail (a test fails)
2. CRITIC analyzes where the failure occurred
3. Reflexion reflects on why it failed
4. Retry

Related concepts

• Reflexion: learning from failure
• Self-Refine: improvement from self-feedback
• ReAct: a tool-use framework
• SelfCheckGPT (EMNLP 2023, arXiv 2303.08896): a verification-family method that detects hallucination through the consistency of N samples

Summary

CRITIC attempts to revise an answer through model interpretation of external tool results. Both the results and their interpretation can be wrong, so accuracy is not automatic. Because results are confirmed through search, calculators, and code execution, it works best where the correct answer is externally verifiable. Fact checking, math and computation, and writing code fit that condition. In areas with no suitable verification tool, such as creative writing or subjective opinion, a self-feedback method like Self-Refine is the better fit.

CRITIC requires a suitable verification tool, and external calls cost money. Verifying every claim individually is also difficult.