Chain of Thought (CoT) Step-by-Step Reasoning

"Let's think step by step"

The Idea

Chain of Thought (CoT) is a prompting technique that draws out an LLM's ability to solve complex problems by showing it the intermediate reasoning steps.

CoT comes in two forms. Wei et al. 2022 (Google) proposed few-shot CoT, which includes the reasoning process in the examples, and Kojima et al. 2022 showed zero-shot CoT, which adds only the single sentence 'Let's think step by step'. Every experimental result in this post refers to few-shot CoT (Wei et al.).

Core Mechanism

Standard prompting: given a problem, the model outputs the answer directly.
CoT prompting: given a problem, the model reaches the answer through step 1, step 2, and step 3.

Side-by-Side Example

Standard prompting

Q: What is 2+3? A: 5

Chain-of-thought prompting

Q: What is 2+3? First think of 2 items, then add 3 more, giving 5 in total. A: 5

Emergent Ability

CoT comes with a condition: it does not work on every model.

Model size	Effect of CoT
Small models	Almost no change in performance
Large models	Performance improves through step-by-step reasoning

The CoT effect is mostly observed in models above a certain scale. CoT from a small model produces output that is "fluent but not logical".

Experimental Results

All the numbers below are as reported by Wei et al. 2022 (arXiv:2201.11903) for PaLM 540B.

Arithmetic Reasoning (GSM8K)

Model	Standard prompting	CoT prompting
PaLM 540B	17.9%	56.9% (about 3x)

Commonsense Reasoning (StrategyQA)

Previous best: 69.4%
PaLM 540B + CoT: 75.6%

Sports Understanding

95.4% (above the compared baseline)

Ablation Study

Experiments run to verify "why does CoT work?":

Variant	Result	Meaning
Show equations only	No effect	Natural language explanation is required
Add meaningless tokens	No effect	It is not simply extra computation time
Reasoning after the answer	No effect	Order matters

Conclusion: making the actual reasoning process explicit is what matters.

Robustness Checks

Prompt Sensitivity

CoT prompts written independently by three authors -> all effective
Random samples drawn from the GSM8K training data -> works
Reordering the examples -> relatively little effect

Conclusion: what matters is making the reasoning explicit, not any particular style.

Out-of-Distribution Tests

Generalization

Last-letter concatenation: trained on 2 words -> solves 3- and 4-word problems
Coin flip: trained on 2 flips -> solves problems with more flips

Meaning: the model learns an abstract logical structure, not simple pattern matching.

Limits

Whether the network actually "thinks" is unresolved
Data cost when scaling up through fine-tuning
No guarantee of correct reasoning (hallucination is possible)
Limited benefit on small models or simple tasks

A Shift in Perspective

Before	After
How do we train the model to be smarter?	How do we use an already smart model better?

CoT moves attention from training methods to usage methods. Large models already hold the reasoning capability, and showing an example of the thinking process is enough to draw it out.

What Came Next

CoT -> ReAct: adds tool use to interact with the external environment.
CoT -> ToT: explores multiple paths to find the best reasoning path.
CoT -> Reflexion: adds self-reflection to learn from failure.

Summary

CoT is a prompting technique that draws out complex problem-solving ability by showing the model intermediate reasoning steps. The effect appears mainly in models above a certain scale, and the ablation study shows that making the actual reasoning explicit in natural language is the essential part. It does not guarantee correct reasoning, hallucination remains, and the benefit is limited on small models or simple tasks. CoT is the starting point for later techniques such as ReAct, ToT, and Reflexion.

Tree of Thought (ToT): expansion into multiple paths
ReAct: combines action
Reflexion: adds self-reflection