Chain of Thought (CoT) Step-by-Step Reasoning
How CoT prompts intermediate steps, how its effects depend on model scale and task, and what the ablation studies establish
Chain of Thought prompts a model to produce intermediate steps before its final answer.
The idea
Chain of Thought (CoT) prompts a large language model (LLM) to generate intermediate reasoning before the final answer. Wei et al. (2022) proposed few-shot CoT with worked examples, while Kojima et al. (2022) showed zero-shot CoT using only "Let's think step by step."
| Method | Prompt content | Representative setting |
|---|---|---|
| Standard prompting | Problem and final answer | Direct-answer baseline |
| Few-shot CoT | Problem, intermediate reasoning, and final answer | Main experiments by Wei et al. |
| Zero-shot CoT | A short instruction to reason step by step | Experiments by Kojima et al. |
In a multi-step problem, each partial result becomes input to the next step. If a cafeteria has 18 tables with six trays each and 37 trays are returned, the model must compute 108 and then subtract 37 to reach 71.
The effect of scale
In the experiments by Wei et al., the benefit became clearer as model scale increased. Smaller models sometimes produced fluent steps that did not follow logically, so CoT is not a guarantee that applies to every model.
| Model condition | Observed tendency | Scope of interpretation |
|---|---|---|
| Smaller models | Little gain or unstable reasoning quality | Generating steps does not create missing capability |
| Larger models | Better results on complex arithmetic and commonsense reasoning | Limited to the evaluated models and tasks |
Experiments and ablations
Wei et al. (arXiv:2201.11903) reported that PaLM 540B rose from 17.9% with standard prompting to 56.9% with CoT on GSM8K. It reached 75.6% on StrategyQA, but these numbers belong to that model and its few-shot example setting.
| Task | Standard prompting | Few-shot CoT | Source scope |
|---|---|---|---|
| GSM8K | 17.9% | 56.9% | PaLM 540B, Wei et al. 2022 |
| StrategyQA | Paper's comparison baseline | 75.6% | PaLM 540B, Wei et al. 2022 |
The ablations separate the role of natural-language reasoning by task. Equations alone produced a smaller gain on GSM8K than natural-language rationales, but they still improved over standard prompting on one-step and two-step arithmetic datasets.
| Variant | Observation | Interpretation |
|---|---|---|
| Equations only | Small GSM8K gain, improvement on short arithmetic tasks | The need for natural language depends on task complexity |
| Meaningless extra tokens | Did not reproduce the CoT gain | Extra output length alone does not explain the result |
| Reasoning after the answer | Weaker than placing reasoning before the answer | Intermediate steps must precede answer generation |
Robustness and generalization
Examples written independently by the authors and examples sampled from GSM8K also improved performance. Last-letter concatenation and coin-flip tasks generalized to longer inputs, though those results cover the synthetic tasks selected by the paper.
Limits and later techniques
A CoT output is not a proof that the answer is correct. A model can produce fluent but invalid intermediate steps, and long reasoning adds token cost and latency.
| Limit | Practical response |
|---|---|
| Limited gain on small models and simple tasks | Compare the baseline and CoT on the same evaluation set |
| Intermediate steps may be false | Verify with a calculator, search, or code execution |
| Longer reasoning increases cost and latency | Cap the reasoning budget by task difficulty |
Later techniques address different weaknesses. ReAct adds tool actions, Tree of Thought searches multiple paths, and Reflexion records lessons from failed attempts.
Related concepts
- Tree of Thought (ToT): expands and evaluates several reasoning paths.
- ReAct: inserts external tool calls between reasoning steps.
- Reflexion: uses lessons from failure in the next attempt.
Summary
CoT generates intermediate reasoning so a model can connect partial results in a multi-step problem. Results from Wei et al. showed gains on complex reasoning tasks with large models, while the equations-only ablation varied with task length. Model scale and problem difficulty therefore need evaluation against a direct-answer baseline. Generated reasoning can still be wrong, so an external tool or test should verify the final answer.