OpenAI o1 and DeepSeek-R1 Reasoning Models

The shift toward training models to think longer before answering, compared across two branches: o1 and DeepSeek-R1.

What reasoning models changed

A reasoning model does not answer immediately. It works through several internal steps and then emits only the conclusion. Earlier language models built an answer by appending tokens in one pass. On hard math and science problems that approach skipped the intermediate reasoning and got things wrong often.

OpenAI o1 split that intermediate reasoning into an internal stage called reasoning tokens. The user sees only the final answer while the thinking unfolds inside the model. OpenAI's o1 announcement in September 2024 was the first public description of this structure.

Placing the output path of an earlier model next to o1 gives the following picture.

Thinking at length is not the goal in itself. o1 learned which lines of thought lead to correct answers, and the mechanism behind that learning is reinforcement learning.

OpenAI o1

Training o1 rests on reinforcement learning (RL), a method that rewards the outcome of an action and adjusts the policy toward higher reward. When the answer can be graded automatically, as with math problems, the reward can be assigned without human input.

The training loop repeats the following four steps millions of times.

• Rewards are not assigned by humans; they are computed automatically from whether the answer is right.
• What gets learned is not the answer itself but the distribution over reasoning paths that lead to the answer.
• It works particularly well in domains with unambiguous grading, such as math and coding.

One peculiar property falls out of this. Attaching a thinking-inducing prompt such as 'Let's think step by step' to o1 actually degrades performance. The model has already learned its optimal reasoning procedure, and the external instruction interferes with it. That is a signal that prompting techniques which worked on earlier models can backfire on reasoning models.

DeepSeek-R1

DeepSeek-R1 is an open-source model that reached reasoning performance comparable to o1. DeepSeek published the technical report and the weights together in January 2025. Unlike the closed o1, both the training method and the model itself can be inspected directly.

The most striking result is DeepSeek-R1-Zero. The usual recipe is supervised fine-tuning (SFT) on correct-answer examples first, then RL to refine. R1-Zero skipped the SFT stage entirely and trained with pure RL.

Without any correct-answer examples, behaviors such as thinking at length, checking its own work, and revisiting a wrong step appeared spontaneously during training (emergence). What matters is that reasoning ability was not written directly into the data; it surfaced as a byproduct of reward optimization.

How GRPO differs from PPO

The algorithm underneath this training is GRPO (Group Relative Policy Optimization). GRPO generates several answers to the same question and adjusts the policy using relative scores within that group. The comparison point, PPO (Proximal Policy Optimization), requires training a separate critic network that estimates the value of each answer. That network consumes additional compute and memory.

GRPO drops the critic network and uses the group mean as the baseline.

5 answers generated for the same question, scores: +5, 0, 0, +5, +3
group mean = 2.6
 
relative advantage = each score - mean
  answer 1: +5 - 2.6 = +2.4   (better than average -> reinforce)
  answer 2:  0 - 2.6 = -2.6   (worse than average -> suppress)

The policy is pushed so that better-than-average answers appear more often and worse ones less often. Because learning needs only group comparison and no separate value estimate, the same budget buys more reasoning attempts. GRPO was introduced in DeepSeek's earlier DeepSeekMath work.

Spending more compute at inference time

The paradigm running through both o1 and R1 is test-time compute scaling. Instead of pouring resources into training to raise performance, the weight moves toward making the model think longer at inference.

The idea reproduces without a massive RL run. The s1 method released in January 2025 (Muennighoff et al.) is one example. It fine-tunes on only 1,000 hard problems and adds budget forcing, which controls thinking length in tokens.

Budget forcing extends or cuts off thinking time using special tokens.

A 32B model fine-tuned with s1 reported 81% on AIME 2024 (s1 paper, 2025-01). Substantial scores are reachable through inference-time compute alone, without large datasets or complicated RL.

Thinking without limit does not keep helping, however. Past a certain point, extra thinking barely moves the score, and a saturation point arrives. That is why a mechanism for budgeting thinking, such as budget forcing, is needed alongside it.

Performance and limits

The representative benchmarks are AIME 2024 and GPQA Diamond. AIME 2024 is the American Invitational Mathematics Examination. GPQA Diamond covers graduate-level Google-proof question answering in the sciences. Exact numbers shift with the sampling method, so the values below are the representative figures each official announcement reported.

The R1-Zero row illustrates the effect of inference-time compute well. Taking a single answer gives 71%, while sampling 16 and choosing by majority vote raises it to 86.7% (DeepSeek-R1 technical report, 2025-01). The same model scores differently once more resources go into inference.

The limits are equally clear.

• Cost: reasoning tokens make responses slower and increase token charges, which is excessive for easy tasks.
• Opacity: o1 does not expose its thinking, so there is no external way to verify why it produced a given answer.
• Saturation: scores stop improving past a point, so compute cannot be poured in indefinitely.

The selection criterion is straightforward. Reasoning models help on math, coding, and complex reasoning where grading is unambiguous. Ordinary models are faster and cheaper for short summaries or simple transformations. A model with open weights such as R1 has an advantage in research and on-premises settings where the thinking process has to be inspected directly.

Summary

o1 and DeepSeek-R1 solve the same goal in different ways. Both make the model think longer before answering and refine that thinking with reinforcement learning against gradeable rewards. o1 is a closed product that hides its reasoning tokens, while R1 publishes weights and method and trains with GRPO without a critic network.

The keyword linking the two is test-time compute scaling. The weight shifted from growing training to spending more compute at inference, and s1 with budget forcing showed that the effect reproduces at small scale. Cost, opacity, and the saturation point remain, so reasoning models are not the right answer for every task.

In practice, choosing a model by how gradeable and how hard the task is works well. Use a reasoning model on hard problems with a clear right and wrong answer, an ordinary model elsewhere, and weigh cost and latency alongside accuracy.