Test-Time Compute Scaling

An approach that raises performance by spending more compute at inference time instead of during training.

Concept

Test-time compute scaling improves performance by using more compute at inference time rather than during training.

The paradigm shift

• Old paradigm: bigger model + more data + more training = better performance
• New paradigm: same model + more thinking at inference = better performance

Resource allocation compared

• Old: 90% training, 10% inference, with most resources invested in training
• New: 50% training, 50% inference, with equal resources given to inference time

Snell et al. 2024

The canonical work that systematized this paradigm is Snell et al., "Scaling LLM Test-Time Compute Optimally" (arXiv 2408.03314, 2024). It scales inference compute along two axes.

1. Verifier-based search: score multiple reasoning paths with a process reward model (PRM) and use those scores for best-of-N or tree search
2. Adaptive revision of the response distribution: update the output distribution according to prompt difficulty

The central idea is compute-optimal allocation. Distributing inference compute according to problem difficulty raises efficiency by more than 4x over best-of-N in some settings. On problems where a small model already achieves a reasonable success rate, it can beat a 14x larger model at the same FLOPs.

Reward models: PRM vs ORM

The quality of verifier-based search depends on the reward model.

• Outcome reward model (ORM): grades only the final answer. Simple, but it misses errors in intermediate reasoning.
• Process reward model (PRM): grades every reasoning step. OpenAI's "Let's Verify Step by Step" (arXiv 2305.20050) trained a PRM on 800,000 step labels (PRM800K) and reached 78.2% at the same sample count, above the ORM's 72.4%.

s1

s1, published in January 2025, implements inference scaling with very little data (arXiv 2501.19393).

Training method

1. Data collection: just 1,000 hard problems
2. Fine-tuning: train on that data alone
3. Budget forcing: control thinking time by inserting 'Wait' and manipulating the end-of-thinking delimiter

Performance

s1-32B, produced by fine-tuning the base model (26.7%) on 1,000 examples, beats o1-preview on AIME 2024. The o1-preview (44.6%) and o1 (79.8%) figures in the table follow Table 1 of the s1 paper (arXiv 2501.19393).

Budget Forcing

The core technique in s1 is budget forcing.

How it works

It does not add dedicated special tokens. To make the model think more, it suppresses the end-of-thinking delimiter and appends the plain string "Wait", which keeps the reasoning going. To make it stop, it appends the delimiter followed by "Final Answer:" so the model commits to an answer.

• Extend thinking: suppress the delimiter + append "Wait" → induce further reasoning
• End thinking: delimiter + "Final Answer:" → emit the final answer

Example

Normal generation: "The answer is 42." - the answer is emitted immediately.

Budget forcing: "Hmm..." → Wait → "To solve this, first..." → Wait → Wait → "Therefore..." → Final Answer: → "42" - the answer comes after several rounds of deeper thinking.

Controlling thinking time

• Easy problem: Wait x 2 - a quick answer
• Medium problem: Wait x 10 - moderate thinking
• Hard problem: Wait x 50 - deep thinking

How compute relates to performance

Key findings

Dynamic compute adjustment

Adjusting by problem difficulty

Easy problem: "1 + 1 = ?" → Wait x 1 → "2" - answered immediately with minimal thinking

Hard problem: "an AIME problem" → Wait x 50 → answered after deep analysis

Confidence-based adjustment

s1 itself controls with a fixed budget, meaning a set number of Waits. The confidence-based scheme below is a more general variant of that idea: assess confidence after an initial round of thinking and adjust compute accordingly.

• High confidence: answer right away
• Low confidence: add a Wait and think more

Cost efficiency

Cost comparison

• Training a bigger model: thousands of GPUs x several months = $millions
• Increasing inference time: extra token cost x only when needed = $thousands

Implementation

Fixed repetition

wait_tokens is the number of times "Wait" is appended to extend thinking. After repeating that fixed budget, "Final Answer:" forces an answer.

def think_longer(model, question, wait_tokens=10):
    prompt = question
    for _ in range(wait_tokens):
        prompt += model.generate(prompt, max_tokens=50)
        prompt += "Wait"
    prompt += "Final Answer:"
    return model.generate(prompt)

Adaptive adjustment

max_waits is the upper bound on thinking extensions, and confidence > 0.9 is the threshold for deciding that no further thinking is needed. Once confidence passes the threshold, the loop stops before reaching the bound.

def adaptive_thinking(model, question, max_waits=50):
    prompt = question
    for i in range(max_waits):
        response = model.generate(prompt, max_tokens=50)
 
        # confidence check
        confidence = model.evaluate_confidence(response)
        if confidence > 0.9:
            break
 
        prompt += response + "Wait"
 
    prompt += "Final Answer:"
    return model.generate(prompt)

Best-of-N Sampling

Best-of-N sampling solves the same problem N times and picks the best answer.

Solve the same problem N times:

Attempt 1: answer A / Attempt 2: answer B / Attempt 3: answer A / Attempt 4: answer A / Attempt 5: answer C

Select the best one or take a majority vote → answer A

Selection criteria

Limits

1. Saturation point: thinking forever does not improve results forever
2. Rising cost: inference cost is still cost
3. Latency: users wait longer
4. Problem type: not effective on every problem

Where it fits

Summary

Test-time compute scaling raises the performance of an unchanged model by spending more compute at inference. Snell et al. showed that compute-optimal allocation matched to problem difficulty is more efficient than best-of-N. The budget forcing in s1 controls thinking time using nothing more than "Wait" insertion and delimiter manipulation, while best-of-N sampling picks the best of several responses.

Gains saturate past a certain amount of inference compute, and the extra tokens bring cost and latency with them. The approach suits math and coding problems where accuracy matters, and does not suit work such as real-time conversation where speed matters.

Related concepts

• OpenAI o1: internal reasoning tokens
• DeepSeek-R1: open-source reasoning model
• Budget forcing: technique for controlling thinking time