Test-Time Compute Scaling

An approach that raises performance by spending more compute at inference time instead of during training.

Concept

Test-time compute scaling improves performance by using more compute at inference time rather than during training.

flowchart LR
    P["Problem"] --> G["Generate reasoning paths"]
    G --> V["Verifier scoring"]
    V --> B["Search within budget"]
    B --> A["Select highest-scoring answer"]

Measure training and inference compute separately. More compute does not guarantee better accuracy.

Snell et al. 2024

Snell et al. study inference-compute allocation in their ICLR 2025 paper (arXiv 2408.03314). It scales inference compute along two axes.

1. Verifier-based search: score multiple reasoning paths with a process reward model (PRM) and use those scores for best-of-N or tree search
2. Adaptive revision of the response distribution: update the output distribution according to prompt difficulty

The paper allocates search and sequential revision according to problem difficulty and compute budget. Its results are specific to the evaluated math problems and model conditions; they do not show that any small model can replace a larger one.

PRM vs ORM

The quality of verifier-based search depends on the reward model.

• Outcome reward model (ORM): grades only the final answer. Simple, but it misses errors in intermediate reasoning.
• Process reward model (PRM): grades every reasoning step. Let's Verify Step by Step (ICLR 2024, arXiv 2305.20050) compares process and outcome supervision on MATH. This comparison does not guarantee that PRMs outperform ORMs for all verifiers and tasks.

s1

s1 was released in early 2025 and published at EMNLP 2025 (arXiv 2501.19393). The figures below use Table 1 from the initial v1 paper.

Training method

1. Data collection: just 1,000 hard problems
2. Fine-tuning: train on that data alone
3. Budget forcing: control thinking time by inserting 'Wait' and manipulating the end-of-thinking delimiter

Performance

The 56.7% result for s1-32B includes supervised fine-tuning and budget forcing. Table 1 in v1 reports 50.0% without budget forcing. Training and inference conditions differ across the listed models, so this comparison cannot isolate the effect of inference compute.

Budget Forcing

The core technique in s1 is budget forcing.

How it works

It does not add dedicated special tokens. To make the model think more, it suppresses the end-of-thinking delimiter and appends the plain string "Wait", which keeps the reasoning going. To make it stop, it appends the delimiter followed by "Final Answer:" so the model commits to an answer.

• Extend thinking: suppress the delimiter + append "Wait" → induce further reasoning
• End thinking: delimiter + "Final Answer:" → emit the final answer

Example

Normal generation: "The answer is 42." (the answer is emitted immediately)

Budget forcing: "Hmm..." → Wait → "To solve this, first..." → Wait → Wait → "Therefore..." → Final Answer: → "42" (the answer comes after several rounds of deeper thinking)

The number of Waits and token limit are settings to select with evaluation data. No universal fixed counts for easy and hard problems are provided here.

Dynamic compute adjustment

Allocating budgets by difficulty requires separate evaluation.

Confidence-based adjustment

s1 itself controls with a fixed budget, meaning a set number of Waits. The confidence-based scheme below is a more general variant. It assesses confidence after an initial round of thinking and adjusts compute accordingly.

• High confidence: answer right away
• Low confidence: add a Wait and think more

Cost efficiency

Additional reasoning costs include generated tokens, verifier calls, and latency. A comparison with training cost must use a common operating period and request volume. No fixed monetary cost or improvement ratio applies generally.

Implementation

The code below is unexecuted pseudocode. It illustrates iteration budgets rather than reproducing the actual s1 delimiter control or a model API.

Fixed repetition

wait_tokens is the number of times "Wait" is appended to extend thinking. After repeating that fixed budget, "Final Answer:" forces an answer.

def think_longer(model, question, wait_tokens=10):
    prompt = question
    for _ in range(wait_tokens):
        prompt += model.generate(prompt, max_tokens=50)
        prompt += "Wait"
    prompt += "Final Answer:"
    return model.generate(prompt)

Adaptive adjustment

max_waits is the upper bound on thinking extensions, and confidence > 0.9 is the threshold for deciding that no further thinking is needed. Once confidence passes the threshold, the loop stops before reaching the bound.

def adaptive_thinking(model, question, max_waits=50):
    prompt = question
    for i in range(max_waits):
        response = model.generate(prompt, max_tokens=50)
 
        # confidence check
        confidence = model.evaluate_confidence(response)
        if confidence > 0.9:
            break
 
        prompt += response + "Wait"
 
    prompt += "Final Answer:"
    return model.generate(prompt)

Best-of-N Sampling

Best-of-N sampling solves the same problem N times and picks the best answer.

Solve the same problem N times:

Attempt 1: answer A / Attempt 2: answer B / Attempt 3: answer A / Attempt 4: answer A / Attempt 5: answer C

Select the best one or take a majority vote → answer A

Selection criteria

Limits

1. Saturation point: thinking forever does not improve results forever
2. Rising cost: inference cost is still cost
3. Latency: users wait longer
4. Problem type: not effective on every problem

Where it fits

Related concepts

• OpenAI o1: internal reasoning tokens
• DeepSeek-R1: open-source reasoning model
• Budget forcing: technique for controlling thinking time

Summary

Test-time compute scaling evaluates ways to spend more compute at inference. Snell et al. showed that compute-optimal allocation matched to problem difficulty is more efficient than best-of-N. Budget forcing in s1 controls thinking time through "Wait" insertion and delimiter manipulation. Best-of-N sampling picks the best of several responses.

Gains saturate past a certain amount of inference compute, and the extra tokens bring cost and latency with them. Decide whether to apply it by measuring accuracy and latency budgets for the task.