Test-Time Compute
The Big Idea
For most of deep learning's history, the recipe for a smarter model was "train a bigger model on more data." That is train-time compute: you pay once, up front, and every future answer comes out of the same fixed forward pass. Test-time compute (also called inference-time compute) is the complementary lever: spend more computation per question, at inference time, to get a better answer from the same trained model.
Concretely, instead of emitting one quick answer, the model can think longer (produce a long internal chain-of-thought), generate many candidate solutions and pick the best, or search over reasoning steps. This is the foundation of the modern class of reasoning models — OpenAI's o1 / o3, DeepSeek-R1, and similar systems — which are trained to produce long chains-of-thought and then "think" before answering.
One sentence: a fixed model can get measurably more accurate if you let it do more work at inference — by sampling more, reasoning longer, or searching harder — rather than only by making the model itself larger.
Why Spending Compute at Inference Helps
A single greedy decode commits to one path through an enormous space of possible reasoning chains. If the first step is wrong, the answer is usually wrong. Test-time compute attacks this in two complementary ways:
1. Reason longer (sequential compute)
Generate more intermediate tokens — explore the problem, try sub-approaches, check work, backtrack — before committing to a final answer. A longer chain-of-thought lets the model decompose a hard problem into easy steps it can actually do.
2. Try many times (parallel compute)
Sample several independent attempts and aggregate them — by majority vote (self-consistency) or by scoring each candidate with a verifier and keeping the best (best-of-N). Even a noisy model that is right more often than not becomes far more reliable when you pool independent attempts.
The Techniques
| Technique | What it does | Compute used |
|---|---|---|
| Chain-of-Thought | Prompt the model to "think step by step," generating intermediate reasoning before the answer. | Sequential (longer output) |
| Self-Consistency | Sample many CoT paths at temperature > 0; take the majority vote over the final answers. | Parallel (N samples) |
| Best-of-N / Verifier reranking | Generate N candidates, score each with a reward model / verifier, keep the highest-scoring one. | Parallel (N + scoring) |
| Tree of Thoughts / search | Beam or tree search over reasoning steps, expanding promising branches and pruning bad ones (often with a process reward model). | Both (branching search) |
| RL-trained long reasoning | Train the model with reinforcement learning to natively produce long, self-correcting chains-of-thought (o1/o3, DeepSeek-R1). Reasoning length is learned, not just prompted. | Sequential (long CoT) |
Key distinction: self-consistency and best-of-N both spend parallel compute, but they aggregate differently. Self-consistency needs no extra model — it just counts which answer appears most often. Best-of-N needs a verifier to score candidates, which can pick a rare-but-correct answer that majority vote would miss.
The Math of Self-Consistency
Why does voting over samples beat a single sample? Suppose a single reasoning attempt produces the correct final answer with probability p, and wrong attempts scatter across d distractor answers. Draw N independent samples and take the answer that appears most often. The correct answer wins whenever its vote count exceeds every distractor's.
In the simplest case (binary: one right answer, one wrong answer, odd N), the majority is correct exactly when more than half of the samples are correct — a binomial tail:
Here C(N, k) is the binomial coefficient (number of ways to choose which k of the N samples are correct), pk is the probability those k are correct, and (1−p)N−k that the rest are wrong. By the law of large numbers, as N → ∞ this tends to 1 when p > 0.5 and to 0 when p < 0.5. That is the whole story:
Voting only helps if the model is better than chance. When p > 0.5, more samples drive accuracy toward 100%. When p < 0.5, more samples drive it toward 0% — you would be amplifying a systematic error. The demo below computes the exact multi-distractor version.
Interactive: Self-Consistency / Best-of-N Simulator
Set the single-sample accuracy p, the number of samples N, and how many distinct wrong answers the model tends to give. The blue curve is the exact majority-vote accuracy as a function of compute (∝ N). Then run real random trials and watch the empirical rate (amber) converge to the theory.
How often one reasoning attempt gets the final answer right.
Odd N avoids most ties.
Wrong votes spread over more answers → easier for the truth to win a plurality.
Try this:
- Keep p > 50% and raise N — accuracy climbs above p, with diminishing returns.
- Set p < 50% — voting makes things worse; you must be better than chance.
- Run trials and watch the amber dot approach the blue curve.
Notice the diminishing returns: the curve is steep at first and flattens — doubling compute does not double accuracy. And below 50% single-sample accuracy, the curve bends the wrong way. This is the test-time scaling curve in miniature.
The Test-Time Scaling Curve & the Tradeoff
Empirically, accuracy on hard reasoning benchmarks rises smoothly as you pour in more inference compute — more samples, longer chains-of-thought, deeper search. Plotted against log compute, the gains look roughly linear over a wide range before saturating. This mirrors training scaling laws, but the knob is now inference.
What you gain
- • Higher accuracy on math, code, and multi-step reasoning
- • Adjustable per query — spend more only on hard problems
- • No retraining; works on an already-deployed model
- • A small model + lots of test-time compute can rival a much larger model
What you pay
- • Latency: long reasoning / many samples are slow to return
- • Cost: compute (and tokens) scale roughly ∝ N or ∝ reasoning length
- • Diminishing returns: each doubling buys less accuracy
- • Can amplify errors if the base model is below chance on a task
The central engineering decision is a compute-accuracy tradeoff: how many samples / how long a chain is worth the extra latency and dollars for this query? A simple lookup needs none; a competition math problem may justify hundreds of samples or minutes of reasoning.
Test-Time Compute vs. Bigger Models
When should you spend your compute budget on inference rather than on a larger model? Research on test-time scaling (e.g. work from DeepMind and others) found that the answer depends on difficulty:
| Situation | Better lever | Why |
|---|---|---|
| Easy / medium problems within the model's reach | Test-time compute | The model already "knows" the answer; sampling/search reliably surfaces it, often beating a bigger model at equal total FLOPs. |
| Very hard problems beyond the base model's competence | A stronger / larger model | If single-sample accuracy is near chance, no amount of voting helps — you need more capability, i.e. more pretraining or RL. |
| Latency-critical, high-volume serving | Bigger model, one pass | A single fast call from a capable model beats waiting on long reasoning per request. |
The two levers also compound: reasoning models like o1/o3 and DeepSeek-R1 are first made capable of long reasoning via RL training, and then spend test-time compute to use it. Training and inference scaling are not rivals so much as a pipeline.
Common Pitfalls
Voting a below-chance model
Self-consistency amplifies whatever the model is biased toward. If that bias is wrong (p < 0.5), more samples make it more confidently wrong.
Correlated samples
The math assumes independent attempts. At low temperature samples are near-identical, so voting buys little. You need enough diversity for the pooling to help.
A weak or hackable verifier (best-of-N)
Best-of-N is only as good as its scorer. A miscalibrated reward model can rank a confident wrong answer above a correct one, and aggressive search can "reward-hack" it.
Overthinking easy questions
Spending hundreds of samples or a long chain on a trivial query wastes latency and money for no accuracy gain. Budget compute to difficulty.
Key Takeaways
- Test-time (inference-time) compute trades extra computation per query for higher accuracy from a fixed trained model.
- Core techniques: chain-of-thought (reason longer), self-consistency (majority-vote many CoT paths), best-of-N with a verifier, tree/beam search over reasoning steps, and RL-trained long reasoning (o1/o3, DeepSeek-R1).
- Self-consistency works by the binomial vote: when single-sample accuracy p > 0.5, more samples push accuracy toward 100%; when p < 0.5, they push it toward 0%.
- Accuracy rises with inference compute along a scaling curve, but with diminishing returns — and at real cost in latency and dollars (∝ N or reasoning length).
- Test-time compute beats scaling the model on problems within reach of the base model; truly hard problems still need more underlying capability.