Test-Time Compute Optimization
Spending inference compute where it pays off
A trained model still has one big knob left at inference time: how much compute to spend answering each request. You can call a small model or a large one; take one sample or vote over many; let a reasoning model think for two steps or twenty. More compute usually buys more accuracy — but in production every extra GPU-second costs money and adds latency, and your users are waiting behind a service-level agreement (SLA).
Test-time compute optimization is the discipline of spending that inference budget efficiently: maximizing quality per dollar and per millisecond, not maximizing compute. The core realization is that not all queries are equally hard. Most production traffic is easy — a small, cheap model nails it. A minority is genuinely hard and deserves the expensive treatment. If you can tell which is which, you can route accordingly and buy most of the big model's accuracy at a fraction of its cost.
This lesson is about cost and latency. Whether extra test-time compute makes a model more accurate — self-consistency voting, best-of-N, longer chains of thought, and the scaling curves behind them — is covered in Test-Time Compute & Reasoning. Here we assume those techniques work and ask the production question: given a budget and an SLA, how do you allocate that compute so you spend it only where it earns its keep?
The accuracy ↔ cost ↔ latency frontier
Every inference strategy lands somewhere on a three-way tradeoff surface. Pushing one axis usually moves the others:
- Accuracy / quality — how often the system is right (or how good the answer is on your eval).
- Cost — compute per request: model size, number of samples, tokens generated, KV-cache memory. This is what shows up on the bill.
- Latency — wall-clock time to first token and to completion, which your SLA bounds (e.g. p95 < 2 s).
You do not get to pick a point off the surface — you pick the best achievable point given your constraints. A useful way to think about it: for a fixed accuracy target, find the cheapest/fastest strategy that hits it; or for a fixed budget and latency SLA, find the strategy that maximizes accuracy. That chosen point is your operating point. The whole game of this lesson is moving your operating point toward the efficient frontier — the set of strategies you cannot improve on one axis without paying on another.
Quality per dollar, not max quality. Always escalating every query to the biggest model and 32-way voting is a valid point on the surface — usually the most accurate one — but it is almost never the right operating point. It burns budget on the 80% of traffic a small model already answers correctly. The win comes from making compute adaptive to difficulty.
Adaptive compute: routing & model cascades
The central technique is adaptive compute: vary how much compute each query gets based on how hard it appears to be. Easy queries get the cheap path; hard queries escalate. A few common patterns:
- Model cascade. Try a small, cheap model first. If it is confident, return its answer. If not, escalate to a larger model (and possibly a third tier). Most traffic stops at tier one.
- Router / dispatcher. A lightweight classifier looks at the query up front and sends it to the right model — cheap model for "what's 2+2", frontier model for a multi-step proof — without necessarily running the cheap model first.
- Confidence-based early exit. Within a single model, stop computing once the answer is settled — exit early from intermediate layers, or stop a reasoning chain as soon as the answer stabilizes, rather than always running to a fixed depth.
- Difficulty-adaptive sampling. For self-consistency, draw few samples on easy queries and many on hard ones — instead of a fixed N for all traffic.
All of these need a difficulty signal: something that tells you whether a query is easy or hard before you have spent the big compute. Common signals are the cheap model's own confidence (token log-probabilities, calibrated probability, or the agreement among a few cheap samples), a learned router's score, or features of the query itself (length, presence of math, retrieval coverage). The signal is never perfect — that is exactly the tradeoff you tune below.
Why cascades work. If 70% of queries are answered correctly and confidently by a model that is 10× cheaper, you only pay for the big model on the remaining 30%. Average cost collapses toward the cheap model's, while accuracy stays close to the big model's — provided your confidence signal mostly escalates the queries the cheap model would have gotten wrong.
Interactive: a model-cascade routing simulator
Below is a stream of 2,000 queries — a mix of easy and hard ones. A cheap model answers each query first and reports a confidence. If that confidence falls below your escalation threshold, the query is re-sent to an expensive model (12× the cost, much higher latency) and we take its answer instead. Move the threshold and watch the operating point trace the frontier.
The shape of the green accuracy curve is the whole point. It rises steeply as the first escalations kick in — because a good confidence signal escalates exactly the queries the cheap model was about to miss — then flattens, because the remaining escalations are mostly queries the cheap model already had right. Cost, by contrast, rises almost linearly with the escalation rate. The efficient operating point sits at the knee, where you have captured most of the accuracy lift before cost runs away.
More techniques for cheaper, faster inference
Routing decides which compute to spend. These techniques make each unit of compute go further — they reduce cost and latency without changing the answer (or barely changing it):
| Technique | What it does | Mainly helps |
|---|---|---|
| Prompt / prefix caching | Reuse the cached KV state for a shared prompt prefix (system prompt, few-shot examples) instead of recomputing it every request. | Cost + latency (TTFT) |
| KV caching | Store the keys/values of already-generated tokens so each new token is O(1) attention over the cache, not a full recompute of the sequence. | Latency + cost |
| Speculative decoding | A small draft model proposes several tokens; the big model verifies them in one parallel forward pass. Accepted tokens are free speed-ups; the output distribution is unchanged. | Latency (same accuracy) |
| Dynamic stopping of reasoning | Halt a chain of thought once the answer is settled or a confidence threshold is met, rather than always generating to a fixed token budget. | Cost + latency |
| Adaptive self-consistency | Scale the number of voting samples with difficulty; stop sampling early once the vote has a clear, stable winner. | Cost |
| Batching / quantization | Group concurrent requests to saturate the GPU; serve a lower- precision (e.g. INT8/FP8) model to cut memory and time per token. | Cost + throughput |
Lossless vs. lossy. Caching and speculative decoding are lossless — they produce the same output faster/cheaper, so use them by default. Routing, early exit, dynamic stopping, and adaptive sampling are lossy: they trade a little accuracy for a lot of savings, so they need a difficulty signal and a tuned operating point. Quantization sits in between depending on how aggressive it is.
SLAs, budgets, and choosing an operating point
In production you do not tune the threshold by eye — you tune it against explicit constraints. Two framings dominate:
- Latency SLA. A bound like p95 latency < 2 s. Note this is a tail guarantee: escalation adds the big model's latency on top of the cheap call, so a high escalation rate blows the p95 even if the average looks fine. Sometimes you escalate in parallel (run both, take the better) to protect latency at the cost of always paying for the big model.
- Cost budget. A target average cost per request (or per day). The escalation rate is your main lever: average cost ≈ c_cheap + (escalation rate) × c_big, so you set the threshold to hit the budgeted escalation rate.
The disciplined recipe: build the frontier curve from a representative traffic sample (exactly what the demo does), draw your constraint as a line on it — a horizontal cost/latency cap or a vertical accuracy floor — and pick the threshold where you just satisfy the constraint with the best value on the other axis. Then monitor: if traffic shifts harder over time, your escalation rate and costs drift, so the operating point needs re-tuning.
Pitfall: a miscalibrated confidence signal. Cascades and early exit only work if the difficulty signal correlates with actual errors. If the cheap model is overconfident on queries it gets wrong, you will fail to escalate the very queries that needed it — accuracy stays low no matter the threshold. Calibrate the signal (e.g. temperature scaling, or use agreement across a few cheap samples) before trusting a cascade in production.
Key Takeaways
- The goal of test-time compute optimization is maximum quality per dollar and per millisecond, not maximum compute. Always using the biggest model and the most samples is rarely the right operating point.
- Inference lives on an accuracy ↔ cost ↔ latency frontier. You choose an operating point on it, bounded by your cost budget and latency SLA.
- Adaptive compute — model cascades, routers, confidence-based early exit, and difficulty-adaptive sampling — spends more only on the hard queries, capturing most of the big model's accuracy at a fraction of the cost.
- Cascades depend on a calibrated difficulty/confidence signal; a miscalibrated one escalates the wrong queries and breaks the whole tradeoff.
- Lossless wins (prompt/KV caching, speculative decoding) cut cost and latency for free — use them by default; lossy wins (early exit, dynamic stopping) need a tuned operating point.
- The accuracy-vs-threshold curve has a knee: most of the accuracy gain comes from the first escalations while cost keeps rising linearly — operate at the knee.
- Whether more test-time compute raises accuracy at all is a separate question — see Test-Time Compute & Reasoning.