Scaling Laws in Production
From a research curve to a deployment budget
Scaling laws tell you how a model's loss falls as you add parameters, data, and compute. That is a statement about training. But a model that ships is not trained once and forgotten — it is run on inference billions of times, and every one of those queries costs money. This lesson is about the other half of the story: turning scaling-law intuition into deployment and cost decisions.
This is the economics / engineering companion to the Scaling Laws lesson, which covers the science (the power-law form, Chinchilla-optimal training, the loss-vs-compute curve). Read that first if the terms below are new. Here we assume the curve and ask: given it, what model do you actually deploy, and what does it cost to keep running?
The key reframing: a bigger model is not just more expensive to train — it is more expensive forever. Each extra parameter is extra arithmetic on every single query. For a popular product, that ongoing cost dwarfs the one-time training bill, and it flips the "just train the biggest model you can afford" instinct on its head.
Two costs, two shapes
Total cost of ownership (TCO) for a deployed model splits cleanly into two terms with very different shapes:
Training — one-time, fixed
You pay it once before launch. Standard estimate:
N = parameters, D = training tokens. The factor 6 is FLOPs per parameter per token for a forward + backward pass. This is a flat horizontal line on a cost-vs-traffic chart: it does not change no matter how many users you get.
Inference — ongoing, scales with use
You pay it on every query, for the life of the product:
N = parameters, T = tokens per query, Q = total queries over the model's lifetime. The factor 2 is FLOPs per parameter per token for a single forward pass. This term grows linearly with traffic — its slope is set by model size.
Putting them together, the lifetime bill is:
Notice N appears in both terms. Doubling the parameter count roughly doubles the training bill and doubles the cost of every future query. That second effect is the one people forget — and it is the one that decides the design of high-traffic systems.
Why popular products want a smaller, longer-trained model
The Chinchilla result says that to minimise loss for a fixed training budget, you should balance N and D at roughly 20 tokens per parameter. That is the right answer when you only pay for training. But it is the wrong objective for a product that will serve trillions of queries.
If inference dominates your lifetime cost, you would happily spend more training compute to get a smaller model of equal quality, because the smaller model is cheaper to serve on every query forever. Concretely: take a small model and train it far past the Chinchilla ratio — over-training, e.g. 150–1000+ tokens/param. You spend more on the one-time training run, but you ship a model with fewer parameters that hits your quality bar and is permanently cheaper at the edge.
This is the inference-aware (or compute-optimal-for-deployment) view. The LLaMA family is the famous example: LLaMA-7B saw ~1T tokens (~140 tokens/param), well past Chinchilla-optimal. It is not training-compute-optimal, but it is excellent total-cost-optimal once you account for serving it to millions of users.
The slogan to remember: "compute-optimal" always begs the question — optimal for what? A lab benchmarking a one-off run optimises training compute. A company serving a product optimises lifetime TCO, and that almost always pushes toward smaller, longer-trained, and compressed models.
Interactive: the training-vs-inference TCO calculator
Compare two models. The big model is trained roughly Chinchilla-optimal; the small model is smaller but over-trained (more tokens/param). Then drag the lifetime query volume slider and watch the two total-cost curves. At low volume the fixed training cost dominates; as volume rises, the per-query slope takes over and the lines cross. Try the quantization toggle to push the small model's curve down further and move the crossover left.
Drag from a research prototype (left) to a hit consumer product (right).
Big model
Small model
Cost model is illustrative (training ≈ 6·N·D FLOPs, inference ≈ 2·N·T FLOPs per query, one $/FLOP price) but qualitatively correct: the big model has a higher fixed training cost AND a higher slope (per-query cost), so its total-cost line starts higher and rises faster. Real serving adds memory, networking, batching, and utilisation effects.
What to notice. The big model's line starts higher (more training FLOPs) andclimbs faster (more FLOPs per query). So once you are to the right of the crossover, the gap only widens — and consumer products live far to the right, where serving trillions of queries makes the small model dramatically cheaper. Shrink the small model or enable quantization and the crossover slides toward zero.
Cutting inference cost: distillation and quantization
Over-training is one lever for a cheap-to-serve model. The other two attack the deployed model directly, shrinking N or the cost per parameter without retraining from scratch:
- Distillation. Train a small "student" to mimic a large "teacher." You get a model with far fewer parameters that retains much of the teacher's quality — directly lowering the N in 2·N·T·Q.
- Quantization. Serve weights/activations in INT8 or INT4 instead of FP16. Same N, but each operation moves fewer bytes and runs on cheaper kernels, cutting effective per-query cost (often 2–4×) with little quality loss.
- Pruning / sparsity. Remove weights or use mixture-of-experts so only part of the network runs per token, lowering the active parameter count.
These are covered in depth in Model Optimization and Inference Optimization for LLMs. Here the point is purely economic: every one of these techniques exists because per-query cost, multiplied across billions of queries, is where the money goes.
Capacity planning and the latency ceiling on model size
Cost is not the only thing that bounds model size — latency and throughput targets do too, and they often bite first. A request must return within some budget (say 300 ms to first token), and your fleet must sustain the peak QPS (queries per second). Both depend on N.
| Quantity | Rough relationship | Consequence |
|---|---|---|
| Per-token latency | ∝ N / (mem bandwidth) | Bigger model ⇒ slower tokens. A hard latency SLA caps N. |
| GPUs needed | ∝ N · T · QPS / (GPU FLOP/s) | Double N or double traffic ⇒ roughly double the fleet (and the bill). |
| Memory to hold weights | ∝ N · bytes/param | A model may need several GPUs just to fit, before serving a single user. |
Batching is the main lever that makes serving economical: combining many requests into one forward pass amortises the cost of loading weights, raising throughput per GPU. But batching trades off against latency, and there is a ceiling — past some model size you simply cannot hit both your latency SLA and your QPS without an unaffordable number of accelerators. That ceiling, not accuracy, is often what decides the deployed model size.
GPU economics in one line: you rent accelerators by the hour whether or not they are busy, so the metric that matters is queries served per GPU-hour. Everything above — smaller models, quantization, distillation, batching — is ultimately a way to push that number up.
Build vs. buy, and the size-vs-quality dial
The TCO lens also frames the strategic choice of where the model comes from:
| Option | Cost shape | Best when |
|---|---|---|
| Call a hosted API | $0 fixed, $/token usage | Low/uncertain volume, fast iteration, no infra team. |
| Self-host an open model | Infra fixed cost + cheaper $/token at scale | High, steady volume; per-token API margin now exceeds your ops cost. |
| Train / over-train your own | Large fixed training cost + lowest $/token | Huge volume, need a specific small model, lifetime inference dominates. |
It is the same crossover from the calculator: more fixed cost is only worth it if enough lifetime volume amortises it. Within any option, the size-vs-quality dial applies — you rarely need the biggest model. Pick the smallest model that clears your quality bar on your task, because every parameter above that bar is pure ongoing cost. Often a distilled or over-trained small model, or a large model reserved only for the hard fraction of traffic (a routing/cascade setup), is the cheapest way to hit the bar.
Common pitfalls
Optimising training compute for a product. Chinchilla-optimal minimises training loss per FLOP, not lifetime cost. For anything high-traffic, over-train a smaller model instead.
Ignoring the slope. Two models with the same launch cost can have wildly different TCO — the bigger one's per-query slope makes it lose badly at scale. Always project to your expected lifetime volume, not just day one.
Picking model size by benchmark alone. Latency SLAs and QPS may make the "best" model physically unservable. Size to what you can serve within budget, then maximise quality inside that box.
Key Takeaways
- TCO = training (one-time, ≈ 6·N·D) + inference (ongoing, ≈ 2·N·T·Q). Parameter count N drives both.
- Training is a flat line; inference is a sloped line. They cross at a query volume that decides which model is cheaper overall.
- For popular products, lifetime inference cost dominates — so it can be cheaper to train a smaller model longer (over-training past Chinchilla) to lower per-query cost forever.
- "Compute-optimal" depends on the objective: training-optimal (Chinchilla) ≠ deployment-optimal (inference-aware).
- Distillation, quantization, pruning/MoE, and batching all exist to push down per-query cost / raise queries per GPU-hour.
- Latency and throughput (QPS) targets put a hard ceiling on feasible model size — often before cost does.
- Build-vs-buy is the same crossover: more fixed cost (self-host, train your own) only pays off above enough lifetime volume.
- This is the deployment-economics counterpart to the Scaling Laws science lesson — read them together.