Model Optimization
Why Shrink a Model?
A modern large language model can have billions of parameters. Stored at full precision, a 7-billion-parameter model occupies roughly 28 GB of memory just for its weights — more than fits on most consumer GPUs, and expensive to serve at scale. Model optimization is the set of techniques for making a trained model smaller, faster, and cheaper to run while preserving as much of its quality as possible.
The three levers we care about are memory (does it fit on the hardware?), latency (how fast is each prediction?), and cost (compute and energy per request). The four workhorse techniques are quantization, pruning, knowledge distillation, and parameter-efficient fine-tuning such as LoRA. They are complementary — real deployments often stack several together.
The central trade-off: almost every optimization buys memory/speed by giving up a little fidelity. The art is finding the point where you save a lot and lose almost nothing measurable.
Quantization: Fewer Bits per Number
By default, neural-network weights are stored as 32-bit floating-point numbers (FP32) — 4 bytes each. Quantization represents those same weights (and sometimes the activations) using fewer bits. Each step down roughly halves the memory:
| Format | Bits | Bytes / param | 7B model | Typical quality |
|---|---|---|---|---|
| FP32 | 32 | 4 | ≈ 28 GB | baseline |
| FP16 / BF16 | 16 | 2 | ≈ 14 GB | essentially lossless |
| INT8 | 8 | 1 | ≈ 7 GB | small, usually negligible loss |
| INT4 | 4 | 0.5 | ≈ 3.5 GB | noticeable but often acceptable loss |
Mechanically, quantization maps a continuous range of real-valued weights onto a small set of discrete levels. An n-bit format has 2ⁿ levels (INT8 has 256, INT4 has just 16). A scale factor maps the original range onto those levels: q = round(w / scale), and the value is reconstructed as ŵ = q × scale. The gap between w and ŵ is the rounding (quantization) error — the price of using fewer bits.
Post-Training Quantization (PTQ)
Take an already-trained model and quantize the weights afterward — no retraining needed. Fast and cheap; works very well down to INT8 and increasingly to INT4. A small calibration set is often used to pick good scale factors.
Quantization-Aware Training (QAT)
Simulate the rounding during training (or fine-tuning) so the model learns weights that are robust to it. More expensive than PTQ, but recovers more accuracy — especially valuable at very low bit-widths where PTQ starts to hurt.
Interactive: Quantization Memory Explorer
Pick a model size and a precision. The bars show the weight-memory footprint at each precision so you can see exactly how many bytes per parameter buys you — and how much you save by dropping bits.
8-bit integer — small, well-tolerated quality loss
Memory footprint by precision
The math: footprint = (number of parameters) × (bytes per parameter). A 7B model in INT8 = 7B × 1 bytes = 7.00 GB. Halving the bit-width halves the memory. (Real deployments also need extra memory for the KV cache and activations, so these are the weight-only minimums.)
Interactive: Snapping Weights to a Grid
Memory savings have a cost: precision. Below, ten continuous weights (blue) are snapped onto the discrete grid of levels available at a given bit-width (orange). Fewer bits means a coarser grid and larger rounding errors. This is quantization made tangible — the same trade the explorer above summarizes as “quality retained.”
| Original | Quantized | Error |
|---|---|---|
| -0.92 | -0.867 | +0.053 |
| -0.61 | -0.600 | +0.010 |
| -0.37 | -0.333 | +0.037 |
| -0.18 | -0.200 | -0.020 |
| 0.05 | 0.067 | +0.017 |
| 0.23 | 0.200 | -0.030 |
| 0.44 | 0.467 | +0.027 |
| 0.58 | 0.600 | +0.020 |
| 0.71 | 0.733 | +0.023 |
| 0.88 | 0.867 | -0.013 |
Drag the slider down toward 1-bit and watch the grid get coarser: fewer levels means each weight snaps farther from its true value, so the rounding error grows. Push it up toward 6-bit and the snapped values hug the originals. This is the precision-loss side of the memory-savings trade.
Pruning: Removing Weights Entirely
Trained networks are over-parameterized — many weights contribute almost nothing. Pruning removes them, typically by setting the smallest-magnitude weights to zero, producing a sparse model. Sparsity is the fraction of weights that are zero; a 90%-sparse layer keeps only 10% of its connections.
Unstructured pruning
Zero out individual weights anywhere in the matrix. Achieves very high sparsity with little accuracy loss, but the leftover non-zeros are scattered — you need specialized sparse kernels or hardware to actually realize a speedup.
Structured pruning
Remove whole units — entire neurons, channels, attention heads, or layers. Lower achievable sparsity, but the result is a genuinely smaller dense model that runs faster on ordinary hardware with no special kernels.
Pruning is usually followed by a short fine-tuning pass so the remaining weights compensate for their departed neighbors. It pairs well with quantization: prune first, then quantize what remains.
Knowledge Distillation: Learning from a Teacher
Instead of shrinking a big model directly, knowledge distillation trains a small student model to imitate a large teacher. The key insight: the teacher's full probability distribution carries more information than just the correct label. When a teacher classifying an image of a dog assigns 70% dog, 25% wolf, 5% cat, those “soft” relative probabilities tell the student that wolves look more dog-like than cats do — a signal the one-hot label throws away.
To expose this structure, the logits are softened with a temperature T in the softmax:
Here zᵢ are the teacher's logits. A higher T (e.g. 2–4) flattens the distribution, revealing the small probabilities the student should match. The student is trained on a blend of two losses: a distillation loss (match the teacher's soft probabilities, usually KL divergence at temperature T) and the standard cross-entropy loss against the true hard labels.
Canonical example — DistilBERT: a student distilled from BERT that keeps roughly 97% of BERT's language-understanding performance while being about 40% smaller and 60% faster. Distillation produces a permanently smaller architecture, not just a compressed copy of the original.
LoRA & Parameter-Efficient Fine-Tuning
Optimization is not only about inference — adapting a large model to a new task is also expensive if you must retrain all its weights. Parameter-efficient fine-tuning (PEFT) freezes the giant pretrained model and trains only a tiny number of new parameters.
The most popular method, LoRA (Low-Rank Adaptation), observes that the weight update needed for a new task is typically low-rank. Rather than learning a full update matrix ΔW (which is the same size as W), LoRA factors it into two small matrices:
The base weights W stay frozen; only the low-rank A and B are trained. For a 4096×4096 layer (≈16.8M parameters), a rank-8 LoRA adapter trains only 8×4096 + 4096×8 ≈ 65K parameters — well under 1% of the original. The adapters are tiny to store, you can swap a different adapter per task, and at inference they can be merged back into W for zero added latency. Combining quantization of the frozen base with LoRA adapters (“QLoRA”) makes fine-tuning huge models possible on a single GPU.
Comparing the Techniques
| Technique | What it does | Mainly helps | Retraining? |
|---|---|---|---|
| Quantization | Fewer bits per weight/activation | Memory & speed | PTQ: no · QAT: yes |
| Pruning | Remove weights/neurons (sparsity) | Memory & (structured) speed | Usually fine-tune after |
| Distillation | Train a small student on a teacher | Memory, speed & cost | Yes (trains the student) |
| LoRA / PEFT | Train tiny low-rank adapters | Cheap fine-tuning & storage | Yes, but only the adapters |
Common Pitfalls
Outliers break naive quantization. A few large-magnitude activations can blow up the scale factor and crush precision for everything else. Modern methods handle these outliers specially (per-channel scales, mixed precision).
Unstructured sparsity is not free speed. Zeroing 90% of weights does not give a 10× speedup on a standard GPU unless you have sparse kernels — the matrix is still the same shape.
Always measure quality on your task. “99% quality retained” is an average; the loss may concentrate on exactly the hard cases you care about. Benchmark the compressed model on real evaluation data before shipping.
Key Takeaways
- Model optimization trades a little quality for large gains in memory, latency, and cost so big models can actually be deployed.
- Memory = parameters × bytes/param. Halving the bit-width halves the memory: a 7B model is ≈28 GB (FP32), ≈14 GB (FP16), ≈7 GB (INT8), ≈3.5 GB (INT4).
- Quantization snaps continuous weights onto 2ⁿ discrete levels; fewer bits means a coarser grid and larger rounding error. PTQ is cheap and post-hoc; QAT simulates rounding during training to recover accuracy.
- Pruning removes weights — unstructured for high sparsity, structured for real speedups on ordinary hardware.
- Knowledge distillation trains a small student to match a teacher's temperature-softened soft probabilities (e.g. DistilBERT: ~97% of BERT, ~40% smaller).
- LoRA / PEFT freezes the base model and trains a low-rank update ΔW = B·A — fine-tuning huge models with under 1% of the parameters.
- These techniques are complementary and routinely stacked (e.g. prune → quantize, or QLoRA).