Model Deployment
Introduction
Training a model is only half the job. A model sitting in a notebook or a checkpoint file produces no value until it is deployed — wired up so that real systems and users can send it inputs and get predictions back, reliably, fast enough, and at acceptable cost. Model deployment is the engineering discipline of getting a trained model into production and keeping it healthy there.
The hard parts are rarely about the math. They are about latency budgets, throughput under load, cost per prediction, safe rollouts, and the subtle ways that a model that scored 95% offline can quietly misbehave once it meets live traffic. This lesson walks through the patterns and pitfalls, and lets you play with the single most important tradeoff in serving: latency versus throughput versus cost.
The core mental shift: training optimizes a model for accuracy; serving optimizes a system for latency, throughput, cost, and reliability — all while keeping predictions correct. Those are different objectives, and they pull in different directions.
Serving patterns: when do predictions happen?
The first design decision is when predictions are computed relative to when they are needed. There are three canonical patterns.
Batch (offline) inference
Predictions are precomputed on a schedule (e.g. nightly) for a known set of inputs and written to a store (database, warehouse, cache). At request time you just look up the answer — microseconds, no model in the hot path. Great for recommendations, churn scores, lead ranking. Downsides: predictions are stale between runs, and you cannot score inputs you have not seen yet.
Online (real-time) inference
The model runs synchronously behind an API: a request comes in, the model computes a prediction, the response goes back — typically within tens to hundreds of milliseconds. Needed when the input is only known at request time (fraud check on this transaction, this user's feed, this chat turn). This is where latency, throughput, and autoscaling really matter.
Streaming inference
The model is embedded in a continuous event pipeline (Kafka, Flink, Spark Streaming). Events flow in, get scored as they arrive, and results flow downstream — near-real-time but asynchronous, not a blocking request/response. Used for real-time anomaly detection, monitoring, and feature pipelines. (Token-by-token LLM "streaming responses" are a different, related idea: an online request that streams its output progressively.)
| Pattern | Latency to user | Freshness | Typical use |
|---|---|---|---|
| Batch / offline | ~0 (cache lookup) | Stale until next run | Recommendations, scoring known entities |
| Online / real-time | ms (in the hot path) | Always fresh | Fraud, search ranking, LLM chat |
| Streaming | Near-real-time, async | Continuously updated | Event scoring, monitoring pipelines |
Deployment surfaces: where does the model run?
Once you know when you serve, you choose where. The same model can live on very different surfaces depending on latency, scale, and privacy requirements.
REST / gRPC microservice
The model is wrapped in a long-running service behind an HTTP/REST or gRPC endpoint. gRPC (binary, HTTP/2, streaming) is typically lower-latency and higher-throughput than JSON REST, and is the default for internal model-to-model traffic. The workhorse of online serving.
Serverless functions
Deploy as a function (AWS Lambda, Cloud Run, etc.) that scales to zero and bills per invocation. Cheap for spiky, low-volume traffic. Watch out for cold starts (loading a large model on first request) and per-invocation memory/time limits — usually a poor fit for big GPU models.
Edge / on-device
The model runs on the phone, browser, or IoT device (Core ML, TensorFlow Lite, ONNX Runtime, WebGPU). Zero network latency, works offline, keeps data private — at the cost of tight compute/memory budgets, so models are quantized and distilled to fit.
Packaging: containers
In the cloud, models are almost always shipped as Docker containers pinning the exact runtime, dependencies, and weights, then orchestrated with Kubernetes. Containers make "works on my machine" reproducible and let you scale replicas up and down.
Model servers
Rather than hand-roll a Flask app, production systems use purpose-built model servers that handle batching, multi-model hosting, GPU scheduling, metrics, and versioning out of the box:
- TorchServe — serving for PyTorch models.
- NVIDIA Triton Inference Server — multi-framework (PyTorch, TensorFlow, ONNX, TensorRT), dynamic batching, concurrent model execution on GPU.
- vLLM (and TGI, TensorRT-LLM) — specialized LLM serving with continuous batching and PagedAttention KV-cache management to push tokens/sec on expensive GPUs.
The key serving metrics
Three numbers dominate every serving decision. You cannot maximize all three at once — that is the whole game.
Latency
How long one request takes. Report it as percentiles, never just the mean: p50 (median), p95, p99. The tail (p95/p99) matters most — if 1% of users wait 2 seconds, that is the experience that drives complaints, and a page that fans out to 100 services hits its p99 on almost every request.
Throughput
How much work the system clears per second — requests/sec for classic models, tokens/sec for LLMs. Determines how much hardware you need and is the lever that controls cost-per-prediction.
Cost & utilization
GPUs are expensive; an idle GPU is wasted money. Autoscaling adds/removes replicas with load to keep utilization high without dropping requests. High throughput per GPU = low cost per prediction.
The latency ↔ throughput ↔ cost triangle
Batching — running several requests through the model in a single forward pass — is the canonical knob. A GPU has fixed per-call overhead, so processing 32 requests together is far cheaper per request than 32 separate calls: throughput and utilization soar, and cost-per-prediction drops. But a request must now wait for the batch to fill, so tail latency rises. Batch size 1 is the lowest-latency, lowest-throughput extreme; large batches are the opposite. The simulator below lets you feel exactly where the sweet spot is.
Interactive: the batching tradeoff simulator
This is a discrete-event simulation of a real inference server. Requests arrive as a Poisson process at rate λ. The server forms a batch — waiting until it is full (your batch size) or until a batching window elapses — then runs the whole batch in one forward pass that costs 5 ms + 1.5 ms × (batch size) (fixed GPU overhead plus a marginal cost per item). Each request's latency is its queue wait plus the batch compute time.
Try this: start at batch size 1 and slide it up — watch throughput (blue) climb while p95 latency (red) climbs too. Then crank the arrival rate until the blue throughput curve can no longer reach the orange λ line: the server is overloaded, the queue explodes, and latency runs away. That is the entire latency/throughput/cost tradeoff in one picture.
How fast requests show up.
Max requests run together in one pass.
Max time to wait for a batch to fill.
The green dashed line marks your current batch size. Push it left for low latency / low throughput; push it right for high throughput at the cost of tail latency. Raise the arrival rate until the blue curve can no longer reach the orange λ line — that is overload, and the queue (and latency) explode.
Versioning & safe rollout strategies
Models change constantly — retrained on new data, swapped for better architectures. Every model artifact should be versioned (weights + code + preprocessing + config) so any prediction can be traced to exactly what produced it and any release can be rolled back. You never just flip the whole fleet to a new model and hope. Instead:
Shadow deployment
The new model receives a copy of live traffic but its predictions are not served to users — they are only logged and compared against the current model. Zero user risk; lets you validate latency and prediction quality on real traffic before trusting it.
Canary release
Route a small slice of real traffic (say 1% → 5% → 25%) to the new version while watching metrics (latency, errors, business KPIs). If it looks healthy, ramp up; if not, roll back having affected only a handful of users.
Blue-green deployment
Stand up the new version ("green") fully alongside the running one ("blue"), then switch all traffic over at once. Instant cutover and instant rollback (flip back to blue) — at the cost of running two full environments during the transition.
These pair naturally with A/B testing, where you split traffic to measure which model actually moves business metrics, not just offline accuracy.
Separate training and serving — and beware skew
Training and serving are different systems with different needs. Training is throughput-oriented, batch, offline, and tolerant of slow iteration; serving is latency-oriented, online, and must be highly available. Keep them separate: the serving stack should load a frozen, versioned artifact, not import your training code. But that separation creates the single most notorious production bug.
The train/serve skew pitfall
Training/serving skew happens when the features fed to the model differ between training and serving — even subtly. The model was fit on one distribution and is now scored on another, so accuracy silently degrades in production while every offline metric still looks great. Classic causes:
- Features computed by a Python/pandas pipeline in training but re-implemented in Java/SQL at serving time — small differences in rounding, null handling, time zones, or default values diverge.
- Normalization stats (mean/σ) recomputed at serving time instead of reusing the exact values from training.
- Time-travel / leakage: a feature uses data that was not yet available at the real prediction moment, so serving sees a different (correct-but-later) value.
- Different text tokenization, image resizing, or missing-value imputation between the two paths.
Mitigations: share one feature-transformation code path across train and serve, use a feature store that guarantees the same computation in both, log serving features and compare their distribution to training, and write unit tests asserting identical outputs for identical inputs.
| Aspect | Training | Serving |
|---|---|---|
| Optimized for | Throughput, accuracy | Latency, availability |
| Mode | Offline, batch | Online, per-request |
| Data access | Full historical dataset | One example, point-in-time |
| Failure cost | Re-run the job | User-facing outage |
Common pitfalls
✗ Watch out for
- • Reporting only mean latency and ignoring the p95/p99 tail.
- • Train/serve skew from divergent feature pipelines.
- • Cold starts on serverless with large models.
- • No rollback path — flipping 100% of traffic at once.
- • Provisioning for average load, then falling over at peak.
- • Forgetting that a model degrades over time (data drift) once deployed.
✓ Good practice
- • Define a latency SLO in percentiles (e.g. p99 < 200 ms).
- • Tune batch size to the SLO, not just for max throughput.
- • Version every artifact; make rollback one command.
- • Roll out via shadow → canary → full.
- • Share one feature path across train and serve.
- • Monitor latency, errors, utilization, and prediction drift in prod.
Key Takeaways
- Deployment turns a trained artifact into a service: it must be fast, reliable, and cheap enough, not just accurate.
- Choose a serving pattern by when predictions are needed: batch/offline (precompute), online (synchronous API), or streaming (event pipeline).
- Choose a surface by latency and privacy needs: REST/gRPC microservice, serverless, or edge/on-device — usually packaged as a Docker container and served by TorchServe, Triton, or vLLM.
- Latency (measure p50/p95/p99, mind the tail), throughput (req/s or tokens/sec), and cost form a triangle you cannot fully win.
- Batching raises throughput and GPU utilization (lowering cost per prediction) but adds latency, since requests wait to fill a batch; size it to your latency SLO.
- When arrival rate exceeds capacity, the queue grows without bound and latency explodes — autoscale or shed load before that point.
- Version every model and roll out safely with shadow, canary, and blue-green strategies so you can always roll back.
- Keep training and serving separate, but beware train/serve skew — share one feature pipeline so features are computed identically in both.