Edge AI & Mobile Deployment
What is Edge AI?
Edge AI means running a model directly on the device that captures the data — a phone, a smartwatch, a camera, a car, or a tiny microcontroller — instead of sending the data to a server and waiting for a response. The model lives on the edge of the network, next to the user, rather than in a centralized cloud datacenter.
When you unlock your phone with your face, get live captions without internet, or say a wake word that triggers a voice assistant, a model is very likely running on-device. The same idea scales down to microcontrollers with kilobytes of memory — a field often called TinyML.
Cloud inference
- Huge models, lots of RAM and GPUs
- Needs a network round trip
- Data leaves the device
- Per-query server cost
On-device (edge) inference
- Small models, tight memory budget
- No round trip — instant response
- Data stays local (private)
- Works offline, no per-query cost
Why run models on-device?
Low latency
No network round trip. A cloud call can take 100–500 ms of network time before the model even runs; on-device the result is ready in milliseconds. Essential for real-time tasks like camera effects, AR, and autocorrect.
Privacy
The raw data — your photos, voice, health signals, keystrokes — never leaves the device. Nothing to intercept, store, or breach on a server. This is often a hard requirement for medical and personal data.
Offline capability
Works on a plane, in a tunnel, or in a remote area with no connectivity. The model is right there, so flaky or absent networks do not break the feature.
No per-query server cost
Inference runs on hardware the user already paid for. There are no GPU servers to rent and scale, so a feature that runs billions of times a day costs the provider essentially nothing per call.
The constraints
The cloud has nearly unlimited resources. The edge does not. Fitting a useful model onto a device means living within four hard limits:
| Constraint | Why it bites | Typical edge budget |
|---|---|---|
| Memory / RAM | Weights + activations must fit in RAM. A 7B model in FP32 is ~28 GB — far beyond a phone. | KB (MCU) → few GB (phone) |
| Compute | Limited FLOPs/ops per second; big models are simply too slow to feel interactive. | <1 GOPS → tens of TOPS |
| Battery / thermal | Inference drains the battery and heats the device, which then throttles to protect itself. | millijoules per inference |
| Model size (storage) | The model file ships inside the app. Hundreds of MB bloat downloads and updates. | ~KB to low hundreds of MB |
Interactive: on-device feasibility explorer
Pick a target device, choose a base model, then apply compression and watch the footprint, latency, energy, and accuracy update live. A green bar means the model fits the device's RAM budget; red means it is too big. The goal: get a model to fit without giving up too much accuracy.
1. Pick a target device
Modern flagship with a dedicated NPU / Neural Engine
2. Choose a base model
3. Quantize (precision)
4 bytes per weight · FP32 (full)
4. Compress further
The model fits in memory and responds fast enough to feel interactive.
Numbers are illustrative but follow the real relationships: footprint = parameters × bytes-per-weight, latency scales with compute work divided by device throughput, and lower precision both shrinks and speeds up the model at a small accuracy cost.
Techniques to fit a model on-device
There are two complementary strategies: compress an existing model so it needs less memory and compute, or design an efficient architecture that is small and fast from the start. Most of the compression methods below are covered in depth in the model optimization lesson.
Quantization (FP32 → FP16 → INT8 → INT4)
Store weights (and often activations) in fewer bits. FP32 uses 4 bytes per weight; INT8 uses 1 byte — an immediate 4× reduction in size — and INT4 halves it again to 8×. Integer math is also faster and more energy-efficient on hardware with INT8/INT4 datapaths. INT8 quantization is typically near-lossless (often within ~1% accuracy); INT4 costs a bit more and usually needs careful calibration or quantization-aware training.
Pruning
Remove weights (or whole channels/neurons) that contribute little to the output, then fine-tune to recover accuracy. Unstructured pruning zeros individual weights for sparsity; structured pruning removes entire channels so the model is genuinely smaller and faster on standard hardware. Many networks tolerate 50%+ sparsity with minimal accuracy loss.
Knowledge distillation
Train a small student network to mimic a large teacher. The student learns from the teacher's soft probability outputs (which carry richer information than hard labels), so it reaches accuracy close to the teacher at a fraction of the size. DistilBERT, for example, keeps ~97% of BERT's performance with ~40% fewer parameters.
Efficient architectures
Design the network to be cheap from the start. The flagship trick is the depthwise-separable convolution used by MobileNet: it splits a standard convolution into a depthwise step (one filter per input channel) plus a 1×1 pointwise step that mixes channels. This cuts the compute of a standard convolution by roughly a factor of N (the number of output channels) — often an 8×–9× reduction for a 3×3 kernel — with only a small accuracy drop. Other examples: EfficientNet (compound width/depth/resolution scaling), SqueezeNet (fire modules), and small language models like Phi and Gemma built to run on a phone.
Why depthwise-separable convolutions are so cheap
A standard convolution with a K×K kernel mapping M input channels to N output channels costs about K·K·M·N·H·W operations. A depthwise-separable convolution does the same job in two cheap steps that cost roughly K·K·M·H·W + M·N·H·W. The ratio of the two is about 1/N + 1/K² — so for many output channels and a 3×3 kernel that is roughly an 8× to 9× reduction in compute.
Hardware accelerators & runtimes
Shrinking the model is half the story; the other half is using dedicated silicon and an efficient runtime. Modern devices ship NPUs (Neural Processing Units) built specifically for the multiply-accumulate-heavy math of neural nets — they are far more energy-efficient than running on the CPU.
Accelerators (the silicon)
- Apple Neural Engine — NPU in Apple A/M-series chips
- Qualcomm Hexagon NPU — Snapdragon AI accelerator
- Google Edge TPU — Coral devices, INT8 inference
- GPU — mobile GPUs via Metal / Vulkan / WebGPU
- Mobile CPU — the fallback; works everywhere, slowest
Runtimes (the software)
- TensorFlow Lite (LiteRT) — Android, MCUs (TFLite Micro)
- Core ML — Apple's on-device runtime
- ONNX Runtime — cross-platform, many backends
- llama.cpp / GGUF — quantized LLMs on CPU & more
- WebGPU / WebAssembly — models in the browser
The core tradeoff: accuracy ↔ size ↔ latency ↔ energy
Every edge decision moves along these four axes at once, and they pull against each other. You cannot maximize all of them — deployment is the art of finding the point that is “good enough” on accuracy while fitting the device's memory, latency, and battery budgets.
| Action | Size | Latency | Energy | Accuracy |
|---|---|---|---|---|
| Quantize (INT8) | ↓ 4× | ↓ | ↓ | ~ same (−1%) |
| Quantize (INT4) | ↓ 8× | ↓↓ | ↓↓ | slight drop |
| Prune | ↓ | ↓ (if structured) | ↓ | ~ same after fine-tune |
| Distill | ↓↓ | ↓↓ | ↓↓ | small drop |
| Bigger base model | ↑↑ | ↑↑ | ↑↑ | ↑ |
Common pitfalls
Watch out for
- Measuring only model size, ignoring activation/KV-cache RAM
- Aggressive INT4 quantization without calibration → accuracy cliff
- Unstructured pruning that does not actually speed up dense hardware
- Thermal throttling: fast for one inference, slow when sustained
- Assuming an NPU exists — many low-end devices fall back to CPU
Good practice
- Profile on the real target device, not a desktop
- Start with an efficient architecture, then quantize
- Use INT8 as the safe default; reach for INT4 only when needed
- Fine-tune after pruning/quantization to recover accuracy
- Match the runtime to the hardware (Core ML on iOS, TFLite on Android)
Key Takeaways
- Edge AI runs models on-device (phones, IoT, MCUs, browsers) instead of in the cloud — for low latency, privacy, offline use, and zero per-query server cost.
- The edge is resource-constrained: limited RAM, compute, battery/thermal, and storage bound what you can deploy.
- Quantization (FP32→INT8 = 4× smaller, INT4 = 8×), pruning, and distillation compress existing models; efficient architectures like MobileNet (depthwise-separable convolutions, ~8–9× less compute) are small by design.
- Dedicated accelerators (NPUs, Apple Neural Engine, Edge TPU) and efficient runtimes (TFLite, Core ML, ONNX Runtime, GGUF/llama.cpp, WebGPU) make on-device inference fast and energy-efficient.
- Deployment is a four-way tradeoff between accuracy, size, latency, and energy — the goal is the smallest, fastest model that is still accurate enough to fit the device.