AI Daddy › Inference Optimization
On-Device and Edge Deployment · Inference Optimization
Not every model has to run in someone else's cloud. Running LLMs locally, on a laptop, a workstation GPU, a phone, or an edge box, is a real deployment…
On-Device and Edge Deployment
Not every model has to run in someone else's cloud. Running LLMs locally, on a laptop, a workstation GPU, a phone, or an edge box, is a real deployment target in 2026, driven by privacy, offline operation, latency, and cost at steady volume. The catch is that the tool that makes local models easy to try (Ollama) is not the tool that serves them in production (vLLM), and the two get conflated constantly. This chapter sorts out the runtime stack, the prototype-to-production path, the hardware limits, and when local actually beats an API.
Table of Contents
The Runtime Stack
The key mental model: these tools are not substitutes. They occupy different layers.
| Tool | Layer | What it is for |
|---|
| Ollama | Experience layer / local daemon | One-command model pull and run, OpenAI-ish API, single-user dev. Builds on llama.cpp, and uses Apple MLX on Apple Silicon in recent versions. |
| LM Studio | Experience layer / GUI | A desktop GUI for browsing and running local models. Single-user focused. |
| llama.cpp | Inference engine | Portable C/C++ CPU/GPU inference (the GGUF format); runs almost anywhere; powers the experience-layer tools. |
| MLX | Inference engine | Apple's array framework; the fastest Apple Silicon path; research and fine-tuning. |
| vLLM | Serving system | High-throughput concurrent serving with PagedAttention and continuous batching; OpenAI-compatible. The production answer. |
| TGI / TensorRT-LLM | Serving system | Hugging Face's and NVIDIA's high-throughput servers; production. |
| ExecuTorch | Embedded/mobile runtime | PyTorch-native on-device inference (phone to microcontroller); reached 1.0 in late 2025 and ships in apps for billions of users. |
| Core ML / ONNX Runtime / MLC LLM | Embedded/mobile runtime | Apple on-device, cross-platform, and compile-to-many-targets (including browser/WebGPU) respectively. |
Why Ollama Is Not a Production Server
Ollama and LM Studio are excellent for prototyping and wrong for a shared production endpoint, for an architectural reason worth teaching.
Ollama defaults to serving requests with very limited parallelism and queues excess requests first-in-first-out; a full queue returns an error. Each parallel slot also statically multiplies the context memory allocation. LM Studio is built for single-user scenarios without rate limiting or auth. Neither is designed to turn concurrent demand into throughput.
vLLM is, via two mechanisms: PagedAttention (the KV cache stored in non-contiguous blocks like OS paging, cutting the 60-80% KV memory waste of naive serving to under ~4%) and continuous batching (swap a finished request out and a queued one in mid-batch). The clearest first-party benchmark, from Red Hat: on a single datacenter GPU running an 8B model, vLLM reached roughly 793 tokens/sec versus Ollama's ~41, about 19x, with far lower tail latency, and even a tuned Ollama trailed across all concurrency levels.
The takeaway is not "vLLM is tuned better." It is structural: Ollama and LM Studio serialize, vLLM batches continuously and pages the KV cache. For one user the difference is small; under concurrency it becomes roughly 16-20x. One caveat for honesty: most public head-to-head numbers run on a datacenter GPU to isolate the software difference, so do not read "vLLM beats Ollama" as "GPU beats Mac."
When Local Beats Cloud (and When It Does Not)
Lean local or edge when:
- Privacy or regulated data that cannot leave the box (HIPAA, GDPR, contractual residency). A caveat to teach: the major API providers now offer zero-data-retention enterprise tiers, so "privacy" alone no longer automatically decides for local.
- Offline or air-gapped operation (field devices, critical infrastructure).
- A latency floor: on-device removes the network round trip (often 50-200ms), which matters for tight interactive loops; total response latency still depends on the model and hardware.
- Cost at steady, high volume: a reserved GPU is reported to break even against frontier APIs somewhere around a few million tokens per day, above which owning hardware wins because you stop paying per token. The exact break-even is workload-dependent.
Stay on cloud APIs when you need frontier quality, have spiky or unpredictable demand (you would pay for idle GPUs; batch endpoints at ~50% off often beat local at medium volume), run low-to-moderate volume (below break-even, total cost favors APIs), or lack the ops capacity to run vLLM with autoscaling and monitoring. The 2026 consensus is usually a hybrid: small, private, offline, or cost-sensitive paths local, heavy or frontier or spiky paths to the cloud, within one product.
Quantization for Local Serving
Quantization is what makes local serving viable; the Quantization Deep Dive covers the math, so here is just the deployment layer.
GGUF is the local-model format used by llama.cpp, Ollama, and LM Studio. Common quant levels trade quality for size: Q4_K_M is the practical sweet spot (roughly 1-3% quality loss versus FP16 at about a quarter of the size), Q5_K_M is noticeably better for code and reasoning at under ~1% loss, Q8_0 is effectively lossless at about half FP16, and Q2/Q3 save the most memory but degrade math and reasoning by 5-10% or more.
The VRAM rule of thumb:
VRAM (GB) ≈ (params in billions × bits per weight) / 8 # model weights only
then add the KV cache (it grows with context length times concurrent requests) plus roughly 10-20% runtime overhead. So a 7B model's weights are roughly 14GB at FP16, ~7.7GB at Q8_0, and ~4.5GB at Q4_K_M, before that overhead. The operating rule everyone repeats: use the highest-quality quant that fits with 10-20% headroom for KV cache, activations, and context.
Hardware
A model-size-to-hardware guide (Q4 quant assumed; planning guidance, not guarantees):
| Model size (Q4) | Min VRAM/RAM | Realistic hardware |
|---|
| 1-3B | 4-6 GB | Any modern GPU; high-end phones (NPU); AI PCs |
| 7-8B | 8 GB | Mainstream GPU; 16 GB Mac |
| 13-14B | 12 GB | Upper-mainstream GPU; 16-24 GB Mac |
| 32-35B | 24 GB | A 24 GB consumer GPU; 36-48 GB Mac |
| 70B | ~40 GB+ | High-end or dual GPU; 64 GB+ Mac; or a datacenter card |
| 200B+ | 48 GB+, often multi-GPU / 128 GB+ unified | Multi-GPU rigs; large-unified-memory workstations |
Notes:
- Consumer GPUs top out at 24-32 GB of VRAM, which is the binding constraint on local model size.
- Apple Silicon shares one memory pool between CPU and GPU, so system RAM doubles as VRAM, letting a large-RAM Mac hold models a same-priced discrete GPU cannot. Apple's MLX path keeps improving: Ollama's MLX backend (preview) reports sizable prefill and decode gains on Apple Silicon from exploiting unified memory, and a separate update adds NVFP4, NVIDIA's 4-bit floating-point format (not Apple's), reported around 20% faster than Q4_K_M.
- NPUs in phones and AI PCs advertise high TOPS, but a teaching nuance: TOPS alone does not predict LLM speed, because limited operator support and memory bandwidth gate real performance. NPUs suit lightweight, battery-efficient tasks; discrete GPUs still win for heavy local inference.
- Mobile is bandwidth-bound and memory-constrained: realistic on-phone models are sub-1B to about 3B, available app RAM is often under 4 GB even on flagships, and mobile memory bandwidth is 30-50x below a datacenter GPU. The on-device standard is 4-bit quantization.
Prototype to Production
- Prototype with Ollama (CLI) or LM Studio (GUI) on a GGUF Q4_K_M model; validate quality and prompts on the smallest model that passes.
- Pick the largest model and best quant that fits the target hardware with KV-cache headroom.
- Switch the serving engine for any concurrent endpoint: vLLM (NVIDIA or AMD), TensorRT-LLM (max NVIDIA), or TGI. Keep the OpenAI-compatible API so application code barely changes.
- For mobile or edge, export to ExecuTorch, Core ML, or ONNX Runtime / MLC LLM, quantize to 4-bit, and budget for under 4 GB of RAM and the bandwidth limit.
Common pitfalls: treating Ollama or LM Studio as a server (it serializes under load); forgetting the KV cache when sizing memory (long context times parallel slots can dominate); over-quantizing (Q2/Q3 hurts reasoning); conflating "vLLM beats Ollama" with "GPU beats Mac"; assuming NPU TOPS equals LLM speed; and mismatching engine to hardware (vLLM is GPU-centric, MLX is Apple-only, llama.cpp is the portability fallback).
Maturity: server-side local serving is production-mature (vLLM is widely deployed with an OpenAI-compatible API). On-device and mobile is production-ready for small models (sub-1B to 3B) and not for frontier ones. NPU-as-LLM-engine is still early; a discrete GPU and a large-unified-memory Mac remain the serious local paths in 2026.
Interview Questions
Q: A team prototyped on Ollama and wants to ship it as a shared API. What changes and why?
Strong answer:
Ollama is the wrong tool for a shared endpoint. It serves with limited parallelism and queues excess requests first-in-first-out, so under concurrency latency spikes and requests start failing. The fix is to switch the serving engine to vLLM (or TensorRT-LLM or TGI), keeping the same OpenAI-compatible API so the app barely changes. vLLM wins structurally, not by tuning: PagedAttention stores the KV cache in non-contiguous blocks to eliminate most of the memory waste, and continuous batching swaps finished requests out and queued ones in mid-batch, so concurrent demand becomes throughput. First-party benchmarks show roughly an order-of-magnitude higher throughput and far lower tail latency under load. I would also right-size the model and quant to the target GPU with KV-cache headroom, and add autoscaling and monitoring, which Ollama does not provide.
Q: When would you choose local or on-device inference over a cloud API?
Strong answer:
When data cannot leave the box for privacy or residency reasons, when the system must work offline or air-gapped, when I need the lowest possible latency by cutting the network round trip, or when I have steady high volume where a reserved GPU beats per-token pricing, which is reported to break even around a few million tokens a day. I would stay on an API for frontier quality, spiky demand where idle GPUs waste money, low volume below break-even, or when the team lacks the ops capacity to run a serving stack. In practice it is usually a hybrid: small, private, or offline paths run local on quantized models, and heavy or frontier or bursty paths go to the cloud. On phones specifically, I would plan for sub-1B-to-3B models, since mobile is memory and bandwidth constrained.
References
Next: Prompt Engineering Fundamentals