On device and edge AI is production ready in 2026: run quantized LLMs in browser via WebGPU and on mobile with Flutter, LiteRT, and Core ML for private, offline inference.
For most of the last decade, "AI on the edge" meant a cramped image classifier or a wake-word detector — useful, but nobody's idea of intelligence. I built a couple of those myself and quietly filed them under "neat, not transformative." That era is over. In 2026 I can load a small language model into a browser tab over WebGPU, or run a quantized model directly inside a Flutter app, and get genuinely useful text out of it with zero network round-trips. The demos finally survive contact with production, and paying customers.
But "practical" is not the same as "always the right call." I've now shipped on-device inference into real apps at Shpper, watched it save real money, and also watched it heat up a tester's phone until the OS killed the process mid-sentence. So this isn't a hype piece. It's a field report on on-device and edge AI — what actually works today, where it bites, and how I decide between the phone in someone's pocket and a GPU in a datacenter.
Two independent things converged, and both had to land for this to stop being a science project.
First, quantization got genuinely good. Shrinking model weights from 16-bit floats down to 4-bit integers used to wreck quality — you'd get a model that hallucinated its own name. Modern schemes changed that. Instead of naively rounding every weight, they keep a per-group scale factor and lean on techniques that protect the weights that matter most (the "salient" ones that dominate a layer's output). This family of methods — weight-only quantization with per-group scales, activation-aware weighting, and the k-quant formats popularized by the llama.cpp ecosystem — keeps small models coherent at 4 bits while their memory footprint drops by roughly 4x. A 3B-parameter model that wanted serious server hardware in fp16 now fits in under 2 GB and runs on a mid-range phone or a browser tab without swapping itself to death.
Second, the runtimes matured into something you'd actually depend on. In the browser you now have real, boring, production-grade options: WebGPU gives you GPU-accelerated inference through transformers.js, WebLLM, or MediaPipe's LLM Inference API, with a WebAssembly (WASM) fallback for machines where WebGPU isn't available. On mobile you've got LiteRT (the runtime formerly known as TensorFlow Lite), ONNX Runtime Mobile, MLC-LLM, llama.cpp compiled for ARM, and platform-native paths like Core ML on iOS that tap the Neural Engine (the NPU). The plumbing to feed a model bytes and get tokens back is no longer the hard part. The hard parts moved up the stack — to UX, memory, and honest expectation-setting.
Be clear-eyed about the size classes, because they define what's realistic. Parameter count is the single best predictor of both what a model can do and what it costs you in RAM and heat.
If your feature needs frontier-level reasoning, none of these are the answer — and pretending otherwise is how you ship something that demos well and disappoints daily. A useful mental model: pick the smallest size class that clears the quality bar for one specific task, then stop. On-device is a discipline of narrowing scope, not chasing capability.
The browser story surprised me most, because I'd written it off. Here's roughly what pulling a small model into a page looks like with transformers.js:
import { pipeline } from '@huggingface/transformers';// Runs on WebGPU when available, falls back to WASM automatically.const generator = await pipeline( 'text-generation', 'onnx-community/some-small-instruct-model', { device: 'webgpu', dtype: 'q4', // 4-bit weights keep the download and RAM sane },);const output = await generator('Summarize this in one sentence: ...', { max_new_tokens: 64,});console.log(output[0].generated_text);That's the happy path. The two things that actually decide whether users keep the tab open are cold start and streaming.
The model has to be downloaded — tens to hundreds of megabytes depending on which size class you picked — and then compiled for the GPU before the first token. If you block your UI on that, you've lost. Cache aggressively (the browser's Cache API or the origin private file system will keep the weights around after the first visit), show an honest progress bar tied to real bytes, and pre-warm the model during idle time rather than the moment the user clicks. The transformers.js pipeline accepts a progress_callback — wire it to your loading state so people see movement instead of a dead spinner:
const generator = await pipeline('text-generation', modelId, { device: 'webgpu', dtype: 'q4', progress_callback: (p) => { if (p.status === 'progress') { updateBar(p.file, p.loaded / p.total); // real bytes, not a fake animation } },});Once warm, it's fast and completely offline. Do the heavy work in a Web Worker so tokenization and the WebGPU dispatch don't jank the main thread, and stream tokens with a TextStreamer so the first word appears in a couple hundred milliseconds instead of after the whole generation finishes. Perceived latency is the whole game here, and streaming is what wins it — a model that produces its answer word by word feels twice as fast as one that dumps the same text all at once after a pause.
import { TextStreamer } from '@huggingface/transformers';const streamer = new TextStreamer(generator.tokenizer, { skip_prompt: true, callback_function: (token) => appendToUI(token), // paint each token as it lands});await generator(prompt, { max_new_tokens: 128, streamer });One trap worth naming: feature-detect WebGPU (navigator.gpu) before you commit to the GPU path, and have a real plan for the WASM fallback — it works, but it's several times slower, and a 4B model on WASM will test your users' patience. Sometimes the right call is "WebGPU or bust": you show a graceful "your browser can't run this locally" message and fall back to the cloud. Decide that policy up front instead of discovering it in a bug report.
For Flutter — where I spend most of my days — the pattern is to keep the model runtime on the native side and expose a thin Dart interface. You do not want to marshal large tensors across the platform channel on every call. You load the model once, keep it resident, and stream results back over an EventChannel.
class OnDeviceModel { static const _method = MethodChannel('app/on_device_llm'); static const _events = EventChannel('app/on_device_llm/stream'); Future<void> load(String assetPath) => _method.invokeMethod('load', {'path': assetPath}); Stream<String> generate(String prompt, {int maxTokens = 128}) { _method.invokeMethod('generate', { 'prompt': prompt, 'maxTokens': maxTokens, }); // Native side pushes tokens as they're produced. return _events.receiveBroadcastStream().cast<String>(); } Future<void> unload() => _method.invokeMethod('unload');}The heavy lifting lives in a LiteRT, MLC-LLM, or Core ML session on the platform side; Dart just orchestrates. A few hard-won rules that separate a shipping feature from a one-star review:
maxTokens, throttle repeated calls, and remember that a thermally throttled phone gets slower mid-generation — your steady latency quietly degrades exactly when the user is leaning on the feature.At Shpper we had a feature that needed to categorize and clean up user-entered text — short, high-volume, privacy-sensitive. Cloud was the obvious first instinct. It was also the wrong one: every keystroke-adjacent call would have been a per-token bill, a network round-trip on flaky UAE-to-somewhere connections, and a pile of user data leaving the device for no good reason.
A ~1B quantized model on-device handled it. First-token latency dropped from a jittery cloud round-trip to a steady, predictable local response. Our inference bill for that feature went to literally zero — it runs on hardware the user already paid for. And the compliance conversation got dramatically shorter, because the honest answer to "where does this data go?" became "nowhere." That last point is worth more than the money.
Let me be blunt about where each side wins, because the marketing on both sides is dishonest.
Where on-device AI clearly wins:
Where cloud LLMs still win, and will keep winning:
I don't treat this as either/or, and neither should you. My default rule is boring and it works: match the model to the task, not the task to the model.
Reach for on-device inference when the job is narrow and well-defined and at least one of these is true: the data is sensitive, the feature must survive offline, latency needs to be rock-steady, or you call it so often that cloud cost would balloon. Concretely — text classification, smart replies, semantic search over local data, redaction and PII stripping, on-device transcription, and summarizing a document the user already holds on their device. These are jobs a small model does well, and the constraints reward keeping them local.
Reach for the cloud when you need real reasoning, large context windows, genuinely current knowledge, or a uniform experience across every device — and the network cost and privacy profile are acceptable. Don't torture a 3B model into a task that wants a frontier model. It'll technically respond, and it'll quietly be wrong in ways that erode trust.
The best architecture is often both tiers, cooperating. A pattern I keep reaching for:
This routing layer is where the engineering leverage lives. Get the on-device/cloud split right and you get instant UX, lower spend, and a stronger privacy story all at once — which almost never happens in this job, where usually you pick two. Keep the seam between the tiers clean and explicit (one interface, two implementations behind a router) so you can move the boundary later without a rewrite.
I'm not going to pretend I have a crystal ball, but the direction is clear enough to plan around. Phone NPUs keep getting faster and the runtimes keep getting better at using them, so the size class that runs comfortably on-device drifts upward every year — today's painful 7B is next year's comfortable one. Quantization keeps improving, squeezing more quality out of fewer bits. And the browser is quietly becoming a first-class inference target, which means "install nothing, runs locally, works offline" stops being a contradiction.
None of that closes the capability gap with frontier cloud models — that gap moves too, because the big models keep getting bigger. What changes is the floor. Every year, more of the "just needs to be good enough, fast, and private" work slides onto the device, and the cloud gets reserved for the genuinely hard problems. Build your architecture so that boundary can move without a rewrite, and you'll age well.
transformers.js in the browser; LiteRT, ONNX Runtime, MLC-LLM, and Core ML on mobile).TextStreamer.On-device and edge AI graduated from party trick to production tool, and in 2026 that's not aspirational — it's shipping in apps I maintain. It won't replace frontier cloud models; the capability gap is real and I'm not going to hand-wave it away. But for narrow, latency-sensitive, privacy-critical, or offline-first features, keeping inference on the device is often the better engineering choice, not merely the cheaper one.
So profile your actual task before you reach for an API key. If a small quantized model handles it, keep it on the device — you'll get instant latency, zero marginal cost, and a privacy story that makes hard conversations easy. Save the cloud for the problems that genuinely need a bigger brain, build a clean seam between the two tiers, and let them cover for each other. That's not a compromise. That's just good architecture.