Run an LLM Entirely in Your Browser: A Live WebGPU Demo

For two years the story of large language models was a story about data centers — racks of GPUs, API keys, and every prompt travelling across the internet to someone else’s server. This article tells a different story, and it proves it on the spot: a real language model that downloads into this browser tab and runs entirely on your own machine. No server. No API key. Nothing you type ever leaves your device. Scroll to the next section and try it.

1. An entire AI model — in this tab

The idea sounds implausible at first. Language models are supposed to be enormous, expensive things that live in the cloud. But a model does not have to be enormous to be useful, and your laptop or phone already contains a remarkably capable parallel processor: its GPU.

What changed in 2026 is that two pieces finally lined up. Browsers gained WebGPU, a standard way to run heavy computation on that GPU. And the open-source community produced genuinely small instruction-tuned models — in the hundreds of millions of parameters — that are coherent enough to be interesting. Put those together with a compiler that turns a model into GPU code, and the browser becomes an inference engine.

The consequences are not subtle. Inference that runs locally is private by construction, works offline, answers with no network latency, and costs nothing per request. Those are exactly the properties that matter for sensitive documents, regulated industries, and anything that should not depend on a third-party API. The demo below is small on purpose — but the architecture it shows scales all the way up to on-device assistants.

2. Try it: the Local AI Playground

Pick one of the smallest usable models, load it into this tab, and chat with it. The first load downloads the model once (you will see the progress); after that it is cached on your device and starts instantly. Everything you type is processed locally — you can prove that later by watching your network tab or pulling the plug.

A note on expectations: these are tiny models — from 135 million to 1.5 billion parameters, a fraction of the size of a frontier model. The smallest (SmolLM2-135M) is an English-centric curiosity that will make mistakes, lose the thread, and occasionally say something charmingly wrong; the multilingual Qwen models are noticeably more coherent. That contrast is the point. That is the point. You are watching the floor of what runs comfortably in a browser, on hardware you already own, for free. The rest of this article explains how it works and where this floor is genuinely useful.

3. How it works: WebGPU turns your GPU into an inference engine

A language model is, mechanically, a long sequence of matrix multiplications. GPUs exist to do exactly that — thousands of multiply-adds in parallel. The historical problem was that web pages had no way to reach the GPU for general math; WebGL could draw triangles, but it was never meant for compute.

WebGPU is the modern successor. It exposes the GPU through compute shaders — small programs that run massively in parallel — and gives a web page low-level, explicit control over buffers and pipelines. It is the browser-native cousin of native APIs like Vulkan, Metal and Direct3D 12.

The demo above is powered by WebLLM, which sits on top of the MLC/Apache TVM compiler stack. The model’s operations are compiled ahead of time into WebGPU shader code, so when you load a model the browser ships those shaders to your GPU and feeds the quantized weights through them. There is no Python, no CUDA, and crucially no server in the loop — the JavaScript runtime only orchestrates; the GPU does the arithmetic.

One practical detail matters for deployment: because WebGPU does its own thing on the GPU, this approach does not require the cross-origin isolation headers (COOP/COEP) that multithreaded-WebAssembly engines need. That means it runs on an ordinary static website with no special server configuration at all.

4. Tiny models, quantized: SmolLM2 and Qwen2.5 at 4-bit

The models in the playground are open-weight releases chosen because they are small enough to download in seconds-to-minutes and still follow instructions:

• SmolLM2-135M (Hugging Face) — the tiniest option, a research-grade model that’s astonishingly small but English-centric and decidedly quirky (great for “how tiny can it go?”, weak in German).
• Qwen2.5-0.5B and 1.5B (Alibaba) — strongly multilingual, handling German and English well. The 0.5B is the reliable sweet spot (the default); the 1.5B is the “smartest” option in exchange for a bigger download.

Even “1.5B” would be too large to ship comfortably at full precision. The trick is quantization. The Qwen variants here are q4f16_1: the weights are stored at 4 bits each (instead of 16- or 32-bit floating point) while computation runs in 16-bit. Four bits per weight cuts the download roughly four-fold versus fp16 — which is how a 1.5-billion-parameter model fits in ~1.1 GB instead of several. SmolLM2-135M is small enough that it ships only in 16-bit (~270 MB; there is no 4-bit build for it). There is a quality cost to quantization, but for these sizes it is a sensible trade. For the deeper mechanics, see QLoRA and quantization and Small language models and edge AI.

5. Download once, run offline forever

The first time you load a model, WebLLM fetches its weights from a public CDN and stores them in the browser’s Cache API / IndexedDB. That initial download is the only moment the network is touched. On every subsequent visit the model loads straight from local storage — the progress bar barely appears, and the “cached on this device” badge lights up.

This is what makes the privacy claim concrete rather than rhetorical. Once the weights are cached, you can turn off your Wi-Fi and the model keeps answering. Try it: load a model, disconnect, and keep chatting. Nothing breaks, because there was never anything to send.

(Browsers can evict cached storage under heavy pressure, in which case the weights re-download once — so the honest phrasing is “instant and offline once cached,” not “forever no matter what.”)

6. Reading the tokens/second meter

After each answer the playground shows a small stat line measured on your hardware. Two numbers matter:

• Prefill — how fast the model reads and processes your prompt.
• Decode — how fast it writes the answer, token by token.

A token is a chunk of text, usually a short word or word-piece; “tokens per second” is simply the generation rate. On a recent discrete GPU a 0.5B model can decode dozens to over a hundred tokens per second; on an integrated laptop GPU or a phone it will be slower. The point of showing the meter is honesty: you are seeing the real speed of your device, not a benchmark from a data center. For more on tokens and context, see Tokenization and context windows; for the broader picture of what drives inference speed, see LLM inference performance.

7. The privacy proof: zero network during inference

Most “private AI” claims ask you to trust a privacy policy. This one you can verify in under a minute:

• Watch the network. Open your browser’s developer tools, go to the Network tab, and send a prompt. After the one-time weight download, no requests fire while the model generates. The text appears out of pure local computation.
• Pull the plug. Disconnect from the internet entirely (once the model is cached) and keep chatting. If anything were being sent to a server, it would stop working. It doesn’t.

That is a categorically different guarantee from “we promise not to log your prompts.” There is no endpoint, no request, no copy of your text anywhere but the tab in front of you. For regulated data — health, legal, HR, internal documents — that distinction is the whole game. See Secure AI integration for how this fits into a wider data-protection strategy.

8. Limits — and honesty

In-browser models are not magic, and pretending otherwise would undercut the point. The real constraints:

• Capability. A model this small (135M–1.5B) has shallow world knowledge and weak multi-step reasoning. It is good at short rewrites, simple Q&A, classification and tone changes — not at research, maths or long chains of logic.
• First-load weight. The initial download (≈270 MB to 1.1 GB) is real friction on a slow connection. Caching fixes the second visit, not the first.
• Hardware variance. WebGPU coverage is high but not universal, and a low-memory device can run out of GPU memory on the larger model — which is why the demo suggests the smallest model when that happens.
• Context length. Tiny models keep only a short conversation in view before quality degrades.

None of this is a flaw in the demo; it is the honest shape of the technology. Knowing the floor is what lets you design for it.

9. When on-device inference is the right architecture

Browser-native inference is not a replacement for frontier models — it is a different tool for a different job. It is the right call when:

• Privacy is non-negotiable — sensitive text that must never reach a third-party server.
• The task is narrow — autocomplete, redaction, classification, tone-shifting, structured extraction, simple on-page assistants.
• Offline or low-latency matters — field tools, kiosks, in-app helpers that must respond instantly.
• Cost should scale to zero — millions of small interactions where per-API-call pricing would be absurd.

In production, the strongest pattern is usually hybrid: a small on-device model handles the common, private, latency-sensitive cases, and a larger cloud model is called only for the rare hard ones — with a routing layer deciding between them. The browser demo you just used is the smallest possible version of that idea, running on hardware you already own. For how on-device inference fits into a complete, data-sovereign architecture, see the sovereign AI stack 2026.

The takeaway is simple. AI does not have to live in someone else’s data center. In 2026, a meaningful amount of it can live in a browser tab — private, offline-capable, and free at the point of use. That changes which products are possible, and where sensitive data has to go. If that is the kind of system you want to build, that is exactly the kind we build.