CPU vs GPU Mental Model

A CPU is a latency machine: a handful of very fast cores, huge caches, deep branch predictors — built to finish one unpredictable thing as soon as possible. A GPU is a throughput machine: thousands of simple cores that are individually slow, but all execute the same instruction over different data in lockstep.

The programming consequence: you stop writing loops and start writing the body of the loop. You describe what happens to element i, and the hardware runs that description for every i at once. Work that fits this shape — image filters, physics, matrix math, neural network layers — gets a 10–100× speedup. Work that branches per element or chains dependent steps does not.

The Pipeline

Every WebGPU program is the same five-step ritual. The API is verbose because it is explicit— all the validation happens when you build the pipeline, so the per-frame or per-inference hot path is just "set, dispatch, submit".

Where the Data Lives

GPU memory and the JS heap are separate worlds. Crossing between them is the expensive part, which is why every serious WebGPU program is shaped the same way: copy in once, run many passes on-GPU, copy out once.

Your First Compute Shader

The canonical hello-world: double a million floats. Two files of mental model — the TypeScript side builds and launches the pipeline, the WGSL side is the loop body that runs once per element.

The Shader

WGSL is WebGPU's shading language — Rust-flavored, strictly typed, validated in the browser. @workgroup_size(64) declares 64 threads per workgroup; the bounds check exists because the dispatch rounds up.

Reading Results Back

Workgroups& Invocations

Threads are launched in a two-level grid. The shader declares a workgroup size (threads per group, up to 256); the dispatch call supplies the grid of workgroups. Total threads = workgroup size × dispatch counts, in up to three dimensions.

Two Coordinate Systems

Matrix Multiply = AI

Strip a transformer to its skeleton and it is matmul all the way down: attention scores, projections, and feed-forward layers are all A × B on large matrices. In a typical LLM forward pass, well over 90% of the FLOPs are matrix multiplies— which is why "can the browser do AI" reduces to "can the browser do fast matmul".

Naive vs Tiled

The naive kernel is compute-cheap but bandwidth-starved: every output cell re-fetches an entire row of A and column of B from slow global memory. The classic fix is tiling — each workgroup copies a block of A and B into shared memory once, synchronizes, and lets all its threads reuse the block.

LLMs in the Browser

With WebGPU compute underneath, an LLM runtime in a tab is just an engineering problem: fetch quantized weights, upload them to GPU buffers once, and run the decode loop as compute passes. Three stacks dominate:

Streaming Tokens

The Quantization Ladder

Model weights are just numbers, and numbers can be stored at different precisions. Quantization rounds fp32 weights down to fp16, int8, or int4 — trading a small accuracy loss for a 2–8× smaller memory footprint and proportionally faster inference, because LLM decoding is bound by how fast weights stream from memory.

This chart is the entire feasibility argument for on-device AI. The same 8B model that needs a datacenter card at fp32 fits beside your browser tabs at int4 — and a 1–3B model at int4 fits on a phone. Perplexity loss from fp16 to int4 is measurable but small; for chat, summarization, and extraction tasks most users cannot tell.

Privacy & Cost Wins

Running the model where the data already lives changes the product conversation, not just the architecture diagram.

When the Cloud Still Wins

Ship Checklist

WebGPU demos are easy; WebGPU products meet users on five-year-old hardware with sixteen tabs open. Work through the list — it is the distance between the two.