On-device AI in 2026: what fits on a laptop

Running AI models on a laptop has gone from a hobby project to something you can reasonably plan a product around. On-device AI in 2026 is good enough for summarising, rewriting, searching your own files, describing images and a fair amount of coding help, all without sending anything to a server. But the practical limits are real, and they are mostly about memory, not about how clever the model is. This post is our working notes on what fits on a laptop and why.

The numbers below are approximate and rounded. Model formats and runtimes change quickly, and your results will depend on the exact model, the runtime and what else your machine is doing. Treat them as a way to reason about the problem, not as benchmarks.

Quantisation: how big models get small

A language model is mostly a very large set of numbers called weights. A model described as "8B" has around eight billion of them. In their original training format each weight usually takes 16 bits, or two bytes, so an 8B model needs about 16 GB just to hold its weights. That is already every byte on a 16 GB laptop, before the operating system has loaded.

Quantisation stores each weight with fewer bits. At 8 bits the same model needs roughly 8 GB. At 4 bits it needs roughly 4 GB, plus a little overhead for the scaling values that let the runtime recover approximate original numbers. Popular 4-bit formats in practice work out at around 4.5 to 5 bits per weight on average, so an 8B model lands somewhere near 5 GB.

The surprising part is how little quality you lose. Going from 16-bit to 8-bit is close to invisible for most tasks. Going to 4-bit costs a little more, and the loss shows up first on precise work such as maths, code and following long, detailed instructions. Below 4 bits, quality drops faster, and a smaller model at 4-bit is often a better choice than a larger model squeezed to 2 or 3 bits. For most laptop use, 4-bit is the sensible default.

Weights are not the whole story. While a model generates text it keeps a cache of the conversation so far, often called the KV cache. It grows with the length of the context. For an 8B-class model, 8,000 tokens of context can take roughly another gigabyte, and much more for long documents or larger models. Leave room for it.

Memory bandwidth matters more than compute

When people compare chips for AI they often look at compute figures, measured in TOPS or TFLOPS. For running a language model on one machine for one person, the more important number is usually memory bandwidth.

Here is why. To generate each new token, a dense model has to read essentially all of its weights from memory once. The actual arithmetic is quick. So the speed limit is how fast the chip can stream those gigabytes. A useful rule of thumb:

tokens per second (upper bound) ≈ memory bandwidth ÷ model size in memory

120 GB/s ÷ 5 GB (8B at 4-bit)  ≈ 24 tokens/s at best
120 GB/s ÷ 19 GB (32B at 4-bit) ≈ 6 tokens/s at best

Real numbers come in below the upper bound, often somewhere between half and three quarters of it. But the rule explains most of what you see. Doubling the bandwidth roughly doubles generation speed. Halving the model size through quantisation does the same.

Compute does matter for the other half of the work, which is reading your prompt. Processing a long prompt or a big document happens in parallel and is limited by raw compute rather than bandwidth. That is why a laptop can feel quick when chatting but slow to start when you paste in a fifty page PDF.

One more wrinkle: some newer models use a mixture-of-experts design, where only a fraction of the weights are used for each token. They generate faster than their total size suggests, because less has to be read per token, but they still need enough memory to hold every weight.

Unified memory on Apple silicon

This is where laptop architecture makes a big difference. On most Windows and Linux laptops with a separate graphics card, the GPU has its own memory, often 8 GB, sometimes 12 or 16. That memory is very fast, but a model that does not fit in it has to spill into slower system memory, and performance falls off a cliff.

Apple silicon Macs use unified memory. The CPU, GPU and Neural Engine share one pool, so a MacBook with 32 GB can give most of those 32 GB to a model running on the GPU. Bandwidth also scales with the chip tier: recent base chips offer somewhere around 100 to 150 GB/s, Pro chips roughly twice that, and Max chips around 400 to 550 GB/s. The exact figures vary by generation, and Apple lists them on each chip's specification page.

By default macOS keeps part of that memory back for the system, so the GPU cannot use the whole pool. On a 16 GB machine, plan on something like 10 to 11 GB being realistically available for a model while you keep a browser and an editor open.

Some newer Windows laptops take a similar approach, with large pools of fast memory shared between CPU and GPU. It is the design, not the brand, that matters here.

Where NPUs fit

Most new laptops now include a neural processing unit, a block of the chip designed for AI workloads at low power. Apple has had the Neural Engine for years. Recent Intel, AMD and Qualcomm laptop chips all have NPUs as well, and Microsoft's Copilot+ PC programme set a bar of 40 TOPS or more.

NPUs are very efficient for small models that run constantly, such as background blur on video calls, live captions, speech recognition or on-device image classification. For larger language models the picture is more mixed. The popular open-source runtimes for local language models mostly run on the GPU and CPU, and NPU support differs between vendors and tools. That is improving, but as of mid 2026 we would not buy a laptop for its NPU TOPS figure if running language models is the main goal. Memory size and bandwidth are the numbers to look at.

What fits on 16 and 32 GB

The chart at the top of this post shows approximate weight memory for common model sizes. Here is how we read it, with the usual caveats: 4-bit quantisation, moderate context lengths and a normal set of other apps open.

• 8 GB. Models of 1B to 3B parameters. Good for autocomplete, classification, short rewrites and extracting data from text. Anything bigger will fight the operating system for memory.
• 16 GB. Models around 7B to 8B run comfortably. This is the tier where local chat and summarising become genuinely useful. A 12B to 14B model can load, but it will be tight and slower.
• 32 GB. Models around 14B run comfortably, with room for longer contexts. Models around 30B at 4-bit are possible and often very capable, but they take most of the machine.
• 64 GB and up. A 70B-class model at 4-bit needs around 40 GB for weights alone. It works, at modest speed, on high-memory machines with good bandwidth.

If you are buying a laptop now and think local models will matter to you, memory is the thing to over-buy, because on most thin laptops it cannot be upgraded later. We made the same point about development machines in general in our post about the quiet desk.

Privacy, and when cloud still wins

The strongest argument for on-device AI is privacy. When a model runs locally, your documents, photos and notes never leave the machine. There is no server log to leak, no retention policy to read and no question about your data ending up in a training set. It also works offline, and it has no per-request cost, which changes what you are willing to try. Running a model over every file on a disk is unthinkable at cloud prices and unremarkable on your own laptop.

That is why we think local models belong in a lot of ordinary desktop apps, doing small, specific jobs well rather than trying to be a general assistant.

Cloud models still win in several places. The largest frontier models are far beyond what any laptop can hold, and they are noticeably better at hard reasoning, long complex tasks and broad knowledge. Very long contexts, hundreds of thousands of tokens, are impractical locally because of the cache size. Heavy use also costs battery: a laptop running a large model flat out will get hot and drain quickly, while a request to a data centre costs your battery almost nothing.

For product design, the split we keep coming back to is simple. Private, frequent, small tasks run locally. Rare, hard tasks go to the cloud, and only when the user has chosen to send them. Getting that boundary right, and making it visible in the interface, is most of the work.