Your Mac isn't short of memory. It's short of bandwidth.
I run a 26 billion parameter model on a laptop. Only 4 billion of it works on any given word, and that unlocks a complete new world.
Your Mac isn’t short of memory
I can run large models, since I have 128 GB of memory. But the bottleneck is the memory bandwidth. The larger models take longer to produce an output token, because there is more work to do per token, and the work scales with the parameter count.
The way I picture that is that you have to do lots of matrix multiplication and calculations, given each token. The bigger the model, the more maths calculations you have to do. You have to send them always back and forth between the GPU, CPU, and the unified memory. This sending back and forth is then becoming a bottleneck.
Capacity decides whether a model runs at all. Bandwidth decides how fast it answers.
What a mixture of experts actually is
Imagine you have all the parameters in your memory, and you can also group them. These groups are called experts. What the mixture of experts model does is, per task, it only uses a subset of them.
Imagine a room full of 100 experts, and you have a question. You would need to walk up to everyone and ask them the question and get the answer. Say they are super fast and it takes a minute each. That is a hundred minutes. If you only ask a subgroup of those experts, let’s say just four, then you would only need four minutes to already have a pretty good answer without asking all 100.
The picture is not quite that tidy in practice: the experts are not neatly labelled, and a model does not keep one for maths and one for French. The router picks by patterns nobody chose, and researchers argue about how specialised the experts really are.
The other kind is dense: every parameter works on every word, every time.
You can have a stellar CPU and GPU and enough memory, but you’ll always be bottlenecked by the memory bandwidth.
So 26b-a4b tells me that we have, in total, 26 billion parameters sitting in memory. However, at any given time, only 4 billion will be active. That is why those mixtures of experts work better, in my understanding, because you have to send less back and forth.
What it buys, and what it costs
Honestly, when I learned that only four billion parameters are active, I was quite surprised how well it still works. And by how much faster it is versus a dense model.
The research says roughly what you would expect, with one caveat. Judged against a dense model of the same total size, a mixture tends to come out slightly behind, because a lot of what it is carrying sits idle. Judged against a dense model of the same active size, it comes out ahead. Epoch AI puts the inference economics that way and the main survey agrees, though it is a tendency rather than a law: at least one paper finds a mixture beating its dense counterpart on equal resources.
The one place people say dense wins is coding. I have not tested that, and the source I took it from does not actually compare the two. It just ranks the 31 billion dense model above the 26 billion mixture on quality. So treat it as a rumour until somebody runs it.
What I can vouch for is text at scale. I’m running this massive tweet sentiment pipeline, just for you to get a sense of the sheer scale that we have here. We run 2B tokens/month, and that’s only on nightly runs. The task is basically processing about half a million tweets per night in various languages: Polish, Portuguese, English, standard German, Japanese, Chinese and Arabic. It doesn’t struggle to process those.
The other thing it manages to do is to pair that sentiment with a geopolitical market on Polymarket. One from June: the market asked whether 60 ships would transit the Strait of Hormuz on any day by 30 June. Fox News tweeted that 69 vessels had passed through on the Wednesday, the highest since 1 March. The model read the tweet, pulled out the number, and worked out that it mattered for that market. The information was not priced in yet. That one made money.
Which one to pick
- Start with what fits. Memory capacity decides whether a model runs at all. Nothing else matters until it does.
- Use a dense model if throughput is not your problem. If you are not really bottlenecked by high-throughput token usage, dense is the simpler choice.
- Use a mixture when volume is the constraint. Two billion tokens a month is when it stops being a preference.
- Try them both. Get a sense of the feel for the way they perform on your own task. Nobody’s table beats ten minutes with your own prompts.
What comes next
I still haven’t tried to use the dense models for coding, or the mixture models for coding. What I would need is a side-by-side: a small coding task, run once with the dense model and once with the mixture, and then judge the outcome.
There is also a distillation corner of this world that trains small models to match the big frontier ones inside one narrow domain, and at least one startup will do it on demand.
Next: nine runs on my own Mac. Three model sizes, three compression tricks, and the tokens per second to go with them.
Which model have you never actually measured?
Sources
- Epoch AI, MoE versus dense models at inference — the same-total and same-active comparison.
- A Survey on Mixture of Experts in LLMs — sparse activation trades some quality for compute.
- Dense vs Sparse Pretraining at Tiny Scale — a controlled matched-active and matched-total comparison, at very small scale.
- MoE Can Surpass Dense LLMs Under Strictly Equal Resource — the counter-result, and the reason this is a tendency rather than a law.
- The Myth of Expert Specialisation in MoEs — why the experts are not neatly labelled.
- Distil Labs, how Knowunity cut its LLM bill — distillation into a narrow domain.
Several of these are preprints rather than peer-reviewed papers. Read them before you rely on them.
Get the next episode first
Every week: the progress unfiltered, plus what I can't post publicly. The build, the number, green or red.