I ran 120B MoE models on my 32GB M1 Max, using under 16 GiB

I've been trying to run large language models on my MacBook Pro. It has an M1 Max and 32GB of RAM, so a 60GB model obviously won't fit entirely in memory.

I wanted to see how far I could get with a different approach: keep most of a mixture-of-experts (MoE) model on the SSD, bring in what it needs, and put an actual limit on how much memory the inference process can use. Not just a number in a configuration file, but something I could check against macOS's own memory counters.

I ended up with a prototype that can run gpt-oss-120b and Qwen3.5-122B-A10B with a 16 GiB process-memory budget. It also has a small browser chat and an OpenAI-compatible local API. I'm not releasing the engine code yet, but I wanted to share the results and, perhaps more importantly, the mistakes I made along the way.

The useful numbers

On gpt-oss-120b, using the original MXFP4 experts and BF16 non-expert weights, I compared my engine against TUFF 10.2.2 on the same Mac. I first made sure the weights matched: I checked a sample of 144 expert tensors byte-for-byte, and all 471 non-expert tensors matched the files TUFF loaded.

Across eight prompts and three alternating rounds, my engine averaged 7.84 tokens/s, versus 5.78 tokens/s for TUFF. That's about 36% faster in this setup. The advantage showed up on every prompt, although the amount varied.

The difference got bigger when another process used some of the laptop's memory. That process held 4 or 8 GiB and touched all its allocated pages every five seconds. I repeated the tests three times at each level, alternating which engine ran first.

At 4 GiB of competing load, my engine lost about 3% of its speed; TUFF lost about 23%. At 8 GiB, both suffered, as you'd expect on a 32GB machine.

What surprised me about memory

At first glance, TUFF looks much lighter in Activity Monitor: roughly 7.4 GiB of process footprint, versus 15.6 GiB for my engine. But that isn't the whole picture.

During the tests, TUFF filled 14.5–15.2 GiB of the macOS file cache with model data, and system-wide compressed memory rose from roughly 2.4 GiB at idle to 9.3–9.6 GiB. My engine kept the model-file cache nearly empty, with compressed memory around its idle level. Both reduced the system's free memory by roughly 17–18 GiB, and neither caused swap usage to increase.

I also measured how long the competing process needed to make one pass over its own 4 GiB allocation. With my engine running, the median was 4 ms. With TUFF, it was 352 ms. The 95th-percentile values were 146 ms and 606 ms, respectively.

That was the result I hadn't expected. I started out trying to make the model faster, but for a laptop, whether the rest of the machine stays responsive may matter just as much.

There is a limit to this: with 8 GiB of competing memory use, both engines made the other process slow (around 1.3 seconds median). This is not a claim that my engine somehow creates extra physical memory.

I had to throw away some good-looking results

One early benchmark looked better than it really was. macOS was quietly keeping about 8 GB of model data in the OS file cache, outside the inference process's footprint. I thought I was measuring SSD reads, but some of those reads were coming from RAM.

After I fixed the test setup, performance dropped. Since then I've checked model-file residency with mincore, disk traffic against both kernel counters and iostat, and the process's physical footprint using macOS kernel measurements rather than RSS alone.

Correctness was another rabbit hole. An MLX multiplication kernel could take a different numerical path depending on how the expert tensors were arranged in memory. I also had a one-token bug in prefix-cache reuse. Both bugs were subtle enough that normal chat output looked fine.

I now compare complete generated answers against unmodified mlx-lm, using the same input tokens and prefill/cache schedule. In the current reference suite, all 9 of 9 test cases matched for each 120B-class model, including generated token IDs and hashes of the logits. I also ran a 200-request stress test for gpt-oss (17,952 matching MoE decode checks) and 100 cancel-and-retry cycles without finding corruption. Those are test results, not a proof that every possible input is bit-exact.

A few important caveats

This is still one Mac, one request at a time, and mostly decode benchmarks. TUFF ran via its CLI with explicit cache settings because its server's automatic configuration wouldn't serve this model on my machine. My engine ran as a persistent API server. TUFF also adds its own system prompt, so we did not compare the texts of its answers with mine. An earlier single-run test showed TUFF prefilling a long prompt faster, by about 39%.

I also learned the hard way why comparing different quantizations is misleading. My first comparison used 8-bit non-expert weights while TUFF used BF16, and the apparent speed advantage was larger. The results above are from the later test using matching BF16 non-expert weights. The Qwen speed measurements from an earlier stage of the project are not included in this head-to-head; they need another pass after a correctness fix that slowed its prefill.

Expert offloading itself isn't new. I'm not claiming to have invented it. My focus has been on making it work with current large MoE models, keeping the memory budget predictable, and checking that the output really matches the reference implementation.

Where I go from here

I'd like to test this on other Apple Silicon machines, especially ones with different RAM sizes and SSD speeds. I'd also like an independent comparison with other inference backends. So far, all the numbers come from my own laptop, scripts and logs.

I'm still deciding what to do with the project. I could keep developing it, find people interested in working on it, or leave it as an experiment for a while. For now, I thought the results were worth sharing.

If you work on local inference, MoE models or Apple Silicon, I'd genuinely like to hear what you'd test next. I'm also open to talking about independent evaluation or collaboration.