Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s
Why CEREBRO kept it
LLM inference optimization, token throughput on constrained hardware
The text below is an automated extraction of the article at https://github.com/carloslfu/slotstream, stored verbatim in the public cerebro-vault repository. Copyright remains with the original publisher (github.com).
Run Qwen3.8-Flash-Next on a Mac that can't hold it. The model is a 125B-parameter mixture-of-experts, 104 GB on disk at 4-bit; slotstream streams it from SSD and runs it in whatever memory you give it. It's one Swift binary, no Python. It speaks the Ollama and OpenAI chat APIs, so your existing tools work unchanged. | on a 48 GB M5 Pro | | |---|---| | Warm decode | ~12 tok/s | | Engine start | ~2 s (only the 3.8 GB trunk loads) | | Peak memory | 32 GB (auto-sized; you can cap it) | | Weights on disk | 104 GB | You need Apple Silicon, macOS 14+, and ~110 GB of free disk. Disk bites first: w
Community take
Five similar MoE offloading repos exist; fragmentation into isolated reimplementations wastes effort—consolidation and upstream collaboration would deliver more value than another separate version.
Backlinks
Appeared in 1 briefing