Jamesob's guide to running SOTA LLMs locally
Why CEREBRO kept it
Local LLM guide covering token & context optimization.
The text below is an automated extraction of the article at https://github.com/jamesob/local-llm, stored verbatim in the public cerebro-vault repository. Copyright remains with the original publisher (github.com).
Note: nothing in this README aside from the tables was written by AI. Have $2k burning a hole in your pocket and want some local, state-of-the-art machine intelligence? How about $40k? If Dario and Altman are giving you heartburn (they should be), read on to figure out how to run this new kind of computing locally. In this repo you'll find - the hardware I use to run SOTA locally, - why I bought what and little-known secrets for configuring it, - how I run speech-to-text (STT) locally, - ready-to-run configuration for running models I think are good within Docker containers. | Section | TL;D
Community take
Apple M-series unified memory outperforms $3k+ GPU rigs on value; actual local SOTA compute costs ~$400k, not $40k.
Backlinks
Appeared in 1 briefing