WebLLM: high-performance in-browser LLM inference engine
Why CEREBRO kept it
In-browser LLM inference, enables agentic patterns.
The text below is an automated extraction of the article at https://github.com/mlc-ai/web-llm, stored verbatim in the public cerebro-vault repository. Copyright remains with the original publisher (github.com).
High-Performance In-Browser LLM Inference Engine. Documentation | Blogpost | Paper | Examples WebLLM is a high-performance in-browser LLM inference engine that brings language model inference directly onto web browsers with hardware acceleration. Everything runs inside the browser with no server support and is accelerated with WebGPU. WebLLM is fully compatible with OpenAI API. That is, you can use the same OpenAI API on any open source models locally, with functionalities including streaming, JSON-mode, function-calling (WIP), etc. We can bring a lot of fun opportunities to build AI assistant
Community take
Project effectively abandoned with WebGPU incompatibilities across platforms and no updates since Gemma 2; Transformers.js is the recommended maintained alternative.
Backlinks
Appeared in 1 briefing