A from-scratch inference engine — no PyTorch, no llama.cpp, no framework — compiled to WebAssembly. The model downloads once, then everything happens on your machine. Nothing you type is sent anywhere.
This is a demo, and it is the slow way to run this engine. A browser gives up the two things the engine is built on: memory-mapping the model (natively the file is the arena — RAM cost equals file size) and multi-core SIMD. In a tab it runs single-threaded on a copy in memory. The same engine on a real machine runs a 7B-class mixture-of-experts model at 10 tok/s on a 2018 laptop CPU. Numbers here should never be compared with those.