HNseen ×2NEW
I'm making LLMs work on small and old devices(github.com)
The project "smolmaxxing" aims to maximize the ability to run large language models (LLMs) on small and old devices, such as TV boxes, old phones, and single-board computers. By utilizing custom kernels, smarter memory use, and weight streaming, the project has successfully run models with up to 16 billion parameters on devices like a 3rd-gen Xiaomi TV Box and a 13-year-old iPad Air, achieving impressive performance metrics, including 24-26 tokens per second on the iPad Air with a 1.7B-parameter model, demonstrating the potential to unlock the hidden capabilities of everyday devices.