A developer created a 250M parameter quantized LLM from scratch, trained on 30B tokens, deployable in 60 MB on CPU with a novel disk-based long context system supporting up to 100M tokens.
I trained a 250M parameter model from scratch on 30B tokens of fineweb. It’s quantized to under 2 bits so the whole deployment is 60 MB and it needs about 80 MB of RAM to run. Runs around 400 tok/s on a normal laptop CPU, no GPU needed. How the long context works: the most recent 2048 tokens stay in fp16 like a normal KV cache. Everything older gets compressed to 1 bit and written to disk, about 320 bytes per token, so 1 million tokens of history is roughly 320 MB on disk. From the start the model was trained to retrieve from that disk cache, up to 100M tokens. Due to a limited budget it wasn’t trained to reason over those tokens, only retrieve and answer from them. Language modeling quality of the base model, measured on held-out English web text that was never seen in training (educational web pages, 2,048 token windows): cross entropy 3.15 nats per token, perplexity 23.3, 0.99 bits per byte. The vocabulary is also not a normal embedding table. Every token is a fixed 512-bit code, 8.4 MB for all 131k tokens, zero trained parameters. I tested it on WordSim-353 (human word similarity ratings): my table scores 0.619 Spearman correlation vs 0.029 for random codes. Test script is in the repo. Some outputs (settings included so you know I’m not cherry picking, all reproducible from the repo): “Explain photosynthesis in two sentences.” (greedy) Photosynthesis is a process in which plants convert sunlight into chemical energy, which is then used to produce oxygen and other chemicals. This process is called photosynthesis. “Write a short poem about the sea.” (temp 0.25, top-k 30, rep 1.15, seed 2) The waves had swept over, and they were crashing against each other like rocks on top of one another. The wind was strong but not very gentle as it tried to push them towards their destination without lifting any weight or pulling away from those who stood in front of her with a stiff smile that made the sea look even more majestic than before. “What is the serial number of device Grus-189?” where the answer sits 50.6 million tokens deep in the archive on disk (archive mode, k=16) SN-442976 It’s a 250M model so expect mistakes on open facts, I’m not claiming it beats anything big. You can also fine-tune it, the full kit with a demo and before/after numbers is included. Master weights for fine-tuning are in the repo too: https://github.com/QLNI/SHADOW-250M-Instruct https://huggingface.co/NODEMIND/SHADOW-250M
Trained a 75M parameter LLM called KeyLM from scratch on 18B tokens, achieving competitive instruction-following scores against larger models while using fewer parameters and less data.
A developer trained a 1.1B-parameter LLM from scratch on 20B tokens for about $200, using fineweb-edu for pretraining and LoRA finetuning on OpenHermes. The project includes open-source code, model weights, and a demo website.
This paper validates distillation and quantization as cost-effective methods to expand the Apertus LLM family to new sizes and hardware formats, producing Apertus-v1.1 models with up to 4B parameters trained on 1.7T tokens.
A developer successfully ran a 5.2 million parameter MoE LLM quantized to INT4 on an ESP32 Dev Kit V1 using only 81KB of SRAM by streaming experts from flash, achieving about 5 tokens per second.
This paper presents a framework for quantizing vision-language models to 2.7 bits per parameter, enabling efficient mobile deployment by compressing the Llama 3.2 11B Vision Instruct model to 3.7 GB while preserving performance on visual QA tasks.