@savipww: a 744B parameter model just booted on a laptop with 25 gigs of ram and i read the repo twice before i believed it repo …
Summary
A 744B parameter mixture-of-experts model boots on a laptop with 25GB RAM by storing expert weights on SSD and only loading the active ~40B parameters per token, enabling local inference despite the model's size.
View Cached Full Text
Cached at: 07/21/26, 06:38 AM
a 744B parameter model just booted on a laptop with 25 gigs of ram and i read the repo twice before i believed it
repo is in the first reply if you want to run it yourself
the whole trick is that a model this big only fires about 40 billion of those params on any single word
so the engine parks the small dense core in your ram and leaves the thousands of experts sitting on your ssd pulling in only the few each token actually needs
that is how 370 gigs of weights fit onto a machine that has 25
pure c with zero dependencies int4 weights and an attention state crushed 57 times smaller so the state never blows up
is it fast honestly no on a bare 25gb box it crawls but on a real gpu rig it holds a few tokens a second
what gets me is how honest it is the dashboard paints every expert as a live galaxy and never once pretends the disk isnt doing the heavy lifting
this is the most i have wanted to actually sit down and try a local model all year
savip (@savipww): i found open sourced a tool that turns any brand into a website you scroll through like a movie
you scroll and the camera flies from outside a 3d scene straight into it then flows into the next one with no cuts
every scene is generated from the brand so the whole page feels
Similar Articles
@bkdgiffug: A model with 2.78 trillion parameters, runnable on 8GB of memory? Sounds absurd, but this project is indeed tinkering w…
A project demonstrates a 2.78-trillion-parameter model that can run on just 8GB of memory using a pure C99 inference engine with no BLAS, frameworks, or GPU reliance, trading speed for extreme memory efficiency (about 26.5 seconds/token at 8GB, 5.6 seconds/token with 128GB+).
How i managed to run a 193B Parameter model using only 24gb of Ram
Describes Iris Ai, a system that routes queries across 8 specialized LLMs on consumer hardware, achieving large-model performance with low memory by keeping only one model active at a time and dynamic model swapping.
@GPTWare: Uhhhh WTF is this???
Colibri runs the 744B parameter GLM-5.2 MoE model on a laptop with 25GB RAM by activating only ~40B parameters per token and streaming experts from disk, all in a single 2,400-line C file with no GPU required.
@MaximeRivest: 4B at a time, we could get to 10T-parameter models: a true mixture of experts. Jev showed us we can route pretty decent…
Maxime Rivest argues that 4B-parameter expert models could be combined into a true mixture-of-experts reaching 10T total parameters, enabling frontier-level performance at negligible cost on edge hardware, as small models fine-tuned on few synthetic examples already rival frontier systems.
I tried running a 1.56TB MoE model on a 6GB RTX 4050 Laptop, Here’s the result
Testing a 1.56TB Mixture-of-Experts model on a 6GB RTX 4050 laptop, requiring patched memory streaming with NVMe to achieve 0.106 tokens/s decode speed.