Fractale-350M-base is a 386M-parameter base model pretrained from scratch with a novel trained fast-weight memory bank of 8 vectors replacing long context, fully open-sourced including weights, code, and research log.
Some of you may remember my post about the research project behind this: a trained fast-weight memory, with the paper and the full research log at github.com/kkuette/thought-bank. This is the follow-up. The first public model of the series is out. Quick context: solo researcher, one RTX 3090 for everything below 97M params, openly working with Claude for implementation and write-ups. Direction and judgment are mine. What it is. Fractale-350M-base is a 386M-param base model pretrained from scratch on 10B tokens (code + English web) around one idea: the model's only long-term memory is a bank of 8 vectors it writes to itself, one gist vector per 512-token chunk, oldest evicted FIFO. The bank is read back as fast weights: each slot expands through a hypernet into a small low-rank MLP that the token stream passes through. Not retrieval, not attention over stored text. The memory becomes part of the forward pass. The context window is deliberately tiny (512-token chunks). Each chunk is a separate forward pass, so anything older than the current chunk can only reach the prediction through those 8 vectors. The pretraining objective forces the bank to matter: predict the opening of the next, never-seen chunk of a document from the bank alone (blank input). What's measurable. GAP = CE(reset bank) minus CE(carried bank) on held-out documents: +9.4 nats on code (CE 12.9 reset vs 3.45 carried) and +7.3 nats on web at the final checkpoint, flat from 2 to 8 chunks written (no FIFO cliff). And the gap widened from both sides during training: the bank-only arm kept sharpening while the no-bank arm degraded, meaning the model grew more dependent on its memory as pretraining progressed. At smaller scales, with exact controls: content is addressable by cues, survives eviction for 2000+ steps, and transfers docstring to code in both directions. From the paper (3M-scale mechanism, DOI on Zenodo): a single 13-token presentation installs a never-trained rule at 0.79 to 1.00 accuracy on unseen queries, where test-time training fits its own examples and transfers nothing, at 138x the cost. What it is NOT. It's a base model, not a chat model: no instruction tuning, no alignment, and at 386M on 10B tokens the raw fluency is what you'd expect. The bank carries the gist (domain, register, structure, announced facts), not verbatim text. Ask it to quote line 3 and it can't. And the GAP compares the model to itself without its memory, not to a matched-compute attention baseline. That steelman (yes, the gated-DeltaNet one I promised here a while back) lives in the research repo. The fun part: the memory is a state you hold in your hand. from fractale import BankSession sess = BankSession.from_pretrained("fractale-lm/Fractale-350M-base") sess.read(open("long_doc.txt").read()) # any length, no growing prompt print(sess.continuation(32)) # predicted from the 8 notes ALONE print(sess.continuation(32, use_bank=False)) # amnesic control: the difference IS the memory sess.save_bank("doc.bank") # a few kB; restore tomorrow, or transplant into another session Everything is open: model and card (training mix, NaN incident included) at huggingface.co/fractale-lm/Fractale-350M-base, usage kit at github.com/fractale-lm/fractale, research log with exact repro commands for every claim at github.com/kkuette/thought-bank, paper on Zenodo. The repo also ships a script that reproduces the entire pretrain from a pinned commit (repro/phase1). Total cost of the run: about $320 of rented 8xA100 time, self-funded. Phase 2 (exploratory): teaching it to use the memory deliberately. Instruction tuning where remembering is a behaviour, then RL with the bank as working memory. Happy to answer anything, and genuinely interested in what breaks when you poke it. Links - Weights + model card: https://huggingface.co/fractale-lm/Fractale-350M-base - Usage kit (inference differs from a classic LM, you carry a bank state instead of a growing prompt): https://github.com/fractale-lm/fractale - Research repo (training code, findings, probes): https://github.com/kkuette/thought-bank - Full phase-1 reproduction script (8x A100, pinned commit): https://github.com/fractale-lm/fractale/tree/main/repro/phase1
This paper introduces a trained fast-weight memory mechanism for a 3M-parameter transformer that installs never-trained rules at inference using forward-only test-time training, showing no transfer. The work is fully reproducible on a single RTX 3090.
Memtensor Research Group released Metis, a family of LLMs (4B/9B/27B) that internalize memory into the backbone, eliminating external RAG. The model performs memory read/write in a single forward pass and deploys with frozen weights like a standard LLM.
Shibai-700M-Base is a 700M parameter LLaMA-based model pre-trained on 18B tokens of English, math, and Python code, achieving coherent generation despite being under-trained compared to larger models, and notable for its efficient training on a single RTX 5070 Ti GPU.
DWARF-55M-Base is a new language model using a nearly all-sparse attention architecture (DSQG) with a single full causal attention layer, achieving reliable retrieval up to 2048 tokens and extrapolating to 3x that context. It is released as a research prototype for community experimentation.
NVIDIA releases Nemotron-3-Ultra, a 550B-parameter open-weight model with a hybrid architecture combining Mamba-2, MoE, and attention, supporting up to 1M token context and configurable reasoning mode.