Introducing DWARF-55M-Base

Finally after months of research, the very first model made from the DWARF architecture is available for folks to check out and experiment with! DWARF is a nearly all-sparse attention architecture that uses 9 Dynamic Sparse Query-Gather (DSQG) layers as the backbone for transportation and a single full causal attention layer at 25% layer depth. For example, if there are 32 layers in the model it requires only a single full causal attention layer at L7 as the global mixer, achieving consistently reliable retrieval up to the trained context of 2048. Initial results also show that training at only N=2048 produces consistently reliable retrieval out to 3 times it's trained N (in this case, N=6144), but this has yet to be thoroughly tested to confirm for sure. DSQG replaces full attention with a fixed, causal set of nearby and long-range token offsets, so each token consults a small multi-scale sample of its past rather than every earlier token. It then learns which sampled positions matter and modulates their values using query/key probe signals, aiming to preserve useful content-sensitive communication at much lower attention cost. Because DSQG decodes only a fixed number of offsets for each new token rather than scanning the whole history, its per-token KV-cache bandwidth and attention work are nearly-entirely O(1) with context length. This is a base model for a research prototype trained on 10B tokens of Dolma3 Mix 150B, using a ChatML-modified OLMo1 Base tokenizer (50282 tokens). The code on the Github allows folks to experiment with the HISA path, which replaces full attention at L3, to get competitive performance to the full attention hybrid but is purely sparse in nature. The HISA path is currently experimental and not yet hammered out all the way. Huggingface Github