Olmo Hybrid: From Theory to Practice and Back

# Olmo Hybrid: From Theory to Practice and Back
Source: https://arxiv.org/abs/2604.03444
Authors: William Merrill (https://arxiv.org/search/cs?searchtype=author&query=Merrill,+W), Yanhong Li (https://arxiv.org/search/cs?searchtype=author&query=Li,+Y), Tyler Romero (https://arxiv.org/search/cs?searchtype=author&query=Romero,+T), Anej Svete (https://arxiv.org/search/cs?searchtype=author&query=Svete,+A), Caia Costello (https://arxiv.org/search/cs?searchtype=author&query=Costello,+C), Pradeep Dasigi (https://arxiv.org/search/cs?searchtype=author&query=Dasigi,+P), Dirk Groeneveld (https://arxiv.org/search/cs?searchtype=author&query=Groeneveld,+D), David Heineman (https://arxiv.org/search/cs?searchtype=author&query=Heineman,+D), Bailey Kuehl (https://arxiv.org/search/cs?searchtype=author&query=Kuehl,+B), Nathan Lambert (https://arxiv.org/search/cs?searchtype=author&query=Lambert,+N), Chuan Li (https://arxiv.org/search/cs?searchtype=author&query=Li,+C), Kyle Lo (https://arxiv.org/search/cs?searchtype=author&query=Lo,+K), Saumya Malik (https://arxiv.org/search/cs?searchtype=author&query=Malik,+S), DJ Matusz (https://arxiv.org/search/cs?searchtype=author&query=Matusz,+D), Benjamin Minixhofer (https://arxiv.org/search/cs?searchtype=author&query=Minixhofer,+B), Jacob Morrison (https://arxiv.org/search/cs?searchtype=author&query=Morrison,+J), Luca Soldaini (https://arxiv.org/search/cs?searchtype=author&query=Soldaini,+L), Finbarr Timbers (https://arxiv.org/search/cs?searchtype=author&query=Timbers,+F), Pete Walsh (https://arxiv.org/search/cs?searchtype=author&query=Walsh,+P), Noah A. Smith (https://arxiv.org/search/cs?searchtype=author&query=Smith,+N+A), Hannaneh Hajishirzi (https://arxiv.org/search/cs?searchtype=author&query=Hajishirzi,+H), Ashish Sabharwal (https://arxiv.org/search/cs?searchtype=author&query=Sabharwal,+A)

View PDF (https://arxiv.org/pdf/2604.03444)

> Abstract: Recent work has demonstrated the potential of non-transformer language models, especially linear recurrent neural networks (RNNs) and hybrid models that mix recurrence and attention. Yet there is no consensus on whether the potential benefits of these new architectures justify the risk and effort of scaling them up. To address this, we provide evidence for the advantages of hybrid models over pure transformers on several fronts. First, theoretically, we show that hybrid models do not merely inherit the expressivity of transformers and linear RNNs, but can express tasks beyond both, such as code execution. Putting this theory to practice, we train Olmo Hybrid, a 7B-parameter model largely comparable to Olmo 3 7B but with the sliding window layers replaced by Gated DeltaNet layers. We show that Olmo Hybrid outperforms Olmo 3 across standard pretraining and mid-training evaluations, demonstrating the benefit of hybrid models in a controlled, large-scale setting. We find that the hybrid model scales significantly more efficiently than the transformer, explaining its higher performance. However, it's unclear why greater expressivity on specific formal problems should result in better scaling or superior performance on downstream tasks unrelated to those problems. To explain this apparent gap, we return to theory and argue why increased expressivity should translate to better scaling efficiency, completing the loop. Overall, our results suggest that hybrid models mixing attention and recurrent layers are a powerful extension to the language modeling paradigm: not merely to reduce memory during inference, but as a fundamental way to obtain more expressive models that scale better during pretraining.

## Submission history

From: William Merrill [view email (https://arxiv.org/show-email/6449a88f/2604.03444)] **[v1](https://arxiv.org/abs/2604.03444v1)** Fri, 3 Apr 2026 20:36:34 UTC (1,124 KB) **[v2](https://arxiv.org/abs/2604.03444v2)** Tue, 7 Apr 2026 06:15:48 UTC (1,124 KB) **[v3]** Thu, 16 Apr 2026 18:50:04 UTC (1,124 KB)