I trained a 15M "decision-only" model two ways — linear recurrence vs Transformer attention. Accuracy was a coin flip; latency and training cost weren't.[R]

Reddit r/MachineLearning Papers
linear-attention recurrent-models transformers long-context decision-models efficiency reproducible-research

Summary

A researcher trained two budget-matched 15M-parameter 'decision-only' models — a linear recurrent backbone vs a standard Transformer — and found accuracy was a statistical coin flip, while the linear model was 2.37× faster at 8k-token inference and ~6–7× faster to train. The post frames the negative accuracy result as the key finding: small decision models are accuracy-starved regardless of substrate.

Inspired by TypeSafe's Jev (a non-generative model that only outputs bounded decisions — choices, probabilities, calibrated confidence), I built two small decision models end-to-end on identical budgets and tasks to answer one question: which substrate makes the better Jev? Repo: https://github.com/stube123890-hue/jev-decision-lab Setup (V4, the main result): - ~15.09M params each, budget-matched to ±0.02% - jev-v8a: linear recurrent backbone (my Gated Segmented State Space architecture from linear-attention-lab) + decision head - jev-transformer: standard attention backbone + identical head - New task: long-context retrieval-decision — given a 2k–8k character passage, decide which of 4 sentences appears in it (2000 train / 500 val / 500 test) - Trained on a Colab T4, everything reproducible in the notebooks Results: - Accuracy: noise. v8a 26.8% vs transformer 28.2% on test (chance = 25%). Neither learns much at this scale — reporting the loss straight. - Latency: real crossover. Transformer faster below ~1550 tokens; at T=8192 the linear model is 2.37× faster (108.6 vs 257.7 ms). O(T) vs O(T²) showing up exactly where theory says it should. - Training speed: the big gap. 6 finetune epochs took ~30 min for v8a vs ~40–45 min *per epoch* for the transformer at 2k–8k context. That's a ~6–7× gap that gets worse with scale. Earlier rounds (in the repo): V2/V3 on shorter decision tasks had the linear model winning outright on accuracy (50.4 vs 46.2, 66.2 vs 61.0). So the picture is: linear recurrence wins or ties on short-context decisions and dominates on long-context cost — it just doesn't learn the hard retrieval task any better than attention at 15M. Currently running: V5, a bidirectional variant of both (15M, same budget). Will update the repo when it lands. Happy to answer questions on the architecture or the training setup. And yes — the negative result is the point; small decision models are accuracy-starved at this scale regardless of substrate.
Original Article

Similar Articles

Comparing Transformers and Hybrid Models at the Token Level

Lobsters Hottest

This paper analyzes token-level prediction differences between transformers and hybrid attention-recurrent models using Olmo 3 and Olmo Hybrid, finding that hybrids improve on semantic state tracking while transformers excel at n-gram copying and syntactic bracket matching.