Tag
A detailed landscape mapping of Post-Transformer architectures for 2026 — including Mamba-2/3, RWKV-7, BDH (Dragon Hatchling), xLSTM, Titans, and various hybrid models — comparing core mechanisms, attention variants, and test-time memory under practical compute and energy budgets.
A 150M-parameter non-transformer architecture achieves state-of-the-art cost-efficiency on ARC-AGI-1, validated by Transformer co-author Łukasz Kaiser, suggesting that recurrent latent reasoning can replace brute-force scaling.
Pathway's 150M-parameter BDH-CQ model achieves 29.5% on ARC-AGI-1 at a record-low cost of $0.0007 per task, using recurrent memory and latent reasoning instead of long token chains. The architecture may be the breakthrough Andrew Curran teased, with OpenAI researcher Lukasz Kaiser as an investor and adviser.
Pathway's BDH, a post-transformer architecture, reportedly matches GPT-2 scaling from 10M to 1B parameters while training from scratch on standard GPUs.
An overview of the current state and future outlook of continual learning in mid-2026, covering memory approaches including external memory, in-state memory, and weight updates, with analysis of various models like TTT, Titans, and Dragon Hatchling.
A co-author of the seminal 'Attention is All You Need' paper has argued that the field should move beyond transformers, and a debate hosted by Pathway explores this topic.