Pathway's BDH(post-transformer arch) matches GPT2 scaling from 10M to 1B params trained from scratch. runs on Normal GPUs
Summary
Pathway's BDH, a post-transformer architecture, reportedly matches GPT-2 scaling from 10M to 1B parameters while training from scratch on standard GPUs.
Similar Articles
Scaling Interpretable Transformers with Parity Bottleneck Layers
Introduces the ParityTransformer, a GPT-2-scale architecture with Deep Parity Bottleneck layers that make intermediate representations interpretable-by-design, efficiently enforcing sparsity without the memory costs of traditional wide bottlenecks, and demonstrating competitive performance on sparse probing tasks.
Humanoid-GPT: Scaling Data and Structure for Zero-Shot Motion Tracking
Humanoid-GPT is a GPT-style Transformer pre-trained on a billion-scale motion corpus, achieving zero-shot generalization for whole-body motion tracking across unseen motions and tasks.
@IntuitMachine: The One Change That Lets Small Models Outperform Their Size 1/ Everyone knows you need a 70B model to beat GPT-4 on com…
A thread introducing the Atomic Task Graph (ATG), a DAG-based execution substrate that allows a small 8B model to outperform GPT-4 on complex agent tasks by storing plan structure in a graph instead of linear text, reducing context bloat and enabling localized failure recovery.
@aakashgupta: Karpathy told Dwarkesh that a 1 billion parameter model, trained on clean data, could hit the intelligence of today's 1…
Andrej Karpathy claimed to Dwarkesh Patel that a 1B-parameter model trained on ultra-clean data could match today's 1.8T-parameter frontier models, implying 1,800× effective compression.
Tiny Scale Is All I Can Spare To Play With Transformer
A student introduces Silia, a novel transformer architecture that combines attention and FFN into a unified operation to save parameters at scales ≤10M, achieving comparable performance to GPT-2 with fewer parameters despite limited compute resources.