llm-scaling

The author reflects on past LLM scaling eras (model size, chain-of-thought, agents) and speculates about the next big scaling dimension, suggesting recursive self-improvement as a potential breakthrough.

This paper introduces Hidden Decoding, a sequence-length scaling method for LLMs that adds internal computation per token by expanding each token into multiple streams with independent embeddings, using Stream-Factorized Attention to keep costs low. Experiments on models up to 617B parameters show consistent improvements over baselines, demonstrating a practical fixed-backbone scaling path.