Tag
This paper introduces KITE, a KV-invariant transformer expansion method that efficiently scales LLMs by reducing inference costs while maintaining performance. It presents the SST model that achieves lower training loss and reduced inference cost compared to baselines.
Introduces MoME, a context-aware memory mechanism for LLMs that uses a mixture of slots to handle token polysemy, improving over baselines in pretraining experiments.
The article speculates whether Google's lag behind OpenAI and Anthropic in frontier AI reflects poor execution or a fundamental disagreement with the scaling hypothesis for achieving AGI. The author questions if Google believes LLMs are ultimately a bubble and that AGI requires a fundamentally different approach.
The article proposes a conceptual 3-tier embodied AI architecture to address the limitations of scaling LLMs for agency, emphasizing physical cost auditing and offline sleep cycles.
The author reflects on past LLM scaling eras (model size, chain-of-thought, agents) and speculates about the next big scaling dimension, suggesting recursive self-improvement as a potential breakthrough.
This paper introduces Hidden Decoding, a sequence-length scaling method for LLMs that adds internal computation per token by expanding each token into multiple streams with independent embeddings, using Stream-Factorized Attention to keep costs low. Experiments on models up to 617B parameters show consistent improvements over baselines, demonstrating a practical fixed-backbone scaling path.