Tag
The WeChat WeLM team published a paper introducing the Hidden Decoding method, which extends computation through hidden flows without increasing the Transformer backbone parameters, training the WeLM-HD4-80B and WeLM-HD4-617B MoE models, surpassing autoregressive baselines on multiple benchmarks.
This paper introduces Hidden Decoding, a sequence-length scaling method for LLMs that adds internal computation per token by expanding each token into multiple streams with independent embeddings, using Stream-Factorized Attention to keep costs low. Experiments on models up to 617B parameters show consistent improvements over baselines, demonstrating a practical fixed-backbone scaling path.