Tag
The paper introduces full-bandwidth transformers, which use latent feedback to enhance autoregressive models by allowing non-verbalized computation to re-enter the stack, improving performance with negligible decoding overhead.
A new transformer variant, the full-bandwidth transformer, feeds back top-layer hidden states through a gated linear unit to improve reasoning and efficiency without altering the core architecture. Trained up to 400B tokens, it matches standard transformers trained with 1.5x more data while producing shorter reasoning traces.