Tag
The paper 'On the Entropy Calibration of Language Models' interprets rising entropy as increasing diffusion of accessible pathways in autoregressive generation, proposing that scaling has limited benefits due to heavy-tailed data and suggesting a pathway-aware decoding alternative.
This paper from the Kathleen series shows that an attention-free, byte-level model with ~0.5M parameters can beat a parameter-matched transformer on WikiText-103 language modeling and generation, introduces a non-parametric 'Form Distance' metric for evaluating text realism, and demonstrates that retrieval-augmented decoding from the model's own training corpus improves generation quality.
This tutorial covers how to use a transformer model from training to inference, focusing on autoregressive generation, prefill vs. decode phases, and key-value caching for efficient inference.
This paper proposes SPG, a structure-aware patent generation method that jointly predicts claim topology and content during autoregressive decoding using a pointer head and a preference optimization stage, achieving significant improvements in parent link recovery and antecedent consistency on the HUPD-DCG benchmark.
This paper introduces SPARC, a spectral-algebraic theory explaining the self-correction blind spot in autoregressive language models, where models fail to correct their own errors but can fix identical external errors. The theory proves the blind spot arises when the spectral radius of an error-propagation operator is at least one, derives a threshold for correction markers, and provides convergence guarantees for RL-based self-correction training.
MV-Forcing introduces a diffusion framework that combines temporal and view-wise autoregression to generate long, multi-view consistent videos of dynamic scenes, using a 4D geometric bridge and spatio-temporal distillation to enable arbitrary-length generation from a few-step student model.
This article explains why AI tokens are expensive: the autoregressive generation process requires predicting token by token and repeatedly computing the entire context, causing computational cost to grow linearly with context length; coding agents, due to multi-turn interactions and file reads, quickly accumulate very long contexts, further increasing token costs.
Memento is a subject-reconstruction-guided framework that improves long-form video generation by preserving recurring subjects through memory-based reconstruction and dual-query mechanisms, achieving state-of-the-art performance in long-term subject consistency and cross-shot coherence.
TBD-VLA introduces a discrete vision-language-action framework that combines block diffusion with autoregressive generation to achieve efficient temporal action modeling and faster inference, significantly outperforming prior VLA approaches in simulation and real-world manipulation tasks.
This academic paper develops a theoretical framework for online learning with autoregressive chain-of-thought reasoning, analyzing mistake bounds under end-to-end and trajectory supervision models.