Tag
该论文提出让谐振序列模型（ResonatorLM）的读出相位依赖当前 token，在保持 FFT 卷积训练路径的同时引入输入依赖性，使在 256 token 训练、1024 token 测试时困惑度从 12.81 降至 3.76，显著改善长度泛化。
This paper investigates why standard neural architectures fail to generalize to longer inputs in dynamic programming, using geometric analysis with tropical semiring theory to reveal structural limitations in compositions.
This paper shows that state-of-the-art vision-language models fail at length generalization in visual reasoning due to 'global shortcuts', and demonstrates that combining local foveated perception with recurrence enables robust out-of-distribution generalization.
New paper introduces PoPE, a positional encoding that decouples content and position, addressing a fundamental flaw in RoPE used in many LLMs like Qwen, Gemma, DeepSeek. Presented at ICML2026.
This paper systematically studies in-context retrieval at million-token scale, introducing BlockSearch, a 0.6B LM retriever, and analyzing attention dilution. The model matches or outperforms dense retrieval on benchmarks like MS MARCO and NQ, and significantly outperforms on tasks requiring different similarity notions, highlighting the potential of in-context retrieval while emphasizing attention control under extreme context growth.
Wall Attention generalizes diagonal forget gates to softmax attention, enabling state-of-the-art length extrapolation from 4k to 160k+ context zero-shot and outperforming RoPE and FoX in pretraining. It is released as a drop-in replacement with open-source Triton kernels.