Tag
UC Berkeley, Apple, EPFL, Harvard, MIT, and Google DeepMind researchers show that two-layer transformers trained on sorting converge to one of two mechanistically distinct solution classes depending on initialization, where only the two-attention-layer solution length-generalizes perfectly.
This paper examines how program inspection of Transformer Programs reveals failures hidden by token accuracy metrics across nine algorithmic tasks, showing how inspecting extracted Python code can guide training data improvements and boost sequence accuracy.
论文提出 Recurrent Looped Transformer (RLT)，将层分为并行因果编码器和循环解码器，使计算路径随序列长度增长而每 token 成本固定，在奇偶校验、S5 置换追踪和模运算等算法任务上显著优于标准 Transformer 的长度外推能力。
New paper finds that emergent capabilities in LLMs arise randomly from learning sparse attention patterns; the bottleneck is learning which tokens to attend to, which is slow and unpredictable.
XLGoBench introduces a synthetic benchmark of algorithmic tasks to detect cross-lingual skill gaps in LLMs, demonstrating persistent gaps across multiple state-of-the-art models.