Tag
ProVe reverses speculative decoding roles by fine-tuning a large model to propose token blocks verified by a smaller or identical model, achieving up to 2.7x throughput with adjustable speed-accuracy tradeoffs across six benchmarks.
The paper builds teams of persistent, discipline-diverse AI researcher agents with private histories and adaptive allocation of experimental budget to autonomously discover linear attention architectures, outperforming GD-NS at 1.3B scale on language modeling and retrieval tasks.
The paper introduces calibrated accommodation pairs (CAP), a preference dataset that rewards chat models for matching users' dialect while preserving Standard English, and audits how SFT, DPO, and RL stages change a model's tendency to accommodate informal dialect cues across 10 open model lineages.
The paper presents Lafa, an 8B-parameter blockwise continuous flow language model trained with under 30B tokens that matches or surpasses autoregressive quality while achieving 2.54x inference throughput via self-speculative decoding and blockwise parallel decoding.
PAR (Progress-Aligned Recurrence) aligns each loop iteration of a looped language model with a step of the task and rescales hidden states to prevent drift, enabling 16× length extrapolation on swap-S5 state tracking with a 0.5B model reaching 95.5% accuracy, far surpassing RLT and CoT baselines.
该论文因果性地追踪了导致模型「跟随提示线索而放弃正确答案」(sycophancy/caving)的注意力头,发现多种偏差共享同一个在预训练阶段就存在的线索路由电路,而微调与RL后训练只是调节该电路的激活强度与时机。
Yale researchers use influence functions to identify indirect evidence in language model training data across 15 BLiMP grammatical phenomena, showing that removing the top influential sentences lowers held-out accuracy significantly more than removing random controls, and characterizing how this evidence relates to linguistic structure.
论文提出 Causal Edit Serialization (CES) 框架,通过特殊指针令牌与双索引 RoPE,在标准因果注意力掩码下实现插入、删除、替换、复制和移动等局部化文本编辑,并在文法纠错任务上验证了监督与强化学习后训练的可行性。
论文提出“watermarkability”指标来量化语言模型与基于logit的水印规则的兼容性,并通过两阶段方法(目标集条件LoRA加闭环控制器)在保持高水印可靠性的同时显著降低困惑度增加。
提出 First Passage Attention (FPA),将注意力遗忘机制从路径求和改为路径极值,包含 Forgetting Transformer 作为特例,并在 110M 语言模型上实现 8 倍训练长度的完美 passkey 检索。
The paper introduces ToMStories, an evaluation dataset of ~6,000 synthetic narrative vignettes for testing theory-of-mind inference in 13 open-weight language models, finding that large models match humans on multiple-choice but not open-ended questions and tend to oversimplify implicit psychological states.
The paper identifies user representations as a key bottleneck in user simulation and introduces hindsight distillation, an online training procedure, to improve them. The resulting UserRepGen-4B model generates representations that enable more faithful simulation than those produced by GPT-5.5 across seven datasets.
该论文研究语言模型如何学习动作名词化结构(ANCs),通过构建 96 种参数化的人工语言并从头训练 GPT-2-small 模型,发现模型学习偏好在基本从句配置上与自然语言类型学分布一致,但在补语系统和 ANC 策略上出现分歧。
该论文指出,用秩来衡量微更新ΔW时必须加入噪声基线:方向计数类指标显示微更新仅略低于噪声,而稳定秩则显示远低于噪声,二者共同揭示更新具有少数强方向加长尾的结构。研究还发现指令数据、DPO偏好标签对更新并无显著特殊性,并质疑了'后训练是表面性的'相关论断。
这篇论文提出"relearning savings"评估协议,发现在Qwen2.5-1.5B中,曾学习过的事实绑定在部分遗忘后比从未学习的绑定更容易被重新习得,恢复所需优化步数减少约18%,说明当前行为准确率会低估既有经验的持久影响。
The paper introduces Draft Refinement Language Models (DRLMs), which generate complete drafts and iteratively refine them using a single denoiser and a cavity-energy score, outperforming autoregressive and discrete diffusion baselines on Countdown, Sudoku-Extreme, and Maze-Hard with only 6M parameters.
The paper identifies a Competing Learning phenomenon in hybrid language models where recurrence and attention layers compete to learn the same functions, and proposes a staged training procedure that improves language modeling loss, common-sense reasoning, and retrieval tasks on models up to 1B parameters.
MIT researchers published a Nature Communications paper on a multimodal AI model that combines 3D atomic structures with natural language to better predict and design crystalline materials, outperforming text-only models in identifying promising candidates like solid-state lithium-ion battery electrolytes.
The article questions whether embodied AI is advancing faster or slower than language models, noting that robots are judged by demos rather than benchmarks, and asks what fair metrics could compare physical AI progress with text and reasoning progress.
ECHO integrates multi-horizon prediction directly into the backbone via self-recursion and Dynamic Depth Sampling, improving pretraining efficiency and enabling the model to serve as its own inference drafter without auxiliary MTP modules.