generalization

The paper introduces 'model information,' an information-theoretic measure of how much a trained diffusion model's outputs depend on its training dataset, and uncovers two scaling regimes: a nonlinear regime marking the transition from memorization to imperfect generalization, and a linear generalization regime. It proposes a principled scaling law that induces extrapolatable scaling laws for memorization and sample quality.

该论文定义了一种与 Fisher 信息矩阵对齐的参数分布，使得在保证低损失的同时获得与 Fisher 信息矩阵零特征值数量成正比的参数对数体积，从而得到一个随 Fisher 信息矩阵期望秩缩放的高概率 PAC 泛化误差界。

This paper compares human and large language model linguistic productivity in extending novel verbs from active to passive contexts, finding that humans generalize based on abstract syntactic form even when no meaning can be assigned, while LLMs succeed only when a meaning similar to an existing verb exists.

This arXiv paper studies how to distributionally align LLM output distributions with human reference distributions and evaluates how well this alignment generalizes to unseen questions, finding that same-domain generalization works but cross-topic generalization is unstable and can degrade model capabilities.

ParetoRSI-Gym 提出一个多领域递归自我改进（RSI）评估框架，在固定模型与训练管线的前提下让智能体自主搜索与筛选数据，结果显示最强系统（Claude Code + Opus-5）在五个领域上带来平均 9.2% 的相对泛化提升，且各智能体展现出独特的数据选择策略。

提出 Harness Scaling 这一新的强化学习扩展维度，通过在多样化执行配置上训练以提升智能体模型对未见 harness 的泛化能力，并在多个基准上验证其优于单一 harness 训练的效果。

The paper shows that ReLU networks on modular addition can maintain 100% training accuracy and near-flat training cross-entropy while hidden features evolve and held-out accuracy rises above 90%, proving this coexistence with finite-window theorems and reproducing it in deeper networks and transformers.

Maxime Rivest shares a trained model that classifies any abstract against any given taxonomy, matching Qwen 27B and Jev-level superhuman accuracy at an order of magnitude lower cost and latency, while reflecting on the specialization vs. generalization tradeoff.

INCONVENIENT 是一个用于评估 LLM 在便利采样偏差下泛化能力的基准，通过对训练分布中特征-目标关系进行可控扰动，涵盖 200 个表格数据集并生成约 15,500 个偏移训练分布，揭示了真实部署场景中的性能退化问题。

A theoretical study of denoising score matching showing how rank constraints and architectural bottlenecks in score networks affect generalization, deriving asymptotically precise training and test errors via truncated SVD in a random-feature model and identifying a natural rank threshold tied to the data subspace's orthogonal complement.

该论文研究高维场景下的持续线性回归，推导了梯度下降有限步数训练时泛化损失的闭式表达，并证明最优步长按 lnT/T 衰减，其效果渐近等价于最优 L2 正则化。研究还探讨了对角正则化条件以及任务变异性与隐式正则化之间的权衡，并通过实验验证了理论预测。

提出了一种新的可扩展性原则——'有效连贯认知速度'(S_C^eff)，用于衡量AI系统在现实变化速度超出训练分布时维持可靠行为的能力，并展示了受意识机制启发的Co⁴架构相比Transformer和gated-MLP基线在此指标上表现更优。

PreciseCoder 提出在单元测试奖励之外加入语义验证的局部编辑奖励信号，显著减少模型在调试时的多余改动并防止幻觉式 bug 修改；在 4B/9B/27B 模型上将平均编辑行数从 7-15 行降至 3 行，并能泛化到仓库级程序修复，其推理轨迹还能提升 GPT-5.6 Sol 等前沿模型的编辑精度。

本文提出RoleWorld，一种符号化世界建模方法，通过可复用的功能角色抽象（functional role abstractions）实现LLM智能体在不同环境间的知识迁移，在MARS基准上展示了跨环境的稳健泛化能力。

本文提出层级子任务执行（HSE）框架与 RoboCombine 基准，研究具身操作中技能重组的组合泛化极限，指出失败源于「起始状态差距」，并提出 PRTI 轻量后训练策略以提升跨子任务边界的执行成功率。

该论文提出“人格层级模型”（Persona Hierarchy Model）来解释LLM微调中的上下文泛化现象：修改共享默认人格的微调会带来更广泛的迁移，而局部人格的改变则更具上下文特异性。作者还提出人格保持正则化（PPR），在RL训练中将奖励黑客行为从42–55%降至最多0.2%，同时保留准确率增益。

MemCo提出了一种以记忆为中心的协作框架，通过维护互补的局部与全局记忆空间，帮助LLM智能体在未见过的交互环境中实现更好的泛化，实验表明其在交互式决策基准上提升了任务成功率并减少了冗余探索。

论文提出 Recurrent Looped Transformer (RLT)，将层分为并行因果编码器和循环解码器，使计算路径随序列长度增长而每 token 成本固定，在奇偶校验、S5 置换追踪和模运算等算法任务上显著优于标准 Transformer 的长度外推能力。

This paper systematically evaluates how LLMs generalize across four dimensions—domain shift, adversarial perturbations, compositionality, and length increase—in time and event expression extraction, finding that inductive prompting generalizes most consistently while gains from scale, architecture, and other prompting strategies are uneven and dimension-specific.

The paper discovers "forking," a generalization failure in NanoGPT autoresearch where training and validation loss sharply diverge at epoch boundaries when models with an over-encoding n-gram memory branch are trained under data replay. It mechanistically explains the phenomenon and cautions about unintended side effects of techniques produced by autoresearch agents.