scaling-laws

该论文提出将 agentic RL 数据分解为任务、环境、轨迹三个维度并拟合各维度的 scaling laws，发现环境多样性 > 任务 > 轨迹的扩展指数排序，并据此推导出 Chinchilla 风格的计算最优数据配方，最高可获得 1.7 倍计算效率提升。

A scaling study over ~600 training runs finds that full BPTT on looped transformers outperforms truncated alternatives at matched compute, with two fully trained loops emerging as the compute-optimal setting that beats non-looped models with ~27% fewer parameters.

Inception Labs 发布 Mercury 3，这是迄今最强的扩散式 LLM，在多项基准上超越 GPT-6 Luna 和 Claude Haiku 5.5，同时以 1000+ tokens/秒的速度快 4-8 倍。模型规模比 Mercury 1 大 40 倍以上，支持 100 万 token 上下文，并以极具竞争力的价格面向生产环境开放。

MIT Technology Review explainer on how brute-force data center scaling is hitting ceilings as classic scaling laws (Moore's, Dennard's, Koomey's, Kryder's, Keck's) slow, and how fundamental physical limits of computing still leave room for progress. Draws on Feynman's 1959 'There's Plenty of Room at the Bottom' lecture to frame the next frontier below today's engineered limits.

This paper presents systematic evidence that vision-language models exhibit inverse scaling at the subtask level, with larger models underperforming smaller ones on tasks like spatial reasoning, visual perception, and STEM questions, suggesting model selection should be task-aware.

提出了一种新的可扩展性原则——'有效连贯认知速度'(S_C^eff)，用于衡量AI系统在现实变化速度超出训练分布时维持可靠行为的能力，并展示了受意识机制启发的Co⁴架构相比Transformer和gated-MLP基线在此指标上表现更优。

微软公布了其最新的 MoE 基础模型 MAI-Base-2（80B 激活参数 / 2.8T 总参数），在自研评测中超越所有开源权重模型，预训练 FLOPs 效率比 DeepSeek-V4-Pro-Base 高 4-12 倍，使用 16k 张 GB200 GPU 在 38 天内完成训练。

The author argues that scaling laws are missing a parameter H — human effort/time invested in producing learning signal — and that most recent AI gains (ChatGPT's RLHF, coding/math verifiable tasks, data vendor value) stem from scaling H rather than model size or data alone.

该论文通过受控消融实验研究原生多模态预训练中视觉信号对扩展行为的影响，发现视觉内容作为可扩展锚点不仅能提升多模态理解，其收益还能延伸到纯文本空间推理任务（如定位、导航和符号规划），同时不损害整体语言性能。

该论文将 Best-of-N 采样建模为正确与错误响应极端奖励尾部之间的竞争，利用极值理论刻画了三种渐近机制并推导了有限样本缩放定律，解释了奖励 hacking 导致的性能退化，并提出轻量级 tail-tension 准则来判断额外采样何时有害。

Concordia University and Mila researchers derive exact scaling laws for prediction regret in a bottlenecked bigram model with Zipfian token statistics, isolating capacity-driven scaling from optimization effects and matching predictions on WikiText2 and OpenWebText.

Harvard researchers develop a statistical learning theory of size generalization, proving an exponential separation in sample complexity between in-domain and cross-size learning, along with dimension-free scaling laws and matching minimax lower bounds.

该论文提出了一种自动生成多样室内布局和快速开源近似光线追踪器的流程，用128M合成样本预训练ResNet-101和Swin-T编码器-解码器，微调后在五个室内无线电地图预测任务上平均降低15%误差，并展示了可预测的数据缩放规律。

YerevaNN researchers show that for image self-distillation, sample repetition effects are dataset-dependent and can violate standard scaling law assumptions — a 1.5% random subset of Maxar even outperforms the full 50M-image pool under equal compute — and propose a modified Chinchilla-style law accounting for unique samples and repetition.

斯坦福大学新开设 CS 312「Deep Learning Alchemy」课程，由 Tatsunori Hashimoto 和 Suhas Kotha 主讲，聚焦深度学习中以经验为主的知识——超参数缩放、优化器不变性（μP、Muon）、损失景观与训练稳定性等，全部作业代码、测验和讲义公开，可视为 CS 336 的续作。

The paper introduces COUNT (Counting Offline Unlocks N-gram Transformers), a method that builds n-gram embeddings from precomputed counts stored on host memory without learned parameters, improving validation loss for 1B-parameter LMs and enabling training-free domain adaptation at inference time.

提出ShellElastic，一种将隐藏维度组织为稠密核心与递增外壳的弹性Transformer架构，使单次全宽前向即可同时包含所有子网络的精确隐藏状态；并引入μE扩展最大化更新参数化（μP），保证子网络在超网络加宽或增加外壳时仍被最大化更新且超参数可迁移，性能优于MatFormer等弹性基线并逼近分别训练的模型。

Multi-Stream Attention Residuals (MSAR) 将注意力残差（AttnRes）从单流扩展为多头深度注意力，通过多尺度因果卷积引入流多样性，在 18B/28B MoE 规模的 12 项下游基准上一致优于标准残差、mHC 和 AttnRes，scaling law 实验显示相对标准残差基线有 1.58 倍的 iso-loss 算力加速。

RoboJEPA 基于 JEPA 架构，在涵盖 12 种机器人具身的大型真实机器人数据集上训练，首次为多具身机器人世界模型建立了 scaling law：其 latent rollout 的“想象力误差”随计算量呈二阶幂律变化，并可作为真实机器人规划性能的可靠代理。该模型参数量达 8B，是迄今最大的 JEPA 预测模型，支持零样本部署为长时程规划的机器人智能体，并开源全部检查点与训练/部署代码。

Banbury Road unveils Kardashev-0.7, a population of 32 distinct models trained jointly with reinforcement learning to pursue Population Scaling, treating model count and collaboration as a new scaling axis toward complementary specializations.