Tag
Nikita Pavlichenko, former head of pretraining for JetBrains Mellum, systematically evaluated LLM architecture changes since 2020 through 1,129 training experiments. He concluded that architectural gains had largely been exhausted by the DeepSeek-V3 generation, with fine-grained MoE and linear attention representing the most substantial advances, while the impact of training methods such as Muon and inference kernels is often underestimated.
该论文研究随机舍入（stochastic rounding）如何影响低精度训练中的预测风险，针对两层线性网络推导了匹配的上下界，并给出了随计算量、模型规模、训练时长与精度变化的 scaling laws，揭示前向与后向舍入对最优计算缩放的不同影响。
A new paper by Randall Balestriero and Yann LeCun argues that paired-view training (e.g., image-caption pairs), not sheer scale or next-word prediction, is what produces unified world representations—showing a 109M-parameter model outperforming a 72B language model in representational alignment.
A thread presenting research that fits scaling laws to determine how much compute to invest in synthetic rewrites for improving data quality, and whether a model trained on rewrites can surpass the rewriting model itself.
OpenEBFM introduces Energy-Based Foundation Models that separate generation into energy steering and token generation, showing better data, parameter, and compute scaling than matched GPT baselines at up to 1B parameters, with full code, checkpoints, and an EBFM chatbot released as open source.
Apple's new paper introduces SynthAPI, showing that for tool-use agents, diversity of apps and database states matters more than raw task count, deriving a data-mix scaling law that predicts held-out generalization far better than task-count-only laws.
The paper turns the Platonic Representation Hypothesis into a theorem, proving that JEPAs trained separately on multiple modalities converge to shared representations up to a single rotation, and shows that next-token prediction is essentially a JEPA with 'bad views' — making it far less efficient for cross-modal alignment (good views need 42x fewer samples).
Cartesia 分享关于分层网络（H-Nets）的研究博客，展示无需分词器的字节级语言模型在受控实验中实现 3-5 倍计算效率提升，2 阶段 H-Net 以 10 倍更少计算与数据超越匹配的开源模型，并在代码等领域表现出色。
Cartesia 的 From Tokens to Bytes 系列第一篇探讨 Hierarchical Networks (H-Nets) 的扩展规律，展示学习式 tokenization 带来 3-5 倍计算效率提升，2 阶段 H-Net 以 10 倍更少算力和数据超越匹配的开源模型，并呈现 in-context tokenization 等新行为以及在生物与视觉等跨模态任务上的潜力。
Albert Gu announces a new version of hierarchical networks (H-Nets) that claim 3-5x efficiency gains in scaling law experiments, matching an open-source model trained on 11T tokens with roughly 10x less data and compute, as a step toward end-to-end learning from raw bytes.
An empirical study of reinforcement learning scaling laws for an actor-critic algorithm in the Starpilot and Craftax environments, showing that Elo-based performance metrics are more predictable than mean episode return and enabling state-of-the-art results in Craftax through compute scaling.
The paper introduces 'model information,' an information-theoretic measure of how much a trained diffusion model's outputs depend on its training dataset, and uncovers two scaling regimes: a nonlinear regime marking the transition from memorization to imperfect generalization, and a linear generalization regime. It proposes a principled scaling law that induces extrapolatable scaling laws for memorization and sample quality.
A Stanford CS312 (Deep Learning Alchemy) instructor posts exam-style prediction questions testing deep learning intuition, asking whether learning-rate or weight-decay rescaling can compensate for an 8× batch size increase in an AdamW-trained Transformer.
Dario Amodei 回忆 GPT-2 发布时给他的震撼，称当时就预见到规模再放大一百万倍后模型将在所有方面超越人类，并认为这是现代决定性的经济与科学时刻。
A Cornell University article/survey explaining how to build modern discrete diffusion language models, covering iterative parallel refinement, fast sampling at thousands of tokens per second, and their potential to improve scaling laws over autoregressive LLMs.
The paper proposes a rewriting scaling law that predicts loss from model size, training tokens, rewrite fraction, and data epochs while accounting for rewrite generation compute, finding optimal rewrite fractions increase with compute and that models trained on rewrites can outperform their rewriters.
提出「先知后画」的混合学习范式，用自回归预训练获取语义知识、扩散后训练进行高保真渲染，并建立计算最优扩展律，仅用6,600 H100 GPU小时训出3B文生图模型，在GenEval和DPG-Bench上取得86/87的成绩。
论文提出IEC框架，研究语言模型在受控指令负载（如无意义的单词替换规则）下的行为与性能退化，揭示了规则数量、替换密度、规则位置等因素遵循的数学规律，并在GPT-4o mini上验证，为模型建立可复现的"能力护照"。
The article surveys the dense wave of AI breakthroughs over the past month across mathematics, biology, medicine, chemistry, physics, energy, neuroscience, and robotics. Researchers have dubbed the phenomenon "alien mathematics" and "alien minds," and note that three multipliers — scaling laws combined with compute, training methods, and AI's ability to self-improve — are pushing the capability curve toward vertical.
在 PyTorchCon 2026 上，Sara Hooker（Coatue/Adaption AI）发表主旨演讲，探讨扩展更大模型的局限以及向自适应智能的转变，涵盖后训练、测试时扩展、自适应计算和硬件协同设计，并展示 AutoScientist 相关案例。