pretraining

A recent paper worth noting proposes a “unified training” algorithm that integrates pre-training, mid-training, and post-training into a single algorithmic workflow, simplifying infrastructure and providing a framework for better scientific research.

The paper shows for the first time that poisoning just 0.1% of pretraining data can implant a remote-code-execution backdoor in a coding agent, which executes attacker-chosen commands when a trigger appears while behaving normally otherwise; the 4B Qwen3 model reaches ~92% attack success rate.

A CMU-led paper introduces 'unified training,' showing that pretraining, midtraining, SFT, and RL can be collapsed into a single likelihood-based algorithm where RL objectives are treated as maximum-likelihood fitting to model-generated targets, outperforming staged training in several accuracy-compute and continual-learning experiments.

Carnegie Mellon paper shows that pretraining, fine-tuning, and RL can collapse into a single training rule (MaxRL as likelihood training on correct answers), with blended data runs outperforming staged training on reasoning and coding benchmarks while preserving general skills.

The author recommended the technical reports for MAI-Thinking-1 and Qwen3.8-Flash-Next, considering them the best learning materials for understanding the details of training frontier large language models, covering topics such as data processing, pretraining, reinforcement learning, architecture choices, and training stability.

OpenEBFM introduces Energy-Based Foundation Models that separate generation into energy steering and token generation, showing better data, parameter, and compute scaling than matched GPT baselines at up to 1B parameters, with full code, checkpoints, and an EBFM chatbot released as open source.

A tweet speculates that OpenAI's rumored GPT-6.5 Bel was pretrained with over 10 trillion parameters, suggesting it could represent a major leap once post-training is applied.

该论文指出权重归一化（WN）训练的收益主要源于其诱导出的有效学习率（ELR）轨迹，而非归一化本身，并据此提出 AIR（仿射平方根倒数衰减）调度策略。在 128B token 的 MoE 预训练实验中，AIR 在训练损失、下游性能和超参数鲁棒性上均优于 cosine 等调度方案。

An independent researcher trained Apex-3, a 21B-parameter MoE (3.36B active) from scratch on 70B tokens, releasing a stable checkpoint and sharing training-log evidence of emergent behaviors. The clearest finding is a sharp phase transition in in-context learning around 1.3B-1.5B tokens, while arithmetic gains appear continuous in probability despite looking abrupt in exact match.

Researchers train ModernBERT-based Icelandic encoders from 12M to 1B parameters on 9B tokens of open data, showing that a 1B model trained from scratch beats IceBERT on all eleven tasks, and that warm-starting from an English Ettin encoder yields the best scores across every benchmark while highlighting the importance of vocabulary transplantation.

ARMOR-DP is a modular framework for multilingual BabyLM that dynamically adjusts language sampling based on per-language validation loss, showing that the weakest language shifts during training and that dynamic protection outperforms fixed Chinese-protection baselines.

该论文提出了一个统一的六维分析框架，系统综述了2017–2026年大模型训练全流程（数据与分词、预训练、SFT与偏好对齐/强化学习）的395篇核心文献，并揭示了跨阶段效应与多个开放问题。

The paper proposes a rewriting scaling law that predicts loss from model size, training tokens, rewrite fraction, and data epochs while accounting for rewrite generation compute, finding optimal rewrite fractions increase with compute and that models trained on rewrites can outperform their rewriters.

提出了名为 Nestor 的嵌套稀疏机制，将条件计算从 MoE 扩展到注意力算子，使稀疏性层级组合，在同等计算与稀疏度下优于传统 MoE，并在长上下文与领域持续预训练中表现更佳。

Mirendil 发布其首个自研基础模型 Fern，支持最高 10M token 上下文窗口，专注于大规模 agentic 任务，如全代码库审计、日志调试和完整轨迹分析。

The paper introduces Latin Modern BERT (LAMB), a ModernBERT-based encoder pretrained on over 74B Latin tokens with morphologically informed masking, alongside two new Latin corpora (Acervus and Corpus Trinum), and shows it outperforms 15 existing encoders on 16 of 17 benchmark subtasks.

RynnVLA-Latent 提出用 30.1K 小时异构视频学习 3D-grounded 潜在动作，预训练 4B 视觉-语言-动作策略，在 LIBERO（98.8%）、RoboTwin 2.0 及真实机器人操作任务上取得领先成功率，无需针对基准的专门调优。

A Hugging Face repo publishes a claimed leaked mid-pretraining checkpoint of Anthropic's unreleased ~14T-parameter Mythos-class model codenamed "numbat" (referenced in a Claude Code source leak), including 772 safetensors shards (~7.72 TB), a full reference implementation, and training logs at step 1.83M of ~1.96M.

论文提出 Causal Edit Serialization (CES) 框架，通过特殊指针令牌与双索引 RoPE，在标准因果注意力掩码下实现插入、删除、替换、复制和移动等局部化文本编辑，并在文法纠错任务上验证了监督与强化学习后训练的可行性。

该研究系统考察了在预训练与后训练阶段切换 AdamW 和 Muon 优化器的代价，发现最终性能主要由两个优化器各自独立的效果解释，切换优化器几乎没有额外的性能惩罚，后训练优化器的选择可基本独立于预训练优化器。