Tag
据《星报日报》记者多方供应链消息，DeepSeek 下一代大模型 DeepSeek V5 计划在华为昇腾芯片上进行预训练，此前昇腾主要用于 DeepSeek 模型的推理任务。这将是国产算力首次支撑大模型预训练阶段，并伴随 DeepSeek 与华为在昇腾软件栈上的深度协同开源。
Han Junlin announces joining AMI Labs as an MTS to work on world model pre-training and capability emergence, expressing excitement about building AI for the real world with a scaling mindset.
该论文提出词级未登录词交叉熵（UCE）作为衡量预训练语料与零样本基准词面对齐的无模型指标，并通过受控预训练实验表明语料-基准的词面对齐能一致地预测零样本准确率，同时在文档选择与提示路由层面展示了可操控的收益。
HuGe 提出了一族人形通用视觉-语言-动作（VLA）模型，利用超过2000小时的第一人称人类与机器人数据进行预训练，并通过伪本体感知与视觉适配实现全身控制，在 HumanoidArena 仿真基准上达到 SOTA，同时在 Unitree G1 实体机器人上验证了其有效性。
TempoPFN 发表于 NeurIPS 2026，提出一个合成时间序列生成器与增强库，用于预训练线性 RNN 时间序列基础模型，实现零样本预测，并已开源代码与模型权重。
微软公布了其最新的 MoE 基础模型 MAI-Base-2（80B 激活参数 / 2.8T 总参数），在自研评测中超越所有开源权重模型，预训练 FLOPs 效率比 DeepSeek-V4-Pro-Base 高 4-12 倍，使用 16k 张 GB200 GPU 在 38 天内完成训练。
该论文通过受控消融实验研究原生多模态预训练中视觉信号对扩展行为的影响，发现视觉内容作为可扩展锚点不仅能提升多模态理解，其收益还能延伸到纯文本空间推理任务（如定位、导航和符号规划），同时不损害整体语言性能。
An independent researcher documents increasing MFU from 21% to 50% while pre-training a 21B MoE model on a single NVIDIA GH200 by combining existing optimization techniques, sharing code, logs, and per-stage metrics for budget-constrained LLM builders.
UltraBERT revisits bidirectional encoder pre-training design choices, showing performance saturation beyond a few trillion tokens, masking-ratio trade-offs between classification and retrieval, and proposes a dual-LM-head distillation approach achieving state-of-the-art on GLUE (91.0) and BEIR (47.7).
研究人员提出BioBigBird，一个基于稀疏注意力机制、可处理最长4096 token的双向生物医学预训练语言模型，并通过多任务学习联合优化NER和关系抽取，在BLURB基准上取得具有竞争力的结果。
OpenESM 提出能量引导模型（ESM），将能量引导的潜状态优化与 token 生成统一到单一架构中，在大规模语料上进行能量引导预训练，表现出良好的扩展性并在多数下游任务上超越 GPT，同时完全开源模型权重与训练基础设施。
A Reddit post questions how Qwen's 27B model achieves performance comparable to or better than GPT-4o, speculating whether better or higher-quality pretraining data or other new techniques explain its efficiency.
Aleph Alpha describes how they scaled pre-training of their 30B-A3B MoE model (revealed to be Kolibri Origin) from 16 to 512 NVIDIA B200 GPUs using a hierarchical hyperparameter search approach, achieving 35.3% MFU and near-linear scaling just 6% below ideal, trained with a custom fork of torchtitan over InfiniBand/NVLink.
A paper from Meta Superintelligence Labs shows that pre-trained LLMs equipped with a light inference harness can act as capable agents and even outperform RL post-trained counterparts when given a sufficient test-time sampling budget.
A new paper reveals "mode-hopping" in LLM pre-training, where models repeatedly toggle between shallow pattern-matching and genuine generalization even as training loss stays flat — suggesting more pre-training does not always guarantee better generalization.
该论文提出"模式跳变"(mode-hopping)概念，指出LM预训练中平滑下降的损失与稳定基准分数会掩盖模型答题方式的突变（如OLMo3-32B在算术提示上81%→0%→81.7%的反复），并通过六项廉价行为测试排除了评估指标非线性与局部优化波动等简单解释。
The article questions whether Nvidia's Vera Rubin will accelerate LLM pre-training and explores if models with over 10 trillion parameters are feasible, considering constraints like data, power, and cost.
This paper introduces EoupCT, a novel framework that estimates and orthogonalizes unknown pre-training gradients to mitigate catastrophic forgetting in continual fine-tuning of large language models.
This paper systematically evaluates masking strategies for EEG foundation models across MAE and JEPA frameworks, identifying optimal configurations and a novel failure mode, achieving comparable performance to REVE with reduced compute.
Teacher He Jiyan and 7 PhD students from Zhongguancun College trained a 7B model from scratch in 3 months, achieving state-of-the-art performance in the 7B model category, with insights on efficient training and data quality.