Tag
An independent researcher trained Apex-3, a 21B-parameter MoE (3.36B active) from scratch on 70B tokens, releasing a stable checkpoint and sharing training-log evidence of emergent behaviors. The clearest finding is a sharp phase transition in in-context learning around 1.3B-1.5B tokens, while arithmetic gains appear continuous in probability despite looking abrupt in exact match.
该论文提出"模式跳变"(mode-hopping)概念，指出LM预训练中平滑下降的损失与稳定基准分数会掩盖模型答题方式的突变（如OLMo3-32B在算术提示上81%→0%→81.7%的反复），并通过六项廉价行为测试排除了评估指标非线性与局部优化波动等简单解释。
This paper introduces CurveShift, an analysis method that separates overall ability gains from difficulty-specific improvements in LLM agents. Using METR time-horizon data and LiveCodeBench, it finds that most apparent shifts toward harder tasks are ceiling effects, though a genuine hard-task effect exists for reasoning models in competitive programming.
This paper formalizes analogical reasoning in Transformers using category theory, introduces synthetic tasks to study its emergence, and reveals that it arises from geometric alignment of relational structures and functor application, with signatures also found in pretrained LLMs. The work was accepted as a Spotlight at ICML 2026.
Discusses that the mathematics used by AI is mainly linear algebra, calculus, etc., from before the 19th century, but emerging phenomena such as Scaling Law, emergent abilities, double descent, in-context learning, and representation geometry lack mathematical explanation. Analogizes to the clouds in physics in 1900, suggesting it may drive the development of 21st-century mathematics.
Discusses the structural weakness of current evaluation methods for LLMs, which fail to anticipate qualitative shifts in capability, and argues that developing proactive evaluation infrastructure is the critical bottleneck for safe capability jumps.