Tag
A technical discussion arguing that Shampoo, Muon, and Adam all drop per-sample covariance terms when approximating second moments (like the Fisher Information Matrix), and questioning why using the true second moments underperforms these approximations.
CS312 course experiments explore weight-averaging and EMA through the river-valley loss landscape theory, showing EMA slightly loses to LR decay on their model organism, complemented by DatologyAI's pretraining trick of skipping LR decay for a running weight average and merging checkpoints.
This paper proposes using quantum minimum finding (Dürr–Høyer with BBHT search) combined with an Interaction-Guided Schedule Oracle to jointly select layer-group learning-rate schedules for language-model fine-tuning, demonstrating closed-loop training of a 1.5B-parameter model whose benchmark results match uniform-rate baselines.
Meta and Mila researchers show that temporary weight decay during Muon-pretrained Transformers can appear harmful at removal yet improve final loss, demonstrating delayed effects driven by persistent weight-norm deficits that should be evaluated at the full training budget.
A-RSI 通过监控训练日志中的奖励变化与准确率来决定何时刷新任务池,结合模型自编写程序生成可独立验证的多模态问答对,在零外部数据条件下将基准平均性能从 57.77 提升到 61.05。
This paper traces how subject–verb number agreement circuits emerge and sparsify across training in Pythia-1b and Pythia-410m models, finding that behavior stabilizes early while the underlying circuit keeps reorganizing, shifting attention heads, and shrinking to a strikingly sparse final form.
实验表明 PBT 并不能加速 grokking,但在固定超参数全部失效的'死区'内可以成功逃逸并诱导 grokking;复盘显示真正起作用的是学习率/权重衰减的调度路径而非种群间的权重拷贝。
A research post introducing 'reactive optimizers': an agent watches training dynamics and rewrites the optimizer's source code during training, matching or beating the best hand-tuned recipes on image and language tasks while autonomously guarding against loss spikes and overfitting. The method is released via the open-source OPAL package.
《Mathematical Theory of the Transformer》是一部长达825页的数学专著,系统阐述了Transformer架构的理论基础,涵盖嵌入、自注意力、矩阵分解、谱分析、信息几何、训练动力学与优化等主题。
The paper identifies a Competing Learning phenomenon in hybrid language models where recurrence and attention layers compete to learn the same functions, and proposes a staged training procedure that improves language modeling loss, common-sense reasoning, and retrieval tasks on models up to 1B parameters.
本文提出一种受控诊断任务,区分语言模型在上下文学习中的“复制示例”与“组合应用规则”两种行为,发现提示控制、答案偏好、无候选构造和迁移能力在训练中沿不同时间线发展,且合成定义文档训练能提升组合式响应。
该论文提出了 Spectrally Targeted Muon,通过仅对超过或低于阈值的奇异值进行正交化,在 SGD 和 Muon 之间插值,并在 CIFAR-10 与 NanoGPT speedrun 上分析了 Muon 成功的真正原因:放大过小的奇异方向而非抑制大方向。
This paper investigates the life cycle of massive activations in transformers, showing that weight decay causally regulates their scale — with an AdamW-preconditioned growth vs. decay balance model predicting peak magnitudes scaling as λ^{-1/2}.
This paper audits whether gradient-conflict metrics (cosine similarity, conflict rates) actually predict the understanding–generation trade-off in unified multimodal models, using a controlled testbed (GridUMM) where the true trade-off is computable. Across 63 configurations and 372 checkpoints, no directional conflict metric reliably correlates with the trade-off, and a dose-response intervention suppressing conflict leaves the trade-off flat, while eff_rank and training loss outperform conflict geometry as diagnostics.
This paper proves that the learning-rate schedule acts as the averaging operator that determines whether data-ordering effects appear in model parameters, showing a 12.29σ arrangement effect under constant learning rate that collapses to zero under cosine decay. It also shows that conflicting-convention training shifts which correct answer form the model commits to without changing overall capability, a phenomenon exact-match benchmarks cannot detect.
This paper analyzes why pointwise clipping of the forward-KL objective in on-policy self-distillation (OPSD) can backfire: it proves and empirically shows that clipped objectives can push token probabilities away from the teacher, causing persistent token repetitions in reasoning outputs.
This paper identifies "co-cheating" in self-evolving search agents, where a proposer and solver converge on shared errors that inflate internal reward without improving true correctness. It proposes CrossFit, a cross-fitted verification scheme that substantially reduces false agreement and boosts performance on seven search benchmarks with Qwen3.5-4B/9B models.
This monograph develops a unified account of training and inference dynamics in Power Law Decoder Representation language models (PLDR-LLMs), covering exact identities, renormalization, and experimental findings.
This paper examines subliminal learning in large language models and introduces liminal training as a method to reduce unintended trait acquisition during fine-tuning while preserving task performance.
The paper studies training dynamics for two-layer neural networks learning orthogonal multi-index models under standard initialization, revealing incremental learning and competitive parameter reallocation, and introduces a symmetry-based finite-width approximation for analysis.