training-dynamics

Tag

Cards List
#training-dynamics

Why don't optimizes like Shampoo, SOAP, and Adam measure the actual per sample second moments like the Fischer Information Matrix does? [d]

Reddit r/MachineLearning ↗ · yesterday

A technical discussion arguing that Shampoo, Muon, and Adam all drop per-sample covariance terms when approximating second moments (like the Fisher Information Matrix), and questioning why using the true second moments underperforms these approximations.

0 favorites 0 likes
#training-dynamics

@ginojirat: we run tens of experiments on weight-averaging and EMA in CS312! > some theory (or folk tales) first: the river valley …

X AI KOLs Timeline ↗ · 3d ago Cached

CS312 course experiments explore weight-averaging and EMA through the river-valley loss landscape theory, showing EMA slightly loses to LR decay on their model organism, complemented by DatologyAI's pretraining trick of skipping LR decay for a running weight average and merging checkpoints.

0 favorites 0 likes
#training-dynamics

Hybrid Quantum Selection of Learning Rates in Language-Model Training: Termination Criteria and an Interaction-Guided Schedule Oracle

arXiv cs.LG ↗ · 4d ago Cached

This paper proposes using quantum minimum finding (Dürr–Høyer with BBHT search) combined with an Interaction-Guided Schedule Oracle to jointly select layer-group learning-rate schedules for language-model fine-tuning, demonstrating closed-loop training of a 1.5B-parameter model whose benchmark results match uniform-rate baselines.

0 favorites 0 likes
#training-dynamics

Delayed Effects of Temporary Weight Decay in Muon Pretraining

arXiv cs.CL ↗ · 6d ago Cached

Meta and Mila researchers show that temporary weight decay during Muon-pretrained Transformers can appear harmful at removal yet improve final loss, demonstrating delayed effects driven by persistent weight-norm deficits that should be evaluated at the full training budget.

0 favorites 0 likes
#training-dynamics

A-RSI: Adaptive Task Refresh for Zero-Data Multimodal Self-Improvement

Hugging Face Daily Papers ↗ · 2026-10-28 Cached

A-RSI 通过监控训练日志中的奖励变化与准确率来决定何时刷新任务池,结合模型自编写程序生成可独立验证的多模态问答对,在零外部数据条件下将基准平均性能从 57.77 提升到 61.05。

0 favorites 0 likes
#training-dynamics

The Emergence and Sparsification of a Syntactic Circuit

arXiv cs.CL ↗ · 2026-10-27 Cached

This paper traces how subject–verb number agreement circuits emerge and sparsify across training in Pythia-1b and Pythia-410m models, finding that behavior stabilizes early while the underlying circuit keeps reorganizing, shifting attention heads, and shrinking to a strikingly sparse final form.

0 favorites 0 likes
#training-dynamics

PBT doesn’t make grokking faster, but it can escape a hyperparameter range where nothing groks [P]

Reddit r/MachineLearning ↗ · 2026-10-26

实验表明 PBT 并不能加速 grokking,但在固定超参数全部失效的'死区'内可以成功逃逸并诱导 grokking;复盘显示真正起作用的是学习率/权重衰减的调度路径而非种群间的权重拷贝。

0 favorites 0 likes
#training-dynamics

@SimulatedAnneal: 1/9 Current optimizers are static during training. What if the optimizer could think and react while the network trains…

X AI KOLs Following ↗ · 2026-10-21 Cached

A research post introducing 'reactive optimizers': an agent watches training dynamics and rewrites the optimizer's source code during training, matching or beating the best hand-tuned recipes on image and language tasks while autonomously guarding against loss spikes and overfitting. The method is released via the open-source OPAL package.

0 favorites 0 likes
#training-dynamics

@antoniolupetti: "Mathematical Theory of the Transformer" is an extensive, 825-page mathematical monograph exploring the theoretical fou…

X AI KOLs Timeline ↗ · 2026-10-20 Cached

《Mathematical Theory of the Transformer》是一部长达825页的数学专著,系统阐述了Transformer架构的理论基础,涵盖嵌入、自注意力、矩阵分解、谱分析、信息几何、训练动力学与优化等主题。

0 favorites 0 likes
#training-dynamics

Competing Learning in Hybrid Architectures

Hugging Face Daily Papers ↗ · 2026-10-19 Cached

The paper identifies a Competing Learning phenomenon in hybrid language models where recurrence and attention layers compete to learn the same functions, and proposes a staged training procedure that improves language modeling loss, common-sense reasoning, and retrieval tasks on models up to 1B parameters.

0 favorites 0 likes
#training-dynamics

From Copying to Composition: How Language Models Develop In-Context Learning

arXiv cs.CL ↗ · 2026-10-15 Cached

本文提出一种受控诊断任务,区分语言模型在上下文学习中的“复制示例”与“组合应用规则”两种行为,发现提示控制、答案偏好、无候选构造和迁移能力在训练中沿不同时间线发展,且合成定义文档训练能提升组合式响应。

0 favorites 0 likes
#training-dynamics

Spectrally Targeted Muon

arXiv cs.LG ↗ · 2026-10-09 Cached

该论文提出了 Spectrally Targeted Muon,通过仅对超过或低于阈值的奇异值进行正交化,在 SGD 和 Muon 之间插值,并在 CIFAR-10 与 NanoGPT speedrun 上分析了 Muon 成功的真正原因:放大过小的奇异方向而非抑制大方向。

0 favorites 0 likes
#training-dynamics

The Life Cycle of a Massive Activation: Stochastic Birth, Weight-Decay-Driven Growth, and Competitive Consolidation

Hugging Face Daily Papers ↗ · 2026-10-05 Cached

This paper investigates the life cycle of massive activations in transformers, showing that weight decay causally regulates their scale — with an AdamW-preconditioned growth vs. decay balance model predicting peak magnitudes scaling as λ^{-1/2}.

0 favorites 0 likes
#training-dynamics

Does Gradient Conflict Predict the Understanding--Generation Trade-off? A Controlled Audit of Conflict-Metric Validity in Unified Multimodal Models

arXiv cs.LG ↗ · 2026-10-02 Cached

This paper audits whether gradient-conflict metrics (cosine similarity, conflict rates) actually predict the understanding–generation trade-off in unified multimodal models, using a controlled testbed (GridUMM) where the true trade-off is computable. Across 63 configurations and 372 checkpoints, no directional conflict metric reliably correlates with the trade-off, and a dose-response intervention suppressing conflict leaves the trade-off flat, while eff_rank and training loss outperform conflict geometry as diagnostics.

0 favorites 0 likes
#training-dynamics

Conflicting Supervision Moves Commitment, Not Capability: A 12.29{\sigma} arrangement effect that is exactly zero under a convention-agnostic score

arXiv cs.AI ↗ · 2026-10-02 Cached

This paper proves that the learning-rate schedule acts as the averaging operator that determines whether data-ordering effects appear in model parameters, showing a 12.29σ arrangement effect under constant learning rate that collapses to zero under cosine decay. It also shows that conflicting-convention training shifts which correct answer form the model commits to without changing overall capability, a phenomenon exact-match benchmarks cannot detect.

0 favorites 0 likes
#training-dynamics

When Clipping Reverses Correction: Failure Dynamics of Pointwise Forward-KL On-Policy Self-Distillation

arXiv cs.CL ↗ · 2026-10-01 Cached

This paper analyzes why pointwise clipping of the forward-KL objective in on-policy self-distillation (OPSD) can backfire: it proves and empirically shows that clipped objectives can push token probabilities away from the teacher, causing persistent token repetitions in reasoning outputs.

0 favorites 0 likes
#training-dynamics

False Frontiers: Diagnosing and Mitigating Co-Cheating in Self-Evolving Search Agents

Hugging Face Daily Papers ↗ · 2026-09-30 Cached

This paper identifies "co-cheating" in self-evolving search agents, where a proposer and solver converge on shared errors that inflate internal reward without improving true correctness. It proposes CrossFit, a cross-fitted verification scheme that substantially reduces false agreement and boosts performance on seven search benchmarks with Qwen3.5-4B/9B models.

0 favorites 0 likes
#training-dynamics

Training and Inference Dynamics of PLDR-LLMs: Row-Map Collapse, Renormalization, and Predictive Reduction

Hugging Face Daily Papers ↗ · 2026-09-28 Cached

This monograph develops a unified account of training and inference dynamics in Power Law Decoder Representation language models (PLDR-LLMs), covering exact identities, renormalization, and experimental findings.

0 favorites 0 likes
#training-dynamics

On Mitigation of Subliminal Learning in Large Language Models

arXiv cs.CL ↗ · 2026-09-22 Cached

This paper examines subliminal learning in large language models and introduces liminal training as a method to reduce unintended trait acquisition during fine-tuning while preserving task performance.

0 favorites 0 likes
#training-dynamics

Learning Orthogonal Multi-Index Models Beyond Small Initialization: Incremental Learning, Competitive Dynamics and Symmetry

arXiv cs.LG ↗ · 2026-09-11 Cached

The paper studies training dynamics for two-layer neural networks learning orthogonal multi-index models under standard initialization, revealing incremental learning and competitive parameter reallocation, and introduces a symmetry-based finite-width approximation for analysis.

0 favorites 0 likes
Next →
← Back to home

Submit Feedback