Tag
DiffusionRVGE 提出一种基于回归的价值梯度估计方法,通过对中心化奖励与中心化去噪图像进行回归来构建扩散模型的 RL 更新方向,无需奖励导数或穿过采样轨迹的反向传播,在匹配 DiffusionNFT 效果的同时显著降低训练计算成本。
The paper introduces Success-Referenced On-Policy Distillation (SR-OPD), which uses a student's own successful rollouts as references to selectively allocate teacher supervision, achieving comparable reasoning performance on math benchmarks using only 3.46–5.02% of the teacher-input tokens required by vanilla on-policy distillation.
Pivot-SD(EMNLP 2026 Oral)提出一种面向掩码扩散语言模型的高效自蒸馏方法,通过信息增益识别关键的"pivot"去噪步骤并仅对这些步骤进行奖励/惩罚,仅用200个问题在数学与代码任务上以更少算力超越全序列SFT和在线RL基线。
SMAT introduces a simple and efficient merge-aware training method that jointly optimizes expert loss and expected loss at simulated merged parameters, improving merged model performance with minimal computational overhead.
The paper introduces Matryoshka Language Model Suites, which nests multiple model sizes (500M, 1.5B, 3B) into a single architecture for joint training, allowing smaller models to benefit from distillation with the largest model.
Aurora-80K is a modern tiny language model with 80 thousand parameters, trained on 40M tokens and released on Hugging Face with benchmark results.
Dion3 is a revised Muon optimizer that reduces computational and communication overhead via Gram Newton-Schulz, symmetric GEMM kernels, and megabatching, achieving up to 6x faster optimizer steps while matching or improving loss.
LazyTrain is a research system that optimizes LLM training on limited hardware by formulating checkpoint selection, activation placement, recomputation, and CPU-GPU-NVMe communication overlap as a mixed-integer scheduling problem, achieving ~1.24x TFLOPS improvement over matched baselines.
This paper introduces MOSAIC, a framework that jointly optimizes sparse Mixture-of-Experts model architecture and hardware systems for large-scale pretraining, showing that compute-optimal sparsity is not necessarily cluster-optimal when MFU, communication costs, and parallel layouts are considered.
OmniLens is a scalable lens method for interpreting LLM hidden states, using low-rank translators and Subset-KL to reduce parameters and memory, enabling a dense ensemble of 482 lenses on LLaMA-3.3-70B at substantially lower cost.
This paper presents HiFloat4 format and Rollout-ResQ for end-to-end FP4 reinforcement learning post-training of LLMs, achieving a 1.1% accuracy gap to BF16 on Qwen2.5 models.
DomainPilot introduces a domain-level loss-guided two-stage framework for data mixture optimization in LLM fine-tuning, achieving improvements on MMLU-Redux, AIME24, LiveCodeBench v5, and BFCL v3 without increasing data volume or training cost.
The paper proposes A*-Inspired Batch Selection (A*-BS), a lightweight strategy that treats mini-batch scheduling as a heuristic search to improve CNN training efficiency. On MedMNIST tasks, a simple CNN with A*-BS outperforms deeper ResNet-18/50 baselines in accuracy and training speed.
Describes a workflow to build a task-specific local model in under a day using 25 hand-written examples, synthetic data expansion, LoRA fine-tuning, and quantization for CPU inference.
Proposes Efficient Long-horizon Optimization (ELO) learning, a meta-training algorithm that reallocates compute to longer horizons and uses decoupled progressive expert supervision, improving learned optimizers' performance on long-unroll tasks and out-of-distribution generalization. ELO-Celo2 consistently outperforms AdamW and matches Muon on language modeling tasks.
TurnOPD introduces turn-level budgeting for on-policy distillation of long-horizon agents, addressing inefficiencies in vanilla OPD by adaptive rollout-depth and progressive turn-normalized loss budgeting, achieving better accuracy under equal training budgets.
This paper proposes SiamJEPA, which uses masked Siamese student encoders with an EMA teacher network in JEPA models, showing improved representation separability and training efficiency compared to single-encoder variants and MAE.
The paper introduces Ember, a lightweight optimizer for embedding and LM-head matrices that exploits gradient geometry to improve efficiency and performance across supervised finetuning, RL, and pretraining, while using far less optimizer state than Adam.
Introduces Duplicated Latent Residual (DLR), a training-only, parameter-free plug-in for low-rank pre-training that improves perplexity across LLaMA models from 60M to 7B parameters, and can be folded into the model after training with zero inference cost.
This paper introduces FRESCO, an Echo State Network architecture operating entirely in the frequency domain to achieve O(N) complexity for dense recurrent updates, matching state-of-the-art performance on benchmarks while reducing computational costs.