Tag
本文对隐式思维链（latent CoT）是否真正进行推理进行了系统研究，覆盖21种模型-任务组合，提出一个统一分类法，区分链条的表示内容与其在模型中的因果作用，揭示不同训练配方下latent reasoning的多样机制。
This paper analyzes curriculum learning in large language models by examining optimization dynamics across difficulty levels, proposing a new method called Transfer-aware Dynamic Curriculum Sampling (TDCS) that dynamically adjusts training data based on transfer relationships.
This paper introduces a method to predict best-of-N inference scaling gains for language models using cheap statistics from a single labeled validation-set sampling pass. A compact predictor with three core features achieves Spearman ρ=0.90 with actual gains, enabling screening of configurations before expensive reward-model scoring.
This paper identifies a failure mode in masked diffusion language models where confidence-based decoding leads to high-confidence errors on complex reasoning tasks, and shows that confidence-aligned training exacerbates this issue while random masking preserves reasoning performance.
This paper investigates the 'small-vs-large gap', where training on fewer samples with more repetitions can lead to faster learning and compute savings compared to using larger datasets, attributing the speedup to layer-wise growth enabled by sampling biases. The findings suggest that smaller datasets with repetition can be proactively leveraged as favorable inductive biases, particularly in reasoning tasks.
Self-Distillation Zero (SD-Zero) is a novel training method that converts sparse binary rewards into dense token-level supervision through dual-role training where a model acts as both generator and reviser, achieving 10%+ improvements on math and code reasoning benchmarks with higher sample efficiency than RL approaches.