Addressing Performance Saturation for LLM RL via Precise Entropy Curve Control
Summary
This paper introduces Entrocraft, a rejection-sampling method for RL that controls entropy schedules to prevent performance saturation in LLMs. It demonstrates improved generalization and training longevity, allowing smaller models to outperform larger baselines.
View Cached Full Text
Cached at: 05/13/26, 12:20 AM
Paper page - Addressing Performance Saturation for LLM RL via Precise Entropy Curve Control
Source: https://huggingface.co/papers/2604.26326
Abstract
Entrocraft, a rejection-sampling approach for reinforcement learning, addresses performance saturation in large language models by customizing entropy schedules and improving generalization and training longevity.
Reinforcement learning(RL) has enabled complex reasoning abilities in large language models (LLMs). However, most RL algorithms suffer fromperformance saturation, preventing continued gains as RL training scales. This problem can be characterized by the collapse of entropy, a key diagnostic for exploration in RL. Existing attempts focus on preventingentropy collapsethrough regularization or clipping. However, their resulting entropy curves often exhibit instability in the long term, which hinders performance gains. In this paper, we introduce Entrocraft, a simplerejection-samplingapproach that realizes user-customizedentropy scheduleby biasing theadvantage distributions. Entrocraft requires no objective regularization and isadvantage-estimator-agnostic. Theoretically, we relate per-step entropy change to the advantage distribution under minimal assumptions. This explains the behavior of existing RL and entropy-preserving methods. Entrocraft also enables a systematic study ofentropy schedules, which reveals that linear annealing, which starts high and decays to a slightly lower target, performs best. Empirically, Entrocraft addressesperformance saturation, significantly improvinggeneralization,output diversity, and long-term training. It enables a 4B model to outperform an 8B baseline, sustains improvement for up to 4x longer before plateauing, and raises pass@K by 50% over the baseline.
View arXiv pageView PDFProject pageGitHub2Add to collection
Get this paper in your agent:
hf papers read 2604\.26326
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2604.26326 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2604.26326 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2604.26326 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
Breaking Entropy Bounds: Accelerating RL Training via MTP with Rejection Sampling
Bebop proposes entropy-aware multi-token prediction with rejection sampling and a novel TV loss to accelerate RL training of LLMs, achieving up to 1.8x speedup. The method addresses the degradation of acceptance rates during RL by optimizing training objectives.
Revisiting Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learning
This paper proposes Adaptive Entropy Regularization (AER), a framework that dynamically balances exploration and exploitation in LLM reinforcement learning by addressing policy entropy collapse through difficulty-aware coefficient allocation and initial-anchored target entropy. Experiments on mathematical reasoning benchmarks demonstrate consistent improvements in both accuracy and exploration capability.
ERR+: Sequential Entropy Resolution for Efficient and Decisive LLM Reasoning
The paper proposes ERR+, a two-phase reinforcement learning framework that improves LLM reasoning by rewarding entropy drops in thinking phases, leading to better accuracy and efficiency across multiple datasets.
Taming the Thinker: Conditional Entropy Shaping for Adaptive LLM Reasoning
This paper introduces Conditional Entropy Shaping (CES), a framework that dynamically controls token-level response entropy in LLMs to balance reasoning depth and conciseness, achieving improved accuracy while reducing response length on mathematical benchmarks.
EntroRouter: Learning Efficient Model Routing via Entropy Regulation
EntroRouter proposes a single-round model routing framework that uses entropy regulation to balance accuracy and computational cost, achieving 98.3% of the strongest expert's accuracy while reducing costs by 48.25%.