TGRL: Temperature-Grouped Reinforcement Learning for Efficient Exploration in LLMs

Hugging Face Daily Papers Papers
reinforcement-learning llm exploration temperature-control efficiency rlvr

Summary

TGRL proposes a temperature-grouped reinforcement learning method to enhance exploration in large language models, achieving faster training and improved performance across multiple benchmarks.

Efficient exploration often remains a central bottleneck in reinforcement learning with verifiable rewards (RLVR). Although temperature control and test-time scaling strategies can increase rollout diversity of large language models (LLMs), they either expand the sample budget at rollout time or leave the benefit of exploration unquantified. To this end, we propose Temperature-Grouped Reinforcement Learning (TGRL), which turns temperature-induced diversity into an explicit training signal. For each prompt, TGRL partitions its rollout group into low- and high-temperature subsets, estimates exploration gain through their reward contrast, and allocates this group-level signal as token-level credit using Jensen--Shannon (JS) divergence between the corresponding temperature-scaled next-token distributions induced by the same logits. Notably, TGRL reaches equivalent accuracy up to 36% faster than strong RLVR baselines without expanding the rollout budget. Across 11 benchmarks from diverse domains, TGRL broadly improves over strong RLVR baselines: it improves the six-benchmark math average by 1.6% at 32B, raises CodeForces rating by 196.7 points and LiveCodeBench Pass@16 by 4.4%, and improves ALFWorld/WebShop success rates by 6.3%/4.9%. Comprehensive ablations and wall-clock analysis confirm the efficacy of all proposed components. Code is available at https://github.com/1229095296/TGRL/tree/main.
Original Article
View Cached Full Text

Cached at: 09/30/26, 04:22 AM

Paper page - TGRL: Temperature-Grouped Reinforcement Learning for Efficient Exploration in LLMs

Source: https://huggingface.co/papers/2609.33589 Published on Sep 27

·

Submitted byhttps://huggingface.co/lin1111987

zihanon Sep 30

Abstract

Efficientexplorationoftenremainsacentralbottleneckinreinforcementlearningwithverifiablerewards(RLVR).Althoughtemperaturecontrolandtest-timescalingstrategiescanincreaserolloutdiversityoflargelanguagemodels(LLMs),theyeitherexpandthesamplebudgetatrollouttimeorleavethebenefitofexplorationunquantified.Tothisend,weproposeTemperature-GroupedReinforcementLearning(TGRL),whichturnstemperature-induceddiversityintoanexplicittrainingsignal.Foreachprompt,TGRLpartitionsitsrolloutgroupintolow-andhigh-temperaturesubsets,estimatesexplorationgainthroughtheirrewardcontrast,andallocatesthisgroup-levelsignalastoken-levelcreditusingJensen--Shannon(JS)divergencebetweenthecorrespondingtemperature-scalednext-tokendistributionsinducedbythesamelogits.Notably,TGRLreachesequivalentaccuracyupto36%fasterthanstrongRLVRbaselineswithoutexpandingtherolloutbudget.Across11benchmarksfromdiversedomains,TGRLbroadlyimprovesoverstrongRLVRbaselines:itimprovesthesix-benchmarkmathaverageby1.6%at32B,[email protected]%,andimprovesALFWorld/WebShopsuccessratesby6.3%/4.9%.Comprehensiveablationsandwall-clockanalysisconfirmtheefficacyofallproposedcomponents.Codeisavailableathttps://github.com/1229095296/TGRL/tree/main.

View arXiv pageView PDFGitHub2Add to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.33589 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.33589 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.33589 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

From Trainee to Trainer: LLM-Designed Training Environment for RL with Multi-Agent Reasoning

arXiv cs.CL

This paper proposes the LLM-as-Environment-Engineer framework, where a policy model analyzes failures to automatically redesign the training environment for reinforcement learning, and introduces MAPF-FrozenLake as a controllable testbed. The framework, using Qwen3-4B, outperforms larger models like GPT and Gemini, showing that policy learning improves the model's ability to diagnose weaknesses.

ExpRL: Exploratory RL for LLM Mid-Training

Hugging Face Daily Papers

ExpRL is a new RL-based mid-training method that uses human-written reference solutions as dense reward scaffolds (never shown to the policy) to improve LLM reasoning, achieving significant gains on hard math benchmarks like AIME-2026.

Discovering Reinforcement Learning Interfaces with Large Language Models

Hugging Face Daily Papers

This paper introduces LIMEN, an LLM-guided evolutionary framework that automatically discovers reinforcement learning interfaces by jointly optimizing observation mappings and reward functions from raw simulator states. The approach reduces manual engineering effort and demonstrates that co-designing observations and rewards outperforms optimizing either component alone.