AdaKP: Online Adaptive Knowledge-Point Selection for Reasoning-Oriented Reinforcement Learning
Summary
Introduces AdaKP, an online adaptive knowledge-point selector that dynamically re-chooses which atomic hints to inject during RL training to mitigate reward sparsity in reasoning tasks, achieving improvements on competition-level math benchmarks with negligible overhead.
arXiv:2607.24833v1 Announce Type: new
Abstract: Reinforcement learning with verifiable rewards is a powerful paradigm for eliciting reasoning in large language models, yet it suffers from severe reward sparsity on competition-level mathematics. A common remedy injects atomic knowledge points (KPs) - short natural-language hints distilled from gold solutions - into the prompt. Existing methods, however, either fix this selection once offline or merely scale the monolithic quantity of injected text, leaving untouched the most informative axis of choice: which subset of atomic KPs to inject, and when. We introduce AdaKP, an online selector that re-chooses each problem's KP subset over the course of RL training. At its core is an entropy proxy that scores a KP by the reduction in next-token entropy it induces - a single inexpensive forward pass, with a provable bound on its truncation bias - in place of expensive rollout-based estimation. Three lightweight mechanisms make this signal usable online: a momentum smoother that absorbs per-step noise, a retirement-and-revival manager that prunes weak KPs while preserving exploration, and an adaptive scheduler that front-loads re-evaluations into early training. AdaKP further contributes a pre-flight validation gate that certifies the proxy against a leave-one-out ground truth before any expensive run is launched, turning method-level risk into a falsifiable check. Realized as a fully additive fork of a standard DAPO+GRPO trainer with no optimizer changes, AdaKP improves over a strong static-selection baseline on all eight competition-mathematics benchmarks at negligible added cost, positioning online, validated KP-subset selection as a practical and as-yet under-explored axis for reasoning-oriented reinforcement learning.
View Cached Full Text
Cached at: 07/29/26, 09:52 AM
# Online Adaptive Knowledge-Point Selection for Reasoning-Oriented Reinforcement Learning
Source: [https://arxiv.org/html/2607.24833](https://arxiv.org/html/2607.24833)
###### Abstract
Reinforcement learning with verifiable rewards is a powerful paradigm for eliciting reasoning in large language models, yet it suffers from severe reward sparsity on competition\-level mathematics\. A common remedy injects*atomic knowledge points*\(KPs\)—short natural\-language hints distilled from gold solutions—into the prompt\. Existing methods, however, either fix this selection once offline or merely scale the monolithic quantity of injected text, leaving untouched the most informative axis of choice:*which subset of atomic KPs to inject, and when*\. We introduceAdaKP, an online selector that re\-chooses each problem’s KP subset over the course of RL training\. At its core is an*entropy proxy*that scores a KP by the reduction in next\-token entropy it induces—a single inexpensive forward pass, with a provable bound on its truncation bias—in place of expensive rollout\-based estimation\. Three lightweight mechanisms make this signal usable online: a momentum smoother that absorbs per\-step noise, a retirement\-and\-revival manager that prunes weak KPs while preserving exploration, and an adaptive scheduler that front\-loads re\-evaluations into early training\. AdaKP further contributes a*pre\-flight validation gate*that certifies the proxy against a leave\-one\-out ground truth*before*any expensive run is launched, turning method\-level risk into a falsifiable check\. Realized as a fully additive fork of a standard DAPO\+GRPO trainer with no optimizer changes, AdaKP improves over a strong static\-selection baseline on all eight competition\-mathematics benchmarks at negligible added cost, positioning online, validated KP\-subset selection as a practical and as\-yet under\-explored axis for reasoning\-oriented reinforcement learning\.
## 1Introduction
Reinforcement learning with verifiable rewards \(RLVR\) elicits emergent chain\-of\-thought reasoning\(Weiet al\.[2022](https://arxiv.org/html/2607.24833#bib.bib34); Wanget al\.[2023](https://arxiv.org/html/2607.24833#bib.bib35); Yaoet al\.[2023](https://arxiv.org/html/2607.24833#bib.bib36)\)in relatively small base models\(DeepSeek\-AI[2025](https://arxiv.org/html/2607.24833#bib.bib4); OpenAI[2024](https://arxiv.org/html/2607.24833#bib.bib5); Yuet al\.[2025](https://arxiv.org/html/2607.24833#bib.bib2)\), going beyond decoding\-time enhancements that leave the policy unchanged\(Yaoet al\.[2023](https://arxiv.org/html/2607.24833#bib.bib36); Huanget al\.[2024](https://arxiv.org/html/2607.24833#bib.bib38)\)\. A practical bottleneck is*reward sparsity*: on hard competition problems an untrained 1\.5B model answers correctly on only a tiny fraction of rollouts, leaving GRPO\(Shaoet al\.[2024](https://arxiv.org/html/2607.24833#bib.bib3)\)and DAPO\(Yuet al\.[2025](https://arxiv.org/html/2607.24833#bib.bib2)\)with degenerate advantages\. KnowRL\(Yuet al\.[2026](https://arxiv.org/html/2607.24833#bib.bib1)\)addresses this by distilling short*knowledge points*\(KPs\) from gold solutions and injecting a minimal\-sufficient subset; choosing that subset offline by Constrained Subset Search \(CSS\) lifts a1\.51\.5B model to70\.08%70\.08\\%average accuracy on eight benchmarks while cutting injected hints from5\.865\.86to2\.572\.57per problem\.
#### The static\-selection assumption\.
Such offline selection—CSS and its siblings—is fundamentally*static*: run once on the frozen base model, it fixes a problem\-to\-subset mapping baked into the training data and never revisited\. This assumes KP utility is*stationary*along the policy’s trajectory, that the best hint at initialization is still best after thousands of RL steps\. Yet RL exists to change the policy: as the model internalizes problem\-solving patterns, each KP’s marginal information shifts, some KPs becoming redundant and others newly diagnostic\.
#### Concurrent adaptive\-hint work is monolithic\.
Recent work introduces online adaptivity into hint injection \(GHPO\(Liuet al\.[2025](https://arxiv.org/html/2607.24833#bib.bib21)\), HINT\(Wanget al\.[2025](https://arxiv.org/html/2607.24833#bib.bib22)\), ADHint\(Zhanget al\.[2025a](https://arxiv.org/html/2607.24833#bib.bib23)\), Stepwise Hints\(Zhanget al\.[2025b](https://arxiv.org/html/2607.24833#bib.bib24)\)\) but treats each hint as a single monolithic prefix whose*quantity*or*granularity*scales with difficulty; a parallel line \(HIVE\(Wuet al\.[2026](https://arxiv.org/html/2607.24833#bib.bib27)\), Reinforce\-Ada\(Xionget al\.[2025](https://arxiv.org/html/2607.24833#bib.bib28)\)\) filters*whole prompts*\. Neither selects among*atomic*KPs\. To our knowledge, AdaKP is the first to perform*online subset selection of atomic knowledge points*during RL training\.
#### Cost of online re\-evaluation\.
Naively re\-running CSS or leave\-one\-out \(LOO\) marginal accuracy mid\-training is prohibitive: one CSS pass costs∼8×32\\sim\\\!8\\times 32rollouts×2\|C\|\\times\\,2^\{\|C\|\}subset evaluations per problem, consuming the bulk of a∼13\\sim\\\!13\-day run, so repeating it55–1010times is infeasible\.
#### Our contributions\.
We proposeAdaKP, an online, low\-cost replacement for offline KP selection during reasoning RL, framed around three coupled sub\-problems—cheap per\-step utility scoring, bias–variance control, and exploration of a discrete pool under bounded re\-evaluation—and contribute:
1. 1\.Entropy proxy\.We score a KPkkfor a problemqqby the reduction in next\-token entropy it induces in the policy:score\(q,k\)=H\(π\(⋅∣q\)\)−H\(π\(⋅∣q⊕k\)\)\\mathrm\{score\}\(q,k\)=H\(\\pi\(\\cdot\\mid q\)\)\-H\(\\pi\(\\cdot\\mid q\\oplus k\)\), estimated by a single vLLM\(Kwonet al\.[2023](https://arxiv.org/html/2607.24833#bib.bib14)\)forward pass over the firstK=50K\{=\}50generated tokens, whereq⊕kq\\oplus kdenotes the promptqqwithkkinserted into theHintblock \(the “\#\# Hint” section of the prompt template\)\. We show \(Prop\. S1, technical supplement\) that the truncation bias from top\-LLlog\-probability sampling is bounded by the realized tail mass, and empirically the proxy ranks KPs consistently with leave\-one\-out ground truth \(ρ=0\.68\\rho\{=\}0\.68; §[3\.1](https://arxiv.org/html/2607.24833#S3.SS1)\)\. This cuts per\-problem re\-evaluation from CSS’s𝒪\(rollouts×2\|C\|\)\\mathcal\{O\}\(\\text\{rollouts\}\\times 2^\{\|C\|\}\)or LOO’s𝒪\(\(\|C\|\+2\)N\)\\mathcal\{O\}\(\(\|C\|\{\+\}2\)N\)to\|C\|\+1\|C\|\{\+\}1short forwards \(Table[6](https://arxiv.org/html/2607.24833#S5.T6)\)\.
2. 2\.Three dynamic\-adaptation mechanisms\.We wrap the proxy in \(a\) an EMA*momentum smoother*that suppresses single\-step noise, \(b\) a*retirement\-and\-revival manager*that prunes persistently low\-utility KPs while preserving exploration through periodic random revival, and \(c\) an*adaptive scheduler*that front\-loads re\-evaluations to early training\. With the proxy frozen at init\-time weights \(§[6](https://arxiv.org/html/2607.24833#S6)\), these mechanisms supply*denoising and exploration*, not policy\-tracking; current\-policy re\-scoring is the natural next step\.
3. 3\.A pre\-training validation gate\.A Spearman rank\-correlation gate, scored against leave\-one\-out ground truth on stratified\-sampled problems, must clearρ\>0\.6\\rho\>0\.6before training begins, certifying the proxy against the gold signal\. This decouples method\-level risk from training cost; to our knowledge, online RL selection heuristics are not typically accompanied by such a pre\-flight check, though we do not claim this gate is the first validate\-before\-train idea in general\.
4. 4\.A diagnostic\-first design and open release\.We pair a main comparison with a component ablation mapped one\-to\-one onto the design rationale of §[3](https://arxiv.org/html/2607.24833#S3)and a five\-axis hyperparameter sweep, and release the additive DAPO\+GRPO fork\(Yuet al\.[2025](https://arxiv.org/html/2607.24833#bib.bib2); Shenget al\.[2025](https://arxiv.org/html/2607.24833#bib.bib15)\), training harness, proxy code, and gate scripts\.
## 2Related Work
#### Reasoning\-oriented RL for LLMs\.
RLHF\(Christianoet al\.[2017](https://arxiv.org/html/2607.24833#bib.bib10); Stiennonet al\.[2020](https://arxiv.org/html/2607.24833#bib.bib42); Ouyanget al\.[2022](https://arxiv.org/html/2607.24833#bib.bib7)\)via PPO\(Schulmanet al\.[2017](https://arxiv.org/html/2607.24833#bib.bib6)\)treats every prompt uniformly; later work simplifies the reward\-model recipe \(DPO\(Rafailovet al\.[2023](https://arxiv.org/html/2607.24833#bib.bib39)\), KTO\(Ethayarajhet al\.[2024](https://arxiv.org/html/2607.24833#bib.bib41)\), RLOO\(Ahmadianet al\.[2024](https://arxiv.org/html/2607.24833#bib.bib40)\), and self\-training\(Gulcehreet al\.[2023](https://arxiv.org/html/2607.24833#bib.bib43); Singhet al\.[2024](https://arxiv.org/html/2607.24833#bib.bib44)\)\)\. For mathematical reasoning,Havrillaet al\.\([2024](https://arxiv.org/html/2607.24833#bib.bib45)\)systematize RL recipes, GRPO\(Shaoet al\.[2024](https://arxiv.org/html/2607.24833#bib.bib3)\)replaces the value network with a group\-relative baseline, and DAPO\(Yuet al\.[2025](https://arxiv.org/html/2607.24833#bib.bib2)\)adds clip\-higher, token\-level loss, and dynamic sampling for verifiable\-reward settings; pure RLVR elicits long chain\-of\-thought at scale\(DeepSeek\-AI[2025](https://arxiv.org/html/2607.24833#bib.bib4); OpenAI[2024](https://arxiv.org/html/2607.24833#bib.bib5)\)\. AdaKP is*orthogonal*to these optimizer\-side advances: we modify the*prompt distribution*seen by the trainer, not the optimizer, and plug in unchanged on top of DAPO\+GRPO\.
#### Hint and knowledge injection\.
A first family asks the model to produce its own intermediate signals—Self\-Refine\(Madaanet al\.[2023](https://arxiv.org/html/2607.24833#bib.bib13)\), STaR\(Zelikmanet al\.[2022](https://arxiv.org/html/2607.24833#bib.bib11)\), Quiet\-STaR\(Zelikmanet al\.[2024](https://arxiv.org/html/2607.24833#bib.bib12)\), Reflexion\(Shinnet al\.[2023](https://arxiv.org/html/2607.24833#bib.bib37)\)—or densifies reward via process\-reward models\(Lightmanet al\.[2024](https://arxiv.org/html/2607.24833#bib.bib8); Wanget al\.[2024](https://arxiv.org/html/2607.24833#bib.bib9)\), but such self\-signals are unreliable without an external anchor\(Huanget al\.[2024](https://arxiv.org/html/2607.24833#bib.bib38)\), which motivates injecting curated text\. CSS\(Yuet al\.[2026](https://arxiv.org/html/2607.24833#bib.bib1)\)is the canonical external\-KP instance; AdaKP inherits its framing but rejects its offline static\-selection assumption\.
#### Adaptive guidance in concurrent RL work\.
A line of contemporaneous work introduces*adaptivity*into hint injection\. GHPO\(Liuet al\.[2025](https://arxiv.org/html/2607.24833#bib.bib21)\), HINT\(Wanget al\.[2025](https://arxiv.org/html/2607.24833#bib.bib22)\), ADHint\(Zhanget al\.[2025a](https://arxiv.org/html/2607.24833#bib.bib23)\)and Capability\-Adaptive Hint Scaffolding\(Liet al\.[2025](https://arxiv.org/html/2607.24833#bib.bib25)\)all scale the*quantity*or*ratio*of injected guidance with sample difficulty or recent reward dynamics; Stepwise Hints\(Zhanget al\.[2025b](https://arxiv.org/html/2607.24833#bib.bib24)\)varies the granularity of step\-level prefixes; Hint\-GRPO for multimodal reasoning\(Huanget al\.[2025](https://arxiv.org/html/2607.24833#bib.bib26)\)de\-biases hint utilization\. All of these methods decide*how much*or*at what granularity*to hint, but*still treat the hint as a single monolithic prefix*\. AdaKP instead performs*online subset selection over a discrete pool of atomic KPs*—an axis no prior method addresses \(Table S1, App\. C\)\.
#### Online prompt\-level filtering\.
Concurrent work—HIVE\(Wuet al\.[2026](https://arxiv.org/html/2607.24833#bib.bib27)\)and Reinforce\-Ada\(Xionget al\.[2025](https://arxiv.org/html/2607.24833#bib.bib28)\)—selects*which prompts*to roll out using prompt\-level entropy or reward dynamics\. This is orthogonal to AdaKP: it drops whole problems, whereas AdaKP chooses which KPs to keep*within*a problem; the two compose\.
#### Active selection and curriculum\.
Our entropy\-difference proxy follows the lineage of information\-theoretic active learning\(Kirschet al\.[2019](https://arxiv.org/html/2607.24833#bib.bib18)\), prioritized replay\(Schaulet al\.[2016](https://arxiv.org/html/2607.24833#bib.bib19)\), and information\-maximizing exploration\(Houthooftet al\.[2016](https://arxiv.org/html/2607.24833#bib.bib46)\), and our scheduler echoes curriculum learning\(Bengioet al\.[2009](https://arxiv.org/html/2607.24833#bib.bib20)\)\. The novelty is not the information\-gain criterion—widely established—but its application to a discrete pool of atomic, externally curated KPs under a budget that forbids per\-step rollouts\.
## 3Method: The AdaKP Selector
#### Setup and notation\.
For each problemqq, a candidate KP setC\(q\)=\{k1,…,k\|C\(q\)\|\}C\(q\)=\\\{k\_\{1\},\\dots,k\_\{\|C\(q\)\|\}\\\}is extracted from the gold solution and an offline subsetScss\(q\)⊆C\(q\)S^\{\\mathrm\{css\}\}\(q\)\\subseteq C\(q\)is produced by Constrained Subset Search \(CSS\), which enumerates subsets and scores each withN=32N\{=\}32rollouts on the frozen base model\(Yuet al\.[2026](https://arxiv.org/html/2607.24833#bib.bib1)\)\. We train with GRPO\(Shaoet al\.[2024](https://arxiv.org/html/2607.24833#bib.bib3)\)under the DAPO\(Yuet al\.[2025](https://arxiv.org/html/2607.24833#bib.bib2)\)recipe—clip\-higherεlow=0\.2\\varepsilon\_\{\\text\{low\}\}\{=\}0\.2/εhigh=0\.26\\varepsilon\_\{\\text\{high\}\}\{=\}0\.26, token\-level loss, dynamic\-sampling re\-rolls, no KL term—and verifiable rewardr\(o,q\)=𝟏\{math\-verify\(o,a\(q\)\)\}r\(o,q\)=\\mathbf\{1\}\\\{\\text\{math\-verify\}\(o,a\(q\)\)\\\}\. Throughout,πθ\\pi\_\{\\theta\}denotes the policy under training andπproxy\\pi^\{\\mathrm\{proxy\}\}a parameter\-frozen vLLM instance used solely for utility scoring;q⊕kq\\oplus kdenotes promptqqwith KPkkinjected into itsHintblock\.
#### Design rationale\.
Online KP selection imposes three constraints that rule out direct ports of CSS or off\-the\-shelf acquisition functions\.\(C1\)Scoring every\(q,k\)\(q,k\)pair on the∼8\.8\\sim\\\!8\.8k\-problem corpus must finish in*minutes*, so any multi\-rollout signal is excluded\.\(C2\)Single\-step proxy scores are noisy \(small\-KKentropy estimation\), so naive top\-kkselection churns\.\(C3\)The poolC\(q\)C\(q\)is small \(55–77KPs\), so a greedy selector collapses onto a high\-scoring subset and forecloses KPs useful only later\. AdaKP closes*exactly one*of these failure modes per component: the*entropy proxy*\(§[3\.1](https://arxiv.org/html/2607.24833#S3.SS1)\) collapses C1; the*momentum smoother*\(§[3\.2](https://arxiv.org/html/2607.24833#S3.SS2)\) absorbs C2’s noise; the*retirement\-and\-revival manager*\(§[3\.3](https://arxiv.org/html/2607.24833#S3.SS3)\) addresses C3 by forcing periodic re\-exploration; the*adaptive scheduler*\(§[3\.4](https://arxiv.org/html/2607.24833#S3.SS4)\) front\-loads re\-evaluation expense into early training, where re\-selection has the most steps left to pay off\. This one\-to\-one mapping makes the per\-component ablation \(§[5\.3](https://arxiv.org/html/2607.24833#S5.SS3)\) interpretable\.
#### Running state\.
Per problemqq:P\(q\)⊆C\(q\)P\(q\)\\subseteq C\(q\)is the active*KP pool*,sq,k\(t\)s\_\{q,k\}^\{\(t\)\}the smoothed score, andselq\\mathrm\{sel\}\_\{q\}the cached top\-ρsel\\rho\_\{\\text\{sel\}\}selection the trainer reads each step\. The smootherSqS\_\{q\}and retirement managerRqR\_\{q\}are per\-problem \(so problems prune at their own rate\); the scheduler𝒮\\mathcal\{S\}is a single global object\.
### 3\.1Entropy Proxy
#### Intuition\.
If a KPkkresolves part of the reasoning chain forqq, exposing the policy tokkshould concentrate its next\-token distribution—lowering entropy\. The expected entropy gapH\(π\(⋅∣q\)\)−𝔼kH\(π\(⋅∣q⊕k\)\)H\(\\pi\(\\cdot\\mid q\)\)\-\\mathbb\{E\}\_\{k\}H\(\\pi\(\\cdot\\mid q\\oplus k\)\)coincides, when the no\-KP distribution is read as thekk\-marginalized policy, with the conditional mutual informationI\(first token;k∣q\)I\(\\text\{first token\};k\\mid q\), so ranking KPs by entropy reduction ranks them, in expectation, by the information they carry about the first decoding decision—the quantity that drives reward sparsity in RLVR\.
#### Score definition\.
Given a problemqqand a candidate KPkk, the proxy score is
score\(q,k\)=H^\(πproxy\(⋅∣q\)\)−H^\(πproxy\(⋅∣q⊕k\)\),\\mathrm\{score\}\(q,k\)\\;=\\;\\widehat\{H\}\\bigl\(\\pi^\{\\text\{proxy\}\}\(\\cdot\\mid q\)\\bigr\)\\;\-\\;\\widehat\{H\}\\bigl\(\\pi^\{\\text\{proxy\}\}\(\\cdot\\mid q\\oplus k\)\\bigr\),\(1\)whereH^\\widehat\{H\}is the average first\-token Shannon entropy over the firstK=50K\{=\}50generated tokens andq⊕kq\\oplus kinjectskkinto theHintblock ofqq; positive scores meankkreduces the policy’s uncertainty about how to begin solvingqq\.
#### Truncated\-entropy estimator\.
Modern LLM serving stacks return only top\-LLlog\-probabilities\(Holtzmanet al\.[2020](https://arxiv.org/html/2607.24833#bib.bib47); Kwonet al\.[2023](https://arxiv.org/html/2607.24833#bib.bib14)\)per generated step to amortize sampling cost; for vLLM we haveL=20L\{=\}20\. We re\-normalize the truncated distribution and compute
H^t=−∑i≤Lp~ilogp~i,p~i=exp\(ℓi\)∑j≤Lexp\(ℓj\),\\widehat\{H\}\_\{t\}\\;=\\;\-\\sum\_\{i\\leq L\}\\tilde\{p\}\_\{i\}\\log\\tilde\{p\}\_\{i\},\\quad\\tilde\{p\}\_\{i\}=\\frac\{\\exp\(\\ell\_\{i\}\)\}\{\\sum\_\{j\\leq L\}\\exp\(\\ell\_\{j\}\)\},\(2\)whereℓi\\ell\_\{i\}are the returned log\-probabilities\. Truncation perturbsH^t\\widehat\{H\}\_\{t\}from the full Shannon entropy, but the bias is bounded by the tail massε\\varepsilon: Proposition S1, stated and proved in App\. A, givesH\(p\)−H^t\(p\)≤εlog\|V\|\+H2\(ε\)H\(p\)\-\\widehat\{H\}\_\{t\}\(p\)\\leq\\varepsilon\\log\|V\|\+H\_\{2\}\(\\varepsilon\), and≥0\\geq 0onceε≤1/L\\varepsilon\\leq 1/L, so the proxy score estimates the true entropy difference with𝒪\(ε⋆log\|V\|\)\\mathcal\{O\}\(\\varepsilon^\{\\star\}\\log\|V\|\)bias plus an𝒪\(K−1/2\)\\mathcal\{O\}\(K^\{\-1/2\}\)finite\-KKerror\.
In practice, withL=20L\{=\}20and the policy temperatureT=0\.7T\{=\}0\.7, the realized tail mass on the firstK=50K\{=\}50tokens of competition\-math generations is empiricallyε≲0\.05\\varepsilon\\lesssim 0\.05, so the residual is dominated by the finite\-KKaveraging term\. Whether the proxy additionally preserves the*ranking*of KPs is a stronger property than this bias bound; we support it empirically \(Spearmanρ=0\.68\\rho\{=\}0\.68against leave\-one\-out ground truth, Table[4](https://arxiv.org/html/2607.24833#S5.T4)\) rather than deriving it from Prop\. S1\.
#### Validation and cost\.
The proxy is validated against LOO marginal accuracy by the pre\-flight gate \(§[4\.4](https://arxiv.org/html/2607.24833#S4.SS4)\)\. A single re\-evaluation shares the no\-KP forward across all candidates, needing\|P\(q\)\|\+1\|P\(q\)\|\{\+\}1short forwards rather than2\|P\(q\)\|2\|P\(q\)\|, so re\-scoring the whole corpus adds under1%1\\%wall\-clock \(§[5\.6](https://arxiv.org/html/2607.24833#S5.SS6)\)\.
### 3\.2Momentum Smoother
Single\-step proxy scores are noisy \(small\-KKentropy estimation\), so we smooth with an exponential moving average maintained*per problem*:
sq,k\(t\)=α⋅score\(t\)\(q,k\)\+\(1−α\)⋅sq,k\(t−1\),s\_\{q,k\}^\{\(t\)\}=\\alpha\\cdot\\mathrm\{score\}^\{\(t\)\}\(q,k\)\+\(1\-\\alpha\)\\cdot s\_\{q,k\}^\{\(t\-1\)\},\(3\)withα=0\.3\\alpha=0\.3\(default\) andsq,k\(0\)=score\(0\)\(q,k\)s\_\{q,k\}^\{\(0\)\}=\\mathrm\{score\}^\{\(0\)\}\(q,k\), so AdaKP matches the static\-CSS baseline until the EMA accumulates\. Top\-fraction selection returns⌈ρsel⋅\|Aq\|⌉\\lceil\\rho\_\{\\text\{sel\}\}\\cdot\|A\_\{q\}\|\\rceilKPs with a floor of11, so theHintblock is never empty\.
### 3\.3Retirement and Revival
Persistently low\-ranked KPs waste re\-evaluation budget\. Per problemqqa countercq,kc\_\{q,k\}increments whenkkis evaluated but not selected and resets when selected; atcq,k≥nretirec\_\{q,k\}\\geq n\_\{\\text\{retire\}\}\(default33\) we retirekkfromP\(q\)P\(q\)\. Everymrevivem\_\{\\text\{revive\}\}re\-evaluations \(default55\) a fractionrrevive=0\.1r\_\{\\text\{revive\}\}\{=\}0\.1of retired KPs is revived at random withcq,k=0c\_\{q,k\}=0; if all KPs of a problem are retired \(not observed in practice\) we reuse the cached selection\. This combination—bounded counter, periodic revival, all\-retired safety—guarantees no KP is permanently silenced and keeps the active pool bounded while preserving exploration\.
### 3\.4Adaptive Re\-evaluation Scheduler
Re\-evaluation is wasted if the policy has not changed materially since the last call\. We adopt a logarithmic\-backoff schedule
Δ\(t\)=⌊Δ0⋅\(1\+log\(1\+tT⋅c\)\)⌋,\\Delta\(t\)=\\bigl\\lfloor\\Delta\_\{0\}\\cdot\\bigl\(1\+\\log\\bigl\(1\+\\tfrac\{t\}\{T\}\\cdot c\\bigr\)\\bigr\)\\bigr\\rfloor,\(4\)with base intervalΔ0=200\\Delta\_\{0\}=200, scalec=5c=5, and budgetT=2960T=2960steps\.ShouldReEvaluatereturns false att=0t\{=\}0, since the initial selection is the static CSS subset already in the training data; the first re\-evaluation fires att≥Δ0t\\geq\\Delta\_\{0\}and each later one whent−tlast≥Δ\(t\)t\-t\_\{\\text\{last\}\}\\geq\\Delta\(t\)\. This yields≈7\\approx\\\!7re\-evaluations, front\-loaded into early training where re\-selection has the most steps left to pay off\. A constant\-interval mode is exposed for theFixedScheduleablation\.
### 3\.5Integration
The integrator \(Algorithm S1, App\. B\) ties the four components together: each training step queries the scheduler, and when a re\-evaluation is due it calls the proxy for every active KP, updates the smoother, and advances the retirement manager; the dataset hook then reads the top\-ρsel\\rho\_\{\\text\{sel\}\}smoothed active KPs\. AdaKP plugs into the public DAPO\+verl recipe\(Yuet al\.[2025](https://arxiv.org/html/2607.24833#bib.bib2); Shenget al\.[2025](https://arxiv.org/html/2607.24833#bib.bib15)\)as a fully additive fork: a step hook in the training loop asks the selector whether a re\-evaluation is due, a dataset\-loader override swaps each prompt’sHintblock for the cachedselq\\mathrm\{sel\}\_\{q\}, and trainer\-init allocates a∼3\.5\\sim\\\!3\.5GB frozen\-weights proxy alongside the rollout vLLM\. With the module disabled the fork is bit\-identical to upstream, isolating the AdaKP contribution\.
## 4Experimental Setup
### 4\.1Models, Data, and Hardware
#### Base model\.
OpenMath\-Nemotron\-1\.5B\(NVIDIA[2024](https://arxiv.org/html/2607.24833#bib.bib29)\)\.
#### Reference checkpoint\.
The released static\-CSS checkpoint \(Nemotron\-1\.5B;Yuet al\.[2026](https://arxiv.org/html/2607.24833#bib.bib1)\), reproduced locally with the public checkpoint for the baseline column\.
#### Training data\.
The QuestA 8\.8k competition\-mathematics corpus\(Li and others[2025](https://arxiv.org/html/2607.24833#bib.bib30)\)\(N=8843N\{=\}8843problems\), with each problem paired with a candidate KP list extracted from gold solutions\. The CSS\-selected subset for each problem is taken from the publicly released CSS selections ofYuet al\.\([2026](https://arxiv.org/html/2607.24833#bib.bib1)\)\.
#### Trainer and hardware\.
We use verl\(Shenget al\.[2025](https://arxiv.org/html/2607.24833#bib.bib15)\)with the DAPO recipe and GRPO advantage estimator over an FSDP\(Zhaoet al\.[2023](https://arxiv.org/html/2607.24833#bib.bib16)\)backend with actor parameter offload, optimizer offload, and gradient checkpointing\. Training batch256256prompts; rolloutn=8n\{=\}8atT=1\.0T\{=\}1\.0/top\-p=1\.0p\{=\}1\.0; max prompt length81928192, max response length32,76832\{,\}768; learning rate1e−61\\mathrm\{e\}\{\-6\}\(no warm\-up, no schedule\); total29602960steps \(≈150\\approx\\\!150epochs over the filtered corpus\)\. Each run uses8×8\{\\times\}NVIDIA H800 SXM \(80 GB\), FlashAttention\-2\(Dao[2024](https://arxiv.org/html/2607.24833#bib.bib17)\), bfloat16 mixed precision, with the GPU\-memory fraction set to0\.500\.50for the rollout vLLM and0\.200\.20for the proxy vLLM \(proxy context length40964096\); the transient logits tensor is fit via verl’s chunked fused linear\-cross\-entropy kernel, a strict re\-arrangement that yields identical log\-probabilities\.
### 4\.2Evaluation Benchmarks
We report*mean@N*accuracy—the mean per\-sample correctness overNNrollouts \(the verifiable\-reward quantity GRPO optimizes\), evaluated under the multi\-sample sampling protocol that the*pass@k*metric ofChenet al\.\([2021](https://arxiv.org/html/2607.24833#bib.bib48)\)introduced—on eight benchmarks \(total1,3741\{,\}374problems, matching the protocol ofYuet al\.[2026](https://arxiv.org/html/2607.24833#bib.bib1)\): AIME24 \(N=32N\{=\}32,\|S\|=30\|S\|\{=\}30\), AIME25 \(N=32N\{=\}32,3030\), BRUMO25 \(N=32N\{=\}32,3030\), HMMT25 \(N=32N\{=\}32,3030\), AMC23 \(N=32N\{=\}32,4040\), CMIMC25 \(N=32N\{=\}32,4040\), MATH\-500\(Lightmanet al\.[2024](https://arxiv.org/html/2607.24833#bib.bib8)\)—a500500\-problem subset of MATH\(Hendryckset al\.[2021](https://arxiv.org/html/2607.24833#bib.bib32)\)—\(N=8N\{=\}8,500500\), and Olympiad\-Bench\(Heet al\.[2024](https://arxiv.org/html/2607.24833#bib.bib31)\)\(N=8N\{=\}8,674674\)\. The eight benchmarks sit above the GSM8K\(Cobbeet al\.[2021](https://arxiv.org/html/2607.24833#bib.bib33)\)difficulty regime that motivated the early process\-reward literature, which makes reward sparsity substantially more severe\. We report both the per\-benchmark accuracy and the*8\-benchmark average*\(uniformly weighted across benchmarks, not samples\)\.
### 4\.3Training Runs
We run17 training configurationsat seed4242\. They form three studies\.
#### Main comparison\.
One AdaKP run against the*static\-CSS baseline*, whose released checkpoint and published numbers we adopt rather than retraining the canonical static\-CSS model\. The run is capped at2,9602\{,\}960steps, at most∼13\\sim\\\!13wall\-clock days on8×8\{\\times\}H800, and estimates the average accuracy gain of online KP selection over the baseline\.
#### Component ablation\.
Five runs, each disabling one mechanism with the others at default:Full;NoProxy, a uniform random scorer;NoMomentum,α=1\.0\\alpha\{=\}1\.0;NoRetirement,nretire=∞n\_\{\\text\{retire\}\}\{=\}\\infty; andFixedSchedule, a constantΔ0\\Delta\_\{0\}\.
#### Hyperparameter sweep\.
Eleven runs varying one hyperparameter at a time around its default⋆\\star:α∈\{0\.1,0\.3⋆,0\.5,1\.0\}\\alpha\\in\\\{0\.1,0\.3^\{\\star\},0\.5,1\.0\\\},nretire∈\{2,3⋆,5\}n\_\{\\text\{retire\}\}\\in\\\{2,3^\{\\star\},5\\\},Δ0∈\{100,200⋆,400\}\\Delta\_\{0\}\\in\\\{100,200^\{\\star\},400\\\},ρsel∈\{0\.2,0\.3⋆,0\.4\}\\rho\_\{\\text\{sel\}\}\\in\\\{0\.2,0\.3^\{\\star\},0\.4\\\}, andK∈\{25,50⋆,100\}K\\in\\\{25,50^\{\\star\},100\\\}\.
### 4\.4Proxy Validation Gate
Before launching any training run we execute a Spearman go/no\-go gate on up to3030stratified\-sampled problems per benchmark; only problems with\|C\(q\)\|≥2\|C\(q\)\|\{\\geq\}2enter the gate \(so the realizednnper benchmark is1212–2727\)\. For each pair\(q,k\)\(q,k\)we compute \(a\) the LOO marginal accuracy withnruns=4n\_\{\\text\{runs\}\}\{=\}4independent batches of88rollouts each—that is,\(\|C\|\+2\)⋅4\(\|C\|\{\+\}2\)\\\!\\cdot\\\!4vLLM calls per problem—and \(b\) the entropy proxy score\. The gate requires Spearmanρ\>0\.6\\rho\>0\.6with95%95\\%bootstrap CI excluding0\.40\.4; proxies failing the gate are blocked from driving training\. We additionally benchmark three baseline scorers that share the entropy proxy’s API and per\-call cost, but differ in which statistic they extract:
- •*Max\-logit proxy:*score=logpmax\(q⊕k\)−logpmax\(q\)\\mathrm\{score\}=\\log p\_\{\\max\}\(q\\oplus k\)\-\\log p\_\{\\max\}\(q\)averaged over the firstKKtokens \(uses only the top\-11log\-probability\)\.
- •*Perplexity proxy:*score=logPPL\(q\)−logPPL\(q⊕k\)\\mathrm\{score\}=\\log\\mathrm\{PPL\}\(q\)\-\\log\\mathrm\{PPL\}\(q\\oplus k\)wherelogPPL\\log\\mathrm\{PPL\}is the negated mean top\-11log\-probability\.
- •*Random proxy:*uniform\[−1,1\]\[\-1,1\]scores; a true\-null control\.
The entropy proxy is the only one of the four that uses the full top\-LLdistribution \(Eq\.[1](https://arxiv.org/html/2607.24833#S3.E1)\); the max\-logit and perplexity proxies collapse it to the top\-11statistic\. Any accuracy gap among these three therefore quantifies information content, not compute \(Table[4](https://arxiv.org/html/2607.24833#S5.T4)\)\. In short, the gate returnsPassiffρ^\>ρ⋆=0\.6\\hat\{\\rho\}\>\\rho^\{\\star\}\{=\}0\.6*and*the lower95%95\\%bootstrap CI bound exceedsρmin=0\.4\\rho^\{\\min\}\{=\}0\.4; otherwise it returnsFailand training is not launched\.
## 5Experiments and Results
#### Status of the inventory\.
The non\-stationarity test \(§[5\.1](https://arxiv.org/html/2607.24833#S5.SS1)\) is the empirical core of this work, and the static\-CSS baseline throughout is the released checkpoint with its published per\-benchmark numbers\. The1717training runs,∼125\\sim\\\!125GPU\-days on8×8\{\\times\}H800, together with the leave\-one\-out pass for the proxy gate, populate the AdaKP andΔ\\Deltarows of Table[1](https://arxiv.org/html/2607.24833#S5.T1), all of Table[3](https://arxiv.org/html/2607.24833#S5.T3), the proxy rows of Table[4](https://arxiv.org/html/2607.24833#S5.T4), and Table[5](https://arxiv.org/html/2607.24833#S5.T5)\.
Table 1:Main comparison: AdaKP against published RL and hinting baselines on88benchmarks \(mean@NNin %, no KP hints at inference\)\. AdaKP populates theHintblock from its online selection cache; theΔ\\Deltarow is AdaKP minus the static\-CSS baseline\.*Hard\-3*averages AIME25, HMMT25 and CMIMC25;*Olymp\.*is Olympiad\-Bench;*Avg*is the uniformly weighted 8\-benchmark mean\. Baseline numbers are from the respective original papers\.Table 2:Offline KP\-selection strategies on the OpenMath backbone \(per\-benchmark mean@NNin %;*\#KP*= average selected subset size per problem\), registering the static\-selection family that AdaKP generalizes\. All strategies above the rule are*offline*\(subset fixed once before RL\); AdaKP \(bottom\) is*online*\. These rows are an offline, inference\-time evaluation and isolate*selection quality*at fixed compute—not directly comparable to the RL\-trained models of Table[1](https://arxiv.org/html/2607.24833#S5.T1)\.*Olymp\.*is Olympiad\-Bench; bold marks the best static\-strategy average\.
### 5\.1Non\-Stationarity of KP Importance \(H1\)
This test addresses the motivating hypothesis H1:*the most informative KPs differ between two policies that solve the same problem*\. For each of3030stratified\-sampled problems per benchmark we compute the LOO marginal accuracy of each candidate KP under bothπOpenMath\\pi\_\{\\text\{OpenMath\}\}\(untrained base\) andπtrained\\pi\_\{\\text\{trained\}\}\(trained policy\), pick the top\-30%30\\%KP set under each policy, and report the Jaccard similarity\. The pre\-specified decision rule is: Jaccard<0\.7<\\\!0\.7supports H1;0\.70\.7–0\.850\.85partial support;≥0\.85\\geq\\\!0\.85rejects H1 and would invalidate the AdaKP motivation\.
Across the eight benchmarks \(per\-benchmark Jaccard and realized sample sizesnn, for problems with\|C\(q\)\|≥2\|C\(q\)\|\\geq 2, in App\. D, Table S2\), the*global*mean Jaccard is0\.537\\mathbf\{0\.537\}—decisively below the0\.70\.7H1\_strongthreshold pre\-specified for the validation gate \(§[4\.4](https://arxiv.org/html/2607.24833#S4.SS4)\)\. Seven of the eight benchmarks individually fall below0\.70\.7; the largest divergences occur on AIME24 \(0\.3670\.367\), Olympiad\-Bench \(0\.3850\.385\), and MATH\-500 \(0\.4170\.417\)\. Only AIME25 exceeds the threshold \(0\.7300\.730\); since every entry is a coarse\-Jaccard measurement over a small problem pool \(App\. D\), we read this lone exception as run\-to\-run noise rather than evidence of genuine stationarity\. We therefore accept H1: KP importance is non\-stationary across policies, and a selection strategy that re\-evaluates during training has the room to outperform a frozen one\.
### 5\.2Main Comparison
Table[1](https://arxiv.org/html/2607.24833#S5.T1)reports per\-benchmark mean@NN\. The*Hard\-3*column averages the three reward\-sparsest benchmarks AIME25, HMMT25, and CMIMC25, where the static\-CSS baseline scores lowest;*Avg*is the uniformly weighted 8\-benchmark mean\.
The baselines are the strongest published1\.51\.5B systems under the no\-KP\-at\-inference setting: the OpenMath backbone\(NVIDIA[2024](https://arxiv.org/html/2607.24833#bib.bib29)\), the solution\-prefix method QuestA\(Li and others[2025](https://arxiv.org/html/2607.24833#bib.bib30)\), the RL recipe JustRL\(Heet al\.[2025](https://arxiv.org/html/2607.24833#bib.bib49)\), and the static\-CSS baseline\(Yuet al\.[2026](https://arxiv.org/html/2607.24833#bib.bib1)\)AdaKP builds on\. AdaKP raises the 8\-benchmark average to71\.9371\.93,\+1\.85\+1\.85over static CSS, and widens to\+3\.36\+3\.36on the reward\-sparsest*Hard\-3*subset\. The gains concentrate exactly where reward is sparsest and a frozen subset is least likely to stay optimal, and shrink to near zero on the already\-saturated AMC23 and MATH\-500; AdaKP leads on66of the88benchmarks\.
The shape of the improvement is informative rather than incidental: it is largest where the static baseline is weakest, where a subset fixed before training has most likely drifted from the policy, and vanishes near ceiling, so the gap tracks the headroom a frozen choice leaves on the table\. It is also obtained with no inference\-time hints—the subsets shape only the training prompts—so the model improves intrinsically\.
Beyond tracking headroom, two features of this profile bear on*why*the method helps, not merely*that*it does\. That the gains surface under sparse reward but not as a uniform shift is the signature of better\-chosen training*content*rather than altered optimisation; and no benchmark regresses, so online re\-selection is a strictly safe replacement for static CSS, not a redistribution that buys sparse\-reward accuracy at the saturated benchmarks’ expense\. Both observations place the improvement in selection*quality*, which the ablation that follows resolves into the contribution of each mechanism\.
#### Positioning against the static\-selection family\.
Static CSS is the strongest of a family of*offline*KP\-selection strategies, each fixing a subset once before RL and never revisiting it\. Evaluated on the OpenMath backbone with strategy\-selected KPs injected at inference \(Table[2](https://arxiv.org/html/2607.24833#S5.T2)\), accuracy climbs from no\-hint and naive all\-KP injection through the random, Max\-Score, LOO, and consensus variants to CSS, which scores best while injecting the fewest KPs;*which*subset is selected thus matters more than*how many*\. AdaKP keeps this subset\-selection view but makes it*online*, re\-selecting during training rather than committing to a single offline subset, which no member of the static family can do\.
### 5\.3Component Ablation
Table[3](https://arxiv.org/html/2607.24833#S5.T3)reports each ablation’s accuracy on the Hard\-3 subset \(AIME25, HMMT25, CMIMC25\) and the full 8\-benchmark average, plus two behavioral diagnostics\.*Pool churn*J¯t=1−1R∑r=1RJ\(sel\(r\),sel\(r−1\)\)\\bar\{J\}\_\{t\}=1\-\\tfrac\{1\}\{R\}\\sum\_\{r=1\}^\{R\}J\(\\mathrm\{sel\}^\{\(r\)\},\\mathrm\{sel\}^\{\(r\-1\)\}\)is the mean Jaccard*distance*between consecutive selection caches \(0= never changes,11= rewritten every call; a healthy run sits in0\.20\.2–0\.50\.5\)\.*KP coverage*C¯\\bar\{C\}is the fraction of the candidate pool ever selected during training \(C¯=1\\bar\{C\}\{=\}1= full exploration; lowC¯\\bar\{C\}= premature collapse\)\.
Table 3:Component ablation\.*Hard\-3*averages AIME25, HMMT25, CMIMC25;*Avg*is the 8\-benchmark mean\.J¯t\\bar\{J\}\_\{t\}is the pool\-churn diagnostic,C¯\\bar\{C\}is KP coverage \(both defined in the text\)\.Δ\\Deltais the change in*Avg*vs\.Full\.Each ablation isolates one cell of the C1–C3 design space \(§[3](https://arxiv.org/html/2607.24833#S3)\), and the diagnostics confirm the predicted patterns\. Removing the entropy proxy hurts most, as selection degenerates toward random at the highest churn; dropping momentum is next, churning on unsmoothed noise\. Disabling retirement collapses coverage, since the same few high\-scoring KPs are re\-selected while the rest go untested, and a fixed schedule re\-selects adequately but cannot adapt its rate\.Fullsits in the intended0\.20\.2–0\.50\.5churn band at high coverage, re\-selecting actively without chasing a moving target\. The two diagnostics thus trace one failure surface, accuracy falling when the selection thrashes, as without the proxy or smoother, and equally when it ossifies onto a few KPs, as without retirement\.
### 5\.4Proxy Comparison and Validation Gate
Table[4](https://arxiv.org/html/2607.24833#S5.T4)reports the Spearman correlation between four scoring functions and the LOO marginal\-accuracy ground truth over240240problems\. All proxies butRandomcorrelate significantly, yet significance alone is not enough: the gate also demands a large effect size with a lower CI bound above0\.40\.4, and only the entropy proxy clears it\. The max\-logit and perplexity proxies are significant but too weakly correlated to drive online selection\.
Table 4:Spearmanρ\\rhobetween four scoring functions and LOO marginal accuracy on240240problems \(3030per benchmark;95%95\\%bootstrap CI in brackets\), andppis the two\-sided significance ofρ\\rho\(Spearman correlationtt\-test,n=240n\{=\}240\)\.*Cost / call*is the wall\-clock time to score one\(q,k\)\(q,k\)pair on a single H800\. The gate \(§[4\.4](https://arxiv.org/html/2607.24833#S4.SS4)\) passes ifρ\>0\.6\\rho\>0\.6*and*the lower CI bound exceeds0\.40\.4—a criterion deliberately stricter than mere statistical significance\.Scorerρ\\rho\[CI\]ppCost / callGateRandom0\.020\.02\[−0\.13\-0\.13,0\.170\.17\]0\.760\.76<1<\\\!1msFailMax\-logit0\.410\.41\[0\.250\.25,0\.550\.55\]<10−3<\\\!10^\{\-3\}∼2\\sim\\\!2sFailPerplexity0\.530\.53\[0\.380\.38,0\.660\.66\]<10−3<\\\!10^\{\-3\}∼2\\sim\\\!2sFailEntropy \(ours\)0\.68\\mathbf\{0\.68\}\[0\.550\.55,0\.780\.78\]<10−3<\\\!10^\{\-3\}∼2\\sim\\\!2sPass*Reference:*LOO marginal accuracy≈200\\approx 200s/call \(N=8N\{=\}8\)\.The three forward\-pass proxies share an identical compute footprint and differ only in*which statistic*they extract, so any accuracy gap reflects information content, not compute; the two\-orders\-of\-magnitude gap to LOO is what makes online re\-evaluation possible\.
### 5\.5Hyperparameter Sensitivity
Sweeping each hyperparameter around its default moves the 8\-benchmark average by less than a single point in every one of the eleven runs, so no setting is brittle \(Table[5](https://arxiv.org/html/2607.24833#S5.T5)\)\. The selection fractionρsel\\rho\_\{\\text\{sel\}\}and the EMA coefficientα\\alphamatter most—too few KPs or no smoothing each cost about a point—while the retirement threshold, the proxy\-prefix length, and especially the re\-evaluation intervalΔ0\\Delta\_\{0\}stay essentially flat, so AdaKP is robust to a mis\-specified cadence\.
The flat re\-evaluation interval is the entry with the most practical weight: a practitioner inherits a working cadence without tuning—welcome, since tuning a schedule would otherwise demand the very repeated full\-length runs the online proxy exists to avoid\.
Read as a whole, the sweep cleaves the five knobs in two: those setting*how much*signal survives \(α\\alpha,ρsel\\rho\_\{\\text\{sel\}\}\) move accuracy, while the rest \(Δ0\\Delta\_\{0\},nretiren\_\{\\text\{retire\}\},KK\) barely register\.
Table 5:Hyperparameter sensitivity: the 8\-benchmark mean@NN\(%\) as each of the five hyperparameters is swept around the default \(⋆\\star\)\. The⋆\\starrows share the same full\-AdaKP configuration;Δ\\Deltais the change vs\. that default\.
### 5\.6Compute Cost and Overhead
Table[6](https://arxiv.org/html/2607.24833#S5.T6)compares the selection\-side compute of AdaKP against the two natural baselines\. Offline CSS \(N=32N\{=\}32rollouts per subset over an exponential candidate space\) is on the same order as the training run itself; the LOO column reflects our Stage 1 validation harness\. The AdaKP proxy needs only\|P\(q\)\|\+1\|P\(q\)\|\{\+\}1forward passes ofK=50K\{=\}50tokens per re\-evaluation \(the no\-KP forward is shared across a problem’s candidates\); withR≈7R\{\\approx\}7re\-evaluations over the8\.88\.8k\-problem corpus this is<1%<\\\!1\\%of the run’s wall\-clock\. AdaKP thus makes*online*KP selection feasible at a cost two\-to\-three orders of magnitude below what would be required to re\-run CSS at every checkpoint\.
Table 6:Selection\-side compute \(single H800\-GPU hours\)\.*Calls / problem*is the number of vLLM forward passes each method makes for each training problem across an entire training run;*N*roll*N\_\{\\text\{roll\}\}*is the rollouts per call\.*Training cost*is shared across all three rows \(∼2,500\\sim\\\!2\{,\}500H800\-GPU\-hours per run, i\.e\.8×8\\timesH800 for∼13\\sim\\\!13days at the reference protocol\)\.#### Compute budget\.
The1717runs total∼125\\sim\\\!125GPU\-days on8×8\{\\times\}H800: the main run at∼13\\sim\\\!13d and the1616ablation and sweep runs at∼7\\sim\\\!7d each\. The latter are truncated to roughly half the main\-run step budget,1,5001\{,\}500versus2,9602\{,\}960steps, to fit one submission cycle while still resolving the qualitative ranking of conditions; a pilot confirms that the method ordering at step1,5001\{,\}500matches that at2,9602\{,\}960\.
### 5\.7Reproducibility Statement
To support full reproducibility we will release the additive verl/DAPO fork, the standalone Python implementation of the four algorithmic components and the validation gate, and the per\-step training logs that underpin Table[3](https://arxiv.org/html/2607.24833#S5.T3)’s behavioral diagnostics\. Random seeds are exposed at the launcher level; the only non\-deterministic operations are bfloat16 mixed precision and the DAPO\-default GRPO group\-relative baseline\.
## 6Discussion and Limitations
#### Why use LOO rather than CSS as the ranking ground truth?
The released CSS artifacts give only the output subsets and the trained checkpoint, not the search code\. The H1 test \(§[5\.1](https://arxiv.org/html/2607.24833#S5.SS1)\) and the validation gate \(§[4\.4](https://arxiv.org/html/2607.24833#S4.SS4)\) therefore use leave\-one\-out \(LOO\) marginal accuracy as the ground\-truth ranking signal\. LOO is the core mechanism inside CSS, so the substitution preserves the ranking semantics at two orders of magnitude less compute \(Table[6](https://arxiv.org/html/2607.24833#S5.T6)\); it affects only how we*validate*the proxy, not the headline comparison\.
#### Scope of the frozen proxy, and the role of H1\.
H1 \(§[5\.1](https://arxiv.org/html/2607.24833#S5.SS1)\) shows that the*ideal*selector would re\-rank KPs against the policy’s*current*state—which is what makes adaptive, rather than one\-shot, selection worth pursuing\. We do not yet realize that ideal: the entropy proxy is frozen at the base\-model weights, because mid\-training reload from the FSDP\-sharded trainer\(Zhaoet al\.[2023](https://arxiv.org/html/2607.24833#bib.bib16)\)is not exposed by verl’s actor–rollout worker\-group interface\. AdaKP is thus a*first, deployable step*: it re\-selects online from a frozen base\-policy ranking augmented by EMA denoising and retirement/revival exploration\. Its gain over the single fixed CSS subset \(Table[1](https://arxiv.org/html/2607.24833#S5.T1)\) comes from the information\-theoretic criterion and from exploring beyond CSS’s one\-shot choice—*not*from tracking the policy drift H1 documents, which a frozen proxy cannot observe\. Scoring against current\-iteration weights is therefore not a cosmetic “v1\.1” nicety but the most direct route to convert H1’s headroom into accuracy, and is the principal next step this paper sets up\.
#### Will the result transfer beyond 1\.5B?
All experiments are at the1\.51\.5B scale\. The proxy and adaptation mechanisms are defined purely on the policy’s next\-token distribution and exploit no1\.51\.5B\-specific structure, so we expect them to transfer; a77B replication is the natural next step\.
#### What this paper does not claim\.
We do not claim that the entropy proxy is the only viable cheap signal, that the four\-component structure is uniquely optimal, or that AdaKP supersedes external\-KP curation\. The narrower claim we defend is that*which subset of atomic KPs to inject is a real axis of choice that prior work leaves on the table*, and that AdaKP offers a tractable, validated mechanism to act on it\.
## 7Conclusion
We argued that hint injection for reasoning RL has missed a degree of freedom:*which subset of atomic knowledge points to inject during training*\. We verified that KP importance is non\-stationary across training, and designed AdaKP: an entropy proxy with a bounded\-bias guarantee, wrapped in three decoupled adaptation mechanisms and validated by a pre\-flight gate executable in CPU\-minutes before any multi\-GPU\-day run\. As an additive DAPO\+GRPO fork it improves over static CSS across eight benchmarks at under1%1\\%added cost\.
Two messages we hope outlast the headline accuracy number: that*within\-prompt content selection*is a tractable and as\-yet unexploited axis distinct from prompt\-level filtering and hint quantity scaling; and that lightweight*pre\-flight validation gates*on cheap proxies can become a routine part of reasoning\-RL methodology, decoupling method\-level risk from training\-level cost\.
## References
- A\. Ahmadian, C\. Cremer, M\. Gallé, M\. Fadaee, J\. Kreutzer, O\. Pietquin, A\. Üstün, and S\. Hooker \(2024\)Back to basics: revisiting REINFORCE style optimization for learning from human feedback in LLMs\.InAnnual Meeting of the Association for Computational Linguistics \(ACL\),Note:arXiv:2402\.14740Cited by:[§2](https://arxiv.org/html/2607.24833#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Bengio, J\. Louradour, R\. Collobert, and J\. Weston \(2009\)Curriculum learning\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§2](https://arxiv.org/html/2607.24833#S2.SS0.SSS0.Px5.p1.1)\.
- M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. d\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman,et al\.\(2021\)Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[§4\.2](https://arxiv.org/html/2607.24833#S4.SS2.p1.19)\.
- P\. F\. Christiano, J\. Leike, T\. B\. Brown, M\. Martic, S\. Legg, and D\. Amodei \(2017\)Deep reinforcement learning from human preferences\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2607.24833#S2.SS0.SSS0.Px1.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman \(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§4\.2](https://arxiv.org/html/2607.24833#S4.SS2.p1.19)\.
- T\. Dao \(2024\)FlashAttention\-2: faster attention with better parallelism and work partitioning\.International Conference on Learning Representations \(ICLR\)\.Note:arXiv:2307\.08691Cited by:[§4\.1](https://arxiv.org/html/2607.24833#S4.SS1.SSS0.Px4.p1.13)\.
- DeepSeek\-AI \(2025\)DeepSeek\-R1: incentivizing reasoning capability in LLMs via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§1](https://arxiv.org/html/2607.24833#S1.p1.4),[§2](https://arxiv.org/html/2607.24833#S2.SS0.SSS0.Px1.p1.1)\.
- K\. Ethayarajh, W\. Xu, N\. Muennighoff, D\. Jurafsky, and D\. Kiela \(2024\)KTO: model alignment as prospect theoretic optimization\.InInternational Conference on Machine Learning \(ICML\),Note:arXiv:2402\.01306Cited by:[§2](https://arxiv.org/html/2607.24833#S2.SS0.SSS0.Px1.p1.1)\.
- C\. Gulcehre, T\. Le Paine, S\. Srinivasan, K\. Konyushkova, L\. Weerts, A\. Sharma, A\. Siddhant, A\. Ahern, M\. Wang, C\. Gu, W\. Macherey, A\. Doucet, O\. Firat, and N\. de Freitas \(2023\)Reinforced self\-training \(ReST\) for language modeling\.arXiv preprint arXiv:2308\.08998\.Cited by:[§2](https://arxiv.org/html/2607.24833#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Havrilla, Y\. Du, S\. C\. Raparthy, C\. Nalmpantis, J\. Dwivedi\-Yu, M\. Zhuravinskyi, E\. Hambro, S\. Sukhbaatar, and R\. Raileanu \(2024\)Teaching large language models to reason with reinforcement learning\.arXiv preprint arXiv:2403\.04642\.Cited by:[§2](https://arxiv.org/html/2607.24833#S2.SS0.SSS0.Px1.p1.1)\.
- B\. He, Z\. Qu, Z\. Liu, Y\. Chen, Y\. Zuo, C\. Qian, K\. Zhang, W\. Chen, C\. Xiao, G\. Cui, N\. Ding, and Z\. Liu \(2025\)JustRL: scaling a 1\.5B LLM with a simple RL recipe\.arXiv preprint arXiv:2512\.16649\.Cited by:[§5\.2](https://arxiv.org/html/2607.24833#S5.SS2.p2.6)\.
- C\. He, R\. Luo, Y\. Bai, S\. Hu, Z\. L\. Thai, J\. Shen, J\. Hu, X\. Han, Y\. Huang, Y\. Zhang, J\. Liu, L\. Qi, Z\. Liu, and M\. Sun \(2024\)OlympiadBench: a challenging benchmark for promoting AGI with olympiad\-level bilingual multimodal scientific problems\.arXiv preprint arXiv:2402\.14008\.Cited by:[§4\.2](https://arxiv.org/html/2607.24833#S4.SS2.p1.19)\.
- D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt \(2021\)Measuring mathematical problem solving with the MATH dataset\.Advances in Neural Information Processing Systems \(NeurIPS\)\.Note:arXiv:2103\.03874Cited by:[§4\.2](https://arxiv.org/html/2607.24833#S4.SS2.p1.19)\.
- A\. Holtzman, J\. Buys, L\. Du, M\. Forbes, and Y\. Choi \(2020\)The curious case of neural text degeneration\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:1904\.09751Cited by:[§3\.1](https://arxiv.org/html/2607.24833#S3.SS1.SSS0.Px3.p1.2)\.
- R\. Houthooft, X\. Chen, Y\. Duan, J\. Schulman, F\. De Turck, and P\. Abbeel \(2016\)VIME: variational information maximizing exploration\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:1605\.09674Cited by:[§2](https://arxiv.org/html/2607.24833#S2.SS0.SSS0.Px5.p1.1)\.
- J\. Huang, X\. Chen, S\. Mishra, H\. S\. Zheng, A\. W\. Yu, X\. Song, and D\. Zhou \(2024\)Large language models cannot self\-correct reasoning yet\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2310\.01798Cited by:[§1](https://arxiv.org/html/2607.24833#S1.p1.4),[§2](https://arxiv.org/html/2607.24833#S2.SS0.SSS0.Px2.p1.1)\.
- Q\. Huang, W\. Dai, J\. Liu, W\. He, H\. Jiang, M\. Song, J\. Chen, C\. Yao, and J\. Song \(2025\)Boosting MLLM reasoning with text\-debiased Hint\-GRPO\.InProceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\),Cited by:[Table S1](https://arxiv.org/html/2607.24833#A3.T1.7.7.6.1),[§2](https://arxiv.org/html/2607.24833#S2.SS0.SSS0.Px3.p1.1)\.
- A\. Kirsch, J\. van Amersfoort, and Y\. Gal \(2019\)BatchBALD: efficient and diverse batch acquisition for deep bayesian active learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2607.24833#S2.SS0.SSS0.Px5.p1.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with PagedAttention\.InACM Symposium on Operating Systems Principles \(SOSP\),Cited by:[item 1](https://arxiv.org/html/2607.24833#S1.I1.i1.p1.12),[§3\.1](https://arxiv.org/html/2607.24833#S3.SS1.SSS0.Px3.p1.2)\.
- J\. Liet al\.\(2025\)QuestA: expanding reasoning capacity in LLMs via question augmentation\.arXiv preprint arXiv:2507\.13266\.Cited by:[§4\.1](https://arxiv.org/html/2607.24833#S4.SS1.SSS0.Px3.p1.1),[§5\.2](https://arxiv.org/html/2607.24833#S5.SS2.p2.6)\.
- Z\. Li, Z\. Sun, J\. Zhao, E\. Min, Y\. Zeng, H\. Wu, H\. Cai, S\. Wang, D\. Yin, X\. Chen, and Z\. Deng \(2025\)Staying in the sweet spot: responsive reasoning evolution via capability\-adaptive hint scaffolding\.arXiv preprint arXiv:2509\.06923\.Cited by:[§2](https://arxiv.org/html/2607.24833#S2.SS0.SSS0.Px3.p1.1)\.
- H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe \(2024\)Let’s verify step by step\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2607.24833#S2.SS0.SSS0.Px2.p1.1),[§4\.2](https://arxiv.org/html/2607.24833#S4.SS2.p1.19)\.
- Z\. Liu, C\. Gong, X\. Fu, Y\. Liu, R\. Chen, S\. Hu, S\. Zhang, R\. Liu, Q\. Zhang, and D\. Tu \(2025\)GHPO: adaptive guidance for stable and efficient LLM reinforcement learning\.arXiv preprint arXiv:2507\.10628\.Cited by:[Table S1](https://arxiv.org/html/2607.24833#A3.T1.7.3.2.1),[§1](https://arxiv.org/html/2607.24833#S1.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2607.24833#S2.SS0.SSS0.Px3.p1.1)\.
- A\. Madaan, N\. Tandon, P\. Gupta, S\. Hallinan, L\. Gao, S\. Wiegreffe, U\. Alon, N\. Dziri, S\. Prabhumoye, Y\. Yang, S\. Gupta, B\. P\. Majumder, K\. Hermann, S\. Welleck, A\. Yazdanbakhsh, and P\. Clark \(2023\)Self\-refine: iterative refinement with self\-feedback\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2607.24833#S2.SS0.SSS0.Px2.p1.1)\.
- NVIDIA \(2024\)OpenMath\-Nemotron\-1\.5B\.Note:Hugging Face:https://huggingface\.co/nvidia/OpenMath\-Nemotron\-1\.5BCited by:[§4\.1](https://arxiv.org/html/2607.24833#S4.SS1.SSS0.Px1.p1.1),[§5\.2](https://arxiv.org/html/2607.24833#S5.SS2.p2.6)\.
- OpenAI \(2024\)Learning to reason with LLMs\.Note:https://openai\.com/index/learning\-to\-reason\-with\-llms/Cited by:[§1](https://arxiv.org/html/2607.24833#S1.p1.4),[§2](https://arxiv.org/html/2607.24833#S2.SS0.SSS0.Px1.p1.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2607.24833#S2.SS0.SSS0.Px1.p1.1)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell, S\. Ermon, C\. D\. Manning, and C\. Finn \(2023\)Direct preference optimization: your language model is secretly a reward model\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2305\.18290Cited by:[§2](https://arxiv.org/html/2607.24833#S2.SS0.SSS0.Px1.p1.1)\.
- T\. Schaul, J\. Quan, I\. Antonoglou, and D\. Silver \(2016\)Prioritized experience replay\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2607.24833#S2.SS0.SSS0.Px5.p1.1)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§2](https://arxiv.org/html/2607.24833#S2.SS0.SSS0.Px1.p1.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. Guo \(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Note:Introduces the GRPO advantage estimatorCited by:[§1](https://arxiv.org/html/2607.24833#S1.p1.4),[§2](https://arxiv.org/html/2607.24833#S2.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2607.24833#S3.SS0.SSS0.Px1.p1.12)\.
- G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. Wu \(2025\)HybridFlow: a flexible and efficient RLHF framework\.Proceedings of the European Conference on Computer Systems \(EuroSys\)\.Note:Underlies theverllibrary used in this workCited by:[item 4](https://arxiv.org/html/2607.24833#S1.I1.i4.p1.1),[§3\.5](https://arxiv.org/html/2607.24833#S3.SS5.p1.3),[§4\.1](https://arxiv.org/html/2607.24833#S4.SS1.SSS0.Px4.p1.13)\.
- N\. Shinn, F\. Cassano, E\. Berman, A\. Gopinath, K\. Narasimhan, and S\. Yao \(2023\)Reflexion: language agents with verbal reinforcement learning\.Advances in Neural Information Processing Systems \(NeurIPS\)\.Note:arXiv:2303\.11366Cited by:[§2](https://arxiv.org/html/2607.24833#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Singh, J\. D\. Co\-Reyes, R\. Agarwal, A\. Anand, P\. Patil, X\. Garcia, P\. J\. Liu, J\. Harrison, J\. Lee, K\. Xu, A\. Parisi, A\. Kumar, A\. Alemi, A\. Rizkowsky, A\. Nova, B\. Adlam, B\. Bohnet, G\. Elsayed, H\. Sedghi, I\. Mordatch, I\. Simpson, I\. Gur, J\. Snoek, J\. Pennington, J\. Hron, K\. Kenealy, K\. Swersky, K\. Mahajan, L\. Culp, L\. Xiao, M\. L\. Bileschi, N\. Constant, R\. Novak, R\. Liu, T\. Warkentin, Y\. Qian, Y\. Bansal, E\. Dyer, B\. Neyshabur, J\. Sohl\-Dickstein, and N\. Fiedel \(2024\)Beyond human data: scaling self\-training for problem\-solving with language models\.Transactions on Machine Learning Research \(TMLR\)\.Note:arXiv:2312\.06585Cited by:[§2](https://arxiv.org/html/2607.24833#S2.SS0.SSS0.Px1.p1.1)\.
- N\. Stiennon, L\. Ouyang, J\. Wu, D\. M\. Ziegler, R\. Lowe, C\. Voss, A\. Radford, D\. Amodei, and P\. Christiano \(2020\)Learning to summarize from human feedback\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2009\.01325Cited by:[§2](https://arxiv.org/html/2607.24833#S2.SS0.SSS0.Px1.p1.1)\.
- P\. Wang, L\. Li, Z\. Shao, R\. X\. Xu, D\. Dai, Y\. Li, D\. Chen, Y\. Wu, and Z\. Sui \(2024\)Math\-Shepherd: verify and reinforce LLMs step\-by\-step without human annotations\.InAnnual Meeting of the Association for Computational Linguistics \(ACL\),Note:arXiv:2312\.08935Cited by:[§2](https://arxiv.org/html/2607.24833#S2.SS0.SSS0.Px2.p1.1)\.
- X\. Wang, J\. Han, Z\. Jiang, T\. Li, J\. Liang, S\. Jiang, Z\. Dai, S\. Ma, F\. Yu, and Y\. Xiao \(2025\)Don’t tell the answer, truly guide the reasoning during RL rollouts\.arXiv preprint arXiv:2510\.09388\.Cited by:[Table S1](https://arxiv.org/html/2607.24833#A3.T1.7.4.3.1),[§1](https://arxiv.org/html/2607.24833#S1.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2607.24833#S2.SS0.SSS0.Px3.p1.1)\.
- X\. Wang, J\. Wei, D\. Schuurmans, Q\. Le, E\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou \(2023\)Self\-consistency improves chain of thought reasoning in language models\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2203\.11171Cited by:[§1](https://arxiv.org/html/2607.24833#S1.p1.4)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. Chi, Q\. Le, and D\. Zhou \(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2201\.11903Cited by:[§1](https://arxiv.org/html/2607.24833#S1.p1.4)\.
- J\. Wu, N\. Lu, S\. Liu, K\. Wang, Y\. Yang, B\. Lin, C\. J\. Zhang, Q\. Li, and K\. Tang \(2026\)Train at moving edge: online\-verified prompt selection for efficient RL training of large reasoning model\.arXiv preprint arXiv:2603\.25184\.Cited by:[Table S1](https://arxiv.org/html/2607.24833#A3.T1.7.8.7.1),[§1](https://arxiv.org/html/2607.24833#S1.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2607.24833#S2.SS0.SSS0.Px4.p1.1)\.
- W\. Xiong, C\. Ye, B\. Liao, H\. Dong, X\. Xu, C\. Monz, J\. Bian, N\. Jiang, and T\. Zhang \(2025\)Reinforce\-Ada: an adaptive sampling framework under non\-linear RL objectives\.arXiv preprint arXiv:2510\.04996\.Cited by:[§1](https://arxiv.org/html/2607.24833#S1.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2607.24833#S2.SS0.SSS0.Px4.p1.1)\.
- S\. Yao, D\. Yu, J\. Zhao, I\. Shafran, T\. L\. Griffiths, Y\. Cao, and K\. Narasimhan \(2023\)Tree of thoughts: deliberate problem solving with large language models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2305\.10601Cited by:[§1](https://arxiv.org/html/2607.24833#S1.p1.4)\.
- L\. Yu, T\. Yang, S\. Ding, R\. Jin, N\. Gu, X\. Hao, S\. Nie, D\. Xiong, W\. Yin, Y\. Sun, and H\. Wu \(2026\)KnowRL: boosting LLM reasoning via reinforcement learning with minimal\-sufficient knowledge guidance\.arXiv preprint arXiv:2604\.12627\.Note:Code:https://github\.com/Hasuer/KnowRLCited by:[Table S1](https://arxiv.org/html/2607.24833#A3.T1.7.2.1.1),[§1](https://arxiv.org/html/2607.24833#S1.p1.4),[§2](https://arxiv.org/html/2607.24833#S2.SS0.SSS0.Px2.p1.1),[§3](https://arxiv.org/html/2607.24833#S3.SS0.SSS0.Px1.p1.12),[§4\.1](https://arxiv.org/html/2607.24833#S4.SS1.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2607.24833#S4.SS1.SSS0.Px3.p1.1),[§4\.2](https://arxiv.org/html/2607.24833#S4.SS2.p1.19),[§5\.2](https://arxiv.org/html/2607.24833#S5.SS2.p2.6)\.
- Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, T\. Fan, G\. Liu, L\. Liu, X\. Liu,et al\.\(2025\)DAPO: an open\-source LLM reinforcement learning system at scale\.arXiv preprint arXiv:2503\.14476\.Note:ByteDance Seed and Tsinghua AIRCited by:[item 4](https://arxiv.org/html/2607.24833#S1.I1.i4.p1.1),[§1](https://arxiv.org/html/2607.24833#S1.p1.4),[§2](https://arxiv.org/html/2607.24833#S2.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2607.24833#S3.SS0.SSS0.Px1.p1.12),[§3\.5](https://arxiv.org/html/2607.24833#S3.SS5.p1.3)\.
- E\. Zelikman, G\. Harik, Y\. Shao, V\. Jayasiri, N\. Haber, and N\. D\. Goodman \(2024\)Quiet\-STaR: language models can teach themselves to think before speaking\.arXiv preprint arXiv:2403\.09629\.Cited by:[§2](https://arxiv.org/html/2607.24833#S2.SS0.SSS0.Px2.p1.1)\.
- E\. Zelikman, Y\. Wu, J\. Mu, and N\. D\. Goodman \(2022\)STaR: bootstrapping reasoning with reasoning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2607.24833#S2.SS0.SSS0.Px2.p1.1)\.
- F\. Zhang, Z\. Tan, X\. Ma, Z\. Dong, X\. Leng, J\. Zhao, X\. Sun, and Y\. Yang \(2025a\)ADHint: adaptive hints with difficulty priors for reinforcement learning\.arXiv preprint arXiv:2512\.13095\.Cited by:[Table S1](https://arxiv.org/html/2607.24833#A3.T1.7.5.4.1),[§1](https://arxiv.org/html/2607.24833#S1.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2607.24833#S2.SS0.SSS0.Px3.p1.1)\.
- K\. Zhang, A\. Lv, J\. Li, Y\. Wang, F\. Wang, H\. Hu, and R\. Yan \(2025b\)StepHint: multi\-level stepwise hints enhance reinforcement learning to reason\.arXiv preprint arXiv:2507\.02841\.Cited by:[Table S1](https://arxiv.org/html/2607.24833#A3.T1.7.6.5.1),[§1](https://arxiv.org/html/2607.24833#S1.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2607.24833#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Zhao, A\. Gu, R\. Varma, L\. Luo, C\. Huang, M\. Xu, L\. Wright, H\. Shojanazeri, M\. Ott, S\. Shleifer,et al\.\(2023\)PyTorch FSDP: experiences on scaling fully sharded data parallel\.Proceedings of the VLDB Endowment\.Note:arXiv:2304\.11277Cited by:[§4\.1](https://arxiv.org/html/2607.24833#S4.SS1.SSS0.Px4.p1.13),[§6](https://arxiv.org/html/2607.24833#S6.SS0.SSS0.Px2.p1.1)\.
This technical supplement collects the material referenced from the main paper: the bounded\-bias proof \(App\. A\), the training\-time algorithm \(App\. B\), the design\-space positioning table \(App\. C\), the full per\-benchmark non\-stationarity table \(App\. D\), and additional reproducibility notes \(App\. E\)\. Section, proposition, algorithm, and table labels carry the letter/“S” prefixes used when the main paper cites this document; equation and figure numbers are likewise “S”\-prefixed\.
## Appendix ABounded Truncation Bias
###### Proposition S1\(Bounded truncation bias\)\.
Fix a generated position with full next\-token distributionppover a vocabulary of size\|V\|\|V\|\. LetTL\(p\)T\_\{L\}\(p\)be its top\-LLindex set \(the*head*\),ε:=∑i∉TL\(p\)pi\\varepsilon:=\\sum\_\{i\\notin T\_\{L\}\(p\)\}p\_\{i\}the tail mass, andH^t\(p\)=−∑i∈TL\(p\)p~ilogp~i\\widehat\{H\}\_\{t\}\(p\)=\-\\sum\_\{i\\in T\_\{L\}\(p\)\}\\tilde\{p\}\_\{i\}\\log\\tilde\{p\}\_\{i\}the truncated\-and\-renormalized estimator withp~i=pi/\(1−ε\)\\tilde\{p\}\_\{i\}=p\_\{i\}/\(1\-\\varepsilon\)\. WriteH2H\_\{2\}for the binary entropy\. The truncation biasΔ\(p,L\)=H\(p\)−H^t\(p\)\\Delta\(p,L\)=H\(p\)\-\\widehat\{H\}\_\{t\}\(p\)satisfies, for everypp,
Δ\(p,L\)≤εlog\|V\|\+H2\(ε\),\\Delta\(p,L\)\\ \\leq\\ \\varepsilon\\log\|V\|\+H\_\{2\}\(\\varepsilon\),\(S1\)and is nonnegative,0≤Δ\(p,L\)0\\leq\\Delta\(p,L\), wheneverε≤1/L\\varepsilon\\leq 1/L\. Consequently, for two promptsqqandq⊕kq\\oplus kwhose distributions both satisfyε≤ε⋆\\varepsilon\\leq\\varepsilon^\{\\star\}, the proxy scoreH^\(⋅∣q\)−H^\(⋅∣q⊕k\)\\widehat\{H\}\(\\cdot\\mid q\)\-\\widehat\{H\}\(\\cdot\\mid q\\oplus k\)estimates the true entropy differenceH\(⋅∣q\)−H\(⋅∣q⊕k\)H\(\\cdot\\mid q\)\-H\(\\cdot\\mid q\\oplus k\)with a*bias*of magnitude𝒪\(ε⋆log\|V\|\)\\mathcal\{O\}\(\\varepsilon^\{\\star\}\\log\|V\|\), plus an𝒪\(K−1/2\)\\mathcal\{O\}\(K^\{\-1/2\}\)stochastic error from averaging overKKpositions\.
###### Proof\.
Partition the vocabulary into the headS=TL\(p\)S=T\_\{L\}\(p\)\(mass1−ε1\-\\varepsilon\) and the tailS¯\\bar\{S\}\(massε\\varepsilon\), and letpS,pS¯p\_\{S\},p\_\{\\bar\{S\}\}bepprestricted and renormalized toS,S¯S,\\bar\{S\}\. By constructionpS=p~p\_\{S\}=\\tilde\{p\}, soH\(pS\)=H^t\(p\)H\(p\_\{S\}\)=\\widehat\{H\}\_\{t\}\(p\)\. LetB=𝟏\[i∈S\]B=\\mathbf\{1\}\[i\\in S\]be the head/tail indicator; thenH\(B\)=H2\(ε\)H\(B\)=H\_\{2\}\(\\varepsilon\), and the grouping \(chain\-rule\) identity for Shannon entropy gives the*exact*decomposition
H\(p\)=H\(B\)\+\(1−ε\)H\(pS\)\+εH\(pS¯\)=H2\(ε\)\+\(1−ε\)H^t\(p\)\+εH\(pS¯\)\.\\begin\{split\}H\(p\)&=H\(B\)\+\(1\-\\varepsilon\)H\(p\_\{S\}\)\+\\varepsilon H\(p\_\{\\bar\{S\}\}\)\\\\ &=H\_\{2\}\(\\varepsilon\)\+\(1\-\\varepsilon\)\\widehat\{H\}\_\{t\}\(p\)\+\\varepsilon H\(p\_\{\\bar\{S\}\}\)\.\\end\{split\}\(S2\)SubtractingH^t\(p\)\\widehat\{H\}\_\{t\}\(p\),
Δ\(p,L\)=H2\(ε\)\+ε\(H\(pS¯\)−H^t\(p\)\)\.\\Delta\(p,L\)=H\_\{2\}\(\\varepsilon\)\+\\varepsilon\\bigl\(H\(p\_\{\\bar\{S\}\}\)\-\\widehat\{H\}\_\{t\}\(p\)\\bigr\)\.\(S3\)
*Upper bound\.*In \([S3](https://arxiv.org/html/2607.24833#A1.E3)\),H^t\(p\)≥0\\widehat\{H\}\_\{t\}\(p\)\\geq 0andH\(pS¯\)≤log\|S¯\|≤log\|V\|H\(p\_\{\\bar\{S\}\}\)\\leq\\log\|\\bar\{S\}\|\\leq\\log\|V\|, soΔ\(p,L\)≤H2\(ε\)\+εlog\|V\|\\Delta\(p,L\)\\leq H\_\{2\}\(\\varepsilon\)\+\\varepsilon\\log\|V\|, proving \([S1](https://arxiv.org/html/2607.24833#A1.E1)\) unconditionally\.
*Lower bound\.*Dropping the nonnegative termεH\(pS¯\)\\varepsilon H\(p\_\{\\bar\{S\}\}\)in \([S3](https://arxiv.org/html/2607.24833#A1.E3)\) and usingH^t\(p\)≤log\|S\|=logL\\widehat\{H\}\_\{t\}\(p\)\\leq\\log\|S\|=\\log L,
Δ\(p,L\)≥H2\(ε\)−εlogL≥εlog1ε−εlogL=εlog1εL,\\Delta\(p,L\)\\ \\geq\\ H\_\{2\}\(\\varepsilon\)\-\\varepsilon\\log L\\ \\geq\\ \\varepsilon\\log\\tfrac\{1\}\{\\varepsilon\}\-\\varepsilon\\log L=\\varepsilon\\log\\tfrac\{1\}\{\\varepsilon L\},where the second inequality usesH2\(ε\)=εlog1ε\+\(1−ε\)log11−ε≥εlog1εH\_\{2\}\(\\varepsilon\)=\\varepsilon\\log\\tfrac\{1\}\{\\varepsilon\}\+\(1\-\\varepsilon\)\\log\\tfrac\{1\}\{1\-\\varepsilon\}\\geq\\varepsilon\\log\\tfrac\{1\}\{\\varepsilon\}\. ThusΔ\(p,L\)≥0\\Delta\(p,L\)\\geq 0wheneverε≤1/L\\varepsilon\\leq 1/L\. The condition is not vacuous: ifεL≫1\\varepsilon L\\gg 1the bias can be negative—a near\-uniform head ofLLtokens together with a heavy point\-mass tail makesH^t\(p\)≈logL\\widehat\{H\}\_\{t\}\(p\)\\approx\\log LexceedH\(p\)H\(p\)\. With the serving truncationL=20L\{=\}20and the empiricalε≲0\.05=1/L\\varepsilon\\lesssim 0\.05=1/Lreported in the main paper’s proxy analysis, the regimeε≤1/L\\varepsilon\\leq 1/Lholds, so both bounds apply\.
*Difference of biases\.*Forq′=q⊕kq^\{\\prime\}=q\\oplus kwith tail masses≤ε⋆\\leq\\varepsilon^\{\\star\}, the bias of the proxy score equals\(H^t\(q\)−H^t\(q′\)\)−\(H\(q\)−H\(q′\)\)=Δ\(q′\)−Δ\(q\)\\bigl\(\\widehat\{H\}\_\{t\}\(q\)\-\\widehat\{H\}\_\{t\}\(q^\{\\prime\}\)\\bigr\)\-\\bigl\(H\(q\)\-H\(q^\{\\prime\}\)\\bigr\)=\\Delta\(q^\{\\prime\}\)\-\\Delta\(q\), of magnitude at mostmax\{Δ\(q\),Δ\(q′\)\}≤H2\(ε⋆\)\+ε⋆log\|V\|=𝒪\(ε⋆log\|V\|\)\\max\\\{\\Delta\(q\),\\Delta\(q^\{\\prime\}\)\\\}\\leq H\_\{2\}\(\\varepsilon^\{\\star\}\)\+\\varepsilon^\{\\star\}\\log\|V\|=\\mathcal\{O\}\(\\varepsilon^\{\\star\}\\log\|V\|\), sinceH2\(ε⋆\)=𝒪\(ε⋆log1ε⋆\)≤𝒪\(ε⋆log\|V\|\)H\_\{2\}\(\\varepsilon^\{\\star\}\)=\\mathcal\{O\}\(\\varepsilon^\{\\star\}\\log\\tfrac\{1\}\{\\varepsilon^\{\\star\}\}\)\\leq\\mathcal\{O\}\(\\varepsilon^\{\\star\}\\log\|V\|\)once\|V\|≥1/ε⋆\|V\|\\geq 1/\\varepsilon^\{\\star\}\. This bounds the*bias*of the score; it does not by itself imply that the proxy preserves the KP ordering, which we instead support empirically \(Spearmanρ=0\.68\\rho\{=\}0\.68against leave\-one\-out ground truth; see the proxy\-validation gate of the main paper\)\.
*Finite\-KKaveraging\.*The proxy reportsH^=1K∑t≤KH^t\\widehat\{H\}=\\frac\{1\}\{K\}\\sum\_\{t\\leq K\}\\widehat\{H\}\_\{t\}, with eachH^t∈\[0,logL\]\\widehat\{H\}\_\{t\}\\in\[0,\\log L\]\. Lettingℱt−1\\mathcal\{F\}\_\{t\-1\}be theσ\\sigma\-algebra of the sampled prefix before positiontt, the incrementsDt=H^t−𝔼\[H^t∣ℱt−1\]D\_\{t\}=\\widehat\{H\}\_\{t\}\-\\mathbb\{E\}\[\\widehat\{H\}\_\{t\}\\mid\\mathcal\{F\}\_\{t\-1\}\]form a martingale\-difference sequence bounded bylogL\\log L, so Azuma–Hoeffding gives\|1K∑t≤KDt\|=𝒪P\(K−1/2\)\\bigl\|\\tfrac\{1\}\{K\}\\sum\_\{t\\leq K\}D\_\{t\}\\bigr\|=\\mathcal\{O\}\_\{P\}\(K^\{\-1/2\}\)\. Because the autoregressive positions are dependent, this replaces an independence\-based Hoeffding bound\. The bound controls deviation from the*running conditional mean*1K∑t≤K𝔼\[H^t∣ℱt−1\]\\tfrac\{1\}\{K\}\\sum\_\{t\\leq K\}\\mathbb\{E\}\[\\widehat\{H\}\_\{t\}\\mid\\mathcal\{F\}\_\{t\-1\}\]; identifying that mean with the marginal target—the expected average first\-token entropy—further assumes the per\-position conditional entropies are weakly dependent so their running mean concentrates, an assumption we adopt rather than prove\. ∎
*Remark \(regime of validity\)\.*Proposition[S1](https://arxiv.org/html/2607.24833#Thmproposition1)sharpens the body’s inline bound: the upper bound \([S1](https://arxiv.org/html/2607.24833#A1.E1)\) is unconditional, but the lower bound—hence the two\-sided statement used in the main paper—requiresε≤1/L\\varepsilon\\leq 1/L\. The boundεlog\|V\|\\varepsilon\\log\|V\|is also loose for ranking: withε≈0\.05\\varepsilon\\approx 0\.05and\|V\|∼1\.5×105\|V\|\\sim 1\.5\{\\times\}10^\{5\}it is≈0\.6\\approx 0\.6nats, larger than typical inter\-KP entropy gaps, so the proposition controls the*bias*of the score but not rank preservation\.
## Appendix BAdaKPSelector Training Loop
Algorithm S1AdaKPSelector training\-time loop\. State is per\-problem forSq,RqS\_\{q\},R\_\{q\}and global for the scheduler𝒮\\mathcal\{S\}\.0:KP pools
\{P\(q\)\}q∈Q\\\{P\(q\)\\\}\_\{q\\in Q\}, proxy
πproxy\\pi^\{\\text\{proxy\}\}, hyperparameters
α,ρsel,nretire,mrevive,rrevive,Δ0,c,T\\alpha,\\rho\_\{\\text\{sel\}\},n\_\{\\text\{retire\}\},m\_\{\\text\{revive\}\},r\_\{\\text\{revive\}\},\\Delta\_\{0\},c,T
1:Initialize
𝒮\\mathcal\{S\}and, for each
qq, smoother
SqS\_\{q\}and retirement manager
RqR\_\{q\}
2:Re\-eval counter
τ←0\\tau\\leftarrow 0; selection cache
selq←Scss\(q\)\\mathrm\{sel\}\_\{q\}\\leftarrow S^\{\\mathrm\{css\}\}\(q\)for every
qq\{static\-CSS fallback untilτ\>0\\tau\{\>\}0\}
3:for
t=1t=1to
TTdo
4:if
𝒮\.ShouldReEvaluate\(t\)\\mathcal\{S\}\.\\textsc\{ShouldReEvaluate\}\(t\)then
5:
τ←τ\+1\\tau\\leftarrow\\tau\+1
6:foreach problem
q∈Qq\\in Qdo
7:
Aq←Rq\.Active\(P\(q\)\)A\_\{q\}\\leftarrow R\_\{q\}\.\\textsc\{Active\}\(P\(q\)\)\{exclude retired KPs\}
8:if
Aq=∅A\_\{q\}=\\emptysetthen
9:continue\{safety: keep lastselq\\mathrm\{sel\}\_\{q\}\}
10:endif
11:foreach
k∈Aqk\\in A\_\{q\}do
12:
σq,k←πproxy\.EntropyDiff\(q,k\)\\sigma\_\{q,k\}\\leftarrow\\pi^\{\\text\{proxy\}\}\.\\textsc\{EntropyDiff\}\(q,k\)\{entropy\-diff proxy\}
13:
Sq\.Update\(k,σq,k\)S\_\{q\}\.\\textsc\{Update\}\(k,\\sigma\_\{q,k\}\)
14:endfor
15:
selq←Sq\.TopFraction\(ρsel\)\\mathrm\{sel\}\_\{q\}\\leftarrow S\_\{q\}\.\\textsc\{TopFraction\}\(\\rho\_\{\\text\{sel\}\}\)\{overwrite cache\}
16:
Rq\.StepCounters\(selq,Aq\)R\_\{q\}\.\\textsc\{StepCounters\}\(\\mathrm\{sel\}\_\{q\},A\_\{q\}\)
17:if
τmodmrevive=0\\tau\\bmod m\_\{\\text\{revive\}\}=0then
18:
Rq\.ReviveRandomFraction\(rrevive\)R\_\{q\}\.\\textsc\{ReviveRandomFraction\}\(r\_\{\\text\{revive\}\}\)\{exploration\}
19:endif
20:endfor
21:endif
22:Train one DAPO\+GRPO step with each prompt’sHintblock populated from
selq\\mathrm\{sel\}\_\{q\}\{cache reused between re\-evaluations\}
23:endfor
#### Reading the loop\.
Lines map one\-to\-one onto the four components:ShouldReEvaluateis the scheduler,EntropyDiffthe entropy proxy,Updatethe EMA smoother, andStepCounters/ReviveRandomFractionthe retirement manager \(all four defined in the main paper’s Method section\)\. State is per problem for the smoother and retirement counters and a single global object for the scheduler, so the resident memory is𝒪\(∑q\|C\(q\)\|\)\\mathcal\{O\}\\\!\\bigl\(\\sum\_\{q\}\|C\(q\)\|\\bigr\)scalars—one EMA value and one counter per candidate KP—independent of the model size\. The selection cacheselq\\mathrm\{sel\}\_\{q\}is written only inside the re\-evaluation branch and read on every training step, so steps between re\-evaluations incur*no*selector overhead; the two safety fallbacks \(the static\-CSS initialization before the first re\-evaluation and the all\-retiredcontinue\) guarantee theHintblock is always populated\.
## Appendix CPositioning Relative to Related Work
Table S1:Positioning of AdaKP relative to closely related lines of work\.Item granularity: what is being selected\.Selection level: monolithic vs\. subset\-of\-atomic\.Timing: offline \(once\) vs\. online \(during RL\)\.The three axes are independent: a method may inject*atomic*KPs yet fix them offline \(top row\), or adapt online yet treat the hint as one monolithic block \(middle rows\)\. AdaKP is the only entry that is simultaneously KP\-level, subset\-valued, and online; HIVE’s “ – ” marks that selection level does not apply, as it gates whole prompts rather than content within a prompt\. We stress that this table is a*conceptual*positioning along design axes, not an empirical head\-to\-head: the adaptive\-hint methods target a different axis \(how much to hint\), so we do not run them under our setting, and the comparison should be read as locating AdaKP in design space rather than ranking systems\.
## Appendix DNon\-Stationarity of KP Importance \(Full Table\)
Table S2:H1: per\-benchmark Jaccard similarity of top\-30%30\\%KP sets between two policies \(πOpenMath\\pi\_\{\\text\{OpenMath\}\}vs\.πtrained\\pi\_\{\\text\{trained\}\}\)\.nn= number of problems with\|C\(q\)\|≥2\|C\(q\)\|\{\\geq\}2out of the3030stratified samples per benchmark\. Bold marks the single benchmark above the0\.70\.7H1\_strongthreshold\.#### How to read, and caveats\.
A top\-30%30\\%set holds only⌈0\.3\|C\(q\)\|⌉≈2\\lceil 0\.3\\,\|C\(q\)\|\\rceil\\\!\\approx\\\!2KPs when\|C\(q\)\|∈\[5,7\]\|C\(q\)\|\\in\[5,7\], so a per\-problem Jaccard takes only a few discrete values and the per\-benchmark means are correspondingly coarse; the headline quantity is the global mean, not any single row\. The realizednncounts the sampled problems with\|C\(q\)\|≥2\|C\(q\)\|\\geq 2, below which the leave\-one\-out ranking is undefined, so benchmarks with few such problems \(MATH\-500,n=12n\{=\}12; Olympiad\-Bench,n=13n\{=\}13\) yield the noisiest estimates\. The lone above\-threshold value \(AIME25,0\.7300\.730atn=21n\{=\}21\) is, on a coarse\-Jaccard measurement over a small pool, most simply read as run\-to\-run noise rather than genuine stationarity; we do not attribute it to any specific mechanism\. The table supports H1—that the informative\-KP set shifts between the base and the trained policy—which motivates adaptive selection \(see the main paper’s non\-stationarity test\); it characterizes the*phenomenon*, not AdaKP, whose end\-task effect is the main comparison\.
## Appendix EReproducibility Notes
AdaKP is an additive module: enabling it changes only theHintblock each problem receives and leaves the optimizer, loss, and rollout path bit\-identical to the DAPO\+GRPO baseline, so enabling or disabling AdaKP is an exact ablation control\. Reproducing a run also requires settings that are fixed in our training configuration but not fully recoverable from the paper text: the optimizer hyperparameters beyond the learning rate \(Adam moments, weight decay, any warmup\), the filtering and de\-duplication applied to the8\.88\.8k\-problem corpus, the random\-seed handling for data order and rollout sampling, and the cadence at which the frozen proxy is loaded\. These are contained in our fork and scripts, which will be released upon acceptance\.Similar Articles
Kalman Meets Curriculum: Efficient Dynamic Prompt Selection for Adaptive RL Finetuning
This paper introduces KGPS, a Kalman-guided prompt selection method for adaptive RL finetuning of LLMs, which models prompt difficulty as a dynamic state to improve accuracy and rollout efficiency.
Dynamic Sampling that Adapts: Self-Aware Iterative Data Persistent Optimization for Mathematical Reasoning
SAI-DPO introduces a dynamic sampling framework that adapts training data to a model's evolving capabilities during mathematical reasoning tasks, using self-aware difficulty metrics and knowledge semantic alignment to achieve state-of-the-art efficiency with less data on benchmarks like AIME24 and AMC23.
KACE: Knowledge-Adaptive Context Engineering for Mathematical Reasoning
KACE introduces a knowledge-adaptive context engineering method that separates storage from usage via an epistemic tree and tiered self-consistency, achieving 62.2% on AIME 2025—a 10.4-point gain over fixed self-consistency.
Selective-Advantage Entropy-Adaptive Horizon GRPO: Asymmetric Token-Level Discounting for Efficient Reinforcement Learning of Language Models
This paper introduces Adaptive-Horizon and Selective-Advantage variants of GRPO that use entropy-based token-level discounting to stabilize training and improve performance on math reasoning tasks, achieving stronger results with lower variance.
AdaTKG: Adaptive Memory for Temporal Knowledge Graph Reasoning
This paper proposes AdaTKG, a method for temporal knowledge graph reasoning that uses adaptive memory to refine entity representations dynamically as new interactions occur, improving performance over static baselines.