From Memorization to Absorption: Mixed-Policy RL for Continual Knowledge Injection

# Mixed-Policy RL for Continual Knowledge Injection
Source: [https://arxiv.org/html/2608.25243](https://arxiv.org/html/2608.25243)
## From Memorization to Absorption: Mixed\-Policy RL for Continual Knowledge Injection

Fan ZhaoAffiliation:University of California, MercedEmail:[fanzhao@ucmerced\.edu](mailto:)Zhiyu AnAffiliation:University of California, MercedEmail:[zan7@ucmerced\.edu](mailto:)Wan DuAffiliation:University of California, MercedEmail:[wdu3@ucmerced\.edu](mailto:)

###### Abstract

Continual knowledge injection is essential for keeping large language models up\-to\-date in a fast\-evolving world\. Existing methods rely on supervised fine\-tuning \(SFT\), which memorizes injected facts in their training format but fails to generalize across paraphrasing, document combinations, and reasoning\. To address this, we propose Golden\-GRPO Injection \(GRIN\), a three\-stage self\-learning framework for continual knowledge injection\. Golden\-GRPO is a mixed\-policy reinforcement learning algorithm designed specifically for knowledge injection, which injects a golden answer to provide learning signal even when on\-policy rollouts fail on novel facts\. We further introduceBlankandCounter, two document\-level benchmarks targeting novel acquisition and counterfactual overwrite respectively, each evaluating single\-fact recall, multi\-source retrieval, and inferential reasoning\. Our experiments establish a clear empirical claim: mixed\-policy reinforcement learning enables knowledge absorption beyond what supervised fine\-tuning can achieve\. GRIN substantially outperforms SFT and mixed\-policy RL baselines on the harder question types while matching them on basic fact recall\.

## 1Introduction

Large Language Models\(LLMs\)[Yang et al\. \(2025\)](https://arxiv.org/html/2608.25243#bib.bib2);[Liu et al\. \(2024\)](https://arxiv.org/html/2608.25243#bib.bib3);[Touvron et al\. \(2023\)](https://arxiv.org/html/2608.25243#bib.bib4)excel in knowledge intensive applications such as search assistants and knowledge based question answering tasks[Yue \(2025\)](https://arxiv.org/html/2608.25243#bib.bib5);[Xu et al\. \(2024\)](https://arxiv.org/html/2608.25243#bib.bib6)owing to the wealth of factual knowledge acquired during pre\-training phase[Dong et al\. \(2019\)](https://arxiv.org/html/2608.25243#bib.bib1)\. However, as a static fixed\-parameter model in a fast changing world, the LLMs may easily become outdated[Zhang et al\. \(2025a\)](https://arxiv.org/html/2608.25243#bib.bib7);[Lin et al\. \(2025\)](https://arxiv.org/html/2608.25243#bib.bib9)\. The conventional response to this has been to discard the old model and train a new one from scratch, incorporating updated data alongside architectural improvements[Yang et al\. \(2025\)](https://arxiv.org/html/2608.25243#bib.bib2)\. This cycle, while effective, is costly, time\-consuming, and treats knowledge updating as a byproduct of periodic retraining rather than as a first class objective\. Life\-long learning, or continual knowledge injection[Wu et al\. \(2024b\)](https://arxiv.org/html/2608.25243#bib.bib16), aims to solve this problem by post\-training LLMs on the latest real world corpora\.

Existing continual knowledge injection approaches[Ovadia et al\. \(2024\)](https://arxiv.org/html/2608.25243#bib.bib10);[He et al\. \(2025\)](https://arxiv.org/html/2608.25243#bib.bib11)can be broadly grouped by their training data format\. The first group trains directly on raw domain corpora, allowing large\-scale knowledge coverage but often failing at recall during inference, as models struggle to surface the injected knowledge when faced with question\-style queries[Jang et al\. \(2021\)](https://arxiv.org/html/2608.25243#bib.bib12);[Cheng et al\. \(2023\)](https://arxiv.org/html/2608.25243#bib.bib13)\. The second group constructs diverse QA pairs from domain knowledge to bridge the gap between training and evaluation formats, improving recall[Zhang et al\. \(2025a\)](https://arxiv.org/html/2608.25243#bib.bib7);[Jiang et al\. \(2025\)](https://arxiv.org/html/2608.25243#bib.bib14)\. Both groups rely on Supervised Fine\-Tuning \(SFT\) loss for knowledge memorization, and they share two common limitations\. First, the injected knowledge often fails to generalize to unseen queries beyond the training distribution, limiting practical applicability[Krishnamurthy et al\. \(2024\)](https://arxiv.org/html/2608.25243#bib.bib15)\. Second, they fail to explicitly account for the model’s existing parametric beliefs, leaving outdated or incorrect prior knowledge unaddressed and allowing it to interfere with newly injected facts during inference\.

Recent work[Chu et al\. \(2025\)](https://arxiv.org/html/2608.25243#bib.bib8)has shown that SFT tends to encourage memorization rather than generalizable internalization, in contrast to reinforcement learning\. While this finding was established for reasoning tasks, we hypothesize that the same pattern manifests in knowledge injection and that reinforcement learning offers a path beyond surface\-level memorization\. However, existing mixed\-policy RL methods[Yan et al\. \(2026\)](https://arxiv.org/html/2608.25243#bib.bib31);[Zhang et al\. \(2025b\)](https://arxiv.org/html/2608.25243#bib.bib44)are designed for reasoning, where the model already possesses the underlying capability and RL only needs to provide directional guidance\. Knowledge injection demands a stronger pull yet their off\-policy gradients vanish on unlearned facts \(Appendix[E](https://arxiv.org/html/2608.25243#A5)\)\.

We therefore propose GRIN, a three\-stage framework for continual knowledge injection that follows the SFT\-then\-RL pipeline with a stage of diverse problem\-set construction in between\. The base model extracts QA pairs for Stage 1 SFT, samples diverse questions and golden answers for Stage 2, and is then trained on this pool via reinforcement learning in Stage 3\. A key challenge of applying RL to novel knowledge is that base\-model rollouts often fail entirely on unlearned facts, leaving no positive reward signal\. We address this with Golden\-GRPO, a mixed\-policy RL algorithm that injects the golden answer as an off\-policy trajectory when on\-policy rollouts fail, guaranteeing meaningful learning signal at every training step\.

To evaluate continual knowledge injection rigorously, we introduce two complementary benchmarks:BlankandCounter\.Blanktargets novel knowledge acquisition, andCountertargets the prior belief overwrite\. Both benchmarks evaluate three question types: single\-fact recall, multi\-source retrieval, and inferential reasoning\. These question types are designed to distinguish surface memorization from genuine knowledge absorption\.Counteradditionally reports fail@k, measuring whether the model’s prior beliefs resurface in any of k samples, to capture overwrite reliability\.

In summary, our contributions are three\-fold:

We propose GRIN, a three\-stage framework for continual knowledge injection that requires no external teacher model, built on Golden\-GRPO, a mixed\-policy reinforcement learning algorithm specifically designed for the knowledge injection setting, producing knowledge that is absorbed into the model’s parametric reasoning\.

We introduceBlankandCounter, two document\-level benchmarks for evaluating continual knowledge injection along orthogonal axes, novel acquisition and counterfactual overwrite, each with a three\-tier evaluation protocol that separates recall from generalization\.

We conduct extensive experimentto empirically show that GRIN substantially outperforms both supervised and mixed\-policy RL baselines on multi\-source retrieval, inferential reasoning, and counterfactual overwrite, with comparable performance on basic fact recall\.

## 2Related Works

### 2\.1Continual Knowledge Injection

Approaches to continual knowledge injection generally fall into two categories: retrieval\-based methods, which provide knowledge through external context at inference time, and parameter\-updating methods, which incorporate knowledge directly into the model’s weights through training[Zhang et al\. \(2025a\)](https://arxiv.org/html/2608.25243#bib.bib7);[Jiang et al\. \(2024\)](https://arxiv.org/html/2608.25243#bib.bib28)\.Retrieval\-Augmented Generation \(RAG\)\.RAG injects knowledge non\-parametrically by retrieving relevant passages from an external corpus at inference time and concatenating them with the query[Izacard et al\. \(2023\)](https://arxiv.org/html/2608.25243#bib.bib20);[Vu et al\. \(2024\)](https://arxiv.org/html/2608.25243#bib.bib18);[Wu et al\. \(2024a\)](https://arxiv.org/html/2608.25243#bib.bib19);[Lewis et al\. \(2020\)](https://arxiv.org/html/2608.25243#bib.bib17)\. While effective as a search mechanism, RAG does not constitute true knowledge injection: facts remain external to the model’s parameters, so the LLM cannot reason over them natively or consolidate them with prior knowledge[Su et al\. \(2025\)](https://arxiv.org/html/2608.25243#bib.bib24);[Levy et al\. \(2024\)](https://arxiv.org/html/2608.25243#bib.bib25)\. Retrieval also depends on the model recognizing that external knowledge is needed, which often fails when partial knowledge is sufficient for confident but incorrect answers[Asai et al\. \(2024\)](https://arxiv.org/html/2608.25243#bib.bib21);[Mallen et al\. \(2023\)](https://arxiv.org/html/2608.25243#bib.bib22);[Yin et al\. \(2023\)](https://arxiv.org/html/2608.25243#bib.bib23)\. These limitations are intrinsic to retrieval\-based methods\. We therefore focus on parameter\-updating methods, which integrate facts directly into the model’s parametric knowledge\.

Knowledge Injection via Training\.Training\-based approaches incorporate knowledge into the model’s parameters through gradient updates[Xu et al\. \(2023\)](https://arxiv.org/html/2608.25243#bib.bib26)\. The simplest variant is continued pre\-training on raw documents[Xu et al\. \(2023\)](https://arxiv.org/html/2608.25243#bib.bib26);[Mecklenburg et al\. \(2024\)](https://arxiv.org/html/2608.25243#bib.bib27), but this often yields shallow memorization without effective recall or reasoning[Jiang et al\. \(2024\)](https://arxiv.org/html/2608.25243#bib.bib28)\. To close this gap, recent methods operate on instruction\-tuned models and train on QA or instruction formatted derivatives of the raw documents[Zhang et al\. \(2025a\)](https://arxiv.org/html/2608.25243#bib.bib7);[Ovadia et al\. \(2024\)](https://arxiv.org/html/2608.25243#bib.bib10)\. However, these methods primarily focus on diversifying the synthesized QA pairs while still rely on supervised fine\-tuning \(SFT\) as the training signal\. Recent study[Chu et al\. \(2025\)](https://arxiv.org/html/2608.25243#bib.bib8)has shown that SFT tends to memorize training data rather than acquire generalizable rules, motivating our use of reinforcement learning to enable genuine knowledge absorption\.

### 2\.2Guided Reinforcement Learning

Recent work augments on\-policy RL with external guidance to overcome the inability to acquire capabilities outside the base policy’s sampling distribution\. ReLIFT[Ma et al\. \(2025\)](https://arxiv.org/html/2608.25243#bib.bib29)and FLAME[Lin et al\. \(2024\)](https://arxiv.org/html/2608.25243#bib.bib30)interleave SFT within the RL training loop to provide learning signal when rollouts fail\. Another line of work[Yan et al\. \(2026\)](https://arxiv.org/html/2608.25243#bib.bib31);[Huang et al\. \(2026\)](https://arxiv.org/html/2608.25243#bib.bib43);[Zhang et al\. \(2025b\)](https://arxiv.org/html/2608.25243#bib.bib44)mixes on\-policy rollouts with off\-policy reasoning traces, balancing imitation and exploration to transfer reasoning skills\. These methods all target reasoning, where the model has the underlying capability but needs guidance to elicit it\. Knowledge injection presents a different challenge: on\-policy rollouts often fail to sample the injected facts from the base model, leaving the RL stage with no positive reward signal\. Our work addresses this with Golden\-GRPO, extending the SFT\-then\-RL framework with mixed\-policy RL in the final stage\.

## 3Preliminary

Group Relative Policy Optimization \(GRPO\)[Shao et al\. \(2024\)](https://arxiv.org/html/2608.25243#bib.bib36)is a widely used on\-policy RL algorithm that estimates per\-trajectory advantage via group\-relative normalization\. For each questionqq, the current policyπθo​l​d\\pi\_\{\\theta\_\{old\}\}samples a group ofNNtrajectories\{τ1,τ2,…,τN\}\\\{\\tau\_\{1\},\\tau\_\{2\},\.\.\.,\\tau\_\{N\}\\\}, each scored by a reward functionR⁡\(τi\)R\(\\tau\_\{i\}\)\. The advantage of theii\-th trajectory is computed by standardizing rewards within the group:

Ai=R⁡\(τi\)−mean​\(\{R⁡\(τj\)\}j=1N\)std​\(\{R⁡\(τj\)\}j=1N\)A\_\{i\}=\\frac\{R\(\\tau\_\{i\}\)\-\\text\{mean\}\(\\\{R\(\\tau\_\{j\}\)\\\}^\{N\}\_\{j=1\}\)\}\{\\text\{std\}\(\\\{R\(\\tau\_\{j\}\)\\\}^\{N\}\_\{j=1\}\)\}\(1\)The policy is then updated using a per\-token PPO\-clipped objective:

𝒥GRPO​\(θ\)\\displaystyle\\mathcal\{J\}\_\{\\text\{GRPO\}\}\(\\theta\)=𝔼\[min\(ri,t\(θ\)⋅Ai,\\displaystyle=\\mathbb\{E\}\\bigg\[\\min\\Big\(r\_\{i,t\}\(\\theta\)\\cdot A\_\{i\},\(2\)clip\(ri,t\(θ\),1−ε,1\+ε\)⋅Ai\)\]\\displaystyle\\text\{clip\}\\big\(r\_\{i,t\}\(\\theta\),\\,1\-\\varepsilon,\\,1\+\\varepsilon\\big\)\\cdot A\_\{i\}\\Big\)\\bigg\]whereri,t​\(θ\)=πθ​\(yi,t\)/πθo​l​d​\(yi,t\)r\_\{i,t\}\(\\theta\)=\\pi\_\{\\theta\}\(y\_\{i,t\}\)/\\pi\_\{\\theta\_\{old\}\}\(y\_\{i,t\}\)is the importance sampling ratio between the current and rollout\-time policies, andε\\varepsilonis the clipping bound\. The clipping prevents large policy updates, stabilizing training\.

Table 1:Representative examples from BLANK and COUNTER across all three question types\. For COUNTER, counterfactual replacements are shown with the original fact in parentheses\.
## 4Benchmarks: BLANK and COUNTER

We introduce two document\-level benchmarks for continual knowledge injection that together provide a full picture of an injection method’s behavior along two orthogonal axes\. The first axis is the injection mode\.Blankevaluates the acquisition of knowledge the model does not currently hold, whileCounterevaluates the overwrite of knowledge the model already believes\. These are two distinct capabilities that prior work tends to omit or conflate[Zhang et al\. \(2025a\)](https://arxiv.org/html/2608.25243#bib.bib7);[Ji et al\. \(2025\)](https://arxiv.org/html/2608.25243#bib.bib32);[Jiang et al\. \(2024\)](https://arxiv.org/html/2608.25243#bib.bib28)\. The second axis is the evaluation type\. Each benchmark measures three increasingly demanding capabilities: single\-fact recall, multi\-source retrieval, and inferential reasoning that distinguished surface memorization from joint retrieval and reasoning\.Counteradditionally reports fail@k, measuring whether the model’s prior beliefs resurface during overwrite\. Crossing these axes yields a fine\-grained diagnostic view that localizes where each method succeeds or fails, exposing failure modes that aggregate accuracy obscures\.

All filtering and prior\-belief sampling described below uses Qwen3\-4B[Yang et al\. \(2025\)](https://arxiv.org/html/2608.25243#bib.bib2)as the base model; the construction pipeline is model\-agnostic and can be re\-applied to any target model\.

### 4\.1Document Collection

#### 4\.1\.1BLANK

Blanktargets the setting in which the base model encounters knowledge it has never seen during pretraining\. We construct it from two sources of real\-world wiki content: passages from the TimeQA dataset[Chen et al\. \(2021\)](https://arxiv.org/html/2608.25243#bib.bib33)on which the base model has zero prior knowledge \(verified by probing the model on TimeQA’s associated questions without context\), and wiki pages created after the base model’s training cutoff\. TimeQA contributes passages whose accompanying questions are documented to be challenging for current models[Zhou et al\. \(2026\)](https://arxiv.org/html/2608.25243#bib.bib41), requiring more than surface\-level retrieval, making it a natural source of difficult evaluation\. The post\-cutoff source broadens topical coverage beyond TimeQA’s temporally\-grounded historical content\.

Blankcontains 776 raw corpora after filtering\.

#### 4\.1\.2COUNTER

Countertargets the setting in which injected knowledge must overwrite the model’s existing beliefs\. EachCounterdocument is a parallel\-universe rewrite of a wiki page in which named entities, dates, and other concrete facts have been replaced with internally consistent alternatives\.

Prior\-belief sampling\.To guarantee that eachCounterdocument contradicts the base model’s current beliefs, we sample those beliefs directly by prompting the base model to draft a wiki page on a given subject and treating the resulting text as a representation of the model’s current beliefs about that subject\. The drafted wiki then serves as the original against which the counterfactual is constructed\. This design ensures that the counterfactual contradicts what the model actually believes, including any hallucinations or outdated information, rather than what the real wiki page asserts\.

Parallel\-universe rewriting\.Given the original wiki, we prompt gemini\-3\.1\-pro\-preview to produce a parallel\-universe version in which concrete facts, including proper nouns, dates, numerical values and named entities, are replaced with plausible, internally consistent counterfacts, while preserving document structure and entity\-substitution consistency throughout\. The full sampling and rewriting prompt is provided in Appendix[A](https://arxiv.org/html/2608.25243#A1)\.

Countercontains 252 raw corpora after filtering\.

### 4\.2QA Pair Generation

Both benchmarks share a three\-tier evaluation protocol designed to probe injection at increasing levels of difficulty\. Single\-fact recall measures direct retrieval; multi\-source retrieval and inferential reasoning measure progressively stronger forms of generalization\. Evaluation scores are reported separately to localize the failure mode of each method\. We additionally report a per\-benchmark average, representing overall acquisition capability and overwrite capability\. All questions are LLM\-generated unless otherwise noted\. Full prompts are provided in Appendix[B](https://arxiv.org/html/2608.25243#A2)\.

Single\-fact recall\.Direct factoid questions whose answers appear in a single sentence of the source document\.

Multi\-source retrieval\.Questions composed of N sub\-questions \(N from 2 to 4\) targeting distinct facts that span across paragraphs or documents, testing joint recall in a single response\.

Inferential reasoning\.Questions requiring inference that combines multiple facts from the source document, including temporal reasoning, comparison, and conditional inference\. For TimeQA\-derived pages, this category includes the original TimeQA questions alongside additional LLM\-generated questions\.

fail@k\(Counteronly\) For each single fact recall question, we sample k responses and count an item as a fail if the prior fact appears in any sample, measuring the prior\-belief leakage\.

Table[1](https://arxiv.org/html/2608.25243#S3.T1)shows example document excerpts and questions of each type for both benchmarks\.

## 5GRIN: Golden\-GRPO Injection

GRIN is a three\-stage framework for continual knowledge injection that operates entirely from the base model, requiring no external teacher \(Figure[1](https://arxiv.org/html/2608.25243#S5.F1)\)\. Stages 1 and 2 use the base model to extract atomic QA pairs and to sample a diverse \(question, golden\-answer\) pool from each corpus\. Stage 3 trains the model on this pool using Golden\-GRPO, our mixed\-policy reinforcement learning algorithm that injects the off\-policy golden answer to provide learning signal even when on\-policy rollouts fail on novel facts, producing knowledge that is absorbed into the model’s parametric reasoning rather than memorized in surface form\.

![Refer to caption](https://arxiv.org/html/2608.25243v1/bi8Au_arrow_border.png)

Figure 1:Overview of the GRIN framework\. Stage 1 builds QA\-style SFT data from raw corpora; Stage 2 samples diverse \(question, golden answer\) pairs grounded in each document; Stage 3 trains the model via Golden\-GRPO, a mixed\-policy RL objective that injects the golden answer as an off\-policy trajectory alongside on\-policy rollouts\.### 5\.1Stage 1: Self\-Extracted Fact Injection

The first stage extracts atomic facts from source corpora and trains the model to recall them via supervised fine\-tuning \(SFT\)\. We use only the base model itself for fact extraction, consistent with our self\-supervised framing\.

LetD=\{d1,d2,d3,…,dN\}D=\\\{d\_\{1\},d\_\{2\},d\_\{3\},\.\.\.,d\_\{N\}\\\}denote the target corpora, and letπθ\\pi\_\{\\theta\}denote the base model with parameterθ\\theta\. For each corpusdid\_\{i\}, we segment it into sentences through the NLTK[Bird \(2006\)](https://arxiv.org/html/2608.25243#bib.bib34)sentence tokenizerSS:

S⁡\(di\)=\{si,1,si,2,si,3,…,si,Mi\}S\(d\_\{i\}\)=\\\{s\_\{i,1\},s\_\{i,2\},s\_\{i,3\},\.\.\.,s\_\{i,M\_\{i\}\}\\\}\(3\)For each sentencesi,js\_\{i,j\}, we use base modelπθ\\pi\_\{\\theta\}with fixed promptPe​x​tP\_\{ext\}to produce atomic factoid question\-answer pairs:

Fi,j=πθ​\(Pe​x​t⊕si,j\)=\{qi,j,k,ai,j,k\}k=1Ki,jF\_\{i,j\}=\\pi\_\{\\theta\}\(P\_\{ext\}\\oplus s\_\{i,j\}\)=\\\{q\_\{i,j,k\},a\_\{i,j,k\}\\\}^\{K\_\{i,j\}\}\_\{k=1\}\(4\)where each\(q,a\)\(q,a\)is a question\-answer pair encoding a single fact from the sentencesi,js\_\{i,j\}\. The full extraction prompt is given in Appendix[C](https://arxiv.org/html/2608.25243#A3)\.

Aggregating across all sentences in all corpora creates a training setF1=⋃i,jFi,jF\_\{1\}=\\bigcup\_\{i,j\}F\_\{i,j\}\. We then fine\-tuneπθ\\pi\_\{\\theta\}through standard cross\-entropy loss over the answer tokens:

ℒ1​\(θ\)=−𝔼\(q,a\)∼F1​log​πθ​\(a\|q\)\\mathcal\{L\}\_\{1\}\(\\theta\)=\-\\mathbb\{E\}\_\{\(q,a\)\\sim F\_\{1\}\}\\text\{log\}\\pi\_\{\\theta\}\(a\|q\)\(5\)
Stage 1 provides the model with parametric access to the injected facts through SFT\. However, as shown in Table[2](https://arxiv.org/html/2608.25243#S5.T2), SFT alone produces brittle memorization\. Facts are recalled in their training format but fail on alternative phrasings or compositional queries, which motivates the following stages\.

### 5\.2Stage 2: Corpus\-Conditioned Sampling

The second stage constructs the \(question, golden\-answer\) pool used as training data for Golden\-GRPO in Stage 3\. Following prior work that uses pre\-query tokens to elicit diverse instruction data from language models[Xu et al\. \(2025\)](https://arxiv.org/html/2608.25243#bib.bib35), we adapt this technique to a corpus\-conditioned setting: for each corpusdi∈Dd\_\{i\}\\in D, we prepend corpus and a system prompt before a pre\-query tokenTq​u​e​r​yT\_\{query\}to sample questions grounded in the document content, then produce the golden answer a via a pre\-answer tokenTa​n​sT\_\{ans\}:

q∼πθ​\(Tq​u​e​r​y⊕di\),a⋆∼πθ​\(Ta​n​s⊕di⊕q\)q\\sim\\pi\_\{\\theta\}\(T\_\{query\}\\oplus d\_\{i\}\),a^\{\\star\}\\sim\\pi\_\{\\theta\}\(T\_\{ans\}\\oplus d\_\{i\}\\oplus q\)\(6\)Each corpus is sampled multiple times with duplicates removed, resulting a poolF2=\{\(qi,j,ai,j∗\)\}F\_\{2\}=\\\{\(q\_\{i,j\},a^\{\*\}\_\{i,j\}\)\\\}that varies in phrasing, granularity, and fact coverage\. Example outputs are provided in Appendix[C](https://arxiv.org/html/2608.25243#A3)\. Two design choices distinguishF2F\_\{2\}fromF1F\_\{1\}\. First, asF2F\_\{2\}is used as RL training data rather than SFT targets, we prioritize coverage over individual question quality, that noisy questions that would mislead SFT training may still yield meaningful learning signal through Golden\-GRPO reward function\. Second, more importantly,F2F\_\{2\}is sampled at the corpus level, producing questions that span multiple facts and capture inter\-sentence relationships, exactly the patterns that single\-sentence extraction cannot generate\.

### 5\.3Stage 3: Golden\-GRPO

In knowledge injection, on\-policy RL on unlearned facts frequently produces zero reward rollouts, yielding zero advantage with no meaningful learning signal\. Mixed\-policy RL restores non\-zero reward to the group by adding an off\-policy answer, but importance\-weighted off\-policy gradients vanish whenπθ​\(a⋆\)\\pi\_\{\\theta\}\(a^\{\\star\}\)is small, leaving only the on\-policy disadvantage signal, which drives forgetting without producing recall\. We therefore design Golden\-GRPO, which replaces the importance\-weighted off\-policy branch with a direct supervised gradient scaled by the off\-policy advantage\. This gradient is strong when the model has not yet learned the fact and naturally diminishes as the model improves, smoothly transitioning training toward on\-policy exploration\. We provide a formal analysis of these dynamics in Appendix[E](https://arxiv.org/html/2608.25243#A5)and empirical evidence in Section[6](https://arxiv.org/html/2608.25243#S6)\.

Rollout Injection\.For each\(q,a⋆\)∈F2\(q,a^\{\\star\}\)\\in F\_\{2\}, we form a rollout group consisting ofNo​nN\_\{on\}on\-policy trajectories sampled from the current policy, together with the off\-policy golden answer:

𝒢⁡\(q\)=\{τ1,τ2,…,τNo​n\}∪\{τ⋆\}\\mathcal\{G\}\(q\)=\\\{\\tau\_\{1\},\\tau\_\{2\},\.\.\.,\\tau\_\{N\_\{on\}\}\\\}\\cup\\\{\\tau^\{\\star\}\\\}\(7\)whereτi∼πθo​l​d\(⋅\|q\)\\tau\_\{i\}\\sim\\pi\_\{\\theta\_\{old\}\}\(\\cdot\|q\)fori=1,2,…,No​ni=1,2,\.\.\.,N\_\{on\}, andτ⋆\\tau^\{\\star\}is the trajectory corresponding to the golden answera⋆a^\{\\star\}\. Each trajectory receives a scalar rewardR⁡\(τi\)R\(\\tau\_\{i\}\)\.

Training Objective\.Recall that mixed\-policy RL fails in knowledge injection because the gradient toward the golden answer collapses whenπθ​\(a⋆\)\\pi\_\{\\theta\}\(a^\{\\star\}\)is small\. Our design protects the gradient at two points\.

First, to preserve the absolute magnitude ofA⋆A^\{\\star\}in high\-variance early\-training groups, we adopt Dr\. GRPO–style group\-relative advantages[Liu et al\. \(2025\)](https://arxiv.org/html/2608.25243#bib.bib37), which omit the standard\-deviation normalization used in GRPO:

Ai=R⁡\(τi\)−mean​\(\{R⁡\(τj\)\}j∈𝒢⁡\(q\)\)A\_\{i\}=R\(\\tau\_\{i\}\)\-\\text\{mean\}\(\\\{R\(\\tau\_\{j\}\)\\\}\_\{j\\in\\mathcal\{G\}\(q\)\}\)\(8\)
Standard GRPO’s std normalization shrinks advantages precisely when within\-group reward variance is high, which is the exact situation in early knowledge injection training, where a correct off\-policy trajectory sits alongside still\-failing on\-policy rollouts\.

We further remove the importance ratio and clip from the off\-policy branch and replace them with a direct supervised gradient scaled by the off\-policy advantage:

𝒥\\displaystyle\\mathcal\{J\}Golden\-GRPO\(θ\)=𝔼\[Ai⋅log⁡πθ​\(yi,t∣yi,<t,q\)⏟off\-policy branch\\displaystyle\{\}\_\{\\text\{Golden\-GRPO\}\}\(\\theta\)=\\ \\mathbb\{E\}\\bigg\[\\underbrace\{A\_\{i\}\\cdot\\log\\pi\_\{\\theta\}\(y\_\{i,t\}\\mid y\_\{i,<t\},q\)\}\_\{\\text\{off\-policy branch\}\}\(9\)\+min⁡\(ri,t⋅Ai,clip​\(ri,t,εl,εh\)⋅Ai\)⏟on\-policy branch\]\\displaystyle\+\\underbrace\{\\min\\Big\(r\_\{i,t\}\\cdot A\_\{i\},\\ \\text\{clip\}\\big\(r\_\{i,t\},\\varepsilon\_\{l\},\\varepsilon\_\{h\}\)\\cdot A\_\{i\}\\Big\)\}\_\{\\text\{on\-policy branch\}\}\\bigg\]The off\-policy supervised branch contains no importance\-sampling ratio, guaranteeing the model is pulled directly towarda⋆a^\{\\star\}, scaled by the off\-policy advantageA⋆A^\{\\star\}\. This term dominates early updates when on\-policy rollouts fail to produce correct answers\. As the model improves and on\-policy rollouts begin producing correct answers,A⋆A^\{\\star\}shrinks relative to the on\-policy group, and the gradient transitions naturally toward on\-policy exploration\. A formal analysis of this gradient transition is provided in Appendix[E](https://arxiv.org/html/2608.25243#A5), with empirical results in Section[6](https://arxiv.org/html/2608.25243#S6)\.

Reward Design\.The reward functionR⁡\(τi\)R\(\\tau\_\{i\}\)combines four components, with distinct roles:

•Format\.Following RL training for mathematical reasoning, we prompt the model to recall relevant facts and then provide the final answer between<answer\>\.\.\.</answer\>tags\. A correctly\-formatted output receives a format reward of\+0\.5\+0\.5\.

•ROUGE\-L\.With valid format, we extract the content inside the<answer\>tags and compute ROUGE\-L score againsta⋆a^\{\\star\}, producing a smooth reward in\[0,0\.5\]\[0,0\.5\]based on lexical overlap\.

•Exact Match\.To capture exact factual correctness, we extract a target keywordk⋆k^\{\\star\}froma⋆a^\{\\star\}and award\+1\+1when the answer containsk⋆k^\{\\star\}as an exact substring\.

•Multi\-answer penalty\.We observed that the model occasionally emits multiple<answer\>blocks when confused\. We apply a−0\.25\-0\.25penalty for any output containing more than one<answer\>block to prevent this reward\-hacking pattern\.

Table 2:Main results onBlankandCounterwith Qwen3\-4B\. Training\-free methods \(Closed\-book, Open\-book, RAG\) are reference points and are excluded from ranking\.Boldmarks the best andunderlinemarks the second\-best per column within the training\-based group; fail@k \(↓\\downarrow\) applies toCounteronly\.Table 3:Cross\-model results on COUNTER with Llama3\.2\-3B\. We evaluate on COUNTER as counter facts are model irrelevant\. GRIN leads on three of four accuracy metrics and is competitive on fail@k, indicating that the contribution generalizes beyond Qwen3\-4B\.

## 6Experiment

### 6\.1Experimental Setup

Models and Infrastructure\.We use Qwen3\-4B[Yang et al\. \(2025\)](https://arxiv.org/html/2608.25243#bib.bib2)as the base model for all main results\. All experiments are run on 2×H200 GPUs\. We evaluate onBlankandCounter\(Section[4](https://arxiv.org/html/2608.25243#S4)\), reporting per\-type accuracy and the per\-benchmark average; forCounterwe additionally report fail@k with k=5\. Accuracy is judged by an LLM \(gemini\-3\.5\-flash\), with prompt details provided in Appendix[D](https://arxiv.org/html/2608.25243#A4)\.

Baselines\.We compare GRIN against both training\-free and training\-based methods to fully evaluate how mixed\-policy reinforcement learning contributes to continual knowledge injection\. For training\-free baselines,closed\-bookqueries the base model without context and serves as a sanity check, that near\-zero accuracy confirms our evaluation questions cannot be answered from the model’s current knowledge base alone\.Open\-bookprompts the base model with the source wiki page in the context and serves as an upper\-bound reference for knowledge injection\.RAG[Lewis et al\. \(2020\)](https://arxiv.org/html/2608.25243#bib.bib17)retrieves at inference time with top\-k set to 3\. For training\-based methods,PIT[Jiang et al\. \(2024\)](https://arxiv.org/html/2608.25243#bib.bib28)trains with QA pairs positioned before their corresponding document texts\.Self\-tuning[Zhang et al\. \(2025a\)](https://arxiv.org/html/2608.25243#bib.bib7)extends PIT with self\-generated diverse training data, andAutonomous Learning \(AL\)[Ji et al\. \(2025\)](https://arxiv.org/html/2608.25243#bib.bib32)extends supervised training with offline direct preference optimization \(DPO\)[Rafailov et al\. \(2023\)](https://arxiv.org/html/2608.25243#bib.bib38)\.

Implementation\.We use the AdamW[Loshchilov and Hutter \(2017\)](https://arxiv.org/html/2608.25243#bib.bib39)optimizer with learning rate 2e\-5 for both Stage 1 SFT and Stage 3 Golden\-GRPO training, with an effective batch size of 512\. Stage 1 is trained for 5 epochs and Stage 3 Golden\-GRPO for 3 epochs with 8 on\-policy rollouts\. We set the KL coefficient to 5 which is substantially higher than the 0\.01–0\.1 range typical in RLHF, as we empirically observe knowledge injection induces large parameter shifts in early training\.

### 6\.2Main Results

Benchmark Validation\.The training\-free references in Table[2](https://arxiv.org/html/2608.25243#S5.T2)show that the benchmarks are well\-designed\. Closed\-book accuracy is near zero on bothBlankandCounter, with averages of 4\.33% and 2\.33% respectively, confirming that the evaluation questions cannot be answered from the base model’s parametric knowledge alone\. Open\-book achieves high accuracy on single\-fact recall but only 36\.61% and 45\.43% on inferential reasoning, indicating that the inferential questions are genuinely difficult\. RAG performs strongly on single\-fact recall but collapses on multi\-source retrieval as our multi\-source questions span facts across pages that retrieval cannot reliably surface\. Additionally, a close\-sourced model \(gemini\-3\.1\-flash\-lite\) with documents in context reaches 94\.05% onBlankand 99\.51% onCounter, confirming our benchmark questions are answerable from the corpora\.

SFT memorize but does not generalize\.Among training based baselines, PIT, AL, and Self\-Tuning all have respectable single\-fact recall on bothBlankandCounter, but degrade sharply on multi\-source retrieval and inferential reasoning tasks\. This is the characteristic failure mode of supervised injection, that facts are bound to their training questions forms but cannot be retrieved under different phrasings or question types\. This can be clearly observed on Self\-Tuning, the strongest baseline, which achieves54\.54%54\.54\\%single fact recall inBlank, but drops to only7\.10%7\.10\\%on inferential reasoning, an almost 8 times drop across question types that rely on the same underlying knowledge\.

Golden\-GRPO enables absorption\.GRIN is competitive with the strongest SFT\-based baseline on single\-fact recall, confirming that the RL objective does not sacrifice basic recall\. On the more generalized question types, the gap is more significant\. GRIN reaches 21\.16% multi\-source and 31\.69% inferential onBlank, compared to Self\-Tuning’s 11\.26% and 7\.10%\. TheCounterresults show the same pattern, and GRIN’s fail@k of 24\.05% is lower than every other training\-based method, indicating that GRIN can not only learn new facts, but suppresses prior beliefs as well\.

Golden\-GRPO’s design is necessary\.We further shown in ablation that the performance gain is not the result of diverse sampled data alone\. Continue training on the same Stage 2 data on Stage 1 model with SFT yields similar performance to other SFT baselines on multi\-source retrieval and inferential reasoning, empirically proving that the training method, not the data, is the key obstacle to generalize the injected knowledge\. On\-policy RL alone is even less effective, that vanilla GRPO without off\-policy injection collapses performance to near zero on BLANK \(1\.83% average\), confirming that without the off\-policy golden trajectory, on\-policy rollouts produce no learning signal on facts the model has not yet acquired\. Moreover, existing mixed\-policy RL method is not suffice as well\. Replacing Golden\-GRPO with LUFFY[Yan et al\. \(2026\)](https://arxiv.org/html/2608.25243#bib.bib31)outperforms SFT on generalization but underperforms ours, indicating that mixed\-policy RL is a viable direction and the design choices in Golden\-GRPO closed the remaining gap\.

Cross\-model generalization\.To verify that GRIN’s gains are not specific to Qwen3\-4B, we replicate theCounterexperiment with Llama3\.2\-3B[Grattafiori et al\. \(2024\)](https://arxiv.org/html/2608.25243#bib.bib40)as the base model\.Counterreplacements contradict real\-world facts, so for any base model they serve as valid injection targets, that either overwriting existing beliefs or acquiring novel ones\. Table[3](https://arxiv.org/html/2608.25243#S5.T3)reports results: GRIN leads on three of four accuracy metrics and is competitive on fail@k, reproducing the memorization\-versus\-absorption pattern observed with Qwen3\-4B and indicating that the contribution generalizes across base models\.

Figure 2:AverageCounteraccuracy across training epochs\. SFT\-based baselines plateau or degrade well below GRIN \(dashed\) even at 50 epochs, showing the generalization gap is not closed by additional compute\.Compute\-matched training\.A natural question is whether GRIN’s advantage stems simply from its higher training compute relative to SFT based baselines\. To test this, we continual train all baselines for additional epochs, resulting in a total of 50 epochs, which match or even exceed the total training cost of Golden\-GRPO\. Figure[2](https://arxiv.org/html/2608.25243#S6.F2)reportsCounteraverage accuracy across training epochs\. Training\-based baselines plateau beyond 15 epochs, well below GRIN’s 43\.65% average accuracy\. This indicates that the gap is not closed by additional training\. SFT based methods reach a ceiling determined by their training objective, while GRIN’s RL objective is capable of extract gains from the same data, unlocking performance that SFT cannot reach regardless of compute\.

## 7Conclusion

In this paper, we presented GRIN, a three\-stage framework for continual knowledge injection built around Golden\-GRPO, a mixed policy reinforcement learning algorithm tailored to this setting\. Motivated by the failure of supervised methods to absorb knowledge beyond their training format, and by the vanishing off\-policy gradient that limits standard mixed\-policy RL on novel facts, Golden\-GRPO produces knowledge that is absorbed into the model’s parametric reasoning rather than memorized in surface form\. Across novel acquisition \(Blank\) and counterfactual overwrite \(Counter\) benchmarks, GRIN substantially outperforms supervised and mixed\-policy RL baselines on multi\-source retrieval and inferential reasoning\.

## Limitations

While our experimental results have demonstrated that mixed\-policy reinforcement learning enables knowledge absorption beyond what SFT achieves, several aspects of this work remain open for further investigation\.

Single\-round injection vs\. Lifelong learning\.Our benchmarks evaluate a single round of knowledge injection, where the model learns from a group of corpora\. However, the goal of lifelong learning is continual training on the target model as the real world evolves\. A successful single\-round injection does not guarantee successful multi\-round injection\. Each round of training shapes the model’s parameter space in ways that may affect both the retention of previously injected knowledge and the model’s capacity to absorb future knowledge\. How Golden\-GRPO behaves under repeated injection rounds, whether it accumulates knowledge stably, suffers from catastrophic forgetting, or degrades the model’s ability to learn remains an open question, and require further investigation\.

Knowledge domain coverage\.Our benchmarks \(BlankandCounter\) are constructed from wiki\-style, entity\-grounded text\. The behavior of GRIN on other knowledge formats, like procedural knowledge, code, mathematical content, or structured data, is unexplored\. The reward function \(ROUGE\-L and Exact Match\) is tailored to factual recall and may need adaptation for domains where correctness is harder to measure with lexical signals\.

Resource Requirements\.Golden\-GRPO requires sampling on\-policy rollouts at each training step, which scales with model size, dataset size and rollout count\. At the 3–4B scale we evaluate, this overhead is modest and is justified by our compute\-matched experiments\. At frontier scales, however, the cost of rollout sampling on a trillion\-parameter model with world\-scale knowledge updates may approach the cost of full retraining\. In such cases, full retraining remains preferable because it offers more than knowledge updates, as it also allows architectural improvements \(e\.g\. new attention mechanism[Team et al\. \(2026\)](https://arxiv.org/html/2608.25243#bib.bib42)\) that knowledge injection cannot provide\. Our method is therefore most applicable to scenarios where retraining the base model is not an option, like task\-specific customization, company\-internal knowledge updates, or rapid deployment of new factual content\. As foundation model architectures mature and retraining shifts toward serving primarily for parametric knowledge updates, Golden\-GRPO’s specialized approach to knowledge injection may become attractive at larger scales as well\.

## Ethical considerations

TheCounterbenchmark contains counterfactual statements that contradict real\-world facts; these are constructed strictly as evaluation infrastructure and are not intended for deployment\.

## References

- Asaiet al\.\(2024\)A\. Asai, Z\. Wu, Y\. Wang, A\. Sil, and H\. HajishirziSelf\-rag: learning to retrieve, generate, and critique through self\-reflection\.InInternational conference on learning representations,Vol\.2024,pp\. 9112–9141\.Cited by:[§2\.1](https://arxiv.org/html/2608.25243#S2.SS1.p1.1)\.
- Bird \(2006\)S\. BirdNLTK: the natural language toolkit\.InProceedings of the COLING/ACL 2006 interactive presentation sessions,pp\. 69–72\.Cited by:[Appendix F](https://arxiv.org/html/2608.25243#A6.p1.1),[§5\.1](https://arxiv.org/html/2608.25243#S5.SS1.p2.1)\.
- Chenet al\.\(2021\)W\. Chen, X\. Wang, and W\. Y\. WangA dataset for answering time\-sensitive questions\.arXiv preprint arXiv:2108\.06314\.Cited by:[Appendix F](https://arxiv.org/html/2608.25243#A6.p1.1),[§4\.1\.1](https://arxiv.org/html/2608.25243#S4.SS1.SSS1.p1.1)\.
- Chenget al\.\(2023\)D\. Cheng, S\. Huang, and F\. WeiAdapting large language models via reading comprehension\.InThe Twelfth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.25243#S1.p2.1)\.
- Chuet al\.\(2025\)T\. Chu, Y\. Zhai, J\. Yang, S\. Tong, S\. Xie, D\. Schuurmans, Q\. V\. Le, S\. Levine, and Y\. MaSft memorizes, rl generalizes: a comparative study of foundation model post\-training\.arXiv preprint arXiv:2501\.17161\.Cited by:[§1](https://arxiv.org/html/2608.25243#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.25243#S2.SS1.p2.1)\.
- Donget al\.\(2019\)L\. Dong, N\. Yang, W\. Wang, F\. Wei, X\. Liu, Y\. Wang, J\. Gao, M\. Zhou, and H\. HonUnified language model pre\-training for natural language understanding and generation\.Advances in neural information processing systems32\.Cited by:[§1](https://arxiv.org/html/2608.25243#S1.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[Appendix F](https://arxiv.org/html/2608.25243#A6.p1.1),[§6\.2](https://arxiv.org/html/2608.25243#S6.SS2.p5.1)\.
- Heet al\.\(2025\)B\. He, X\. He, R\. Shao, S\. Shu, X\. Xue, M\. Cheng, H\. Li, and Z\. LingSelect to know: an internal\-external knowledge self\-selection framework for domain\-specific question answering\.Preprint\.Cited by:[§1](https://arxiv.org/html/2608.25243#S1.p2.1)\.
- Huanget al\.\(2026\)L\. Huang, X\. Cheng, C\. Zhao, G\. Shen, J\. Yang, X\. Feng, Y\. Gu, X\. Yu, and B\. QinBootstrapping exploration with group\-level natural language feedback in reinforcement learning\.arXiv preprint arXiv:2603\.04597\.Cited by:[Appendix E](https://arxiv.org/html/2608.25243#A5.p1.1),[§2\.2](https://arxiv.org/html/2608.25243#S2.SS2.p1.1)\.
- Izacardet al\.\(2023\)G\. Izacard, P\. Lewis, M\. Lomeli, L\. Hosseini, F\. Petroni, T\. Schick, J\. Dwivedi\-Yu, A\. Joulin, S\. Riedel, and E\. GraveAtlas: few\-shot learning with retrieval augmented language models\.Journal of Machine Learning Research24\(251\),pp\. 1–43\.Cited by:[§2\.1](https://arxiv.org/html/2608.25243#S2.SS1.p1.1)\.
- Janget al\.\(2021\)J\. Jang, S\. Ye, S\. Yang, J\. Shin, J\. Han, G\. Kim, S\. J\. Choi, and M\. SeoTowards continual knowledge learning of language models\.arXiv preprint arXiv:2110\.03215\.Cited by:[§1](https://arxiv.org/html/2608.25243#S1.p2.1)\.
- Jiet al\.\(2025\)K\. Ji, J\. Chen, A\. Gao, W\. Xie, X\. Wan, and B\. WangUnlocking llms’ self\-improvement capacity with autonomous learning for domain adaptation\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 21051–21067\.Cited by:[§4](https://arxiv.org/html/2608.25243#S4.p1.1),[§6\.1](https://arxiv.org/html/2608.25243#S6.SS1.p2.1)\.
- Jianget al\.\(2025\)K\. Jiang, H\. Jiang, N\. Jiang, Z\. Gao, J\. Bi, Y\. Ren, B\. Li, Y\. Du, L\. Liu, and Q\. LiKORE: enhancing knowledge injection for large multimodal models via knowledge\-oriented augmentations and constraints\.arXiv preprint arXiv:2510\.19316\.Cited by:[§1](https://arxiv.org/html/2608.25243#S1.p2.1)\.
- Jianget al\.\(2024\)Z\. Jiang, Z\. Sun, W\. Shi, P\. Rodriguez, C\. Zhou, G\. Neubig, X\. Lin, W\. Yih, and S\. IyerInstruction\-tuned language models are better knowledge learners\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 5421–5434\.Cited by:[§2\.1](https://arxiv.org/html/2608.25243#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2608.25243#S2.SS1.p2.1),[§4](https://arxiv.org/html/2608.25243#S4.p1.1),[§6\.1](https://arxiv.org/html/2608.25243#S6.SS1.p2.1.5)\.
- Krishnamurthyet al\.\(2024\)A\. Krishnamurthy, K\. Harris, D\. J\. Foster, C\. Zhang, and A\. SlivkinsCan large language models explore in\-context?\.Advances in Neural Information Processing Systems37,pp\. 120124–120158\.Cited by:[§1](https://arxiv.org/html/2608.25243#S1.p2.1)\.
- Levyet al\.\(2024\)M\. Levy, A\. Jacoby, and Y\. GoldbergSame task, more tokens: the impact of input length on the reasoning performance of large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 15339–15353\.Cited by:[§2\.1](https://arxiv.org/html/2608.25243#S2.SS1.p1.1)\.
- Lewiset al\.\(2020\)P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel,et al\.Retrieval\-augmented generation for knowledge\-intensive nlp tasks\.Advances in neural information processing systems33,pp\. 9459–9474\.Cited by:[§2\.1](https://arxiv.org/html/2608.25243#S2.SS1.p1.1),[§6\.1](https://arxiv.org/html/2608.25243#S6.SS1.p2.1.4)\.
- Linet al\.\(2025\)J\. Lin, S\. Wang, X\. Guo, J\. Shun, and Y\. ZhuTemporal reasoning with large language models augmented by evolving knowledge graphs\.arXiv preprint arXiv:2509\.15464\.Cited by:[§1](https://arxiv.org/html/2608.25243#S1.p1.1)\.
- Linet al\.\(2024\)S\. Lin, L\. Gao, B\. Oguz, W\. Xiong, J\. Lin, W\. Yih, and X\. ChenFlame: factuality\-aware alignment for large language models\.Advances in Neural Information Processing Systems37,pp\. 115588–115614\.Cited by:[§2\.2](https://arxiv.org/html/2608.25243#S2.SS2.p1.1)\.
- Liuet al\.\(2024\)A\. Liu, B\. Feng, B\. Xue, B\. Wang, B\. Wu, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang, C\. Ruan,et al\.Deepseek\-v3 technical report\.arXiv preprint arXiv:2412\.19437\.Cited by:[§1](https://arxiv.org/html/2608.25243#S1.p1.1)\.
- Liuet al\.\(2025\)Z\. Liu, C\. Chen, W\. Li, P\. Qi, T\. Pang, C\. Du, W\. S\. Lee, and M\. LinUnderstanding r1\-zero\-like training: a critical perspective\.arXiv preprint arXiv:2503\.20783\.Cited by:[§5\.3](https://arxiv.org/html/2608.25243#S5.SS3.p4.1)\.
- Loshchilov and Hutter \(2017\)I\. Loshchilov and F\. HutterDecoupled weight decay regularization\.arXiv preprint arXiv:1711\.05101\.Cited by:[§6\.1](https://arxiv.org/html/2608.25243#S6.SS1.p3.1)\.
- Maet al\.\(2025\)L\. Ma, H\. Liang, M\. Qiang, L\. Tang, X\. Ma, Z\. H\. Wong, J\. Niu, C\. Shen, R\. He, Y\. Li,et al\.Learning what reinforcement learning can’t: interleaved online fine\-tuning for hardest questions\.arXiv preprint arXiv:2506\.07527\.Cited by:[§2\.2](https://arxiv.org/html/2608.25243#S2.SS2.p1.1)\.
- Mallenet al\.\(2023\)A\. Mallen, A\. Asai, V\. Zhong, R\. Das, D\. Khashabi, and H\. HajishirziWhen not to trust language models: investigating effectiveness of parametric and non\-parametric memories\.InProceedings of the 61st annual meeting of the association for computational linguistics \(volume 1: Long papers\),pp\. 9802–9822\.Cited by:[§2\.1](https://arxiv.org/html/2608.25243#S2.SS1.p1.1)\.
- Mecklenburget al\.\(2024\)N\. Mecklenburg, Y\. Lin, X\. Li, D\. Holstein, L\. Nunes, S\. Malvar, B\. Silva, R\. Chandra, V\. Aski, P\. K\. R\. Yannam,et al\.Injecting new knowledge into large language models via supervised fine\-tuning\.arXiv preprint arXiv:2404\.00213\.Cited by:[§2\.1](https://arxiv.org/html/2608.25243#S2.SS1.p2.1)\.
- Ovadiaet al\.\(2024\)O\. Ovadia, M\. Brief, M\. Mishaeli, and O\. ElishaFine\-tuning or retrieval? comparing knowledge injection in llms\.InProceedings of the 2024 conference on empirical methods in natural language processing,pp\. 237–250\.Cited by:[§1](https://arxiv.org/html/2608.25243#S1.p2.1),[§2\.1](https://arxiv.org/html/2608.25243#S2.SS1.p2.1)\.
- Rafailovet al\.\(2023\)R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. FinnDirect preference optimization: your language model is secretly a reward model\.Advances in neural information processing systems36,pp\. 53728–53741\.Cited by:[§6\.1](https://arxiv.org/html/2608.25243#S6.SS1.p2.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§3](https://arxiv.org/html/2608.25243#S3.p1.1)\.
- Suet al\.\(2025\)W\. Su, Y\. Tang, Q\. Ai, J\. Yan, C\. Wang, H\. Wang, Z\. Ye, Y\. Zhou, and Y\. LiuParametric retrieval augmented generation\.InProceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 1240–1250\.Cited by:[§2\.1](https://arxiv.org/html/2608.25243#S2.SS1.p1.1)\.
- Teamet al\.\(2026\)K\. Team, G\. Chen, Y\. Zhang, J\. Su, W\. Xu, S\. Pan, Y\. Wang, Y\. Wang, G\. Chen, B\. Yin,et al\.Attention residuals\.arXiv preprint arXiv:2603\.15031\.Cited by:[Limitations](https://arxiv.org/html/2608.25243#Sx1.p4.1)\.
- Touvronet al\.\(2023\)H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar,et al\.Llama: open and efficient foundation language models\.arXiv preprint arXiv:2302\.13971\.Cited by:[§1](https://arxiv.org/html/2608.25243#S1.p1.1)\.
- Vuet al\.\(2024\)T\. Vu, M\. Iyyer, X\. Wang, N\. Constant, J\. Wei, J\. Wei, C\. Tar, Y\. Sung, D\. Zhou, Q\. Le,et al\.Freshllms: refreshing large language models with search engine augmentation\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 13697–13720\.Cited by:[§2\.1](https://arxiv.org/html/2608.25243#S2.SS1.p1.1)\.
- Wuet al\.\(2024a\)K\. Wu, E\. Wu, and J\. ZouClasheval: quantifying the tug\-of\-war between an llm’s internal prior and external evidence\.Advances in neural information processing systems37,pp\. 33402–33422\.Cited by:[§2\.1](https://arxiv.org/html/2608.25243#S2.SS1.p1.1)\.
- Wuet al\.\(2024b\)T\. Wu, L\. Luo, Y\. Li, S\. Pan, T\. Vu, and G\. HaffariContinual learning for large language models: a survey\.arXiv preprint arXiv:2402\.01364\.Cited by:[§1](https://arxiv.org/html/2608.25243#S1.p1.1)\.
- Xuet al\.\(2023\)Y\. Xu, M\. Namazifar, D\. Hazarika, A\. Padmakumar, Y\. Liu, and D\. Hakkani\-TurKilm: knowledge injection into encoder\-decoder language models\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 5013–5035\.Cited by:[§2\.1](https://arxiv.org/html/2608.25243#S2.SS1.p2.1)\.
- Xuet al\.\(2024\)Y\. Xu, S\. He, J\. Chen, Z\. Wang, Y\. Song, H\. Tong, G\. Liu, J\. Zhao, and K\. LiuGenerate\-on\-graph: treat llm as both agent and kg for incomplete knowledge graph question answering\.InProceedings of the 2024 conference on empirical methods in natural language processing,pp\. 18410–18430\.Cited by:[§1](https://arxiv.org/html/2608.25243#S1.p1.1)\.
- Xuet al\.\(2025\)Z\. Xu, F\. Jiang, L\. Niu, Y\. Deng, R\. Poovendran, Y\. Choi, and B\. Y\. LinMagpie: alignment data synthesis from scratch by prompting aligned llms with nothing\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 76346–76382\.Cited by:[Appendix C](https://arxiv.org/html/2608.25243#A3.p1.1),[§5\.2](https://arxiv.org/html/2608.25243#S5.SS2.p1.1)\.
- Yanet al\.\(2026\)J\. Yan, Y\. Li, Z\. Hu, Z\. Wang, G\. Cui, X\. Qu, Y\. Cheng, and Y\. ZhangLearning to reason under off\-policy guidance\.Advances in Neural Information Processing Systems38,pp\. 117157–117186\.Cited by:[Appendix E](https://arxiv.org/html/2608.25243#A5.p1.1),[§1](https://arxiv.org/html/2608.25243#S1.p3.1),[§2\.2](https://arxiv.org/html/2608.25243#S2.SS2.p1.1),[§6\.2](https://arxiv.org/html/2608.25243#S6.SS2.p4.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[Appendix F](https://arxiv.org/html/2608.25243#A6.p1.1),[§1](https://arxiv.org/html/2608.25243#S1.p1.1),[§4](https://arxiv.org/html/2608.25243#S4.p2.1),[§6\.1](https://arxiv.org/html/2608.25243#S6.SS1.p1.1)\.
- Yinet al\.\(2023\)Z\. Yin, Q\. Sun, Q\. Guo, J\. Wu, X\. Qiu, and X\. HuangDo large language models know what they don’t know?\.InFindings of the association for Computational Linguistics: ACL 2023,pp\. 8653–8665\.Cited by:[§2\.1](https://arxiv.org/html/2608.25243#S2.SS1.p1.1)\.
- Yue \(2025\)M\. YueA survey of large language model agents for question answering\.arXiv preprint arXiv:2503\.19213\.Cited by:[§1](https://arxiv.org/html/2608.25243#S1.p1.1)\.
- Zhanget al\.\(2025a\)X\. Zhang, B\. Peng, Y\. Tian, J\. Zhou, Y\. Zhang, H\. Mi, and H\. MengSelf\-tuning: instructing llms to effectively acquire new knowledge through self\-teaching\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 5688–5724\.Cited by:[§1](https://arxiv.org/html/2608.25243#S1.p1.1),[§1](https://arxiv.org/html/2608.25243#S1.p2.1),[§2\.1](https://arxiv.org/html/2608.25243#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2608.25243#S2.SS1.p2.1),[§4](https://arxiv.org/html/2608.25243#S4.p1.1),[§6\.1](https://arxiv.org/html/2608.25243#S6.SS1.p2.1.6)\.
- Zhanget al\.\(2025b\)X\. Zhang, Y\. Zhang, H\. Sun, K\. Feng, C\. Lu, C\. Yang, and H\. MengCritique\-grpo: advancing llm reasoning with natural language and numerical feedback\.arXiv preprint arXiv:2506\.03106\.Cited by:[§1](https://arxiv.org/html/2608.25243#S1.p3.1),[§2\.2](https://arxiv.org/html/2608.25243#S2.SS2.p1.1)\.
- Zhouet al\.\(2026\)X\. Zhou, C\. Jin, C\. Eickhoff, Z\. Guo, and S\. A\. BahrainianWhen silence is golden: can llms learn to abstain in temporal qa and beyond?\.arXiv preprint arXiv:2602\.04755\.Cited by:[§4\.1\.1](https://arxiv.org/html/2608.25243#S4.SS1.SSS1.p1.1)\.

## Appendix APrompts used for corpus construction

We construct theCountercorpora using two API calls per subject\. Prompt 1 \(Table[4](https://arxiv.org/html/2608.25243#A1.T4)\) elicits the base model’s prior knowledge of a subject as a short Wikipedia\-style passage\. Prompt 2 \(Table[5](https://arxiv.org/html/2608.25243#A1.T5)\) takes that passage and rewrites it into a parallel\-universe version in which only identifiable concrete facts are replaced, while structure and the subject entity are preserved\.

Table 4:Prompt 1: sampling the base model’s current beliefs about a subject\.Table 5:Prompt 2: rewriting a passage from Prompt 1 into a “parallel\-universe” version\.
## Appendix BPrompts used for question generation

We construct the question pool forBlankandCounterusing three LLM prompts, one per question type\. Prompt 1 \(Table[6](https://arxiv.org/html/2608.25243#A2.T6)\) generates single\-fact recall questions from a single sentence of the source document\. Prompt 2 \(Table[7](https://arxiv.org/html/2608.25243#A2.T7)\) generates multi\-source retrieval questions composed ofN∈\{2,3,4\}N\\in\\\{2,3,4\\\}sub\-questions targeting distinct facts that span different paragraphs or documents\. Prompt 3 \(Table[8](https://arxiv.org/html/2608.25243#A2.T8)\) generates inferential reasoning questions that require combining multiple facts from the source document\. ForBlank’s inferential reasoning category, we additionally include the original TimeQA questions for TimeQA\-derived passages; Prompt 3 is used only for the LLM\-augmented portion and for all post\-cutoff andCounterpassages\.

Table 6:Prompt for generating single\-fact recall questions\.Table 7:Prompt for generating multi\-source retrieval questions\.Table 8:Prompt for generating inferential reasoning questions\.
## Appendix CExample Stage 2 QA pair for Golden\-GRPO

To illustrate Stage 2’s corpus\-conditioned sampling mechanism, we show how the pre\-query token elicits a diverse question from the base model conditioned on a source corpus\. Following[Xu et al\. \(2025\)](https://arxiv.org/html/2608.25243#bib.bib35), we construct a prompt by prepending the corpus and a system prompt before the pre\-query token, and let the base model continue from there\. The model’s continuation is then truncated at the first end\-of\-turn marker to extract the sampled question\. Table[9](https://arxiv.org/html/2608.25243#A3.T9)shows a representative example using aCountercorpus on Apple’s stock listing \(where the real\-world fact is Apple on NASDAQ underAAPL, but the corpus has been rewritten to place Apple on the NYSE underAPL\)\.

Table 9:Stage 2 question sampling using a pre\-query token\.Table 10:A representative \(question, golden\-answer\) data pair from Stage 2 sampling onCounter\.
## Appendix DJudge Prompts

We use two LLM\-as\-judge prompts for evaluation\. The accuracy judge \(Table[11](https://arxiv.org/html/2608.25243#A4.T11)\) determines whether a model’s answer contains the same factual information as the reference answer; this is used for all per\-type accuracy scores reported in Tables 2 and 3\. The fail@k judge \(Table[12](https://arxiv.org/html/2608.25243#A4.T12)\) is used only forCounter, classifying each ofkksampled answers as agreeing with the counterfactual training text, the original Wikipedia prior, or neither\.

Table 11:Prompt used by the accuracy judge to score per\-type accuracy\.Table 12:Prompt used by the fail@k judge\.
## Appendix EGradient dynamics of Golden\-GRPO vs\. importance\-weighted mixed\-policy RL

This appendix formalizes the gradient\-vanishing problem of importance\-weighted mixed\-policy RL in the knowledge injection setting, and shows why Golden\-GRPO’s direct supervised gradient design avoids it\. We use LUFFY\([Yan et al\., 2026](https://arxiv.org/html/2608.25243#bib.bib31)\)and GOLF[Huang et al\. \(2026\)](https://arxiv.org/html/2608.25243#bib.bib43)as two representative importance\-weighted mixed\-policy baselines that share the same structural failure mode\.

##### LUFFY’s mixed\-policy objective\.

LUFFY combines on\-policy rollouts with off\-policy trajectories drawn from a reference policyπϕ\\pi\_\{\\phi\}\(in our case, the LLM that generated the golden answer\):

𝒥LUFFY​\(θ\)=1Z​\(CLOSE\\displaystyle\\mathcal\{J\}\_\{\\text\{LUFFY\}\}\(\\theta\)=\\frac\{1\}\{Z\}\\bigg\(∑j=1Noff∑t=1\|τj\|CLIP​\(r^j,tLUFFY,A^j,ε\)⏟off\-policy branch\\displaystyle\\underbrace\{\\sum\_\{j=1\}^\{N\_\{\\text\{off\}\}\}\\sum\_\{t=1\}^\{\|\\tau\_\{j\}\|\}\\text\{CLIP\}\\big\(\\hat\{r\}\_\{j,t\}^\{\\text\{LUFFY\}\},\\hat\{A\}\_\{j\},\\varepsilon\\big\)\}\_\{\\text\{off\-policy branch\}\}OPEN\+∑i=1Non∑t=1\|τi\|CLIP​\(ri,t​\(θ\),A^i,ε\)⏟on\-policy branch\)\\displaystyle\+\\underbrace\{\\sum\_\{i=1\}^\{N\_\{\\text\{on\}\}\}\\sum\_\{t=1\}^\{\|\\tau\_\{i\}\|\}\\text\{CLIP\}\\big\(r\_\{i,t\}\(\\theta\),\\hat\{A\}\_\{i\},\\varepsilon\\big\)\}\_\{\\text\{on\-policy branch\}\}\\bigg\)where the off\-policy importance ratio isr^j,tLUFFY=πθ​\(τj,t∣q,τj,<t\)/πϕ​\(τj,t∣q,τj,<t\)\\hat\{r\}\_\{j,t\}^\{\\text\{LUFFY\}\}=\\pi\_\{\\theta\}\(\\tau\_\{j,t\}\\mid q,\\tau\_\{j,<t\}\)/\\pi\_\{\\phi\}\(\\tau\_\{j,t\}\\mid q,\\tau\_\{j,<t\}\)\.

##### GOLF’s mixed\-policy objective\.

GOLF takes a similar form but constructs the off\-policy ratio differently\. Rather than drawing trajectories from an external reference policy, GOLF augments the original promptxxwith natural\-language feedback or guidancepagg​\(x\)p\_\{\\text\{agg\}\}\(x\)that helps the model produce a correct answer, and treats the resulting trajectory as off\-policy data\. The off\-policy ratio is:

rj,tGOLF​\(θ\)=πθ​\(τj,t∣x,τj,<t\)πθold​\(τj,t∣pagg​\(x\),τj,<t\)r\_\{j,t\}^\{\\text\{GOLF\}\}\(\\theta\)=\\frac\{\\pi\_\{\\theta\}\(\\tau\_\{j,t\}\\mid x,\\tau\_\{j,<t\}\)\}\{\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(\\tau\_\{j,t\}\\mid p\_\{\\text\{agg\}\}\(x\),\\tau\_\{j,<t\}\)\}Here the numerator and denominator condition on*different prompts*: the numerator on the plain queryxx, and the denominator on the augmented querypagg​\(x\)p\_\{\\text\{agg\}\}\(x\)that explicitly includes guidance\.

##### Why both ratios vanish in knowledge injection\.

Both LUFFY and GOLF suffer from the same structural problem: the denominator’s probability for the off\-policy trajectoryτ⋆\\tau^\{\\star\}is substantially larger than the numerator’s, making the ratio small precisely when the model has not yet learned the injected fact\.

For LUFFY, the reference policyπϕ\\pi\_\{\\phi\}assigns high probability to its own generated trajectory, so the denominatorπϕ​\(τ⋆\)\\pi\_\{\\phi\}\(\\tau^\{\\star\}\)is large\. The numeratorπθ​\(τ⋆\)\\pi\_\{\\theta\}\(\\tau^\{\\star\}\)is small in early training because the model has not learned the fact\. The ratior^LUFFY=πθ/πϕ\\hat\{r\}^\{\\text\{LUFFY\}\}=\\pi\_\{\\theta\}/\\pi\_\{\\phi\}collapses\.

For GOLF, the asymmetry is even more pronounced\. The augmented promptpagg​\(x\)p\_\{\\text\{agg\}\}\(x\)is*explicitly designed*to makeτ⋆\\tau^\{\\star\}likely, that it contains feedback or guidance that steers the model toward the correct answer\. Thusπθold​\(τ⋆∣pagg​\(x\)\)\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(\\tau^\{\\star\}\\mid p\_\{\\text\{agg\}\}\(x\)\)is by construction high, whileπθ​\(τ⋆∣x\)\\pi\_\{\\theta\}\(\\tau^\{\\star\}\\mid x\)on the plain prompt remains small until the model has actually learned the fact\. The ratiorGOLF=πθ\(⋅∣x\)/πθold\(⋅∣pagg\(x\)\)r^\{\\text\{GOLF\}\}=\\pi\_\{\\theta\}\(\\cdot\\mid x\)/\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(\\cdot\\mid p\_\{\\text\{agg\}\}\(x\)\)collapses for the same reason as LUFFY\.

In both cases, the gradient contribution of the off\-policy branch is proportional to this small ratio, regardless of how large the off\-policy advantageA^⋆\\hat\{A\}^\{\\star\}is\.

##### Golden\-GRPO’s design\.

Golden\-GRPO removes the importance ratio entirely from the off\-policy branch and replaces it with a direct supervised gradient scaled by the off\-policy advantage:

𝒥offGolden=A⋆⋅log⁡πθ​\(yi,t∣yi,<t,q\)\\mathcal\{J\}\_\{\\text\{off\}\}^\{\\text\{Golden\}\}=A^\{\\star\}\\cdot\\log\\pi\_\{\\theta\}\(y\_\{i,t\}\\mid y\_\{i,<t\},q\)The gradient with respect toθ\\thetais:

∇θ𝒥offGolden=A⋆⋅∇θ​log​πθ​\(yi,t∣yi,<t,q\)\\nabla\_\{\\theta\}\\mathcal\{J\}\_\{\\text\{off\}\}^\{\\text\{Golden\}\}=A^\{\\star\}\\cdot\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{i,t\}\\mid y\_\{i,<t\},q\)The magnitude depends only on the advantageA⋆A^\{\\star\}and the standard supervised gradient∇θ​log​πθ\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\. Critically, this does*not*vanish whenπθ​\(τ⋆∣q\)\\pi\_\{\\theta\}\(\\tau^\{\\star\}\\mid q\)is small; the supervised gradient is well\-defined and stable across the range ofπθ\\pi\_\{\\theta\}values encountered during training\.

##### Three\-phase gradient analysis\.

To make the self\-regulating behavior concrete, we trace the gradient composition through three representative training phases\. For simplicity, assume the reward is binary:R⁡\(τ\)=RmaxR\(\\tau\)=R\_\{\\max\}if the trajectory is correct,00otherwise\. Golden\-GRPO usesNonN\_\{\\text\{on\}\}on\-policy rollouts plus one off\-policy golden trajectoryτ⋆\\tau^\{\\star\}withR⁡\(τ⋆\)=RmaxR\(\\tau^\{\\star\}\)=R\_\{\\max\}\.

Phase 1: All on\-policy rollouts incorrect\.The group reward distribution is\{Rmax,0,0,…,0\}\\\{R\_\{\\max\},0,0,\\ldots,0\\\}, giving group meanR¯=Rmax/\(Non\+1\)\\bar\{R\}=R\_\{\\max\}/\(N\_\{\\text\{on\}\}\+1\)\. The advantages are:

A⋆=Rmax⋅NonNon\+1,Ai=−RmaxNon\+1A^\{\\star\}=R\_\{\\max\}\\cdot\\frac\{N\_\{\\text\{on\}\}\}\{N\_\{\\text\{on\}\}\+1\},\\quad A\_\{i\}=\-\\frac\{R\_\{\\max\}\}\{N\_\{\\text\{on\}\}\+1\}The off\-policy advantageA⋆A^\{\\star\}is large and positive, while on\-policy advantages are small and negative\. The gradient is dominated by the off\-policy supervised term, pulling the model strongly towardτ⋆\\tau^\{\\star\}\.

Phase 2: Mixed rollouts\.SupposekkofNonN\_\{\\text\{on\}\}on\-policy rollouts are correct, the rest incorrect\. The group mean rises toR¯=\(k\+1\)​Rmax/\(Non\+1\)\\bar\{R\}=\(k\+1\)R\_\{\\max\}/\(N\_\{\\text\{on\}\}\+1\), and the advantages become:

A⋆=Acorrect=Rmax⋅Non−kNon\+1A^\{\\star\}=A\_\{\\text\{correct\}\}=R\_\{\\max\}\\cdot\\frac\{N\_\{\\text\{on\}\}\-k\}\{N\_\{\\text\{on\}\}\+1\}Awrong=−Rmax⋅k\+1Non\+1A\_\{\\text\{wrong\}\}=\-R\_\{\\max\}\\cdot\\frac\{k\+1\}\{N\_\{\\text\{on\}\}\+1\}The off\-policy advantage has shrunk; correct on\-policy rollouts now share the same positive advantage as the golden trajectory, so they contribute positively to the policy update\. The model reinforces both the golden trajectory and its own correct outputs simultaneously, and the off\-policy branch’s relative dominance over the total gradient diminishes askkgrows\.

Phase 3: All on\-policy rollouts correct\.The group reward distribution becomes uniformlyRmaxR\_\{\\max\}, givingR¯=Rmax\\bar\{R\}=R\_\{\\max\}andA⋆=Ai=0A^\{\\star\}=A\_\{i\}=0for all trajectories\. The entire objective gradient vanishes for this question\. The model has converged on this fact, and further training resources are naturally redirected to questions where the rollout group still shows advantage variance\.

##### Why this matters\.

This three\-phase progression demonstrates Golden\-GRPO’s self\-regulating property in concrete form: the off\-policy gradient is largest when the model needs it most \(Phase 1\), shrinks gracefully as the model learns \(Phase 2\), and reaches zero exactly when the model has internalized the fact \(Phase 3\)\. The transition emerges automatically from the advantage formula and does not require any external schedule or annealing\. Importantly, the model is never*over\-pulled*toward the golden trajectory: once on\-policy rollouts succeed at the same rate as the off\-policy reference, the supervised gradient towardτ⋆\\tau^\{\\star\}disappears, preventing the model from overfitting onto the specific phrasing of the LLM\-generated golden answer\.

## Appendix FReproducibility Statement\.

We use Qwen3\-4B[Yang et al\. \(2025\)](https://arxiv.org/html/2608.25243#bib.bib2), Llama3\.2\-3B[Grattafiori et al\. \(2024\)](https://arxiv.org/html/2608.25243#bib.bib40), TimeQA[Chen et al\. \(2021\)](https://arxiv.org/html/2608.25243#bib.bib33), and NLTK[Bird \(2006\)](https://arxiv.org/html/2608.25243#bib.bib34)under their respective open\-source licenses\. Code and benchmark data will be made publicly available; a code and data archive accompanies this submission for reviewer access\.

## Appendix GThe Use of Large Language Models \(LLMs\)

Large language models were used solely to aid in polishing the writing of this paper\. They were not used for research ideation, methodology, analysis, or concluding\. The authors take full responsibility for all content\.