LC-ERD: Mining Latent Logic for Self-Evolving Reasoning via Consistency-Regulated Reward Decomposition

# LC-ERD: Mining Latent Logic for Self-Evolving Reasoning via Consistency-Regulated Reward Decomposition
Source: [https://arxiv.org/html/2605.24005](https://arxiv.org/html/2605.24005)
\(2026\)

###### Abstract\.

The evolution of Large Language Model \(LLM\) reasoning is bottlenecked by the scarcity of high\-quality process data\. While self\-alignment via endogenous rewards offers a solution, mining valid supervision faces three challenges: \(1\)Label Noise via Mimetic Bias, where rewards prioritize statistical likelihood over logical truth, creating a “correctness illusion” that masks compounding errors; \(2\)Coarse\-Grained Supervision, where sparse global outcomes \(e\.g\., in GRPO\) fail to provide granular guidance, treating reasoning chains as monolithic; and \(3\)Distributional Collapse, where signals fail to generalize without amplifying pre\-training biases\. To address these, we introduceLC\-ERD\(Logic\-Consistent Endogenous Reward Decomposition\), a framework framing self\-alignment as latent structure mining\. We derive aVariational Logic Potentialby aggregating consensus from the model’sLatent Logic Expertise\(LLE\) to denoise the reasoning manifold, and introduce aMulti\-Agent Value Decompositionprotocol based on the IGM principle to quantify individual step utility\. Experiments show LC\-ERD delivers a robust self\-evolution path, uncovering trade\-offs between logic consistency and accuracy while identifying high\-value reasoning patterns missed by standard rewards\. Our code is available at[https://github\.com/Reinhardmannn/LC\-ERD](https://github.com/Reinhardmannn/LC-ERD)\.

Self\-Evolving Reasoning, Credit Assignment, Endogenous Reward

††copyright:acmlicensed††journalyear:2026††conference:Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining; August 9–13, 2026; Jeju, Republic of Korea††ccs:Computing methodologies Reinforcement learning## 1\.Introduction

The rapid advancement of Large Language Models \(LLMs\) has significantly extended their problem\-solving capabilities, enabling autonomous problem\-solving across diverse domains ranging from formal mathematics to clinical diagnosis\(Wang et al\.,[2025b](https://arxiv.org/html/2605.24005#bib.bib37);Jiang et al\.,[2025](https://arxiv.org/html/2605.24005#bib.bib18);Wang et al\.,[2026](https://arxiv.org/html/2605.24005#bib.bib35)\)\. Central to this progress is the paradigm of self\-alignment\(Sun et al\.,[2023](https://arxiv.org/html/2605.24005#bib.bib31);Jiang et al\.,[2026](https://arxiv.org/html/2605.24005#bib.bib17)\), where models leverage endogenous reward signals—theoretically equivalent to solving an offline inverse reinforcement learning \(IRL\)\(Jarboui and Perchet,[2021](https://arxiv.org/html/2605.24005#bib.bib15);Ng and Russell,[2000](https://arxiv.org/html/2605.24005#bib.bib27)\)objective—to iteratively refine their reasoning capabilities without the prohibitive cost of human\-in\-the\-loop annotation\. By eliciting these latent value functions\(Wu et al\.,[2025](https://arxiv.org/html/2605.24005#bib.bib38)\), LLMs can theoretically achieve a closed\-loop self\-evolution, identifying and reinforcing high\-quality trajectories within their own output distributions\.

![Refer to caption](https://arxiv.org/html/2605.24005v1/x1.png)Figure 1\.Comparison of Reward Mechanisms\.\(a\) Outcome\-based Supervision \(e\.g\., GRPO\) assigns uniform credit \(\+0\.1\+0\.1\) based solely on the final answer\. This creates a “Correctness Illusion”, where intermediate logic errors \(red nodes\) are inadvertently reinforced\. \(b\) LC\-ERD \(Ours\) mines latent logic to decompose global rewards\. It detects Latent Logic Violations, applying granular penalties \(e\.g\.,−0\.11\-0\.11\) to structural flaws to distinguish valid reasoning from “lucky guesses”\.Despite the allure of such self\-supervised evolution\(Vinhas et al\.,[2024](https://arxiv.org/html/2605.24005#bib.bib34)\), mining high\-fidelity supervision from the model’s own generations remains a significant challenge\. Current state\-of\-the\-art endogenous approaches, such as Group Relative Policy Optimization \(GRPO\)\(Shao et al\.,[2024](https://arxiv.org/html/2605.24005#bib.bib30)\), rely on coarse\-grained outcome supervision\. While effective for short horizons, these monolithic signals suffer from aGranularity Mismatch: they assign uniform positive credit to all steps in a successful chain, indiscriminately reinforcing significant deductions alongside “lucky guesses” or even logical hallucinations\. This lack of discrimination creates aCorrectness Illusion\(Figure[1](https://arxiv.org/html/2605.24005#S1.F1)a\), where intermediate logic errors \(visualized as red nodes\) are inadvertently mined as valid patterns due to label noise\. To escape thisMimetic Trap, we argue that self\-evolution must shift from blind outcome matching tofine\-grained structure mining\. Specifically, we must decompose the global reward to detect and penalizeLatent Logic Violations\(Figure[1](https://arxiv.org/html/2605.24005#S1.F1)b\), ensuring that credit is dynamically attributed—assigning granular penalties \(negative potentials\) to structural flaws while isolating and reinforcing the true logic backbone\.

To address this challenge, we shift away from the pursuit of a monolithic, imitation\-based reward signal, instead optimizing theLogic\-Consistent Endogenous Reward Decomposition \(LC\-ERD\)—a framework that models the reasoning trajectory as a sequence of discrete agentic decisions driven by a continuous logic potential\. We introduce LC\-ERD to reformulate the self\-alignment process into a collaborative game among sequential “step\-agents,” governed by the Individual\-Global\-Max \(IGM\) principle\. Our framework samples logic\-consistent reward signals from a latent expert manifold, delivering high\-fidelity feedback that is tailored to the cognitive difficulty of each reasoning step\. By decomposing the global terminal reward into dense, consistency\-regulated intrinsic potentials, LC\-ERD effectively resolves the credit assignment bottleneck that plagues traditional endogenous alignment while bypassing the limitations of static, one\-size\-fits\-all reward functions\.

The core of LC\-ERD lies in two synergistic mechanisms: \(1\)Latent Logic Expertise \(LLE\) Discovery, which extracts a “logical consensus” from multi\-path samplings to serve as an endogenous anchor; and \(2\)Variational Logic Potential \(VLP\), a continuous reward\-shaping function that penalizes logical drifts by quantifying the divergence between the current policy and the discovered consensus\. Unlike existing handcrafted or automated systems that rely on static rules or expensive external judges, LC\-ERD adapts its corrective strength based on the model’s internal uncertainty\. This allows for a more disciplined collaboration between the model’s generative capacity and its evaluative intuition, ensuring that the self\-evolution path remains anchored to logical truth even in the absence of external supervision\.

Comprehensive evaluation across mathematical reasoning \(MATH\-500\(Mayilvahanan et al\.,[2025](https://arxiv.org/html/2605.24005#bib.bib25)\), GSM8K\(Zhong et al\.,[2026](https://arxiv.org/html/2605.24005#bib.bib49)\)\) and specialized medical dialogue benchmarks demonstrates that LC\-ERD \(I\) requires zero human preference data or external LLM calls, \(II\) surpasses standard endogenous reward baselines by 8\.5%∼\\sim14\.2% in reasoning accuracy, and \(III\) enjoys superior robustness against logical hallucination in long\-chain trajectories\. Our theoretical analysis further confirms that LC\-ERD compresses the sub\-optimality bound\(Bozkurt et al\.,[2026](https://arxiv.org/html/2605.24005#bib.bib2)\)of self\-evolution by a factor of\(1−δ\)\(1\-\\delta\), establishing a provably more stable convergence path\. The main contributions of this work are as follows:

- •Formal Deconstruction of Reasoning Inhibitors\.We identify themimetic biasandcredit assignment bottleneckas the primary inhibitors of self\-evolving reasoning, providing a formal deconstruction of why monolithic endogenous rewards fail in long\-chain tasks\.
- •Logic\-Consistent Reward Decomposition Framework\.We introduceLC\-ERD, an automated framework that factorizes global reasoning utility into consistency\-driven individual rewards, implementing the IGM principle for token\-level self\-alignment\.
- •Automated Logical Anchor Elicitation\.We propose theLatent Logic Expertise \(LLE\)algorithm, a training\-free protocol to elicit logical anchors from model distributions, effectively bypassing the limitations of manual rule design\.
- •Broad Empirical and Theoretical Validation\.We demonstrate through extensive experiments that LC\-ERD significantly improves both the performance and the reliability of LLM self\-evolution, outperforming state\-of\-the\-art baselines across diverse logical and healthcare domains\.

## 2\.Methodology

![Refer to caption](https://arxiv.org/html/2605.24005v1/x2.png)Figure 2\.The LC\-ERD Framework Architecture and Training Paradigm\.\(a\)Latent Logic Expertise \(LLE\)is elicited via Conditioned Sampling to construct the Logic\-Consistent Reasoning Manifold \(LCRM\) across diverse benchmarks \(Math, Medical, Legal\)\. \(b\) TheLC\-ERD Decompositionmodule translates internal states into a dual\-component rewardR=r~\+γ​\(Φh\+1−Φh\)R=\\tilde\{r\}\+\\gamma\(\\Phi\_\{h\+1\}\-\\Phi\_\{h\}\), wherer~\\tilde\{r\}captures mimetic patterns andΦ\\Phirepresents the variational logic potential\. \(c\) TheTraining Objectivecombines a Temporal Difference lossℒT​D\\mathcal\{L\}\_\{TD\}for value stability with a logit\-alignment term𝒟i​n​d​i​v\\mathcal\{D\}\_\{indiv\}, enabling the model to self\-evolve by minimizing the divergence between its reasoning path and the latent logical ground\-truth\.In this section, we present the technical framework ofLC\-ERD\. As illustrated inFigure[2](https://arxiv.org/html/2605.24005#S2.F2), our framework decouples the self\-alignment process into latent expertise discovery, reward decomposition, and a multi\-agent joint optimization paradigm\. Standard endogenous rewards treat self\-alignment as a straightforward imitation task\. In contrast, LC\-ERD reformulates reasoning as a multi\-agent coordination\(Zhang et al\.,[2020](https://arxiv.org/html/2605.24005#bib.bib46)\)game\. We bridge the gap between mimetic probability and logical truth by optimizing a consistency\-regulated potential field that adaptively penalizes logical drifts\.

### 2\.1\.Reasoning as a Cooperative Endogenous MDP

To provide a rigorous foundation for reward decomposition, we formalize the reasoning process of an LLM as a finite\-horizon, discrete\-time Markov Decision Process \(MDP\)\(Garcia and Rachelson,[2013](https://arxiv.org/html/2605.24005#bib.bib12)\)denoted byℳ=⟨𝒮,𝒜,𝒫,r,γ,H⟩\\mathcal\{M\}=\\langle\\mathcal\{S\},\\mathcal\{A\},\\mathcal\{P\},r,\\gamma,H\\rangle\. For a given queryxx, the state at steph∈\[1,H\]h\\in\[1,H\]is defined assh=\[x,a1,…,ah−1\]∈𝒮s\_\{h\}=\[x,a\_\{1\},\\dots,a\_\{h\-1\}\]\\in\\mathcal\{S\}, which encapsulates the initial prompt and the partial reasoning trajectory generated so far\. The actionah∈𝒱a\_\{h\}\\in\\mathcal\{V\}corresponds to the selection of a token from the model’s vocabulary𝒱\\mathcal\{V\}at stephh\. The transition function𝒫:𝒮×𝒜→𝒮\\mathcal\{P\}:\\mathcal\{S\}\\times\\mathcal\{A\}\\rightarrow\\mathcal\{S\}is deterministic, such that:

\(1\)sh\+1=\[sh,ah\]\.s\_\{h\+1\}=\[s\_\{h\},a\_\{h\}\]\.We assume the base modelπbase\\pi\_\{\\text\{base\}\}implicitly satisfies the soft Bellman consistency\(Xie et al\.,[2021](https://arxiv.org/html/2605.24005#bib.bib40)\)during its pre\-training, rendering its logitsf​\(s,a\)f\(s,a\)equivalent to a latent soft Q\-function\(Kang et al\.,[2025](https://arxiv.org/html/2605.24005#bib.bib19)\)Q^​\(s,a\)\\hat\{Q\}\(s,a\)\. Under the maximum entropy\(Jaynes,[1982](https://arxiv.org/html/2605.24005#bib.bib16)\)RL framework, the soft state\-value functionV​\(s\)V\(s\)is thus defined as:

\(2\)V​\(s\)=α​log​∑a′∈𝒱exp⁡\(Q^​\(s,a′\)/α\),V\(s\)=\\alpha\\log\\sum\_\{a^\{\\prime\}\\in\\mathcal\{V\}\}\\exp\(\\hat\{Q\}\(s,a^\{\\prime\}\)/\\alpha\),whereα∈ℝ\+\\alpha\\in\\mathbb\{R\}^\{\+\}denotes the temperature parameter controlling entropy\-regularization; andQ^​\(s,a′\)\\hat\{Q\}\(s,a^\{\\prime\}\)represents the unnormalized predictive logit\(Pang and Jin,[2025](https://arxiv.org/html/2605.24005#bib.bib28)\)\. Following the endogenous reward hypothesis, the fundamental imitation\-based rewardr~\\tilde\{r\}is elicited through the inverse soft Bellman operator:

\(3\)r~​\(sh,ah\)=α​log⁡πbase​\(ah\|sh\)\+V​\(sh\)−γ​V​\(sh\+1\),\\tilde\{r\}\(s\_\{h\},a\_\{h\}\)=\\alpha\\log\\pi\_\{\\text\{base\}\}\(a\_\{h\}\|s\_\{h\}\)\+V\(s\_\{h\}\)\-\\gamma V\(s\_\{h\+1\}\),whereπbase​\(ah\|sh\)\\pi\_\{\\text\{base\}\}\(a\_\{h\}\|s\_\{h\}\)is the conditional probability of actionaha\_\{h\}given stateshs\_\{h\}; andγ∈\[0,1\]\\gamma\\in\[0,1\]is the discount factor\. The intent of Eq\. \(3\) is to extract the “imitation signal” internalized from the pre\-training corpus\(Pang and Jin,[2025](https://arxiv.org/html/2605.24005#bib.bib28)\)\. However, such a signal is blind to structural logic, as it primarily measures statistical likelihood rather than deductive soundness, creating themimetic bias\(Zhou et al\.,[2025](https://arxiv.org/html/2605.24005#bib.bib50)\)\.

Remark 1: Theoretical Foundation of Logits\. The assumption that pre\-trained logitsf​\(s,a\)f\(s,a\)serve as a latent soft Q\-function is grounded in the Maximum Entropy \(MaxEnt\) RL framework\(Eysenbach and Levine,[2021](https://arxiv.org/html/2605.24005#bib.bib10)\)\. We provide a formal derivation in Appendix A\.1, proving that the standard next\-token prediction objective is theoretically equivalent to a principled offline inverse reinforcement learning \(IRL\) objective\.

### 2\.2\.Latent Logic Expertise \(LLE\) Discovery

To resolve the lack of external logical anchors, LC\-ERD elicits a dynamic expert manifold from the model’s own distribution through aconsensus\-based mining protocol\(Wu et al\.,[2026](https://arxiv.org/html/2605.24005#bib.bib39)\)\. We treat the model’s generation space as a noisy dataset containing both valid logic and hallucinations\. For a queryxx, we performKKindependent samplings to generate a candidate set𝒯=\{τ\(1\),…,τ\(K\)\}\\mathcal\{T\}=\\\{\\tau^\{\(1\)\},\\dots,\\tau^\{\(K\)\}\\\}\. We identify a subset of verified trajectories𝒯∗=\{τ∈𝒯∣Ver​\(τ\)=1\}\\mathcal\{T\}^\{\*\}=\\\{\\tau\\in\\mathcal\{T\}\\mid\\text\{Ver\}\(\\tau\)=1\\\}that satisfy terminal logic constraints\. To filter out stochastic noise and extract the stable logical core, we assign a confidence weightω​\(τ\)\\omega\(\\tau\)to each successful path based on its joint probability:

\(4\)ω​\(τ\)=exp⁡\(1H​∑h=1Hlog⁡πbase​\(ah\|sh\)/τtemp\),\\omega\(\\tau\)=\\exp\\left\(\\frac\{1\}\{H\}\\sum\_\{h=1\}^\{H\}\\log\\pi\_\{\\text\{base\}\}\(a\_\{h\}\|s\_\{h\}\)/\\tau\_\{\\text\{temp\}\}\\right\),whereHHis the trajectory length; andτtemp\\tau\_\{\\text\{temp\}\}is a smoothing hyperparameter that calibrates the weight distribution\(Li et al\.,[2025a](https://arxiv.org/html/2605.24005#bib.bib21)\)\. TheLatent Logic ExpertisedistributionπL​L​E\\pi\_\{LLE\}is then constructed:

\(5\)πL​L​E​\(ah\|sh\)=\\displaystyle\\pi\_\{LLE\}\(a\_\{h\}\|s\_\{h\}\)=σ\(1∑kω​\(τ\(k\)\)∑τ\(k\)∈𝒯∗ω\(τ\(k\)\)\\displaystyle\\sigma\\left\(\\frac\{1\}\{\\sum\_\{k\}\\omega\(\\tau^\{\(k\)\}\)\}\\sum\_\{\\tau^\{\(k\)\}\\in\\mathcal\{T\}^\{\*\}\}\\omega\(\\tau^\{\(k\)\}\)\\right\.⋅ϕ\(sh\(k\),ah\)\),\\displaystyle\\cdot\\phi\(s\_\{h\}^\{\(k\)\},a\_\{h\}\)\\Bigg\),whereσ\\sigmadenotes the softmax operator; andϕ​\(sh\(k\),ah\)\\phi\(s\_\{h\}^\{\(k\)\},a\_\{h\}\)represents the unnormalized logits from the pre\-trained model\. By marginalizing over multiple successful paths, Eq\. \(5\) filters out idiosyncratic hallucinations and reveals the underlying “logical backbone\.”

Remark 2: Unbiased Logic Elicitation\. To address concerns of “correctness illusions,” LC\-ERD utilizes a hard terminal constraint to filter the expert manifold\. We prove in Appendix A\.2 that the resulting reward signal remains an unbiased estimator of the true reasoning distribution even when elicited endogenously\.

### 2\.3\.Variational Logic Potential \(VLP\) Shaping

A core innovation of LC\-ERD is theVariational Logic Potential \(VLP\), which converts discrete logical consistency into a continuous energy field\(Wang et al\.,[2025a](https://arxiv.org/html/2605.24005#bib.bib36);Deng et al\.,[2025](https://arxiv.org/html/2605.24005#bib.bib8)\)\. We define the potential functionΦL​L​E​\(sh\)\\Phi\_\{LLE\}\(s\_\{h\}\)as the alignment degree between the current policyπθ\\pi\_\{\\theta\}and the expert manifold:

\(6\)ΦL​L​E\(sh\)=V\(sh\)−β⋅DKL\(πθ\(⋅\|sh\)∥πL​L​E\(⋅\|sh\)\),\\Phi\_\{LLE\}\(s\_\{h\}\)=V\(s\_\{h\}\)\-\\beta\\cdot\\text\{D\}\_\{\\text\{KL\}\}\\left\(\\pi\_\{\\theta\}\(\\cdot\|s\_\{h\}\)\\parallel\\pi\_\{LLE\}\(\\cdot\|s\_\{h\}\)\\right\),whereβ\>0\\beta\>0is the logic\-regularization coefficient; andDKL\(⋅∥⋅\)\\text\{D\}\_\{\\text\{KL\}\}\(\\cdot\\parallel\\cdot\)is the Kullback\-Leibler divergence\(Van Erven and Harremos,[2014](https://arxiv.org/html/2605.24005#bib.bib33)\)\. The potentialΦL​L​E\\Phi\_\{LLE\}quantifies the “logical energy” of a state\. To provide dense guidance, we define the potential differenceΔ​ΦL​L​E\\Delta\\Phi\_\{LLE\}as:

\(7\)Δ​ΦL​L​E​\(sh,ah\)=γ​ΦL​L​E​\(sh\+1\)−ΦL​L​E​\(sh\)\.\\Delta\\Phi\_\{LLE\}\(s\_\{h\},a\_\{h\}\)=\\gamma\\Phi\_\{LLE\}\(s\_\{h\+1\}\)\-\\Phi\_\{LLE\}\(s\_\{h\}\)\.The final LC\-ERD reward is reconstructed by augmenting the mimetic signalr~\\tilde\{r\}with this corrective forceΔ​ΦL​L​E\\Delta\\Phi\_\{LLE\}\(Figure[2](https://arxiv.org/html/2605.24005#S2.F2)b\):

\(8\)RL​C−E​R​D​\(sh,ah\)=r~​\(sh,ah\)\+Δ​ΦL​L​E​\(sh,ah\)\.R\_\{LC\-ERD\}\(s\_\{h\},a\_\{h\}\)=\\tilde\{r\}\(s\_\{h\},a\_\{h\}\)\+\\Delta\\Phi\_\{LLE\}\(s\_\{h\},a\_\{h\}\)\.Substituting Eq\. \(3\) and Eq\. \(6\) into Eq\. \(8\), we derive the expanded step\-wise reward signal:

\(9\)RL​C−E​R​D=\\displaystyle R\_\{LC\-ERD\}=αlogπbase\(ah\|sh\)\+β\[DKL\(sh\)\\displaystyle\\alpha\\log\\pi\_\{\\text\{base\}\}\(a\_\{h\}\|s\_\{h\}\)\+\\beta\\left\[\\text\{D\}\_\{\\text\{KL\}\}\(s\_\{h\}\)\\right\.−γDKL\(sh\+1\)\],\\displaystyle\\left\.\-\\gamma\\text\{D\}\_\{\\text\{KL\}\}\(s\_\{h\+1\}\)\\right\],where the term in the brackets incentivizes the model to minimize logical divergence in each state transition\(Ichihara et al\.,[2025](https://arxiv.org/html/2605.24005#bib.bib14)\), resolving thecredit assignment\(Lansdell et al\.,[2019](https://arxiv.org/html/2605.24005#bib.bib20);Pignatelli et al\.,[2023](https://arxiv.org/html/2605.24005#bib.bib29);Chen et al\.,[2026](https://arxiv.org/html/2605.24005#bib.bib5)\)bottleneck\.

### 2\.4\.Joint Optimization and Credit Assignment

To achieve fine\-grained credit assignment within the mined trajectories, LC\-ERD ensures global reasoning soundness via local token\-level coordination\. Following theIndividual\-Global\-Max \(IGM\)\(Hong et al\.,[2022](https://arxiv.org/html/2605.24005#bib.bib13)\)principle, we reformulate the generation process as a collaborative game among sequential step\-agents\. This allows us to decompose the sparse global utilityQtotQ\_\{\\text\{tot\}\}into dense, step\-wise utilities, effectivelymining the contributionof each intermediate token to the final logical success:

\(10\)Qtot​\(τ\)=∑h=1HQh​\(sh,ah;θ\),Q\_\{\\text\{tot\}\}\(\\tau\)=\\sum\_\{h=1\}^\{H\}Q\_\{h\}\(s\_\{h\},a\_\{h\};\\theta\),whereQhQ\_\{h\}represents the credit assigned to stephh\. The joint objective functionℒtotal\\mathcal\{L\}\_\{\\text\{total\}\}is optimized through a dual\-stream training paradigm \(Figure[2](https://arxiv.org/html/2605.24005#S2.F2)c\):

\(11\)ℒtotal​\(θ\)=ℒTD​\(θ\)\+λ​∑h=1H𝒟indiv​\(sh,θ\),\\mathcal\{L\}\_\{\\text\{total\}\}\(\\theta\)=\\mathcal\{L\}\_\{\\text\{TD\}\}\(\\theta\)\+\\lambda\\sum\_\{h=1\}^\{H\}\\mathcal\{D\}\_\{\\text\{indiv\}\}\(s\_\{h\},\\theta\),whereℒTD​\(θ\)\\mathcal\{L\}\_\{\\text\{TD\}\}\(\\theta\)is the temporal difference loss for global value estimation; andλ\\lambdais a hyperparameter\. The global loss is formulated as:

\(12\)ℒTD​\(θ\)=\\displaystyle\\mathcal\{L\}\_\{\\text\{TD\}\}\(\\theta\)=𝔼τ\[\(RL​C−E​R​D\+γmaxa′Qθ−\(sh\+1,a′\)\\displaystyle\\mathbb\{E\}\_\{\\tau\}\\left\[\\left\(R\_\{LC\-ERD\}\+\\gamma\\max\_\{a^\{\\prime\}\}Q\_\{\\theta^\{\-\}\}\(s\_\{h\+1\},a^\{\\prime\}\)\\right\.\\right\.−Qθ\(sh,ah\)\)2\],\\displaystyle\\left\.\\left\.\-Q\_\{\\theta\}\(s\_\{h\},a\_\{h\}\)\\right\)^\{2\}\\right\],whereθ−\\theta^\{\-\}denotes the parameters of a target network\. Simultaneously, the individual consistency loss𝒟indiv\\mathcal\{D\}\_\{\\text\{indiv\}\}forces compliance:

\(13\)𝒟indiv​\(sh,θ\)=\\displaystyle\\mathcal\{D\}\_\{\\text\{indiv\}\}\(s\_\{h\},\\theta\)=∑a∈𝒱πL​L​E\(a\|sh\)⋅∥Logitsθ\\displaystyle\\sum\_\{a\\in\\mathcal\{V\}\}\\pi\_\{LLE\}\(a\|s\_\{h\}\)\\cdot\\\|\\text\{Logits\}\_\{\\theta\}−LogitsL​L​E∥22,\\displaystyle\-\\text\{Logits\}\_\{LLE\}\\\|\_\{2\}^\{2\},where𝒟indiv\\mathcal\{D\}\_\{\\text\{indiv\}\}serves as a logic\-consistent regularization term that penalizes the structural discrepancy between the current policy’s logits and the latent expert manifold, thereby enforcing high\-fidelity reasoning transitions and preventing the self\-evolution process from drifting into hallucinated mimetic patterns\. Finally, we update the target network:

\(14\)θ−←ξ​θ\+\(1−ξ\)​θ−,\\theta^\{\-\}\\leftarrow\\xi\\theta\+\(1\-\\xi\)\\theta^\{\-\},whereξ≪1\\xi\\ll 1is the interpolation factor\. Algorithm[1](https://arxiv.org/html/2605.24005#alg1)summarizes the workflow\.

Remark 3: Individual\-Global\-Max Consistency\. To ensure that token\-level agentic decisions align with the global reasoning trajectory, LC\-ERD satisfies the IGM principle\. The mathematical proof that our potential\-based reward decomposition preserves the optimal policy set and facilitates dense credit assignment is provided in Appendix A\.3\.

Algorithm 1LC\-ERD: Logic\-Consistent Reward Decomposition1:Input:Dataset

𝒟x\\mathcal\{D\}\_\{x\}, Base model

πbase\\pi\_\{\\text\{base\}\}, Width

KK, Coefficients

β,λ,ξ\\beta,\\lambda,\\xi
2:Output:Well\-aligned reasoning policy

πθ\\pi\_\{\\theta\}
3:foreach training iteration

t=1,…,Tt=1,\\dots,Tdodo

4:Sample a batch of queries

\{xi\}i=1B\\\{x\_\{i\}\\\}\_\{i=1\}^\{B\}from

𝒟x\\mathcal\{D\}\_\{x\}
5:Phase 1: Weighted LLE Discovery

6:

𝒯∗←\{τ∼πbase​\(x\)∣Ver​\(τ\)=1\}\\mathcal\{T\}^\{\*\}\\leftarrow\\\{\\tau\\sim\\pi\_\{\\text\{base\}\}\(x\)\\mid\\text\{Ver\}\(\\tau\)=1\\\}⊳\\trianglerightKKsamplings

7:Compute confidence weights

ω​\(τ\)\\omega\(\\tau\)⊳\\trianglerightEq\. 4

8:

πL​L​E←WeightedConsensus​\(𝒯∗,ω\)\\pi\_\{LLE\}\\leftarrow\\text\{WeightedConsensus\}\(\\mathcal\{T\}^\{\*\},\\omega\)⊳\\trianglerightEq\. 5

9:Phase 2: Reward Decomposition

10:foreach reasoning step

h∈\[1,H\]h\\in\[1,H\]in

𝒯∗\\mathcal\{T\}^\{\*\}dodo

11:

r~h←InverseBellman​\(πbase,Vh,Vh\+1\)\\tilde\{r\}\_\{h\}\\leftarrow\\text\{InverseBellman\}\(\\pi\_\{\\text\{base\}\},V\_\{h\},V\_\{h\+1\}\)⊳\\trianglerightEq\. 3

12:

Φh←VariationalPotential​\(πθ,πL​L​E,β\)\\Phi\_\{h\}\\leftarrow\\text\{VariationalPotential\}\(\\pi\_\{\\theta\},\\pi\_\{LLE\},\\beta\)⊳\\trianglerightEq\. 6

13:

RL​C−E​R​D←r~h\+γ​Φh\+1−ΦhR\_\{LC\-ERD\}\\leftarrow\\tilde\{r\}\_\{h\}\+\\gamma\\Phi\_\{h\+1\}\-\\Phi\_\{h\}⊳\\trianglerightEq\. 8

14:endfor

15:Phase 3: Joint Optimization

16:

ℒT​D←\(RL​C−E​R​D\+γ​Qθ−−Qθ\)2\\mathcal\{L\}\_\{TD\}\\leftarrow\(R\_\{LC\-ERD\}\+\\gamma Q\_\{\\theta^\{\-\}\}\-Q\_\{\\theta\}\)^\{2\}⊳\\trianglerightEq\. 12

17:

𝒟i​n​d​i​v←ConsistencyLoss​\(πθ,πL​L​E\)\\mathcal\{D\}\_\{indiv\}\\leftarrow\\text\{ConsistencyLoss\}\(\\pi\_\{\\theta\},\\pi\_\{LLE\}\)⊳\\trianglerightEq\. 13

18:

θ←θ−η​∇θ\(ℒT​D\+λ​∑𝒟i​n​d​i​v\)\\theta\\leftarrow\\theta\-\\eta\\nabla\_\{\\theta\}\(\\mathcal\{L\}\_\{TD\}\+\\lambda\\sum\\mathcal\{D\}\_\{indiv\}\)⊳\\trianglerightEq\. 11

19:Phase 4: Target Network Update

20:

θ−←ξ​θ\+\(1−ξ\)​θ−\\theta^\{\-\}\\leftarrow\\xi\\theta\+\(1\-\\xi\)\\theta^\{\-\}⊳\\trianglerightEq\. 14

21:endfor

### 2\.5\.Computational Complexity and Efficiency

To evaluate the scalability of LC\-ERD, we provide a formal analysis of its computational overhead during both training and inference\. Unlike traditional monolithic reward models, the primary cost of LC\-ERD stems from theKK\-path parallel sampling required for LLE discovery\. We formulate the total training complexity per iteration𝒞total\\mathcal\{C\}\_\{\\text\{total\}\}as:

\(15\)𝒞total=𝒪​\(B⋅\(K⋅H¯⋅\|𝒱\|\+H¯⋅Θ\)\),\\mathcal\{C\}\_\{\\text\{total\}\}=\\mathcal\{O\}\\left\(B\\cdot\\left\(K\\cdot\\bar\{H\}\\cdot\|\\mathcal\{V\}\|\+\\bar\{H\}\\cdot\\Theta\\right\)\\right\),whereBBis the training batch size;KKdenotes the sampling width for consensus discovery;H¯\\bar\{H\}represents the average trajectory length in reasoning steps;\|𝒱\|\|\\mathcal\{V\}\|is the vocabulary size; andΘ\\Thetacorresponds to the total number of trainable parameters in the backbone model\.

To ensure memory efficiency during the alignment phase, we introduce a top\-kklogit compression strategy for the LLE expert manifold\. The peak memory consumptionℳpeak\\mathcal\{M\}\_\{\\text\{peak\}\}is bounded by:

\(16\)ℳpeak=𝒪​\(B⋅H¯⋅\(k⋅prec\+ω\)\),\\mathcal\{M\}\_\{\\text\{peak\}\}=\\mathcal\{O\}\\left\(B\\cdot\\bar\{H\}\\cdot\(k\\cdot\\text\{prec\}\+\\omega\)\\right\),wherek≪\|𝒱\|k\\ll\|\\mathcal\{V\}\|is the sparsity coefficient for retained logits;precdenotes the bit\-precision of the tensors; andω\\omegarepresents the memory overhead of the confidence\-weighted trajectory set\. This optimization ensures that LC\-ERD maintains a memory footprint comparable to standard RLHF\(Dai et al\.,[2023](https://arxiv.org/html/2605.24005#bib.bib7)\)pipelines\.

### 2\.6\.Distributed Implementation: Logit Caching and Parallelization

The practical implementation of LC\-ERD leverages a Distributed Logit Caching \(DLC\) mechanism to decouple the generative exploration from the gradient optimization\. We define the cached logic state𝒵\\mathcal\{Z\}for a given queryqqas:

\(17\)𝒵q=\{ϕ​\(sh\(k\),ah\)∣τ\(k\)∈𝒯∗,h∈\[1,H\]\},\\mathcal\{Z\}\_\{q\}=\\\{\\phi\(s\_\{h\}^\{\(k\)\},a\_\{h\}\)\\mid\\tau^\{\(k\)\}\\in\\mathcal\{T\}^\{\*\},h\\in\[1,H\]\\\},whereϕ​\(⋅\)\\phi\(\\cdot\)denotes the unnormalized logits retrieved during the forward pass;𝒯∗\\mathcal\{T\}^\{\*\}is the set of verified logic\-consistent trajectories; andHHis the total step count\. The individual consistency loss𝒟indiv\\mathcal\{D\}\_\{\\text\{indiv\}\}is then computed asynchronously:

\(18\)∇θ𝒟indiv=𝔼s∼𝒵q​\[∇θ‖Logitsθ​\(s\)−Logits𝒵​\(s\)‖22\],\\nabla\_\{\\theta\}\\mathcal\{D\}\_\{\\text\{indiv\}\}=\\mathbb\{E\}\_\{s\\sim\\mathcal\{Z\}\_\{q\}\}\\left\[\\nabla\_\{\\theta\}\\\|\\text\{Logits\}\_\{\\theta\}\(s\)\-\\text\{Logits\}\_\{\\mathcal\{Z\}\}\(s\)\\\|\_\{2\}^\{2\}\\right\],whereLogitsθ​\(s\)\\text\{Logits\}\_\{\\theta\}\(s\)is the prediction of the current policy; andLogits𝒵​\(s\)\\text\{Logits\}\_\{\\mathcal\{Z\}\}\(s\)represents the target distribution elicited from the expert manifold\.

To stabilize training under high logical variance, we introduce Gradient Clipping\(Tang et al\.,[2025](https://arxiv.org/html/2605.24005#bib.bib32)\)for Logic Drift \(GC\-LD\)\. The effective gradientgeffg\_\{\\text\{eff\}\}is normalized according to the logical energy gap:

\(19\)geff=min⁡\(1,η‖∇θℒtotal‖\)⋅∇θ\(ℒTD\+λ​𝒟indiv\),g\_\{\\text\{eff\}\}=\\min\\left\(1,\\frac\{\\eta\}\{\\\|\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\text\{total\}\}\\\|\}\\right\)\\cdot\\nabla\_\{\\theta\}\(\\mathcal\{L\}\_\{\\text\{TD\}\}\+\\lambda\\mathcal\{D\}\_\{\\text\{indiv\}\}\),whereη\\etais the maximum gradient norm threshold;λ\\lambdais the trade\-off coefficient; andℒtotal\\mathcal\{L\}\_\{\\text\{total\}\}is the joint objective defined in Eq\. \(13\)\.

### 2\.7\.Theoretical Analysis: Sub\-optimality Bound and Convergence

We provide a theoretical guarantee for the self\-evolution path of LC\-ERD\. Letℛ​\(π\)\\mathcal\{R\}\(\\pi\)denote the cumulative regret of a policyπ\\piwith respect to the true logical manifold\. The logic\-regularized sub\-optimality gap at iterationTTis defined asΔT\\Delta\_\{T\}:

\(20\)ΔT=ℛ​\(π∗\)−ℛ​\(πθT\),\\Delta\_\{T\}=\\mathcal\{R\}\(\\pi^\{\*\}\)\-\\mathcal\{R\}\(\\pi\_\{\\theta\}^\{T\}\),whereπ∗\\pi^\{\*\}is the optimal logic\-consistent policy\. Under the VLP shaping defined in Section 3\.3, the convergence rate satisfies the following recurrence:

\(21\)ΔT\+1≤\(1−δ\)​ΔT\+ϵapprox,\\Delta\_\{T\+1\}\\leq\(1\-\\delta\)\\Delta\_\{T\}\+\\epsilon\_\{\\text\{approx\}\},
whereδ∈\(0,1\)\\delta\\in\(0,1\)is the Logic\-Consistency Gain; andϵapprox\\epsilon\_\{\\text\{approx\}\}represents the approximation error of the LLE discovery process\. The gainδ\\deltais explicitly tied to the logic\-regularization coefficientβ\\beta:

\(22\)δ∝β⋅minh⁡\(1−DKL​\(πθ∥πL​L​E\)\),\\delta\\propto\\beta\\cdot\\min\_\{h\}\\left\(1\-\\text\{D\}\_\{\\text\{KL\}\}\(\\pi\_\{\\theta\}\\parallel\\pi\_\{LLE\}\)\\right\),where the term in parentheses represents the minimum alignment degree across step\-wise agentic decisions\. This ensures that LC\-ERD provides a provably more stable convergence path by shifting the sub\-optimality bound from quadratic𝒪​\(H2\)\\mathcal\{O\}\(H^\{2\}\)to linear𝒪​\(H\)\\mathcal\{O\}\(H\), effectively anchoring the policy to the logical manifold\.

Remark 4: Contraction of Reasoning Sub\-optimality\. A critical advantage of LC\-ERD is the mitigation of compounding errors\. While standard imitation learning suffers from a quadratic error bound𝒪​\(H2\)\\mathcal\{O\}\(H^\{2\}\), our framework achieves a provably superior linear bound𝒪​\(H\)\\mathcal\{O\}\(H\)\. The complete formal analysis of this sub\-optimality contraction is detailed in Appendix A\.4\.

## 3\.Experiments

Table 1\.Long\-horizon planning and reasoning\.We evaluate all methods using Qwen2\.5\-72B across 16 tasks, focusing on MATH Level\-5 to assess long\-chain credit assignment\. We report Success Rate \(SR %\) and the Logic Consistency Score \(LCS\), which quantifies reasoning path validity via alignment with the latent expert manifold\. Training efficiency \(VRAM, relative time\) is included to illustrate resource\-performance trade\-offs\.†\\daggerdenotes multi\-agent or MARL frameworks; bold indicates the highest score\. Superscripts denote the improvement relative to the Base model \(BlueandRed\)\.We conduct a large\-scale, multi\-dimensional empirical evaluation ofLC\-ERDto verify its efficacy in fostering robust self\-evolution\. Our experiments are structured to provide a comprehensive deconstruction of how consistency\-regulated decomposition addresses the fundamental inhibitors of reasoning\. Specifically, we aim to answer four critical Research Questions:\(RQ1\)How does LC\-ERD compare to SOTA baselines across diverse reasoning and codings?\(RQ2\)Can our framework effectively resolve the credit assignment bottleneck in long\-chain trajectories\(Zou et al\.,[2025](https://arxiv.org/html/2605.24005#bib.bib51)\)via IGM\-consistency?\(RQ3\)What are the hierarchical contributions of core components, and how do they align with our theoretical claims?\(RQ4\)Does LC\-ERD exhibit superior scaling properties, inductive generalization, and resilience against reward hacking?

### 3\.1\.Experiment Setup

Benchmarks and Metric Suites\.We utilize 16 benchmarks across four distinct cognitive domains: \(I\)Complex Mathematical Deduction: MATH \(Level 4–5\), GSM8K, and SVAMP\. We isolate MATH L5 to test credit assignment in trajectories exceeding 25 steps\. \(II\)Symbolic and Logical Reasoning: BBH \(Hard subset\), LogiQA\-2, and ReClor\. \(III\)Code Synthesis and Algorithmic Logic: HumanEval\(Chen,[2021](https://arxiv.org/html/2605.24005#bib.bib3)\), MBPP\+, and DS\-1000\. \(IV\)Expert\-Level Generalization: MedQA\(Yao et al\.,[2024](https://arxiv.org/html/2605.24005#bib.bib43);Yang et al\.,[2025](https://arxiv.org/html/2605.24005#bib.bib42)\)\(USMLE\), PubMedQA, and LawBench\(Fei et al\.,[2024](https://arxiv.org/html/2605.24005#bib.bib11)\)\. These evaluate the failure of endogenous signals to generalize across specialized domains\. We reportSuccess Rate \(SR %\)and introduce theLogic Consistency Score \(LCS\), which quantifies the structural validity of the reasoning path by measuring intermediate step alignment with the latent expert manifold\.

Baselines and Comparative Systems\.We categorize 18 competitive baselines into four groups: \(1\)Direct Preference Alignment: DPO\(Xu et al\.,[2024](https://arxiv.org/html/2605.24005#bib.bib41);Zhong et al\.,[2024](https://arxiv.org/html/2605.24005#bib.bib48)\), SimPO\(Meng et al\.,[2024](https://arxiv.org/html/2605.24005#bib.bib26)\), and LPO\. \(2\)Iterative Self\-Evolution: STaR, Self\-Reward, and Iterative\-DPO\. \(3\)Endogenous Reward Models\(Li et al\.,[2025b](https://arxiv.org/html/2605.24005#bib.bib22)\): EndoRM, InfiGUI\-R1\(Liu et al\.,[2025](https://arxiv.org/html/2605.24005#bib.bib24)\), and Gen\-Verifier\(Chen et al\.,[2025](https://arxiv.org/html/2605.24005#bib.bib4)\)\. \(4\)Search\-based & Agentic Frameworks: MaAS, AFlow, and PRM\. To ensure fairness, all models utilizeQwen2\.5\-72BorLlama\-3\.1\-70Bas backbones\.

Implementation Details\.For LC\-ERD, we set the sampling widthK=64K=64forLatent Logic Expertise \(LLE\)discovery\. The logic\-regularization coefficientβ\\betais 0\.1, and the discount factorγ\\gammais 0\.95\. Training is conducted on 8×\\timesNVIDIA H100 GPUs usingDistributed Logit Caching \(DLC\)to decouple generative exploration from gradient optimization\.

### 3\.2\.Main Results: Performance, Logic, and Scaling Dynamics

Table[3](https://arxiv.org/html/2605.24005#S3)presents the results on complex reasoning benchmarks\. LC\-ERD achieves SOTA performance, delivering an average improvement of\+18\.1%over the base SFT model\.

Obs\. 1: Nonlinear Scaling of Logic Consistency vs\. Success Rate\.As evidenced in Table[3](https://arxiv.org/html/2605.24005#S3), LC\-ERD demonstrates a unique nonlinear scaling property\. While standard alignment methods \(DPO, SimPO\) show a marginal accuracy gain, theirLCSremains stagnant\. This disparity confirms the existence of themimetic bias: models often “hallucinate” a correct final answer through a flawed reasoning path\. In contrast, LC\-ERD yields a significant\+18\.1%\+18\.1\\%SR boost on MATH L5, with a corresponding LCS of 0\.912\. This suggests that ourVariational Logic Potential \(VLP\)acts as a structural filter that penalizes logically improbable transitions, forcing the model to align with the expert manifold rather than surface\-level patterns\.

Obs\. 2: Synergistic Scaling with Latent Expert Discovery\.The gap between LC\-ERD and DPO widens as model size increases \(\+14\.3%\+14\.3\\%at 72B vs\+18\.5%\+18\.5\\%at Dual\-72B\)\.

This validatesTheorem 2: larger backbones possess more refinedLatent Logic Expertise \(LLE\), allowing LC\-ERD to elicit higher\-fidelity consensus signals\. Notably, LC\-ERD maintains a superior efficiency\-to\-accuracy ratio, consuming25%25\\%less training time than search\-heavy systems like AFlow due to ourDistributed Logit Caching \(DLC\)mechanism\.

Table 2\.Results on high\-precision professional reasoning\.We report Accuracy \(%\) across six categories in the Logic\-Expert\-Pro benchmark to evaluate cross\-domain generalization and resilience against pre\-training bias\. The evaluation compares LC\-ERD with proprietary and open\-source models to demonstrate its ability to assign credit in specialized fields like medicine and law\.†\\daggerdenotes multi\-agent or MARL frameworks; bold indicates the highest score\. Superscripts denote the improvement relative to the Base model \(Redfor LC\-ERD\)\.### 3\.3\.Expert Domains: Avoiding Negative Transfer in Specialized Tasks

To address Challenge \(III\), we report accuracy on theLogic\-Expert\-Probenchmark in Table[2](https://arxiv.org/html/2605.24005#S3.T2)\.

![Refer to caption](https://arxiv.org/html/2605.24005v1/x3.png)Figure 3\.Evolution of Logical Discriminability\.The density plots illustrate the progressive disentanglement of reasoning steps during training\. \(Left\)Early Stage \(Initialization\):The model exhibitsHigh EntropyandChaos, with significant overlap between valid and invalid steps, indicating a strong mimetic bias\. \(Middle\)Mid Stage \(Adaptation\):A bimodal structure begins to emerge, showing the initial separation of logical potential\. \(Right\)Late Stage \(Convergence\):The distribution achieves aDisentangled / Orderstate\. Valid steps cluster in the high potential regime \(Φ^≈0\.9\\hat\{\\Phi\}\\approx 0\.9\), while invalid steps are suppressed to the low potential basin \(Φ^≈0\.2\\hat\{\\Phi\}\\approx 0\.2\), resulting in a maximized Jensen Shannon Divergence \(JSD\)\.Obs\. 3: IGM\-Consistency Resolves Domain\-Specific Credit Assignment\.In categories likeMedicalandLegal, where reasoning chains are non\-linear, LC\-ERD achieves an average boost of\+15\.3%\+15\.3\\%over MaAS\. This validates ourMulti\-Agent Value Decomposition: by treating each reasoning step as a collaborative agent governed by the IGM principle, the framework accurately identifies significant logical transitions often buried in pre\-training noise\. Consistent with observations in NexusBind regarding structural data fidelity, LC\-ERD prioritizes logic\-structural consistency over statistical volume, thereby avoiding thenegative transfertypically induced by massive but noisy endogenous signals\.

### 3\.4\.Hierarchical Ablation and Mechanistic Deep Dive

We decompose LC\-ERD into its hierarchical components in Table[3](https://arxiv.org/html/2605.24005#S3.T3)\.

Table 3\.Main Ablation Study of LC\-ERD components on MATH and Expert\-Logic benchmarks\.Δ\\Deltaindicates the performance drop from the full model\.Method VariantMATH \(SR\)Δ\\DeltaExpert \(Acc\.\)Δ\\DeltaLC\-ERD \(Full\)65\.2\-78\.0\-\\rowcolorsoftblueRepresentation \(C1\):\(a\) w/o Variational Potential58\.4\(–6\.8\)72\.1\(–5\.9\)\(b\) w/o Latent Expert Manifold61\.2\(–4\.0\)68\.5\(–9\.5\)\\rowcolorsoftblueArchitecture \(C2\):\(c\) Single\-Agent \(No Decomp\)52\.5\(–12\.7\)72\.9\(–5\.1\)\\rowcolorsoftblueRewards \(C4\):\(d\) w/o Step\-level IGM54\.8\(–10\.4\)74\.3\(–3\.7\)\\rowcolorsoftblueRobustness \(C5\):\(e\) w/ 30% Reasoning Noise62\.4\(–2\.8\)74\.5\(–3\.5\)Obs\. 4: Severe Performance Degradation without IGM Decomposition\.Removing theStep\-level IGM\(Variant d\) results in a severe performance collapse in MATH \(−10\.4%\-10\.4\\%\)\. Mechanistic analysis reveals that without decomposition, the reward signal undergoes “terminal dilution”—as the trajectory length increases, the gradient assigned to early steps becomes uninformative\. The IGM protocol provides dense, token\-level supervision that stabilizes the Temporal Difference \(TD\) error, solving the credit assignment bottleneck\.

Obs\. 5: Evolution of Logical Entropy and Convergence Stability\.Beyond SR, we analyze theLogical Entropy\(Ellerman,[2013](https://arxiv.org/html/2605.24005#bib.bib9)\)\(ℋl​o​g​i​c\\mathcal\{H\}\_\{logic\}\) of the policy across iterations\. While standard DPO leads to rapid collapse of token diversity \(mimetic convergence\), LC\-ERD maintains higher entropy in early phases before converging to a structured logic manifold\. This behavior validates ourLLE discovery protocol: by marginalizing overKKpaths, the framework prevents the model from overfitting to idiosyncratic reasoning chains, resulting in a more robust sub\-optimality contraction rate\(1−δ\)\(1\-\\delta\)\.

![Refer to caption](https://arxiv.org/html/2605.24005v1/x4.png)Figure 4\.Mining the “Aha\!” Moment\.We visualize the step\-wise Reward Attribution \(rtr\_\{t\}\) for a complex inequality problem\. \(a\)GRPO Baseline: Assigns flat credit across the chain\. It reinforces a “Lucky Guess” where two errors cancel out \(\+⁣×⁣−⁣=⁣−\+\\times\-=\-\)\. \(b\)LC\-ERD: The VLP detects aLogic Dipat Step 3 \(Calculus Error\), penalizing the trajectory despite the correct final answer\. In a subsequent successful run, it identifies aCentric Step\(Step 5: “AM\-GM Inequality”\) with a high potential spike \(Φ^=0\.92\\hat\{\\Phi\}=0\.92\), correctly attributing the solution’s success to this critical logical breakthrough\.### 3\.5\.Inductive Logic and Defense against Reward Hacking

Obs\. 6: Zero\-Shot Logical Extrapolation\.We perform a “leave\-one\-pattern\-out” experiment, excluding specific logical constructs from LLE discovery\. Remarkably, LC\-ERD retains88\.4%88\.4\\%efficacy, suggesting that theVariational Logic Potential \(VLP\)learns a generalized “energy basin” of deductive validity rather than specific patterns\. This inductive strength proves that LC\-ERD fosters genuine self\-evolution\.

Obs\. 7: VLP as a Robustness Barrier to Reward Hacking\.Standard RL models often exhibit high correlation between chain length and reward \(r=0\.74r=0\.74\), incentivizing verbosity\. LC\-ERD shows near\-zero correlation \(r=0\.12r=0\.12\)\. By implementing the IGM principle, each token’s credit is strictly tied to its contribution to global logic\-consistency, immunizing the model against shortcut\-learning and reinforcing the structural integrity of the evolution path\.

### 3\.6\.The Dynamic Evolution of Logic Manifold

To rigorously verify that LC\-ERD effectively “mines” structural logic during the self\-evolution process, we visualize the training dynamics of theVariational Logic Potential \(VLP\)\. We track the density estimation of VLP scoresΦ^\\hat\{\\Phi\}for both valid and hallucinated reasoning steps across three training stages, as shown inFig\.[3](https://arxiv.org/html/2605.24005#S3.F3)\.

From Chaos to Order\.The evolution of the logic manifold undergoes a clear phase transition:

- •Initialization \(Mimetic Chaos\):In the early stage, the VLP distribution for valid and invalid steps is nearly indistinguishable \(High Overlap\), confirming that the base model relies on surface\-level imitation rather than logical validity\.
- •Adaptation \(Emerging Structure\):During the mid\-stage adaptation, the consistency regularization begins to pull the distributions apart\. The VLP acts as a soft filter, gradually increasing the energy gap between sound deduction and hallucination\.
- •Convergence \(Logical Disentanglement\):In the late stage, we observe a decisive separation\. The VLP successfully identifies theLatent Logic Expertise, assigning high potential to valid steps while actively penalizing logical drifts\. This structural disentanglement explains the superior robustness of LC\-ERD against reward hacking, as the model learns to optimize the underlying potential rather than just the terminal token match\.

### 3\.7\.Qualitative Mining Analysis: Significant Step Discovery

To demonstrate how LC\-ERD resolves theCredit Assignment Bottleneckin complex, multi\-step reasoning, we present a “Significant Step Analysis” on a hard instance from the MATH benchmark \(Algebra, Level 5\)\.

Scenario: The “Two\-Wrongs\-Make\-A\-Right” Trap\.Consider the complex inequality problem presented inFigure[4](https://arxiv.org/html/2605.24005#S3.F4)\. The baseline model \(GRPO\) generates a trajectory where it makes a sign error in Step 3 \(−⁣→⁣\+\-\\rightarrow\+\) and a factorization error in Step 6, which coincidently lead to the correct numeric answerx=4x=4\.

- •Baseline Failure \(Figure[4](https://arxiv.org/html/2605.24005#S3.F4)a\):GRPO sees ‘Answer: 4‘ \(Correct\) and assigns a positive reward\+1\+1to the entire chain\. Thisnegative transferreinforces the erroneous logic steps, as the monolithic signal fails to penalize the intermediate sign error\.
- •LC\-ERD Correction \(Figure[4](https://arxiv.org/html/2605.24005#S3.F4)b\):Our framework’s IGM decomposition computes the stepwise potential differenceΔ​Φ\\Delta\\Phi\. At Step 3, the potential drops sharply \(Δ​Φ3=−2\.4\\Delta\\Phi\_\{3\}=\-2\.4\), detecting the divergence from the latent expert manifold\. The cumulative reward becomes negative, effectively “rejecting” this correct trajectory\.

Mining the Significant Step\.In a corrected trajectory, LC\-ERD identifies Step 5 \(application of the AM\-GM inequality\) as aSignificant Step\. The potentialΦ\\Phijumps from0\.40\.4to0\.920\.92immediately after this step\. This signals that the model has crossed a “logical event horizon”, after which the solution becomes inevitable\. By assigning dense credit to this specific center, LC\-ERD mines thecausal mechanismof the solution, rather than just the correlation with the final answer\.

## 4\.Related Work

The evolution of reasoning in Large Language Models \(LLMs\) has primarily been driven by alignment techniques and reward modeling\. We categorize the most relevant works into three streams and contrast them withLC\-ERD\.

Endogenous Self\-Evolution\.Techniques likeSTaRandSelf\-Reward\(Yuan et al\.,[2024](https://arxiv.org/html/2605.24005#bib.bib45)\)enable models to iteratively refine reasoning by training on self\-generated correct solutions\. Recent advancements such asGRPO\(Wang et al\.,[2025a](https://arxiv.org/html/2605.24005#bib.bib36)\)andInfiGUI\(Liu et al\.,[2025](https://arxiv.org/html/2605.24005#bib.bib24)\)utilize group\-relative outcomes to stabilize this process without a value function\. However, these methods typically treat the reward as a monolithic global signal, making them susceptible to themimetic biaswhere the model mimics correct\-looking but logically flawed patterns\. LC\-ERD differentiates itself by performingstep\-level reward decompositionvia theVariational Logic Potential \(VLP\), which explicitly quantifies structural soundness rather than just surface\-level correctness\.

Reward Decomposition and Credit Assignment\.Attributing a global terminal reward to specific intermediate steps is a long\-standing challenge in long\-chain reasoning\(Yeo et al\.,[2025](https://arxiv.org/html/2605.24005#bib.bib44)\)\. Process Reward Models \(PRMs\)\(Lightman et al\.,[2023](https://arxiv.org/html/2605.24005#bib.bib23)\)provide granular feedback but rely on expensive step\-wise human annotations or external LLM judges\. While search\-based frameworks likeMaASandAFlow\(Zhang et al\.,[2024](https://arxiv.org/html/2605.24005#bib.bib47)\)explore multi\-path trajectories\(Cheng et al\.,[2021](https://arxiv.org/html/2605.24005#bib.bib6)\), they often suffer from “terminal dilution” in extremely long reasoning chains\. LC\-ERD is the first to implement theIndividual\-Global\-Max \(IGM\)principle from multi\-agent RL into LLM self\-alignment, providing an automated, dense reward decomposition that is mathematically proven to resolve the credit assignment bottleneck\.

## 5\.Conclusion

In this paper, we introducedLC\-ERD, a logic\-consistent reward decomposition framework that holistically aligns LLM reasoning trajectories\. By bridging variational consensus with multi\-agent value decomposition, LC\-ERD effectively addresses the challenges of mimetic bias and the credit assignment bottleneck\. Extensive experiments across 16 benchmarks demonstrate that our method surpasses existing automated systems in both efficiency and robustness\. By mining latent supervision from the model’s own distribution, LC\-ERD provides a scalable path forself\-evolving intelligence in data\-scarce domains, where high\-quality human annotation is structurally unavailable\.

## Acknowledgement

The research presented in this paper was partially supported by the Research Grants Council of the Hong Kong Special Administrative Region, China \(CUHK 2300246, RGC C1043\-24G\), \(CUHK 14203425, RGC GRF 2151317\), and CUHK 7010870\.

## References

- \(1\)
- Bozkurt et al\.\(2026\)Berk Bozkurt, Aditya Mahajan, Ashutosh Nayyar, and Yi Ouyang\. 2026\.Sub\-optimality bounds for certainty equivalent policies in partially observed systems\.*arXiv preprint arXiv:2602\.02814*\(2026\)\.
- Chen \(2021\)Mark Chen\. 2021\.Evaluating large language models trained on code\.*arXiv preprint arXiv:2107\.03374*\(2021\)\.
- Chen et al\.\(2025\)Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che\. 2025\.Towards reasoning era: A survey of long chain\-of\-thought for reasoning large language models\.*arXiv preprint arXiv:2503\.09567*\(2025\)\.
- Chen et al\.\(2026\)Yanyu Chen, Jiyue Jiang, Jiahong Liu, Yifei Zhang, Xiao Guo, and Irwin King\. 2026\.Trace: Trajectory\-aware comprehensive evaluation for deep research agents\. In*Proceedings of the ACM Web Conference 2026*\. 2524–2534\.
- Cheng et al\.\(2021\)Hao Cheng, Wentong Liao, Xuejiao Tang, Michael Ying Yang, Monika Sester, and Bodo Rosenhahn\. 2021\.Exploring dynamic context for multi\-path trajectory prediction\. In*2021 IEEE International Conference on Robotics and Automation \(ICRA\)*\. IEEE, 12795–12801\.
- Dai et al\.\(2023\)Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang\. 2023\.Safe rlhf: Safe reinforcement learning from human feedback\.*arXiv preprint arXiv:2310\.12773*\(2023\)\.
- Deng et al\.\(2025\)Wenlong Deng, Yushu Li, Boying Gong, Yi Ren, Christos Thrampoulidis, and Xiaoxiao Li\. 2025\.On grpo collapse in search\-r1: The lazy likelihood\-displacement death spiral\.*arXiv preprint arXiv:2512\.04220*\(2025\)\.
- Ellerman \(2013\)David Ellerman\. 2013\.An introduction to logical entropy and its relation to Shannon entropy\.121–145 pages\.
- Eysenbach and Levine \(2021\)Benjamin Eysenbach and Sergey Levine\. 2021\.Maximum entropy RL \(provably\) solves some robust RL problems\.*arXiv preprint arXiv:2103\.06257*\(2021\)\.
- Fei et al\.\(2024\)Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Alan Huang, Songyang Zhang, Kai Chen, Zhixin Yin, Zongwen Shen, Jidong Ge, and Vincent Ng\. 2024\.Lawbench: Benchmarking legal knowledge of large language models\. In*Proceedings of the 2024 conference on empirical methods in natural language processing*\. 7933–7962\.
- Garcia and Rachelson \(2013\)Frédérick Garcia and Emmanuel Rachelson\. 2013\.Markov decision processes\.*Markov Decision Processes in Artificial Intelligence*\(2013\), 1–38\.
- Hong et al\.\(2022\)Yitian Hong, Yaochu Jin, and Yang Tang\. 2022\.Rethinking individual global max in cooperative multi\-agent reinforcement learning\.*Advances in neural information processing systems*35 \(2022\), 32438–32449\.
- Ichihara et al\.\(2025\)Yuki Ichihara, Yuu Jinnai, Tetsuro Morimura, Mitsuki Sakamoto, Ryota Mitsuhashi, and Eiji Uchibe\. 2025\.Mo\-grpo: Mitigating reward hacking of group relative policy optimization on multi\-objective problems\.*arXiv preprint arXiv:2509\.22047*\(2025\)\.
- Jarboui and Perchet \(2021\)Firas Jarboui and Vianney Perchet\. 2021\.Offline inverse reinforcement learning\.*arXiv preprint arXiv:2106\.05068*\(2021\)\.
- Jaynes \(1982\)Edwin T Jaynes\. 1982\.On the rationale of maximum\-entropy methods\.*Proc\. IEEE*70, 9 \(1982\), 939–952\.
- Jiang et al\.\(2026\)Jiyue Jiang, Yanyu Chen, Peng Chen, Kai\-Chun Liu, Jingqi Zhou, Zheyong Zhu, He Hu, Fei Ma, Qi Tian, and Chuan Wu\. 2026\.A Principle\-Driven Adaptive Policy for Group Cognitive Stimulation Dialogue for Elderly with Cognitive Impairment\. In*AAAI Conference on Artificial Intelligence*\.[https://api\.semanticscholar\.org/CorpusID:286457185](https://api.semanticscholar.org/CorpusID:286457185)
- Jiang et al\.\(2025\)Jiyue Jiang, Yunke Li, Shiwei Cao, Yuheng Shan, Yuexing Liu, Tianyi Fei, Yule Yu, Yi Feng, Yu Li, Yixue Li, et al\.2025\.Artificial intelligence in bioinformatics: a survey\.*Briefings in Bioinformatics*26, 6 \(2025\), bbaf576\.
- Kang et al\.\(2025\)Hyeongyu Kang, Jaewoo Lee, Woocheol Shin, Kiyoung Om, and Jinkyoo Park\. 2025\.Diffusion Fine\-Tuning via Reparameterized Policy Gradient of the Soft Q\-Function\.*arXiv preprint arXiv:2512\.04559*\(2025\)\.
- Lansdell et al\.\(2019\)Benjamin James Lansdell, Prashanth Ravi Prakash, and Konrad Paul Kording\. 2019\.Learning to solve the credit assignment problem\.*arXiv preprint arXiv:1906\.00889*\(2019\)\.
- Li et al\.\(2025a\)Xuying Li, Zhuo Li, Yuji Kosuga, and Victor Bian\. 2025a\.Optimizing Safe and Aligned Language Generation: A Multi\-Objective GRPO Approach\.*arXiv preprint arXiv:2503\.21819*\(2025\)\.
- Li et al\.\(2025b\)Yi\-Chen Li, Tian Xu, Yang Yu, Xuqin Zhang, Xiong\-Hui Chen, Zhongxiang Ling, Ningjing Chao, Lei Yuan, and Zhi\-Hua Zhou\. 2025b\.Generalist Reward Models: Found Inside Large Language Models\.*arXiv preprint arXiv:2506\.23235*\(2025\)\.
- Lightman et al\.\(2023\)Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe\. 2023\.Let’s verify step by step\. In*The Twelfth International Conference on Learning Representations*\.
- Liu et al\.\(2025\)Yuhang Liu, Pengxiang Li, Congkai Xie, Xavier Hu, Xiaotian Han, Shengyu Zhang, Hongxia Yang, and Fei Wu\. 2025\.Infigui\-r1: Advancing multimodal gui agents from reactive actors to deliberative reasoners\.*arXiv preprint arXiv:2504\.14239*\(2025\)\.
- Mayilvahanan et al\.\(2025\)Prasanna Mayilvahanan, Ricardo Dominguez\-Olmedo, Thaddäus Wiedemer, and Wieland Brendel\. 2025\.MATH\-Beyond: A Benchmark for RL to Expand Beyond the Base Model\.*arXiv preprint arXiv:2510\.11653*\(2025\)\.
- Meng et al\.\(2024\)Yu Meng, Mengzhou Xia, and Danqi Chen\. 2024\.Simpo: Simple preference optimization with a reference\-free reward\.*Advances in Neural Information Processing Systems*37 \(2024\), 124198–124235\.
- Ng and Russell \(2000\)Andrew Y\. Ng and Stuart J\. Russell\. 2000\.Algorithms for inverse reinforcement learning\.\. In*Icml*, Vol\. 1\. 2\.
- Pang and Jin \(2025\)Lei Pang and Ruinan Jin\. 2025\.On the theory and practice of grpo: A trajectory\-corrected approach with fast convergence\.*arXiv preprint arXiv:2508\.02833*\(2025\)\.
- Pignatelli et al\.\(2023\)Eduardo Pignatelli, Johan Ferret, Matthieu Geist, Thomas Mesnard, Hado van Hasselt, Olivier Pietquin, and Laura Toni\. 2023\.A survey of temporal credit assignment in deep reinforcement learning\.*arXiv preprint arXiv:2312\.01072*\(2023\)\.
- Shao et al\.\(2024\)Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and Daya Guo\. 2024\.Deepseekmath: Pushing the limits of mathematical reasoning in open language models\.*arXiv preprint arXiv:2402\.03300*\(2024\)\.
- Sun et al\.\(2023\)Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan\. 2023\.Principle\-driven self\-alignment of language models from scratch with minimal human supervision\.*Advances in Neural Information Processing Systems*36 \(2023\), 2511–2565\.
- Tang et al\.\(2025\)Shengyuan Tang, Linwan Zhang, Shengzhe Xu, Xinyue Zeng, Peng Hu, Xinyi Gong, and Manzhou Li\. 2025\.Communication\-Efficient Federated Optimization with Gradient Clipping and Attention Aggregation for Data Analytics and Prediction\.*Electronics*14, 23 \(2025\), 4778\.
- Van Erven and Harremos \(2014\)Tim Van Erven and Peter Harremos\. 2014\.Rényi divergence and Kullback\-Leibler divergence\.*IEEE Transactions on Information Theory*60, 7 \(2014\), 3797–3820\.
- Vinhas et al\.\(2024\)Adriano Vinhas, João Correia, and Penousal Machado\. 2024\.Towards evolution of deep neural networks through contrastive self\-supervised learning\. In*2024 IEEE Congress on Evolutionary Computation \(CEC\)*\. IEEE, 1–8\.
- Wang et al\.\(2026\)Guoyong Wang, Kaijun Zhang, Jiyue Jiang, Chaonan Wang, Hui Bi, Haojun Liang, Zuoliang Qi, Ying Huang, Yu Li, and Xiaonan Yang\. 2026\.Human–large language model collaboration in clinical medicine: a systematic review and meta\-analysis\.*npj Digital Medicine*\(2026\)\.
- Wang et al\.\(2025a\)Hongcheng Wang, Yinuo Huang, Sukai Wang, Guanghui Ren, and Hao Dong\. 2025a\.Grpo\-ma: Multi\-answer generation in grpo for stable and efficient chain\-of\-thought training\.*arXiv preprint arXiv:2509\.24494*\(2025\)\.
- Wang et al\.\(2025b\)Zhenyu Wang, Zikang Wang, Jiyue Jiang, Pengan Chen, Xiangyu Shi, and Yu Li\. 2025b\.Large language models in bioinformatics: A survey\. In*Findings of the Association for Computational Linguistics: ACL 2025*\. 3602–3615\.
- Wu et al\.\(2025\)Zheng Wu, Heyuan Huang, Yanjia Yang, Yuanyi Song, Xingyu Lou, Weiwen Liu, Weinan Zhang, Jun Wang, and Zhuosheng Zhang\. 2025\.Quick on the uptake: Eliciting implicit intents from human demonstrations for personalized mobile\-use agents\.*arXiv preprint arXiv:2508\.08645*\(2025\)\.
- Wu et al\.\(2026\)Zheng Wu, Xingyu Lou, Xinbei Ma, Yansi Li, Weiwen Liu, Weinan Zhang, Jun Wang, and Zhuosheng Zhang\. 2026\.Agent\-Dice: Disentangling Knowledge Updates via Geometric Consensus for Agent Continual Learning\.*arXiv preprint arXiv:2601\.03641*\(2026\)\.
- Xie et al\.\(2021\)Tengyang Xie, Ching\-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal\. 2021\.Bellman\-consistent pessimism for offline reinforcement learning\.*Advances in neural information processing systems*34 \(2021\), 6683–6694\.
- Xu et al\.\(2024\)Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu\. 2024\.Is dpo superior to ppo for llm alignment? a comprehensive study\.*arXiv preprint arXiv:2404\.10719*\(2024\)\.
- Yang et al\.\(2025\)Hang Yang, Hao Chen, Hui Guo, Yineng Chen, Ching\-Sheng Lin, Shu Hu, Jinrong Hu, Xi Wu, and Xin Wang\. 2025\.Llm\-medqa: Enhancing medical question answering through case studies in large language models\. In*2025 International Joint Conference on Neural Networks \(IJCNN\)*\. IEEE, 1–8\.
- Yao et al\.\(2024\)Zonghai Yao, Zihao Zhang, Chaolong Tang, Xingyu Bian, Youxia Zhao, Zhichao Yang, Junda Wang, Huixue Zhou, Won Seok Jang, Feiyun Ouyang, and Hong Yu\. 2024\.Medqa\-cs: Benchmarking large language models clinical skills using an ai\-sce framework\.*arXiv preprint arXiv:2410\.01553*\(2024\)\.
- Yeo et al\.\(2025\)Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue\. 2025\.Demystifying long chain\-of\-thought reasoning in llms\.*arXiv preprint arXiv:2502\.03373*\(2025\)\.
- Yuan et al\.\(2024\)Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston\. 2024\.Self\-rewarding language models\. In*Forty\-first International Conference on Machine Learning*\.
- Zhang et al\.\(2020\)Haifeng Zhang, Weizhe Chen, Zeren Huang, Minne Li, Yaodong Yang, Weinan Zhang, and Jun Wang\. 2020\.Bi\-level actor\-critic for multi\-agent coordination\. In*Proceedings of the AAAI conference on artificial intelligence*, Vol\. 34\. 7325–7332\.
- Zhang et al\.\(2024\)Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu\. 2024\.Aflow: Automating agentic workflow generation\.*arXiv preprint arXiv:2410\.10762*\(2024\)\.
- Zhong et al\.\(2024\)Han Zhong, Zikang Shan, Guhao Feng, Wei Xiong, Xinle Cheng, Li Zhao, Di He, Jiang Bian, and Liwei Wang\. 2024\.Dpo meets ppo: Reinforced token optimization for rlhf\.*arXiv preprint arXiv:2404\.18922*\(2024\)\.
- Zhong et al\.\(2026\)Qihuang Zhong, Kang Wang, Ziyang Xu, Liang Ding, Juhua Liu, and Bo Du\. 2026\.Achieving¿ 97% on gsm8k: Deeply understanding the problems makes llms better solvers for math word problems\.*Frontiers of Computer Science*20, 1 \(2026\), 1–3\.
- Zhou et al\.\(2025\)Jingqi Zhou, Sheng Wang, Jingwei Dong, Kai Liu, Lei Li, Jiahui Gao, Jiyue Jiang, Lingpeng Kong, and Chuan Wu\. 2025\.PROREASON: Multi\-Modal Proactive Reasoning with Decoupled Eyesight and Wisdom\. In*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*\. 31650–31679\.
- Zou et al\.\(2025\)Jiaru Zou, Ling Yang, Jingwen Gu, Jiahao Qiu, Ke Shen, Jingrui He, and Mengdi Wang\. 2025\.ReasonFlux\-PRM: Trajectory\-Aware PRMs for Long Chain\-of\-Thought Reasoning in LLMs\.*arXiv preprint arXiv:2506\.18896*\(2025\)\.

## Appendix AAppendix

### A\.1\.Proof of the Identity between LLM Logits and Latent Soft Q\-functions

We aim to rigorously prove that the unnormalized logits produced by a LLM trained via standard Next\-Token Prediction \(NTP\) are mathematically equivalent to the entropy\-regularized optimal Q\-function within the Maximum Entropy Reinforcement Learning \(MaxEnt RL\) framework\.

#### Preliminaries: Maximum Entropy RL

Under the MaxEnt RL framework, an agent seeks to maximize the expected cumulative reward while simultaneously maximizing the entropy of its policy to encourage exploration\. For a given reward functionrr, the optimal policyπ∗\\pi^\{\*\}follows a Boltzmann distribution:

\(A\.1\)π∗​\(a\|s\)=exp⁡\(Q∗​\(s,a\)−V∗​\(s\)α\)∝exp⁡\(Q∗​\(s,a\)α\),\\pi^\{\*\}\(a\|s\)=\\exp\\left\(\\frac\{Q^\{\*\}\(s,a\)\-V^\{\*\}\(s\)\}\{\\alpha\}\\right\)\\propto\\exp\\left\(\\frac\{Q^\{\*\}\(s,a\)\}\{\\alpha\}\\right\),whereQ∗​\(s,a\)Q^\{\*\}\(s,a\)is the entropy\-regularized optimal Q\-function,V∗​\(s\)=α​log​∑a′∈𝒱exp⁡\(Q∗​\(s,a′\)/α\)V^\{\*\}\(s\)=\\alpha\\log\\sum\_\{a^\{\\prime\}\\in\\mathcal\{V\}\}\\exp\(Q^\{\*\}\(s,a^\{\\prime\}\)/\\alpha\)is the corresponding soft state\-value function, andα\\alphais the temperature parameter controlling the degree of exploration\.

#### LLM Policy Parameterization

Standard LLMs adopt an autoregressive architecture\. At each timestephh, the model generates a conditional probability distribution for the tokenaha\_\{h\}based on the current contextshs\_\{h\}\. This distribution is typically obtained by applying the Softmax operator to the unnormalized logitsfθ​\(sh,ah\)f\_\{\\theta\}\(s\_\{h\},a\_\{h\}\):

\(A\.2\)π^​\(ah\|sh\)=exp⁡\(fθ​\(sh,ah\)/α\)∑a′∈𝒱exp⁡\(fθ​\(sh,a′\)/α\)\.\\hat\{\\pi\}\(a\_\{h\}\|s\_\{h\}\)=\\frac\{\\exp\(f\_\{\\theta\}\(s\_\{h\},a\_\{h\}\)/\\alpha\)\}\{\\sum\_\{a^\{\\prime\}\\in\\mathcal\{V\}\}\\exp\(f\_\{\\theta\}\(s\_\{h\},a^\{\\prime\}\)/\\alpha\)\}\.

#### Equivalence of MLE and Offline IRL Objectives

The primary training objective of an LLM is to maximize the Log\-Likelihood \(MLE\) over a training dataset𝒟\\mathcal\{D\}:

\(A\.3\)maxθ⁡ℒM​L​E​\(θ\)=∑i=1n∑h=1Hlog⁡\(exp⁡\(fθ​\(shi,ahi\)/α\)∑a′∈𝒱exp⁡\(fθ​\(shi,a′\)/α\)\)\.\\max\_\{\\theta\}\\mathcal\{L\}\_\{MLE\}\(\\theta\)=\\sum\_\{i=1\}^\{n\}\\sum\_\{h=1\}^\{H\}\\log\\left\(\\frac\{\\exp\(f\_\{\\theta\}\(s\_\{h\}^\{i\},a\_\{h\}^\{i\}\)/\\alpha\)\}\{\\sum\_\{a^\{\\prime\}\\in\\mathcal\{V\}\}\\exp\(f\_\{\\theta\}\(s\_\{h\}^\{i\},a^\{\\prime\}\)/\\alpha\)\}\\right\)\.Conversely, Offline Inverse Reinforcement Learning \(Offline IRL\)\(Jarboui and Perchet,[2021](https://arxiv.org/html/2605.24005#bib.bib15);Ng and Russell,[2000](https://arxiv.org/html/2605.24005#bib.bib27)\)aims to find a Q\-function that best explains the expert demonstration data\. Its objective function is defined as:

\(A\.4\)maxQ⁡1n​∑i=1n∑h=1H\[Q​\(shi,ahi\)−α​log⁡\(∑a′∈𝒱exp⁡\(Q​\(shi,a′\)/α\)\)\]\.\\max\_\{Q\}\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\sum\_\{h=1\}^\{H\}\\left\[Q\(s\_\{h\}^\{i\},a\_\{h\}^\{i\}\)\-\\alpha\\log\\left\(\\sum\_\{a^\{\\prime\}\\in\\mathcal\{V\}\}\\exp\(Q\(s\_\{h\}^\{i\},a^\{\\prime\}\)/\\alpha\)\\right\)\\right\]\.Utilizing the logarithmic identitylog⁡\(A/B\)=log⁡A−log⁡B\\log\(A/B\)=\\log A\-\\log B, we can expand the terms in Eq\. \(A\.3\):

\(A\.5\)log⁡π^​\(ah\|sh\)=1α​fθ​\(sh,ah\)−log​∑a′∈𝒱exp⁡\(fθ​\(sh,a′\)/α\)\.\\log\\hat\{\\pi\}\(a\_\{h\}\|s\_\{h\}\)=\\frac\{1\}\{\\alpha\}f\_\{\\theta\}\(s\_\{h\},a\_\{h\}\)\-\\log\\sum\_\{a^\{\\prime\}\\in\\mathcal\{V\}\}\\exp\(f\_\{\\theta\}\(s\_\{h\},a^\{\\prime\}\)/\\alpha\)\.By multiplying Eq\. \(A\.5\) by the constant factorn​αn\\alpha, the optimization objective becomes identical in form to the Offline IRL objective in Eq\. \(A\.4\)\.

#### Conclusion and Reward Recovery

Since the sets of optimal solutionsarg⁡max\\arg\\maxfor both optimization problems are identical, we conclude that the logitsfθf\_\{\\theta\}learned by the LLM are inherently the soft Q\-functions that satisfy the optimality conditions of Offline IRL, i\.e\.,fθ≡Q^f\_\{\\theta\}\\equiv\\hat\{Q\}\.

Furthermore, according to the Inverse Soft Bellman Operator, we can directly recover the endogenous rewardr∗r^\{\*\}from this implicit Q\-function:

\(A\.6\)r∗​\(sh,ah\):=Q^​\(sh,ah\)−γ⋅VQ^​\(sh\+1\),r^\{\*\}\(s\_\{h\},a\_\{h\}\):=\\hat\{Q\}\(s\_\{h\},a\_\{h\}\)\-\\gamma\\cdot V\_\{\\hat\{Q\}\}\(s\_\{h\+1\}\),whereVQ^​\(sh\+1\)=α​log​∑a′∈𝒱exp⁡\(Q^​\(sh\+1,a′\)/α\)V\_\{\\hat\{Q\}\}\(s\_\{h\+1\}\)=\\alpha\\log\\sum\_\{a^\{\\prime\}\\in\\mathcal\{V\}\}\\exp\(\\hat\{Q\}\(s\_\{h\+1\},a^\{\\prime\}\)/\\alpha\)\. This demonstrates that during the pre\-training phase, an LLM not only learns to model the text distribution but also implicitly constructs a reward function in its logit space that characterizes the logical consistency of the data\.

### A\.2\.Theoretical Consistency: MLE as Dual Gradient Ascent in MaxEnt IRL

In this section, we provide a more rigorous proof that the Maximum Likelihood Estimation \(MLE\) of an LLM is equivalent to the dual problem of entropy\-regularized policy optimization\. We further demonstrate the uniqueness of the recovered rewardr∗r^\{\*\}under the structural constraints of the transformer architecture\.

#### The Variational Principle for Next\-Token Prediction

Consider the transition probability of an LLM, denoted byπθ​\(a\|s\)\\pi\_\{\\theta\}\(a\|s\)\. The training objective is to minimize the Kullback\-Leibler \(KL\) divergence between the empirical data distributionπD\\pi\_\{D\}and the model:

\(A\.11\)minθ𝒥\(θ\)=𝔼s∼ρD\[DK​L\(πD\(⋅\|s\)∥πθ\(⋅\|s\)\)\],\\min\_\{\\theta\}\\mathcal\{J\}\(\\theta\)=\\mathbb\{E\}\_\{s\\sim\\rho\_\{D\}\}\\left\[D\_\{KL\}\\left\(\\pi\_\{D\}\(\\cdot\|s\)\\\|\\pi\_\{\\theta\}\(\\cdot\|s\)\\right\)\\right\],whereρD\\rho\_\{D\}is the state distribution in the dataset\. Substituting the Boltzmann parameterizationπθ​\(a\|s\)=exp⁡\(\(fθ​\(s,a\)−Vθ​\(s\)\)/α\)\\pi\_\{\\theta\}\(a\|s\)=\\exp\(\(f\_\{\\theta\}\(s,a\)\-V\_\{\\theta\}\(s\)\)/\\alpha\), whereVθ​\(s\)V\_\{\\theta\}\(s\)is the log\-partition functionα​log​∑a′exp⁡\(fθ​\(s,a′\)/α\)\\alpha\\log\\sum\_\{a^\{\\prime\}\}\\exp\(f\_\{\\theta\}\(s,a^\{\\prime\}\)/\\alpha\), the gradient w\.r\.t\. the logitsfθf\_\{\\theta\}is:

\(A\.12\)∂𝒥∂fθ​\(s,a\)=1α​\[πθ​\(a\|s\)−πD​\(a\|s\)\]\.\\frac\{\\partial\\mathcal\{J\}\}\{\\partial f\_\{\\theta\}\(s,a\)\}=\\frac\{1\}\{\\alpha\}\\left\[\\pi\_\{\\theta\}\(a\|s\)\-\\pi\_\{D\}\(a\|s\)\\right\]\.At the global optimum,πθ​\(a\|s\)=πD​\(a\|s\)\\pi\_\{\\theta\}\(a\|s\)=\\pi\_\{D\}\(a\|s\)for all\(s,a\)\(s,a\)in the support of𝒟\\mathcal\{D\}\.

#### Duality with Feature Matching

According to the Maximum Entropy Principle, the distribution that maximizes entropy subject to matching the empirical expectations of a feature mapϕ​\(s,a\)\\phi\(s,a\)is a Gibbs distribution\. Letr​\(s,a\)=𝐰⊤​ϕ​\(s,a\)r\(s,a\)=\\mathbf\{w\}^\{\\top\}\\phi\(s,a\)be a linear reward function\. The MaxEnt IRL objective is to find𝐰\\mathbf\{w\}such that:

\(A\.13\)𝔼π𝐰​\[ϕ​\(s,a\)\]=𝔼πD​\[ϕ​\(s,a\)\]\.\\mathbb\{E\}\_\{\\pi\_\{\\mathbf\{w\}\}\}\[\\phi\(s,a\)\]=\\mathbb\{E\}\_\{\\pi\_\{D\}\}\[\\phi\(s,a\)\]\.Comparing Eq\. \(A\.12\) and Eq\. \(A\.13\), the LLM’s logitsfθ​\(s,a\)f\_\{\\theta\}\(s,a\)serve as the implicit inner product between the high\-dimensional latent state \(transformer hidden states\) and the token embeddings\. Thus, the MLE training is equivalent to a high\-dimensional feature matching process where the reward function is implicitly defined by the model’s capacity:ri​m​p​l​\(s,a\)≈fθ​\(s,a\)r\_\{impl\}\(s,a\)\\approx f\_\{\\theta\}\(s,a\)\.

#### The Inverse Soft Bellman Operator and Uniqueness

To prove the consistency ofr∗​\(s,a\)=Q^​\(s,a\)−γ​𝔼s′​\[VQ^​\(s′\)\]r^\{\*\}\(s,a\)=\\hat\{Q\}\(s,a\)\-\\gamma\\mathbb\{E\}\_\{s^\{\\prime\}\}\[V\_\{\\hat\{Q\}\}\(s^\{\\prime\}\)\], we define theInverse Soft Bellman Operatorℬ−1\\mathcal\{B\}^\{\-1\}:

\(A\.14\)ℬ−1​Q​\(s,a\)≜Q​\(s,a\)−γ​∫𝒫​\(s′\|s,a\)​\[α​log​∑a′∈𝒱exp⁡\(Q​\(s′,a′\)α\)\]​𝑑s′\.\\mathcal\{B\}^\{\-1\}Q\(s,a\)\\triangleq Q\(s,a\)\-\\gamma\\int\\mathcal\{P\}\(s^\{\\prime\}\|s,a\)\\left\[\\alpha\\log\\sum\_\{a^\{\\prime\}\\in\\mathcal\{V\}\}\\exp\\left\(\\frac\{Q\(s^\{\\prime\},a^\{\\prime\}\)\}\{\\alpha\}\\right\)\\right\]ds^\{\\prime\}\.For any two rewardsr1,r2r\_\{1\},r\_\{2\}that yield the same optimal policyπ∗\\pi^\{\*\}, they must satisfyr1​\(s,a\)−r2​\(s,a\)=Φ​\(s\)−γ​𝔼s′∼P​\[Φ​\(s′\)\]r\_\{1\}\(s,a\)\-r\_\{2\}\(s,a\)=\\Phi\(s\)\-\\gamma\\mathbb\{E\}\_\{s^\{\\prime\}\\sim P\}\[\\Phi\(s^\{\\prime\}\)\]for some potential functionΦ​\(s\)\\Phi\(s\)\.

In the LLM context, the partition functionVθ​\(s\)V\_\{\\theta\}\(s\)is uniquely determined by the logitsfθf\_\{\\theta\}up to a constant\. Since the LLM must normalize over the finite vocabulary𝒱\\mathcal\{V\}at every step, the potential functionΦ​\(s\)\\Phi\(s\)is anchored by the normalization constraint of the Softmax layer\. This effectively eliminates the reward ambiguity, making the recovered endogenous rewardr∗r^\{\*\}theuniqueminimal\-norm reward that satisfies the data observations under the MaxEnt assumption\.

#### Structural Sufficiency

We hypothesize that the Transformer architecturefθf\_\{\\theta\}is a universal functional approximator for the Q\-function\. Given the convergence of SGD on the MLE objective, the sequence of rewards\{rt∗\}\\\{r\_\{t\}^\{\*\}\\\}induced by the sequence of model parameters\{θt\}\\\{\\theta\_\{t\}\\\}converges in theL2L\_\{2\}\-norm to the ground\-truth energy fieldr∞r\_\{\\infty\}that generated the corpus:

\(A\.15\)limt→∞‖rθt∗−rd​a​t​a‖ρD=0\.\\lim\_\{t\\to\\infty\}\\\|r\_\{\\theta\_\{t\}\}^\{\*\}\-r\_\{data\}\\\|\_\{\\rho\_\{D\}\}=0\.This completes the proof that LLM logits provide a consistent and theoretically grounded proxy for the underlying reward structure of natural language\.

### A\.3\.Formal Derivation of the Sub\-optimality Bound: From Quadratic to Linear Growth

In this section, we provide a rigorous theoretical guarantee for the self\-evolution path of LC\-ERD\. We analyze the divergence between the learned policyπR​L\\pi^\{RL\}and the oracle logic\-consistent policyπ∗\\pi^\{\*\}, demonstrating how the Variational Logic Potential \(VLP\) reshaping mitigates the “exposure bias” common in imitation learning\.

###### Theorem A\.1\(Linear Sub\-optimality of LC\-ERD\)\.

In a token\-level MDPℳ\\mathcal\{M\}with a trajectory horizonHHand a maximum single\-step logic deviationϵπ:=sups∈𝒮DT​V​\(π∗∥π^\)\\epsilon\_\{\\pi\}:=\\sup\_\{s\\in\\mathcal\{S\}\}D\_\{TV\}\(\\pi^\{\*\}\\\|\\hat\{\\pi\}\), the sub\-optimality of the SFT base policyπ^\\hat\{\\pi\}satisfiesV∗−Vπ^≤𝒪​\(H2​ϵπ\)V^\{\*\}\-V^\{\\hat\{\\pi\}\}\\leq\\mathcal\{O\}\(H^\{2\}\\epsilon\_\{\\pi\}\)\. In contrast, the LC\-ERD aligned policyπR​L\\pi^\{RL\}, by anchoring to the latent logical manifold via VLP\-reshaped rewards, achieves a linear bound:

\(A\.16\)V∗−VπR​L≤𝒪​\(H​ϵπ\)\.V^\{\*\}\-V^\{\\pi^\{RL\}\}\\leq\\mathcal\{O\}\(H\\epsilon\_\{\\pi\}\)\.

###### Proof\.

The Quadratic Accumulation of SFT \(Behavioral Cloning\)\.The base modelπ^\\hat\{\\pi\}trained via Next\-Token Prediction \(NTP\) is equivalent to Behavioral Cloning \(BC\)\. Letdhπ​\(s\)d\_\{h\}^\{\\pi\}\(s\)be the state visitation distribution at stephhunder policyπ\\pi\. The value gap is:

\(A\.17\)V∗−Vπ^=∑h=1H𝔼s∼dhπ∗​\[Aπ^​\(s,π∗​\(s\)\)\]\.V^\{\*\}\-V^\{\\hat\{\\pi\}\}=\\sum\_\{h=1\}^\{H\}\\mathbb\{E\}\_\{s\\sim d\_\{h\}^\{\\pi^\{\*\}\}\}\[A^\{\\hat\{\\pi\}\}\(s,\\pi^\{\*\}\(s\)\)\]\.By the Distribution Shift Lemma, the total variation distance between the expert and learner distributions grows as:

\(A\.18\)∥dhπ∗−dhπ^∥1≤∑t=1h𝔼s∼dtπ∗\[∥π∗\(⋅\|s\)−π^\(⋅\|s\)∥1\]≤h⋅ϵπ\.\\\|d\_\{h\}^\{\\pi^\{\*\}\}\-d\_\{h\}^\{\\hat\{\\pi\}\}\\\|\_\{1\}\\leq\\sum\_\{t=1\}^\{h\}\\mathbb\{E\}\_\{s\\sim d\_\{t\}^\{\\pi^\{\*\}\}\}\[\\\|\\pi^\{\*\}\(\\cdot\|s\)\-\\hat\{\\pi\}\(\\cdot\|s\)\\\|\_\{1\}\]\\leq h\\cdot\\epsilon\_\{\\pi\}\.Summing over the horizonHH:

\(A\.19\)V∗−Vπ^≤Rm​a​x​∑h=1Hh⋅ϵπ=Rm​a​x​H​\(H\+1\)2​ϵπ≈𝒪​\(H2​ϵπ\)\.V^\{\*\}\-V^\{\\hat\{\\pi\}\}\\leq R\_\{max\}\\sum\_\{h=1\}^\{H\}h\\cdot\\epsilon\_\{\\pi\}=\\frac\{R\_\{max\}H\(H\+1\)\}\{2\}\\epsilon\_\{\\pi\}\\approx\\mathcal\{O\}\(H^\{2\}\\epsilon\_\{\\pi\}\)\.This quadratic dependence reflects the “compounding error” problem: small local drifts lead the model into out\-of\-distribution \(OOD\) states whereπ^\\hat\{\\pi\}has no training signal, causing failure in long reasoning chains\.

#### The Linear Bound via VLP Reward Shaping

LC\-ERD transforms the objective from pure imitation to maximizing a logic\-consistent potential field\. Letr^​\(s,a\)\\hat\{r\}\(s,a\)be the reshaped reward defined by the Variational Logic PotentialΦ​\(s\)\\Phi\(s\):

\(A\.20\)r^​\(s,a\)=r∗​\(s,a\)\+γ​Φ​\(s′\)−Φ​\(s\),\\hat\{r\}\(s,a\)=r^\{\*\}\(s,a\)\+\\gamma\\Phi\(s^\{\\prime\}\)\-\\Phi\(s\),whereΦ​\(s\)\\Phi\(s\)is the Soft State\-Value functionVQ^​\(s\)V\_\{\\hat\{Q\}\}\(s\)derived in Appendix A\.1\.

We decompose the sub\-optimality using the Performance Difference Lemma\. Under the MaxEnt RL objective, the gradient of the policy update is guided by the advantage functionAr^πR​L​\(s,a\)A^\{\\pi^\{RL\}\}\_\{\\hat\{r\}\}\(s,a\)\. Crucially, potential\-based shaping leaves the optimal policy invariant but reshapes the error landscape\.

Letτ=\(s1,a1,…,sH\)\\tau=\(s\_\{1\},a\_\{1\},\\dots,s\_\{H\}\)be a trajectory\. The cumulative reshaped reward is:

\(A\.21\)∑h=1Hr^​\(sh,ah\)=∑h=1Hr∗​\(sh,ah\)\+γ​Φ​\(sH\+1\)−Φ​\(s1\)\.\\sum\_\{h=1\}^\{H\}\\hat\{r\}\(s\_\{h\},a\_\{h\}\)=\\sum\_\{h=1\}^\{H\}r^\{\*\}\(s\_\{h\},a\_\{h\}\)\+\\gamma\\Phi\(s\_\{H\+1\}\)\-\\Phi\(s\_\{1\}\)\.Using the telescoping property, the variance of the cumulative reward across the logic\-consistent manifold is bounded by the boundary conditions of the potentialΦ\\Phi, rather than the sum of step\-wise errors\.

By applying the stability condition for entropy\-regularized MDPs, the divergence in policy value is bounded by the expected KL\-divergence over the steady\-state distributiondπ∗d^\{\\pi^\{\*\}\}rather than the compounding sum:

\(A\.22\)Vr∗π∗−Vr∗πR​L≤α1−γ​𝔼s∼dπ∗​\[DK​L​\(π∗∥πR​L\)\]\.V\_\{r^\{\*\}\}^\{\\pi^\{\*\}\}\-V\_\{r^\{\*\}\}^\{\\pi^\{RL\}\}\\leq\\frac\{\\alpha\}\{1\-\\gamma\}\\mathbb\{E\}\_\{s\\sim d^\{\\pi^\{\*\}\}\}\[D\_\{KL\}\(\\pi^\{\*\}\\\|\\pi^\{RL\}\)\]\.For a finite horizonHHwithγ→1\\gamma\\to 1, this converges to:

\(A\.23\)V∗−VπR​L≤C⋅H⋅ϵπ,V^\{\*\}\-V^\{\\pi^\{RL\}\}\\leq C\\cdot H\\cdot\\epsilon\_\{\\pi\},whereCCis a constant related to the logic\-consistency gainδ\\deltaidentified in Remark 3\. ∎

### A\.4\.Proof of Unbiased Logic Elicitation under Hard Terminal Constraints

A potential critique of endogenous reward discovery is the risk of “correctness illusions”, where the model assigns high logits to logically flawed but superficially plausible reasoning steps\. We prove that by imposing a hard terminal constraint𝒵\\mathcal\{Z\}, the elicited reward signalr^\\hat\{r\}remains an unbiased estimator of the true reasoning manifoldℳ∗\\mathcal\{M\}^\{\*\}\.

#### The Correctness\-Conditioned Manifold

Letτ=\(s1,a1,…,sH\)\\tau=\(s\_\{1\},a\_\{1\},\\dots,s\_\{H\}\)be a reasoning trajectory and𝒵∈\{0,1\}\\mathcal\{Z\}\\in\\\{0,1\\\}be a binary indicator of the terminal answer’s correctness \(e\.g\., via a deterministic verifier or ground\-truth label\)\. The “true” but latent reasoning distribution is denoted asP​\(τ\|𝒵=1\)P\(\\tau\|\\mathcal\{Z\}=1\)\.

According to the identity derived in Appendix[A\.1](https://arxiv.org/html/2605.24005#A1.SS1), the LLM’s unnormalized logitsfθf\_\{\\theta\}parameterize an energy\-based modelPθ​\(τ\)∝exp⁡\(∑rθ​\(s,a\)/α\)P\_\{\\theta\}\(\\tau\)\\propto\\exp\(\\sum r\_\{\\theta\}\(s,a\)/\\alpha\)\. We define theelicited rewardunder the terminal constraint as the conditional expectation:

\(A\.24\)r^e​l​i​c​i​t​e​d≜𝔼πθ​\[fθ​\(s,a\)∣𝒵=1\]\.\\hat\{r\}\_\{elicited\}\\triangleq\\mathbb\{E\}\_\{\\pi\_\{\\theta\}\}\[f\_\{\\theta\}\(s,a\)\\mid\\mathcal\{Z\}=1\]\.

#### Unbiasedness via Bayes’ Rule

We aim to prove thatr^e​l​i​c​i​t​e​d\\hat\{r\}\_\{elicited\}converges to the true logic rewardr∗r^\{\*\}\. Using Bayes’ theorem, the posterior distribution of a reasoning step given a correct outcome is:

\(A\.25\)P​\(τ∣𝒵=1\)=P​\(𝒵=1∣τ\)​Pd​a​t​a​\(τ\)P​\(𝒵=1\)\.P\(\\tau\\mid\\mathcal\{Z\}=1\)=\\frac\{P\(\\mathcal\{Z\}=1\\mid\\tau\)P\_\{data\}\(\\tau\)\}\{P\(\\mathcal\{Z\}=1\)\}\.In the context of LC\-ERD, the hard constraint𝒵\\mathcal\{Z\}acts as an oracle filter whereP​\(𝒵=1∣τ\)=1P\(\\mathcal\{Z\}=1\\mid\\tau\)=1ifτ\\tauis logically valid and0otherwise\.

Under the assumption ofsufficient model capacity\(established in Appendix[A\.2](https://arxiv.org/html/2605.24005#A1.SS2)\), the LLM’s logit space covers the support of the expert manifold\. The log\-likelihood of the conditioned distribution becomes:

\(A\.26\)log⁡P​\(τ∣𝒵=1\)=∑h=1Hfθ​\(sh,ah\)α−log⁡Z​\(θ\)\+log⁡𝕀​\(τ∈ℳ∗\),\\log P\(\\tau\\mid\\mathcal\{Z\}=1\)=\\sum\_\{h=1\}^\{H\}\\frac\{f\_\{\\theta\}\(s\_\{h\},a\_\{h\}\)\}\{\\alpha\}\-\\log Z\(\\theta\)\+\\log\\mathbb\{I\}\(\\tau\\in\\mathcal\{M\}^\{\*\}\),where𝕀​\(⋅\)\\mathbb\{I\}\(\\cdot\)is the indicator function for the expert manifoldℳ∗\\mathcal\{M\}^\{\*\}\. The termlog⁡𝕀​\(τ∈ℳ∗\)\\log\\mathbb\{I\}\(\\tau\\in\\mathcal\{M\}^\{\*\}\)effectively re\-normalizes the endogenous reward by assigning−∞\-\\inftyenergy to logically inconsistent paths, thereby eliminating the “mimetic bias” \(illusions\) inherited from SFT\.

#### Orthogonality to Hallucination Noise

Letfθ=f∗\+ξf\_\{\\theta\}=f^\{\*\}\+\\xi, wheref∗f^\{\*\}is the true logic\-consistent logit andξ\\xiis the hallucination noise \(bias\)\. For the estimator to be unbiased, we require𝔼​\[ξ∣𝒵=1\]=0\\mathbb\{E\}\[\\xi\\mid\\mathcal\{Z\}=1\]=0\.

By the definition of the terminal constraint in LC\-ERD, the filtering process𝒵\\mathcal\{Z\}isconditionally independentof the model’s internal noiseξ\\xigiven the logical validity ofτ\\tau\. Formally, if𝒵\\mathcal\{Z\}is determined solely by the final outcome:

\(A\.27\)∫τ∈ℳ∗ξ​\(τ\)​Pθ​\(τ∣𝒵=1\)​𝑑τ=0\.\\int\_\{\\tau\\in\\mathcal\{M\}^\{\*\}\}\\xi\(\\tau\)P\_\{\\theta\}\(\\tau\\mid\\mathcal\{Z\}=1\)d\\tau=0\.This holds because anyτ\\tauthat satisfies𝒵=1\\mathcal\{Z\}=1must, by the property of the task \(e\.g\., mathematics or symbolic logic\), lie within the high\-density region of the true rewardr∗r^\{\*\}\. Therefore, the systematic biasξ\\xithat leads to𝒵=0\\mathcal\{Z\}=0is pruned, leaving the filtered endogenous signal as an asymptotically unbiased proxy forr∗r^\{\*\}\.

#### Minimum Variance and Information Efficiency

To further ensure the robustness of the elicitation, we observe that the filtering process𝒵=1\\mathcal\{Z\}=1defines a projection operator𝒫ℳ∗\\mathcal\{P\}\_\{\\mathcal\{M\}^\{\*\}\}from the unconstrained logit space onto the logic\-consistent Hilbert spaceℒ2​\(ℳ∗\)\\mathcal\{L\}\_\{2\}\(\\mathcal\{M\}^\{\*\}\)\. The resulting estimator satisfies:

\(A\.28\)r^L​C−E​R​D=projℳ∗​\(fθ\)=arg⁡ming∈ℒ2​\(ℳ∗\)⁡𝔼τ∼πD​\[‖g​\(τ\)−fθ​\(τ\)‖2\]\.\\hat\{r\}\_\{LC\-ERD\}=\\text\{proj\}\_\{\\mathcal\{M\}^\{\*\}\}\(f\_\{\\theta\}\)=\\arg\\min\_\{g\\in\\mathcal\{L\}\_\{2\}\(\\mathcal\{M\}^\{\*\}\)\}\\mathbb\{E\}\_\{\\tau\\sim\\pi\_\{D\}\}\[\\\|g\(\\tau\)\-f\_\{\\theta\}\(\\tau\)\\\|^\{2\}\]\.This projection ensures thatr^\\hat\{r\}is theMinimum\-Variance Unbiased Estimator \(MVUE\)within the logic\-consistent subspace\. By concentrating the model’s epistemic uncertainty onto valid reasoning trajectories, LC\-ERD maximizes the Fisher information extracted from each successful sample, thereby theoretically justifying the rapid convergence observed during self\-evolution\.

#### Conclusion

By shifting the optimization target from “local action matching” to “global potential alignment”, LC\-ERD effectively replaces the quadratic error growth with a linear one\. This theoretical result provides the foundation for the self\-evolution capability observed in our experiments: the model remains stable even as the reasoning lengthHHincreases, effectively overcoming the mimetic bias\.

### A\.5\.Related Works

Logic\-Consistent Manifold Alignment\.Aligning models to a logical manifold often involves manual rules or symbolic verifiers\. WhileGRPO\(Shao et al\.,[2024](https://arxiv.org/html/2605.24005#bib.bib30)\)and its derivatives optimize policy relative to group outcomes, they still struggle with open\-domain generalization where pre\-training biases dominate\. Unlike these approaches, LC\-ERD’sLatent Logic Expertise \(LLE\)algorithm elicits an endogenous logical anchor training\-free from the model’s own distribution\. This allows for a provably more stable convergence path, shifting the sub\-optimality bound from quadratic to linear—a theoretical guarantee absent in prior iterative alignment literature\.