Learning Stateful Predictive Knowledge From Experience

# Learning Stateful Predictive Knowledge From Experience
Source: [https://arxiv.org/html/2607.28638](https://arxiv.org/html/2607.28638)
Yan Song1, 8222Corresponding authors\. Emails:<yan\.song\.24@ucl\.ac\.uk\>and<junwang@cs\.ucl\.ac\.uk\>\.Xidong Feng1Bo Liu2Xinyu Cui3Haotian Fu4Zichen Liu2 Mengyue Yang5Cheng Deng6Jian Zhao7Jun Wang1222Corresponding authors\. Emails:<yan\.song\.24@ucl\.ac\.uk\>and<junwang@cs\.ucl\.ac\.uk\>\. 1AI Centre, Department of Computer Science, University College London 2National University of Singapore3Institute of Automation, Chinese Academy of Sciences 4Brown University5University of Bristol6University of Edinburgh 7Zhongguancun Academy8AI Lab, The Yangtze River Delta

###### Abstract

As large language model \(LLM\) agents increasingly learn from experience, they primarily rely on trajectory\-level reflection to extract insights\. Viewed through the lens of predictive knowledge\(Suttonet al\.,[2011](https://arxiv.org/html/2607.28638#bib.bib125)\), we argue that this approach operates on episodic hindsight rather than predictive foresight, yielding brittle, path\-dependent heuristics\. To address this, we proposeStateful Knowledge Learning\(SKL\)\. SKL shifts the agent’s focus from trajectory\-level summarization to maintaining Stateful Knowledge: explicit, declarative predictive assessments anchored to state\. We first demonstrate a motivating example showing how stateful knowledge provides granularity, enhances generalization, and enables knowledge bootstrapping\. To further scale up the idea, we introduce two algorithms via self\-distillation \(SKL\-SD\) and reinforcement learning \(SKL\-RL\), training agents to autonomously extract state\-grounded predictive knowledge from experience and learn to leverage it for policy making\. Experiments on interactive environments \(WebShop, ScienceWorld\) and a complex reasoning task \(ChessPuzzles\) demonstrate that equipping models with the inherent ability to learn stateful predictive knowledge significantly outpaces current reflection\-based training paradigms\. Code available at[https://github\.com/YanSong97/Stateful\_Knowledge\_Learning](https://github.com/YanSong97/Stateful_Knowledge_Learning)\.

## 1Introduction

The frontier of artificial intelligence is increasingly shifting towards ’Era of Experience,’ where agents acquire capabilities and knowledge through continuous, grounded interactions with their environments rather than relying solely on static human data\(Silver and Sutton,[2025](https://arxiv.org/html/2607.28638#bib.bib2)\)\. Recent efforts have made significant progress for large language model agent, including Reflexion\(Shinnet al\.,[2023](https://arxiv.org/html/2607.28638#bib.bib76)\), advanced memory system\(Zhouet al\.,[2025](https://arxiv.org/html/2607.28638#bib.bib133)\), and more recent learning\-based methods\(Zhanget al\.,[2025b](https://arxiv.org/html/2607.28638#bib.bib97); Shiet al\.,[2026b](https://arxiv.org/html/2607.28638#bib.bib91)\)\. These frameworks typically rely on the idea of self\-reflection: analyzing past interaction trajectories to summarize errors and extract insights, which can subsequently guide the agent’s next cycle of rollout\.

This growing reliance on experiential learning prompts a fundamental question:what exactly constitutes "knowledge" for an LLM agent to learn from experience?While "knowledge" is a broad concept with numerous definitions\(Zagzebski,[2017](https://arxiv.org/html/2607.28638#bib.bib116)\)and instances \(static facts or mathematical logic\), for the context of agent interacting with dynamic environment, we believepredictive knowledgeoffers a highly principled framework\(Suttonet al\.,[2011](https://arxiv.org/html/2607.28638#bib.bib125); Littman and Sutton,[2001](https://arxiv.org/html/2607.28638#bib.bib115)\)to capture the core of agent environment knowledge–Much of an agent’s knowledge about the world is intrinsically predictive—meaning it can be translated into statements and predictions about potential future outcomes\. For an LLM agent, this means articulating forward\-looking assessment, whether it is estimating the exact numerical distance to a sub\-goal, deducing hidden transition dynamics governing the environment, or predicting whether a strategic opportunity exists in the following steps\.

Viewed through this lens of predictive knowledge, the dominant trajectory\-level reflection paradigms reveal a structural limitation: they operate on episodic hindsight rather than predictive foresight\. A fundamental premise of predictive knowledge is that predictions about the future must be state\-grounded, not trajectory\-grounded\(Suttonet al\.,[2011](https://arxiv.org/html/2607.28638#bib.bib125)\)\. The future unfolds based on the current state, regardless of the historical path taken to reach it\. When agents evaluate entire trajectories post\-hoc, they conflate universal environment dynamics with specific historical sequences\. Consequently, they yield brittle, path\-dependent heuristics \(e\.g\., "I should avoid moving right early"\) instead of verifiable, state\-grounded predictions \(e\.g\., "Moving right is hazardous when a pit is adjacent"\)\. To genuinely harness predictive knowledge, we need to shift our focus from trajectory summarization toStateful Knowledge: explicit, declarative predictive assessments maintained for the encountered state\.

In this paper, we proposeStateful Knowledge Learning \(SKL\), enabling LLM agents to extract, bootstrap and finally learn predictive knowledge from experience\. The paper is organized as follows:

Understanding Stateful Knowledge\.Through a motivating example on a toy stochastic environment, Section[2](https://arxiv.org/html/2607.28638#S2)presents how stateful knowledge works and two benefits that can emerge via direct prompting: \(1\) finer granularity and enhanced generalization compared with trajectory\-level knowledge\. \(2\) knowledge bootstrapping that propogates predictive knowledge backward from successor states\.

Extracting and Learning from Stateful Knowledge\.Section[3](https://arxiv.org/html/2607.28638#S3)further presents how we can scale up the idea to \(1\) Train to enhance LLM agent’s capability on extracting stateful knowledge from experience and, \(2\) Finally learn from the extracted stateful knowledge and use it to enhance agent policy\. We introduce two variants \(SKL\-SD and SKL\-RL\) with self\-distillation and RL that train the agent to autonomously aggregate state\-level experiences and perform predictive knowledge bootstrapping\. Extensive experiments are conducted in Section[4](https://arxiv.org/html/2607.28638#S4)on two widely\-used agentic benchmarks \(WebShop, ScienceWorld\) and a highly complex, contamination\-free reasoning task \(ChessPuzzles\)\. Our results empirically validate that equipping models with the inherent ability to learn stateful predictive knowledge significantly outpaces current reflection\-based training paradigms\.

![Refer to caption](https://arxiv.org/html/2607.28638v1/x1.png)Figure 1:Reflect on trajectories and reflect on stateful knowledge \(root state update only and fully bootstrapping variants\)\. The main difference iswhat informationis aggregated to updatewhich representations\. Stateful Knowledge Reflection aggregates subsequent outcomes originating from the same structural state within an experience buffer to surgically update knowledge at that specific state\.
## 2Understanding stateful predictive knowledge: a motivating example

### 2\.1Reflect on trajectories versus on stateful knowledge

We provide a motivating example to demonstrate how different methods extract information under the same interaction budget\. We distinguish our state\-grounded approach from standard reflection paradigms as follows \(also as illustrated in Figure[1](https://arxiv.org/html/2607.28638#S1.F1)\):

Trajectory Reflection \(Hindsight\):At iterationii, guided by a plan from the previous trial, the agentπ\\piinteracts with environmentEEto form a complete trajectoryτ​\(s0\)=\(s0,a0,s1,…,sT\)\\tau\(s\_\{0\}\)=\(s\_\{0\},a\_\{0\},s\_\{1\},\.\.\.,s\_\{T\}\)starting ats0s\_\{0\}\. The agent then reflects on the entirety ofτ\\tauto generate a global critique and an updated previous plan for iterationi\+1i\+1\(e\.g\.,Reflexion\(Shinnet al\.,[2023](https://arxiv.org/html/2607.28638#bib.bib76)\)\)\. The process repeats until the maximum number of iterationsII\.

Stateful Knowledge Reflection \(Foresight\):Throughout the iterations, the agentπ\\piconsistently maintains an explicit, state\-indexed predictive knowledge table\{zs\}s∈𝒮\\\{z\_\{s\}\\\}\_\{s\\in\\mathcal\{S\}\}for each visited states∈𝒮s\\in\\mathcal\{S\}by querying:"What does the current state predict about future outcomes?"\. At each iteration, decisionsaaare conditioned directly on this declarative assessmentzsz\_\{s\}, i\.e\.,a∼π\(⋅\|zs,s\)a\\sim\\pi\(\\cdot\|z\_\{s\},s\), to interact with the environmentEE\. During the reflection phase, rather than summarizing from a single trajectory, the agent first aggregates outcomes associated with a specific statessacross multiple previous trials, denoted as\{τnH​\(s\)\}n=1N\\\{\\tau\_\{n\}^\{H\}\(s\)\\\}\_\{n=1\}^\{N\}at budget\(N,H\)\(N,H\)\.NNis the number of considered trials andHHis the maximum horizon\. Then the agent generates a refined knowledgez^s∼π\(⋅\|\{τnH\(s\)\}n=1N,s,zs\)\\hat\{z\}\_\{s\}\\sim\\pi\(\\cdot\|\\\{\\tau\_\{n\}^\{H\}\(s\)\\\}\_\{n=1\}^\{N\},s,z\_\{s\}\)based on the contexts, update the knowledge table and proceed to the next iteration\. Refer to Figure[1](https://arxiv.org/html/2607.28638#S1.F1)for a more detailed illustration\.

![Refer to caption](https://arxiv.org/html/2607.28638v1/x2.png)\(a\)Iterative quantitative performance\.
![Refer to caption](https://arxiv.org/html/2607.28638v1/x3.png)\(b\)Qualitative examples

Figure 2:Slippery FrozenLake Analysis\.\(a\) Comparison oftrajectory\-levelandstateful reflection\. Thegreydashed line represents the baseline for retrying the game ten times without any self\-reflection and success once\. Good move ratio refers to the probability of a move not leading to falling\. \(b\) Representative examples of agent behaviour in different iterative loops\.To understand how stateful knowledge works, we utilize a stochastic variant of the FrozenLake environment\(Fenget al\.,[2024](https://arxiv.org/html/2607.28638#bib.bib10)\), where the agent must maneuver toward a goal tile while avoiding deadly pits on the icy surface\. Crucially, the "icy" surface introduces transition stochasticity: an intended directional movement \(e\.g\., "Up"\) has a probability of slipping into orthogonal directions \(e\.g\., "Left" or "Right"\)\. While many recent LLM\-agent focus on deterministic transitions, this slipping mechanic introduces an extra layer of complexity that perfectly exposes the myopia and brittleness of trajectory\-grounded reasoning, since an optimal decision might fail by chance, while a fatal blunder might luckily succeed\. Agents relying on trajectory hindsight are easily misled\.

We prompt a strong modelQwen3\-235B\-A22B\(Yanget al\.,[2025](https://arxiv.org/html/2607.28638#bib.bib126)\)\. To prevent the agent from cheating by pre\-trained knowledge about FrozenLake \(the default isP=\(1/3,1/3\)P=\(1/3,1/3\), so when move up,P​\(”​u​p​”\|”​u​p​”\)=P​\(”​l​e​f​t​”\|”​u​p​”\)=P​\(”​r​i​g​h​t​”\|”​u​p​”\)=13P\("up"\|"up"\)=P\("left"\|"up"\)=P\("right"\|"up"\)=\\frac\{1\}\{3\}\), we also test a variant by setting the slipping probability toP=\(0,2/3\)P=\(0,2/3\)\. To maintain the same interaction budget with the trajectory baseline, the stateful agent only records and updates the initial statezs0z\_\{s\_\{0\}\}in the knowledge table\. Knowledge for the remaining states is regenerated, taking the new root\-state knowledge into account\. For metrics, we evaluate the agent’s success rate of reaching the goal and good move ratio \(the average frequency of actions that yield the lowest death rate\)\. We use 50 maps for training and the remaining 50 as the held\-out test set\.

We report both performance among iterations on training environments and the final test performance \(after iteration 10\) in Figure[2](https://arxiv.org/html/2607.28638#S2.F2), which strongly validates our predictive knowledge hypothesis:

Robustness to Stochasticity \(Overcoming Hindsight Bias\):State\-level reflection always presents the best performance in all metrics and scenarios\. As shown in the qualitative example in Figure[2\(b\)](https://arxiv.org/html/2607.28638#S2.F2.sf2), trajectory critiques suffer from severe hindsight bias: they prematurely conclude that certain movements are deterministic after a single lucky success, leading to brittle policies\. Conversely, stateful reflection successfully predicts and uncovers the asymmetric slipping dynamics\.

Generalization and Transfer \(State vs\. Path\):When transferred to a newly generated testing map after 10 iterations, all agents experience a performance drop\. However, on the tweaked scenario, trajectory\-level reflection shows a drastic drop \(\-33%\) while stateful knowledge drop is mild \(\-16%\)\. This indicates that state\-specific predictive insights \(e\.g\., "slipping only occurs to the right from this type of tile"\) are fundamentally more generalizable than the trajectory\-based baseline\.

![Refer to caption](https://arxiv.org/html/2607.28638v1/x4.png)Figure 3:Left: An example reasoning trace from the fully bootstrapping variant at the final iteration\. The agent defines and utilizesrecoverabilityfor reasoning\.Middle: The average frequency of the keywordrecoverabilityacross training iterations, showing a steady upward trend\.Right: The relative distribution of keywords along the trajectory horizon; notably, the occurrence of the keywordirrecoverableshifts backward from terminal states toward the initial state as learning progresses\.
### 2\.2Knowledge bootstrapping

Another benefit of stateful knowledge is knowledge bootstrapping\. This key idea shares similarity with temporal difference learning \(TD\)\(Suttonet al\.,[1998](https://arxiv.org/html/2607.28638#bib.bib3)\): because of the temporal relation between states \(e\.g\.,st,at→st\+1s\_\{t\},a\_\{t\}\\rightarrow s\_\{t\+1\}\), an agent can update its current assessment onsts\_\{t\}based on the knowledge of successor statesst\+1s\_\{t\+1\}\. Maintaining stateful knowledge enables the propagation of semantic insights across the temporal axis of interaction – a feature trajectory reflection does not have\.

To demonstrate this, we deploy aFully Bootstrapping Agentin the FrozenLake environment, where the knowledge of all visited states in a trajectory is updated\. The aim is to show how knowledge can propagate across all the visited states in the trajectory \(workflow also illustrated in Figure[1](https://arxiv.org/html/2607.28638#S1.F1)\)\. Now the agent updates recursively backwards from the terminal statesTs\_\{T\}to the root states0s\_\{0\}, thinking based on the concatenation of context:

\[st,\{τnH\}n=1N,zst\+H,zst\]\[s\_\{t\},\\\{\\tau\_\{n\}^\{H\}\\\}\_\{n=1\}^\{N\},z\_\{s\_\{t\+H\}\},z\_\{s\_\{t\}\}\]The context includes the current state, aggregated local experience over horizonHHand the pre\-existing stateful knowledge of the destination statest\+Hs\_\{t\+H\}and current state\. This setup encourages the agent to comprehend its own self\-generated intermediate assessments and adjust its current understanding accordingly, achieving temporal consistency of knowledge\. For instance, if an action leads to a successor statezst\+Hz\_\{s\_\{t\+H\}\}that the agent already recognizes as high\-risk, it must updatezstz\_\{s\_\{t\}\}to reflect this derived danger, effectively "thinking ahead" byHHsteps\. Figure[3](https://arxiv.org/html/2607.28638#S2.F3)illustrates the performance of the fully bootstrapping agent with a look\-ahead horizonH=1H=1, providing a clear visualization of how predictive knowledge propagates temporally across states\.

Emergent Semantic Bootstrapping\.The fully bootstrapping agent reveals an emergent semantic bootstrapping behavior\. We show an example of such emergence in Figure[3](https://arxiv.org/html/2607.28638#S2.F3), tracing how the agent develops a strategic concept calledrecoverabilityduring training, which is an emergent concept defined by the agent itself, denoting if there exists a safe policy from the state to the goal\. As the agent iterates, the keyword "recoverable" \(and its variants\) occurs with increasing frequency, indicating the consolidation of this abstract concept\. Crucially, as also observed in Figure[3](https://arxiv.org/html/2607.28638#S2.F3), the relative position of the assertive keywordirrecoverablestarts from terminal states, then gradually shifts back toward the root state\. This trend confirms that the agent is successfully discovering critical bottlenecks and propagating consistently that information backward through its natural language knowledge base, allowing it to preemptively identify fatal sequences long before they conclude\.

## 3Training LLM agent to extract and learn from stateful knowledge

The temporal consistency and robustness of Knowledge Bootstrapping \(Section[2\.2](https://arxiv.org/html/2607.28638#S2.SS2)\) enable us to scale Stateful Knowledge Learning on more complex tasks, such as with more complex state representation or longer horizons\. Since current off\-the\-shelf models are typically not trained to be state\-aware\(Shojaeeet al\.,[2025](https://arxiv.org/html/2607.28638#bib.bib20)\), we aim to internalize this capability into the model’s parameters\. Revealed by previous experiments, the key to a Stateful Knowledge Learning \(SKL\) agent is a system that learns to iteratively refine its stateful knowledgezzby repeating the following two steps:

1\. Extracting stateful knowledge from experience: Aggregates state\-specific outcomes across multiple visits \(with budgetN,HN,H\):\{τnH​\(st\)\}n=1N=\{\(st,zst,at\(n\),st\+1\(n\),…,st\+H\(n\)\)\}n=1N\\\{\\tau\_\{n\}^\{H\}\(s\_\{t\}\)\\\}\_\{n=1\}^\{N\}=\\\{\(s\_\{t\},z\_\{s\_\{t\}\},a^\{\(n\)\}\_\{t\},s^\{\(n\)\}\_\{t\+1\},\.\.\.,s\_\{t\+H\}^\{\(n\)\}\)\\\}\_\{n=1\}^\{N\}as informative contexts\. Higher budgets increase therepresentational power\. A bootstrapping loop then extracts new knowledgez^st\\hat\{z\}\_\{s\_\{t\}\}based on the aggregated experience as well as the evaluative signals of its successor statezst\+Hz\_\{s\_\{t\+H\}\}, acting as atemporal regularizerthat ensures assessments are grounded in the agent’s own predictive logic\. We follow the idea of knowledge bootstrapping \(Section[2\.2](https://arxiv.org/html/2607.28638#S2.SS2)\) for the remaining parts of this paper \(by settingH<TH<T\)\.

2\. Internalize stateful knowledge to guide policy: The extracted knowledge is internalized and leveraged to guide policy optimization\. While Section[2](https://arxiv.org/html/2607.28638#S2)utilizes stateful knowledge as a plug\-in context for inference, we propose more scalable variants to inject stateful knowledge into model parameters\.

We introduce two primary training variants that perform steps 1 and 2 differently:SKL Self\-Distillation \(SKL\-SD\)andSKL Reinforcement Learning \(SKL\-RL\), see Figure[4](https://arxiv.org/html/2607.28638#S3.F4)for visualization\. The core distinction lies in the state aggregation method: SKL\-SD samples model\-free rollouts from replay buffer, whereas SKL\-RL employs online, model\(simulator\)\-based search within a single episode\. Detailed pseudocode is provided in Appendix[A](https://arxiv.org/html/2607.28638#A1), and a comparison with contemporary reflection\-based methods is available in Appendix[C](https://arxiv.org/html/2607.28638#A3)\.

### 3\.1State\-Knowledge\-Learning Self\-Distillation \(SKL\-SD\)

SKL\-SD extracts stateful knowledge through model\-free rollout and bootstrapping, internalizing state\-aware behavior via quality\-filtered self\-distillation\. Drawing inspiration from recent self\-reflection paradigms\(Zhanget al\.,[2025b](https://arxiv.org/html/2607.28638#bib.bib97); Shiet al\.,[2026b](https://arxiv.org/html/2607.28638#bib.bib91)\), we construct a multi\-faceted dataset𝒟SKL\-SD=\[𝒟base,𝒟distill,𝒟reflect\]\\mathcal\{D\}\_\{\\text\{SKL\-SD\}\}=\[\\mathcal\{D\}\_\{\\text\{base\}\},\\mathcal\{D\}\_\{\\text\{distill\}\},\\mathcal\{D\}\_\{\\text\{reflect\}\}\]generated through four iterative steps:

I\. Model\-free Stateful Rollout: The agent perform knowledge\-grounded reasoning to generate trajectoriesτ=\(s0,\(zs0,a0\),s1,\(zs1,a1\),…,sT,R\)\\tau=\(s\_\{0\},\(z\_\{s\_\{0\}\},a\_\{0\}\),s\_\{1\},\(z\_\{s\_\{1\}\},a\_\{1\}\),\.\.\.,s\_\{T\},R\)\. Successful trajectories are added to𝒟base\\mathcal\{D\}\_\{\\text\{base\}\}, while all trajectories are stored in an experience bufferℬ\\mathcal\{B\}for subsequent state aggregation\.

II\. Bootstrapping Loop with a Replay Buffer: On failed trajectories, the agent initiates a backward bootstrapping loop fromsT−1s\_\{T\-1\}back tos0s\_\{0\}\. It sequentially updates the state knowledge by aggregating outcomes from the experience buffer\{τnH​\(st\)\}n=1N∼ℬ\\\{\\tau\_\{n\}^\{H\}\(s\_\{t\}\)\\\}\_\{n=1\}^\{N\}\\sim\\mathcal\{B\}as well as the successor assessments to construct informative context𝒞t\\mathcal\{C\}\_\{t\}:

𝒞t=\[st,\{τnH​\(st\)\}n=1N,zst,zst\+H\]\\mathcal\{C\}\_\{t\}=\[s\_\{t\},\\\{\\tau\_\{n\}^\{H\}\(s\_\{t\}\)\\\}\_\{n=1\}^\{N\},z\_\{s\_\{t\}\},z\_\{s\_\{t\+H\}\}\]
wherests\_\{t\}refers to the target state being evaluated,\{τnH​\(st\)\}n=1N\\\{\\tau\_\{n\}^\{H\}\(s\_\{t\}\)\\\}\_\{n=1\}^\{N\}refers to a set of aggregated partial rollouts sampled from the experience buffer, all originating atsts\_\{t\}, at bootstrapping budget\(H,N\)\(H,N\)\.zstz\_\{s\_\{t\}\}refers to the current knowledge \(prior belief\) associated with the target state, andzst\+Hz\_\{s\_\{t\+H\}\}represents the successor stateful knowledge, providing the bootstrapping signal fromHHsteps ahead111zst\+Hz\_\{s\_\{t\+H\}\}can be retrieved from experience𝝉\\bm\{\\tau\}or the updated knowledgez^st\+H\\hat\{z\}\_\{s\_\{t\+H\}\}within the same bootstrapping loop\. The agent then generates refined knowledgez^st∼πθ\(⋅\|𝒞t\)\\hat\{z\}\_\{s\_\{t\}\}\\sim\\pi\_\{\\theta\}\(\\cdot\|\\mathcal\{C\}\_\{t\}\), resulting in a sequence𝒛^=\(z^s0,…,z^sT−1\)\\hat\{\\bm\{z\}\}=\(\\hat\{z\}\_\{s\_\{0\}\},\.\.\.,\\hat\{z\}\_\{s\_\{T\-1\}\}\)\.

III\. Knowledge Verification: We verify𝒛^\\hat\{\\bm\{z\}\}by performing a second rollout using the refined knowledge as a pre\-filled context\. Successful second rollouts will be added to𝒟distill\\mathcal\{D\}\_\{\\text\{distill\}\}and the corresponding state\-wise update𝒛→𝒛^\\bm\{z\}\\rightarrow\\hat\{\\bm\{z\}\}correcting the first failure is recorded in𝒟reflect\\mathcal\{D\}\_\{\\text\{reflect\}\}\.

IV\. Internalize Knowledge to Guide Policy: The agent is fine\-tuned by maximizing the likelihood of these traces using policy gradients loss \(e\.g\., GRPO\(Guoet al\.,[2025a](https://arxiv.org/html/2607.28638#bib.bib11)\)\) on the full trajectory\. We also incorporate an auxiliary SFT loss to enhance state\-wise bootstrapping capabilities:

ℒSKG\-SD=ℒPG​\(𝒟base\)\+λ1⋅ℒPG​\(𝒟distill\)\+λ2⋅ℒSFT​\(𝒟reflect\)\\mathcal\{L\}\_\{\\text\{SKG\-SD\}\}=\\mathcal\{L\}\_\{\\text\{PG\}\}\(\\mathcal\{D\}\_\{\\text\{base\}\}\)\+\\lambda\_\{1\}\\cdot\\mathcal\{L\}\_\{\\text\{PG\}\}\(\\mathcal\{D\}\_\{\\text\{distill\}\}\)\+\\lambda\_\{2\}\\cdot\\mathcal\{L\}\_\{\\text\{SFT\}\}\(\\mathcal\{D\}\_\{\\text\{reflect\}\}\)\(1\)
Intuitively, SKL\-SD can be viewed as a “stateful” version of recent self\-reflection training methods with data filtering \(Appendix[C](https://arxiv.org/html/2607.28638#A3)\), aiming to maximize the token probability of the first and the second stateful rollout if they hit the target with or without knowledge bootstrapping, and reinforce the bootstrapping behaviour itself if it successfully helps the agent correct its mistake\. We name it "Self\-Distillation" \(or to be more precise "Self\-Imitation"\) as it essentially performs supervised learning over heuristically quality\-filtered experience\.

![Refer to caption](https://arxiv.org/html/2607.28638v1/x5.png)Figure 4:Illustration of two training variants: SKL\-SD and SKL\-RL\.
### 3\.2State\-Knowledge\-Learning Reinforcement Learning \(SKL\-RL\)

SKL\-RL serves as the online counterpart to SKL\-SD, interacting with the simulatorEsimE^\{\\text\{sim\}\}and extracting stateful knowledge entirely within an online rollout episode, linking the quality of the bootstrapping loop directly to the task outcome\. In this variant, the agent is assumed to have access to a state\-settable environmental simulator, performinglocalized parallel simulations\(Appendix[D](https://arxiv.org/html/2607.28638#A4)\) at each statests\_\{t\}to collectreal\-timeaggregated experiences before synthesizing updated knowledge and committing to an action\.

I\. & II\. Online Model\-based Rollout and Bootstrapping Loop: At each environmental time\-steptt, the agent first interacts with simulatorEsimE^\{\\text\{sim\}\}and generates multiple look\-ahead rollouts𝝉NH​\(st\)=\{st,\(zst,at\(n\)\),st\+1\(n\),…,st\+H\(n\)\}n=1N\\bm\{\\tau\}\_\{N\}^\{H\}\(s\_\{t\}\)=\\\{s\_\{t\},\(z\_\{s\_\{t\}\},a^\{\(n\)\}\_\{t\}\),s^\{\(n\)\}\_\{t\+1\},\.\.\.,s^\{\(n\)\}\_\{t\+H\}\\\}\_\{n=1\}^\{N\}\. These trajectories, operating under a bootstrapping budget of\(H,N\)\(H,N\), are all rooted at the current statests\_\{t\}and can be sampled via heuristic policies or from the agent’s own knowledge\-based understanding\. These localized simulations provide the necessary context for a real\-time bootstrapping step, synthesizing refined state knowledgez^st\\hat\{z\}\_\{s\_\{t\}\}to ground the final action selectionata\_\{t\}for the real environment:

z^st∼πθ\(⋅\|𝝉NH\(st\),st,zst\),at∼πθ\(⋅\|z^st,st\),t→t\+1\\hat\{z\}\_\{s\_\{t\}\}\\sim\\pi\_\{\\theta\}\(\\cdot\|\\bm\{\\tau\}\_\{N\}^\{H\}\(s\_\{t\}\),s\_\{t\},z\_\{s\_\{t\}\}\)\\,,\\,\\,a\_\{t\}\\sim\\pi\_\{\\theta\}\(\\cdot\|\\hat\{z\}\_\{s\_\{t\}\},s\_\{t\}\)\\;,\\;\\;t\\rightarrow t\+1\(2\)Appendix[D](https://arxiv.org/html/2607.28638#A4)and Figure[8](https://arxiv.org/html/2607.28638#A7.F8)provides a concrete example for illustration\. The complete simulation\-augmented trajectory is then recorded as:

…→st→Parallel Simulations \(conditioned onzst\)→z^st→at→st\+1→…\.\.\.\\rightarrow s\_\{t\}\\rightarrow\\text\{Parallel Simulations \(conditioned on $z\_\{s\_\{t\}\}$\)\}\\rightarrow\\hat\{z\}\_\{s\_\{t\}\}\\rightarrow a\_\{t\}\\rightarrow s\_\{t\+1\}\\rightarrow\.\.\.𝒟SKL\-RL←\[s0,\(𝝉NH​\(s0\),z^s0,a0\),s1,\(𝝉NH​\(s1\),z^s1,a1\),s2​…,sT,R\]\\mathcal\{D\}\_\{\\text\{SKL\-RL\}\}\\leftarrow\[s\_\{0\},\(\\bm\{\\tau\}\_\{N\}^\{H\}\(s\_\{0\}\),\\hat\{z\}\_\{s\_\{0\}\},a\_\{0\}\),s\_\{1\},\(\\bm\{\\tau\}\_\{N\}^\{H\}\(s\_\{1\}\),\\hat\{z\}\_\{s\_\{1\}\},a\_\{1\}\),s\_\{2\}\.\.\.,s\_\{T\},R\]\(3\)
III\. Outcome\-Driven Verification: Unlike the rejection sampling in SKL\-SD, the correctness of the bootstrapping step is implicitly supervised by the final rewardRR\. We optimize these interconnected bootstrapping loops as a long model response using GRPO\.

IV\. Knowledge Internalization: To consolidate the simulation\-derived insights into the agent’s parametric memory, we maintain a distillation dataset𝒟distill=\{\(zst,z^st\)\}st∈𝒮\\mathcal\{D\}\_\{\\text\{distill\}\}=\\\{\(z\_\{s\_\{t\}\},\\hat\{z\}\_\{s\_\{t\}\}\)\\\}\_\{s\_\{t\}\\in\\mathcal\{S\}\}that pairs the prior stateful knowledgezstz\_\{s\_\{t\}\}with its refined, post\-simulation counterpartz^st\\hat\{z\}\_\{s\_\{t\}\}\. This optimization step enables the agent to internalize look\-ahead foresight, allowing it to build upon updated knowledge to guide rollouts in the subsequent iteration:

…→st→Parallel Simulations \(conditioned on​z^st​\)→z^st′→…\\dots\\rightarrow s\_\{t\}\\rightarrow\\text\{Parallel Simulations \(conditioned on \}\\hat\{z\}\_\{s\_\{t\}\}\\text\{\)\}\\rightarrow\\hat\{z\}\_\{s\_\{t\}\}^\{\\prime\}\\rightarrow\\dots
Further details regarding the multi\-turn distillation mechanics are provided in Appendix[E](https://arxiv.org/html/2607.28638#A5)\. The joint optimization objective for SKL\-RL is formulated as:

ℒSKL\-RL=ℒGRPO​\(𝒟SKL\-RL\)\+λ⋅ℒDistill​\(\(zs,z^s\)s∈𝒮\)\\mathcal\{L\}\_\{\\text\{SKL\-RL\}\}=\\mathcal\{L\}\_\{\\text\{GRPO\}\}\(\\mathcal\{D\}\_\{\\text\{SKL\-RL\}\}\)\+\\lambda\\cdot\\mathcal\{L\}\_\{\\text\{Distill\}\}\(\(z\_\{s\},\\hat\{z\}\_\{s\}\)\_\{s\\in\\mathcal\{S\}\}\)\(4\)
Intuitively, SKL\-RL drives the agent to verify its stateful knowledge hypothesis through active simulation, extracting insights from these localized experiences to ground an exploitative action for a specific state\. The inclusion of the distillation objective allows the agent to amortize the extensive computational cost of the look\-ahead search and bootstrapping loop into an efficient, zero\-shot inference step\. In contrast toSKL\-SD, which depends on heavily orchestrated data curation and verification pipelines,SKL\-RLprovides a fully autonomous learning paradigm\. By optimizing directly from sparse environmental rewards, it completely bypasses the bottleneck of manual data curation\.

Crucially, when the simulation traces𝝉NH\\bm\{\\tau\}\_\{N\}^\{H\}are also self\-generated through multi\-turn rollouts, SKL\-RL uniquely couples exploration \(via simulation\) and exploitation \(via knowledge bootstrapping\)\. Optimizing the entire interaction trajectory viaℒGRPO\\mathcal\{L\}\_\{\\text\{GRPO\}\}induces an emergent,self\-adaptive balance: excessive exploration introduces behavioral noise that destabilizes the bootstrapping signal, whereas overly conservative simulation limits the agent’s foresight, leading to myopia\. Our chess reasoning evaluation \(Section[4\.2](https://arxiv.org/html/2607.28638#S4.SS2)\) empirically confirms this dynamic, showing that the agent naturally resolves this tension to find a stable operational equilibrium\.

## 4Experiments

We first evaluate SKL\-SD on established agentic benchmarks \(WebShop and ScienceWorld\) to demonstrate that stateful knowledge learning can be integrated into existing training frameworks to outperform current state\-of\-the\-art reflection training methods\. Subsequently, we explore SKL\-RL on ChessPuzzles, a complex reasoning task that suffers less from data contamination and requires deep look\-ahead and strategic state assessments\.

### 4\.1SKL\-SD on agentic environments

Experiment Setup\.We compare SKL\-SD to other training baselines investigated in\(Shiet al\.,[2026b](https://arxiv.org/html/2607.28638#bib.bib91)\)and follow their exact experiment setups\. While SKL\-SD shares the feedback\-learning DNA of these methods \(Appendix[C](https://arxiv.org/html/2607.28638#A3)\), it replaces trajectory\-level reflection with state\-grounded predictive knowledge bootstrapping\. Detailed setup is in Appendix[F\.1](https://arxiv.org/html/2607.28638#A6.SS1)\. The bootstrapping parameters are set to a horizonH=3H=3and a state\-aggregation budgetN=3N=3\.Qwen2\.5\-7B\-Instructis adopted as the backbone model\.

##### Results

Performance is reported in Table[1](https://arxiv.org/html/2607.28638#S4.T1)\. On Webshop, SKL\-SD reaches 0\.790, improving over the strongest baseline R3L \(0\.757\) by \+3\.3 absolute points, and substantially outpacing standard GRPO \(0\.709\) as well as other reflection\-based methods such as Reflect\-GRPO \(0\.723\) and Critique\-GRPO \(0\.714\)\. In the more difficult ScienceWorld environment, SKL\-SD again achieves the best result \(0\.422 vs\. 0\.403 for R3L and 0\.388 for Critique\-GRPO\)\. Not as impressive as in WebShop, largely due to the partial observability\.

Table 1:Performance comparison on WebShop \(WebS\.\) and ScienceWorld \(SciW\.\)\. Baseline performance borrowed from\(Shiet al\.,[2026b](https://arxiv.org/html/2607.28638#bib.bib91)\)These consistent gains demonstrate thatreplacing trajectory\-level reflection with stateful knowledge learning yields tangible benefits\. A review of the reasoning traces \(see Appendix[F\.2](https://arxiv.org/html/2607.28638#A6.SS2)\) reveals that the SKL\-SD agent develops a structured, hierarchical understanding of the task\. For example in WebShop, at the initial search stage, the agent generates precise notes on required attributes and formulates a contingency plan for potential search failures—a behavior rarely seen in the other tested trajectory\-level training baselines\. During the purchasing phase, the predictive knowledgezzshifts to specific product verification, demonstrating that stateful learning helps the agent maintain a "mental map" of the task progress, leading to more grounded decision\-making\.

##### Practical Consideration

While SKL\-SD introduces additional computational overhead, primarily due to longer model outputs during stateful rollouts and context prefilling for bootstrapping, we employ a sampling strategy to maintain training efficiency\. By bootstrapping on only20%20\\%of states uniformly sampled across a trajectory, we limit the increase in training time to approximately1\.5×1\.5\\timescompared to standard reflection baselines\. Given the performance gains, this represents a favorable trade\-off between compute and capability\. We leave further computationally efficient implementations to future exploration\.

### 4\.2SKL\-RL on ChessPuzzles

We next evaluate how SKL\-RL can operate in a more autonomous way without complex data orchestration\. We test on ChessPuzzles\(Ruosset al\.,[2024](https://arxiv.org/html/2607.28638#bib.bib131)\), a domain where even the strongest LLMs fail due to incomplete internalized chess knowledge\(Liuet al\.,[2025](https://arxiv.org/html/2607.28638#bib.bib105); Hwanget al\.,[2025](https://arxiv.org/html/2607.28638#bib.bib26)\)\. This limited prior knowledge setting is especially diagnostic for our framework, as it directly tests the agent’s ability to learn from experience\. More details on our motivation can be found in Appendix[G\.1](https://arxiv.org/html/2607.28638#A7.SS1)\. And the experimental setup for ChessPuzzle is detailed in Appendix[G\.2](https://arxiv.org/html/2607.28638#A7.SS2)\.

#### 4\.2\.1Experiments with heuristic parallel simulation

To isolate whether the bootstrapping loop can be learned from outcome signals alone, we control simulation quality usingStockfishchess engine\(Romstadet al\.,[2024](https://arxiv.org/html/2607.28638#bib.bib99)\)\. We construct behavior policies by mixing expert moves with random sub\-optimal moves at expert probabilities of12,13\\frac\{1\}\{2\},\\frac\{1\}\{3\}and16\\frac\{1\}\{6\}\. Figure[5](https://arxiv.org/html/2607.28638#S4.F5)shows that higher expert rates yield faster learning and stronger asymptotic performance\. The agent progressively learns to identify winning positions during simulation and steer toward them, with win probability in simulated trajectories rising accordingly\. Response length correlates positively with the difficulty of detecting expert moves, indicating that noisier simulation drives the agent to invest more internal reasoning to extract state\-grounded predictions\. Detailed traces \(Appendix[G\.3](https://arxiv.org/html/2607.28638#A7.SS3)\) further reveal that under high signal \(i\.e\. export move1/21/2\), knowledge bootstrapping converges rapidly to accurate but shallow assessments; whereas under low signal \(i\.e\. random\), it remains structured but increasingly prone to hallucination\. This is a direct manifestation of the exploration–exploitation trade\-off discussed in Section[3\.2](https://arxiv.org/html/2607.28638#S3.SS2)\.

These results establish a fundamental coupling between aggregated experience quality and bootstrapping behavior: better trajectories yield better state knowledge, and the agent’s ability to extract predictive insights tracks this quality closely\.When the agent must actively decide what futures to simulate, it is continually forced to balance exploring uncertain branches against exploiting known strong lines—a trade\-off it learns to navigate autonomously\.We will confirmed this in our next experiments\.

![Refer to caption](https://arxiv.org/html/2607.28638v1/x6.png)Figure 5:Experiments of SKL\-RL with heuristic search simulation on ChessPuzzles\.
#### 4\.2\.2Experiments with self\-generated parallel simulation

We now let the agent decide what to simulate by itself, jointly optimizing experience generation and the bootstrapping loop end\-to\-end\. A complete reasoning trace for SKL\-RL under self\-generated simulation is provided in Appendix[G\.4](https://arxiv.org/html/2607.28638#A7.SS4)\. We use SKL\-RL with parallel search at a budget ofN=3,H=2N=3,H=2, denoted asSKL\-RLN​3​H​2\\text\{SKL\-RL\}\_\{N3H2\}\. GRPO is applied for training on the full SKL\-RL track under lossℒGRPO​\(𝒟SKL\-RL\)\\mathcal\{L\}\_\{\\text\{GRPO\}\}\(\\mathcal\{D\}\_\{\\text\{SKL\-RL\}\}\)\.

Figure[6](https://arxiv.org/html/2607.28638#S4.F6)compares SKL\-RL against existing training baselines under matched training budgets\. SKL\-RLN3H2trained end\-to\-end achieves the strongest performance throughout training and the highest test\-time few\-shot accuracy\. In contrast, naive GRPO and Reflect\-GRPO\(Bensalet al\.,[2025](https://arxiv.org/html/2607.28638#bib.bib98)\)plateau on the training set and show noticeable performance degradation at test time\. Critic\-GRPO\(Zhanget al\.,[2025b](https://arxiv.org/html/2607.28638#bib.bib97)\)suffers from training instability, leading to collapse\. Although R3L achieves the best performance among the remaining baselines, it still exhibits a slight drop on the held\-out test set\.

![Refer to caption](https://arxiv.org/html/2607.28638v1/x7.png)Figure 6:SKL\-RL training with self\-generated parallel simulation, evaluated at matched budgets\.Both the diversity of simulated moves and the response length in the bootstrapping loop undergo clear fluctuations\. The sharp drop of both metrics at around training step7575suggests an early synchronized adjustment between diversity and exploitation, while the gradual increase after step 75 for both move diversity and response length indicates that the agent automatically allocates a larger reasoning budget to support broader exploration, enabling more effective extraction of information from simulated experience\. This empirical result directly echoes our previous assumption about the emergent self\-adaptive behaviour during end\-to\-end training\. When comparing trained reasoning traces \(Appendix[G\.5](https://arxiv.org/html/2607.28638#A7.SS5)\), SKL\-RL produces more outcome\-oriented understanding than R3L\. The agent moves away from chess\-specific clichés and instead grounds decisions in implicit evaluative signals such as material balance and concrete tactics, while trajectory\-level reflection remains coarser and path\-dependent\.

#### 4\.2\.3Experiments with stateful knowledge distillation

We now examine how state\-knowledge distillation guides policy learning, thereby closing the learning loop in SKL\-RL\. The self\-distillation lossℒDistill​\(\{zs,z^s\}s∈𝒮\)\\mathcal\{L\}\_\{\\text\{Distill\}\}\(\\\{z\_\{s\},\\hat\{z\}\_\{s\}\\\}\_\{s\\in\\mathcal\{S\}\}\)internalizes post\-hoc stateful knowledge into the prior state understanding, enabling the agent to anticipate predictive knowledge for future training cycles or to directly produce sophisticated reasoning plans at test time\. To isolate its effect while reducing computational cost, we shorten the ChessPuzzles episode length to 5 and ablate the loss coefficientλ\\lambdaand the update intervalδn\\delta\_\{n\}, where distillation is applied everynntraining steps \(Figure[7](https://arxiv.org/html/2607.28638#S4.F7)\)\.

We first observe that removingℒDistill\\mathcal\{L\}\_\{\\text\{Distill\}\}leads to noticeable performance degradation, consistent with the trend in Figure[6](https://arxiv.org/html/2607.28638#S4.F6), and that this instability worsens as episode length decreases\. In contrast, increasingλ\\lambdaand applying distillation updates more frequently markedly stabilizes training\. Similar to recent works, we also find that the reverse KL, a multi\-turn variant ofOn\-Policy Self\-Distillation\(Zhaoet al\.,[2026](https://arxiv.org/html/2607.28638#bib.bib113); Hübotteret al\.,[2026](https://arxiv.org/html/2607.28638#bib.bib114)\), provides greater stability than the forward KL setting\.

![Refer to caption](https://arxiv.org/html/2607.28638v1/x8.png)Figure 7:Ablation study on SKL\-RL with integrated training loss\.These results suggest thatℒDistill\\mathcal\{L\}\_\{\\text\{Distill\}\}helps stabilize the joint optimization of this compositional reasoning process\. As shown by the reasoning traces in Appendix[G\.6](https://arxiv.org/html/2607.28638#A7.SS6), the SKL\-RL agentupdates stateful predictive knowledge by building upon previously distilled information\. Specifically, it treatsz^\\hat\{z\}from the prior learning epoch as a condensed representation, reusing it to empower the agent to explore and exploit more effectively via continuously accumulated stateful knowledge\.

The SKL\-RL framework exhibits several compelling properties: it significantly reduces reliance on human intervention and complex pipeline engineering, enables self\-adaptive behaviors to emerge purely from end\-to\-end training, and demonstrates a distinct capacity to build new knowledge incrementally on top of prior iterations\. Despite these strengths, in its minimal implementation, SKL\-RL faces severe practical limitations in computational efficiency relative to current state\-of\-the\-art self\-evolving agents\. This is primarily due to the substantial computational cost of state\-wise simulation, which leads to overly long training samples and scales exponentially for longer\-horizon tasks\. Nevertheless, this exploration validates our initial vision for next\-generation autonomous agents: systems capable of naturally extracting predictive information from environmental dynamics, absorbing it into a persistent internal "mental map," and dynamically balancing exploration and exploitation in accordance with their own evolving capabilities\.

## 5Related works

##### Trajectory\-level Self\-Reflection

Current research on LLM\-based agents has focused extensively on trajectory\-level self\-reflection, where agents optimize their reasoning and action sequences based on feedback from a single execution path\. Classic examples include early paradigms such as ReAct\(Yaoet al\.,[2022b](https://arxiv.org/html/2607.28638#bib.bib94)\), Reflexion\(Shinnet al\.,[2023](https://arxiv.org/html/2607.28638#bib.bib76)\), and Self\-Refine\(Madaanet al\.,[2023](https://arxiv.org/html/2607.28638#bib.bib78)\)\. Some recent fine\-tuning methods also value trajectory\-level assessment, but focus on how to internalize the transition from bad action to good action, largely remaining action\-centric\(Wanget al\.,[2025](https://arxiv.org/html/2607.28638#bib.bib1); Sunet al\.,[2023](https://arxiv.org/html/2607.28638#bib.bib77); Wanet al\.,[2025](https://arxiv.org/html/2607.28638#bib.bib16); Zhanget al\.,[2025b](https://arxiv.org/html/2607.28638#bib.bib97); Bensalet al\.,[2025](https://arxiv.org/html/2607.28638#bib.bib98); Shiet al\.,[2026b](https://arxiv.org/html/2607.28638#bib.bib91)\)\. This has raised the issue of myopic reasoning, which was also analysed in\(Wanget al\.,[2026b](https://arxiv.org/html/2607.28638#bib.bib121)\)\. This pervasive reliance on myopic, trajectory\-level evaluation motivates us to move beyond isolated execution paths and explore stateful reflection\.

##### Spirit of statefulness in other LLM agents

Being stateful, or outcome\-aware, is one way to alleviate the issues of myopic\(Wanget al\.,[2026b](https://arxiv.org/html/2607.28638#bib.bib121)\)\. This spirit has been widely adopted in various works, but in different forms\. For example, world model\(Yuet al\.,[2025](https://arxiv.org/html/2607.28638#bib.bib29); Guoet al\.,[2025b](https://arxiv.org/html/2607.28638#bib.bib45); Zhanget al\.,[2025a](https://arxiv.org/html/2607.28638#bib.bib57); Chuet al\.,[2026](https://arxiv.org/html/2607.28638#bib.bib122)\)allows the agent to "dream" or rehearse trajectories before committing to an action\. Relying on heuristic search is also a straightforward method for obtaining extra understanding of the state outcome\([Wuet al\.,](https://arxiv.org/html/2607.28638#bib.bib123); Liuet al\.,[2026](https://arxiv.org/html/2607.28638#bib.bib124); Wanget al\.,[2026b](https://arxiv.org/html/2607.28638#bib.bib121)\)\. Structured memory systems provide another avenue for maintaining statefulness\(Wanget al\.,[2026a](https://arxiv.org/html/2607.28638#bib.bib132); Zhouet al\.,[2025](https://arxiv.org/html/2607.28638#bib.bib133); Wang,[2025](https://arxiv.org/html/2607.28638#bib.bib134); Zhouet al\.,[2026](https://arxiv.org/html/2607.28638#bib.bib135)\)\. A more general abstraction is to summarize experience via a numerical state\-action value estimateQQ, whereQQ\-value captures the optimality of actions in a given state\(Yanet al\.,[2024](https://arxiv.org/html/2607.28638#bib.bib61); Zhanget al\.,[2023](https://arxiv.org/html/2607.28638#bib.bib86); Zhaiet al\.,[2025](https://arxiv.org/html/2607.28638#bib.bib66); Yanet al\.,[2025](https://arxiv.org/html/2607.28638#bib.bib60); Zhanget al\.,[2026](https://arxiv.org/html/2607.28638#bib.bib81); Wanget al\.,[2026b](https://arxiv.org/html/2607.28638#bib.bib121)\)\. However, these value\-oriented representations rely on an auxiliaryQQfunction to assess the outcome quality, rather than reflecting how the agent itself internally judges its action\. Our stand is that the agent’s declarative assessments for the state already contain judgment of the possible outcome of each candidate action, and such assessments can be bootstrapped in the same way as the numericalQQ\-function\. This is analogous to having alanguage value functionwhich was first proposed by\(Fenget al\.,[2024](https://arxiv.org/html/2607.28638#bib.bib10)\), andStateful Knowledge Learningcan be seen as a more general extension of it\.

## 6Conclusions and limitation

In this paper, we have challenged the prevailing reliance on trajectory\-level reflection in LLM agent training\. To address this, we introducedStateful Knowledge Learning \(SKL\), a framework that centers on maintaining explicit, declarative predictive assessments anchored to specific states\. Our motivating examples confirmed that stateful knowledge provides finer granularity and generalization, and also enables knowledge bootstrapping that scales SKL to increasingly complex tasks\. Experiments validate how SKL training variantsSKL\-SDandSKL\-RLcan yield significant performance gains and move toward a more autonomous, self\-evolving agent learning paradigm\.

As LLM agents transition further into the "Era of Experience," moving beyond static datasets toward continuous environmental interaction, the ability to maintain a stateful "mental map" of predictive foresight will be critical\. While our current implementation introduces a trade\-off in training compute, the resulting gains in policy robustness and strategic depth suggest that stateful learning is a more principled foundation for autonomous agents\. Future work will explore the practical implementation to resolve the computational limitation raised in the paper and the potential for transferring stateful predictive knowledge across heterogeneous task domains\.

## References

- On\-policy distillation of language models: learning from self\-generated mistakes\.InThe twelfth international conference on learning representations,Cited by:[Appendix E](https://arxiv.org/html/2607.28638#A5.p3.1)\.
- S\. Bensal, U\. Jamil, C\. Bryant, M\. Russak, K\. Kamble, D\. Mozolevskyi, M\. Ali, and W\. AlShikh \(2025\)Reflect, retry, reward: self\-improving llms via reinforcement learning\.arXiv preprint arXiv:2505\.24726\.Cited by:[Appendix C](https://arxiv.org/html/2607.28638#A3.p2.8),[§F\.1](https://arxiv.org/html/2607.28638#A6.SS1.p1.1),[§4\.2\.2](https://arxiv.org/html/2607.28638#S4.SS2.SSS2.p2.2),[§5](https://arxiv.org/html/2607.28638#S5.SS0.SSS0.Px1.p1.1)\.
- C\. Cheng, A\. Kolobov, D\. Misra, A\. Nie, and A\. Swaminathan \(2023\)Llf\-bench: benchmark for interactive learning from language feedback\.arXiv preprint arXiv:2312\.06853\.Cited by:[Appendix C](https://arxiv.org/html/2607.28638#A3.p1.1)\.
- M\. Chu, X\. B\. Zhang, K\. Q\. Lin, L\. Kong, J\. Zhang, T\. Tu, W\. Ma, Z\. Huang, S\. Yang, W\. Huang,et al\.\(2026\)Agentic world modeling: foundations, capabilities, laws, and beyond\.arXiv preprint arXiv:2604\.22748\.Cited by:[§5](https://arxiv.org/html/2607.28638#S5.SS0.SSS0.Px2.p1.4)\.
- H\. Dong, W\. Xiong, D\. Goyal, Y\. Zhang, W\. Chow, R\. Pan, S\. Diao, J\. Zhang, K\. Shum, and T\. Zhang \(2023\)Raft: reward ranked finetuning for generative foundation model alignment\.arXiv preprint arXiv:2304\.06767\.Cited by:[§F\.1](https://arxiv.org/html/2607.28638#A6.SS1.p1.1)\.
- X\. Feng, B\. Liu, Y\. Song, H\. Fu, Z\. Wan, G\. A\. Koushik, Z\. Hu, M\. Yang, Y\. Wen, and J\. Wang \(2024\)Natural language reinforcement learning\.arXiv preprint arXiv:2411\.14251\.Cited by:[Appendix C](https://arxiv.org/html/2607.28638#A3.p1.1),[§G\.1](https://arxiv.org/html/2607.28638#A7.SS1.p1.1),[§2\.1](https://arxiv.org/html/2607.28638#S2.SS1.p4.1),[§5](https://arxiv.org/html/2607.28638#S5.SS0.SSS0.Px2.p1.4)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, R\. Zhang, R\. Xu, Q\. Zhu, S\. Ma, P\. Wang, X\. Bi,et al\.\(2025a\)Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§F\.1](https://arxiv.org/html/2607.28638#A6.SS1.p1.1),[§3\.1](https://arxiv.org/html/2607.28638#S3.SS1.p6.1)\.
- S\. Guo, O\. D\. Domingues, R\. Avalos, A\. Courville, and F\. Strub \(2025b\)Sample, predict, then proceed: self\-verification sampling for tool use of llms\.arXiv preprint arXiv:2506\.02918\.Cited by:[§5](https://arxiv.org/html/2607.28638#S5.SS0.SSS0.Px2.p1.4)\.
- J\. Hübotter, F\. Lübeck, L\. Behric, A\. Baumann, M\. Bagatella, D\. Marta, I\. Hakimi, I\. Shenfeld, T\. K\. Buening, C\. Guestrin,et al\.\(2026\)Reinforcement learning via self\-distillation\.arXiv preprint arXiv:2601\.20802\.Cited by:[§4\.2\.3](https://arxiv.org/html/2607.28638#S4.SS2.SSS3.p2.2)\.
- D\. Hwang, H\. Lee, J\. Choo, D\. Park, and J\. Park \(2025\)Can large language models develop strategic reasoning? post\-training insights from learning chess\.arXiv preprint arXiv:2507\.00726\.Cited by:[§4\.2](https://arxiv.org/html/2607.28638#S4.SS2.p1.1)\.
- M\. Littman and R\. S\. Sutton \(2001\)Predictive representations of state\.Advances in neural information processing systems14\.Cited by:[§1](https://arxiv.org/html/2607.28638#S1.p2.1)\.
- A\. Liu, Z\. Gong, Y\. Song, Y\. Chen, X\. Liu, H\. Lu, K\. Zhang, and C\. Wei \(2026\)Active reasoning vision\-language models via sequential experimental design\.External Links:[Link](https://api.semanticscholar.org/CorpusID:287948114)Cited by:[§5](https://arxiv.org/html/2607.28638#S5.SS0.SSS0.Px2.p1.4)\.
- J\. Liu, S\. He, J\. Wu, X\. Wang, Y\. Chen, Z\. Kuang, S\. Bao, and Y\. Yao \(2025\)ChessArena: a chess testbed for evaluating strategic reasoning capabilities of large language models\.arXiv preprint arXiv:2509\.24239\.Cited by:[§G\.1](https://arxiv.org/html/2607.28638#A7.SS1.p1.1),[§4\.2](https://arxiv.org/html/2607.28638#S4.SS2.p1.1)\.
- A\. Madaan, N\. Tandon, P\. Gupta, S\. Hallinan, L\. Gao, S\. Wiegreffe, U\. Alon, N\. Dziri, S\. Prabhumoye, Y\. Yang,et al\.\(2023\)Self\-refine: iterative refinement with self\-feedback\.Advances in Neural Information Processing Systems36,pp\. 46534–46594\.Cited by:[§5](https://arxiv.org/html/2607.28638#S5.SS0.SSS0.Px1.p1.1)\.
- X\. Pan, Y\. Chen, Y\. Chen, Y\. Sun, D\. Chen, W\. Zhang, Y\. Xie, Y\. Huang, Y\. Zhang, D\. Gao,et al\.\(2025\)Trinity\-rft: a general\-purpose and unified framework for reinforcement fine\-tuning of large language models\.arXiv preprint arXiv:2505\.17826\.Cited by:[§F\.1](https://arxiv.org/html/2607.28638#A6.SS1.p1.1)\.
- T\. Romstad, M\. Costalba, J\. Kiiski, G\. Linscott, Y\. Nasu, M\. Isozaki, H\. Noda, and T\. S\. D\. Team \(2024\)StockfishExternal Links:[Link](https://stockfishchess.org/)Cited by:[§4\.2\.1](https://arxiv.org/html/2607.28638#S4.SS2.SSS1.p1.3)\.
- A\. Ruoss, G\. Delétang, S\. Medapati, J\. Grau\-Moya, L\. K\. Wenliang, E\. Catt, J\. Reid, C\. A\. Lewis, J\. Veness, and T\. Genewein \(2024\)Amortized planning with large\-scale transformers: a case study on chess\.Advances in Neural Information Processing Systems37,pp\. 65765–65790\.Cited by:[§4\.2](https://arxiv.org/html/2607.28638#S4.SS2.p1.1)\.
- T\. Shi, S\. Chen, B\. Jiang, L\. Song, L\. Yang, and J\. Zhao \(2026a\)Experiential reinforcement learning\.External Links:2602\.13949,[Link](https://arxiv.org/abs/2602.13949)Cited by:[Appendix C](https://arxiv.org/html/2607.28638#A3.p2.8)\.
- W\. Shi, Y\. Chen, Z\. Li, X\. Pan, Y\. Sun, J\. Xu, X\. Zhou, and Y\. Li \(2026b\)R3l: reflect\-then\-retry reinforcement learning with language\-guided exploration, pivotal credit, and positive amplification\.arXiv preprint arXiv:2601\.03715\.Cited by:[Appendix C](https://arxiv.org/html/2607.28638#A3.p2.8),[§F\.1](https://arxiv.org/html/2607.28638#A6.SS1.p1.1),[§1](https://arxiv.org/html/2607.28638#S1.p1.1),[§3\.1](https://arxiv.org/html/2607.28638#S3.SS1.p1.1),[§4\.1](https://arxiv.org/html/2607.28638#S4.SS1.p1.2),[Table 1](https://arxiv.org/html/2607.28638#S4.T1),[Table 1](https://arxiv.org/html/2607.28638#S4.T1.4.2),[§5](https://arxiv.org/html/2607.28638#S5.SS0.SSS0.Px1.p1.1)\.
- N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. Yao \(2023\)Reflexion: language agents with verbal reinforcement learning\.Advances in Neural Information Processing Systems36,pp\. 8634–8652\.Cited by:[§1](https://arxiv.org/html/2607.28638#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.28638#S2.SS1.p2.8),[§5](https://arxiv.org/html/2607.28638#S5.SS0.SSS0.Px1.p1.1)\.
- P\. Shojaee, I\. Mirzadeh, K\. Alizadeh, M\. Horton, S\. Bengio, and M\. Farajtabar \(2025\)The illusion of thinking: understanding the strengths and limitations of reasoning models via the lens of problem complexity\.arXiv preprint arXiv:2506\.06941\.Cited by:[§3](https://arxiv.org/html/2607.28638#S3.p1.1)\.
- M\. Shridhar, X\. Yuan, M\. Côté, Y\. Bisk, A\. Trischler, and M\. J\. Hausknecht \(2020\)ALFWorld: aligning text and embodied environments for interactive learning\.ArXivabs/2010\.03768\.External Links:[Link](https://api.semanticscholar.org/CorpusID:222208810)Cited by:[§G\.1](https://arxiv.org/html/2607.28638#A7.SS1.p1.1)\.
- D\. Silver and R\. S\. Sutton \(2025\)Welcome to the era of experience\.Google AI1\.Cited by:[§1](https://arxiv.org/html/2607.28638#S1.p1.1)\.
- H\. Sun, Y\. Zhuang, L\. Kong, B\. Dai, and C\. Zhang \(2023\)Adaplanner: adaptive planning from feedback with language models\.Advances in neural information processing systems36,pp\. 58202–58245\.Cited by:[§5](https://arxiv.org/html/2607.28638#S5.SS0.SSS0.Px1.p1.1)\.
- R\. S\. Sutton, A\. G\. Barto,et al\.\(1998\)Reinforcement learning: an introduction\.Vol\.1,MIT press Cambridge\.Cited by:[§2\.2](https://arxiv.org/html/2607.28638#S2.SS2.p1.3)\.
- R\. S\. Sutton, J\. Modayil, M\. Delp, T\. Degris, P\. M\. Pilarski, A\. White, and D\. Precup \(2011\)Horde: a scalable real\-time architecture for learning knowledge from unsupervised sensorimotor interaction\.InThe 10th international conference on autonomous agents and multiagent systems\-volume 2,pp\. 761–768\.Cited by:[§1](https://arxiv.org/html/2607.28638#S1.p2.1),[§1](https://arxiv.org/html/2607.28638#S1.p3.1)\.
- Z\. Wan, Y\. Li, X\. Wen, Y\. Song, H\. Wang, L\. Yang, M\. Schmidt, J\. Wang, W\. Zhang, S\. Hu,et al\.\(2025\)Rema: learning to meta\-think for llms with multi\-agent reinforcement learning\.arXiv preprint arXiv:2503\.09501\.Cited by:[§5](https://arxiv.org/html/2607.28638#S5.SS0.SSS0.Px1.p1.1)\.
- J\. Wang \(2025\)Memento\-ii: learning by stateful reflective memory\.arXiv preprint arXiv:2512\.22716\.Cited by:[§5](https://arxiv.org/html/2607.28638#S5.SS0.SSS0.Px2.p1.4)\.
- J\. Wang, R\. Zhao, W\. Wei, Y\. Wang, M\. Yu, J\. Zhou, J\. Xu, and L\. Xu \(2026a\)Comorag: a cognitive\-inspired memory\-organized rag for stateful long narrative reasoning\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 33557–33565\.Cited by:[§5](https://arxiv.org/html/2607.28638#S5.SS0.SSS0.Px2.p1.4)\.
- Z\. Wang, F\. Wu, H\. Wang, X\. Tang, B\. Li, Z\. Yin, Y\. Ma, Y\. Li, W\. Sun, X\. Chen,et al\.\(2026b\)Why reasoning fails to plan: a planning\-centric analysis of long\-horizon decision making in llm agents\.arXiv preprint arXiv:2601\.22311\.Cited by:[§5](https://arxiv.org/html/2607.28638#S5.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2607.28638#S5.SS0.SSS0.Px2.p1.4)\.
- Z\. Wang, K\. Wang, Q\. Wang, P\. Zhang, L\. Li, Z\. Yang, X\. Jin, K\. Yu, M\. N\. Nguyen, L\. Liu,et al\.\(2025\)Ragen: understanding self\-evolution in llm agents via multi\-turn reinforcement learning\.arXiv preprint arXiv:2504\.20073\.Cited by:[§5](https://arxiv.org/html/2607.28638#S5.SS0.SSS0.Px1.p1.1)\.
- \[32\]F\. Wu, W\. Xuan, H\. Qi, A\. Tu, X\. Lu, L\. E\. Li, and Y\. ChoiDeepSearch: overcome the bottleneck of reinforcement learning with verifiable rewards via tree\-based search\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§5](https://arxiv.org/html/2607.28638#S5.SS0.SSS0.Px2.p1.4)\.
- S\. Yan, X\. Yang, Z\. Huang, E\. Nie, Z\. Ding, Z\. Li, X\. Ma, K\. Kersting, J\. Z\. Pan, H\. Schütze,et al\.\(2025\)Memory\-r1: enhancing large language model agents to manage and utilize memories via reinforcement learning\.arXiv preprint arXiv:2508\.19828\.Cited by:[§5](https://arxiv.org/html/2607.28638#S5.SS0.SSS0.Px2.p1.4)\.
- X\. Yan, Y\. Song, X\. Feng, M\. Yang, H\. Zhang, H\. B\. Ammar, and J\. Wang \(2024\)Efficient reinforcement learning with large language model priors\.arXiv preprint arXiv:2410\.07927\.Cited by:[§5](https://arxiv.org/html/2607.28638#S5.SS0.SSS0.Px2.p1.4)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§2\.1](https://arxiv.org/html/2607.28638#S2.SS1.p5.4)\.
- A\. Yang, B\. Yang, B\. Hui, B\. Zheng, B\. Yu, C\. Zhou, C\. Li, C\. Li, D\. Liu, F\. Huang, G\. Dong, H\. Wei, H\. Lin, J\. Tang, J\. Wang, J\. Yang, J\. Tu, J\. Zhang, J\. Ma, J\. Xu, J\. Zhou, J\. Bai, J\. He, J\. Lin, K\. Dang, K\. Lu, K\. Chen, K\. Yang, M\. Li, M\. Xue, N\. Ni, P\. Zhang, P\. Wang, R\. Peng, R\. Men, R\. Gao, R\. Lin, S\. Wang, S\. Bai, S\. Tan, T\. Zhu, T\. Li, T\. Liu, W\. Ge, X\. Deng, X\. Zhou, X\. Ren, X\. Zhang, X\. Wei, X\. Ren, Y\. Fan, Y\. Yao, Y\. Zhang, Y\. Wan, Y\. Chu, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Fan \(2024\)Qwen2 technical report\.arXiv preprint arXiv:2407\.10671\.Cited by:[§F\.1](https://arxiv.org/html/2607.28638#A6.SS1.p1.1)\.
- C\. Yao, Y\. Chen, Y\. Sun, Y\. Chen, W\. Zhang, X\. Pan, Y\. Li, and B\. Ding \(2025\)Group\-relative reinforce is secretly an off\-policy algorithm: demystifying some myths about grpo and its friends\.arXiv preprint arXiv:2509\.24203\.Cited by:[§F\.1](https://arxiv.org/html/2607.28638#A6.SS1.p1.1)\.
- S\. Yao, H\. Chen, J\. Yang, and K\. Narasimhan \(2022a\)WebShop: towards scalable real\-world web interaction with grounded language agents\.ArXivabs/2207\.01206\.External Links:[Link](https://api.semanticscholar.org/CorpusID:250264533)Cited by:[§G\.1](https://arxiv.org/html/2607.28638#A7.SS1.p1.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. R\. Narasimhan, and Y\. Cao \(2022b\)React: synergizing reasoning and acting in language models\.InThe eleventh international conference on learning representations,Cited by:[§5](https://arxiv.org/html/2607.28638#S5.SS0.SSS0.Px1.p1.1)\.
- X\. Yu, B\. Peng, R\. Xu, M\. Galley, H\. Cheng, S\. Nath, J\. Gao, and Z\. Yu \(2025\)Dyna\-think: synergizing reasoning, acting, and world model simulation in ai agents\.arXiv preprint arXiv:2506\.00320\.Cited by:[§5](https://arxiv.org/html/2607.28638#S5.SS0.SSS0.Px2.p1.4)\.
- L\. Zagzebski \(2017\)What is knowledge?\.The Blackwell guide to epistemology,pp\. 92–116\.Cited by:[§1](https://arxiv.org/html/2607.28638#S1.p2.1)\.
- Y\. Zhai, S\. Tao, C\. Chen, A\. Zou, Z\. Chen, Q\. Fu, S\. Mai, L\. Yu, J\. Deng, Z\. Cao,et al\.\(2025\)AgentEvolver: towards efficient self\-evolving agent system\.arXiv preprint arXiv:2511\.10395\.Cited by:[§5](https://arxiv.org/html/2607.28638#S5.SS0.SSS0.Px2.p1.4)\.
- D\. Zhang, L\. Chen, S\. Zhang, H\. Xu, Z\. Zhao, and K\. Yu \(2023\)Large language models are semi\-parametric reinforcement learning agents\.Advances in Neural Information Processing Systems36,pp\. 78227–78239\.Cited by:[§5](https://arxiv.org/html/2607.28638#S5.SS0.SSS0.Px2.p1.4)\.
- K\. Zhang, X\. Chen, B\. Liu, T\. Xue, Z\. Liao, Z\. Liu, X\. Wang, Y\. Ning, Z\. Chen, X\. Fu,et al\.\(2025a\)Agent learning via early experience\.arXiv preprint arXiv:2510\.08558\.Cited by:[§5](https://arxiv.org/html/2607.28638#S5.SS0.SSS0.Px2.p1.4)\.
- S\. Zhang, J\. Wang, R\. Zhou, J\. Liao, Y\. Feng, W\. Zhang, Y\. Wen, Z\. Li, F\. Xiong, Y\. Qi,et al\.\(2026\)MemRL: self\-evolving agents via runtime reinforcement learning on episodic memory\.arXiv preprint arXiv:2601\.03192\.Cited by:[§5](https://arxiv.org/html/2607.28638#S5.SS0.SSS0.Px2.p1.4)\.
- X\. Zhang, H\. Sun, Y\. Zhang, K\. Feng, C\. Lu, C\. Yang, and H\. Meng \(2025b\)Critique\-grpo: advancing llm reasoning with natural language and numerical feedback\.arXiv preprint arXiv:2506\.03106\.Cited by:[Appendix C](https://arxiv.org/html/2607.28638#A3.p2.8),[§F\.1](https://arxiv.org/html/2607.28638#A6.SS1.p1.1),[§1](https://arxiv.org/html/2607.28638#S1.p1.1),[§3\.1](https://arxiv.org/html/2607.28638#S3.SS1.p1.1),[§4\.2\.2](https://arxiv.org/html/2607.28638#S4.SS2.SSS2.p2.2),[§5](https://arxiv.org/html/2607.28638#S5.SS0.SSS0.Px1.p1.1)\.
- S\. Zhao, Z\. Xie, M\. Liu, J\. Huang, G\. Pang, F\. Chen, and A\. Grover \(2026\)Self\-distilled reasoner: on\-policy self\-distillation for large language models\.arXiv preprint arXiv:2601\.18734\.Cited by:[Appendix E](https://arxiv.org/html/2607.28638#A5.p3.1),[§4\.2\.3](https://arxiv.org/html/2607.28638#S4.SS2.SSS3.p2.2)\.
- C\. Zheng, S\. Liu, M\. Li, X\. Chen, B\. Yu, C\. Gao, K\. Dang, Y\. Liu, R\. Men, A\. Yang,et al\.\(2025\)Group sequence policy optimization\.arXiv preprint arXiv:2507\.18071\.Cited by:[§F\.1](https://arxiv.org/html/2607.28638#A6.SS1.p1.1)\.
- H\. Zhou, Y\. Chen, S\. Guo, X\. Yan, K\. H\. Lee, Z\. Wang, K\. Y\. Lee, G\. Zhang, K\. Shao, L\. Yang,et al\.\(2025\)Memento: fine\-tuning llm agents without fine\-tuning llms\.arXiv preprint arXiv:2508\.16153\.Cited by:[§1](https://arxiv.org/html/2607.28638#S1.p1.1),[§5](https://arxiv.org/html/2607.28638#S5.SS0.SSS0.Px2.p1.4)\.
- H\. Zhou, S\. Guo, A\. Liu, Z\. Yu, Z\. Gong, B\. Zhao, Z\. Chen, M\. Zhang, Y\. Chen, J\. Li,et al\.\(2026\)Memento\-skills: let agents design agents\.arXiv preprint arXiv:2603\.18743\.Cited by:[§5](https://arxiv.org/html/2607.28638#S5.SS0.SSS0.Px2.p1.4)\.

## Appendix AAlgorithm pseudocode

0:Policy

πθ\\pi\_\{\\theta\}, Environment

EE, Outcome verifier

RR, Bootstrap budget

\(N,H\)\(N,H\)
0:

πθ\\pi\_\{\\theta\}
1:Initialize global experience buffer

ℬ←∅\\mathcal\{B\}\\leftarrow\\emptyset
2:forEpoch

e=1e=1to

KKdo

3:Initialize dataset

𝒟base,𝒟distill,𝒟reflect←∅,∅,∅\\mathcal\{D\}\_\{\\text\{base\}\},\\mathcal\{D\}\_\{\\text\{distill\}\},\\mathcal\{D\}\_\{\\text\{reflect\}\}\\leftarrow\\emptyset,\\emptyset,\\emptyset
4:\# Base Stateful Rollout

5:Sample base stateful trajectories in batches:

𝝉e=\{τ1,τ2,…,τN\}∼πθold\\bm\{\\tau\}^\{e\}=\\\{\\tau\_\{1\},\\tau\_\{2\},\.\.\.,\\tau\_\{N\}\\\}\\sim\\pi\_\{\\theta\_\{\\text\{old\}\}\}
6:Add transition to global experience buffer

ℬ←τe\\mathcal\{B\}\\leftarrow\\tau^\{e\}
7:foreach base trajectory

τi\\tau\_\{i\}in

𝝉e\\bm\{\\tau\}^\{e\}do

8:if

R​\(τi\)\>0R\(\\tau\_\{i\}\)\>0then

9:

𝒟base←τi\\mathcal\{D\}\_\{\\text\{base\}\}\\leftarrow\\tau\_\{i\}
10:else

11:\# Bootstrapping on failed trajectories

12:fortimestep

t=T−1t=T\-1to

0do

13:Sample aggregated trajectories

\{τnH​\(st\(i\)\)\}n=1N∼ℬ\\\{\{\\tau\}\_\{n\}^\{H\}\(s\_\{t\}^\{\(i\)\}\)\\\}\_\{n=1\}^\{N\}\\sim\\mathcal\{B\}originated at state

st\(i\)∈τis\_\{t\}^\{\(i\)\}\\in\\tau\_\{i\}
14:Generate updated state knowledge

z^st\(i\)e∼πθold\(⋅\|st\(i\),\{τnH\(st\(i\)\)\}n=1N,zst\(i\)e−1,zst\+H\(i\)∗\)\\hat\{z\}^\{e\}\_\{s\_\{t\}^\{\(i\)\}\}\\sim\\pi\_\{\\theta\_\{\\text\{old\}\}\}\\big\(\\cdot\\big\|s\_\{t\}^\{\(i\)\},\\\{\{\\tau\}\_\{n\}^\{H\}\(s\_\{t\}^\{\(i\)\}\)\\\}\_\{n=1\}^\{N\},z\_\{s\_\{t\}^\{\(i\)\}\}^\{e\-1\},z^\{\*\}\_\{s\_\{t\+H\}^\{\(i\)\}\}\\big\)
15:endfor

16:\# Verify updated state knowledge via Rejection Sampling

17:Sampling retried stateful trajectory

τiretry\\tau^\{\\text\{retry\}\}\_\{i\}with pre\-defined state knowledge

\{z^st\(i\)e\}t=0T−1\\\{\\hat\{z\}^\{e\}\_\{s\_\{t\}^\{\(i\)\}\}\\\}\_\{t=0\}^\{T\-1\}
18:if

R​\(τiretry\)\>0R\(\\tau^\{\\text\{retry\}\}\_\{i\}\)\>0then

19:

𝒟distill←τiretry\\mathcal\{D\}\_\{\\text\{distill\}\}\\leftarrow\\tau^\{\\text\{retry\}\}\_\{i\},

𝒟reflect←\(\[st\(i\),\{𝝉nH​\(st\(i\)\)\}n=1N,zst\(i\)e−1,zst\+H\(i\)∗\],z^st\(i\)e\)\\mathcal\{D\}\_\{\\text\{reflect\}\}\\leftarrow\\big\(\\big\[s\_\{t\}^\{\(i\)\},\\\{\\bm\{\\tau\}\_\{n\}^\{H\}\(s\_\{t\}^\{\(i\)\}\)\\\}\_\{n=1\}^\{N\},z\_\{s\_\{t\}^\{\(i\)\}\}^\{e\-1\},z^\{\*\}\_\{s\_\{t\+H\}^\{\(i\)\}\}\\big\],\\hat\{z\}^\{e\}\_\{s\_\{t\}^\{\(i\)\}\}\\big\)
20:endif

21:endif

22:\# Policy Update with SKL\-SD Objective Function

23:

ℒSKL\-SD​\(θ\)=ℒPG​\(𝒟base\)\+λ1​ℒPG​\(𝒟distill\)\+λ2​ℒSFT​\(𝒟reflect\)\\mathcal\{L\}\_\{\\text\{SKL\-SD\}\}\(\\theta\)=\\mathcal\{L\}\_\{\\text\{PG\}\}\(\\mathcal\{D\}\_\{\\text\{base\}\}\)\+\\lambda\_\{1\}\\mathcal\{L\}\_\{\\text\{PG\}\}\(\\mathcal\{D\}\_\{\\text\{distill\}\}\)\+\\lambda\_\{2\}\\mathcal\{L\}\_\{\\text\{SFT\}\}\(\\mathcal\{D\}\_\{\\text\{reflect\}\}\)
24:

θold←arg​minθ⁡\(ℒSKL\-SD​\(θ\)\)\\theta\_\{\\text\{old\}\}\\leftarrow\\operatorname\*\{arg\\,min\}\_\{\\theta\}\(\\mathcal\{L\}\_\{\\text\{SKL\-SD\}\}\(\\theta\)\)
25:endfor

26:endfor

Algorithm 1State\-Knowledge\-Learning Self\-Distillation\(SKL\-SD\)11footnotetext:zst\+H\(i\)z\_\{s\_\{t\+H\}^\{\(i\)\}\}can be retrieved from the buffer or the previous time\-step in the same bootstrapping loop\.0:Policy

πθ\\pi\_\{\\theta\}, Environment

EE, Simulator

EsimE^\{\\text\{sim\}\}, Outcome verifier

RR, Bootstrap budget

\(N,H\)\(N,H\)
0:

πθ\\pi\_\{\\theta\}
1:forEpoch

e=1e=1to

KKdo

2:Initialize dataset

𝒟SKL\-RL,𝒟distill←∅,∅\\mathcal\{D\}\_\{\\text\{SKL\-RL\}\},\\mathcal\{D\}\_\{\\text\{distill\}\}\\leftarrow\\emptyset,\\emptyset, environmental state

sts\_\{t\}
3:\# Aggregated Stateful Rollout via Simulation

4:for

t=0t=0to

TTdo

5:Simulate

NNparallel stateful trajectories

𝝉NH​\(st\)=\{τ1H​\(st\),…,τNH​\(st\)\}\\bm\{\\tau\}\_\{N\}^\{H\}\(s\_\{t\}\)=\\\{\\tau\_\{1\}^\{H\}\(s\_\{t\}\),\.\.\.,\\tau\_\{N\}^\{H\}\(s\_\{t\}\)\\\}with horizon

HH
6:Generate updated state knowledge

z^st∼πθold\(⋅\|st,zst,𝝉NH\(st\)\)\\hat\{z\}\_\{s\_\{t\}\}\\sim\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(\\cdot\|s\_\{t\},z\_\{s\_\{t\}\},\\bm\{\\tau\}\_\{N\}^\{H\}\(s\_\{t\}\)\)
7:Generate exploiting action

at∼πθold\(⋅\|st,z^st\)a\_\{t\}\\sim\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(\\cdot\|s\_\{t\},\\hat\{z\}\_\{s\_\{t\}\}\)
8:Transition

st\+1∼E​\(st,at\)s\_\{t\+1\}\\sim E\(s\_\{t\},a\_\{t\}\),

st←st\+1s\_\{t\}\\leftarrow s\_\{t\+1\}
9:endfor

10:Collect trajectories

𝝉e=\[s0,\(zs0,𝝉NH​\(s0\),z^s0,a0\),s1,\(zs1,𝝉NH​\(s1\),z^s1,a1\),…,sT\]\\bm\{\\tau\}^\{e\}=\[s\_\{0\},\(z\_\{s\_\{0\}\},\\bm\{\\tau\}\_\{N\}^\{H\}\(s\_\{0\}\),\\hat\{z\}\_\{s\_\{0\}\},a\_\{0\}\),s\_\{1\},\(z\_\{s\_\{1\}\},\\bm\{\\tau\}\_\{N\}^\{H\}\(s\_\{1\}\),\\hat\{z\}\_\{s\_\{1\}\},a\_\{1\}\),\.\.\.,s\_\{T\}\]
11:

𝒟SKL\-RL←𝝉e\\mathcal\{D\}\_\{\\text\{SKL\-RL\}\}\\leftarrow\\bm\{\\tau\}^\{e\},

𝒟distill←\{\(zst,z^st\)\}t=0T−1\\mathcal\{D\}\_\{\\text\{distill\}\}\\leftarrow\\\{\(z\_\{s\_\{t\}\},\\hat\{z\}\_\{s\_\{t\}\}\)\\\}\_\{t=0\}^\{T\-1\}
12:\# Policy Update with SKL\-RL Objective Function

13:

ℒSKL\-RL​\(θ\)=ℒGRPO​\(𝒟SKL\-RL\)\+λ​ℒDistill​\(𝒟distill\)\\mathcal\{L\}\_\{\\text\{SKL\-RL\}\}\(\\theta\)=\\mathcal\{L\}\_\{\\text\{GRPO\}\}\(\\mathcal\{D\}\_\{\\text\{SKL\-RL\}\}\)\+\\lambda\\mathcal\{L\}\_\{\\text\{Distill\}\}\(\\mathcal\{D\}\_\{\\text\{distill\}\}\),

θold←arg​minθ⁡\(ℒSKL\-RL​\(θ\)\)\\theta\_\{\\text\{old\}\}\\leftarrow\\operatorname\*\{arg\\,min\}\_\{\\theta\}\(\\mathcal\{L\}\_\{\\text\{SKL\-RL\}\}\(\\theta\)\)
14:endfor

Algorithm 2State\-Knowledge\-Learning Reinforcement Learning\(SKL\-RL\)
## Appendix BNotation

Table 2:Notation Description
## Appendix CCompare SKL\-SD and SKL\-RL to other training methods

TheStateful Knowledge Learningframework aligns with theLearning from Language Feedback\[Chenget al\.,[2023](https://arxiv.org/html/2607.28638#bib.bib106)\]paradigm, as our bootstrapping loop is essentially updating state knowledge using evaluative language generated at subsequent states\. The central insight is thatlanguage feedback conveys richer semantic information than scalar rewards\[Fenget al\.,[2024](https://arxiv.org/html/2607.28638#bib.bib10)\], regardless of whether it comes from external environments, self\-reflection, other models, or human supervision\. Within this paradigm, we here compare our methods to recent training\-based methods designed to improve self\-reflection capabilities\.

Critic\-GRPO\[Zhanget al\.,[2025b](https://arxiv.org/html/2607.28638#bib.bib97)\], Reflect\-Retry\-Reward\[Bensalet al\.,[2025](https://arxiv.org/html/2607.28638#bib.bib98)\], andR3\\text\{R\}^\{3\}L\[Shiet al\.,[2026b](https://arxiv.org/html/2607.28638#bib.bib91)\]explore how evaluative behaviour evolves under reinforcement learning\. These approaches construct datasets consisting of initial experiences𝒟base\\mathcal\{D\}\_\{\\text\{base\}\}, self\-generated reflections\{dreflection\}\\\{d\_\{\\text\{reflection\}\}\\\}, and refined experiences𝒟new\\mathcal\{D\}\_\{\\text\{new\}\}\(trajectory\-level\) or retried experiences𝒟retry\\mathcal\{D\}\_\{\\text\{retry\}\}\(step\-level\), followed by RL training on the combined dataset𝒟=\{𝒟base,dreflect,𝒟new/retry\}\\mathcal\{D\}=\\\{\\mathcal\{D\}\_\{\\text\{base\}\},d\_\{\\text\{reflect\}\},\\mathcal\{D\}\_\{\\text\{new/retry\}\}\\\}\. They also synthesize reflective reasoning paths by concatenating base\-level exploration traces with filtered refinement, denoted as\{𝒟reflectdistill,𝒟retrydistill\}\\\{\\mathcal\{D\}\_\{\\text\{reflect\}\}^\{\\text\{distill\}\},\\mathcal\{D\}\_\{\\text\{retry\}\}^\{\\text\{distill\}\}\\\}\. Table[3](https://arxiv.org/html/2607.28638#A3.T3)lists the typical training objectives \(policy gradient losses\) used for different components\. Critic\-GRPO\[Zhanget al\.,[2025b](https://arxiv.org/html/2607.28638#bib.bib97)\]can be seen as reinforcing with objectives 1&2\. Reflect\-Retry\-Reward\[Bensalet al\.,[2025](https://arxiv.org/html/2607.28638#bib.bib98)\]reinforces on objective 6,R3\\text\{R\}^\{3\}L\[Shiet al\.,[2026b](https://arxiv.org/html/2607.28638#bib.bib91)\]is trained with RL on objective 1&3, and perform SFT on objective 4&5\. Experiential RL\[Shiet al\.,[2026a](https://arxiv.org/html/2607.28638#bib.bib120)\]optimize with respect to objective 2&4\.

In our work,State\-Knowledge\-Learning Self\-Distillation\(see Equation[1](https://arxiv.org/html/2607.28638#S3.E1)\) adopts a similar loss design, but all components follow astatefulformulation where state knowledgezzis generated before the actionaa\. Specifically, we maintain a policy gradient loss over base rolloutsℒPG​\(𝒟base\)\\mathcal\{L\}\_\{\\text\{PG\}\}\(\\mathcal\{D\}\_\{\\text\{base\}\}\)\(analogous to objective 1, but with stateful trajectories and sample filtering\), a policy gradient loss over distilled refined responsesℒPG​\(𝒟distill\)\\mathcal\{L\}\_\{\\text\{PG\}\}\(\\mathcal\{D\}\_\{\\text\{distill\}\}\)\(analogous to objective 2, again in a stateful form and filtered\), and an auxiliary SFT lossℒSFT​\(𝒟reflect\)\\mathcal\{L\}\_\{\\text\{SFT\}\}\(\\mathcal\{D\}\_\{\\text\{reflect\}\}\)that guides the bootstrapping behaviour\. Thus,SKL\-SD can be viewed as a “stateful” version of recent self\-reflection RL training methods; we call it “self\-distillation” or “self\-imitation” because we reinforce only filtered \(positive\) trajectories and discard negative samples\.

In contrast,State\-Knowledge\-Learning Reinforcement Learning\(see Equation[4](https://arxiv.org/html/2607.28638#S3.E4)\) isfundamentally different: it integrates state aggregation and the bootstrapping loop into a single online rollout episode and relies solely on the final outcome reward to jointly shape both generation and evaluation behavior\. This makes it a more unified and autonomous agent training framework\.

Table 3:Training objective for different purposes in works related to other self\-reflection training methods\.
## Appendix DLocalized parallel simulation in SKL\-RL

In the context of SKL\-RL, the localized parallel simulation mechanism acts as an online, text\-based look\-ahead tree exploration\. Because the agent has access to a resettable simulator, it doesn’t just pick an action and move forward blindly; instead, it pauses at the current statests\_\{t\}, freezes the main trajectory, and "probes" the future\. Here is a detailed breakdown of how this parallel simulation operates at each timestep:

1. 1\.State Anchoring\(The Root\): At any given online step, the agent encounters statests\_\{t\}\. The simulator’s state is checkpointed or saved at this exact position, acting as the root node for the simulations\.
2. 2\.Multi\-Branch Spawning\(WidthNN\): From this single root statests\_\{t\}, the agent generates knowledgezstz\_\{s\_\{t\}\}, and instantiatesNNactions for independent, parallel branches \(or simulation workers\)\.
3. 3\.Independent Temporal Expansion\(HorizonHH\): At each subsequent simulation timestep1,…,H1,\\dots,H, every branch rolls out independently\. Each branch samples knowledge and actions according to the agent’s current policy \(or an exploration policy\) and receives independent observations and rewards from its own isolated instance of the environment\.
4. 4\.Experience Aggregation: Once allNNbranches reach the maximum simulation horizonHH\(or hit a terminal state\), their complete rollout histories are gathered into an aggregated experience block𝝉NH\\bm\{\\tau\}\_\{N\}^\{H\}\.

A concrete example is also provided in Figure[8](https://arxiv.org/html/2607.28638#A7.F8)\.

## Appendix EMulti\-turn self\-distillation in SKL\-RL

The primary objective of the distillation lossℒDistill​\(\{zs,z^s\}s∈𝒮\)\\mathcal\{L\}\_\{\\text\{Distill\}\}\(\\\{z\_\{s\},\\hat\{z\}\_\{s\}\\\}\_\{s\\in\\mathcal\{S\}\}\)is to internalize the insights gained from look\-ahead simulations into the agent’s prior stateful knowledge\. By grounding its reasoning in previously distilled knowledge, the agent can more effectively explore alternative trajectories in subsequent iterations and incrementally compound its predictive understanding\. Formally, this iterative refinement process across consecutive iterations can be expressed as:

Iteration​i:\\displaystyle\\text\{Iteration \}i:…→st→\{zst,at,st\+1,zst\+1,…,st\+H\}→z^st¯→at→st\+1→…\\displaystyle\.\.\.\\rightarrow s\_\{t\}\\rightarrow\\\{z\_\{s\_\{t\}\},a\_\{t\},s\_\{t\+1\},z\_\{s\_\{t\+1\}\},\.\.\.,s\_\{t\+H\}\\\}\\rightarrow\\underline\{\\hat\{z\}\_\{s\_\{t\}\}\}\\rightarrow a\_\{t\}\\rightarrow s\_\{t\+1\}\\rightarrow\.\.\.Iteration​i\+1:\\displaystyle\\text\{Iteration \}i\+1:…→st→\{z^st¯,at′,st\+1′,zst\+1′,…,st\+H′\}→z^st′→at′→st\+1′→…\\displaystyle\.\.\.\\rightarrow s\_\{t\}\\rightarrow\\\{\\underline\{\\hat\{z\}\_\{s\_\{t\}\}\},a\_\{t\}^\{\{\}^\{\\prime\}\},s\_\{t\+1\}^\{\{\}^\{\\prime\}\},z\_\{s\_\{t\+1\}^\{\{\}^\{\\prime\}\}\},\.\.\.,s^\{\{\}^\{\\prime\}\}\_\{t\+H\}\\\}\\rightarrow\\hat\{z\}\_\{s\_\{t\}\}^\{\{\}^\{\\prime\}\}\\rightarrow a^\{\{\}^\{\\prime\}\}\_\{t\}\\rightarrow s\_\{t\+1\}^\{\{\}^\{\\prime\}\}\\rightarrow\.\.\.
This process constitutes a teacher\-student framework\. The "student" model represents the agent’s zero\-shot capability to generate stateful knowledge upon first encountering a state, governed by the distributionπ​\(zst\|st,ℋ\)\\pi\(z\_\{s\_\{t\}\}\|s\_\{t\},\\mathcal\{H\}\), whereℋ\\mathcal\{H\}denotes the interaction history context\. The "teacher" model represents the target distributionπ​\(z^st\|𝝉NH,st,ℋ\)\\pi\(\\hat\{z\}\_\{s\_\{t\}\}\|\\bm\{\\tau\}\_\{N\}^\{H\},s\_\{t\},\\mathcal\{H\}\), which produces the refined stateful knowledge conditioned on the outcomes of the look\-ahead simulations𝝉NH\\bm\{\\tau\}\_\{N\}^\{H\}\.The goal of this self\-distillation is to align the student model with the teacher, compelling the agent to consolidate simulation\-derived experience and autonomously build new knowledge\. This alignment can be optimized using either Forward or Reverse Kullback\-Leibler \(KL\) divergence in a multi\-turn setting:

Forward KL:ℒDistillFoward\(θ\)=∑stKL\[πθ\(z\|𝝉NH,st,ℋ\)∥πθ\(z\|st,ℋ\)\]\\textbf\{Forward KL\}:\\mathcal\{L\}\_\{\\text\{Distill\}\}^\{\\text\{Foward\}\}\(\\theta\)=\\sum\_\{s\_\{t\}\}\\text\{KL\}\\bigg\[\\pi\_\{\\theta\}\(z\|\\bm\{\\tau\}\_\{N\}^\{H\},s\_\{t\},\\mathcal\{H\}\)\\bigg\\\|\\pi\_\{\\theta\}\(z\|s\_\{t\},\\mathcal\{H\}\)\\bigg\]\(5\)Reverse KL:ℒDistillReverse\(θ\)=∑stKL\[πθ\(z\|st,ℋ\)∥πθ\(z\|𝝉NH,st,ℋ\)\]\\textbf\{Reverse KL\}:\\mathcal\{L\}\_\{\\text\{Distill\}\}^\{\\text\{Reverse\}\}\(\\theta\)=\\sum\_\{s\_\{t\}\}\\text\{KL\}\\bigg\[\\pi\_\{\\theta\}\(z\|s\_\{t\},\\mathcal\{H\}\)\\bigg\\\|\\pi\_\{\\theta\}\(z\|\\bm\{\\tau\}\_\{N\}^\{H\},s\_\{t\},\\mathcal\{H\}\)\\bigg\]\(6\)
Here, the Forward KL objective serves as a multi\-turn, self\-distillation extension of Supervised\-KD\[Agarwalet al\.,[2024](https://arxiv.org/html/2607.28638#bib.bib117)\], while the Reverse KL acts as a multi\-turn extension of OPSD\[Zhaoet al\.,[2026](https://arxiv.org/html/2607.28638#bib.bib113)\]\.

## Appendix FExperiments on agentic environments

### F\.1Experiment setup

The experimental setup of SKL\-SD on agentic tasks is highly aligned with the setup used by\[Shiet al\.,[2026b](https://arxiv.org/html/2607.28638#bib.bib91)\]as we re\-implement SKL\-SD in their public codebase, and use the exact hyperparameter setting reported in their paper\. The baseline includes rejection sampling methods like RAFT\[Donget al\.,[2023](https://arxiv.org/html/2607.28638#bib.bib127)\], GRPO variants including the default GRPO\[Guoet al\.,[2025a](https://arxiv.org/html/2607.28638#bib.bib11)\], OPMD\[Yaoet al\.,[2025](https://arxiv.org/html/2607.28638#bib.bib128)\], and GSPO\[Zhenget al\.,[2025](https://arxiv.org/html/2607.28638#bib.bib129)\]\. Crucially, we compare against language\-feedback frameworks that utilize self\-reflection or critiques, including Critique\-GRPO\[Zhanget al\.,[2025b](https://arxiv.org/html/2607.28638#bib.bib97)\], Reflect\-GRPO\[Bensalet al\.,[2025](https://arxiv.org/html/2607.28638#bib.bib98)\], and R3L\[Shiet al\.,[2026b](https://arxiv.org/html/2607.28638#bib.bib91)\]\. All experiments are implemented using the Trinity\-RFT framework\[Panet al\.,[2025](https://arxiv.org/html/2607.28638#bib.bib130)\]to ensure parity in infrastructure\. We useQwen2\.5\-7B\-Instruct\[Yanget al\.,[2024](https://arxiv.org/html/2607.28638#bib.bib103)\]as the backbone model across all methods\. The training is operated on a NVIDIA H100 single\-node cluster\.

### F\.2Detailed reasoning trace

Table 4: A reasoning example on WebShop\. Multiple state knowledge revealed at different statesQuery:Find me non\-slip, easy clean computer armoires with PU leather with color: white, and size: 120x60cm, and price lower than 70\.00 dollars\[State Knowledge for Initial Search\]State Understanding:1\. Search Queries:•Ensure the search terms include specific attributes and measurements to match the exact product requirements: \- Color: … \- Size: … \- Additional Features:… \- Price: …2\. Product Verification and Filtering:•After locating a potential product, verify all criteria \(color, size, features, price\) are correctly specified\.•Directly click "Buy Now" if the product matches all specified criteria, without additional checks for availability\.3\. Buy Now on Detailed Product Page:•Ensure that the click on the "Buy Now" button is performed directly on the detailed product page with all filters correctly applied\.•Click the "Buy Now" button if all conditions are verified\.4\. Detailed Review and Double\-Check:•Verify that the product’s page correctly displays all the required attributes and features, using inspection tools if necessary\.•Double\-check that the product’s details, especially dimensions \(size: 120x60cm\) and color \(white\), and price \(≤\\leq$70\.00\) match the criteria\.5\. Error Logging:•If a product does not match all criteria, log the relevant details of the product and try again, ensuring a revisit to the search page with corrected parameters if necessary\.Steps for Implementation:1\. Refine Search Query with Specific Attributes and Dimensions: ……\[State Knowledge for Browsing\]State Understanding:1\. Correct Product Matching: Ensure that the selected product details \(color, size, price\)—all attributes must correctly match the required criteria: 1\. Color: …, 2\. Size: …, 3\. Price: …2\. Product Detail Verification:•Double\-check each product detail on the result page to ensure accuracy\.•Verify that images and descriptions explicitly confirm the exact details being used \(white color, 120x60cm size, etc\)\.……N\. Notes for Future Runs:•Cross\-Verify Attributes: Validate the exact color, size, and price on each detailed product page\.•Double\-check Final Product: If unsure, re\-verify the product details and attributes before clicking Buy Now\.•Snapshots as Evidence: Take screenshots or notes of the product description and attributes to confirm they meet the criteria\.\[State Knowledge for Specific Product Page\]State Understanding:1\. Navigating the URL: Start by ensuring that the correct URL is accessed, specifically B095KCTB3Y, to display the desired product\.2\. Selecting Size and Color Details:•Ensure that the size 120x60cm and color white are explicitly selected and chosen correctly from dropdown options \(if available\)\.•Double\-check that the product details on the page reflect the correct options and attributes \(white, 120x60cm, non\-slip\)\.3\. Direct Purchase Action: Once all conditions \(size and color\) are correctly matched and verified on the product description, click the Buy Now button\.Notes:•Ensure that the search query uses keywords such as "white", "120x60cm", and "pu leather" to precisely target the correct product\.•Directly interact with the specified URL and verify the exact attributes before finalizing the purchase\.•Use the correct Buy Now button to conclude the task\.•For future iterations, focus on refining the search criteria to eliminate confusion\.

## Appendix GExperiments on ChessPuzzles

### G\.1The uniqueness of the chess game for LLM evaluation

Recent research on LLM\-based agents often evaluates performance on relatively simple agentic environments, such as ALFWorld\[Shridharet al\.,[2020](https://arxiv.org/html/2607.28638#bib.bib118)\], WebShop\[Yaoet al\.,[2022a](https://arxiv.org/html/2607.28638#bib.bib119)\], or grid\-based games like FrozenLake and Sokoban\[Fenget al\.,[2024](https://arxiv.org/html/2607.28638#bib.bib10)\]\. While these benchmarks are useful for studying sequential decision\-making and tool\-use behaviors, they typically involve limited strategic depth and can often be solved through short\-horizon reasoning or heuristic exploration\. Consequently, many agent studies primarily report final task completion rates, without closely examining the soundness or internal consistency of the reasoning traces that lead to those outcomes\. In contrast, chess games require substantially richer strategic reasoning\[Liuet al\.,[2025](https://arxiv.org/html/2607.28638#bib.bib105)\]\. Solving a chess puzzle generally involves multi\-step planning, evaluation of alternative move sequences, and understanding tactical motifs such as checks, pins, or forced mates\. These characteristics make chess a more demanding testbed for assessing whether an agent can produce coherent reasoning processes rather than merely reaching the correct final answer\.

Another advantage of chess puzzles is that they are less susceptible to data contamination compared to many popular agent benchmarks\. Many environments have well\-documented task templates that may appear in pretraining corpora, making it difficult to disentangle memorization from genuine reasoning\. In contrast, our experiments indicate that even strong closed\-source models \(e\.g\., GPT\-5\) struggle to solve the ChessPuzzles benchmark in a zero\-shot setting, as illustrated in Table[G\.1](https://arxiv.org/html/2607.28638#A7.SS1)\. This suggests that the model does not simply recall solutions from training data but must instead need to reason about the game dynamics\.

Taken together, these properties make chess puzzles an ideal evaluation setting for studying an agent’s ability to adapt to an environment with minimal prior knowledge of the underlying rules\. This motivates our use of chess as a challenging benchmark to evaluate the effectiveness of our algorithm\.

Table 5: Reasoning of based model in ChessPuzzles tasks\. Both Qwen2\.5\-7B\-Instruct and GPT\-5 fail to provide good reasoning\.Game State: 1Qr5/p4r1k/8/3p2p1/P2P2P1/2P2pKP/8/8 w \- \- 0 49,StockFish’s Move: Qb1Qwen2\.5\-7B\-Instruct\(Nonsensical reasoning\) Let’s analyze the position strategically\. White has a queen and a king, while Black has a rook and a king\. The position is closed, with few pieces on the board, so tactical play is important\.Key candidate moves to consider are: 1\. Qxc8: Captures the pawn and puts pressure on the black king\.\(Misidentified the piece\) 2\. Qa8: Attacks the rook on a1 and also threatens the black king\.\(Misidentified the piece\) 3\. Qc7: Targets the black king directly\.\(Wrong justification\)Now let’sevaluate each move:\(Completely wrong\) 1\.Qxc8: If this move is played, Black will respond with Rxa8\. The resulting position will leave White with a potential fork with the queen, targeting both the black king and the rook\. However, this might be risky as Black could still create threats with the rook on a8\. 2\.Qa8: This move attacks the rook on a1 and threatens to put pressure on the black king\. After Qa8, Black can respond with Rxa8, but this would result in a simplified position where White can focus on controlling the center and potentially queening the pawn on h4 or a5\. 3\.Qc7: This move directly targets the black king, putting it under pressure\. If Qc7 is played, Black can respond with Kf8, which limits the immediate threat\. However, it leaves the black king more exposed\.Considering the strategic implications, Qa8 seems to be the most promising move\. It not only attacks a key piece but also creates a significant threat to the black king\. The resulting position is simpler and allows White to control the center and potentially create winning chances\.<move\>Qa8</move\> GPT\-5\(Better at general reasoning,but fail at detailed analysis\) Strategic evaluation\(Making Sense\):Material is heavily in White’s favor because White has a queen \(on b8\) against Black’s rook \(on f7\) plus some pawns\. The immediate priority is to simplify into a trivially winning endgame by trading queens/major pieces or winning the opposing rook, while also keeping White’s king safe\. Black’s king is on h7 and is relatively exposed; Black’s main counterplay is the rook on f7 plus the advanced f3\-pawn\. Three key candidate moves:1\) Qxc8 Idea: capture the black queen on c8 and trade queens immediately\. Likely response: Black must recapture with Rxc8 \(the rook from f8 goes to c8\)\. Outcome: White has traded queens, but then White is only up material modestly and still has to deal with Black’s active rook and passed f\-pawn\. This is good and likely winning, but it gives Black some practical chances compared to winning the rook outright\.\(Wrong detailed analysis\)2\) Qxa7 Idea: win a pawn with check possibilities later \(queen near the king\)\. Likely response: Black can continue with f2 \(or Rxa7 isn’t possible\), pushing the passed pawn and creating real promotion threats, while White hasn’t reduced Black’s major\-piece activity\. Outcome: This is slower and allows Black counterplay with the f\-pawn and rook coordination\. Not the cleanest conversion\. 3\) Qb7 Idea: directly attack Black’s rook on f7 along the 7th rank and win it\. Likely response: Black’s rook has no squares available on the 7th rank because b7 controls the entire rank; it also cannot capture Qb7 \(rook on f7 can’t move like that\)\(Wrong prediction\), and moving the king doesn’t solve the attack\. The only “defense” would be moving the rook away—but from f7, every legal rook move is along the f\-file or 7th rank, and Qb7 covers f7 and the whole 7th, so the rook is effectively lost\. Outcome: White wins the rook cleanly, after which the position is completely winning with queen versus only pawns\. Conclusion: Qb7 is the most forcing and highest\-value move: it wins Black’s rook immediately with minimal counterplay, converting the advantage far more cleanly than trading queens with Qxc8 or grabbing a pawn with Qxa7\.<move\>Qb7</move\>

### G\.2Experiment Setup

##### Experiment Setup

We also compare SKL\-RL with other training baselines introduced in Section[4\.1](https://arxiv.org/html/2607.28638#S4.SS1), all of which operate via trajectory\-level self\-reflection\. All methods are trained with matched computation budgets, measured by the number of LLM calls during training\. We use Qwen2\.5\-7B\-Instruct as the base model, and apply GRPO for all RL implementations\. We select a subset of the original ChessPuzzles \(rating<1000<1000\) and split it into a train and a test set\.

On ChessPuzzles, agents are prompted to generate state knowledgez​\(st\)z\(s\_\{t\}\)for the current game statests\_\{t\}and decide on the movementata\_\{t\}to make in two consecutive reasoning steps:

1. 1\.State Knowledgeztz\_\{t\}:\[User\]: What is your understanding of the game state: \{state\}\. \[Assistant\]: \.\.\.
2. 2\.Movementata\_\{t\}:\[User\]: You are in stage \{stage\}, based on the state understanding, what is the decided move? \[Assistant\]: \.\.\.

where the state knowledge explicitly queries the model for its internal predictive knowledge of the game state, and the movement query will use the generated understanding as the context, asking the model to directly generate action at each stage \(i\.e\., both during simulation or bootstrapping\)\. This way of reasoning can also avoid the issue of overthinking that causes missing action tokens and make knowledge distillation easier to implement later\. Figure[8](https://arxiv.org/html/2607.28638#A7.F8)illustrates this\. Self\-generated tree search simulation is carried out in a similar way as parallel decoding, where the agent is prompted to generate multiple \(HH\) moves at the same time, and for each rollout branch, the agent independently performs a single\-trajectory rollout until running out of horizon budget\. Table[G\.2](https://arxiv.org/html/2607.28638#A7.SS2.SSS0.Px1)shows the system prompt for ChessPuzzles\.

![Refer to caption](https://arxiv.org/html/2607.28638v1/x9.png)Figure 8:At each statests\_\{t\}, the agent first performs a tree search where each node expansion consists of state knowledge and a simulation move\. The generated simulation experience𝝉NH​\(st\)\\bm\{\\tau\}\_\{N\}^\{H\}\(s\_\{t\}\)is aggregated as the context, and the agent summarizes an updated state understandingz^st∼πθ\(⋅\|𝝉NH\(st\),st,zst\)\\hat\{z\}\_\{s\_\{t\}\}\\sim\\pi\_\{\\theta\}\(\\cdot\|\\bm\{\\tau\}\_\{N\}^\{H\}\(s\_\{t\}\),s\_\{t\},z\_\{s\_\{t\}\}\)as well as a finalized moveata\_\{t\}to transition to the next statest\+1s\_\{t\+1\}\.Table 6: System Prompt used in ChessPuzzles tasks\.A conversation between User and Assistant\. The User asks the best move to make for a given chess board state, and the Assistant solves it\. The Assistant is a professional chess player who can plan a few steps ahead by interacting with a simulator, and then think about the reasoning process and at last provides the User with the answer\.The Assistant’s thinking process must include an understanding of the game state, based on which the Assistant can later decide on the next move\. The understanding should describe how the Assistant analyzes the current position, including: \- A strategic evaluation of the position\. \- A comparison of key candidate moves \(not all of them\) in terms of their possible subsequent outcomes\. \- Do not try to enumerate all possible moves and outcomes, only consider the most likely ones\. The decision on next move must be in SAN notation, strictly using the moving piece and the destination square \(e\.g\., Nf3, Rxf2, c5\)\. Reminder of chess rules: \- Bishops move diagonally\. \- Rooks move horizontally or vertically\. \- Knights jump in an L\-shape\. \- Queens combine rook and bishop movement\. \- Kings move one square in any direction\. \- Pawns move forward, capture diagonally, and can promote\.

### G\.3Ablation on bootstrapping behaviour at different heuristic simulation strategies

Table 7: Reasoning trace example of ablation study for SKL\-RL in Section[4\.2\.1](https://arxiv.org/html/2607.28638#S4.SS2.SSS1)\.\(Test set\) Game State: 2r3k1/p1r2ppp/2N1p3/3pPP2/4b1P1/1P6/P6P/2R2RK1 w \- \- 1 28Simulation Strategy: Expert Move Rate12\\frac\{1\}\{2\}in simulation\(short, frequency\-based\)The given game state is: 2r3k1/p1r2ppp/2N1p3/3pPP2/4b1P1/1P6/P6P/2R2RK1 w \- \- 1 28\. From the simulation trajectories, we can see that the move ‘Nd8‘ is the most frequently chosen move by the player, and it leads to a few different successor states\. Let’s analyze the trajectories:\[Simulation Trajectory\]From the trajectories, we can see that the move ‘Nd8‘ leads to the same or similar states in all cases where it is played\. On the other hand, the move ‘Ne7‘ leads to a different state\. The state after ‘Nd8‘ is: 2rN3k/p1r2ppp/4p3/3pPP2/4b1P1/1P6/P6P/2R2RK1This state is very similar to the original state, suggesting that ‘Nd8‘ is a good move as it maintains the position quality\. The state after ‘Ne7‘ is: 2r4k/p1r1Nppp/4p3/3pPP2/4b1P1/1P6/P6P/2R2RK1This state is different and potentially less favorable, as it involves a knight move that could be less effective in maintaining the position\.Strategic evaluation:•The position is balanced, with both sides having similar material and pawn structures\.•The move ‘Nd8‘ supports the king and maintains the structure, while ‘Ne7‘ leads to a different and potentially less favorable position\.Based on the simulation rollouts and the strategic evaluation, the best move is:Nd8Simulation Strategy: Expert Move Rate16\\frac\{1\}\{6\}in simulation\(more structured, but hallucinate\)The given state is: 2r3k1/p1r2ppp2N1p3/3pPP24b1P1/1P6P6P/2R2RK1w \- \-1 28\. This is a middlegame position with White to move\. Let’s analyze the key features of this position:1\.Material Balance: The material is balanced with two rooks \(R\), two knights \(N\), and one bishop \(B\) for each side\. White has a pawn \(P\) on the 8th rank \(White’s king\-side\), and Black has a pawn on the 7th rank \(Black’s king\-side\)\.2\.King Safety: Both kings are castled on the kingside, and neither side has any immediate threats to their kings\.3\.Pawn Structure: White has a pawn on e2 and pawns on f2 and g2\. Black has pawns on e7 and f7\.4\.Active Pieces: White has a knight on b8 and a rook on the 1st rank\. Black has a knight on a5 and a rook on the 1st rank\.5\.Key Moves: The knight on b8 \(Nb8\) in Traj 3 suggests that it might be a strong move to support the pawn structure and potentially create threats\. The knight moves Ne7 and Nb5 in Traj 1 and 2 suggest support for the king\-side pawns and maintaining control over key squares\. The rook move Na5 in Traj 4 suggests a support move for the knight and maintaining control over key files\.Analysis of Moves•Move: Nb8 \(Traj 3\): Supports the pawn structure, maintains control over key squares, and potentially prepares for future pawn moves\.•Move: Ne7 \(Traj 1\): Supports the king\-side pawns and maintains control over key squares\.•Move: Na5 \(Traj 4\): Supports the knight and maintains control over key files\.Conclusion: The move Nb8 in Traj 3 seems to be a strong candidate as it supports the pawn structure and maintains control over key squares\. Given the similarity in the trajectories, maintaining support and control over key squares appears to be a strong strategy\.Recommendation: The best move for White in the given state is:Nb8Simulation Strategy: Random Movein simulation\(ignore simulation, more explicit reasoning but severe hallucination\)Analysis of the Game StateThe root state of interest is: 2r3k1/p1r2ppp/2N1p3/3pPP2/4b1P1/1P6/P6P/2R2RK1 w \- \- 1 28\.This state is derived from the four trajectories provided\. Let’s break down the key elements of this position:1\.White to Move \(w \- \- 1 28\): White is to move, and it is the 28th move of the game\.2\.Material Balance: White has 2 rooks, 3 knights, 1 king, and 1 pawn\. Black has 2 rooks, 1 knight, 1 king, and 1 pawn\.3\.Positional Evaluation: White has a rook and a knight, while Black has a rook and a knight\. White has a king and a rook, while Black has a king and a rook\. White has a pawn on the 6th rank, while Black has a pawn on the 6th rank\.4\.Key Pieces: White’s rook on d1 and Black’s rook on c4\. White’s knight on g5 and Black’s knight on d8\.5\.Active Play: The game is still in the middlegame, with no clear endgame or decisive advantage\. Both sides have active pieces, and the game is likely to continue with tactical exchanges and positional maneuvers\.Trajectory Analysis: Let’s analyze the trajectories to understand the potential future value of the root state:•Traj 1, 2, 3, 4: The state remains similar, with no clear changes in the position\.Strategic Evaluation•Control of the Board: Both sides have active pieces, and the board is open\.•King Safety: Both kings are relatively safe, but the rooks and knights are more active\.•Pawn Structure: The pawns on the 6th rank are key, and both sides have similar pawn structures\.Candidate Moves: Given the current state, let’s consider the most likely candidate moves:1\.g5: This move might aim to control the g5 square and potentially create a tactical opportunity\.2\.h4: This move might aim to control the h4 square and potentially create a tactical opportunity\.3\.Rfd1: This move might aim to control the d1 square and potentially create a tactical opportunity\.Conclusion: Based on the analysis of the trajectories, the root state is still in a balanced middlegame position with active pieces\. The moves g5, h4, and Rfd1 are all plausible and aim to create tactical opportunities or control key squares\.Recommended Move: Given the strategic evaluation and the potential future value from the trajectories, the moveg5seems to be a strong candidate as it controls a key square and potentially creates tactical opportunities\.Recommended Move: g5

### G\.4A complete example of SKL\-RL with self\-generated simulation

Table 8: Complete reasoning trace of SKL\-RL at statests\_\{t\}\.\[System Prompt\]: \(See Table[G\.2](https://arxiv.org/html/2607.28638#A7.SS2.SSS0.Px1)\)\[User\]\(Environment State𝐬tE\\mathbf\{s\}\_\{t\}^\{E\}\) Current Game State: k1r4r/p4pp1/2Nb2p1/3P4/Q3p3/3nPq1P/P2K1P2/R4R2 w \- \- 4 29Root State Simulationsts\_\{t\}\[User\]\(Root State𝐬t=stE\\mathbf\{s\}\_\{t\}=s\_\{t\}^\{E\}in Simulation\) Current Simulation State:k1r4r/p4pp1/2Nb2p1/3P4/Q3p3/3nPq1P/P2K1P2/R4R2 w \- \- 4 29What is your understanding of the game state: k1r4r/p4pp1/2Nb2p1/3P4/Q3p3/3nPq1P/P2K1P2/R4R2 w \- \- 4 29\[Assistant\]\(Understandingzstz\_\{s\_\{t\}\}at Root State\) \#\#\# Understanding of the Game State The current position is …\[User\] You are now in simulation, your current goal is to move in order to better understand potential future outcomes of the game, so you can either explore or exploit\. You can generate at most 3\(budgetNN\)candidate moves to be tested with the simulator\. Do not generate duplicate moves\.Choose the move from the available move list: …\[Assistant\]\(Simulation Actionat\(i\)a\_\{t\}^\{\(i\)\}at Root State\) <move\>Qxa7</move\><move\>Qa6</move\><move\>Ne7</move\>\[User\]Found 3 valid moves\.\(Next Statest\+1\(i\)s\_\{t\+1\}^\{\(i\)\}in Simulation\)•Traj 1: Subsequent result of move Qxa7: Results: You have successfully solved the game within this simulation\!•Traj 2: ……•Traj 3: ……Expanded State Simulationst\+1s\_\{t\+1\}\[User\]&\[Assistant\]\(Continue Simulation atst\+1\(i\)s\_\{t\+1\}^\{\(i\)\}\) \(Turn 2 simulation\) …Self\-Evaluation/Bootstrapping forz^​\(st\)\\hat\{z\}\(s\_\{t\}\)andata\_\{t\}\[User\]\(Simulation Ends, Start Bootstrapping and Getz^st\\hat\{z\}\_\{s\_\{t\}\}\) \[Summarizing stage\] The simulation ends\. Lets pause, check the interaction history, and rethink again both the benefits and harm of each possible moves at simulation step 1\.\#\#\# Simulation History: …\[User\] Based on the simulation history, what is your updated understanding of the game state:k1r4r/p4pp1/2Nb2p1/3P4/Q3p3/3nPq1P/P2K1P2/R4R2 w \- \- 4 29\[Assistant\]\(Finalized actionata\_\{t\}atsts\_\{t\}\) \#\#\# Updated Understanding of the Game State…\[User\] The simulation ends, your current goal is to output the optimal move for the root state\. Based on previous simulation, your understanding of the game state and your current goal, what is the best move to make?\[Assistant\] <answer\>Qxa7</answer\>

### G\.5Reasoning trace for R3L and SKL\-RL with self\-generated simulation after end\-to\-end training

Table 9: Reasoning trace example ONE of SKL\-RL after end\-to\-end RL training\. SKL\-RL helps with more concise, stateful, outcome\-aware reasoning\.Game State: r1b1k2r/5ppp/2p5/p1q1p1B1/Pp2P1n1/1P3NN1/1P3PPP/3Q1RK1 w kq \- 0 17\[State Understanding\] The current position is a complex middlegame position for White, who is in a strong position due to their queen and rook being active and threatening the black king\. White has a strong initiative and should focus on developing their queenside attack while also preparing to push their queenside pawns\. Let’s consider the most promising moves:1\.Qd8: This move is very strong because it not only attacks the black queen but also threatens to checkmate on the next move if the queen moves\. \(Correctly identify the key benefits\)2\.Kh1: This move is more positional and prepares to push the h\-pawns, which could be a long\-term strategy\.3\.Re1: This move is more defensive but also prepares to support the h\-pawns and control the e\-file\.4\.Nxe5: This move is more tactical and aims to put pressure on the black knight and potentially open up the e5 square for further attacks\. \(analysis seems reasonablebut still at a surface\-level\) Given the strong positional and tactical threats, the move Qd8 stands out as the most aggressive and decisive\. \[Experience Summary/Bootstrapping\] Based on the simulation history, the updated understanding of the game state is as follows:1\. White is in a strong position, having just won a queen and a rook, which is a significant material advantage\.\(wrong\)2\. White has a strong attack on the black king, with the queen and rook threatening to checkmate on the next move\. \(correctly sumarize the key benefits\)3\. The black king is in a vulnerable position, with the queen and rook threatening to deliver checkmate\.4\. The black queen is a potential threat, but it is not well\-placed to counter the white attack\.5\. The white king is safe and can continue to support the attack on the black king\.6\. The board is open and dynamic, with many pieces still active and ready to participate in the attack\.Given this updated understanding, the position is a strong position for White, and the moveQd8was the best move in the simulation history, leading to a win\. \(combine knowledge with simulation result\)

Table 10: Reasoning trace example of R3L\. Trajectory reflection from R3L ignores stateful information\.Game State: r1b1k2r/5ppp/2p5/p1q1p1B1/Pp2P1n1/1P3NN1/1P3PPP/3Q1RK1 w kq \- 0 17\[Trajectory Reflection\] trajectory summary: The Assistant analyzed the chess position and decided on a sequence of moves focusing on gaining positional advantage, promoting pawns, and controlling key squares\. However, the Assistant failed to complete the task within the step limit, as no pawn promotion occurred despite several opportunities\. root cause analysis: Why was the approach suboptimal? Because the Assistant did not prioritize pawn promotion in the key moments\. Why did the Assistant not promote pawns? Because the strategy was too positional and did not explicitly plan for pawn promotion\. Why this positional approach? Because the Assistant optimized for immediate positional advantages rather than focusing on the critical goal of promoting pawns\. Root cause: Lack of explicit focus on the endgame goal of promoting pawns\. The Assistant should have immediately promoted a pawn when given the opportunity, as the game was nearing its end\. trajectory outcome: failure, improvement suggestion: In endgame scenarios, especially in rook and pawn situations, the primary goal should be to promote pawns\. The Assistant must explicitly prioritize pawn promotion and take advantage of opportunities to do so, even if it means sacrificing other positional benefits\. For instance, when the King is closer to the fifth rank \(more pawn promotion squares\), the Assistant should move towards those squares and promote the pawn as soon as possible\. Example: In the current trajectory, when the white king was on g7 or h6, the priority should have been to move towards a pawn promotion square and promote the pawn immediately when legal\. retry from step: 0 \(Coarse\-grained Understanding\)

Table 11: Reasoning trace example TWO of SKL\-RL before and after end\-to\-end RL training\.Game State: 2r2k2/1K1r1p2/5Q2/pp2pN2/P3P3/2P4P/1P3q2/8 w \- \- 3 46; Optimal Move: Kxc8 Experience Summary/Bootstrapping before RLThe current game state is as follows: 2r2k21K1r1p25Q2pp2pN2P3P32P4P1P3q28Updated Understanding of the Game State: 2r2k21K1r1p25Q2pp2pN2P3P32P4P1P3q28\*\*White to move\.\*\*Analysis of Simulation Trajectories\(re\-stating the history\):1\. \*\*Trajectory 1: Kxc8\*\*\- \*\*Initial Move\*\*: Kxc8\- \*\*Opponent’s Move\*\*: f8g8\- \*\*Next State\*\*: 2K3k1/3r1p2/5Q2/pp2pN2/P3P3/2P4P/1P3q2/8 w \- \- 1 47\- \*\*Subsequent Move\*\*: Kb8\- \*\*Opponent’s Move\*\*: g8f8\- \*\*Next State\*\*: 1K3k2/3r1p2/5Q2/pp2pN2/P3P3/2P4P/1P3q2/8 w \- \- 3 482\. \*\*Trajectory 2: Ka6\*\*\- \*\*Initial Move\*\*: Ka6\- \*\*Opponent’s Move\*\*: f8g8\- \*\*Next State\*\*: 2r3k1/3r1p2/K4Q2/pp2pN2/P3P3/2P4P/1P3q2/8 w \- \- 5 47\- \*\*Subsequent Move\*\*: Kxa5\- \*\*Opponent’s Move\*\*: g8f8\- \*\*Next State\*\*: 2r2k2/3r1p2/5Q2/Kp2pN2/P3P3/2P4P/1P3q2/8 w \- \- 1 48Evaluation:1\. \*\*Kxc8 Trajectory\*\*: \- After Kxc8, the board simplifies, and the white king is now in a more exposed position\. The black king is still a threat, and the board is less controlled by the white queen\. \- The subsequent move Kb8 does not significantly improve the position, and the black king continues to be a threat\.2\. \*\*Ka6 Trajectory\*\*: \- After Ka6, the white king is more protected, and the board is less open\. The black king is still a threat, but the white queen has more control over the center and the board\. \- The subsequent move Kxa5 does not significantly improve the position, and the black king continues to be a threat\.\(Superficial evaluation\)Conclusion:While both moves have their merits, Ka6 provides better protection for the king and maintains more control over the board\. Therefore, the updated understanding is that Ka6 is a better move to explore further\.\*\*Next Move: Ka6\*\*\(Surface\-level evaluation, Over\-indexed on “evaluation form” rather than chess substance\)Experience Summary/Bootstrapping after RLThe simulation history has …\.Based on the simulation history, the position after the move Kxc8 would be a position with a queen on the board, while the position after the move Ka6 would be a position with a queen taken\. Therefore, the position after the move Kxc8 is better for the white player\.\(Outcome\-oriented, Evaluates the resulting material outcome\)

Table 12: Reasoning trace example THREE of SKL\-RL before and after end\-to\-end RL training\.Game State: r2r2k1/p4ppp/1p1R3b/n3p3/N7/2P2N2/P3K1PP/3R4 w \- \- 4 22State Understanding Before RLThe current position is as follows: r2r2k1/p4ppp/1p1R3b/n3p3/N7/2P2N2/P3K1PP/3R4 w \- \- 4 22Key Features:1\. \*\*White to Move\*\*: White is in the process of checkmating Black\.2\. \*\*Material\*\*: White has a rook, two knights, and a king\. Black has a king and a rook\.3\. \*\*Positional Evaluation\*\*:\- White’s rook is on the 6th rank, which is a strong position\.\- Black’s rook is on the 7th rank, which is a weak position\.\- White’s knights are well\-placed to attack Black’s king\.\- Black’s king is in the center, with no immediate escape routes\.Strategic Considerations:\- \*\*Checkmate Threat\*\*: White has a strong checkmate threat with the rook and knights\.\- \*\*Rook Control\*\*: The rook on the 6th rank is a key piece, controlling the 6th and 7th ranks\.\- \*\*Knight Maneuvering\*\*: The knights can be used to deliver checkmate or put pressure on Black’s king\.Candidate Moves\(spamming\):1\. \*\*Rd8\*\*: Checkmate with the rook\.2\. \*\*Rd7\*\*: Checkmate with the rook\.3\. \*\*Rxh6\*\*: Captures a piece and puts pressure on Black’s king\.4\. \*\*Rg6\*\*: Check with the rook, putting pressure on Black’s king\.5\. \*\*Rf6\*\*: Check with the rook, putting pressure on Black’s king\.6\. \*\*Rc6\*\*: Check with the rook, putting pressure on Black’s king\.7\. \*\*Rxb6\*\*: Captures a piece and puts pressure on Black’s king\.8\. \*\*Nxb6\*\*: Captures a piece and puts pressure on Black’s king\.9\. \*\*Nc5\*\*: Maneuvering the knight to a key square\.10\. \*\*Ng5\*\*: Maneuvering the knight to a key square\.Reasoning:\- \*\*Rd8\*\*: This move is the most direct and immediate checkmate threat\. If Black’s king cannot escape, this will result in checkmate\.\- \*\*Rd7\*\*: Similar to Rd8, but slightly less direct\. It still puts pressure on Black’s king and can lead to a quick checkmate\.\- \*\*Rxh6\*\*: Captures a piece and puts pressure on Black’s king\. This move is less direct but still a strong threat\.\- \*\*Nxb6\*\*: Captures a piece and puts pressure on Black’s king\. This move is also a strong threat but less direct than the rook moves\.Conclusion:The best move is \*\*Rd8\*\*, as it directly leads to checkmate if Black’s king cannot escape\. This move is the most immediate and decisive\.\*\*Next Move: Rd8\(Prefer to propose pure assertion, not conclusion, spamming candidate moves\)State Understanding After RLThis position is complex and dynamic, with White needing to take advantage of their material superiority and central control\.Candidate moves:\- ’Kd3’: This is too passive and doesn’t address the immediate threats\.\- ’Rxd8’: This sacrifices the rook but takes out a key black rook and opens up the board\.\- ’Kf2’: This centralizes the king and prepares to control key central squares\.After careful consideration, sacrificing the rook seems more aggressive and decisive\.’Rxd8’ is the most dynamic move\. It sacrifices the rook but takes out a key black rook and puts pressure on the black rook’s central position\.Therefore, I’ll go with:Rxd8\(More focused, Material awareness is actively considered\),\(but miss the checkmate advantage\)

### G\.6Reasoning trace for SKL\-RL with self\-distillation loss

Reasoning trace of SKL\-RL after adding self\-distillation objective functionℒDistill\\mathcal\{L\}\_\{\\text\{Distill\}\}\.Game State: r1b1k2r/5ppp/2p5/p1q1p1B1/Pp2P1n1/1P3NN1/1P3PPP/3Q1RK1 w kq \- 0 17\[State Understanding\] Based on the simulation history, our understanding of the game state is as follows:\(imprint from forward KL self\-distillation data\) 1\. ActionQd8appears to be the best action among the three\. It resulted in a perfect game termination with a reward of 1\.0\. This action clearly outperformed the other two actions in terms of long\-term value\. The simulation history forQd8is overwhelmingly positive, and it is the only action that led to a game termination with a perfect reward\. Therefore, we can confidently conclude that Qd8 is the best action to take in this game state\. We should not consider this action in future game play as it has demonstrated superior performance and potential for future success\.\(reasoning with the distilled experience\) 2\. ActionQd7did not perform well in the simulation history\. Although it did not result in a game termination, it did not yield any positive reward\. The simulation history forQd7did not provide any evidence of future value\. In fact, the game terminated with a reward of 0\.0, indicating that the action did not lead to a successful outcome\. The simulation history forQd7did not provide any evidence of future value, and it did not perform as well asQd8in terms of long\-term value\. Therefore, we can conclude thatQd7is not a good action to take in this game state\. 3\. ActionQd6…\[Same inference as 2\]\. However, we should not completely rule out the possibility that Qd6 could perform better in future game play\. We should continue to consider Qd6 as a potential action in future game play, but we should not give it the same level of consideration as Qd8\. We should be cautious in our use of Qd6 and only consider it as a potential action when Qd8 is not available\. We should also be aware that Qd6 did not perform as well as Qd7 in terms of long\-term value\. Therefore, we should give Qd6 the lowest level of consideration among the three actions\.\(careful and reflective about its own decisions\) \[In Brief\]: reasoning with previous knowledge, but not over\-confident\. \[Bootstrapping / Self\-Evaluation\] Based on the simulation history, my updated understanding of the game state is as follows: \#\#\# Detailed Analysis\(Recognize multiple subsequent steps in simulation\):1\.Move Qd8:\- Result: Terminated the game with a reward of 1\.0\.\-Conclusion: Qd8 is the best move and the only move that leads to a win\. It is the optimal move in this state\.2\.Move Qd7:\- Opponent’s move e8f8 led to a state where the game did not terminate\.\- Subsequent moves Qxg4, Qd2, and Rd1 all led to states where the game did not terminate\. \(More explicit trajectory analysis\)\-Conclusion: Qd7 is not a winning move, and it does not guarantee a win\. It is a suboptimal move\.3\.Move Qd6:\- Opponent’s move h8g8 led to a state where the game did not terminate\.\- Subsequent moves Qd2 and Ra1 both led to states where the game did not terminate\.\-Conclusion: Qd6 is not a winning move, and it does not guarantee a win\. It is a suboptimal move\.4\.Move Qd5:\- Opponent’s move h8g8 led to a state where the game did not terminate\.\- Subsequent moves Qd6 led to states where the game did not terminate\.\-Conclusion: Qd5 is not a winning move, and it does not guarantee a win\. It is a suboptimal move\. \#\#\# Summary:\(Well\-rounded consideration\)\-Qd8is the only move that guarantees a win and terminates the game with a reward of 1\.0\.\-Qd7, Qd6, and Qd5do not guarantee a win and do not terminate the game\. They all lead to states where the game continues and does not terminate\. Therefore, the updated understanding is:Q​d​8\\boxed\{Qd8\}is the best move to play in this state\. Qd7, Qd6, and Qd5 are suboptimal moves that do not guarantee a win\. Qd8 is the only move that solves the game\.