Chain-of-Experience for Continual LLM Improvement

# Chain-of-Experience for Continual LLM Improvement
Source: [https://arxiv.org/html/2608.18027](https://arxiv.org/html/2608.18027)
Haoqin Tu\*Affiliation:UC Santa CruzAffiliation:Bytedance Seed\* equal contribution and work done at Bytedance SeedYizhong WangAffiliation:Bytedance Seed\* equal contribution and work done at Bytedance SeedCihang XieAffiliation:UC Santa CruzShen YanAffiliation:Bytedance Seed\* equal contribution and work done at Bytedance Seed

###### Abstract

Humans continuously learn from experience, whereas conventional large language model \(LLM\) evaluations ignore the models’ ability to improve through inference\-time interaction\. In this paper, we study how LLMs learn from iterative experience at test time, a setting we refer to as Chain\-of\-Experience \(CoE\), where models accumulate experiential traces through iterative interactions with self or environmental feedback to form a continual improvement loop beyond zero\-shot inference\. We instantiate CoE with diverse feedback mechanisms, including model self\-feedback and environmental signals such as correctness or public coding test pass rates, and evaluate across math, coding, and knowledge domains using 8 LLMs, including GPT\-5, Gemini\-2\.5 Pro, Claude\-4\.5 Sonnet\. Our study shows that leveraging iterative experience consistently outperforms feedback\-free baselines, achieving substantial gains with self feedback alone, alongside a 5\.6% overall improvement and 19% lower API cost across tasks and models\. We further show that combining complementary feedback channels \(e\.g\., model and correctness signals\) yields additional gains, and that CoE delivers higher accuracy per token than existing test\-time strategies\. We observe a positive correlation between LLM base ability and improvement capacity, and show that models remain robust under weak or spurious feedback, with different feedback contributing to distinct improvement aspects and most gains emerging early in the iterations\.

## 1Introduction

Humans naturally and continuously learn from their experiences, each success or failure contributes to an evolving understanding that informs future decisions\. Contemporary machine learning systems — modern large language models \(LLMs\) in particular\([1](https://arxiv.org/html/2608.18027#bib.bib2);[53](https://arxiv.org/html/2608.18027#bib.bib1);[16](https://arxiv.org/html/2608.18027#bib.bib3);[29](https://arxiv.org/html/2608.18027#bib.bib4);[67](https://arxiv.org/html/2608.18027#bib.bib5)\)— behave quite differently: once trained, they are deployed in a fixed state, treating every inference as an isolated event and ignoring the rich feedback embedded in the problem solving process itself\. In contrast, learning should be a continuous process, LLMs should be able to improve not only during training but also at test\-time by iteratively interacting with their environment — processing feedback, engaging in reflection to constantly update their understanding, or shortly, improving from their own knowledge and experience\([48](https://arxiv.org/html/2608.18027#bib.bib8);[49](https://arxiv.org/html/2608.18027#bib.bib44)\)\.

Inspired by successes in reinforcement learning, search\-based methods explore a vast range of solutions\. Heuristic strategies\([69](https://arxiv.org/html/2608.18027#bib.bib7);[20](https://arxiv.org/html/2608.18027#bib.bib11)\), including majority voting\([59](https://arxiv.org/html/2608.18027#bib.bib9);[56](https://arxiv.org/html/2608.18027#bib.bib67)\)aim to identify higher\-quality answers leveraging verifiers from candidates parallelly generated by LMs\. Nevertheless, the model experience from these methods is still temporary, which is usually consolidated into asingle answerand then discarded, leaving models to restart each problem without accumulated insight\. An alternative paradigm is self\-refinement, where models are enabled to iteratively improve through self\-critique and correction\([33](https://arxiv.org/html/2608.18027#bib.bib13);[46](https://arxiv.org/html/2608.18027#bib.bib12)\), internalizing feedback within a single context\. Extensions such as self\-debugging\([9](https://arxiv.org/html/2608.18027#bib.bib24)\)and Reflexion[46](https://arxiv.org/html/2608.18027#bib.bib12)incorporate external signals like code execution signals, but they remain fragmented and shallow in the use of iterative experience\.

Building on these attempts, we conceptualize Chain\-of\-Experience \(CoE\), wherein models engage in iterative problem\-solving with feedback, and investigate:“How can LLMs evolve from accumulated experience with interactions and feedback to improve during test time?”We use the CoE framework to systematically explore prolonged environment interactions across four feedback types: none, model feedback, executor feedback for code tasks, and correctness feedback for general tasks\.

![Refer to caption](https://arxiv.org/html/2608.18027v1/teaser_v2.png)Figure 1:Summarized results on four benchmarks across math, code, and knowledge over four LLMs\. Iterative problem solving under CoE provides three benefits:*upper:*by incorporatingfeedbackin CoE, the average performance across four LLMs outperforms other test\-time augmentations;*lower left:*models are capable of achieving the better performance with lower API cost withfeedback;*lower right:*LLMs that perform better on the task display better improving capabilities through CoE with moderate to strong Pearson correlation\. We present more explanations regarding baselines in Section[4\.1](https://arxiv.org/html/2608.18027#S4.SS1)\.To evaluate CoE\-based algorithms, we prompt eight state\-of\-the\-art models, including GPT\-5, o3, Gemini\-2\.5 Pro, Claude\-4\.5 Sonnet, across math, coding, and knowledge domains\. Extensive experiments over four feedback types—drawn from existing algorithms and our designed paradigm \(i\.e\., none, executor, model, and correctness feedback\)—show that CoE with feedback consistently yields notable improvements \(Figure[1](https://arxiv.org/html/2608.18027#S1.F1)\)\. In particular, simple CoE methods substantially outperform test\-time scaling approaches that rely on experience from other tasks\([51](https://arxiv.org/html/2608.18027#bib.bib15);[74](https://arxiv.org/html/2608.18027#bib.bib21)\), achieving an average of 7\-9% gain over these algorithms with just self feedback \(average 62\.9% to 71\.0%\)\. Moreover, models leveraging iterative experience and feedback reason more efficiently, delivering a 5\.6% overall improvement with 19% lower API cost and higher accuracy per token across all tasks and models\. Combining complementary feedback channels \(e\.g\., model and correctness/executor signals\) yields further improvements, while memory\-based selection methods applied within task do not outperform full experience trails, suggesting that aggressive compression may discard critical intermediate reasoning\. We further observe a clear positive correlation between learning gains from feedback and base ability, with an average Pearson correlation of \+0\.5 across five benchmarks, indicating that stronger models evolve more effectively from experience\. Finally, our analyses reveal deeper behavioral insights: models remain robust under spurious or weak feedback, most gains occur early in the iterations, and distinct improvement trajectories emerge under different feedback types\.

## 2Related Work

#### Training\-free Test\-time Strategies\.

Large language models can reason without additional training via various test\-time prompting strategies\. Chain\-of\-Thought \(CoT\) elicits step\-by\-step reasoning and improves arithmetic, commonsense, and symbolic tasks\([63](https://arxiv.org/html/2608.18027#bib.bib6)\), spawning a family of “Chain\-of\-X” methods\([71](https://arxiv.org/html/2608.18027#bib.bib20);[27](https://arxiv.org/html/2608.18027#bib.bib19);[21](https://arxiv.org/html/2608.18027#bib.bib17);[12](https://arxiv.org/html/2608.18027#bib.bib16)\)\. Representative variants include contrastive CoT\([12](https://arxiv.org/html/2608.18027#bib.bib16)\), least\-to\-most prompting\([78](https://arxiv.org/html/2608.18027#bib.bib18)\), task\-specialized forms such as Chain\-of\-Explanation, Chain\-of\-Note, and Chain\-of\-Knowledge\([21](https://arxiv.org/html/2608.18027#bib.bib17);[71](https://arxiv.org/html/2608.18027#bib.bib20);[27](https://arxiv.org/html/2608.18027#bib.bib19)\), and Tree\-of\-Thought \(ToT\), which explores multiple reasoning paths via search and outperforms CoT on planning tasks\([69](https://arxiv.org/html/2608.18027#bib.bib7)\)\. With the emergence of large reasoning models such as OpenAI’soseries\([22](https://arxiv.org/html/2608.18027#bib.bib54)\)and DeepSeek R1\([18](https://arxiv.org/html/2608.18027#bib.bib55)\), verifier\-based methods that select among parallel generations have regained attention, including step\-level\([28](https://arxiv.org/html/2608.18027#bib.bib48);[57](https://arxiv.org/html/2608.18027#bib.bib49);[72](https://arxiv.org/html/2608.18027#bib.bib50)\)and output\-level\([76](https://arxiv.org/html/2608.18027#bib.bib53);[6](https://arxiv.org/html/2608.18027#bib.bib51);[31](https://arxiv.org/html/2608.18027#bib.bib52)\)verification\. While effective as post\-processing using external feedback\([30](https://arxiv.org/html/2608.18027#bib.bib56);[58](https://arxiv.org/html/2608.18027#bib.bib57);[55](https://arxiv.org/html/2608.18027#bib.bib58);[60](https://arxiv.org/html/2608.18027#bib.bib59)\), these methods lack the iterative loop for model evolving\. Our CoE is also training\-free, but differs by using feedback to drive iterative self\-evolving during inference\.

#### Learning from Experiences\.

Learning from experience underlies both human intelligence and AI systems\. Reinforcement learning formalizes experience through policy gradients and actor–critic methods\([65](https://arxiv.org/html/2608.18027#bib.bib61);[44](https://arxiv.org/html/2608.18027#bib.bib62)\), achieving success in games, robotics, and control\([47](https://arxiv.org/html/2608.18027#bib.bib10)\), and more recently via post\-training on online generations to improve alignment and reasoning\([3](https://arxiv.org/html/2608.18027#bib.bib63);[17](https://arxiv.org/html/2608.18027#bib.bib64);[45](https://arxiv.org/html/2608.18027#bib.bib65);[70](https://arxiv.org/html/2608.18027#bib.bib66)\)\. Beyond training, experience can accumulate during inference\. For cross\-task experience, Dynamic CheatSheet \(DC\)\([51](https://arxiv.org/html/2608.18027#bib.bib15)\), Agentic Context Engineering \(ACE\)\([74](https://arxiv.org/html/2608.18027#bib.bib21)\), and related approaches\([77](https://arxiv.org/html/2608.18027#bib.bib46);[61](https://arxiv.org/html/2608.18027#bib.bib47);[75](https://arxiv.org/html/2608.18027#bib.bib68)\)maintain persistent inference\-time memories that distill reusable strategies, while agentic scaffolds enable collective experience sharing across agents\([52](https://arxiv.org/html/2608.18027#bib.bib23);[8](https://arxiv.org/html/2608.18027#bib.bib40);[39](https://arxiv.org/html/2608.18027#bib.bib22);[19](https://arxiv.org/html/2608.18027#bib.bib78)\)\. For same\-task experience, Reflexion\([46](https://arxiv.org/html/2608.18027#bib.bib12)\), Self\-Refine\([33](https://arxiv.org/html/2608.18027#bib.bib13)\), Self\-Debug\([9](https://arxiv.org/html/2608.18027#bib.bib24)\), S\*\([26](https://arxiv.org/html/2608.18027#bib.bib45)\), Iteration\-of\-Thought\([40](https://arxiv.org/html/2608.18027#bib.bib25)\), and ReVeal\([25](https://arxiv.org/html/2608.18027#bib.bib26)\)iteratively refine outputs using self\-feedback or execution signals, while recent pipelines further exploit offline model experience for agent improvement\([73](https://arxiv.org/html/2608.18027#bib.bib43);[10](https://arxiv.org/html/2608.18027#bib.bib60)\)\. Most recent works\([66](https://arxiv.org/html/2608.18027#bib.bib77);[54](https://arxiv.org/html/2608.18027#bib.bib76)\)also extend such paradigm to real\-world usage\. In contrast to prior work, we present a unified framework that treats a model’s entire solving history as experience and systematically studies diverse feedback signals to enable continual improvement at test time\.

## 3Model Improvement via CoE

In this section, we first provide general concepts of the iterative problem solving setup, termedChain\-of\-Experience\(CoE\), followed by a detailed discussion of four diverse feedback types to enhance model experience\. Finally, we provide explanations on how we scale up the iteration of experience to probe model learning performance at test\-time\.

### 3\.1Overview

In the traditional question\-answering setting\([50](https://arxiv.org/html/2608.18027#bib.bib36);[41](https://arxiv.org/html/2608.18027#bib.bib35);[43](https://arxiv.org/html/2608.18027#bib.bib37);[5](https://arxiv.org/html/2608.18027#bib.bib14)\), when given a questionQQ, large language models \(LLMs\) will generate a plausible responseAAsampled from the conditional distributionP⁡\(A∣Q\)P\(A\\mid Q\)\. To extend this paradigm into the era of experience\([48](https://arxiv.org/html/2608.18027#bib.bib8)\), we incorporate an environment feedback variableFFto represent the observable consequence or evaluation of responses when grounded in an interactive environment\. Formally, the feedbackffis sampled from the conditional distributionf∼P′​\(F∣Q,A\)f\\sim P^\{\\prime\}\(F\\mid Q,A\), whereP′P^\{\\prime\}is modeled by an environment that can be instantiated as a coding execution environment, an internal world model of the agent, or even a real\-world environment, which provides generative\([38](https://arxiv.org/html/2608.18027#bib.bib38);[34](https://arxiv.org/html/2608.18027#bib.bib39)\)or environment feedback\([45](https://arxiv.org/html/2608.18027#bib.bib65);[33](https://arxiv.org/html/2608.18027#bib.bib13)\)for the given sequence\.

In this paper, we investigate a paradigm that extends the single\-turn formulation into a sequential decision process, where each responseaia\_\{i\}at stepiidepends on the full history and corresponding environmental feedback of prior attempts ; we refer to this setting asChain\-of\-Experience\(CoE\)\. The generative process is defined:

at∼P⁡\(at∣Q,e0,e1,…,et−1\)a\_\{t\}\\sim P\(a\_\{t\}\\mid Q,e\_\{0\},e\_\{1\},\\ldots,e\_\{t\-1\}\)\(1\)whereeie\_\{i\}is theit​hi^\{th\}experience consists of\(ai,fi\)\(a\_\{i\},f\_\{i\}\)\.

![Refer to caption](https://arxiv.org/html/2608.18027v1/coe_v2.png)Figure 2:An overview of our studies iterative improvement loop for LMs\. The modelℳ\\mathcal\{M\}learns by repeatedly interacting with the environmentℰ\\mathcal\{E\}\(e\.g\., model simulation or coding environment\)\. In each cycle,ℳ\\mathcal\{M\}generates actions conditioned on the input and accumulated experience, then receives feedback fromℰ\\mathcal\{E\}to form a new experience\. We investigate four specific feedback in this study: none, execution, model, and correctness feedback\.
### 3\.2Feedback Spectrum

To fully characterize how different forms of experience shape the model’s iterative evolution, we categorize feedback along a spectrum of richness — from completely implicit to strongly explicit signals\. Each feedback type corresponds to a specific instantiation of the environment’s response, which in turn influences the next\-step action via the CoE generative process\. In general, the model can be updated according to

at∼P⁡\(at∣Q,\(a0,f0\),…,\(at−1,ft−1\)\),a\_\{t\}\\sim P\(a\_\{t\}\\mid Q,\(a\_\{0\},f\_\{0\}\),\\ldots,\(a\_\{t\-1\},f\_\{t\-1\}\)\),where the functionfif\_\{i\}is the feedback we employed at thei−i\-th iteration\. Below, we describe these four feedback types used in this study, together with their formal definitions\.

No feedback\.The environment provides no evaluation or signal,i\.e\.,fi=∅f\_\{i\}=\\varnothing\. The experience reduces toei=\(ai,∅\)e\_\{i\}=\(a\_\{i\},\\varnothing\), the next action is sampled from

at∼P⁡\(at∣Q,a0,a1,…,at−1\),a\_\{t\}\\sim P\(a\_\{t\}\\mid Q,a\_\{0\},a\_\{1\},\\ldots,a\_\{t\-1\}\),meaning any improvement must arise from reflection on prior attempts, without external guidance\.

Execution feedback\.For tasks grounded in executable or interactive environments \(e\.g\., coding tasks with interpreters or unit tests\), feedback is generated by running the model’s responseaia\_\{i\}inside the environmentℰ\\mathcal\{E\}:

fi∼P′​\(F∣Q,ai\)=ℰ⁡\(Q,ai\),f\_\{i\}\\sim P^\{\\prime\}\(F\\mid Q,a\_\{i\}\)=\\mathcal\{E\}\(Q,a\_\{i\}\),wherefif\_\{i\}may include execution traces, error messages, runtime logs, or test\-case outcomes\.

Model feedback\.A \(possibly identical\) auxiliary language modelℳfb\\mathcal\{M\}\_\{\\text\{fb\}\}acts as a judge or criticfi=ℳfb​\(Q,ai\)f\_\{i\}=\\mathcal\{M\}\_\{\\text\{fb\}\}\(Q,a\_\{i\}\), wherefif\_\{i\}may include textual critiques, preference scores, or structured evaluations\. This enables refinement even in the absence of an external environment, relying purely on linguistic or preference\-based signals\.

Correctness feedback\.When a domain\-specific verifier is available, the environment supplies binary correctness signals:

fi=𝟏​\{ai​is correct\}∈\{0,1\}\.f\_\{i\}=\\mathbf\{1\}\\\{a\_\{i\}\\text\{ is correct\}\\\}\\in\\\{0,1\\\}\.Such oracle\-like information provides explicit fine\-grained evaluation of success and failure\. Although this type of feedback is often unrealistic in real\-world settings, where ground\-truth verification is costly or unavailable, still, we include it as a high\-signal reference setting to approximate an upper bound on the benefits of iterative refinement\.

## 4Experiments

### 4\.1Experiment Setup

Datasets\.We focus on three different tasks:math,coding, andknowledge\. More specific, we select two benchmarks for each task:AIME 2025\([4](https://arxiv.org/html/2608.18027#bib.bib27)\),OmniMath\([15](https://arxiv.org/html/2608.18027#bib.bib28)\),LiveCodebench \(V6\)\([23](https://arxiv.org/html/2608.18027#bib.bib29)\),LiveBench \(Code\)\([64](https://arxiv.org/html/2608.18027#bib.bib30)\),EvaLearn\([14](https://arxiv.org/html/2608.18027#bib.bib31)\), andGPQA Diamond\([42](https://arxiv.org/html/2608.18027#bib.bib32)\)\. Detailed descriptions are in Appendix[C](https://arxiv.org/html/2608.18027#A3)\.

Baselines\.As for baselines, we examine model skills in either \(1\) utilizing different levels of built\-in reasoning or \(2\) leveraging experience from previously solved problems\. For controlling reasoning depth, OpenAI and Claude models can be tuned to produce varying amounts of reasoning tokens\. For methods that absorb experience from prior examples, we adopt few\-shot in\-context learning \(ICL\)\([5](https://arxiv.org/html/2608.18027#bib.bib14)\)as a standard baseline\. For a more sophisticated approach, we select Dynamic CheatSheet\([51](https://arxiv.org/html/2608.18027#bib.bib15)\)and Agentic Context Engineering \(ACE\)\([74](https://arxiv.org/html/2608.18027#bib.bib21)\), which maintain a continually updated external memory of reusable strategies\. Although follow\-up works\([39](https://arxiv.org/html/2608.18027#bib.bib22);[7](https://arxiv.org/html/2608.18027#bib.bib74)\)introduce finer\-grained refinements in a similar processing loop, we use these two as the representative baseline\. We select the mostk∈\[1,5,8,12,15,20\]k\\in\[1,5,8,12,15,20\]relevant solutions for ICL, DC, and ACE to form their context, we present baseline details in Appendix[D](https://arxiv.org/html/2608.18027#A4)\.

Models\.We focus on the latest language models with inherent reasoning abilities from various developers to probe their improving capabilities during test\-time: GPT\-5\([36](https://arxiv.org/html/2608.18027#bib.bib69)\), GPT\-5\-mini\([36](https://arxiv.org/html/2608.18027#bib.bib69)\), o4\-mini\([37](https://arxiv.org/html/2608.18027#bib.bib70)\), o3\([37](https://arxiv.org/html/2608.18027#bib.bib70)\), o3\-mini\([37](https://arxiv.org/html/2608.18027#bib.bib70)\), Gemini\-2\.5 Pro[13](https://arxiv.org/html/2608.18027#bib.bib71), Claude 4\.5 Sonnet[2](https://arxiv.org/html/2608.18027#bib.bib72)\. We run all experiments for three times and report the mean and standard deviation statistics\. We present detailed prompting configurations in Appendix[A](https://arxiv.org/html/2608.18027#A1)\.

### 4\.2Scaling with Experience

![Refer to caption](https://arxiv.org/html/2608.18027v1/main_results_partial_v3.png)Figure 3:Results of five state\-of\-the\-art LLMs on six benchmarks using different generation techniques\. Models under CoE with different levels of feedback \(correctness/executor,self, andbinary executorfeedback\) generally perform better than the baseline strategies \(no feedback,DC,ICL, and the ones with differentreasoning efforts\)\. Results are averaged over 3 runs and we shade the standard deviation with a lighter color and plot bars showing model performance under different reasoning efforts\. The full results are in Appendix[H](https://arxiv.org/html/2608.18027#A8)\.Findings 1: Performance: The Chain\-of\-Experience setting with feedback boosts reasoning LLMs on various tasks\.

LLMs equipped with feedback consistently outperform almost all baselines and settings\. As shown in Figure[3](https://arxiv.org/html/2608.18027#S4.F3), across six benchmarks, the Chain\-of\-Experience \(CoE\) paradigm—where models iteratively absorb and reuse prior feedback—delivers substantial performance gains \(full results in Appendix[H](https://arxiv.org/html/2608.18027#A8)\)\. With self feedback or executor/correctness feedback \(as an upper bound\), eight modern reasoning models achieve average improvements of 5\.6% and 11\.1% over their no\-feedback counterparts, underscoring the value of explicit outcome\-based signals for model refinement\. For coding\-centric tasks such as LiveBench \(Code\) and LiveCodeBench \(V6\), programmatic executor feedback derived from public test\-case verification drives sharp accuracy gains of 8\.6% on average \(from 66\.4% to 75\.0%\), while self\-judgement feedback still provides a 7\.0% lift\. This indicates that models can internalize both fine\-grained and abstract feedback cues into subsequent reasoning\. On non\-coding tasks \(e\.g\., AIME 2025, OmniMath, GPQA Diamond\), the same trend holds: correctness feedback establishes an upper bound, and self feedback—though noisier—continues to foster improvement \(e\.g\., 75\.1%\>\>67\.1%\>\>62\.5% w/o feedback\)\.

In comparison, although ICL, DC, and ACE remain widely used, none demonstrates reliable scaling under advanced reasoning models\. Averaged across six benchmarks in Table[3](https://arxiv.org/html/2608.18027#A2.T3), ICL, ACE, and DC achieve only 62\.1%, 64\.0%, and 62\.7% respectively, all trailing a simple without baseline \(66\.8%\)\. In contrast, incorporating explicit feedback yields consistent gains: results with model self feedback reaches 71\.0% \(\+7\-9% over ICL/ACE/DC\), while the best feedback signal further improves performance to 79\.3%\. Overall, these results proves that feedback\-driven CoE acts as a more general and effective test\-time scaling framework, enabling LLMs to autonomously improve across domains over no\-feedback inference and other baselines\.

![Refer to caption](https://arxiv.org/html/2608.18027v1/price_v3.png)Figure 4:Total cost of each model over task completion vs\. its best performance within 20 iterations\. LLMs withfeedbackgenerally achieve higher scores with fewer costs \(at the upper left\), while iterative experience without feedback generally falls behind \(at lower right\)\. We provide full results over six benchmarks in Appendix[H](https://arxiv.org/html/2608.18027#A8)\.Findings 2: Efficiency: LLMs with feedback strikes a balance between performance and API calling cost\.

As shown in Figure[4](https://arxiv.org/html/2608.18027#S4.F4), incorporating feedback into CoE not only enhances performance but also improves efficiency — models with feedback tend to achieve higher accuracy at lower API costs \(upper left of sub\-figures\)\. Across tasks, most LLMs using feedback\-based variants \(blue objects\) consistently dominate the no feedback baselines \(gray square\), except Gemini 2\.5 Pro\. Full results are in Figure[12](https://arxiv.org/html/2608.18027#A8.F12), Appendix[H](https://arxiv.org/html/2608.18027#A8)\. For the two coding tasks,self feedbackemerges as a cost\-effective compromise: it captures the majority of the performance gain ofexecutor\(i\.e\., 73\.4% vs\. 75\.0%\) while incurring 13\.4% fewer API calls \(e\.g\., $70\.7 vs\. $81\.6\), and moreover, requires 20% less cost than theno\-feedbackcounterpart\. Similarly, on AIME 2025 and EvaLearn,self feedbackprovides more informative input with substantially lower API cost with 47\.3% and 7\.0% reductions across all eight LLMs, even compared with theno\-feedbacksolution \(e\.g\., $8\.8 vs\. $4\.6 on AIME 25 and $325\.3 vs\. $302\.4 on EvaLearn\), while still yielding an average 4\.4% and 6\.9% accuracy improvement, respectively\. One exception is Gemini 2\.5 Pro, whose self\-feedback produces more verbose output with smaller gains, suggesting it may be less effective at self\-evaluation than other LLMs\. Beyond API cost, we further analyze token\-level efficiency in Table[5](https://arxiv.org/html/2608.18027#A9.T5)\(Appendix[I](https://arxiv.org/html/2608.18027#A9)\)\. CoE with correctness/executor feedback achieves the best accuracy with total token counts comparable to other multi\-round methods, yielding substantially higher return per token\. For example, on AIME 2025, correctness feedback reaches 84\.6% with 108K tokens, while DC uses only 11K tokens yet achieves a lower 74\.7%, and no\-feedback CoE consumes similar tokens \(107K\) but trails at 74\.1%\. This pattern holds consistently across benchmarks, confirming that CoE reallocates compute into feedback iterations and generates less verbose outputs, rather than simply inflating prompt length\.

Figure 5:Zero\-shot performance of models \(Base Capacity\) and the learning gain show positive Pearson correlations \(r\), indicating that better\-performing LLMs possess greater improving capability\. Scores are averaged across 3 runs\.Findings 3: Improving Capability: LLMs that perform better on the task shows higher learning gain during test\-time\.

We calculate the improving capability of a modelℳ\\mathcal\{M\}usingΔℳ=Smax−Sbase1−Sbase\\Delta\_\{\\mathcal\{M\}\}=\\frac\{S\_\{\\text\{max\}\}\-S\_\{\\text\{base\}\}\}\{1\-S\_\{\\text\{base\}\}\}, whereSbaseS\_\{\\text\{base\}\}denotes the model’s initial zero\-shot performance without feedback andSmaxS\_\{\\text\{max\}\}represents its peak accuracy achieved under our CoE setting with model self feedback\. All numbers are averaged across three runs\. In Figure[5](https://arxiv.org/html/2608.18027#S4.F5), we observe a clear positive trend between base performance and improving capability across benchmarks\. For instance, models on both coding taskse\.g\., LiveBench \(Code\) withr=0\.97r=0\.97\) and LiveCodeBench \(V6\) \(r=0\.83r=0\.83\) show strong correlations, indicating that models with stronger initial reasoning ability tend to learn more effectively from feedback\. Although tasks like AIME 2025 \(r=0\.33r=0\.33\) and OmniMath \(r=0\.24r=0\.24\) show relatively weaker correlations, the overall trend remains consistent, with an average task\-level Pearson correlation of 0\.50\. These findings suggest that learning from experience is an emergent property that scales with model capacity — stronger LLMs are inherently better at digesting feedback and improving through interactions\.

Figure 6:Percentages of different reasons for LLMs’ improvement patterns from 6,630 incorrect to correct response pairs\. We employ GPT\-5 to conduct this automatic analysis\.

## 5Further Discussion and Conclusion

To further investigate LLMs’ learning capacity under CoE, we conduct ablations on the iterative problem\-solving setup, including spurious feedback, improvement pattern analysis, dual feedback combinations, and principled experience selection\. We additionally study model behavior under varying feedback strengths \(Appendix[F\.3](https://arxiv.org/html/2608.18027#A6.SS3)\) and extended experience iterations \(Appendix[F\.2](https://arxiv.org/html/2608.18027#A6.SS2)\)\.

Learning from Spurious Feedback\.To investigate the robustness of LLMs under spurious feedback, we design an experiment where models receive exclusivelyincorrectorcorrectfeedback \(e\.g\., always stating “the answer is incorrect” or vice versa\)\. We report the best model performance over 20 iterations in Table[2](https://arxiv.org/html/2608.18027#S5.T2)\. This setup evaluates whether models can recover from \(or even benefit under\) misleading feedback signals\. We find that spurious feedback generally degrades performance by average 7\.6% on AIME 2025 and 2\.6% on GPQA\-Diamond\. Yet, stronger models such as GPT\-5 mini exhibit greater robustness, with only minor drops of 2\.5% and 0\.6%, compared to o4\-mini’s declines of 12\.8% and 4\.6%\.

To further enhance reliability, we introduceSelective Majority Voting\(SelMV\-nn\), which aggregates final answers via majority voting among the firstnnvalid attempts\. Interestingly, on GPQA\-Diamond, SelMV with incorrect feedback surpasses model feedback by 0\.9% \(79\.4%→\\to80\.3%\), while accuracies after SelMV improve by average 1\.2% and 2\.3% on AIME 2025 and GPQA\-Diamond, respectively, underscoring the robustness of reasoning LLMs despite adversarial signals\. Further results on spurious and different levels of feedback are in Appendix[F\.1](https://arxiv.org/html/2608.18027#A6.SS1)and[F\.3](https://arxiv.org/html/2608.18027#A6.SS3)\.

Analysis of Improvement Patterns\.To better understand reasons for the improvement of LLMs through iterative feedback, we design the experiments to analyze the “why” behind each flip from incorrect to correct answer of LLMs\. We collect 6,630 examples across all tasks and five models \(i\.e\., GPT\-5, GPT\-5 mini, o4\-mini, o3, o3\-mini\) and leverage the latest GPT\-5 model to analyze the cause\. To validate this automated analysis, we compute Cohen’s Kappa between GPT\-5 and human ratings on 100 randomly sampled trajectories \(25 per category\), achieving 76\.8% agreement, indicating substantial agreement \(details in Appendix[F\.5](https://arxiv.org/html/2608.18027#A6.SS5)\)\. We define four factors behind the improvement of LLMs: Feedback Fidelity for improving from trail feedback, Self Reflection for referring to self\-reflection, Specification Recall for correcting based on the question and/or format requirements, Random for model improving from other reasons\. We present the detailed criteria of these factors and prompts in Appendix[F\.4](https://arxiv.org/html/2608.18027#A6.SS4)and[K](https://arxiv.org/html/2608.18027#A11)\.

In Figure[6](https://arxiv.org/html/2608.18027#S4.F6), we observe: \(1\) 47\.7% of all improvements are feedback\-driven, confirming that LLMs meaningfully interpret and act on feedback signals\. \(2\) In coding tasks, 30\.0% of improvements stem from specification recall, reflecting the format\- and syntax\-sensitive nature of these tasks\. \(3\) Model\-generated feedback elicits a higher feedback\-related improvement proportion than other sources \(e\.g\., 58\.7% vs\. 41\.1%\), suggesting self\-generated feedback is more contextually aligned\.

Dual Feedback CoE\.To test whether multiple feedback channels are complementary, we combine model feedback with correctness \(math\) or executor feedback \(coding\) using Claude 4\.5 Sonnet\. As shown in Table[1](https://arxiv.org/html/2608.18027#S5.T1), dual feedback shows clear complementarity: on AIME 2025, dual feedback reaches 76\.7%, surpassing correctness\-only \(70\.0%\) and model\-only \(60\.0%\); on LiveBench \(Code\), dual feedback achieves 81\.2% vs\. 78\.1% \(executor\) and 57\.8% \(model\)\. On the more challenging OmniMath, gains are marginal \(73\.5% vs\. 74\.5% correctness\-only\), with correctness signals typically pushing best rounds later\. These results suggest that feedback channels contribute complementary signals, though the marginal benefit depends on task difficulty\. Full results are provided in Appendix[J](https://arxiv.org/html/2608.18027#A10)\.

CoE with Principled Experience Selection\.We further investigate whether memory\-based selection and compression mechanisms add benefit beyond full experience trails by combining CoE with Dynamic CheatSheet \(DC\)\([51](https://arxiv.org/html/2608.18027#bib.bib15)\)and SimpleMem\([32](https://arxiv.org/html/2608.18027#bib.bib75)\)under the same within\-task protocol \(no cross\-task leakage\)\. As shown in Table[1](https://arxiv.org/html/2608.18027#S5.T1), these methods do not outperform pure self\-feedback: on AIME 2025, model feedback alone achieves 60\.0% vs\. 50\.0% \(\+DC\) and 56\.7% \(\+SimpleMem\); on LiveBench \(Code\), 57\.8% vs\. 51\.6% \(\+DC\) and 54\.7% \(\+SimpleMem\), suggesting that aggressive compression may discard critical intermediate reasoning\.

AIME 2025LiveBench \(Code\)OmniMathSettingAccBest RAccBest RAccBest RDual \(Model \+ Corr/Exec\)76\.7R1981\.2R1573\.5R17Correctness / Executor70\.0R1378\.1R1574\.5R17Binary Executor––71\.9R13––Model60\.0R657\.8R1750\.5R9Model \+ DC50\.0R851\.6R1546\.0R10Model \+ SimpleMem56\.7R654\.7R1749\.5R12Table 1:Dual feedback and principled experience selection results using Claude 4\.5 Sonnet\. Dual feedback combines model feedback with correctness \(math\) or executor \(code\) signals\. Memory\-based methods \(DC, SimpleMem\) are applied within\-task with no cross\-task leakage\. Acc: best accuracy \(%\) over 20 iterations; Best R: iteration achieving best performance\.FeedbackAIME 2025GPQA DiamondGPT\-5minio4miniGPT\-5minio4miniSelf93\.391\.179\.978\.8SelMV Self91\.188\.980\.479\.5AllCorrect90\.073\.379\.375\.8SelMVCorrect93\.373\.379\.376\.3Incorrect91\.783\.379\.372\.7SelMVIncorrect89\.786\.782\.877\.8Table 2:The best performance over 20 iterations under constantcorrect” orincorrect” feedback \(e\.g\., “the answer is correct”\)\. Selective majority voting \(SelMV\) helps LLMs maintain performance\. Results are averaged over 3 runs with best scoresemphasized\.Conclusion\.We present a comprehensive analysis of Chain\-of\-Experience \(CoE\), showing that LLMs can improve during inference through iterative feedback and accumulated experience\. Across math, coding, and knowledge tasks, methods in CoE consistently enhances performance and efficiency, demonstrating the effectiveness of feedback\-driven test\-time learning\. Our analysis also reveals a positive correlation between model ability and improvement capacity, and shows that most gains emerge early, even under weak or spurious feedback\. Finally, we analyze various model improvement patterns during this iterative process\.

## 6Acknowledgement

The authors sincerely thank Deyao Zhu, Shu Zhong, and Guang Shi for providing valuable feedback and discussions on the experimental part and presentation of the paper\.

## References

- Achiamet al\.\(2023\)J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§1](https://arxiv.org/html/2608.18027#S1.p1.1)\.
- Anthropic \(2025\)AnthropicIntroducing Claude Sonnet 4\.5\.Note:[https://www\.anthropic\.com/news/claude\-sonnet\-4\-5](https://www.anthropic.com/news/claude-sonnet-4-5)Accessed: 2025\-11\-17Cited by:[§4\.1](https://arxiv.org/html/2608.18027#S4.SS1.p3.1)\.
- Baiet al\.\(2022\)Y\. Bai, A\. Jones, K\. Ndousse, A\. Askell, A\. Chen, N\. DasSarma, D\. Drain, S\. Fort, D\. Ganguli, T\. Henighan,et al\.Training a helpful and harmless assistant with reinforcement learning from human feedback\.arXiv preprint arXiv:2204\.05862\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px2.p1.1)\.
- Balunovićet al\.\(2025\)M\. Balunović, J\. Dekoninck, I\. Petrov, N\. Jovanović, and M\. VechevMathArena: evaluating llms on uncontaminated math competitions\.SRI Lab, ETH Zurich\.External Links:[Link](https://matharena.ai/)Cited by:[1st item](https://arxiv.org/html/2608.18027#A3.I1.i1.p1.1),[§4\.1](https://arxiv.org/html/2608.18027#S4.SS1.p1.1)\.
- Brownet al\.\(2020\)T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.Language models are few\-shot learners\.Advances in neural information processing systems33,pp\. 1877–1901\.Cited by:[§3\.1](https://arxiv.org/html/2608.18027#S3.SS1.p1.1),[§4\.1](https://arxiv.org/html/2608.18027#S4.SS1.p2.1)\.
- Caiet al\.\(2024\)Z\. Cai, M\. Cao, H\. Chen, K\. Chen, K\. Chen, X\. Chen, X\. Chen, Z\. Chen, Z\. Chen, P\. Chu,et al\.Internlm2 technical report\.arXiv preprint arXiv:2403\.17297\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px1.p1.1)\.
- Caiet al\.\(2025\)Z\. Cai, X\. Guo, Y\. Pei, J\. Feng, J\. Chen, Y\. Zhang, W\. Ma, M\. Wang, and H\. ZhouFlex: continuous agent evolution via forward learning from experience\.arXiv preprint arXiv:2511\.06449\.Cited by:[§4\.1](https://arxiv.org/html/2608.18027#S4.SS1.p2.1)\.
- Chenet al\.\(2025a\)S\. Chen, S\. Lin, X\. Gu, Y\. Shi, H\. Lian, L\. Yun, D\. Chen, W\. Sun, L\. Cao, and Q\. WangSwe\-exp: experience\-driven software issue resolution\.arXiv preprint arXiv:2507\.23361\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px2.p1.1)\.
- Chenet al\.\(2023\)X\. Chen, M\. Lin, N\. Schärli, and D\. ZhouTeaching large language models to self\-debug\.arXiv preprint arXiv:2304\.05128\.Cited by:[§1](https://arxiv.org/html/2608.18027#S1.p2.1),[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px2.p1.1)\.
- Chenet al\.\(2025b\)Z\. Chen, Z\. Zhao, K\. Zhang, B\. Liu, Q\. Qi, Y\. Wu, T\. Kalluri, S\. Cao, Y\. Xiong, H\. Tong,et al\.Scaling agent learning via experience synthesis\.arXiv preprint arXiv:2511\.03773\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px2.p1.1)\.
- Chenet al\.\(2025c\)Z\. Chen, X\. Ma, S\. Zhuang, P\. Nie, K\. Zou, A\. Liu, J\. Green, K\. Patel, R\. Meng, M\. Su,et al\.Browsecomp\-plus: a more fair and transparent evaluation benchmark of deep\-research agent\.arXiv preprint arXiv:2508\.06600\.Cited by:[Appendix G](https://arxiv.org/html/2608.18027#A7.p1.1)\.
- Chiaet al\.\(2023\)Y\. K\. Chia, G\. Chen, L\. A\. Tuan, S\. Poria, and L\. BingContrastive chain\-of\-thought prompting\.arXiv preprint arXiv:2311\.09277\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px1.p1.1)\.
- Comaniciet al\.\(2025\)G\. Comanici, E\. Bieber, M\. Schaekermann, I\. Pasupat, N\. Sachdeva, I\. Dhillon, M\. Blistein, O\. Ram, D\. Zhang, E\. Rosen,et al\.Gemini 2\.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities\.arXiv preprint arXiv:2507\.06261\.Cited by:[§4\.1](https://arxiv.org/html/2608.18027#S4.SS1.p3.1)\.
- Douet al\.\(2025\)S\. Dou, M\. Zhang, C\. Huang, J\. Chen, F\. Chen, S\. Liu, Y\. Liu, C\. Liu, C\. Zhong, Z\. Zhang,et al\.EvaLearn: quantifying the learning capability and efficiency of llms via sequential problem solving\.arXiv preprint arXiv:2506\.02672\.Cited by:[5th item](https://arxiv.org/html/2608.18027#A3.I1.i5.p1.1),[§4\.1](https://arxiv.org/html/2608.18027#S4.SS1.p1.1)\.
- Gaoet al\.\(2024\)B\. Gao, F\. Song, Z\. Yang, Z\. Cai, Y\. Miao, Q\. Dong, L\. Li, C\. Ma, L\. Chen, R\. Xu,et al\.Omni\-math: a universal olympiad level mathematic benchmark for large language models\.arXiv preprint arXiv:2410\.07985\.Cited by:[2nd item](https://arxiv.org/html/2608.18027#A3.I1.i2.p1.1),[Appendix E](https://arxiv.org/html/2608.18027#A5.p8.1),[§4\.1](https://arxiv.org/html/2608.18027#S4.SS1.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§1](https://arxiv.org/html/2608.18027#S1.p1.1)\.
- Guanet al\.\(2024\)M\. Y\. Guan, M\. Joglekar, E\. Wallace, S\. Jain, B\. Barak, A\. Helyar, R\. Dias, A\. Vallone, H\. Ren, J\. Wei,et al\.Deliberative alignment: reasoning enables safer language models\.arXiv preprint arXiv:2412\.16339\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px2.p1.1)\.
- Guoet al\.\(2025\)D\. Guo, D\. Yang, H\. Zhang, J\. Song, R\. Zhang, R\. Xu, Q\. Zhu, S\. Ma, P\. Wang, X\. Bi,et al\.Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px1.p1.1)\.
- Hanet al\.\(2026\)Q\. Han, H\. Tu, Z\. Wang, H\. Dai, Y\. Zhou, N\. Lau, A\. A\. Cardenas, Y\. Xu, R\. Xu, C\. Xiong,et al\.VLAA\-gui: knowing when to stop, recover, and search, a modular framework for gui automation\.arXiv preprint arXiv:2604\.21375\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px2.p1.1)\.
- Haoet al\.\(2023\)S\. Hao, Y\. Gu, H\. Ma, J\. J\. Hong, Z\. Wang, D\. Z\. Wang, and Z\. HuReasoning with language model is planning with world model\.arXiv preprint arXiv:2305\.14992\.Cited by:[§1](https://arxiv.org/html/2608.18027#S1.p2.1)\.
- Huanget al\.\(2023\)F\. Huang, H\. Kwak, and J\. AnChain of explanation: new prompting method to generate quality natural language explanation for implicit hate speech\.InCompanion proceedings of the ACM Web conference 2023,pp\. 90–93\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px1.p1.1)\.
- Jaechet al\.\(2024\)A\. Jaech, A\. Kalai, A\. Lerer, A\. Richardson, A\. El\-Kishky, A\. Low, A\. Helyar, A\. Madry, A\. Beutel, A\. Carney,et al\.Openai o1 system card\.arXiv preprint arXiv:2412\.16720\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px1.p1.1)\.
- Jainet al\.\(2024\)N\. Jain, K\. Han, A\. Gu, W\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. StoicaLivecodebench: holistic and contamination free evaluation of large language models for code\.arXiv preprint arXiv:2403\.07974\.Cited by:[3rd item](https://arxiv.org/html/2608.18027#A3.I1.i3.p1.1),[§4\.1](https://arxiv.org/html/2608.18027#S4.SS1.p1.1)\.
- Jimenezet al\.\(2023\)C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. NarasimhanSwe\-bench: can language models resolve real\-world github issues?\.arXiv preprint arXiv:2310\.06770\.Cited by:[Appendix L](https://arxiv.org/html/2608.18027#A12.p1.1)\.
- Jinet al\.\(2025\)Y\. Jin, K\. Xu, H\. Li, X\. Han, Y\. Zhou, C\. Li, and J\. BaiReVeal: self\-evolving code agents via iterative generation\-verification\.arXiv preprint arXiv:2506\.11442\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px2.p1.1)\.
- Liet al\.\(2025\)D\. Li, S\. Cao, C\. Cao, X\. Li, S\. Tan, K\. Keutzer, J\. Xing, J\. E\. Gonzalez, and I\. StoicaS\*: test time scaling for code generation\.arXiv preprint arXiv:2502\.14382\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px2.p1.1)\.
- Liet al\.\(2023\)X\. Li, R\. Zhao, Y\. K\. Chia, B\. Ding, S\. Joty, S\. Poria, and L\. BingChain\-of\-knowledge: grounding large language models via dynamic knowledge adapting over heterogeneous sources\.arXiv preprint arXiv:2305\.13269\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px1.p1.1)\.
- Lightmanet al\.\(2023\)H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. CobbeLet’s verify step by step\.InThe Twelfth International Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px1.p1.1)\.
- Liuet al\.\(2024a\)A\. Liu, B\. Feng, B\. Xue, B\. Wang, B\. Wu, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang, C\. Ruan,et al\.Deepseek\-v3 technical report\.arXiv preprint arXiv:2412\.19437\.Cited by:[§1](https://arxiv.org/html/2608.18027#S1.p1.1)\.
- Liuet al\.\(2024b\)C\. Y\. Liu, L\. Zeng, J\. Liu, R\. Yan, J\. He, C\. Wang, S\. Yan, Y\. Liu, and Y\. ZhouSkywork\-reward: bag of tricks for reward modeling in llms\.arXiv preprint arXiv:2410\.18451\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px1.p1.1)\.
- Liuet al\.\(2025\)C\. Y\. Liu, L\. Zeng, Y\. Xiao, J\. He, J\. Liu, C\. Wang, R\. Yan, W\. Shen, F\. Zhang, J\. Xu,et al\.Skywork\-reward\-v2: scaling preference data curation via human\-ai synergy\.arXiv preprint arXiv:2507\.01352\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px1.p1.1)\.
- Liuet al\.\(2026\)J\. Liu, Y\. Su, P\. Xia, S\. Han, Z\. Zheng, C\. Xie, M\. Ding, and H\. YaoSimpleMem: efficient lifelong memory for llm agents\.arXiv preprint arXiv:2601\.02553\.Cited by:[Appendix J](https://arxiv.org/html/2608.18027#A10.p1.1),[§5](https://arxiv.org/html/2608.18027#S5.p7.1)\.
- Madaanet al\.\(2023\)A\. Madaan, N\. Tandon, P\. Gupta, S\. Hallinan, L\. Gao, S\. Wiegreffe, U\. Alon, N\. Dziri, S\. Prabhumoye, Y\. Yang,et al\.Self\-refine: iterative refinement with self\-feedback\.Advances in Neural Information Processing Systems36,pp\. 46534–46594\.Cited by:[§1](https://arxiv.org/html/2608.18027#S1.p2.1),[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2608.18027#S3.SS1.p1.1)\.
- Mahanet al\.\(2024\)D\. Mahan, D\. Van Phung, R\. Rafailov, C\. Blagden, N\. Lile, L\. Castricato, J\. Fränken, C\. Finn, and A\. AlbalakGenerative reward models\.arXiv preprint arXiv:2410\.12832\.Cited by:[§3\.1](https://arxiv.org/html/2608.18027#S3.SS1.p1.1)\.
- OpenAI \(2024\)OpenAINew embedding models and API updates\.Note:[https://openai\.com/index/new\-embedding\-models\-and\-api\-updates/](https://openai.com/index/new-embedding-models-and-api-updates/)Accessed: 2025\-11\-17Cited by:[Appendix D](https://arxiv.org/html/2608.18027#A4.p2.1)\.
- OpenAI \(2025a\)OpenAIIntroducing GPT\-5\.Note:[https://openai\.com/index/introducing\-gpt\-5/](https://openai.com/index/introducing-gpt-5/)Accessed: 2025\-11\-17Cited by:[§4\.1](https://arxiv.org/html/2608.18027#S4.SS1.p3.1)\.
- OpenAI \(2025b\)OpenAIIntroducing OpenAI o3 and o4\-mini\.Note:[https://openai\.com/index/introducing\-o3\-and\-o4\-mini/](https://openai.com/index/introducing-o3-and-o4-mini/)Accessed: 2025\-11\-17Cited by:[§4\.1](https://arxiv.org/html/2608.18027#S4.SS1.p3.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[§3\.1](https://arxiv.org/html/2608.18027#S3.SS1.p1.1)\.
- Ouyanget al\.\(2025\)S\. Ouyang, J\. Yan, I\. Hsu, Y\. Chen, K\. Jiang, Z\. Wang, R\. Han, L\. T\. Le, S\. Daruki, X\. Tang,et al\.ReasoningBank: scaling agent self\-evolving with reasoning memory\.arXiv preprint arXiv:2509\.25140\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2608.18027#S4.SS1.p2.1)\.
- Radhaet al\.\(2024\)S\. K\. Radha, Y\. N\. Jelyani, A\. Ghukasyan, and O\. GoktasIteration of thought: leveraging inner dialogue for autonomous large language model reasoning\.arXiv preprint arXiv:2409\.12618\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px2.p1.1)\.
- Raffelet al\.\(2020\)C\. Raffel, N\. Shazeer, A\. Roberts, K\. Lee, S\. Narang, M\. Matena, Y\. Zhou, W\. Li, and P\. J\. LiuExploring the limits of transfer learning with a unified text\-to\-text transformer\.Journal of machine learning research21\(140\),pp\. 1–67\.Cited by:[§3\.1](https://arxiv.org/html/2608.18027#S3.SS1.p1.1)\.
- Reinet al\.\(2024\)D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. BowmanGpqa: a graduate\-level google\-proof q&a benchmark\.InFirst Conference on Language Modeling,Cited by:[6th item](https://arxiv.org/html/2608.18027#A3.I1.i6.p1.1),[§4\.1](https://arxiv.org/html/2608.18027#S4.SS1.p1.1)\.
- Robertset al\.\(2020\)A\. Roberts, C\. Raffel, and N\. ShazeerHow much knowledge can you pack into the parameters of a language model?\.arXiv preprint arXiv:2002\.08910\.Cited by:[§3\.1](https://arxiv.org/html/2608.18027#S3.SS1.p1.1)\.
- Schulmanet al\.\(2017\)J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. KlimovProximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px2.p1.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2608.18027#S3.SS1.p1.1)\.
- Shinnet al\.\(2023\)N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. YaoReflexion: language agents with verbal reinforcement learning\.Advances in Neural Information Processing Systems36,pp\. 8634–8652\.Cited by:[§1](https://arxiv.org/html/2608.18027#S1.p2.1),[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px2.p1.1)\.
- Silveret al\.\(2016\)D\. Silver, A\. Huang, C\. J\. Maddison, A\. Guez, L\. Sifre, G\. Van Den Driessche, J\. Schrittwieser, I\. Antonoglou, V\. Panneershelvam, M\. Lanctot,et al\.Mastering the game of go with deep neural networks and tree search\.nature529\(7587\),pp\. 484–489\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px2.p1.1)\.
- Silver and Sutton \(2025\)D\. Silver and R\. S\. SuttonWelcome to the era of experience\.Google AI1\.Cited by:[§1](https://arxiv.org/html/2608.18027#S1.p1.1),[§3\.1](https://arxiv.org/html/2608.18027#S3.SS1.p1.1)\.
- Snellet al\.\(2025\)C\. V\. Snell, J\. Lee, K\. Xu, and A\. KumarScaling llm test\-time compute optimally can be more effective than scaling parameters for reasoning\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.18027#S1.p1.1)\.
- Sutskeveret al\.\(2014\)I\. Sutskever, O\. Vinyals, and Q\. V\. LeSequence to sequence learning with neural networks\.Advances in neural information processing systems27\.Cited by:[§3\.1](https://arxiv.org/html/2608.18027#S3.SS1.p1.1)\.
- Suzgunet al\.\(2025\)M\. Suzgun, M\. Yuksekgonul, F\. Bianchi, D\. Jurafsky, and J\. ZouDynamic cheatsheet: test\-time learning with adaptive memory\.arXiv preprint arXiv:2504\.07952\.Cited by:[Appendix J](https://arxiv.org/html/2608.18027#A10.p1.1),[Appendix D](https://arxiv.org/html/2608.18027#A4.p3.1.1),[§1](https://arxiv.org/html/2608.18027#S1.p4.1),[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2608.18027#S4.SS1.p2.1),[§5](https://arxiv.org/html/2608.18027#S5.p7.1)\.
- Tanget al\.\(2025\)X\. Tang, T\. Qin, T\. Peng, Z\. Zhou, D\. Shao, T\. Du, X\. Wei, P\. Xia, F\. Wu, H\. Zhu,et al\.Agent kb: leveraging cross\-domain experience for agentic problem solving\.arXiv preprint arXiv:2507\.06229\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px2.p1.1)\.
- Teamet al\.\(2023\)G\. Team, R\. Anil, S\. Borgeaud, J\. Alayrac, J\. Yu, R\. Soricut, J\. Schalkwyk, A\. M\. Dai, A\. Hauth, K\. Millican,et al\.Gemini: a family of highly capable multimodal models\.arXiv preprint arXiv:2312\.11805\.Cited by:[§1](https://arxiv.org/html/2608.18027#S1.p1.1)\.
- Tuet al\.\(2026\)H\. Tu, J\. Chen, Z\. Wang, S\. Han, J\. Wu, H\. Chen, H\. Ji, K\. Xiong, J\. Liu, P\. Xia,et al\.VisualClaw: a real\-time, personalized agent for the physical world\.arXiv preprint arXiv:2606\.16295\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px2.p1.1)\.
- Tuet al\.\(2025\)H\. Tu, W\. Feng, H\. Chen, H\. Liu, X\. Tang, and C\. XieVilbench: a suite for vision\-language process reward modeling\.arXiv preprint arXiv:2503\.20271\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2024a\)J\. Wang, Q\. Sun, X\. Li, and M\. GaoBoosting language models reasoning with chain\-of\-knowledge prompting\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 4958–4981\.Cited by:[§1](https://arxiv.org/html/2608.18027#S1.p2.1)\.
- Wanget al\.\(2024b\)P\. Wang, L\. Li, Z\. Shao, R\. Xu, D\. Dai, Y\. Li, D\. Chen, Y\. Wu, and Z\. SuiMath\-shepherd: verify and reinforce llms step\-by\-step without human annotations\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 9426–9439\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2025\)W\. Wang, Z\. Gao, L\. Chen, Z\. Chen, J\. Zhu, X\. Zhao, Y\. Liu, Y\. Cao, S\. Ye, X\. Zhu,et al\.Visualprm: an effective process reward model for multimodal reasoning\.arXiv preprint arXiv:2503\.10291\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2022\)X\. Wang, J\. Wei, D\. Schuurmans, Q\. Le, E\. Chi, S\. Narang, A\. Chowdhery, and D\. ZhouSelf\-consistency improves chain of thought reasoning in language models\.arXiv preprint arXiv:2203\.11171\.Cited by:[§1](https://arxiv.org/html/2608.18027#S1.p2.1)\.
- Wanget al\.\(2024c\)Z\. Wang, Y\. Dong, O\. Delalleau, J\. Zeng, G\. Shen, D\. Egert, J\. Zhang, M\. N\. Sreedhar, and O\. KuchaievHelpsteer 2: open\-source dataset for training top\-performing reward models\.Advances in Neural Information Processing Systems37,pp\. 1474–1501\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2024d\)Z\. Z\. Wang, J\. Mao, D\. Fried, and G\. NeubigAgent workflow memory\.arXiv preprint arXiv:2409\.07429\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px2.p1.1)\.
- Weiet al\.\(2025\)J\. Wei, Z\. Sun, S\. Papay, S\. McKinney, J\. Han, I\. Fulford, H\. W\. Chung, A\. T\. Passos, W\. Fedus, and A\. GlaeseBrowsecomp: a simple yet challenging benchmark for browsing agents\.arXiv preprint arXiv:2504\.12516\.Cited by:[Appendix G](https://arxiv.org/html/2608.18027#A7.p1.1)\.
- Weiet al\.\(2022\)J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou,et al\.Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in neural information processing systems35,pp\. 24824–24837\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px1.p1.1)\.
- Whiteet al\.\(2024\)C\. White, S\. Dooley, M\. Roberts, A\. Pal, B\. Feuer, S\. Jain, R\. Shwartz\-Ziv, N\. Jain, K\. Saifullah, S\. Naidu,et al\.Livebench: a challenging, contamination\-free llm benchmark\.arXiv preprint arXiv:2406\.19314\.Cited by:[4th item](https://arxiv.org/html/2608.18027#A3.I1.i4.p1.1),[§4\.1](https://arxiv.org/html/2608.18027#S4.SS1.p1.1)\.
- Williams \(1992\)R\. J\. WilliamsSimple statistical gradient\-following algorithms for connectionist reinforcement learning\.Machine learning8\(3\),pp\. 229–256\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px2.p1.1)\.
- Xiaet al\.\(2026\)P\. Xia, J\. Chen, X\. Yang, H\. Tu, J\. Liu, K\. Xiong, S\. Han, S\. Qiu, H\. Ji, Y\. Zhou,et al\.MetaClaw: just talk–an agent that meta\-learns and evolves in the wild\.arXiv preprint arXiv:2603\.17187\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px2.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§1](https://arxiv.org/html/2608.18027#S1.p1.1)\.
- Yaoet al\.\(2024\)S\. Yao, N\. Shinn, P\. Razavi, and K\. NarasimhanTau \-bench: a benchmark for tool\-agent\-user interaction in real\-world domains\.arXiv preprint arXiv:2406\.12045\.Cited by:[Appendix L](https://arxiv.org/html/2608.18027#A12.p1.1)\.
- Yaoet al\.\(2023\)S\. Yao, D\. Yu, J\. Zhao, I\. Shafran, T\. Griffiths, Y\. Cao, and K\. NarasimhanTree of thoughts: deliberate problem solving with large language models\.Advances in neural information processing systems36,pp\. 11809–11822\.Cited by:[§1](https://arxiv.org/html/2608.18027#S1.p2.1),[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px1.p1.1)\.
- Yuet al\.\(2025\)Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, L\. Liu,et al\.Dapo: an open\-source llm reinforcement learning system at scale\.arXiv preprint arXiv:2503\.14476\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px2.p1.1)\.
- Yuet al\.\(2023\)W\. Yu, H\. Zhang, X\. Pan, K\. Ma, H\. Wang, and D\. YuChain\-of\-note: enhancing robustness in retrieval\-augmented language models\.arXiv preprint arXiv:2311\.09210\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2024\)D\. Zhang, S\. Zhoubian, Z\. Hu, Y\. Yue, Y\. Dong, and J\. TangRest\-mcts\*: llm self\-training via process reward guided tree search\.Advances in Neural Information Processing Systems37,pp\. 64735–64772\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2025a\)K\. Zhang, X\. Chen, B\. Liu, T\. Xue, Z\. Liao, Z\. Liu, X\. Wang, Y\. Ning, Z\. Chen, X\. Fu,et al\.Agent learning via early experience\.arXiv preprint arXiv:2510\.08558\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2025b\)Q\. Zhang, C\. Hu, S\. Upasani, B\. Ma, F\. Hong, V\. Kamanuru, J\. Rainton, C\. Wu, M\. Ji, H\. Li,et al\.Agentic context engineering: evolving contexts for self\-improving language models\.arXiv preprint arXiv:2510\.04618\.Cited by:[Appendix D](https://arxiv.org/html/2608.18027#A4.p4.1.1),[§1](https://arxiv.org/html/2608.18027#S1.p4.1),[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2608.18027#S4.SS1.p2.1)\.
- Zhaoet al\.\(2024\)A\. Zhao, D\. Huang, Q\. Xu, M\. Lin, Y\. Liu, and G\. HuangExpel: llm agents are experiential learners\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 19632–19642\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px2.p1.1)\.
- Zhenget al\.\(2023a\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.Advances in neural information processing systems36,pp\. 46595–46623\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px1.p1.1)\.
- Zhenget al\.\(2023b\)L\. Zheng, R\. Wang, X\. Wang, and B\. AnSynapse: trajectory\-as\-exemplar prompting with memory for computer control\.arXiv preprint arXiv:2306\.07863\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px2.p1.1)\.
- Zhouet al\.\(2022\)D\. Zhou, N\. Schärli, L\. Hou, J\. Wei, N\. Scales, X\. Wang, D\. Schuurmans, C\. Cui, O\. Bousquet, Q\. Le,et al\.Least\-to\-most prompting enables complex reasoning in large language models\.arXiv preprint arXiv:2205\.10625\.Cited by:[§2](https://arxiv.org/html/2608.18027#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix ADetailed Experimental Setup

Different Reasoning Efforts\.For OpenAI models, we employ the default ‘low’ and ‘high’ in thereasoning​\_​effort\\text\{reasoning\}\\\_\\text\{effort\}parameter to twitch models’ reasoning level\. For Claude 4\.5 Sonnet, we disable the thinking mode and set the thinking budget to 10,000 as the low and high reasoning variants, respectively\.

Decoding Parameters\.For OpenAI models, we use the default decoding parameter with temperature set to 1\.0\. For Gemini, Claude series, we use a temperature of 0\.2 for decoding\.

## Appendix BAveraged Results

We present the averaged best scores over 20 iterations of different methods in Table[3](https://arxiv.org/html/2608.18027#A2.T3)\. From the table, we can clearly draw the conclusion that feedback helps LLMs perform better on all six benchmarks, while existing self\-improving algorithms \(i\.e\., ACE, DC\) do not perform decently on this testing suite\.

MethodAIME 2025LiveCodeBench\(V6\)LiveBench\(Code\)OmniMathGPQADiamondEvaLearnICL71\.83%62\.50%65\.46%53\.12%78\.45%40\.99%ACE71\.98%66\.94%69\.38%50\.33%76\.58%42\.54%DC73\.33%63\.59%68\.58%48\.64%79\.56%42\.68%w/o Feedback77\.78%72\.57%60\.16%65\.17%80\.02%44\.91%Reasoning\-high69\.05%70\.63%55\.46%61\.81%76\.21%39\.58%Reasoning\-low60\.48%61\.03%55\.38%50\.60%72\.92%29\.34%Binary\-Executor–72\.90%71\.65%–––Self82\.22%75\.69%69\.94%67\.52%81\.03%51\.73%Correctness/Executor89\.05%74\.50%75\.78%79\.61%99\.52%57\.05%Table 3:Average performance comparison \(%\) across different LLMs on different datasets\. For baselines, ICL, ACE, DC stands for few\-shot in\-context learning, agentic context engineering, and dynamic cheatsheet, respectively\.
## Appendix CEvaluated Benchmarks

We present detailed descriptions of our evaluated benchmarks below:

- •AIME 2025\[[4](https://arxiv.org/html/2608.18027#bib.bib27)\]is a challenging and universal math benchmark consists of 30 cases from AIME in 2025\.
- •OmniMath\[[15](https://arxiv.org/html/2608.18027#bib.bib28)\]is a universal olympiad level mathematic benchmark consists of a total of 4,428 problems, and we sample 200 examples across all difficulties for evaluation\.
- •LiveCodebench \(V6\)\[[23](https://arxiv.org/html/2608.18027#bib.bib29)\]is an evolving benchmark for challenging code generation\. We select its latest version \(V6\) alone with a total of 175 samples\.
- •LiveBench \(Code\)\[[64](https://arxiv.org/html/2608.18027#bib.bib30)\]is originated from a living benchmark spans across six aspects\. We select the coding aspect, comprising 128 samples for evaluation\.
- •EvaLearn\[[14](https://arxiv.org/html/2608.18027#bib.bib31)\]is the first benchmark that evaluate the experience learning abilities of language models, which includes total 648 examples\.
- •GPQA Diamond\[[42](https://arxiv.org/html/2608.18027#bib.bib32)\]is a challenging multiple\-choice question set in biology, chemistry, and physics, authored by PhD\-level experts\. It consists of 198 examples in total\.

## Appendix DBaselines

Built\-in Reasoning: A native test\-time scaling mechanism implemented in OpenAI models allows the reasoning effort to be adjusted between low and high\. We treat this as a built\-in and straightforward scaling baseline and present further details in Appendix[A](https://arxiv.org/html/2608.18027#A1)

Few\-shot In\-Context Learning \(ICL\)\.In few\-shot ICL, each demonstration is a previously solved question–answer pair\. As the model processes tasks sequentially, all past pairs are stored in an experience buffer\. For a new question, we retrieve thekkmost similar past questions using embeddings from OpenAI’stext\-embedding\-3\-largemodel\[[35](https://arxiv.org/html/2608.18027#bib.bib73)\], and include their original question–answer formats as demonstrations, followed by the new question\. When fewer thankkexamples are available, all prior examples are used\. We evaluate ICL withk∈\[1,5,8,12,15,20\]k\\in\[1,5,8,12,15,20\]\.

Dynamic CheatSheet \(DC\)\[[51](https://arxiv.org/html/2608.18027#bib.bib15)\]\.Dynamic CheatSheet \(DC\) is a test\-time learning method that maintains an adaptive external memory of reusable strategies or code snippets distilled from prior solutions\. As new problems are solved, DC summarizes high\-level strategies from selected past tasks—using ground\-truth answers for clean experience curation—and stores them as structured cheatsheets\. Past tasks are retrieved using the same similarity\-based retrieval as few\-shot ICL; when fewer thankktasks exist, all available examples are used\. Consistent with ICL, cheatsheets are synthesized from the most recentk∈\[1,5,8,12,15,20\]k\\in\[1,5,8,12,15,20\]relevant solutions\.

Agentic Context Engineering \(ACE\)\[[74](https://arxiv.org/html/2608.18027#bib.bib21)\]\.Agentic Context Engineering \(ACE\) is a context adaptation framework that treats prompts as evolving playbooks rather than static demonstrations\. It incrementally distills reusable strategies and domain insights through a generate–reflect–curate process, producing localized context updates that preserve prior knowledge and avoid monolithic rewrites\. Similar to few\-shot ICL, ACE retrieves relevant past trajectories and integrates insights from the most recentk∈\[1,5,8,12,15,20\]k\\in\[1,5,8,12,15,20\]trajectories into its evolving playbook\.

## Appendix EBenchmark Input and Answer Examples

We provide representative input and answer examples for three benchmarks below\.

AIME 2025\.

Solve the following AIME problem step by step and provide the final answer: 2x\+2−x=52^\{x\}\+2^\{\-x\}=5\. Compute22​x\+2−2​x2^\{2x\}\+2^\{\-2x\}\. Answer format: ANSWER: <integer\> Answer: 21

GPQA Diamond\.

Which of the following is \*not\* a property of an ideal gas under standard conditions? A\) Molecules occupy negligible volume B\) Collisions are perfectly elastic C\) Internal energy depends only on temperature D\) Pressure is independent of volume Answer: D

LiveCodeBench \(V6\)\.

Question: "Erase Leaves \-\-\- Given a tree with N vertices, repeatedly remove leaf vertices\. Find the minimum operations to delete vertex 1\." Public: \[\{"input": "9\\n1 2\\n2 3\\n\.\.\.", "output": "5\\n"\}, \.\.\.\] Private: \[\{"input": "2\\n1 2\\n", "output": "1\\n"\}, \.\.\.\]

We use exact match for AIME 2025 and GPQA\-Diamond, LLM\-as\-a\-Judge for OmniMath\[[15](https://arxiv.org/html/2608.18027#bib.bib28)\], and a Python interpreter for coding tasks \(LiveBench\-Code and LiveCodeBench\)\.

## Appendix FFull Discussions

In this section, we present the full version of different discussions in

### F\.1LLM with All Spurious “Correct” Feedback

In Figure[7](https://arxiv.org/html/2608.18027#A6.F7), we further present model performance under two extreme conditions: receiving uniformly “correct” feedback \(e\.g\., the answer is correct\) and the SelMV\-augmented results\. We observe that although performance initially drops after exposure to such incorrect feedback, which suggests temporary confusion in adapting to inconsistent supervision\. The models quickly recover and even improve as they adapt to the underlying pattern\. Interestingly, both GPT\-5 mini and o4\-mini exhibit larger gains when exposed to entirely incorrect feedback, as such feedback compels the models to re\-evaluate their reasoning and verify their outputs\. In contrast, consistently positive feedback tends to induce overconfidence, misleading the models into accepting their initial responses without critical reassessment\. This observation suggests that, paradoxically, constructive noise \(in the form of seemingly negative feedback\) can sometimes stimulate deeper reasoning and enhance robustness in iterative test\-time learning\.

![Refer to caption](https://arxiv.org/html/2608.18027v1/selmv_full.png)Figure 7:Model performance using constant “incorrect” and “correct” feedback\. By leveraging the selective majority voting, LLMs show decent performance when facing spurious feedback on math tasks\.
### F\.2Extended Rounds of Iterations

![Refer to caption](https://arxiv.org/html/2608.18027v1/50_iters_v3.png)Figure 8:Performance of GPT\-5 mini, o4\-mini, and o3\-mini with extended iterations of experience to 50 on math tasks\. We mark performance points within the first 20 iterations of experience in blue, and those from iterations 20 to 50 in green\.To further probe the learning capacity of LLMs, we extend the number of experience iterations from 20 to 50, as shown in Figure[8](https://arxiv.org/html/2608.18027#A6.F8)\. Across both AIME 25 and OmniMath, we observe that most performance gains occur within the first 20 iterations, while later stages yield only marginal improvements \(e\.g\., average 16\.7%\>\>2\.2% on AIME 25; 21\.2%\>\>3\.5% on OmniMath\)\. This trend consistently holds across different models, suggesting that LLMs quickly internalize and consolidate the useful feedback signals in the early stages, after which learning saturates\. These results highlight that the majority of test\-time learning under CoE happens rapidly — shorter adaptation loops in our CoE are sufficient for most reasoning tasks, with less significant returns from prolonged experience accumulation\.

### F\.3Feedback Strength

![Refer to caption](https://arxiv.org/html/2608.18027v1/verifier_ablation_v2.png)Figure 9:Model performance of GPT\-5 and GPT\-5 mini with external model feedback on two math tasks\.For experience with model feedback, one intuitive exploration is to design CoE with different feedback providers\. Specifically, we cluster GPT\-5 and GPT\-5 mini as a pair and allow each to serve as a feedback generator for the other on a mathematical and a coding task, as illustrated in Figure[9](https://arxiv.org/html/2608.18027#A6.F9)\. Interestingly, external model feedback proves to be highly effective on these two benchmarks, where it even slightly surpasses correctness\-based/executor feedback\. For instance, GPT\-5 mini achieves a peak accuracy of 94\.4% with model feedback, compared to 93\.0% with correctness feedback; similarly, GPT\-5 reaches 93\.3% vs\. 92\.2%, showing that high\-quality model\-generated judging signals can substitute explicit correctness supervision\. On the more challenging OmniMath benchmark, GPT\-5 continues to help its mini variant outperform its self\-feedback baseline\. However, because GPT\-5 mini starts with only 62\.5% zero\-shot accuracy, it is unable to provide sufficiently reliable feedback to improve GPT\-5, resulting in inferior performance relative to correctness feedback\. Moreover, both GPT\-5 and GPT\-5 mini underperform their AIME 2025 results \(≥\\geq60% vs\.≥\\geq80%\), and correctness feedback remains clearly superior: GPT\-5 gets 82\.8% with correctness feedback but only 74\.5% with external model feedback\. Overall, these findings highlight a consistent trend: feedback quality correlates with the verifier’s base ability on the task, and model\-generated feedback becomes competitive with correctness supervision only when the verifier is sufficiently strong, suggesting a practical threshold for deploying model\-as\-judge in iterative experience frameworks\.

### F\.4Analysis of Improvement Patterns

We present the detailed criteria that we used for classifying the

- •Feedback Fidelity: Improvements directly driven by external feedback, where the model explicitly incorporates provided guidance or corrections into its next response\.
- •Self Reflection: Improvements arising from the model’s own reasoning, identifying and correcting errors with little reliance on external feedback\.
- •Specification Recall: Adjustments motivated by task instructions or formatting requirements, as the model re\-aligns with the original question or output schema\.
- •Random: Changes with no identifiable cause, typically minor rewording or stylistic variations unrelated to feedback or specification\.

### F\.5Human–GPT Judge Agreement

To validate the GPT\-5\-based automatic improvement analysis, we randomly sample 100 incorrect\-to\-correct trajectory pairs \(25 per category: Feedback Fidelity, Self Reflection, Specification Recall, Random\) from the full set of 6,630 examples\. Two human annotators independently classify each pair into the four categories using the same criteria as the GPT\-5 judge \(Appendix[K](https://arxiv.org/html/2608.18027#A11)\)\. We then compute Cohen’s Kappa between the GPT\-5 labels and the majority human label\.

CategoryAgreement \(%\)Cohen’sκ\\kappaFeedback Fidelity84\.00\.81Self Reflection72\.00\.71Specification Recall80\.00\.78Random68\.00\.63Overall76\.00\.768Table 4:Cohen’s Kappa agreement between GPT\-5 judge and human annotators across four improvement categories on 100 sampled trajectories\. The overallκ\\kappaof 0\.768 indicates substantial agreement\[landis1977measurement\]\.The overallκ\\kappaof 0\.768 falls in the “substantial agreement” range, confirming that GPT\-5 is a reliable proxy for human attribution in this task\. Disagreements concentrate in the Random and Self Reflection categories, where the distinction between stochastic drift and genuine self\-correction can be ambiguous even for human raters\.

## Appendix GBrowseComp\-Plus

We also report model performance on BrowseComp\-Plus[11](https://arxiv.org/html/2608.18027#bib.bib33)\. It is a benchmark to evaluate deep research systems, isolating the effect of the retriever with a local database\. It is sourced from the BrowseComp[62](https://arxiv.org/html/2608.18027#bib.bib34)and we sample 200 examples to accelerate the evaluation\.

In Figure[10](https://arxiv.org/html/2608.18027#A7.F10), we observe that unlike coding and math tasks, BrowseComp\-Plus requires knowledge beyond the scope of the models’ training data\. Consequently, for most models, incorporatingself feedbackleads to a performance decline compared to the no\-feedback setting, highlighting the limitation of relying solely on self feedback in out\-of\-distribution knowledge scenarios\.

![Refer to caption](https://arxiv.org/html/2608.18027v1/browsecomp_plus_main.png)Figure 10:On BrowseComp\-Plus, self\-feedback models fall behind as the task requires external search\-based knowledge\.
## Appendix HFull Results of Performance and Efficiency

We present full results of model performance \(Figure[3](https://arxiv.org/html/2608.18027#S4.F3)\) and API costs \(Figure[12](https://arxiv.org/html/2608.18027#A8.F12)\) regarding eight reasoning LLMs over six benchmarks\.

![Refer to caption](https://arxiv.org/html/2608.18027v1/main_results_full_v3.png)Figure 11:Full results of eight state\-of\-the\-art LLMs on six benchmarks incorporating different generation techniques\. Model accuracies with different levels of feedback \(correctness/executor,self, andbinary executorfeedback\) generally perform better than the baseline strategies\.![Refer to caption](https://arxiv.org/html/2608.18027v1/price_full_v2.png)Figure 12:Full results of total API cost \(in dollar\) vs\. best model performance over 20 iterations\. LLMs with detailed feedback \(i\.e\.,self feedback\) achieves decent results with fewer costs, while CoE without feedback generally falls behind \(at lower right\)\.
## Appendix IToken Analysis

We report aggregated token counts alongside accuracy for representative methods in Table[5](https://arxiv.org/html/2608.18027#A9.T5)\. CoE with feedback achieves higher accuracy at comparable token budgets to other multi\-round methods, demonstrating that feedback\-driven iterations yield higher return per token rather than simply inflating prompt length\.

DatasetMethodTokensAcc \(%\)AIME 2025CEF108,73484\.6SF108,23183\.8NF106,82574\.1DC11,23374\.7OmniMathCEF176,41274\.2SF175,80672\.1NF173,94466\.8DC16,90463\.9LiveCodeBench\(V6\)CEF224,11872\.6SF223,44171\.2NF221,55068\.0DC20,77166\.4Table 5:Token complexity vs\. accuracy across methods\. CEF: Correctness/Executor Feedback; SF: Self Feedback; NF: No Feedback; DC: Dynamic CheatSheet\. Token counts are aggregated across all iterations\.
## Appendix JDual Feedback and Principled Selection

We present full results of dual feedback CoE and principled experience selection experiments using Claude 4\.5 Sonnet\. In the dual feedback setting, model feedback is combined with correctness feedback \(math tasks\) or executor feedback \(coding tasks\) within each iteration\. For principled selection, we integrate Dynamic CheatSheet \(DC\)\[[51](https://arxiv.org/html/2608.18027#bib.bib15)\]and SimpleMem\[[32](https://arxiv.org/html/2608.18027#bib.bib75)\]within the same task \(no cross\-task retrieval\) to test whether memory\-based compression adds benefit beyond full experience trails\.

Dual feedback demonstrates clear complementarity on AIME 2025 and LiveBench \(Code\), where combining two feedback channels surpasses either channel alone\. On the more challenging OmniMath, correctness feedback alone matches dual feedback, suggesting that when the primary signal is already strong, the additional model feedback provides marginal benefit\. Memory\-based selection methods \(DC, SimpleMem\) consistently underperform pure model feedback, indicating that aggressive summarization or retrieval may discard critical intermediate reasoning steps that full experience trails preserve\.

## Appendix KPrompt for Improvement Pattern Analysis

DUAL\-AXIS LIFT/CHANGE ATTRIBUTION JUDGEYou are a \*\*dual\-axis attribution judge\*\*\. Your job is to identify both \*\*why\*\* the model changed and \*\*what\*\* specifically changed between two consecutive attempts on the same problem\. Do \*\*not\*\* decide if the solution is correct overall — correctness labels are provided\. Instead, attribute the observed change along two orthogonal dimensions:\* \*\*Change Driver \(WHY\)\*\* – the motivation or trigger behind the change\. \* \*\*Change Manifestation \(WHAT\)\*\* – the concrete locus or type of modification made\.—I\. Change Driver \(WHY the change occurred\)These categories capture \*the source or motivation\* of the update in the second attempt\.1\. \*\*Feedback Fidelity\*\* – The model directly \*uses\* the provided feedback to modify its output\. \*Signals:\* Edits match failing test or critique location; added clause mirrors feedback\.2\. \*\*Self\-Reflection / Internal Reasoning\*\* – The model self\-identifies an error or improvement without \(using\) explicit feedback or tests a different approach to see if it performs better\. \*Signals:\* “I realized…,” “previously I miscalculated…,” or internally consistent reformulation not prompted by feedback or when there’s no feedback\.3\. \*\*Specification Recall / Compliance Awareness\*\* – The model remembers or re\-aligns with task instructions or formatting requirements\. \*Signals:\* Adds “FINAL ANSWER:” wrapper, adheres to requested schema, restores omitted step explicitly mentioned in prompt\.4\. \*\*Random / Drift / Unknown Driver\*\* – The motivation cannot be inferred; the change appears stochastic or stylistic\. \*Signals:\* Superficial rewording, minor ordering changes, no logical link to feedback or instruction\.—II\. Change Manifestation \(WHAT changed\)These categories describe \*the technical or structural form\* of the change between attempts\.A\. \*\*Structural Plan / Algorithm Revision\*\* – A new high\-level approach or reformulation \(e\.g\., brute force → DP, heuristic → formula\)\. \*Signals:\* Rewritten main structure, new helper functions, change in complexity or data representation\.\*B\. \*\*Local Step Soundness & Invariant Fix\*\* – Correction of a local logic, variable, arithmetic, or invariant violation while keeping the overall plan\. \*Signals:\* Fixed off\-by\-one in loop, corrected variable update, repaired algebraic derivation\.\*C\. \*\*Edge / Boundary Condition Handling\*\* – Added or fixed guard for extreme/special cases \(e\.g\., empty, zero, overflow, tie\)\. \*Signals:\* ‘if n == 0‘, ‘<=‘ ‘<‘, added epsilon, handled ‘len == 1‘\.\*D\. \*\*Output / Format Compliance\*\* – Adjusted presentation or output schema without changing algorithmic content\. \*Signals:\* Added “FINAL ANSWER:,” fixed JSON/CSV layout, printing only required token\.\*E\. \*\*Other / Ambiguous Change\*\* – Cannot clearly assign to A–E or insufficient evidence\. \*Signals:\* Stylistic edits, reordering, or unrelated cleanup\.\*— … …VII\. Constraints\* Do \*\*not\*\* recompute correctness or logic\. \* Focus on identifying the \*\*causal link \(WHY\)\*\* and the \*\*technical locus \(WHAT\)\*\* of the change\. \* Be concise: cite only minimal evidence sufficient to justify your judgment\. \* If code is long, highlight the 1–2 lines most diagnostic of change\. \* If feedback is visible but not followed, set ‘"visible": true, "operationalized": false‘\. \* Always describe the \*learning pattern\* observed in Attempt\_t\+1\.

## Appendix LLimitations

Our evaluation focuses primarily on math, knowledge, and coding benchmarks\. While these domains offer controlled settings to probe iterative improvement, there are interaction\-intensive scenarios where experience unfolds over long horizons[24](https://arxiv.org/html/2608.18027#bib.bib41),[68](https://arxiv.org/html/2608.18027#bib.bib42), to which the CoE paradigm should naturally extend\. In addition, we do not update model parameters in this study\. This choice isolates the Chain\-of\-Experience mechanism as a test\-time paradigm, but it also means that the observed improvements arise from contextual reuse of experience rather than true learning; incorporating weight updates to internalize experience remains an important next step toward training models with persistent the “learning\-from\-experience” abilities\.

## Appendix MDeclaration of AI Tool Usage

During the preparation of this manuscript, we used OpenAI’s GPT\-5 model for minor language refinement and smoothing of the writing\. The AI tool was not used for generating original content, conducting data analysis, or formulating core scientific ideas\. All conceptual development, experimentation, and interpretation were conducted independently without reliance on AI tools\.