PAPER2LLM++：基于研究论文的 LLM 持续自我进化

# Paper2LLM++: Continual Self-Evolution of LLMs from Research Papers
Source: [https://arxiv.org/html/2610.02793](https://arxiv.org/html/2610.02793)
Yilun ZhaoYale UniversityNew Haven, CT 06520, USAEmail:[yilun\.zhao@yale\.edu](mailto:)Wenpeng YinThe Pennsylvania State UniversityUniversity Park, PA 16802, USAEmail:[wenpeng@psu\.edu](mailto:)

###### Abstract

Research on LLMs continually uncovers model limitations, their causes, and potential solutions\. Yet these human discoveries remain largely disconnected from model evolution: an LLM does not automatically learn from new research about its own failures\. We introducePaper2LLM\+\+, a framework for*continual self\-evolution of LLMs from research papers*\. Rather than treating papers merely as knowledge to retrieve,Paper2LLM\+\+uses the growing literature as a stream of evidence and supervision for model improvement\. For each incoming paper, it extracts evidence\-grounded findings, tests whether the reported limitation persists in the current model, and, when needed, converts the findings into candidate learning signals\. Atry–evaluate–commitprocedure integrates an update only when it improves the targeted behavior without substantially forgetting prior improvements or degrading general capabilities\. Across a sequential stream of research\-discovered LLM failures, we show that models can progressively incorporate new findings while retaining earlier gains\.Paper2LLM\+\+thus takes a step toward closing the loop between*human discovery*and*model evolution*, enabling models to continually learn from research about their own limitations and improvements\.111Data & Code will be released upon publication\.

\+\+footnotetext:Corresponding author\.## 1Introduction

A long\-standing goal of AI is to build models that*continue improving after deployment*: recognizing what they still need to learn, acquiring new capabilities, and retaining what they have already learned\. Existing approaches enable models to generate their own training data, refine outputs, or adapt from new inputs\([Wang et al\., 2023](https://arxiv.org/html/2610.02793#bib.bib1);[Madaan et al\., 2023](https://arxiv.org/html/2610.02793#bib.bib2);[Zweiger et al\., 2025](https://arxiv.org/html/2610.02793#bib.bib3)\), but a fundamental question remains:where should the supervision for continual self\-evolution come from?We argue that scientific papers provide a natural and continually growing source\. Research on LLMs constantly discovers new failure modes, identifies when and why they occur, and develops methods for evaluating or mitigating them\. Such findings therefore provide more than knowledge to retrieve: they offer evidence about*what a model should learn next*\. The challenge is to turn this evidence into model\-specific, persistent improvements\.

We introducePaper2LLM\+\+, a framework forcontinual self\-evolution of LLMs from research papers\. For each incoming paper,Paper2LLM\+\+extracts evidence\-grounded findings about a reported limitation and first verifies whether the limitation still exists in the current model\. If so, it translates the finding into complementary forms of synthetic supervision \(*i\.e\.,*direct, contrastive, robustness, and process supervision\) without training on the paper’s evaluation benchmark\. Because neither every paper nor every generated supervision signal is necessarily useful,Paper2LLM\+\+employs a*try–evaluate–commit*procedure that accepts an update only when it improves the targeted behavior while preserving previously acquired improvements and general capabilities\. This creates a continual loop from*human discovery*to*model verification, learning, and evolution*\.

We evaluatePaper2LLM\+\+on 30 failure\-revealing research papers spanning seven failure families, using Llama\-3\.1\-8B\-Instruct\([Grattafiori et al\., 2024](https://arxiv.org/html/2610.02793#bib.bib4)\)and Qwen2\.5\-7B\-Instruct\([Yang et al\., 2024](https://arxiv.org/html/2610.02793#bib.bib5)\)\. In individual\-paper experiments, 24 of 30 papers produce positive held\-out gains\. Across continual paper streams,Paper2LLM\+\+consistently improves the targeted behaviors while outperforming prior self\-evolving and continual learning approaches in preserving previously acquired improvements and general capabilities\. Beyond aggregate performance, our analysis reveals three conditions that determine whether research findings become useful learning signals: they must beteachable, meaning they can be translated into concrete and verifiable supervision;needed, meaning the reported limitation still exists in the current model; andretained, meaning the resulting improvement survives subsequent updates\. These findings suggest that effective self\-evolution requires not simply learning from more research, but selectively determining*what is worth learning, how to learn it, and whether it remains learned*\.

Overall, our contributions are threefold:

- •We are the first to formulatepaper\-driven LLM self\-evolution, treating scientific discoveries about model limitations as learning supervision for continual model improvement\.
- •We proposePaper2LLM\+\+, closing the loop from discovery to improvement, enabling evidence extraction, failure verification, supervision construction, and continual adaptation\.
- •Across 30 papers, seven failure families, and two LLMs, we demonstrate continual improvement and identifyteachability, need, and retentionas key conditions governing when scientific findings can drive effective model self\-evolution\.

Figure 1:Paper2LLM\+\+overview\. \(1\) Paper evidence: extract grounded findings and verify the reported gap\. \(2\) Data generation: translate confirmed gaps into complementary synthetic supervision\. \(3\) Controlled adaptation: select LoRA updates based on target gains, retention, and general capabilities\.
## 2Related Work

#### Continual and self\-evolving LLMs\.

Recent work enables LLMs to continually improve from different forms of incoming experience\. SEAL\([Zweiger et al\., 2025](https://arxiv.org/html/2610.02793#bib.bib3)\)learns to convert new passages or task demonstrations into synthetic supervision for parameter updates, but provides limited protection against forgetting\. Recurrent\-KIF\([Feng et al\., 2025](https://arxiv.org/html/2610.02793#bib.bib6)\)and RLEdit\([Li et al\., 2025b](https://arxiv.org/html/2610.02793#bib.bib7)\)instead assume that supervision is already available: the former continually learns from labeled task datasets using replay and importance\-based parameter fusion, while the latter applies a sequence of explicit knowledge edits through RL\-trained parameter updates with memory and locality constraints\. Other approaches evolve the system without modifying the underlying LLM parameters\. Gödel Agent\([Yin et al\., 2025](https://arxiv.org/html/2610.02793#bib.bib8)\)and Darwin Gödel Machine\([Zhang et al\., 2026a](https://arxiv.org/html/2610.02793#bib.bib9)\)use environmental or benchmark feedback to iteratively improve agent code, tools, and workflows, preserving useful solutions through previous implementations or an agent archive\. ACE\([Zhang et al\., 2026b](https://arxiv.org/html/2610.02793#bib.bib10)\)and LSE\([Chen et al\., 2026a](https://arxiv.org/html/2610.02793#bib.bib11)\)learn from sequential task experience and feedback by evolving external context, with ACE incrementally maintaining a reusable playbook and LSE retaining and branching previous contexts through an evolution tree\. These approaches therefore primarily study how to continually learn from*given data, tasks, edits, or interaction feedback*, and how to retain the resulting improvements\. In contrast, ourPaper2LLM\+\+studies a distinct source of continual learning: a stream of*scientific papers*\. This introduces an upstream challenge absent from conventional continual learning: a paper does not directly specify training examples or rewards, but reports findings about model behavior\.Paper2LLM\+\+must therefore determine*what*the paper teaches, whether the reported limitation is still present in the current model, translate applicable findings into supervision, and only then incorporate and retain the resulting improvement\.

#### Research papers as external knowledge signals\.

Recent benchmarks treat papers as sources of new knowledge, but primarily specify or evaluate update targets rather than provide an executable procedure for model improvement\. ScienceMeter\([Wang et al\., 2026](https://arxiv.org/html/2610.02793#bib.bib12)\)derives scientific claim evaluations from curated papers to measure acquisition, retention, and generalization\. DocMEdit\([Zeng et al\., 2025](https://arxiv.org/html/2610.02793#bib.bib13)\)supplies paired documents containing predefined factual changes, while WikiBigEdit\([Thede et al\., 2025](https://arxiv.org/html/2610.02793#bib.bib14)\)extracts update targets from Wikidata snapshot differences; both leave the actual update procedure to the evaluated method\. Our setting instead asks whether new research papers in stream can be converted into executable supervision, verified against an observed model failure, and incorporated without erasing previously acquired gains\.

## 3Problem Formulation

We study how an LLM can*continually improve from newly published research about LLM failures*\. Consider a modelMt−1M\_\{t\-1\}that receives a stream of papersPt,Pt\+1,…,P\_\{t\},P\_\{t\+1\},\\ldots,where each paper reports an LLM limitation, analyzes when or why it occurs, and may suggest mitigation strategies\.

Our goal is to use each paper as a source of model\-maintenance supervision\. A paper, however, is not itself an executable update\. The reported failure may already be absent fromMt−1M\_\{t\-1\}; the paper usually does not provide a training set tailored toMt−1M\_\{t\-1\}; and an update that addresses the new failure may interfere with previously improved or general capabilities\. This gives rise to three challenges thatPaper2LLM\+\+addresses:

- •𝒞1\\mathcal\{C\}\_\{1\}: Is the reported failure relevant to the current model?A paper reports failures of the models it studied, butMt−1M\_\{t\-1\}may behave differently\. We must first assess whether, and to what extent, the paper\-defined failure manifests inMt−1M\_\{t\-1\}\.
- •𝒞2\\mathcal\{C\}\_\{2\}: How can a failure\-discovering paper be turned into executable supervision?If the failure is present, we must identify what it teaches, such as the failed behavior, its trigger, and any proposed explanation or mitigation, and compile this information into synthetic supervision𝒟^t\\widehat\{\\mathcal\{D\}\}\_\{t\}without training on the evaluation examples\.
- •𝒞3\\mathcal\{C\}\_\{3\}: How can continual self\-evolution preserve previously learned capabilities and general capabilities?As papers arrive sequentially, learning from new evidence may overwrite capabilities acquired from earlier papers or degrade the model’s general capabilities\. Each update must integrate new supervision while preserving prior gains and general model performance\.

## 4Method:Paper2LLM\+\+

We describePaper2LLM\+\+\(illustrated in[Figure1](https://arxiv.org/html/2610.02793#S1.F1)\) to address the three challenges:𝒞1\\mathcal\{C\}\_\{1\}by extracting supervision and confirming the reported issue;𝒞2\\mathcal\{C\}\_\{2\}by generating synthetic supervision data; and𝒞3\\mathcal\{C\}\_\{3\}by updating the LLM while maintaining previously learned and general capabilities\.

### 4\.1Extracting Supervision and Verifying Reported Failures

#### Extract evidence from the paper\.

Given paperPtP\_\{t\}, we use gpt\-oss\-120b\([OpenAI, 2025](https://arxiv.org/html/2610.02793#bib.bib15)\)222Temp\.00, top\-p=1\.0p=1\.0, 16,384/2,048 input/output tokens\.to extract a structured record

ℛt=\(CLOSE\\displaystyle\\mathcal\{R\}\_\{t\}=\(What,Trigger,Why,\\displaystyle\\textsc\{What\},\\textsc\{Trigger\},\\textsc\{Why\},OPENMitigation,Evaluation\)\.\\displaystyle\\textsc\{Mitigation\},\\textsc\{Evaluation\}\)\.whereWhat: the behavior or capability reported to fail, such as “Models that correctly solve GSM8K seed questions often fail on minimally perturbed versions of the same problems, producing an accuracy gap of up to 20%\.”[Li et al\. \(2024a\)](https://arxiv.org/html/2610.02793#bib.bib16)\(we use this paper as a running example below\);Trigger: the condition under which the failure appears, such as “adding operation”, “reversing operations” and “distractor insertion”\.Why: the paper’s own explanation of the failure, when explicitly stated\. For example “often overlook steps or make errors in the middle stages of the solution”;Mitigation: a proposed solution, such as “compositional prompting can help\.”Evaluation: the datasetDtD\_\{t\}, and evaluation protocol used to assess the failure\.

Each field consists of one or more supporting text spans extracted directly from the paper\. If a paper does not provide a field \(especiallyWhyorMitigation\), we storeN/Arather than infer one\. Examples above are truncated for brevity; a complete evidence\-grounded example is provided in[AppendixA\.5](https://arxiv.org/html/2610.02793#A1.SS5)\.

#### Construct the detection and held\-out partitions once\.

The benchmarkDtD\_\{t\}\(further split intoDtdevD\_\{t\}^\{\\mathrm\{dev\}\}andDttestD\_\{t\}^\{\\mathrm\{test\}\}by 30/70\) associated with the paperPtP\_\{t\}is used for two different purposes:DtdevD\_\{t\}^\{\\mathrm\{dev\}\}is the only partition used to decide whether the current LLM checkpoint exhibits the paper\-defined failure, whereasDttestD\_\{t\}^\{\\mathrm\{test\}\}is reserved for final assessment\. Thus:Dt=Dtdev∪Dttest\.D\_\{t\}=D\_\{t\}^\{\\mathrm\{dev\}\}\\cup D\_\{t\}^\{\\mathrm\{test\}\}\.

#### Verify whether the paper\-reported failure exists in the current LLM\.

This step is different from simply measuring how often the current LLMMt−1M\_\{t\-1\}makes mistakes: the goal is to re\-assess the*behavioral effect*reported by the paper using its original evaluation protocol\. Concretely, we runMt−1M\_\{t\-1\}on the detection splitDtdevD\_\{t\}^\{\\mathrm\{dev\}\}and reproduce the comparison used byPtP\_\{t\}\. For papers that report degradation between two matched conditions, each detection instance consists of a pair\(xi,x~i\)\(x\_\{i\},\\tilde\{x\}\_\{i\}\), wherexix\_\{i\}is the original instance andx~i\\tilde\{x\}\_\{i\}is the corresponding instance containing the failure\-triggering change identified by the paper\. We measure

Δt=Score⁡\(Mt−1,\{xi\}\)−Score⁡\(Mt−1,\{x~i\}\),\\Delta\_\{t\}=\\operatorname\{Score\}\(M\_\{t\-1\},\\\{x\_\{i\}\\\}\)\-\\operatorname\{Score\}\(M\_\{t\-1\},\\\{\\tilde\{x\}\_\{i\}\\\}\),using the evaluator specified byPtP\_\{t\}\. A positiveΔt\\Delta\_\{t\}therefore means that the paper\-defined trigger degrades the current model’s behavior\.

For each paper, we follow its original evaluation protocol to reproduce the reported failure; matched pairs are used only when they are part of that protocol\. For example,[Li et al\. \(2024a\)](https://arxiv.org/html/2610.02793#bib.bib16)pair original GSM8K problems with controlled variations, allowing us to test whether the variations cause a performance drop\. We bootstrap the evaluation units to estimate the uncertainty ofΔt\\Delta\_\{t\}and obtain a 95% confidence interval\. We confirm the failure only when this interval is entirely above zero, indicating a performance drop under the paper\-defined failure condition\. Otherwise, we performNo Update; confirmed failures proceed to supervision\.

### 4\.2Generating Paper\-Inspired Learning Data

We do not train the LLM on the benchmark examples inDtD\_\{t\}\. Instead, Qwen2\.5\-32B\-Instruct\([Yang et al\., 2024](https://arxiv.org/html/2610.02793#bib.bib5)\), as the data generator, receives onlyℛt\\mathcal\{R\}\_\{t\}, the structured evidence extracted from the paper, and generates new examples that instantiate the same behavioral gap\. Examples fromDtD\_\{t\}are never provided to the generator\.

The same research finding can be translated into different learning signals, and it is not clear a priori which form is most effective\. We therefore generate up to four candidate supervision datasets\. We illustrate each using GSM\-Plus\([Li et al\., 2024a](https://arxiv.org/html/2610.02793#bib.bib16)\), which reports that LLMs can overlook a newly added condition in a math problem\.

•Direct supervisionteaches the desired behavior directly\. For example, we generate the problem “6 boxes contain 12 pencils each, and 3 pencils are removed from each box,” whose correct answer is5454\. This teaches the model to incorporate the added condition rather than overlook it\.

•Contrastive supervisionteaches which changes should change the answer\. For example, we pair “6 boxes contain 12 pencils each” \(7272\) with “6 boxes contain 12 pencils each, but 3 pencils are removed from each box” \(5454\)\. The two problems differ only in the added condition, making its effect on the answer explicit\.

•Robustness supervisionteaches which variation should*not*change the answer\. For example, we pair “3 pencils are removed from each box” with its paraphrase “3 pencils are taken out of every box\.” As the quantities and logic are unchanged, both should retain the same answer \(5454\)\.

•Process supervisionteaches the intermediate reasoning implicated by the paper\. For the same example, the generated supervision explicitly applies the added condition: “*6×\\times12=72→\\rightarrow72\-6×\\times3=54*”\.

Not every paper supports all four forms of supervision\. For example, process supervision is generated only when the paper identifies a concrete reasoning or verification error\. We generate all applicable candidates and determine which form improves the current model most\.

Table 1:Evaluation suite of 30 failure\-revealing papers and benchmarks for paper\-driven LLM self\-evolution\.
### 4\.3Continual Self\-Evolution

To decide which supervision should be used to evolve the current model, we use a*try–evaluate–commit*procedure: each candidate is first tested through a small temporary update, and*only a candidate that improves the new behavior without substantially degrading previously acquired or general capabilities is allowed to update the model*\.

#### Try each candidate with a temporary update\.

For each candidate supervision dataset𝒟t\(k\)\\mathcal\{D\}\_\{t\}^\{\(k\)\}, we start from the same current modelMt−1M\_\{t\-1\}and perform a lightweight LoRA update, producing a temporary modelMt\(k\)M\_\{t\}^\{\(k\)\}\. These updates are used only for candidate selection and discarded after scoring\.

#### Evaluate candidate supervision and update the model\.

Each temporary modelMt\(k\)M\_\{t\}^\{\(k\)\}is evaluated along three dimensions: i\)New\-behavior gain \(Gt\(k\)G\_\{t\}^\{\(k\)\}\):improvement on the behavioral gap identified by the current paper; ii\)Previous\-behavior degradation \(Pt\(k\)P\_\{t\}^\{\(k\)\}\):performance loss on behaviors acquired from earlier papers; and iii\)General\-capability degradation \(Qt\(k\)Q\_\{t\}^\{\(k\)\}\):performance loss on general LLM capabilities\. We therefore combine these signals as “utility \(UU\)”

Ut\(k\)=Gt\(k\)−λ​Pt\(k\)−μ​Qt\(k\)\.U\_\{t\}^\{\(k\)\}=G\_\{t\}^\{\(k\)\}\-\\lambda P\_\{t\}^\{\(k\)\}\-\\mu Q\_\{t\}^\{\(k\)\}\.To prevent a candidate with high target gain but substantial regression from being selected, we additionally requireGt\(k\)≥τgainG\_\{t\}^\{\(k\)\}\\geq\\tau\_\{\\mathrm\{gain\}\},Pt\(k\)≤τprevP\_\{t\}^\{\(k\)\}\\leq\\tau\_\{\\mathrm\{prev\}\}, andQt\(k\)≤τgen\.Q\_\{t\}^\{\(k\)\}\\leq\\tau\_\{\\mathrm\{gen\}\}\.All five hyperparameters\(λ,μ,τgain,τprev,τgen\)\(\\lambda,\\mu,\\tau\_\{\\mathrm\{gain\}\},\\tau\_\{\\mathrm\{prev\}\},\\tau\_\{\\mathrm\{gen\}\}\)are fixed globally before running the continual\-learning experiments and are shared across all papers and model checkpoints\.

Among candidates that pass, we select the one with the highest utility\. If none passes, the model is left unchanged\. Otherwise, we discard the temporary update and retrain fromMt−1M\_\{t\-1\}on the selected supervision using the full budget\.

#### Preserve successful learning for future evolution\.

After committing an update, we retainMtM\_\{t\}and update two components\. First, we add examples from the selected supervisionDt\(k\)D\_\{t\}^\{\(k\)\}to the replay bufferℬt\\mathcal\{B\}\_\{t\}\. Future updates mix new supervision with replay samples to reduce forgetting\. Second, we freeze up to 50 examples fromDtdevD\_\{t\}^\{\\mathrm\{dev\}\}as a protected probevtv\_\{t\}\. We sample without replacement using seed 42, stratifying by task subtype or perturbation category when available and sampling uniformly otherwise\. If fewer than 50 examples are available, we keep all of them\. We then update the probe:

𝒱t=𝒱t−1∪\{vt\}\.\\mathcal\{V\}\_\{t\}=\\mathcal\{V\}\_\{t\-1\}\\cup\\\{v\_\{t\}\\\}\.These probes are never used for training or synthetic\-data generation\. They remain fixed and are re\-evaluated during later updates to detect forgetting of previously learned behaviors\. Without an update, the states remain unchanged:

Mt=Mt−1,ℬt=ℬt−1,𝒱t=𝒱t−1\.M\_\{t\}=M\_\{t\-1\},\\qquad\\mathcal\{B\}\_\{t\}=\\mathcal\{B\}\_\{t\-1\},\\qquad\\mathcal\{V\}\_\{t\}=\\mathcal\{V\}\_\{t\-1\}\.

## 5Experiment Setup

We evaluate paper\-driven LLM self\-evolution under a shared setup that measures target gains, retention, forgetting, and changes in general capabilities\.

### 5\.1Evaluation Datasets

#### Self\-Evolution Paper Stream Datasets\.

We construct an evaluation suite for paper\-driven self\-evolution from 30 papers that report concrete LLM failures and have corresponding benchmarks for reproducing them\. The resulting suite spans seven families:mathematical robustness, coding and software, instruction operators, knowledge and factuality, context grounding, structured reasoning, and agentic reliability\.[Table1](https://arxiv.org/html/2610.02793#S4.T1)summarizes each failure family, its high\-level LLM failure pattern, and all source papers and benchmarks\.

#### General\-Capability Evaluation Dataset\.

General ability is measured on fixed 500\-example subsets of HellaSwag\([Zellers et al\., 2019](https://arxiv.org/html/2610.02793#bib.bib46)\), WinoGrande\([Sakaguchi et al\., 2020](https://arxiv.org/html/2610.02793#bib.bib47)\), PIQA\([Bisk et al\., 2020](https://arxiv.org/html/2610.02793#bib.bib48)\), BoolQ\([Clark et al\., 2019](https://arxiv.org/html/2610.02793#bib.bib49)\), and ARC\-Challenge\([Clark et al\., 2018](https://arxiv.org/html/2610.02793#bib.bib50)\)\. We report macro\-average accuracy across these five datasets \(2,500 instances in total\)\. These reporting subsets are not used for training or update acceptance\.

Figure 2:Results of Experiment I: individual\-paper self\-evolution\. \(a\) Performance gains vary widely even within the same failure family\. \(b\) 24 of 30 papers show positive held\-out gains, while the remaining failures largely involve retrieval, planning, interaction, or verification capabilities beyond the current self\-evolution framework\.

### 5\.2Paper2LLM\+\+& Baselines

Paper2LLM\+\+uses Llama\-3\.1\-8B\-Instruct\([Grattafiori et al\., 2024](https://arxiv.org/html/2610.02793#bib.bib4)\)and Qwen2\.5\-7B\-Instruct\([Yang et al\., 2024](https://arxiv.org/html/2610.02793#bib.bib5)\)\. We use gpt\-oss\-120b\([OpenAI, 2025](https://arxiv.org/html/2610.02793#bib.bib15)\)for parsing and knowledge\-type classification, and Qwen2\.5\-32B\-Instruct\([Yang et al\., 2024](https://arxiv.org/html/2610.02793#bib.bib5)\)for synthetic data generation\.

Regarding baselines, we compare against recent continual\-learning and lifelong\-editing methods under a unified paper\-to\-update protocol\. All methods process the same papers in the same order and are evaluated with the same held\-out procedure; we adapt only how each method consumes the paper\-derived signal\. Full implementation details and hyperparameters are provided in Appendix[A\.3](https://arxiv.org/html/2610.02793#A1.SS3)\.

- •Text\-to\-LoRA\([Charakorn et al\., 2025](https://arxiv.org/html/2610.02793#bib.bib51)\): we render each extracted paper record as the task description expected by the released hypernetwork, generate one adapter per paper, and merge adapters in paper\-arrival order\.
- •ELDER\([Li et al\., 2025a](https://arxiv.org/html/2610.02793#bib.bib52)\): we convert each paper’s fixed synthetic supervision into a sequence of edits and carry ELDER’s mixture\-of\-LoRA state and router across papers, so that one paper corresponds to one continual editing stage\.
- •RLEdit\([Li et al\., 2025b](https://arxiv.org/html/2610.02793#bib.bib7)\): after pre\-training its editor on the authors’ original editing data, we apply the frozen editor to the paper\-derived edit sets, treating each paper as an editing turn\.
- •UltraEdit\([Gu et al\., 2026](https://arxiv.org/html/2610.02793#bib.bib53)\): we replace the editing stream with paper\-derived supervision and treat each paper as an editing turn while retaining the released editor and update procedure\.
- •StableEdit\([Ma et al\., 2026](https://arxiv.org/html/2610.02793#bib.bib54)\): we use UltraEdit’s paper\-to\-edit conversion and carry its running statistics across papers, using general\-domain replay only to initialize warm\-up statistics\.

### 5\.3Evaluation Setting

We evaluate self\-evolution under two settings: i\)individual learning, where the LLM learns independently from a single paper; and ii\)continual learning, where the LLM sequentially learns from a stream of papers\. For continual learning, we consider bothwithin\-familystreams, containing papers from the same failure family, andcross\-familystreams, containing papers across all seven families\. The main\-table same\-family summary reports the coding, structured\-reasoning, and agentic\-reliability streams\. For the continual settings, we report results averaged over three shuffled stream orders and training seeds\{13,21,42\}\\\{13,21,42\\\}\.

We report the following evaluation metrics\. LetSi​\(M\)S\_\{i\}\(M\)be the held\-out score for paperii, on a 0–100 scale\. LetM0M\_\{0\}denote the base model,Mi−1M\_\{i\-1\}the checkpoint before learning paperii,MiM\_\{i\}the checkpoint after its update decision, andMTM\_\{T\}the final checkpoint\. Definegi=Si​\(Mi\)−Si​\(Mi−1\)g\_\{i\}=S\_\{i\}\(M\_\{i\}\)\-S\_\{i\}\(M\_\{i\-1\}\)\.

- •Target gainmeasures improvement on the current paper’s benchmark relative to the initial model\. For reporting, defineGbase=1T​∑i=1T\[Si​\(Mi\)−Si​\(M0\)\]G\_\{\\mathrm\{base\}\}=\\frac\{1\}\{T\}\\sum\_\{i=1\}^\{T\}\[S\_\{i\}\(M\_\{i\}\)\-S\_\{i\}\(M\_\{0\}\)\]\. This measures improvement available when each paper is processed, including transfer from earlier updates\.gi=Si​\(Mi\)−Si​\(Mi−1\)g\_\{i\}=S\_\{i\}\(M\_\{i\}\)\-S\_\{i\}\(M\_\{i\-1\}\)measures only the current update\. They coincide for single papers\.
- •Retentionmeasures the fraction of a positive incremental gain that survives later updates\. Forℐ\+=\{i:gi\>0\}\\mathcal\{I\}\_\{\+\}=\\\{i:g\_\{i\}\>0\\\}, R=100\|ℐ\+\|​∑i∈ℐ\+Si​\(MT\)−Si​\(Mi−1\)gi\+ϵ,R=\\frac\{100\}\{\|\\mathcal\{I\}\_\{\+\}\|\}\\sum\_\{i\\in\\mathcal\{I\}\_\{\+\}\}\\frac\{S\_\{i\}\(M\_\{T\}\)\-S\_\{i\}\(M\_\{i\-1\}\)\}\{g\_\{i\}\+\\epsilon\},whereϵ\>0\\epsilon\>0stabilizes division\. Retention is undefined whenℐ\+\\mathcal\{I\}\_\{\+\}is empty; high retention alone does not imply a large gain\.
- •Forgettingmeasures the loss between each immediate post\-update score and its final score:F=1T​∑i=1T\[Si​\(Mi\)−Si​\(MT\)\]F=\\frac\{1\}\{T\}\\sum\_\{i=1\}^\{T\}\[S\_\{i\}\(M\_\{i\}\)\-S\_\{i\}\(M\_\{T\}\)\]\. Lower is better; negative values indicate improvement\.
- •General\-capability changemeasures collateral effects:Δgen=15​∑j=15\[Aj​\(MT\)−Aj​\(M0\)\]\\Delta\_\{\\mathrm\{gen\}\}=\\frac\{1\}\{5\}\\sum\_\{j=1\}^\{5\}\[A\_\{j\}\(M\_\{T\}\)\-A\_\{j\}\(M\_\{0\}\)\], whereAjA\_\{j\}is accuracy on general datasetjj\.

Gain, forgetting, and general change are reported in score points; retention is a percentage of acquired gain, not an accuracy\. Means and standard deviations are computed across three orders and seeds\.

## 6Experiment and Analysis

We test individual and continual paper learning, then analyze when findings yield lasting gains\.

### 6\.1Experiment I: Individual\-Paper Self\-Evolution

We first study whether an LLM can evolve by learning from a single research paper\. For each of the 30 papers, we independently start from the same base LLM, applyPaper2LLM\+\+using only that paper, and evaluate the model on the held\-out benchmark using the same setup and metrics\.

[Figure2](https://arxiv.org/html/2610.02793#S5.F2)reports the paper\-level results\. Among 30 papers, 24 achieve positive held\-out gains\. The mean across all 30 papers is \+8\.6 points, including the six negative outcomes, with a reported mean general\-capability change of \-0\.3 points\. Instruction, mathematics, and context papers improve the most, while coding and structured reasoning are harder to improve\. Overall, we find that learning from a paper does not always lead to a stronger model: while many papers produce clear gains on their reported behavioral gaps, others result in little or even negative improvement\. The corresponding family\-level values are reported in[AppendixB](https://arxiv.org/html/2610.02793#A2), Table[B](https://arxiv.org/html/2610.02793#A2)\. This raises an important question:What makes a paper effective for self\-evolution?We analyze this question in𝒬1\\mathcal\{Q\}\_\{1\}\.

Table 2:Continual self\-evolution within \(left\) and across \(right\) paper families\.
### 6\.2Experiment II: Continual Self\-Evolution Within and Across Families

[Table2](https://arxiv.org/html/2610.02793#S6.T2)evaluates continual self\-evolution under both within\-family and cross 30\-paper stream\.

#### Paper2LLM\+\+achieves a stronger gain–retention balance\.

Across both base LLMs and both stream settings,Paper2LLM\+\+consistently combines strong acquisition of new behaviors with high retention of previously learned ones\. For example, on the mixed stream,Paper2LLM\+\+achieves gains of\+7\.1\+7\.1and\+7\.5\+7\.5on Llama and Qwen while retaining83\.7%83\.7\\%and85\.0%85\.0\\%of acquired gains, respectively\. In comparison, RLEdit achieves gains of similar magnitude \(\+7\.9\+7\.9on Llama and\+6\.7\+6\.7on Qwen\) but substantially lower retention \(65\.2%65\.2\\%on Llama and71\.1%71\.1\\%on Qwen\) and higher forgetting\. These results show that effective self\-evolution requires jointly considering knowledge acquisition and retention rather than optimizing either in isolation\. \(Internal ablations: Appendix[B](https://arxiv.org/html/2610.02793#A2), Tables[B](https://arxiv.org/html/2610.02793#A2)–[B](https://arxiv.org/html/2610.02793#A2)\.\)

Longer, mixed\-family streams show more forgetting\.Moving from within\-family streams to the mixed 30\-paper stream generally makes continual learning more difficult\. For most parameter\-updating baselines, retention decreases and forgetting increases as the stream crosses failure families\. For example, RLEdit’s forgetting increases from3\.13\.1to4\.94\.9on Llama and from2\.92\.9to4\.54\.5on Qwen; Text\-To\-LoRA shows a similar increase from4\.24\.2to5\.65\.6and from3\.63\.6to5\.35\.3, respectively\. In contrast,Paper2LLM\+\+shows a smaller increase, from1\.81\.8to2\.42\.4on Llama and from1\.51\.5to2\.12\.1on Qwen, while maintaining strong gains\. The mixed streams differ in both family diversity and length, so these results do not isolate a causal effect of heterogeneity\. They show thatPaper2LLM\+\+limits forgetting in the longer, mixed\-family setting\.

Paper2LLM\+\+better preserves general capabilities\.The mixed\-stream setting also evaluates whether continual self\-evolution degrades general LLM capabilities\.Paper2LLM\+\+limits the general\-capability change to−0\.4\-0\.4on Llama and−0\.5\-0\.5on Qwen, compared with substantially larger degradation for aggressive parameter\-updating methods such as RLEdit \(−1\.6\-1\.6on Llama and−1\.4\-1\.4on Qwen\) and Text\-To\-LoRA \(−2\.0\-2\.0on Llama and−1\.8\-1\.8on Qwen\)\. Thus,Paper2LLM\+\+’s advantage is not simply conservative updating: it achieves substantial continual gains while limiting both forgetting and general\-capability regression\.

### 6\.3Analysis: When Does Learning from Research Papers Work?

Figure 3:Understanding papers’ effective signals for LLM self\-evolution: \(a\)Qualitatively:gains vary across knowledge types\. \(b\)Quantitatively:supervision closure is associated with held\-out gain \(r=0\.87r=0\.87; Appendix[A\.4](https://arxiv.org/html/2610.02793#A1.SS4.SSS0.Px2)\)\.The individual\-paper and continual experiments establish that LLMs can continually learn from research papers\. We next ask a more fundamental question:*when does a research finding actually provide a useful signal for self\-evolution?*

Our analysis suggests that successful paper\-driven self\-evolution depends on three conditions\. First, the finding must beteachable: it must be possible to translate the reported insight into concrete supervision that the model can learn from\. Second, it must beneeded: the current model must still exhibit the behavior that the paper aims to improve\. Third, what is learned must beretainedas subsequent papers introduce additional updates\.[Figure3](https://arxiv.org/html/2610.02793#S6.F3),[Figure4](https://arxiv.org/html/2610.02793#S6.F4), and[Figure5](https://arxiv.org/html/2610.02793#S6.F5)examine these conditions across our 30 papers and continual streams\. Paper\-level values are in Appendix[B](https://arxiv.org/html/2610.02793#A2), Tables[B](https://arxiv.org/html/2610.02793#A2)–[B](https://arxiv.org/html/2610.02793#A2)\.

#### 𝒬1\\mathcal\{Q\}\_\{1\}: When Is a Research Finding Teachable?

We first ask why some papers produce large self\-evolution gains while others produce little improvement\. Before inspecting held\-out outcomes, we use gpt\-oss\-120b\([OpenAI, 2025](https://arxiv.org/html/2610.02793#bib.bib15)\)to annotate each paper by the*primary knowledge or behavior that must be installed to address its reported failure*, rather than by its benchmark domain\. The fine\-grained labels include reasoning patterns, boundaries, constraints, verification strategies, retrieval or memory strategies, planning or interaction strategies, algorithmic procedures, knowledge updates, execution feedback, and safety policies\. For the analysis in[Figure3](https://arxiv.org/html/2610.02793#S6.F3)\(a\), semantically related labels are grouped into broader knowledge types\.

[Figure3](https://arxiv.org/html/2610.02793#S6.F3)\(a\) shows that the answer depends strongly on what the paper asks the model to learn\. Findings involving*constraints, decision boundaries, reasoning patterns, and verification strategies*generally produce larger gains, whereas findings requiring*planning, retrieval, or complex interaction*are substantially harder to acquire\.

A common pattern behind these differences is whether the paper’s finding can be turned into concrete training supervision\. For example, if a paper identifies that the model should change its answer under a particular constraint, we can construct examples that demonstrate when the behavior should change and automatically check whether the resulting behavior is correct\. In contrast, some findings require capabilities that cannot be provided by input–output supervision alone, such as searching for unavailable information, maintaining and updating an external state, interacting with an environment, or executing and verifying steps\.

To describe this distinction, we use each paper’s existingsupervision closureannotation, a continuous measure of how readily its finding can be converted into constructible and verifiable supervision\.[AppendixA\.4](https://arxiv.org/html/2610.02793#A1.SS4.SSS0.Px2)provides a reproducible rubric over five dimensions: explicit target behavior \(ctargetc\_\{\\mathrm\{target\}\}\), constructible supervision \(csupc\_\{\\mathrm\{sup\}\}\), automatic verification \(cverc\_\{\\mathrm\{ver\}\}\), local evaluability \(clocalc\_\{\\mathrm\{local\}\}\), and mechanism sufficiency \(cmechc\_\{\\mathrm\{mech\}\}\)\. Each dimension is scored continuously in\[0,1\]\[0,1\]: a score of00indicates that the property is absent or cannot be operationalized from the paper,0\.50\.5indicates partial or conditional support, and11indicates that the property is explicitly and directly satisfied; intermediate values reflect evidence between these anchors\. Closure is then their unweighted mean:

Ct=ctarget\+csup\+cver\+clocal\+cmech5\.C\_\{t\}=\\frac\{c\_\{\\mathrm\{target\}\}\+c\_\{\\mathrm\{sup\}\}\+c\_\{\\mathrm\{ver\}\}\+c\_\{\\mathrm\{local\}\}\+c\_\{\\mathrm\{mech\}\}\}\{5\}\.\(1\)The Appendix rubric provides a reproducible interpretation of the existing descriptive scores\.

[Figure3](https://arxiv.org/html/2610.02793#S6.F3)\(b\) shows a strong relationship between supervision closure and held\-out self\-evolution gain \(r=0\.87r=0\.87\)\. Papers whose findings can be expressed as clear and verifiable learning signals tend to produce substantially larger improvements\. This association is descriptive; it does not validate the scoring rubric or establish a causal effect\.

*Importantly, requiring an additional mechanism does not automatically make a finding unlearnable\. What matters is whether that mechanism can itself provide a training signal or whether it is the capability that is missing\.*For example, an external checker can make verification useful for learning because it directly provides correctness feedback\. By contrast, when reliable verification is itself the behavior that the model lacks, showing additional input–output examples may not be sufficient\. Similarly, selecting relevant evidence already present in the context can be directly supervised, whereas learning to search for evidence unavailable to the model requires an additional retrieval process\.

![Refer to caption](https://arxiv.org/html/2610.02793v1/figure4_novelty.png)Figure 4:Gain vs\. \{semant\. novelty, behav\. novelty\}
#### 𝒬2\\mathcal\{Q\}\_\{2\}: When Does the Current Model Need the Finding?

Teachability alone does not make a paper useful for self\-evolution\. Even a finding that can be converted into high\-quality supervision provides little value if the current model already handles the reported behavior correctly\. We therefore distinguish what is new in the research literature from what is still new to the current model\.

We characterizesemantic noveltyby how different a paper\-reported failure is from earlier failure schemas in the stream, andbehavioral noveltyby how strongly the current checkpoint still exhibits the reported failure\. Semantic novelty is computed as one minus the maximum cosine similarity between the current failure\-schema embedding and earlier schema embeddings, while behavioral novelty is the paper\-defined failure severity min–max normalized to\[0,1\]\[0,1\]\. Full definitions are provided in[AppendixA\.4](https://arxiv.org/html/2610.02793#A1.SS4.SSS0.Px3)\.

The two forms of novelty capture different situations\. A semantically novel failure may provide little learning value if the current model already handles it well\. Conversely, a familiar failure can remain useful when it exposes an unresolved weakness of the current checkpoint\. The largest gains arise when a paper provides non\-redundant information about behavior the model still needs to improve\. This also motivates a key design choice inPaper2LLM\+\+: a new paper does not automatically trigger an update\.Paper2LLM\+\+first verifies whether the reported behavioral gap remains in the current checkpoint and learns from the paper only when there is a meaningful gap to address\.

Figure 5:Marginal gains decrease but remain positive throughout the 30\-paper stream\.
#### 𝒬3\\mathcal\{Q\}\_\{3\}: What Limits Self\-Evolution as More Papers Arrive?

Two factors could limit long\-horizon self\-evolution\. First, the model may gradually*run out of useful learning signals*: later papers may increasingly describe behaviors that the model already handles\. Second, useful new findings may continue to arrive, but learning them may overwrite improvements acquired from earlier papers\.

[Figure5](https://arxiv.org/html/2610.02793#S6.F5)provides evidence about the first possibility\. The marginal gain from each additional paper decreases from approximately 12 points early in the stream to around 5 points near the 30th paper, but remains positive throughout the studied stream\. Thus, although additional papers provide diminishing returns, useful learning signals do not appear exhausted at this scale\.

The continual\-learning results in[Table2](https://arxiv.org/html/2610.02793#S6.T2)reveal a different challenge: preserving previously acquired behavior\. Parameter\-updating methods can achieve gains on new findings while losing part of earlier improvements, and this effect becomes more pronounced in cross\-family streams\.Paper2LLM\+\+substantially reduces this degradation, maintaining strong gains while achieving higher retention and lower forgetting than competing parameter\-updating approaches across both base LLMs\. This suggests that, at the scale studied here, the central long\-horizon challenge is not simply finding new research signals, but accumulating them without overwriting prior learning\.

## 7Conclusion

This work introducedPaper2LLM\+\+, a framework for continual LLM self\-evolution from research papers\. Across 30 papers and two LLMs, our results show that research findings can drive persistent model improvements, but not all findings are equally useful\. Effective self\-evolution requires findings to be*teachable*,*needed*, and*retained*\. Our analysis further reveals an important boundary: some failures can be addressed through generated supervision, while others require additional mechanisms such as retrieval, planning, interaction, or verification\. Ultimately, scientific literature may serve not only as a record of AI progress, but as a continual source of supervision through which models themselves evolve\.

## References

- Ashury\-Tahanet al\.\(2026\)S\. Ashury\-Tahan, Y\. Mai, R\. C, A\. Gera, Y\. Perlitz, A\. Yehudai, E\. Bandel, L\. Choshen, E\. Shnarch, P\. Liang, and M\. Shmueli\-ScheuerThe Mighty ToRR: A Benchmark for Table Reasoning and Robustness in LLMs\.InProceedings of the First Workshop on Structured Understanding, Retrieval, and Generation in the LLM Era \(SURGeLLM 2026\),San Diego, California, United States,pp\. 16–51\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.surgellm-1.2),ISBN 979\-8\-89176\-406\-4,[Link](https://aclanthology.org/2026.surgellm-1.2/)Cited by:[Table 1](https://arxiv.org/html/2610.02793#S4.T1.2.8.3.1.1)\.
- Berglundet al\.\(2024\)L\. Berglund, M\. Tong, M\. Kaufmann, M\. Balesni, A\. Stickland, T\. Korbak, and O\. EvansThe Reversal Curse: LLMs trained on “A is B”fail to learn “B is A”\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 18623–18642\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/hash/%0A5178b2f2d7c44aa390c0777dc77b3f0c-Abstract-Conference.html)Cited by:[Table 1](https://arxiv.org/html/2610.02793#S4.T1.2.4.3.1.1)\.
- Bisket al\.\(2020\)Y\. Bisk, R\. Zellers, R\. Le Bras, J\. Gao, and Y\. ChoiPIQA: Reasoning about Physical Commonsense in Natural Language\.Proceedings of the AAAI Conference on Artificial Intelligence34\(05\),pp\. 7432–7439\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v34i05.6239),[Link](https://doi.org/10.1609/aaai.v34i05.6239)Cited by:[§5\.1](https://arxiv.org/html/2610.02793#S5.SS1.SSS0.Px2.p1.1)\.
- Charakornet al\.\(2025\)R\. Charakorn, E\. Cetin, Y\. Tang, and R\. T\. LangeText\-to\-LoRA: Instant Transformer Adaption\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 7485–7514\.External Links:[Link](https://proceedings.mlr.press/v267/charakorn25a.html)Cited by:[1st item](https://arxiv.org/html/2610.02793#S5.I1.i1.p1.1),[Table 2](https://arxiv.org/html/2610.02793#S6.T2.2.11.1.1),[Table 2](https://arxiv.org/html/2610.02793#S6.T2.2.4.1.1)\.
- Chenet al\.\(2024\)J\. Chen, H\. Lin, X\. Han, and L\. SunBenchmarking Large Language Models in Retrieval\-Augmented Generation\.Proceedings of the AAAI Conference on Artificial Intelligence38\(16\),pp\. 17754–17762\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v38i16.29728),[Link](https://doi.org/10.1609/aaai.v38i16.29728)Cited by:[Table 1](https://arxiv.org/html/2610.02793#S4.T1.2.6.3.1.1)\.
- Chenet al\.\(2026a\)X\. Chen, C\. Xu, Y\. Wang, B\. Liu, Z\. Yao, and Y\. HeLearning to Self\-Evolve\.InLifelong Agent Workshop at ICLR,External Links:[Link](https://openreview.net/forum?id=zedEdPhmsA)Cited by:[§2](https://arxiv.org/html/2610.02793#S2.SS0.SSS0.Px1.p1.1)\.
- Chenet al\.\(2026b\)Y\. Chen, V\. K\. Singh, J\. Ma, and R\. TangCounterBench: Evaluating and Improving Counterfactual Reasoning in Large Language Models\.Proceedings of the AAAI Conference on Artificial Intelligence40\(36\),pp\. 30350–30358\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v40i36.40287),[Link](https://doi.org/10.1609/aaai.v40i36.40287)Cited by:[Table 1](https://arxiv.org/html/2610.02793#S4.T1.2.8.3.1.1)\.
- Clarket al\.\(2019\)C\. Clark, K\. Lee, M\. Chang, T\. Kwiatkowski, M\. Collins, and K\. ToutanovaBoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),Minneapolis, Minnesota,pp\. 2924–2936\.External Links:[Document](https://dx.doi.org/10.18653/v1/N19-1300),[Link](https://aclanthology.org/N19-1300/)Cited by:[§5\.1](https://arxiv.org/html/2610.02793#S5.SS1.SSS0.Px2.p1.1)\.
- Clarket al\.\(2018\)P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. TafjordThink you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge\.Note:arXiv:1803\.05457v1External Links:[Link](https://arxiv.org/abs/1803.05457v1)Cited by:[§5\.1](https://arxiv.org/html/2610.02793#S5.SS1.SSS0.Px2.p1.1)\.
- Dinget al\.\(2023\)Y\. Ding, Z\. Wang, W\. Ahmad, H\. Ding, M\. Tan, N\. Jain, M\. K\. Ramanathan, R\. Nallapati, P\. Bhatia, D\. Roth, and B\. XiangCrossCodeEval: A Diverse and Multilingual Benchmark for Cross\-File Code Completion\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 46701–46723\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/%0A920f2dced7d32ab2ba2f1970bc306af6-Abstract-%0ADatasets_and_Benchmarks.html)Cited by:[Table 1](https://arxiv.org/html/2610.02793#S4.T1.2.3.3.1.1)\.
- Fenget al\.\(2025\)Y\. Feng, X\. Wang, Z\. Lu, S\. Fu, G\. Shi, Y\. Xu, Y\. Wang, P\. S\. Yu, X\. Chu, and X\. WuRecurrent Knowledge Identification and Fusion for Language Model Continual Learning\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Vienna, Austria,pp\. 27396–27413\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1328),ISBN 979\-8\-89176\-251\-0,[Link](https://aclanthology.org/2025.acl-long.1328/)Cited by:[§2](https://arxiv.org/html/2610.02793#S2.SS0.SSS0.Px1.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan, A\. Yang, A\. Fan, A\. Goyal, A\. Hartshorn, A\. Yang, A\. Mitra, A\. Sravankumar, A\. Korenev, A\. Hinsvark, A\. Rao, A\. Zhang, A\. Rodriguez, A\. Gregerson, A\. Spataru, B\. Roziere, B\. Biron, B\. Tang, B\. Chern, C\. Caucheteux, C\. Nayak, C\. Bi, C\. Marra, C\. McConnell, C\. Keller, C\. Touret, C\. Wu, C\. Wong, C\. C\. Ferrer, C\. Nikolaidis, D\. Allonsius, D\. Song, D\. Pintz, D\. Livshits, D\. Wyatt, D\. Esiobu, D\. Choudhary, D\. Mahajan, D\. Garcia\-Olano, D\. Perino, D\. Hupkes, E\. Lakomkin, E\. AlBadawy, E\. Lobanova, E\. Dinan, E\. M\. Smith, F\. Radenovic, F\. Guzmán, F\. Zhang, G\. Synnaeve, G\. Lee, G\. L\. Anderson, G\. Thattai, G\. Nail, G\. Mialon, G\. Pang, G\. Cucurell, H\. Nguyen, H\. Korevaar, H\. Xu, H\. Touvron, I\. Zarov, I\. A\. Ibarra, I\. Kloumann, I\. Misra, I\. Evtimov, J\. Zhang, J\. Copet, J\. Lee, J\. Geffert, J\. Vranes, J\. Park, J\. Mahadeokar, J\. Shah, J\. van der Linde, J\. Billock, J\. Hong, J\. Lee, J\. Fu, J\. Chi, J\. Huang, J\. Liu, J\. Wang, J\. Yu, J\. Bitton, J\. Spisak, J\. Park, J\. Rocca, J\. Johnstun, J\. Saxe, J\. Jia, K\. V\. Alwala, K\. Prasad, K\. Upasani, K\. Plawiak, K\. Li, K\. Heafield, K\. Stone, K\. El\-Arini, K\. Iyer, K\. Malik, K\. Chiu, K\. Bhalla, K\. Lakhotia, L\. Rantala\-Yeary, L\. van der Maaten, L\. Chen, L\. Tan, L\. Jenkins, L\. Martin, L\. Madaan, L\. Malo, L\. Blecher, L\. Landzaat, L\. de Oliveira, M\. Muzzi, M\. Pasupuleti, M\. Singh, M\. Paluri, M\. Kardas, M\. Tsimpoukelli, M\. Oldham, M\. Rita, M\. Pavlova, M\. Kambadur, M\. Lewis, M\. Si, M\. K\. Singh, M\. Hassan, N\. Goyal, N\. Torabi, N\. Bashlykov, N\. Bogoychev, N\. Chatterji, N\. Zhang, O\. Duchenne, O\. Çelebi, P\. Alrassy, P\. Zhang, P\. Li, P\. Vasic, P\. Weng, P\. Bhargava, P\. Dubal, P\. Krishnan, P\. S\. Koura, P\. Xu, Q\. He, Q\. Dong, R\. Srinivasan, R\. Ganapathy, R\. Calderer, R\. S\. Cabral, R\. Stojnic, R\. Raileanu, R\. Maheswari, R\. Girdhar, R\. Patel, R\. Sauvestre, R\. Polidoro, R\. Sumbaly, R\. Taylor, R\. Silva, R\. Hou, R\. Wang, S\. Hosseini, S\. Chennabasappa, S\. Singh, S\. Bell, S\. S\. Kim, S\. Edunov, S\. Nie, S\. Narang, S\. Raparthy, S\. Shen, S\. Wan, S\. Bhosale, S\. Zhang, S\. Vandenhende, S\. Batra, S\. Whitman, S\. Sootla, S\. Collot, S\. Gururangan, S\. Borodinsky, T\. Herman, T\. Fowler, T\. Sheasha, T\. Georgiou, T\. Scialom, T\. Speckbacher, T\. Mihaylov, T\. Xiao, U\. Karn, V\. Goswami, V\. Gupta, V\. Ramanathan, V\. Kerkez, V\. Gonguet, V\. Do, V\. Vogeti, V\. Albiero, V\. Petrovic, W\. Chu, W\. Xiong, W\. Fu, W\. Meers, X\. Martinet, X\. Wang, X\. Wang, X\. E\. Tan, X\. Xia, X\. Xie, X\. Jia, X\. Wang, Y\. Goldschlag, Y\. Gaur, Y\. Babaei, Y\. Wen, Y\. Song, Y\. Zhang, Y\. Li, Y\. Mao, Z\. D\. Coudert, Z\. Yan, Z\. Chen, Z\. Papakipos, A\. Singh, A\. Srivastava, A\. Jain, A\. Kelsey, A\. Shajnfeld, A\. Gangidi, A\. Victoria, A\. Goldstand, A\. Menon, A\. Sharma, A\. Boesenberg, A\. Baevski, A\. Feinstein, A\. Kallet, A\. Sangani, A\. Teo, A\. Yunus, A\. Lupu, A\. Alvarado, A\. Caples, A\. Gu, A\. Ho, A\. Poulton, A\. Ryan, A\. Ramchandani, A\. Dong, A\. Franco, A\. Goyal, A\. Saraf, A\. Chowdhury, A\. Gabriel, A\. Bharambe, A\. Eisenman, A\. Yazdan, B\. James, B\. Maurer, B\. Leonhardi, B\. Huang, B\. Loyd, B\. D\. Paola, B\. Paranjape, B\. Liu, B\. Wu, B\. Ni, B\. Hancock, B\. Wasti, B\. Spence, B\. Stojkovic, B\. Gamido, B\. Montalvo, C\. Parker, C\. Burton, C\. Mejia, C\. Liu, C\. Wang, C\. Kim, C\. Zhou, C\. Hu, C\. Chu, C\. Cai, C\. Tindal, C\. Feichtenhofer, C\. Gao, D\. Civin, D\. Beaty, D\. Kreymer, D\. Li, D\. Adkins, D\. Xu, D\. Testuggine, D\. David, D\. Parikh, D\. Liskovich, D\. Foss, D\. Wang, D\. Le, D\. Holland, E\. Dowling, E\. Jamil, E\. Montgomery, E\. Presani, E\. Hahn, E\. Wood, E\. Le, E\. Brinkman, E\. Arcaute, E\. Dunbar, E\. Smothers, F\. Sun, F\. Kreuk, F\. Tian, F\. Kokkinos, F\. Ozgenel, F\. Caggioni, F\. Kanayet, F\. Seide, G\. M\. Florez, G\. Schwarz, G\. Badeer, G\. Swee, G\. Halpern, G\. Herman, G\. Sizov, G\. Zhang, G\. Lakshminarayanan, H\. Inan, H\. Shojanazeri, H\. Zou, H\. Wang, H\. Zha, H\. Habeeb, H\. Rudolph, H\. Suk, H\. Aspegren, H\. Goldman, H\. Zhan, I\. Damlaj, I\. Molybog, I\. Tufanov, I\. Leontiadis, I\. Veliche, I\. Gat, J\. Weissman, J\. Geboski, J\. Kohli, J\. Lam, J\. Asher, J\. Gaya, J\. Marcus, J\. Tang, J\. Chan, J\. Zhen, J\. Reizenstein, J\. Teboul, J\. Zhong, J\. Jin, J\. Yang, J\. Cummings, J\. Carvill, J\. Shepard, J\. McPhie, J\. Torres, J\. Ginsburg, J\. Wang, K\. Wu, K\. H\. U, K\. Saxena, K\. Khandelwal, K\. Zand, K\. Matosich, K\. Veeraraghavan, K\. Michelena, K\. Li, K\. Jagadeesh, K\. Huang, K\. Chawla, K\. Huang, L\. Chen, L\. Garg, L\. A, L\. Silva, L\. Bell, L\. Zhang, L\. Guo, L\. Yu, L\. Moshkovich, L\. Wehrstedt, M\. Khabsa, M\. Avalani, M\. Bhatt, M\. Mankus, M\. Hasson, M\. Lennie, M\. Reso, M\. Groshev, M\. Naumov, M\. Lathi, M\. Keneally, M\. Liu, M\. L\. Seltzer, M\. Valko, M\. Restrepo, M\. Patel, M\. Vyatskov, M\. Samvelyan, M\. Clark, M\. Macey, M\. Wang, M\. J\. Hermoso, M\. Metanat, M\. Rastegari, M\. Bansal, N\. Santhanam, N\. Parks, N\. White, N\. Bawa, N\. Singhal, N\. Egebo, N\. Usunier, N\. Mehta, N\. P\. Laptev, N\. Dong, N\. Cheng, O\. Chernoguz, O\. Hart, O\. Salpekar, O\. Kalinli, P\. Kent, P\. Parekh, P\. Saab, P\. Balaji, P\. Rittner, P\. Bontrager, P\. Roux, P\. Dollar, P\. Zvyagina, P\. Ratanchandani, P\. Yuvraj, Q\. Liang, R\. Alao, R\. Rodriguez, R\. Ayub, R\. Murthy, R\. Nayani, R\. Mitra, R\. Parthasarathy, R\. Li, R\. Hogan, R\. Battey, R\. Wang, R\. Howes, R\. Rinott, S\. Mehta, S\. Siby, S\. J\. Bondu, S\. Datta, S\. Chugh, S\. Hunt, S\. Dhillon, S\. Sidorov, S\. Pan, S\. Mahajan, S\. Verma, S\. Yamamoto, S\. Ramaswamy, S\. Lindsay, S\. Lindsay, S\. Feng, S\. Lin, S\. C\. Zha, S\. Patil, S\. Shankar, S\. Zhang, S\. Zhang, S\. Wang, S\. Agarwal, S\. Sajuyigbe, S\. Chintala, S\. Max, S\. Chen, S\. Kehoe, S\. Satterfield, S\. Govindaprasad, S\. Gupta, S\. Deng, S\. Cho, S\. Virk, S\. Subramanian, S\. Choudhury, S\. Goldman, T\. Remez, T\. Glaser, T\. Best, T\. Koehler, T\. Robinson, T\. Li, T\. Zhang, T\. Matthews, T\. Chou, T\. Shaked, V\. Vontimitta, V\. Ajayi, V\. Montanez, V\. Mohan, V\. S\. Kumar, V\. Mangla, V\. Ionescu, V\. Poenaru, V\. T\. Mihailescu, V\. Ivanov, W\. Li, W\. Wang, W\. Jiang, W\. Bouaziz, W\. Constable, X\. Tang, X\. Wu, X\. Wang, X\. Wu, X\. Gao, Y\. Kleinman, Y\. Chen, Y\. Hu, Y\. Jia, Y\. Qi, Y\. Li, Y\. Zhang, Y\. Zhang, Y\. Adi, Y\. Nam, Y\. Wang, Y\. Zhao, Y\. Hao, Y\. Qian, Y\. Li, Y\. He, Z\. Rait, Z\. DeVito, Z\. Rosnbrick, Z\. Wen, Z\. Yang, Z\. Zhao, and Z\. MaThe Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783v3\.External Links:[Link](https://arxiv.org/abs/2407.21783v3)Cited by:[§1](https://arxiv.org/html/2610.02793#S1.p3.1),[§5\.2](https://arxiv.org/html/2610.02793#S5.SS2.p1.1),[Table 2](https://arxiv.org/html/2610.02793#S6.T2.2.3.1.1)\.
- Guet al\.\(2024\)A\. Gu, B\. Roziere, H\. J\. Leather, A\. Solar\-Lezama, G\. Synnaeve, and S\. WangCRUXEval: A Benchmark for Code Reasoning, Understanding and Execution\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 16568–16621\.External Links:[Link](https://proceedings.mlr.press/v235/gu24c.html)Cited by:[Table 1](https://arxiv.org/html/2610.02793#S4.T1.2.3.3.1.1)\.
- Guet al\.\(2026\)X\. Gu, Z\. Huang, J\. Gu, and K\. ZhangUltraEdit: Training\-, Subject\-, and Memory\-Free Lifelong Editing in Language Models\.Transactions on Machine Learning Research\.External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=GoJLp3BlRV)Cited by:[4th item](https://arxiv.org/html/2610.02793#S5.I1.i4.p1.1),[Table 2](https://arxiv.org/html/2610.02793#S6.T2.2.14.1.1),[Table 2](https://arxiv.org/html/2610.02793#S6.T2.2.7.1.1)\.
- Heet al\.\(2024\)X\. He, Y\. Tian, Y\. Sun, N\. V\. Chawla, T\. Laurent, Y\. LeCun, X\. Bresson, and B\. HooiG\-Retriever: Retrieval\-Augmented Generation for Textual Graph Understanding and Question Answering\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 132876–132907\.External Links:[Document](https://dx.doi.org/10.52202/079017-4224),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/%0Aefaf1c9726648c8ba363a5c927440529-Abstract-Conference.html)Cited by:[Table 1](https://arxiv.org/html/2610.02793#S4.T1.2.7.3.1.1)\.
- Hsiehet al\.\(2024\)C\. Hsieh, S\. Sun, S\. Kriman, S\. Acharya, D\. Rekesh, F\. Jia, Y\. Zhang, and B\. GinsburgRULER: What’s the Real Context Size of Your Long\-Context Language Models?\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=kIoBbc76Sy)Cited by:[Table 1](https://arxiv.org/html/2610.02793#S4.T1.2.6.3.1.1)\.
- Huanget al\.\(2025\)K\. Huang, J\. Guo, Z\. Li, X\. Ji, J\. Ge, W\. Li, Y\. Guo, T\. Cai, H\. Yuan, R\. Wang, Y\. Wu, M\. Yin, S\. Tang, Y\. Huang, C\. Jin, X\. Chen, C\. Zhang, and M\. WangMATH\-Perturb: Benchmarking LLMs’ Math Reasoning Abilities against Hard Perturbations\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 25311–25328\.External Links:[Link](https://proceedings.mlr.press/v267/huang25k.html)Cited by:[Table 1](https://arxiv.org/html/2610.02793#S4.T1.2.2.3.1.1)\.
- Jainet al\.\(2025\)N\. Jain, K\. Han, A\. Gu, W\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. StoicaLiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code\.InInternational Conference on Learning Representations,External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/hash/%0A94074dd5a072d28ff75a76dabed43767-Abstract-Conference.html)Cited by:[Table 1](https://arxiv.org/html/2610.02793#S4.T1.2.3.3.1.1)\.
- Janget al\.\(2022\)J\. Jang, S\. Ye, and M\. SeoCan Large Language Models Truly Understand Prompts? A Case Study with Negated Prompts\.Note:arXiv:2209\.12711v1External Links:[Document](https://dx.doi.org/10.48550/arXiv.2209.12711),[Link](https://arxiv.org/abs/2209.12711v1)Cited by:[Table 1](https://arxiv.org/html/2610.02793#S4.T1.2.4.3.1.1)\.
- Jimenezet al\.\(2024\)C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. NarasimhanSWE\-bench: Can Language Models Resolve Real\-world Github Issues?\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 54107–54157\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/hash/%0Aedac78c3e300629acfe6cbe9ca88fb84-Abstract-Conference.html)Cited by:[Table 1](https://arxiv.org/html/2610.02793#S4.T1.2.3.3.1.1)\.
- Liet al\.\(2025a\)J\. Li, Q\. Wang, Z\. Wang, Y\. Zhang, and Z\. MaoELDER: Enhancing Lifelong Model Editing with Mixture\-of\-LoRA\.Proceedings of the AAAI Conference on Artificial Intelligence39\(23\),pp\. 24440–24448\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v39i23.34622),[Link](https://doi.org/10.1609/aaai.v39i23.34622)Cited by:[2nd item](https://arxiv.org/html/2610.02793#S5.I1.i2.p1.1),[Table 2](https://arxiv.org/html/2610.02793#S6.T2.2.12.1.1),[Table 2](https://arxiv.org/html/2610.02793#S6.T2.2.5.1.1)\.
- Liet al\.\(2024a\)Q\. Li, L\. Cui, X\. Zhao, L\. Kong, and W\. BiGSM\-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 2961–2984\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.163),[Link](https://aclanthology.org/2024.acl-long.163/)Cited by:[§4\.1](https://arxiv.org/html/2610.02793#S4.SS1.SSS0.Px1.p1.2),[§4\.1](https://arxiv.org/html/2610.02793#S4.SS1.SSS0.Px3.p2.1),[§4\.2](https://arxiv.org/html/2610.02793#S4.SS2.p2.1),[Table 1](https://arxiv.org/html/2610.02793#S4.T1.2.2.3.1.1)\.
- Liet al\.\(2024b\)Z\. Li, B\. Peng, P\. He, and X\. YanEvaluating the Instruction\-Following Robustness of Large Language Models to Prompt Injection\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Miami, Florida, USA,pp\. 557–568\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.33),[Link](https://aclanthology.org/2024.emnlp-main.33/)Cited by:[Table 1](https://arxiv.org/html/2610.02793#S4.T1.2.4.3.1.1)\.
- Liet al\.\(2025b\)Z\. Li, H\. Jiang, H\. Chen, B\. Bi, Z\. Zhou, F\. Sun, J\. Fang, and X\. WangReinforced Lifelong Editing for Language Models\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 34920–34942\.External Links:[Link](https://proceedings.mlr.press/v267/li25an.html)Cited by:[§2](https://arxiv.org/html/2610.02793#S2.SS0.SSS0.Px1.p1.1),[3rd item](https://arxiv.org/html/2610.02793#S5.I1.i3.p1.1),[Table 2](https://arxiv.org/html/2610.02793#S6.T2.2.13.1.1),[Table 2](https://arxiv.org/html/2610.02793#S6.T2.2.6.1.1)\.
- Linet al\.\(2022\)S\. Lin, J\. Hilton, and O\. EvansTruthfulQA: Measuring How Models Mimic Human Falsehoods\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Dublin, Ireland,pp\. 3214–3252\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.acl-long.229),[Link](https://aclanthology.org/2022.acl-long.229/)Cited by:[Table 1](https://arxiv.org/html/2610.02793#S4.T1.2.5.3.1.1)\.
- Liuet al\.\(2023\)J\. Liu, C\. S\. Xia, Y\. Wang, and L\. ZhangIs Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 21558–21572\.External Links:[Document](https://dx.doi.org/10.52202/075280-0943),[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/%0A43e9d647ccd3e4b7b5baab53f0368686-Abstract-Conference.html)Cited by:[Table 1](https://arxiv.org/html/2610.02793#S4.T1.2.3.3.1.1)\.
- Liuet al\.\(2024a\)N\. F\. Liu, K\. Lin, J\. Hewitt, A\. Paranjape, M\. Bevilacqua, F\. Petroni, and P\. LiangLost in the Middle: How Language Models Use Long Contexts\.Transactions of the Association for Computational Linguistics12,pp\. 157–173\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00638),[Link](https://aclanthology.org/2024.tacl-1.9/)Cited by:[Table 1](https://arxiv.org/html/2610.02793#S4.T1.2.6.3.1.1)\.
- Liuet al\.\(2024b\)X\. Liu, H\. Yu, H\. Zhang, Y\. Xu, X\. Lei, H\. Lai, Y\. Gu, H\. Ding, K\. Men, K\. Yang, S\. Zhang, X\. Deng, A\. Zeng, Z\. Du, C\. Zhang, S\. Shen, T\. Zhang, Y\. Su, H\. Sun, M\. Huang, Y\. Dong, and J\. TangAgentBench: Evaluating LLMs as Agents\.InInternational Conference on Learning Representations,External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/hash/%0Ae9df36b21ff4ee211a8b71ee8b7e9f57-Abstract-Conference.html)Cited by:[Table 1](https://arxiv.org/html/2610.02793#S4.T1.2.8.3.1.1)\.
- Luet al\.\(2024\)P\. Lu, H\. Bansal, T\. Xia, J\. Liu, C\. Li, H\. Hajishirzi, H\. Cheng, K\. Chang, M\. Galley, and J\. GaoMathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts\.InInternational Conference on Learning Representations,External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/hash/%0A663bce02a0050c4a11f1eb8a7f1429d3-Abstract-Conference.html)Cited by:[Table 1](https://arxiv.org/html/2610.02793#S4.T1.2.2.3.1.1)\.
- Maet al\.\(2026\)X\. Ma, W\. Chen, Q\. Liu, D\. Xu, Z\. Zheng, T\. Xu, and E\. ChenMore Edits, More Stable: Understanding the Lifelong Normalization in Sequential Model Editing\.InProceedings of the 43rd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.306,pp\. 84195–84250\.External Links:[Link](https://proceedings.mlr.press/v306/ma26f.html)Cited by:[5th item](https://arxiv.org/html/2610.02793#S5.I1.i5.p1.1),[Table 2](https://arxiv.org/html/2610.02793#S6.T2.2.15.1.1),[Table 2](https://arxiv.org/html/2610.02793#S6.T2.2.8.1.1)\.
- Madaanet al\.\(2023\)A\. Madaan, N\. Tandon, P\. Gupta, S\. Hallinan, L\. Gao, S\. Wiegreffe, U\. Alon, N\. Dziri, S\. Prabhumoye, Y\. Yang, S\. Gupta, B\. P\. Majumder, K\. Hermann, S\. Welleck, A\. Yazdanbakhsh, and P\. ClarkSelf\-Refine: Iterative Refinement with Self\-Feedback\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 46534–46594\.External Links:[Document](https://dx.doi.org/10.52202/075280-2019),[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/%0A91edff07232fb1b55a505a9e9f6c0ff3-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2610.02793#S1.p1.1)\.
- Mirzadehet al\.\(2025\)I\. Mirzadeh, K\. Alizadeh\-Vahid, H\. Shahrokhi, O\. Tuzel, S\. Bengio, and M\. FarajtabarGSM\-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 94743–94765\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/hash/%0Aec2e7a896f8250986b3907f57621ce94-Abstract-Conference.html)Cited by:[Table 1](https://arxiv.org/html/2610.02793#S4.T1.2.2.3.1.1)\.
- OpenAI \(2025\)OpenAIgpt\-oss\-120b & gpt\-oss\-20b Model Card\.External Links:[Link](https://openai.com/index/gpt-oss-model-card/)Cited by:[§A\.1](https://arxiv.org/html/2610.02793#A1.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2610.02793#S4.SS1.SSS0.Px1.p1.1),[§5\.2](https://arxiv.org/html/2610.02793#S5.SS2.p1.1),[§6\.3](https://arxiv.org/html/2610.02793#S6.SS3.SSS0.Px1.p1.1)\.
- Qinet al\.\(2024\)Y\. Qin, S\. Liang, Y\. Ye, K\. Zhu, L\. Yan, Y\. Lu, Y\. Lin, X\. Cong, X\. Tang, B\. Qian, S\. Zhao, L\. Hong, R\. Tian, R\. Xie, J\. Zhou, M\. Gerstein, D\. Li, Z\. Liu, and M\. SunToolLLM: Facilitating Large Language Models to Master 16000\+ Real\-world APIs\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 9695–9717\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/hash/%0A28e50ee5b72e90b50e7196fde8ea260e-Abstract-Conference.html)Cited by:[Table 1](https://arxiv.org/html/2610.02793#S4.T1.2.8.3.1.1)\.
- Reinet al\.\(2024\)D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. BowmanGPQA: A Graduate\-Level Google\-Proof Q&A Benchmark\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=Ti67584b98)Cited by:[Table 1](https://arxiv.org/html/2610.02793#S4.T1.2.5.3.1.1)\.
- Sakaguchiet al\.\(2020\)K\. Sakaguchi, R\. Le Bras, C\. Bhagavatula, and Y\. ChoiWinoGrande: An Adversarial Winograd Schema Challenge at Scale\.Proceedings of the AAAI Conference on Artificial Intelligence34\(05\),pp\. 8732–8740\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v34i05.6399),[Link](https://doi.org/10.1609/aaai.v34i05.6399)Cited by:[§5\.1](https://arxiv.org/html/2610.02793#S5.SS1.SSS0.Px2.p1.1)\.
- Thedeet al\.\(2025\)L\. Thede, K\. Roth, M\. Bethge, Z\. Akata, and T\. HartvigsenWikiBigEdit: Understanding the Limits of Lifelong Knowledge Editing in LLMs\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 59326–59354\.External Links:[Link](https://proceedings.mlr.press/v267/thede25a.html)Cited by:[§2](https://arxiv.org/html/2610.02793#S2.SS0.SSS0.Px2.p1.1)\.
- Valmeekamet al\.\(2023a\)K\. Valmeekam, M\. Marquez, and S\. KambhampatiCan Large Language Models Really Improve by Self\-critiquing Their Own Plans?\.InNeurIPS 2023 Foundation Models for Decision Making Workshop,External Links:[Link](https://openreview.net/forum?id=gGQfkyb0KL)Cited by:[Table 1](https://arxiv.org/html/2610.02793#S4.T1.2.7.3.1.1)\.
- Valmeekamet al\.\(2023b\)K\. Valmeekam, M\. Marquez, A\. Olmo, S\. Sreedharan, and S\. KambhampatiPlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 38975–38987\.External Links:[Document](https://dx.doi.org/10.52202/075280-1693),[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/%0A7a92bcdede88c7afd108072faf5485c8-Abstract.html)Cited by:[Table 1](https://arxiv.org/html/2610.02793#S4.T1.2.7.3.1.1)\.
- Wallaceet al\.\(2024\)E\. Wallace, K\. Xiao, R\. Leike, L\. Weng, J\. Heidecke, and A\. BeutelThe Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions\.Note:arXiv:2404\.13208v1External Links:[Link](https://arxiv.org/abs/2404.13208v1)Cited by:[Table 1](https://arxiv.org/html/2610.02793#S4.T1.2.4.3.1.1)\.
- Wanget al\.\(2026\)Y\. Wang, S\. Feng, Y\. Tsvetkov, and H\. HajishirziScienceMeter: Tracking Scientific Knowledge Updates in Language Models\.InConference on Language Modeling,Note:Accepted at COLM 2026External Links:[Link](https://arxiv.org/abs/2505.24302)Cited by:[§2](https://arxiv.org/html/2610.02793#S2.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2023\)Y\. Wang, Y\. Kordi, S\. Mishra, A\. Liu, N\. A\. Smith, D\. Khashabi, and H\. HajishirziSelf\-Instruct: Aligning Language Models with Self\-Generated Instructions\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Toronto, Canada,pp\. 13484–13508\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.754),[Link](https://aclanthology.org/2023.acl-long.754/)Cited by:[§1](https://arxiv.org/html/2610.02793#S1.p1.1)\.
- Wanget al\.\(2024\)Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang, T\. Li, M\. Ku, K\. Wang, A\. Zhuang, R\. Fan, X\. Yue, and W\. ChenMMLU\-Pro: A More Robust and Challenging Multi\-Task Language Understanding Benchmark\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 95266–95290\.External Links:[Document](https://dx.doi.org/10.52202/079017-3018),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/%0Aad236edc564f3e3156e1b2feafb99a24-Abstract.html)Cited by:[Table 1](https://arxiv.org/html/2610.02793#S4.T1.2.5.3.1.1)\.
- Weiet al\.\(2024\)J\. Wei, K\. Nguyen, H\. W\. Chung, Y\. J\. Jiao, S\. Papay, A\. Glaese, J\. Schulman, and W\. FedusMeasuring short\-form factuality in large language models\.Note:arXiv:2411\.04368v1External Links:[Link](https://arxiv.org/abs/2411.04368v1)Cited by:[Table 1](https://arxiv.org/html/2610.02793#S4.T1.2.5.3.1.1)\.
- Xiaet al\.\(2024\)C\. S\. Xia, Y\. Deng, and L\. ZhangTop Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=zZa7Ke7WAJ)Cited by:[Table 1](https://arxiv.org/html/2610.02793#S4.T1.2.3.3.1.1)\.
- Yanget al\.\(2024\)A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. QiuQwen2\.5 technical report\.arXiv preprint arXiv:2412\.15115v2\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2412.15115),[Link](https://arxiv.org/abs/2412.15115v2)Cited by:[§A\.1](https://arxiv.org/html/2610.02793#A1.SS1.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.02793#S1.p3.1),[§4\.2](https://arxiv.org/html/2610.02793#S4.SS2.p1.1),[§5\.2](https://arxiv.org/html/2610.02793#S5.SS2.p1.1),[Table 2](https://arxiv.org/html/2610.02793#S6.T2.2.10.1.1)\.
- Yinet al\.\(2025\)X\. Yin, X\. Wang, L\. Pan, L\. Lin, X\. Wan, and W\. Y\. WangGödel Agent: A Self\-Referential Agent Framework for Recursively Self\-Improvement\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Vienna, Austria,pp\. 27890–27913\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1354),ISBN 979\-8\-89176\-251\-0,[Link](https://aclanthology.org/2025.acl-long.1354/)Cited by:[§2](https://arxiv.org/html/2610.02793#S2.SS0.SSS0.Px1.p1.1)\.
- Yuet al\.\(2025\)T\. Yu, Y\. Jing, X\. Zhang, W\. Jiang, W\. Wu, Y\. Wang, W\. Hu, B\. Du, and D\. TaoBenchmarking Reasoning Robustness in Large Language Models\.Note:arXiv:2503\.04550v1External Links:[Link](https://arxiv.org/abs/2503.04550v1)Cited by:[Table 1](https://arxiv.org/html/2610.02793#S4.T1.2.2.3.1.1)\.
- Zellerset al\.\(2019\)R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. ChoiHellaSwag: Can a Machine Really Finish Your Sentence?\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,Florence, Italy,pp\. 4791–4800\.External Links:[Document](https://dx.doi.org/10.18653/v1/P19-1472),[Link](https://aclanthology.org/P19-1472/)Cited by:[§5\.1](https://arxiv.org/html/2610.02793#S5.SS1.SSS0.Px2.p1.1)\.
- Zenget al\.\(2025\)L\. Zeng, Z\. Liu, C\. Feng, H\. Huang, and Y\. GuoDocMEdit: Towards Document\-Level Model Editing\.InFindings of the Association for Computational Linguistics: ACL 2025,Vienna, Austria,pp\. 19725–19743\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.1012),ISBN 979\-8\-89176\-256\-5,[Link](https://aclanthology.org/2025.findings-acl.1012/)Cited by:[§2](https://arxiv.org/html/2610.02793#S2.SS0.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2026a\)J\. Zhang, S\. Hu, C\. Lu, R\. Lange, and J\. CluneDarwin Gödel Machine: Open\-Ended Evolution of Self\-Improving Agents\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 104223–104294\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2026/hash/%0Aaa5f5e6eb6f613ec412f1d948dfa21a5-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2610.02793#S2.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2026b\)Q\. Zhang, C\. Hu, S\. Upasani, B\. Ma, F\. Hong, V\. Kamanuru, J\. Rainton, C\. Wu, M\. Ji, H\. Li, U\. Thakker, J\. Y\. Zou, and K\. OlukotunAgentic Context Engineering: Evolving Contexts for Self\-Improving Language Models\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 86069–86100\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2026/hash/%0A8a94ff6f922d995d7d3f4ebf4143e442-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2610.02793#S2.SS0.SSS0.Px1.p1.1)\.
- Zhouet al\.\(2025\)Z\. Zhou, S\. Liu, M\. Ning, W\. Liu, J\. Wang, D\. Wong, X\. Huang, Q\. Wang, and K\. HuangIs Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklist\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 34238–34281\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/hash/%0A54d2d38a56a74387d5916ee40e462295-Abstract-Conference.html)Cited by:[Table 1](https://arxiv.org/html/2610.02793#S4.T1.2.2.3.1.1)\.
- Zweigeret al\.\(2025\)A\. Zweiger, J\. Pari, H\. Guo, Y\. Kim, and P\. AgrawalSelf\-Adapting Language Models\.InAdvances in Neural Information Processing Systems,Vol\.38,pp\. 82334–82365\.External Links:[Document](https://dx.doi.org/10.52202/085713-2483),[Link](https://proceedings.neurips.cc/paper_files/paper/2025/hash/%0A6b41e04c41726e2a60e456d0a2b961ab-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2610.02793#S1.p1.1),[§2](https://arxiv.org/html/2610.02793#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix ACATEGORY 1:Replication Details

### A\.1Model Roles, Paper Input, and Data Isolation

#### Model roles\.

Paper parsing and knowledge\-type classification usegpt\-oss\-120b\([OpenAI, 2025](https://arxiv.org/html/2610.02793#bib.bib15)\)\. Synthetic supervision is generated withQwen2\.5\-32B\-Instruct\([Yang et al\., 2024](https://arxiv.org/html/2610.02793#bib.bib5)\)\. The target backbones are Llama\-3\.1\-8B\-Instruct and Qwen2\.5\-7B\-Instruct\. The parser uses temperature00, top\-p=1\.0p=1\.0, and 16,384/2,048\-token input/output limits\.

#### Paper input and preprocessing\.

For each paperPtP\_\{t\}, the parser extracts the paper\-reported behavior or capability failure, its trigger, an explanation and mitigation when explicitly stated, and the evaluation protocol\. The source paper may contain benchmark descriptions, perturbation rules, failure cases, or representative examples; these are part of the paper evidence\. Source\-published examples are removed from the eligible benchmark pool before the detection/held\-out split\. Human preprocessing is restricted to locating benchmark implementations, standardizing file formats, removing source\-published examples from the eligible benchmark pool, and mapping each paper to its evaluator; humans do not inspect held\-out examples when constructing supervision\.

#### Detection and held\-out partitions\.

Each benchmark is divided once using a stratified 30/70 split with seed 42:

Dt=Dtdev∪Dttest,Dtdev∩Dttest=∅\.D\_\{t\}=D\_\{t\}^\{\\mathrm\{dev\}\}\\cup D\_\{t\}^\{\\mathrm\{test\}\},\\qquad D\_\{t\}^\{\\mathrm\{dev\}\}\\cap D\_\{t\}^\{\\mathrm\{test\}\}=\\varnothing\.\(2\)Stratification prioritizes benchmark metadata such as perturbation type, variant, or position bucket when available and otherwise falls back to evaluator type\. The split is fixed across backbones, methods, candidate forms, stream orders, and training seeds\.

#### Held\-out isolation\.

DttestD\_\{t\}^\{\\mathrm\{test\}\}is inaccessible during paper parsing, failure confirmation, synthetic\-data generation and filtering, temporary LoRA training, candidate selection, full training, replay construction, and stream\-order selection\. Held\-out scores are computed only after the update decision has been finalized\. The generator receives only the extracted paper recordℛt\\mathcal\{R\}\_\{t\}and never receives raw questions, model responses, or gold answers from either benchmark partition\.

#### Contamination checks\.

Generated examples are compared with both benchmark partitions using exact MD5 matching and RapidFuzz string ratio≥97\\geq 97; the checker returns only a rejection flag and never exposes benchmark content to the generator\. Within a candidate dataset, deduplication uses exact hashes followed by MPNet embedding cosine similarity≥0\.93\\geq 0\.93\.

### A\.2Failure Confirmation and Metric Details

#### Failure confirmation\.

The current checkpoint is evaluated onDtdevD\_\{t\}^\{\\mathrm\{dev\}\}using the paper\-defined comparison\. For matched original/perturbed conditions,

Δt=Score⁡\(Mt−1,\{xi\}\)−Score⁡\(Mt−1,\{x~i\}\)\.\\Delta\_\{t\}=\\mathrm\{Score\}\(M\_\{t\-1\},\\\{x\_\{i\}\\\}\)\-\\mathrm\{Score\}\(M\_\{t\-1\},\\\{\\tilde\{x\}\_\{i\}\\\}\)\.\(3\)We bootstrap the paper\-defined evaluation units to obtain a 95% confidence interval and confirm the paper\-defined failure only when this interval lies entirely above zero\. Otherwise, no update is attempted\.

#### Bootstrap implementation\.

We useB=2000B=2000bootstrap replicates with random seed 42\. The interval is a 95% percentile interval: after sorting the 2000 bootstrap means, we use the 50th and 1950th order statistics as the lower and upper endpoints\. For paired comparisons, one bootstrap unit is100×\(base−perturbed\)100\\times\(\\mathrm\{base\}\-\\mathrm\{perturbed\}\)for one matched pair\. For unpaired accuracy\-style evaluations, one unit is100×\(1−correct\)100\\times\(1\-\\mathrm\{correct\}\)for one example\. Resampling is performed over these paper\-defined units\.

#### Retention implementation\.

For paperii, letgi=Si​\(Mi\)−Si​\(Mi−1\)g\_\{i\}=S\_\{i\}\(M\_\{i\}\)\-S\_\{i\}\(M\_\{i\-1\}\)\. Retention is computed only for steps withgi\>0g\_\{i\}\>0:

Ri=Si​\(MT\)−Si​\(Mi−1\)gi\+10−6\.R\_\{i\}=\\frac\{S\_\{i\}\(M\_\{T\}\)\-S\_\{i\}\(M\_\{i\-1\}\)\}\{g\_\{i\}\+10^\{\-6\}\}\.\(4\)The ratio is not clipped\. Steps withgi≤0g\_\{i\}\\leq 0are excluded from the retention average, matching the main\-text definition\.

### A\.3Training, Compute, and Baseline Adaptation

#### Candidate generation\.

For each confirmed paper,Paper2LLM\+\+constructs the applicable subset of four candidate forms: direct, contrastive, robustness, and process supervision\. Qwen2\.5\-32B\-Instruct uses temperature0\.70\.7and top\-p=0\.95p=0\.95\. The standard generation target is 512 examples and the compute\-adjusted target is 320; generation first produces1\.5×1\.5\\timesthe target count to support filtering\. A candidate is invalidated if fewer than 128 examples survive\. The update budget itself uses 128 accepted new examples together with 64 replay examples, so the larger generation pool does not increase the number of new examples seen by an update\.

#### Temporary and full updates\.

Each applicable candidate is trained from the same current checkpoint using a disposable LoRA adapter and the same replay budget\. Candidate utility is

Ut\(k\)=Gt\(k\)−λ​Pt\(k\)−μ​Qt\(k\),U\_\{t\}^\{\(k\)\}=G\_\{t\}^\{\(k\)\}\-\\lambda P\_\{t\}^\{\(k\)\}\-\\mu Q\_\{t\}^\{\(k\)\},\(5\)subject to the globally fixed acceptance constraints

Gt\(k\)≥τgain,Pt\(k\)≤τprev,Qt\(k\)≤τgen\.\\begin\{gathered\}G\_\{t\}^\{\(k\)\}\\geq\\tau\_\{\\mathrm\{gain\}\},\\qquad P\_\{t\}^\{\(k\)\}\\leq\\tau\_\{\\mathrm\{prev\}\},\\\\ Q\_\{t\}^\{\(k\)\}\\leq\\tau\_\{\\mathrm\{gen\}\}\.\\end\{gathered\}\(6\)The selected supervision is retrained fromMt−1M\_\{t\-1\}with the full update budget\.

Table 3:Current training configuration used by the final pipeline\. Sequence lengths are 2,048 for short\-form tasks, 4,096 for code and structured tasks, and 8,192 for long\-context tasks\.The configuration in Table[3](https://arxiv.org/html/2610.02793#A1.T3)replaces the earlier development settings: the final implementation uses 60 temporary steps, a10−410^\{\-4\}learning rate for both temporary and full updates, effective batch size 64, and bfloat16 without quantization\.

#### Replay and protected probes\.

After a successful commit, 64 replay examples are mixed into each future update\. The replay buffer has capacity 600 and preserves a balanced mixture of 60% prior paper\-derived supervision, 20% matched clean\-domain examples, and 20% general\-domain examples\. Up to 50 examples from the currentDtdevD\_\{t\}^\{\\mathrm\{dev\}\}are frozen as a protected probe, sampled without replacement using seed 42 and stratified by subtype or perturbation category when available\. Protected probes are never used for training or synthetic\-data generation\.

#### General audit and reporting sets\.

Candidate acceptance uses a fixed general\-audit setDaudD^\{\\mathrm\{aud\}\}that is separate from the final reporting suite\. Final general\-benchmark results are macro\-averaged over fixed 500\-example subsets of HellaSwag, WinoGrande, PIQA, BoolQ, and ARC\-Challenge \(2,500 examples total\); these reporting subsets are not used for training or update acceptance\.

#### Continual runs\.

The mixed experiment uses three fixed 30\-paper permutations generated with order seeds 101, 202, and 303, paired with training seeds 13, 21, and 42, respectively\. Every method uses the same three order–seed pairs and no order is selected or removed after observing results\. Same\-family experiments use three fixed permutations for each of the coding, structured\-reasoning, and agentic\-reliability streams\.

#### Per\-paper optimization accounting\.

Table[4](https://arxiv.org/html/2610.02793#A1.T4)reports online optimization work performed per paper by the current implementation\. Counts cover optimization during the paper stream; one\-time editor or hypernetwork pre\-training is outside this accounting\.

Table 4:Per\-paper online optimization accounting\. The shared paper stream is not compute matched:Paper2LLM\+\+can spend additional optimization on candidate probing, whereas several editing baselines perform no per\-paper gradient optimization\.
#### Baseline adaptation under the shared paper stream\.

All methods process the same paper order and are evaluated with the same held\-out procedure\. Methods that consume example\-level supervision receive a fixed paper\-derived setKtK\_\{t\}ofN=128N=128input–target pairs\. Detection and held\-out benchmark examples are never supplied to a baseline\. Method\-specific pre\-training is completed before the paper stream and is accounted for separately from per\-paper updates\.

Table 5:Adaptation of continual\-learning and lifelong\-editing baselines to the common paper stream\. The interface standardizes paper\-derived input exposure while preserving each method’s native update mechanism; it does not equalize optimization compute\.

### A\.4Analysis Annotation Protocols

The analysis\-only annotations are created before inspecting held\-out outcomes and are never supplied toPaper2LLM\+\+as update features\.

#### Knowledge type\.

Before inspecting held\-out outcomes, gpt\-oss\-120b assigns each paper the primary knowledge or behavior that must be installed to address the reported failure, rather than the benchmark domain\. Fine\-grained labels include reasoning patterns, boundaries, constraints, verification strategies, retrieval or memory strategies, planning or interaction strategies, algorithmic procedures, knowledge updates, execution feedback, and safety policies\.

#### Supervision closure\.

Closure is a continuous descriptive annotation of whether a reported failure can be converted into executable and verifiable supervision\. Five components are scored continuously in\[0,1\]\[0,1\]: explicit target behavior, constructible supervision, automatic verification, local evaluability, and mechanism sufficiency\. A score of00means the property is absent or cannot be operationalized from the paper,0\.50\.5indicates partial or conditional support, and11means it is explicitly and directly satisfied; intermediate values reflect evidence between these anchors\. The aggregate is

Ct=ctarget\+csup\+cver\+clocal\+cmech5\.C\_\{t\}=\\frac\{c\_\{\\mathrm\{target\}\}\+c\_\{\\mathrm\{sup\}\}\+c\_\{\\mathrm\{ver\}\}\+c\_\{\\mathrm\{local\}\}\+c\_\{\\mathrm\{mech\}\}\}\{5\}\.\(7\)These annotations are not supplied toPaper2LLM\+\+as update features\. For Figure[3](https://arxiv.org/html/2610.02793#S6.F3)\(b\), semantically related fine\-grained labels are merged into the broader knowledge types used in panel \(a\), and the reported Pearsonr=0\.87r=0\.87is computed from the grouped, unrounded analysis values\. The Category 2 table reports raw paper\-level annotations and is not the direct unit of this correlation\.

#### Semantic and behavioral novelty\.

Withϕt=Embed⁡\(FailureSchema⁡\(Pt\)\)\\phi\_\{t\}=\\mathrm\{Embed\}\(\\mathrm\{FailureSchema\}\(P\_\{t\}\)\), semantic novelty is

Ntsem=1−maxi<t⁡cos⁡\(ϕt,ϕi\),N1sem=1\.N\_\{t\}^\{\\mathrm\{sem\}\}=1\-\\max\_\{i<t\}\\cos\(\\phi\_\{t\},\\phi\_\{i\}\),\\qquad N\_\{1\}^\{\\mathrm\{sem\}\}=1\.\(8\)Ifδt\\delta\_\{t\}denotes the model\-specific paper\-defined failure severity, behavioral novelty is

Ntbeh=δt−δminδmax−δmin\.N\_\{t\}^\{\\mathrm\{beh\}\}=\\frac\{\\delta\_\{t\}\-\\delta\_\{\\min\}\}\{\\delta\_\{\\max\}\-\\delta\_\{\\min\}\}\.\(9\)Both quantities are used only for descriptive analysis and are not inputs to the update procedure\.

### A\.5Evidence\-Grounded Extraction Example

WhatSummary:Models that correctly solve GSM8K seed questions can fail on minimally perturbed versions, with a gap up to 20%\.Evidence:“Our benchmark revealed a gap of up to 20% between the accuracy reported by the current model and the accuracy observed in our setting, while human performance remains unaffected due to the unchanged inherent difficulty level of the questions\.” \(pp\. 1–2\)

TriggerSummary:Failures are triggered by missing information, additional operations, reversed reasoning, and distractors; adding an operation causes a drop above 40% for most models except GPT\-4 and GPT\-3\.5\-Turbo\.Evidence:“Except GPT\-4 and GPT\-3\.5\-Turbo, most models, regardless of size and output format, experience an accuracy drop exceeding 40%\.” \(pp\. 6–7\)

Failure patternSummary:Errors include operator misuse, omitted conditions, and incorrect use of irrelevant information\.Evidence:“Example C\.3 shows an example of failure for GPT\-3\.5\-Turbo, where it fails to consider one condition …”; the condition “should be considered after step \[1\] and before step \[2\]\.” \(pp\. 15–18\)

WhySummary:Models can lose intermediate constraints or make mistakes in multi\-step reasoning\.Evidence:“We find that LLMs often overlook steps or make errors in the middle stages of the solution\.” \(pp\. 8–9\)

MitigationSummary:Chain\-of\-thought, least\-to\-most, program\-of\-thought, and compositional prompting help but do not eliminate the gap\.Evidence:“Although compositional prompting improves performance on seed questions and their variations, it does not bridge the performance gap …” \(pp\. 8–9\)

EvaluationSummary:The benchmark evaluates 25 LLMs under four prompting methods and compares solved GSM8K seeds with GSM\-Plus variations\.Evidence:“Our experiments on 25 LLMs and 4 prompting techniques show that … their performances are far from robust\.” \(p\. 1\)

Data qualitySummary:Human validation remains necessary; annotators revised 18\.85% of GPT\-4\-generated variants\.Evidence:“Overall, human annotators revised 18\.85% of the variations produced by GPT\-4, highlighting the importance of human revision\.” \(p\. 4\)

This example shows that each summarized field is tied to a concrete source span rather than inferred from held\-out benchmark outcomes\.

## Appendix BCATEGORY 2:Complementary Results

Unless otherwise noted, model\-dependent values in this section are averaged over Llama\-3\.1\-8B\-Instruct and Qwen2\.5\-7B\-Instruct\.

Table 6:Paper\-level analysis values \(1/2\)\. Together with Table[B](https://arxiv.org/html/2610.02793#A2), this exposes all 30 raw annotations used by the analysis\. Gain is the individual\-paper held\-out gain; Ret\. is paper\-level continual retention\. Cl\.=closure; S/P/V/I denote search/planner/verifier/interaction\.

Table 7:Paper\-level analysis values \(2/2\)\. Values are rounded for compact presentation; the reported correlations are computed from the released unrounded analysis data\.

The internal selection\-loop ablations in Tables[B](https://arxiv.org/html/2610.02793#A2)–[B](https://arxiv.org/html/2610.02793#A2)average results over both target backbones\. In these tables, Eval is the mean held\-out target score immediately after each paper’s update decision; CleanΔ\\Deltaand Gen\.Δ\\Deltaare final changes relative to the base model on the matched clean\-domain and five\-benchmark reporting suites, respectively\.

Table 8:Internal selection\-loop ablation on coding\. It isolates selection behavior inside the pipeline and is not the finalPaper2LLM\+\+configuration reported in the main comparison table\.

Table 9:Internal selection\-loop ablation on structured reasoning\. It separates raw target\-gain selection from regression\-aware selection\.

Table 10:Internal selection\-loop ablation on agentic reliability\. It shows why maximizing immediate gain alone can sacrifice later retention\.

Table 11:Internal selection\-loop ablation on the mixed 30\-paper stream\. These numbers diagnose the internal selection design rather than provide alternate finalPaper2LLM\+\+results\.

Table 12:Independent paper\-driven adaptation by failure family, providing the numerical values underlying Figure 2\(a\)\. Values are averaged over Llama\-3\.1\-8B\-Instruct and Qwen2\.5\-7B\-Instruct\. Gain is averaged from unrounded paper\-level differences, so subtracting the separately rounded Before and After entries can differ by 0\.1 point\.