Unsupervised Post-Training of Foundation Models: A Survey

# Unsupervised Post-Training of Foundation Models: A Survey
Source: [https://arxiv.org/html/2608.24982](https://arxiv.org/html/2608.24982)
Xingbo Yao Zhiyu Guo Aiwei Liu Xuming Hu Weiyu Guo Hui XiongAffiliation:HKUST\(GZ\)Affiliation:HKUSTAffiliation:Xiaohongshu Inc\.Affiliation:WeChat, TencentAffiliation:CUHKAffiliation:AI Robotics†Corresponding authors\.

###### Abstract

Foundation\-model post\-training usually relies on human labels, preference data, stronger teachers, or executable verifiers\. We study*Unsupervised Post\-Training*\(UPT\): update\-bearing adaptation on unlabeled inputs whose learning signal is derived from same\-lineage model artifacts rather than an external oracle\. We catalog 80 strict UPT methods and organize them by the object that supplies the update signal: a prediction statistic, a sample relation, a self\-generated target, or an internal evaluator\. Beyond inventory, we show how the choice of internal signal and task structure determines whether post\-training improves the model or recursively amplifies error\. An orthogonal Input Visibility×\\timesUpdate Persistence view maps deployment regimes and defines a unified framework for UPT selection and evaluation\.

## 1Introduction

Foundation\-model post\-training has so far followed two waves of external supervision\.111We use*foundation models*for both text\-only and multimodal large models; a recurring finding is that the dominant unsupervised mechanisms are modality\-agnostic\.The first wave is*human labels*, including supervised fine\-tuning and preference\-based RL\. The second is*external verifiers*: math checkers, unit tests, and executable environments that license RL with verifiable rewards\([Shao et al\., 2024](https://arxiv.org/html/2608.24982#bib.bib45);[Zhao et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib90);[Liao et al\., 2026](https://arxiv.org/html/2608.24982#bib.bib33)\)\. A third wave, accumulating since 2023 and accelerating through 2025–2026, abandons external supervision and updates the model using*only*unlabeled prompts, text, or target inputs, drawing every update signal from the model’s own samples, distributions, judges, or curricula\([Huang et al\., 2023](https://arxiv.org/html/2608.24982#bib.bib22);[Yuan et al\., 2024](https://arxiv.org/html/2608.24982#bib.bib80);[Zuo et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib95);[Huang et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib24)\)\.

What internal update object is usedto compute the update signal?Prediction\-StatisticOptimizationNLL, entropy, confidencee\.g\.,CPT\-LM,EM\-RLSample\-RelationSupervisionmajority vote, semantic clusterse\.g\.,TTRL,EMPOSelf\-Generated TargetBootstrappingpseudo\-labels, rationales, curriculae\.g\.,Self\-Improve,R\-ZeroInternal EvaluatorBootstrappingself\-judge, reward model, rubrice\.g\.,Self\-Rewarding LM,CoNLAdjacent Methods \(boundary\)no\-update inference\-time; verifier/tool\-assisted; human/seed\-supervised; stronger\-teacher; external\-evaluatore\.g\.,EM\-INF,T3RL,Absolute Zero

Figure 1:Update\-object taxonomy of*Unsupervised Post\-Training*\. Strict UPT methods are grouped by the internal update object\. Dashed boxes denote boundary\-adjacent methods that fail at least one strict UPT check\.UPT enables adaptation when labels and task\-specific verifiers cannot be obtained or transferred\. This setting covers newly arrived domain corpora and open\-ended generation tasks such as dialogue and summarization, where exact\-answer checkers are unavailable\.

We use*UPT*for update\-bearing adaptation whose learning signal comes from same\-lineage model artifacts; §[2](https://arxiv.org/html/2608.24982#S2)operationalizes this scope with four boundary checks\. Existing surveys cover neighboring territory: LLM self\-improvement\([Tao et al\., 2024](https://arxiv.org/html/2608.24982#bib.bib54);[Kumar et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib29)\), test\-time adaptation\([Liang et al\., 2023](https://arxiv.org/html/2608.24982#bib.bib32)\), and reinforced reasoning\([Xu et al\., 2025b](https://arxiv.org/html/2608.24982#bib.bib74);[Chen et al\., 2025b](https://arxiv.org/html/2608.24982#bib.bib7)\)\. None of them, however, enforces the three constraints that define our scope: a real update, no external supervision, and a classification axis defined by the internal object that produces the update signal\. Appendix[E](https://arxiv.org/html/2608.24982#A5)provides the dimension\-wise comparison\.

We organize UPT by the internal object consumed by the update: a prediction statistic, a sample relation, a self\-generated target, or an internal evaluator\. Figure[1](https://arxiv.org/html/2608.24982#S1.F1)shows the four resulting families: Prediction\-Statistic Optimization, Sample\-Relation Supervision, Self\-Generated Target Bootstrapping, and Internal Evaluator Bootstrapping\. A parallel adjacent track maps methods that introduce verifier, tool, seed, teacher, or external\-evaluator signals\. Together, the four families expose a common error chain: an imperfect proxy selects or rewards outputs, the update concentrates the model on them, and the next round reads an even more biased proxy\. The family determines where this loop begins and which safeguard can interrupt it\.

Our contributions are: \(i\) an operational boundary protocol that distinguishes signal provenance from task structure \(§[2](https://arxiv.org/html/2608.24982#S2), Appendix[A](https://arxiv.org/html/2608.24982#A1)\); \(ii\) an update\-object taxonomy of four families and an orthogonal Input Visibility×\\timesUpdate Persistence view \(§[3](https://arxiv.org/html/2608.24982#S3)–§[8](https://arxiv.org/html/2608.24982#S8)\), with the hierarchy and inventory cross\-sections in Appendix[C](https://arxiv.org/html/2608.24982#A3); and \(iii\) a cross\-family synthesis of applicability, error propagation, and deployment checks, supported by empirical and task\-structure audits \(§[7](https://arxiv.org/html/2608.24982#S7)–§[9](https://arxiv.org/html/2608.24982#S9), Appendix[B](https://arxiv.org/html/2608.24982#A2)\)\.222Companion[inventory](https://github.com/yeahjack/awesome-unsupervised-post-training)\.

## 2Scope, Survey Protocol, and Definitions

#### Survey protocol\.

We cover text\-only and multimodal foundation\-model methods from January 2023 to May 2026 that perform a post\-pretraining update on unlabeled prompts, text, or target inputs\. Seed\-and\-snowball search covered continued pretraining, test\-time adaptation, internal consensus, self\-training, self\-rewarding, and multimodal UPT across ACL Anthology, arXiv, Semantic Scholar, and Google Scholar\. The frozen inventory contains 94 method records from 91 papers: 80 strict rows from 78 papers, 8 adjacent rows, and 6 prose\-only boundary or antecedent records\. A paper can contribute more than one method record\. Appendix[A](https://arxiv.org/html/2608.24982#A1)gives database\-specific query templates, criteria, and counts\.

#### Definition\.

We define*Unsupervised Post\-Training*\(UPT\) as any procedure that \(a\) begins from a finetuned foundation model, \(b\) uses unlabeled prompts, text, or target inputs, \(c\) modifies model parameters, adapters, memories, or persistent local state, and \(d\) computes the update signal without external supervision\. External supervision includes ground\-truth answers, verifier feedback, executable or tool verdicts, human labels, and labels from a stronger teacher\.

#### Update objects\.

We organize UPT with an*update\-object taxonomy*\. An internal update object is the model\-derived object that produces the update signal: a prediction statistic, a relation among model samples, a self\-generated target, or an internal evaluator\. The taxonomy is orthogonal to optimizer, task, modality, and training schedule\.

Table 1:Strict UPT methods in Family I \(§[3](https://arxiv.org/html/2608.24982#S3)\)\.*Signal*: NLL, entropy \(Ent\.\), self\-confidence \(Conf\.\), or other geometric/rule\-based statistic \(Geom\./Rule\)\.*Mechanism*: continued pretraining \(CPT\), test\-time training \(TTT\), entropy/confidence minimization \(EM\-min\), entropy/confidence or other internal\-statistic policy\-gradient reward \(EM\-RL\), sample\-local state update \(State\)\.*LC*: long\-context target\.‡Family I/IV bridge case \(§[6](https://arxiv.org/html/2608.24982#S6)\)\.
#### Boundary checks and update\-object rule\.

A method is*strict UPT*if all four checks hold:

- B1\.It performs an explicit update to parameters, adapters, memories, or persistent local state\.
- B2\.The update signal is only from unlabeled inputs & same\-lineage samples or judgments\.
- B3\.No external supervision enters the update\.
- B4\.Any judge, scorer, or reward model used in the update derives from the same model lineage\.

Family assignment follows the object consumed by the gradient\. A multi\-sample majority statistic used as a reward belongs to Sample\-Relation Supervision, whereas one used to build pseudo\-labels, curricula, or preference pairs belongs to Self\-Generated Target Bootstrapping\. The four checks rule out*explicit*external supervision; the separate audit below records structural properties of the task and evaluation protocol\.

#### Task\-structure audit\.

We separately record whether a result relies on an answer extractor or canonicalizer, a finite answer alphabet, a code signature, full\-cohort transductive access, or an open\-ended output space\. These features define equivalence classes that make model samples easier to group and compare; correctness\-bearing executions are treated as external verdicts under[\(B3\)](https://arxiv.org/html/2608.24982#S2.SS0.SSS0.Px4)\. Tableseparates these structural priors from correctness\-bearing supervision in the representative\-evidence audit presented in Appendix[B](https://arxiv.org/html/2608.24982#A2)\.

#### Adjacent methods\.

The adjacent track retains five neighboring paradigms for comparison: no\-update inference\-time optimization; verifier\- or tool\-assisted self\-training; human\- or seed\-supervised bootstrapping; stronger\-teacher or cross\-model distillation; and external reward or evaluator methods\([Liao et al\., 2026](https://arxiv.org/html/2608.24982#bib.bib33);[Zhao et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib90);[Wang et al\., 2023](https://arxiv.org/html/2608.24982#bib.bib57);[Li et al\., 2023](https://arxiv.org/html/2608.24982#bib.bib30)\)\. This parallel organization preserves a precise internal\-signal core while retaining the broader design space\.

#### Formal setup\.

Letfθf\_\{\\theta\}be a foundation model and let𝒟x\\mathcal\{D\}\_\{x\}be a distribution of unlabeled prompts, text, or target inputs\.θ\\thetadenotes the updatable state, including parameters, adapters, memories, or persistent local state\. A UPT procedure has three components\. First, a sampling or aggregation operator𝒮\\mathcal\{S\}produces an internal update objectz=𝒮⁡\(fθ,x\),x∼𝒟x,z=\\mathcal\{S\}\(f\_\{\\theta\},x\),x\\sim\\mathcal\{D\}\_\{x\},such as predictive distribution, a set of rollouts, candidate answers, preference pairs, generated targets, or judge verdicts\. Second, a signal extractorσ\\sigmamaps this object to an update signals=σ⁡\(z\),s=\\sigma\(z\),which may be scalar, pairwise, token or sequence\-level\. Third, an update operator𝒰\\mathcal\{U\}maps these to an updated state:

θ′=𝒰⁡\(θ,x,z,s\)\.\\theta^\{\\prime\}=\\mathcal\{U\}\(\\theta;x,z,s\)\.We callzz, notσ∘𝒮\\sigma\\\!\\circ\\\!\\mathcal\{S\}, the*internal update object*\. The update\-object taxonomy classifies a method by the type ofzzused to compute the update signal\.

Consensus signalRegimeMethodMajoritySemanticSelf\-PathPairwiseSoftTrainTestMultimodalvoteclustercertaintyconsist\.agreementclusterEMPO\([Zhang et al\., 2025b](https://arxiv.org/html/2608.24982#bib.bib86)\)✓✓Intuitor\([Zhao et al\., 2026b](https://arxiv.org/html/2608.24982#bib.bib91)\)✓✓CoVo\([Zhang et al\., 2025a](https://arxiv.org/html/2608.24982#bib.bib84)\)✓✓Co\-rewarding\([Zhang et al\., 2026b](https://arxiv.org/html/2608.24982#bib.bib88)\)✓✓EvoLMM\([Thawakar et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib55)\)✓✓✓TTRL\([Zuo et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib95)\)✓✓ETTRL\([Liu et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib36)\)✓✓ECHO\([Zhao et al\., 2026a](https://arxiv.org/html/2608.24982#bib.bib92)\)✓†✓SPINE\([Wu et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib68)\)✓†✓Self\-Harmony\([Wang et al\., 2026b](https://arxiv.org/html/2608.24982#bib.bib62)\)✓✓DARE\([Du et al\., 2026a](https://arxiv.org/html/2608.24982#bib.bib9)\)✓✓SCOPE\([Wang et al\., 2025a](https://arxiv.org/html/2608.24982#bib.bib61)\)✓✓COMPASS\([Xing et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib71)\)✓✓SCRL\([Yan et al\., 2026](https://arxiv.org/html/2608.24982#bib.bib77)\)✓✓RLCCF\([Yuan et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib81)\)✓✓RoiRL\([Arzhantsev et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib5)\)✓✓EVOL\-RL\([Zhou et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib94)\)✓✓TTRV\([Singh et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib49)\)✓✓✓MM\-UPT\([Wei et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib64)\)✓✓✓Dual Consensus\([Du et al\., 2026b](https://arxiv.org/html/2608.24982#bib.bib10)\)✓✓CSRS\([Yu et al\., 2026](https://arxiv.org/html/2608.24982#bib.bib79)\)✓✓✓EvoQuality\([Wen et al\., 2026a](https://arxiv.org/html/2608.24982#bib.bib65)\)✓✓✓Table 2:Strict UPT methods in Family II \(§[4](https://arxiv.org/html/2608.24982#S4)\)\.*Consensus signal*: which multi\-sample statistic over multiple internal samples drives the gradient\.†marks methods where the consensus is the*selection*mask and a separate intrinsic term shapes advantages or tokens\.

## 3Prediction\-Statistic Optimization

The first family is direct:σ\\sigmareads a scalar fromfθf\_\{\\theta\}at a single observation, and𝒰\\mathcal\{U\}optimizes that scalar\. This makes it the historical baseline, with continued pretraining and test\-time training predating the other three families\. By construction, it generates no pseudo\-labels, constructs no preference pairs, and trains no internal judge: the update object is the predictive quantity \(token NLL, sequence likelihood, entropy, confidence\) or an analogous geometric or rule\-based statistic\. Table[1](https://arxiv.org/html/2608.24982#S2.T1)catalogs 26 strict UPT methods along Signal×\\timesMechanism\.

#### Predictive likelihood minimization\.

The basic instantiation is CPT: minimizing LM loss on unlabeled text\. CPT recipes\([Ke et al\., 2023](https://arxiv.org/html/2608.24982#bib.bib28);[Ibrahim et al\., 2024](https://arxiv.org/html/2608.24982#bib.bib42)\)need no annotation when domain shift is the bottleneck; refinements address failure modes\([Guo et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib17);[Abbes et al\., 2026](https://arxiv.org/html/2608.24982#bib.bib1);[Elhady et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib11)\), and a parallel sub\-line scales context windows by corpus engineering alone\([Liu et al\., 2024](https://arxiv.org/html/2608.24982#bib.bib34);[Fu et al\., 2024](https://arxiv.org/html/2608.24982#bib.bib13);[Xiong et al\., 2024](https://arxiv.org/html/2608.24982#bib.bib72)\)\. The NLL objective also reaches into TTA:TLM\([Hu et al\., 2025a](https://arxiv.org/html/2608.24982#bib.bib21)\)treats the test stream as a CPT corpus;TTT\-NN\([Hardt and Sun, 2024](https://arxiv.org/html/2608.24982#bib.bib25)\)restricts each update to a retrieved neighborhood;Long TTT\([Bansal et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib6)\)extends it to long context;In\-Place TTT\([Feng et al\., 2026](https://arxiv.org/html/2608.24982#bib.bib12)\)interleaves updates with generation\.

#### Entropy and confidence objectives\.

A second sub\-line generalizes the read\-out to*predictive confidence*, exploiting that high\-quality reasoning traces are low\-entropy\. Entropy minimization\([Agarwal et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib2);[Gao et al\., 2025b](https://arxiv.org/html/2608.24982#bib.bib15)\)treats per\-token entropy as a training loss or single\-prompt gradient\. The statistics wrap into policy\-optimization loops:EM\-RL\([Agarwal et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib2)\)recasts entropy as intrinsic reward in GRPO;RENT\([Prabhudesai et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib40)\)andRLSC\([Li et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib31)\)reward self\-confidence with no answer key\.[Zhang et al\. \(2025d\)](https://arxiv.org/html/2608.24982#bib.bib85)show that initialization and update duration shape entropy\- and confidence\-based optimization, motivating stratified evaluation in §[9](https://arxiv.org/html/2608.24982#S9)\.

#### Beyond entropy and confidence\.

A recent line broadens Family I to richer internal statistics \(*Geom\./Rule*column of Table[1](https://arxiv.org/html/2608.24982#S2.T1)\)\.VIGOR\([Wen et al\., 2026b](https://arxiv.org/html/2608.24982#bib.bib66)\)uses policy’s teacher\-forced gradient norm as intrinsic GRPO reward;Latent\-GRPO\([Zhang et al\., 2026a](https://arxiv.org/html/2608.24982#bib.bib89)\)replaces external judges with terminal hidden\-state geometry\. In multimodal settings,SSL\-R1\([Xie et al\., 2026](https://arxiv.org/html/2608.24982#bib.bib70)\)derives rewards from visual self\-supervised puzzles, andSUDER\([Hong et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib19)\)uses reverse\-task likelihood as a prediction statistic bridging Families I and IV \(§[6](https://arxiv.org/html/2608.24982#S6)\)\.

#### Sample\-local state update\.

A third sub\-line moves the update target from full parameters to a small, sample\-local state: a per\-prompt optimization vector\([Hu et al\., 2025b](https://arxiv.org/html/2608.24982#bib.bib20)\), a test\-time LoRA\([Xu et al\., 2025c](https://arxiv.org/html/2608.24982#bib.bib75)\), a steering vector\([Kang et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib27)\), or layer\-wise dynamic adaptation\([Xu et al\., 2026](https://arxiv.org/html/2608.24982#bib.bib76)\)\. These methods optimize the same prediction\-statistic objectives but with a smaller update footprint and shorter persistence; we trace this timing axis explicitly in §[8](https://arxiv.org/html/2608.24982#S8)\.

Generated targetSelection criterionRegimeMethodInstr\.RationaleCurric\.Pref\.Doc\.KBCycleSCConfid\.Maj\.DebateMBRForesightReflectTrainTestSelf\-Tuning†\([Zhang et al\., 2025c](https://arxiv.org/html/2608.24982#bib.bib87)\)✓✓✓KBAlign\([Zeng et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib83)\)✓✓✓CYCLE\-INSTRUCT†\([Shen et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib46)\)✓✓✓Self\-Improve\([Huang et al\., 2023](https://arxiv.org/html/2608.24982#bib.bib22)\)✓✓✓Quiet\-STaR\([Zelikman et al\., 2024](https://arxiv.org/html/2608.24982#bib.bib82)\)✓✓Confident ST\([Jang et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib60)\)✓✓✓GENIUS\([Xu et al\., 2025a](https://arxiv.org/html/2608.24982#bib.bib73)\)✓✓✓LRM Self\-Train\([Shafayat et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib47)\)✓✓✓DTE\([Srivastava et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib35)\)✓✓✓LongMagpie\([Gao et al\., 2025a](https://arxiv.org/html/2608.24982#bib.bib14)\)✓✓Long Self\-Improve\([Li et al\., 2024](https://arxiv.org/html/2608.24982#bib.bib59)\)✓✓✓TTCS\([Yang et al\., 2026a](https://arxiv.org/html/2608.24982#bib.bib78)\)✓✓✓DiSCTT\([Moradi and Mudur, 2026](https://arxiv.org/html/2608.24982#bib.bib38)\)✓✓✓TTSR\([He et al\., 2026](https://arxiv.org/html/2608.24982#bib.bib18)\)✓✓✓R\-Zero\([Huang et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib24)\)✓✓✓ScPO\([Prasad et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib41)\)✓✓✓MACA\([Samanta et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib44)\)✓✓✓LongPO\([Chen et al\., 2025a](https://arxiv.org/html/2608.24982#bib.bib8)\)✓✓RLSF\([van Niekerk et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib56)\)✓✓✓G\-Zero\([Huang et al\., 2026](https://arxiv.org/html/2608.24982#bib.bib23)\)✓✓✓QueST\([Song et al\., 2026](https://arxiv.org/html/2608.24982#bib.bib50)\)✓✓✓V\-Zero\([Wang et al\., 2026a](https://arxiv.org/html/2608.24982#bib.bib63)\)✓✓✓Table 3:Strict UPT methods in Family III \(§[5](https://arxiv.org/html/2608.24982#S5)\)\.*Generated target*: instruction/response \(Instr\.\), rationale, curriculum \(Curric\.\), or preference pair \(Pref\.\)\.*Selection criterion*: document grounding \(Doc\.\), knowledge\-base grounding \(KB\), cycle consistency, self\-consistency \(SC\), internal confidence \(Confid\.\), majority vote \(Maj\.\), multi\-agent debate, minimum\-Bayes risk \(MBR\), stepwise foresight \(Foresight\), self\-reflection \(Reflect\)\.†marks a stage\- or variant\-qualified strict entry\.

## 4Sample\-Relation Supervision

The second family changes whatσ\\sigmacomputes\. Methods in Family I read a scalar from the model’s distribution at a single observation; Family II reads a*relation*across multiple internal update objects, such as rollouts, paraphrases, candidate answers, or multiple agents\. The internal update object is a*multi\-sample statistic*: a cluster mass, a consistency score, a vote, or a contrastive agreement\. Table[2](https://arxiv.org/html/2608.24982#S2.T2)shows that 13/22 methods reduce the relation to a binary majority vote; semantic\-cluster, self\-certainty, pairwise\-agreement, and softened\-frequency variants populate the long tail\.

### 4\.1Self\-Consistency Within a Single Prompt

The simplest relation operates among multiple samples drawn from the same prompt\.EMPO\([Zhang et al\., 2025b](https://arxiv.org/html/2608.24982#bib.bib86)\)clusters rollouts and uses cluster mass or semantic entropy as the reward\.Intuitor\([Zhao et al\., 2026b](https://arxiv.org/html/2608.24982#bib.bib91)\)replaces external rewards with*self\-certainty*, the forward KL to a uniform distribution\.CoVo\([Zhang et al\., 2025a](https://arxiv.org/html/2608.24982#bib.bib84)\)combines path consistency and volatility, whileCo\-rewarding\([Zhang et al\., 2026b](https://arxiv.org/html/2608.24982#bib.bib88)\)co\-evolves a pair of networks and reads their contrastive agreement as the reward, which sidesteps the reward hacking that arises when one network grades itself\.

### 4\.2Consensus and Test\-Time RL

A larger sub\-line scales the relation across many samples and feeds it back into a policy\-optimization loop\. The canonical instance isTTRL\([Zuo et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib95)\)\(Table[2](https://arxiv.org/html/2608.24982#S2.T2), majority\-vote rows\): for each test prompt, it drawsNNrollouts, takes the majority answer as a pseudo\-label, and runs a GRPO step against it\. Refinements form a coherent line: entropy\-regularized exploration\([Liu et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib36)\); paraphrase consistency and self\-play\([Wang et al\., 2026b](https://arxiv.org/html/2608.24982#bib.bib62)\); soft rewards from rollout statistics\([Du et al\., 2026a](https://arxiv.org/html/2608.24982#bib.bib9)\); consensus across model populations\([Yuan et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib81)\); entropy\-shaped advantages\([Zhao et al\., 2026a](https://arxiv.org/html/2608.24982#bib.bib92)\); and entropy\-band token masks\([Wu et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib68)\)\. The same recipe also runs at training time on unlabeled prompts\([Wang et al\., 2025a](https://arxiv.org/html/2608.24982#bib.bib61);[Xing et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib71);[Yan et al\., 2026](https://arxiv.org/html/2608.24982#bib.bib77)\)\.EVOL\-RL\([Zhou et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib94)\)isolates semantic novelty as a mechanism for preserving rollout diversity under majority\-based optimization, exposing a broader diversity–selection trade\-off for sample\-relation methods\.

Follow\-up works generalize the relation beyond a single hard\-majority signal\.Dual Consensus\([Du et al\., 2026b](https://arxiv.org/html/2608.24982#bib.bib10)\)uses an anchor\-explorer voting relation to escape spurious majorities;CSRS\([Yu et al\., 2026](https://arxiv.org/html/2608.24982#bib.bib79)\)softens hard majority rewards into frequency signals over retraced multimodal reasoning sets, whileEvoQuality\([Wen et al\., 2026a](https://arxiv.org/html/2608.24982#bib.bib65)\)turns VLM pairwise judgments into majority\-voted image\-quality pseudo\-rankings\.

### 4\.3Cross\-Modal Extension

The relational recipe transfers to multimodal foundation models through modality\-specific reductions of free\-form outputs\.TTRV\([Singh et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib49)\)ports TTRL to vision\-language models with a frequency\-plus\-entropy reward; on ImageNet, InternVL3\-8B reaches 99\.31% \(vs\. GPT\-4o’s 98\.30%\) and exceeds GPT\-4o by 2\.3 points across eight benchmarks\.MM\-UPT\([Wei et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib64)\)runs a training\-time variant on Qwen2\.5\-VL with self\-generated prompts;EvoLMM\([Thawakar et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib55)\)closes a proposer–solver loop driven by multi\-rollout consistency\. All three derive visual supervision from the model’s own rollouts, without image\-text labels, captioners, or external verifiers\.

## 5Self\-Generated Target Bootstrapping

The third family*constructs*a trainable target from the model distribution, such as an instruction, a rationale, a plan, a debate trace, a curriculum, or a preference pair\. The update operator𝒰\\mathcal\{U\}then applies a standard SFT or DPO step against that object\. We organize the family by the type of object that the model bootstraps \(§[5\.1](https://arxiv.org/html/2608.24982#S5.SS1)–§[5\.3](https://arxiv.org/html/2608.24982#S5.SS3)\)\. Table[3](https://arxiv.org/html/2608.24982#S3.T3)shows that rationales form the largest branch with 8 methods, followed by instructions and preferences with 5 each and curricula with 4\.

### 5\.1Self\-Curated Instructions and Knowledge

The simplest synthetic targets are prompt–response pairs derived from raw documents\.Self\-Tuning\([Zhang et al\., 2025c](https://arxiv.org/html/2608.24982#bib.bib87)\)turns documents into a staged memorization–comprehension–reflection curriculum: next\-token prediction, automatically constructed document tasks, and closed\-book reconstruction\. Its document\-derived self\-teaching component falls within the strict core\.

KBAlign\([Zeng et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib83)\)self\-annotates short\- and long\-dependency question–answer pairs from a textual knowledge base, tunes on them, and then uses its own predictions and generated correction rationales in later rounds\.CYCLE\-INSTRUCT\([Shen et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib46)\)instead instantiates two transformer models from the same base: a forward modelMQ→AM\_\{Q\\rightarrow A\}and a backward modelMA→QM\_\{A\\rightarrow Q\}\. They alternate pseudo\-answer generation, backward reconstruction training, pseudo\-instruction generation, and forward reconstruction training\. The shared\-base dual loop anchors both reconstruction directions to the original text and limits drift across successive rounds of self\-training\.

### 5\.2Self\-Trained Rationales and Curricula

A larger sub\-line bootstraps reasoning targets\. The seminal recipe isSelf\-Improve\([Huang et al\., 2023](https://arxiv.org/html/2608.24982#bib.bib22)\)\(Table[3](https://arxiv.org/html/2608.24982#S3.T3), rationale rows\): generate multiple CoT traces, filter them by self\-consistency, and fine\-tune on the survivors\. Variants extend the recipe to latent thoughts during CPT\([Zelikman et al\., 2024](https://arxiv.org/html/2608.24982#bib.bib82)\), confidence\-selected traces\([Jang et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib60)\), foresight\-resampled sequences\([Xu et al\., 2025a](https://arxiv.org/html/2608.24982#bib.bib73)\), multi\-agent debate trajectories\([Srivastava et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib35)\), long\-context extensions\([Gao et al\., 2025a](https://arxiv.org/html/2608.24982#bib.bib14);[Li et al\., 2024](https://arxiv.org/html/2608.24982#bib.bib59)\), andR\-Zero\([Huang et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib24)\), which splits a base model into a Challenger \(proposing problems at the Solver’s ability boundary\) and a Solver \(training on majority\-vote pseudo\-labels\); both the curriculum and the labels are internally synthesized\. A test\-time variant produces curricula at inference\([Yang et al\., 2026a](https://arxiv.org/html/2608.24982#bib.bib78);[Moradi and Mudur, 2026](https://arxiv.org/html/2608.24982#bib.bib38);[He et al\., 2026](https://arxiv.org/html/2608.24982#bib.bib18)\): the machinery uses consensus \(Family II in isolation\), but the gradient is computed against the synthesized target, so the update\-object rule places it in Family III\.LRM Self\-Train\([Shafayat et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib47)\)follows the same logic: its reward is a binary majority match, but the stated unit of analysis is the pseudo\-target that evolves alongside the solver\.

A recent line extends self\-bootstrapping beyond math curricula\.G\-Zero\([Huang et al\., 2026](https://arxiv.org/html/2608.24982#bib.bib23)\)runs a proposer–generator loop creating hint\-driven preference pairs for open\-ended generation without any external judge\.QueST\([Song et al\., 2026](https://arxiv.org/html/2608.24982#bib.bib50)\)moves the loop to test time, generating query\-conditioned auxiliary problems and fitting a small LoRA adapter before answering;V\-Zero\([Wang et al\., 2026a](https://arxiv.org/html/2608.24982#bib.bib63)\)trains a questioner–solver vision\-language loop on unlabeled images with its own questions and majority\-vote pseudo\-labels\.

### 5\.3Self\-Generated Preference Pairs

A final sub\-line synthesizes preference pairs and optimizes with DPO\-style objectives: ranking responses by self\-consistency\([Prasad et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib41)\); multi\-agent debate consensus\([Samanta et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib44)\);*short\-to\-long*pairs \(same instruction on short vs\. long context\) for long\-context self\-evolution\([Chen et al\., 2025a](https://arxiv.org/html/2608.24982#bib.bib8)\); and ranking chains of thought by an internal answer\-confidence statistic\([van Niekerk et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib56)\)\.Confident STandRLSFselect SFT or DPO targets by confidence, so the update\-object rule assigns both to Family III; Appendix[D](https://arxiv.org/html/2608.24982#A4)contrasts evaluator\-reward variants\.

## 6Internal Evaluator Bootstrapping

Table 4:Strict UPT methods in Family IV \(§[6](https://arxiv.org/html/2608.24982#S6)\)\.‡Family I/IV bridge case \(discussed below\)\.In Family IV, the model also self\-elevates the*judge*: the internal update object is an evaluator \(scorer, reward model, or meta\-judge\) produced and consumed by the same lineage\. Where Family III bootstraps a*trainable target*and trains against it, Family IV bootstraps a*trainable verdict\-emitter*and trains its outputs through this verdict\. Because the verdict can take two natural forms, a pairwise preference or a scalar score, Table[4](https://arxiv.org/html/2608.24982#S6.T4)splits the 10 methods: 4 DPO variants consume preference pairs from the judge, and 6 policy\-gradient variants treat the judge’s score as reward\. Both sub\-lines share one constraint: the evaluator must come from the same model lineage; otherwise Appendix[D](https://arxiv.org/html/2608.24982#A4)classifies the method as adjacent under[\(B4\)](https://arxiv.org/html/2608.24982#S2.SS0.SSS0.Px4)\.

### 6\.1Self\-Rewarding via Internal Judges

The line opens withSelf\-Rewarding LM\([Yuan et al\., 2024](https://arxiv.org/html/2608.24982#bib.bib80)\): one LLM alternates between actor and judge roles; chosen–rejected pairs feed iterative DPO, and both capabilities improve in tandem\. Subsequent methods stabilize the self\-rewarding loop along three dimensions:CREAMenforces cross\-iteration consistency\([Wang et al\., 2025c](https://arxiv.org/html/2608.24982#bib.bib58)\),Meta\-Rewardingintroduces meta\-judgment\([Wu et al\., 2024](https://arxiv.org/html/2608.24982#bib.bib67)\), andTemporal SRLManchors preferences across model generations\([Wang et al\., 2025b](https://arxiv.org/html/2608.24982#bib.bib26)\)\.CSR\([Zhou et al\., 2024](https://arxiv.org/html/2608.24982#bib.bib93)\)extends self\-rewarding to vision\-language models with a calibrated self\-judge; Appendix[D](https://arxiv.org/html/2608.24982#A4)audits its external CLIP reward term\.

### 6\.2Evaluator\-Driven Policy Optimization

A second sub\-line lets the evaluator drive a policy\-gradient update directly: structured multi\-agent debate with critique\-helpfulness as Bradley–Terry\-aggregated reward\([Sui and Hooi, 2026](https://arxiv.org/html/2608.24982#bib.bib52)\); an internal evaluator’s natural\-language meta\-judgments \(“correct? / logically consistent?”\) as scalar rewards\([Rentschler and Roberts, 2026](https://arxiv.org/html/2608.24982#bib.bib43)\); and a multimodal cohort\-visible test\-time recipe via a metacognitive introspector\([Tan et al\., 2026](https://arxiv.org/html/2608.24982#bib.bib53)\)\. Family IV routes reward through an explicit evaluator, whereas Family II derives it from multi\-sample statistics, a distinction documented in Appendix[D](https://arxiv.org/html/2608.24982#A4)\.

The same internal\-evaluator pattern appears in broader self\-evolution loops\.AERO\([Gao et al\., 2026](https://arxiv.org/html/2608.24982#bib.bib16)\)uses self\-generated tasks and counterfactual criticism for KTO\-style updates, andSelf\-Judge\([Wu et al\., 2026](https://arxiv.org/html/2608.24982#bib.bib69)\)uses a same\-model\-lineage frozen judge to modulate actor self\-consistency\. Two unified multimodal systems sit on the Family I/IV boundary: the update\-object rule assignsSUDER\([Hong et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib19)\)to Family I andGvU\([Pan et al\., 2026](https://arxiv.org/html/2608.24982#bib.bib39)\)to Family IV according to the primary signal each update consumes\.

## 7Cross\-Family Synthesis

#### Four sources of leverage\.

UPT converts an internal proxy into an update: Family I exploits predictive statistics; Family II aggregates evidence across samples; Family III builds inspectable targets; and Family IV supplies semantic criteria for open\-ended outputs\. Increasing semantic flexibility also lengthens the feedback path, making consistency, diversity, and independent evaluation increasingly valuable\.

Self\-training theory links improvement from unlabeled data to class\-consistent neighborhoods and expansion\([Wei et al\., 2021](https://arxiv.org/html/2608.24982#bib.bib4)\), while evidence on confirmation bias motivates consistency, diversity, and held\-out checks\([Arazo et al\., 2020](https://arxiv.org/html/2608.24982#bib.bib3)\)\. Appendix[B](https://arxiv.org/html/2608.24982#A2)maps one representative result per family and signal\-shaping task structures to these conditions \(Tablesand\)\.

#### From signal to update\.

The four families differ in where uncertainty is converted into a training decision\. Family I optimizes a statistic at one model state\.LangAdapt CPT\([Elhady et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib11)\)uses token likelihood on domain text, while entropy\- and confidence\-based variants reshape the predictive distribution; this short path depends on alignment between the statistic and downstream behavior rather than proxy sharpness alone\. Family II aggregates rollouts\.TTRL\([Zuo et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib95)\)canonicalizes sampled answers, turns majority\-class membership into a GRPO reward, and updates the policy against that relation\. Agreement is informative when errors are diverse and answer equivalence is stable, but self\-reinforcing when errors correlate\. Family III materializes selected outputs as inspectable, reusable targets\. InR\-Zero\([Huang et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib24)\), a Challenger proposes problems near the Solver’s ability boundary and the Solver trains on synthesized labels\. The resulting data can be inspected and reused, while selection errors can persist after entering the target set\. Family IV inserts a same\-lineage evaluator before the update\.Self\-Rewarding LM\([Yuan et al\., 2024](https://arxiv.org/html/2608.24982#bib.bib80)\)andCoNL\([Sui and Hooi, 2026](https://arxiv.org/html/2608.24982#bib.bib52)\)turn judgments or critiques into rewards for open\-ended outputs, while actor–judge drift becomes part of the dynamics\. Semantic reach grows with aggregation, target construction, and evaluation, together with the longer feedback path through which errors can recur\.

#### Same observable, different gradient path\.

Majority vote and confidence recur across families, so classification follows the object consumed by the gradient\.TTRLandRoiRL\([Arzhantsev et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib5)\)consume majority agreement as reward, placing them in Family II;LRM Self\-Train\([Shafayat et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib47)\)uses it to select solutions and computes an SFT loss against the retained target set, placing it in Family III\.Confident ST\([Jang et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib60)\)andRLSF\([van Niekerk et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib56)\)follow the same target path when confidence selects trajectories before SFT or DPO\. Confidence consumed directly as loss or reward is a Family I statistic; a same\-lineage evaluator’s semantic score consumed as reward is a Family IV object\. This gradient\-path rule remains stable across labels such as self\-training, self\-rewarding, and confidence optimization; Appendix[D](https://arxiv.org/html/2608.24982#A4)records these assignments and the update objects that justify them across the taxonomy\.

#### Reading reported evidence\.

Tablepreserves one mechanism\-matched result per family under its original setup\. For Family I,LangAdapt CPTreduces Basque perplexity from 23\.64 to 3\.35 and raises aggregate downstream accuracy from 27\.43 to 34\.14, separating corpus adaptation from task transfer\. For Family II,TTRLraises AIME 2024 from 12\.9 to 40\.2 and MATH\-500 from 46\.7 to 83\.4, while its GPQA result ties majority reward to the target task structure\. For Family III,Quiet\-STaR\([Zelikman et al\., 2024](https://arxiv.org/html/2608.24982#bib.bib82)\)raises zero\-shot GSM8K from 5\.9 to 10\.9 and CommonsenseQA from 36\.3 to 47\.2, supporting future\-token selection of latent reasoning targets\. For Family IV,CoNLraises AIME 2024 from 60\.0 to 76\.5 and DeepMath from 70\.5 to 87\.1, demonstrating a strong same\-policy semantic signal\. Together, the rows connect the four update objects to corpus fit, answer equivalence, target selection, and evaluator quality; Appendix[B](https://arxiv.org/html/2608.24982#A2)retains the source\-specific metrics, budgets, and adaptation protocols\.

#### Choosing by available signal\.

When adaptation data are raw documents and distribution shift is the main problem, Family I is the direct baseline\. When multiple samples are affordable and outcomes admit a defensible equivalence relation, Family II can exploit agreement\. When generated targets can be inspected before an offline update, Family III offers the clearest data interface\. When outputs are open\-ended and equality is undefined, Family IV supplies learned semantic criteria through evaluator\-mediated rollouts; externally grounded programmatic verifiers remain adjacent\.

#### Safeguards follow the feedback path\.

Family I safeguards track calibration and downstream quality alongside the optimized statistic, distinguishing a sharper proxy from a better model\. Family II tracks sample diversity, wrong\-majority frequency, and agreement under alternative canonicalizers\. Family III tracks target diversity and refreshes the generated dataset before selection errors persist across rounds\. Family IV tracks actor–judge correlation, preference margins, cross\-round consistency, and saturation\. Across families, frozen baselines, rollback checkpoints, and held\-out evaluation test whether gains persist\. Each check acts where the internal proxy becomes a training signal, interrupting error accumulation before the next round reads a more biased proxy\.

## 8Timing of Adaptation

The update\-object taxonomy asks what internal object supplies the signal\. A timing view asks when target inputs become visible and how long the induced change persists\. We call these two axes*Input Visibility*and*Update Persistence*\. They are orthogonal to family membership: the same update object can be redeployed across regimes, so timing decides deployment cost while family decides supervision shape\. Cross\-cutting these two axes yields the regimes of Figure[2](https://arxiv.org/html/2608.24982#S8.F2), which charts adjacent no\-update inference\-time optimization that uses similar internal signals but does not satisfy the explicit\-update requirement of strict UPT\.

\{forest\}

Figure 2:Timing of Adaptation, organized by Input Visibility×\\timesUpdate Persistence and orthogonal to Figure[1](https://arxiv.org/html/2608.24982#S1.F1)\. Each leaf names representative methods; protocols spanning multiple regimes appear in more than one leaf\. Adjacent no\-update inference\-time methods are shown for boundary clarity but are not counted as strict UPT\.#### Pre\-sample regimes\.

The first four regimes differ in how much of the target distribution the update sees\.*Offline corpus UPT*is the broadest, covering continued pretraining, instruction self\-curation, reasoning self\-training, and offline self\-rewarding\([Ke et al\., 2023](https://arxiv.org/html/2608.24982#bib.bib28);[Yuan et al\., 2024](https://arxiv.org/html/2608.24982#bib.bib80);[Huang et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib24), e\.g\.,\); the target distribution is invisible\.*Full\-cohort transductive adaptation*updates over the entire target cohort at once\([Zuo et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib95);[Wei et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib64);[Wang et al\., 2025a](https://arxiv.org/html/2608.24982#bib.bib61);[Xing et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib71);[Zhao et al\., 2026a](https://arxiv.org/html/2608.24982#bib.bib92);[Wang et al\., 2026b](https://arxiv.org/html/2608.24982#bib.bib62)\)\.*Few\-sample target adaptation*sees only a small slice and generalizes to a held\-out remainder\([Singh et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib49);[Gao et al\., 2025b](https://arxiv.org/html/2608.24982#bib.bib15);[Prabhudesai et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib40);[Li et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib31)\); this is a strictly stronger generalization claim than full\-cohort\.*Streaming continual adaptation*is properly online: at samplettthe model uses only prefix1:t−11\{:\}t\{\-\}1, with updates accumulating forward\([Hu et al\., 2025a](https://arxiv.org/html/2608.24982#bib.bib21);[Singh et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib49);[Hedna et al\., 2026](https://arxiv.org/html/2608.24982#bib.bib51);[Liu et al\., 2026](https://arxiv.org/html/2608.24982#bib.bib37)\)\.

#### Within\-sample regimes\.

The remaining two interleave adaptation with prediction\.*Test\-time instance adaptation*fits a small update for the current instance and resets at its boundary\([Hardt and Sun, 2024](https://arxiv.org/html/2608.24982#bib.bib25);[Bansal et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib6);[Hu et al\., 2025b](https://arxiv.org/html/2608.24982#bib.bib20);[Xu et al\., 2025c](https://arxiv.org/html/2608.24982#bib.bib75);[Xu et al\., 2026](https://arxiv.org/html/2608.24982#bib.bib76);[Kang et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib27)\)\.*Within\-sequence adaptation*goes finer still: the update unfolds across chunks or token\-states of one sequence and resets at its boundary\([Feng et al\., 2026](https://arxiv.org/html/2608.24982#bib.bib12);[Sim, 2025](https://arxiv.org/html/2608.24982#bib.bib48)\); the update is to persistent local state, satisfying[\(B1\)](https://arxiv.org/html/2608.24982#S2.SS0.SSS0.Px4)\. Protocols can instantiate multiple regimes\. Figure[2](https://arxiv.org/html/2608.24982#S8.F2)records each realized protocol; Family II×\\timesfull\-cohort transductive is the densest observed cell, exemplified byTTRL\.

## 9Challenges and Future Directions

Recursive error propagation is the central open problem: updates reinforce misordered outputs and shift the distribution from which the next proxy is computed\. The four families expose different links in this chain and suggest distinct research priorities\.

#### Separate signal quality from task structure\.

Consensus can be informative because errors cancel, but only after an extractor defines which outputs agree\. Boxed answers, finite choice sets, and code signatures provide such structure without determining correctness; execution and unit tests return a verdict and therefore cross[\(B3\)](https://arxiv.org/html/2608.24982#S2.SS0.SSS0.Px4)\. Future evaluations should report the fields in Tableand test whether gains survive alternative canonicalizers and open\-ended reformulations\. This matters most for Families II–III; meanwhile, open\-ended methods such asG\-Zero\([Huang et al\., 2026](https://arxiv.org/html/2608.24982#bib.bib23)\)probe the brittle boundary of equality\-based consensus\.

#### Interrupt confidence and majority amplification\.

Family I can turn low entropy into overconfidence; Family II can turn a popular error into a training reward\. Among the 22 Family II methods, 13 use hard majority signals, so evaluations should include diversity and wrong\-majority diagnostics\.EVOL\-RL\([Zhou et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib94)\)adds semantic novelty, whileDual Consensus\([Du et al\., 2026b](https://arxiv.org/html/2608.24982#bib.bib10)\)andCSRS\([Yu et al\., 2026](https://arxiv.org/html/2608.24982#bib.bib79)\)soften or diversify consensus\.T3RL\([Liao et al\., 2026](https://arxiv.org/html/2608.24982#bib.bib33)\)shows that an independent execution channel can break false\-popular modes, motivating independent internal views for strict UPT\.

#### Measure target and judge drift across rounds\.

Family III freezes selected generations into training targets, so confirmation bias can survive even when the next sampling round looks more confident\. Family IV adds evaluator drift: bias amplification, judge saturation, and weak chosen–rejected margins are addressed separately by CREAM, Meta\-Rewarding, and Temporal SRLM\([Wang et al\., 2025c](https://arxiv.org/html/2608.24982#bib.bib58);[Wu et al\., 2024](https://arxiv.org/html/2608.24982#bib.bib67);[Wang et al\., 2025b](https://arxiv.org/html/2608.24982#bib.bib26)\)\. Across these interventions, a common longitudinal protocol would track target diversity, actor–judge correlation, held\-out quality, and perturbation recovery\.

#### Control initialization and timing\.

Intrinsic\-feedback results depend on the starting checkpoint and update duration\.[Zhang et al\. \(2025d\)](https://arxiv.org/html/2608.24982#bib.bib85)show that these factors materially change the outcome of intrinsic\-feedback training\. Cross\-family benchmarks should fix backbone, starting checkpoint, target split, rollout and update budgets, task\-structure prior, and held\-out evaluation to isolate the effects of signal family, initialization, and timing\.

## 10Conclusion

We survey 80 strict UPT methods around a single question: which model\-derived object supplies the update signal? The resulting taxonomy has four families, while Input Visibility and Update Persistence provide an orthogonal map of when target inputs become visible and how long updates persist\. Together, these views clarify UPT’s central trade\-off: more semantically expressive signals support open\-ended outputs but create longer feedback paths through which proxy errors can be reinforced\. Method selection and evaluation should therefore align the update object with task structure and deployment timing, and place safeguards where the proxy enters the update\.

## Limitations

The inventory is frozen in May 2026 and therefore excludes later work\. Representative results retain each source paper’s backbone, budget, metric, and adaptation protocol, supporting mechanism\-level synthesis rather than pooled effect\-size estimation\. Hybrid methods follow the primary update object consumed by the gradient during adaptation\. Finally,[\(B1\)](https://arxiv.org/html/2608.24982#S2.SS0.SSS0.Px4)–[\(B4\)](https://arxiv.org/html/2608.24982#S2.SS0.SSS0.Px4)constrain the update rule rather than the deployment stack\. Because no external oracle scores the updated model, reward hacking and silent per\-problem degradation can accumulate unobserved, so independent held\-out evaluation, monitoring, and red\-teaming remain necessary and are not excluded by the strict boundary; we organize such safeguards by feedback path \(§[7](https://arxiv.org/html/2608.24982#S7)\) but do not evaluate their effectiveness, which requires controlled study\.

## Acknowledgments

This work was supported in part by the National Natural Science Foundation of China \(Grant Nos\. 92370204 and 62506318\); in part by the National Key R&D Program of China \(Grant No\. 2023YFF0725001\); in part by the Guangdong Provincial Key Laboratory of Frontier Basic Science for All\-domain Intelligence; in part by the Guangdong Basic and Applied Basic Research Foundation \(Grant No\. 2023B1515120057\); in part by the Key\-Area Special Project of Guangdong Provincial Ordinary Universities \(Grant No\. 2024ZDZX1007\); in part by the Guangdong Provincial Department of Education Project \(Grant No\. 2024KQNCX028\); in part by the Scientific Research Projects for the Higher\-educational Institutions, Education Bureau of Guangzhou Municipality \(Grant No\. 2024312096\); and in part by the Guangzhou\-HKUST\(GZ\) Joint Funding Program, Education Bureau of Guangzhou Municipality \(Grant No\. 2025A03J3957\)\.

## References

- Abbeset al\.\(2026\)I\. Abbes, G\. Subbaraj, M\. Riemer, N\. Islah, T\. Tabaru, H\. Kingetsu, S\. Chandar, and I\. RishRevisiting replay and gradient alignment for continual pretraining of large language models\.InProceedings of The 4th Conference on Lifelong Learning Agents,Proceedings of Machine Learning Research, Vol\.330,pp\. 465–486\.External Links:[Link](https://proceedings.mlr.press/v330/abbes26a.html)Cited by:[Table 1](https://arxiv.org/html/2608.24982#S2.T1.2.7.1),[§3](https://arxiv.org/html/2608.24982#S3.SS0.SSS0.Px1.p1.1)\.
- Agarwalet al\.\(2025\)S\. Agarwal, Z\. Zhang, L\. Yuan, J\. Han, and H\. PengThe unreasonable effectiveness of entropy minimization in LLM reasoning\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=UfFTBEsLgI)Cited by:[§D\.1](https://arxiv.org/html/2608.24982#A4.SS1.p1.1),[Table 1](https://arxiv.org/html/2608.24982#S2.T1.2.15.1),[Table 1](https://arxiv.org/html/2608.24982#S2.T1.2.17.1),[Table 1](https://arxiv.org/html/2608.24982#S2.T1.2.18.1),[§3](https://arxiv.org/html/2608.24982#S3.SS0.SSS0.Px2.p1.1)\.
- Arazoet al\.\(2020\)E\. Arazo, D\. Ortego, P\. Albert, N\. E\. O’Connor, and K\. McGuinnessPseudo\-labeling and confirmation bias in deep semi\-supervised learning\.In2020 International Joint Conference on Neural Networks,pp\. 1–8\.External Links:[Document](https://dx.doi.org/10.1109/IJCNN48605.2020.9207304),[Link](https://arxiv.org/abs/1908.02983)Cited by:[§7](https://arxiv.org/html/2608.24982#S7.SS0.SSS0.Px1.p2.1)\.
- Arzhantsevet al\.\(2025\)A\. Arzhantsev, O\. Sakhi, and F\. VasileRoiRL: efficient, self\-supervised reasoning with offline iterative reinforcement learning\.InNeurIPS 2025 Workshop on Efficient Reasoning,External Links:[Link](https://openreview.net/forum?id=PeJ1eGGygZ)Cited by:[Table 2](https://arxiv.org/html/2608.24982#S2.T2.2.19.1),[§7](https://arxiv.org/html/2608.24982#S7.SS0.SSS0.Px3.p1.1)\.
- Bansalet al\.\(2025\)R\. Bansal, A\. Zhang, R\. Tiwari, L\. Madaan, S\. S\. Duvvuri, D\. Khatri, D\. Brandfonbrener, D\. Alvarez\-Melis, P\. Bhargava, M\. S\. Kale, and S\. JelassiLet’s \(not\) just put things in context: test\-time training for long\-context llms\.External Links:2512\.13898,[Link](https://arxiv.org/abs/2512.13898)Cited by:[Table 1](https://arxiv.org/html/2608.24982#S2.T1.2.13.1),[§3](https://arxiv.org/html/2608.24982#S3.SS0.SSS0.Px1.p1.1),[§8](https://arxiv.org/html/2608.24982#S8.SS0.SSS0.Px2.p1.1)\.
- Chenet al\.\(2025a\)G\. Chen, X\. Li, M\. Shieh, and L\. BingLongPO: long context self\-evolution of large language models through short\-to\-long preference optimization\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=qTrEq31Shm)Cited by:[Table 3](https://arxiv.org/html/2608.24982#S3.T3.2.20.1),[§5\.3](https://arxiv.org/html/2608.24982#S5.SS3.p1.1)\.
- Chenet al\.\(2025b\)Q\. Chen, L\. Qin, J\. Liu, D\. Peng, J\. Guan, P\. Wang, M\. Hu, Y\. Zhou, T\. Gao, and W\. CheTowards reasoning era: a survey of long chain\-of\-thought for reasoning large language models\.External Links:2503\.09567,[Link](https://arxiv.org/abs/2503.09567)Cited by:[Appendix E](https://arxiv.org/html/2608.24982#A5.p1.1),[§1](https://arxiv.org/html/2608.24982#S1.p3.1)\.
- Daiet al\.\(2025\)L\. Dai, Y\. Xu, J\. Ye, H\. Liu, and H\. XiongSePer: measure retrieval utility through the lens of semantic perplexity reduction\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§D\.1](https://arxiv.org/html/2608.24982#A4.SS1.p1.1)\.
- Denget al\.\(2025\)S\. Deng, K\. Wang, T\. Yang, H\. Singh, and Y\. TianSelf\-improvement in multimodal large language models: a survey\.External Links:2510\.02665,[Link](https://arxiv.org/abs/2510.02665)Cited by:[Appendix E](https://arxiv.org/html/2608.24982#A5.p1.1)\.
- Duet al\.\(2026a\)B\. Du, X\. Huang, and X\. LiDistribution\-aware reward estimation for test\-time reinforcement learning\.External Links:2601\.21804,[Link](https://arxiv.org/abs/2601.21804)Cited by:[Table 2](https://arxiv.org/html/2608.24982#S2.T2.2.14.1),[§4\.2](https://arxiv.org/html/2608.24982#S4.SS2.p1.1)\.
- Duet al\.\(2026b\)K\. Du, M\. Cao, H\. Zhang, Y\. Wang, X\. Huang, and N\. LiDual consensus: escaping from spurious majority in unsupervised rlvr via two\-stage vote mechanism\.External Links:2603\.16223,[Link](https://arxiv.org/abs/2603.16223)Cited by:[Table 2](https://arxiv.org/html/2608.24982#S2.T2.2.23.1),[§4\.2](https://arxiv.org/html/2608.24982#S4.SS2.p2.1),[§9](https://arxiv.org/html/2608.24982#S9.SS0.SSS0.Px2.p1.1)\.
- Elhadyet al\.\(2025\)A\. Elhady, E\. Agirre, and M\. ArtetxeEmergent abilities of large language models under continued pre\-training for language adaptation\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 32174–32186\.External Links:[Link](https://aclanthology.org/2025.acl-long.1547/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1547),ISBN 979\-8\-89176\-251\-0Cited by:[Table 1](https://arxiv.org/html/2608.24982#S2.T1.2.5.1),[§3](https://arxiv.org/html/2608.24982#S3.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2608.24982#S7.SS0.SSS0.Px2.p1.1)\.
- Fenget al\.\(2026\)G\. Feng, S\. Luo, K\. Hua, G\. Zhang, W\. Huang, D\. He, and T\. CaiIn\-place test\-time training\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=dTWfCLSoyl)Cited by:[Table 1](https://arxiv.org/html/2608.24982#S2.T1.2.14.1),[§3](https://arxiv.org/html/2608.24982#S3.SS0.SSS0.Px1.p1.1),[§8](https://arxiv.org/html/2608.24982#S8.SS0.SSS0.Px2.p1.1)\.
- Fuet al\.\(2024\)Y\. Fu, R\. Panda, X\. Niu, X\. Yue, H\. Hajishirzi, Y\. Kim, and H\. PengData engineering for scaling language models to 128k context\.External Links:2402\.10171,[Link](https://arxiv.org/abs/2402.10171)Cited by:[Table 1](https://arxiv.org/html/2608.24982#S2.T1.2.9.1),[§3](https://arxiv.org/html/2608.24982#S3.SS0.SSS0.Px1.p1.1)\.
- Gaoet al\.\(2025a\)C\. Gao, X\. Wu, Z\. Lin, D\. Zhang, and S\. HuLongMagpie: a self\-synthesis method for generating large\-scale long\-context instructions\.External Links:2505\.17134,[Link](https://arxiv.org/abs/2505.17134)Cited by:[Table 3](https://arxiv.org/html/2608.24982#S3.T3.2.12.1),[§5\.2](https://arxiv.org/html/2608.24982#S5.SS2.p1.1)\.
- Gaoet al\.\(2026\)Z\. Gao, J\. Ma, X\. Li, P\. Li, N\. Qu, Y\. Wu, H\. Liu, and J\. LiuAERO: autonomous evolutionary reasoning optimization via endogenous dual\-loop feedback\.External Links:2602\.03084,[Link](https://arxiv.org/abs/2602.03084)Cited by:[§6\.2](https://arxiv.org/html/2608.24982#S6.SS2.p2.1),[Table 4](https://arxiv.org/html/2608.24982#S6.T4.2.9.1)\.
- Gaoet al\.\(2025b\)Z\. Gao, L\. Chen, H\. Luo, J\. Zhou, and B\. DaiOne\-shot entropy minimization\.Note:Work in progressExternal Links:2505\.20282,[Link](https://arxiv.org/abs/2505.20282)Cited by:[Table 1](https://arxiv.org/html/2608.24982#S2.T1.2.16.1),[§3](https://arxiv.org/html/2608.24982#S3.SS0.SSS0.Px2.p1.1),[§8](https://arxiv.org/html/2608.24982#S8.SS0.SSS0.Px1.p1.1)\.
- Guoet al\.\(2025\)Y\. Guo, J\. Fu, H\. Zhang, and D\. ZhaoEfficient domain continual pretraining by mitigating the stability gap\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 32850–32870\.External Links:[Link](https://aclanthology.org/2025.acl-long.1578/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1578),ISBN 979\-8\-89176\-251\-0Cited by:[Table 1](https://arxiv.org/html/2608.24982#S2.T1.2.6.1),[§3](https://arxiv.org/html/2608.24982#S3.SS0.SSS0.Px1.p1.1)\.
- Hardt and Sun \(2024\)M\. Hardt and Y\. SunTest\-time training on nearest neighbors for large language models\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=CNL2bku4ra)Cited by:[Table 1](https://arxiv.org/html/2608.24982#S2.T1.2.12.1),[§3](https://arxiv.org/html/2608.24982#S3.SS0.SSS0.Px1.p1.1),[§8](https://arxiv.org/html/2608.24982#S8.SS0.SSS0.Px2.p1.1)\.
- Heet al\.\(2026\)H\. He, Z\. Rong, L\. Zhao, Y\. Zhao, L\. Yang, and H\. ZhangTTSR: test\-time self\-reflection for continual reasoning improvement\.External Links:2603\.03297,[Link](https://arxiv.org/abs/2603.03297)Cited by:[Table 3](https://arxiv.org/html/2608.24982#S3.T3.2.16.1),[§5\.2](https://arxiv.org/html/2608.24982#S5.SS2.p1.1)\.
- Hednaet al\.\(2026\)M\. R\. Hedna, J\. Strich, M\. Semmann, and C\. BiemannSelf\-calibrating language models via test\-time discriminative distillation\.External Links:2604\.09624,[Link](https://arxiv.org/abs/2604.09624)Cited by:[§8](https://arxiv.org/html/2608.24982#S8.SS0.SSS0.Px1.p1.1)\.
- Honget al\.\(2025\)J\. Hong, Y\. Zhang, G\. Wang, Y\. Liu, J\. Wen, and R\. YanSUDER: self\-improving unified large multimodal models for understanding and generation with dual self\-rewards\.External Links:2506\.07963,[Link](https://arxiv.org/abs/2506.07963)Cited by:[Table 1](https://arxiv.org/html/2608.24982#S2.T1.2.28.1),[§3](https://arxiv.org/html/2608.24982#S3.SS0.SSS0.Px3.p1.1),[§6\.2](https://arxiv.org/html/2608.24982#S6.SS2.p2.1)\.
- Huet al\.\(2025a\)J\. Hu, Z\. Zhang, G\. Chen, X\. Wen, C\. Shuai, W\. Luo, B\. Xiao, Y\. Li, and M\. TanTest\-time learning for large language models\.InProceedings of the 42nd International Conference on Machine Learning,A\. Singh, M\. Fazel, D\. Hsu, S\. Lacoste\-Julien, F\. Berkenkamp, T\. Maharaj, K\. Wagstaff, and J\. Zhu \(Eds\.\),Proceedings of Machine Learning Research, Vol\.267,pp\. 24823–24849\.External Links:[Link](https://proceedings.mlr.press/v267/hu25z.html)Cited by:[Table 1](https://arxiv.org/html/2608.24982#S2.T1.2.11.1),[§3](https://arxiv.org/html/2608.24982#S3.SS0.SSS0.Px1.p1.1),[§8](https://arxiv.org/html/2608.24982#S8.SS0.SSS0.Px1.p1.1)\.
- Huet al\.\(2025b\)Y\. Hu, X\. Zhang, X\. Fang, Z\. Chen, X\. Wang, H\. Zhang, and G\. QiSLOT: sample\-specific language model optimization at test\-time\.External Links:2505\.12392,[Link](https://arxiv.org/abs/2505.12392)Cited by:[Table 1](https://arxiv.org/html/2608.24982#S2.T1.2.21.1),[§3](https://arxiv.org/html/2608.24982#S3.SS0.SSS0.Px4.p1.1),[§8](https://arxiv.org/html/2608.24982#S8.SS0.SSS0.Px2.p1.1)\.
- Huanget al\.\(2026\)C\. Huang, H\. Liu, T\. Zheng, R\. Dai, L\. Huang, J\. Li, Z\. Li, Z\. Wei, Y\. Meng, and J\. HuangG\-zero: self\-play for open\-ended generation from zero data\.External Links:2605\.09959,[Link](https://arxiv.org/abs/2605.09959)Cited by:[Table 3](https://arxiv.org/html/2608.24982#S3.T3.2.22.1),[§5\.2](https://arxiv.org/html/2608.24982#S5.SS2.p2.1),[§9](https://arxiv.org/html/2608.24982#S9.SS0.SSS0.Px1.p1.1)\.
- Huanget al\.\(2025\)C\. Huang, W\. Yu, X\. Wang, H\. Zhang, Z\. Li, R\. Li, J\. Huang, H\. Mi, and D\. YuR\-zero: self\-evolving reasoning llm from zero data\.External Links:2508\.05004,[Link](https://arxiv.org/abs/2508.05004)Cited by:[§1](https://arxiv.org/html/2608.24982#S1.p1.1),[Table 3](https://arxiv.org/html/2608.24982#S3.T3.2.17.1),[§5\.2](https://arxiv.org/html/2608.24982#S5.SS2.p1.1),[§7](https://arxiv.org/html/2608.24982#S7.SS0.SSS0.Px2.p1.1),[§8](https://arxiv.org/html/2608.24982#S8.SS0.SSS0.Px1.p1.1)\.
- Huanget al\.\(2023\)J\. Huang, S\. Gu, L\. Hou, Y\. Wu, X\. Wang, H\. Yu, and J\. HanLarge language models can self\-improve\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 1051–1068\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.67/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.67)Cited by:[§1](https://arxiv.org/html/2608.24982#S1.p1.1),[Table 3](https://arxiv.org/html/2608.24982#S3.T3.2.6.1),[§5\.2](https://arxiv.org/html/2608.24982#S5.SS2.p1.1)\.
- Ibrahimet al\.\(2024\)A\. Ibrahim, B\. Thérien, K\. Gupta, M\. L\. Richter, Q\. Anthony, T\. Lesort, E\. Belilovsky, and I\. RishSimple and scalable strategies to continually pre\-train large language models\.External Links:2403\.08763,[Link](https://arxiv.org/abs/2403.08763)Cited by:[Table 1](https://arxiv.org/html/2608.24982#S2.T1.2.4.1),[§3](https://arxiv.org/html/2608.24982#S3.SS0.SSS0.Px1.p1.1)\.
- Janget al\.\(2025\)H\. Jang, Y\. Jang, S\. Lee, J\. Ok, and S\. AhnSelf\-training large language models with confident reasoning\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 14925–14939\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.806/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.806),ISBN 979\-8\-89176\-335\-7Cited by:[§D\.3](https://arxiv.org/html/2608.24982#A4.SS3.p1.1),[Table 3](https://arxiv.org/html/2608.24982#S3.T3.2.8.1),[§5\.2](https://arxiv.org/html/2608.24982#S5.SS2.p1.1),[§7](https://arxiv.org/html/2608.24982#S7.SS0.SSS0.Px3.p1.1)\.
- Kanget al\.\(2025\)X\. Kang, D\. Shi, and L\. ChenModel whisper: steering vectors unlock large language models’ potential in test\-time\.External Links:2512\.04748,[Link](https://arxiv.org/abs/2512.04748)Cited by:[Table 1](https://arxiv.org/html/2608.24982#S2.T1.2.23.1),[§3](https://arxiv.org/html/2608.24982#S3.SS0.SSS0.Px4.p1.1),[§8](https://arxiv.org/html/2608.24982#S8.SS0.SSS0.Px2.p1.1)\.
- Keet al\.\(2023\)Z\. Ke, Y\. Shao, H\. Lin, T\. Konishi, G\. Kim, and B\. LiuContinual pre\-training of language models\.External Links:2302\.03241,[Link](https://arxiv.org/abs/2302.03241)Cited by:[Table 1](https://arxiv.org/html/2608.24982#S2.T1.2.3.1),[§3](https://arxiv.org/html/2608.24982#S3.SS0.SSS0.Px1.p1.1),[§8](https://arxiv.org/html/2608.24982#S8.SS0.SSS0.Px1.p1.1)\.
- Kumaret al\.\(2025\)K\. Kumar, T\. Ashraf, O\. Thawakar, R\. M\. Anwer, H\. Cholakkal, M\. Shah, M\. Yang, P\. H\.S\. Torr, F\. S\. Khan, and S\. KhanLLM post\-training: a deep dive into reasoning large language models\.External Links:2502\.21321,[Link](https://arxiv.org/abs/2502.21321)Cited by:[Appendix E](https://arxiv.org/html/2608.24982#A5.p1.1),[§1](https://arxiv.org/html/2608.24982#S1.p3.1)\.
- Liet al\.\(2026\)P\. Li, P\. Song, W\. Li, H\. Yao, W\. Guo, Y\. Xu, D\. Liu, and H\. XiongSee&trek: training\-free spatial prompting for multimodal large language model\.Advances in Neural Information Processing Systems38,pp\. 62098–62131\.Cited by:[§D\.1](https://arxiv.org/html/2608.24982#A4.SS1.p1.1)\.
- Liet al\.\(2025\)P\. Li, M\. Skripkin, A\. Zubrey, A\. Kuznetsov, and I\. OseledetsConfidence is all you need: few\-shot rl fine\-tuning of language models\.External Links:2506\.06395,[Link](https://arxiv.org/abs/2506.06395)Cited by:[Table 1](https://arxiv.org/html/2608.24982#S2.T1.2.20.1),[§3](https://arxiv.org/html/2608.24982#S3.SS0.SSS0.Px2.p1.1),[§8](https://arxiv.org/html/2608.24982#S8.SS0.SSS0.Px1.p1.1)\.
- Liet al\.\(2024\)S\. Li, C\. Yang, Z\. Cheng, L\. Liu, M\. Yu, Y\. Yang, and W\. LamLarge language models can self\-improve in long\-context reasoning\.External Links:2411\.08147,[Link](https://arxiv.org/abs/2411.08147)Cited by:[Table 3](https://arxiv.org/html/2608.24982#S3.T3.2.13.1),[§5\.2](https://arxiv.org/html/2608.24982#S5.SS2.p1.1)\.
- Liet al\.\(2023\)X\. Li, P\. Yu, C\. Zhou, T\. Schick, O\. Levy, L\. Zettlemoyer, J\. Weston, and M\. LewisSelf\-alignment with instruction backtranslation\.External Links:2308\.06259,[Link](https://arxiv.org/abs/2308.06259)Cited by:[§D\.1](https://arxiv.org/html/2608.24982#A4.SS1.p3.1),[§2](https://arxiv.org/html/2608.24982#S2.SS0.SSS0.Px6.p1.1)\.
- Lianget al\.\(2023\)J\. Liang, R\. He, and T\. TanA comprehensive survey on test\-time adaptation under distribution shifts\.External Links:2303\.15361,[Link](https://arxiv.org/abs/2303.15361)Cited by:[Appendix E](https://arxiv.org/html/2608.24982#A5.p1.1),[§1](https://arxiv.org/html/2608.24982#S1.p3.1)\.
- Lianget al\.\(2024\)X\. Liang, S\. Song, Z\. Zheng, H\. Wang, Q\. Yu, X\. Li, R\. Li, Y\. Wang, Z\. Wang, F\. Xiong, and Z\. LiInternal consistency and self\-feedback in large language models: a survey\.External Links:2407\.14507,[Link](https://arxiv.org/abs/2407.14507)Cited by:[Appendix E](https://arxiv.org/html/2608.24982#A5.p1.1)\.
- Liaoet al\.\(2026\)R\. Liao, N\. Röhrich, X\. Wang, Y\. Zhang, Y\. Samadzadeh, V\. Tresp, and S\. Yeung\-LevyTool verification for test\-time reinforcement learning\.External Links:2603\.02203,[Link](https://arxiv.org/abs/2603.02203)Cited by:[§D\.1](https://arxiv.org/html/2608.24982#A4.SS1.p2.1),[§1](https://arxiv.org/html/2608.24982#S1.p1.1),[§2](https://arxiv.org/html/2608.24982#S2.SS0.SSS0.Px6.p1.1),[§9](https://arxiv.org/html/2608.24982#S9.SS0.SSS0.Px2.p1.1)\.
- Liuet al\.\(2026\)C\. Liu, Y\. Liu, T\. Wang, Q\. Zhuang, J\. C\. Liang, W\. Yang, R\. Xu, Q\. Wang, D\. Liu, and C\. HanOn\-the\-fly vla adaptation via test\-time reinforcement learning\.External Links:2601\.06748,[Link](https://arxiv.org/abs/2601.06748)Cited by:[§8](https://arxiv.org/html/2608.24982#S8.SS0.SSS0.Px1.p1.1)\.
- Liuet al\.\(2025\)J\. Liu, C\. He, Y\. Lin, M\. Yang, F\. Shen, and S\. LiuETTRL: balancing exploration and exploitation in llm test\-time reinforcement learning via entropy mechanism\.External Links:2508\.11356,[Link](https://arxiv.org/abs/2508.11356)Cited by:[Table 2](https://arxiv.org/html/2608.24982#S2.T2.2.10.1),[§4\.2](https://arxiv.org/html/2608.24982#S4.SS2.p1.1)\.
- Liuet al\.\(2024\)J\. Liu, Z\. Bai, Y\. Zhang, C\. Zhang, Y\. Zhang, G\. Zhang, J\. Wang, H\. Que, Y\. Chen, W\. Su, T\. Ge, J\. Fu, W\. Chen, and B\. ZhengE2\-LLM: efficient and extreme length extension of large language models\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 4243–4253\.External Links:[Link](https://aclanthology.org/2024.findings-acl.252/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.252)Cited by:[Table 1](https://arxiv.org/html/2608.24982#S2.T1.2.8.1),[§3](https://arxiv.org/html/2608.24982#S3.SS0.SSS0.Px1.p1.1)\.
- Moradi and Mudur \(2026\)M\. M\. Moradi and S\. MudurDiSCTT: consensus\-guided self\-curriculum for efficient test\-time adaptation in reasoning\.External Links:2603\.05357,[Link](https://arxiv.org/abs/2603.05357)Cited by:[Table 3](https://arxiv.org/html/2608.24982#S3.T3.2.15.1),[§5\.2](https://arxiv.org/html/2608.24982#S5.SS2.p1.1)\.
- Panet al\.\(2026\)J\. Pan, L\. Li, Y\. Peng, Y\. Tang, S\. Wang, Y\. Sun, H\. Wu, Q\. Huang, and H\. WangLearning to generate via understanding: understanding\-driven intrinsic rewarding for unified multimodal models\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 22174–22184\.Cited by:[§6\.2](https://arxiv.org/html/2608.24982#S6.SS2.p2.1),[Table 4](https://arxiv.org/html/2608.24982#S6.T4.2.11.1)\.
- Prabhudesaiet al\.\(2025\)M\. Prabhudesai, L\. Chen, A\. Ippoliti, K\. Fragkiadaki, H\. Liu, and D\. PathakMaximizing confidence alone improves reasoning\.External Links:2505\.22660,[Link](https://arxiv.org/abs/2505.22660)Cited by:[Table 1](https://arxiv.org/html/2608.24982#S2.T1.2.19.1),[§3](https://arxiv.org/html/2608.24982#S3.SS0.SSS0.Px2.p1.1),[§8](https://arxiv.org/html/2608.24982#S8.SS0.SSS0.Px1.p1.1)\.
- Prasadet al\.\(2025\)A\. Prasad, W\. Yuan, R\. Y\. Pang, J\. Xu, M\. Fazel\-Zarandi, M\. Bansal, S\. Sukhbaatar, J\. E\. Weston, and J\. YuSelf\-consistency preference optimization\.InProceedings of the 42nd International Conference on Machine Learning,A\. Singh, M\. Fazel, D\. Hsu, S\. Lacoste\-Julien, F\. Berkenkamp, T\. Maharaj, K\. Wagstaff, and J\. Zhu \(Eds\.\),Proceedings of Machine Learning Research, Vol\.267,pp\. 49737–49751\.External Links:[Link](https://proceedings.mlr.press/v267/prasad25a.html)Cited by:[Table 3](https://arxiv.org/html/2608.24982#S3.T3.2.18.1),[§5\.3](https://arxiv.org/html/2608.24982#S5.SS3.p1.1)\.
- Rentschler and Roberts \(2026\)M\. Rentschler and J\. RobertsReinforcement learning from meta\-evaluation: aligning language models without ground\-truth labels\.External Links:2601\.21268,[Link](https://arxiv.org/abs/2601.21268)Cited by:[§6\.2](https://arxiv.org/html/2608.24982#S6.SS2.p1.1),[Table 4](https://arxiv.org/html/2608.24982#S6.T4.2.7.1)\.
- Samantaet al\.\(2025\)A\. Samanta, A\. Magesh, R\. Wu, A\. Jain, Y\. Yu, D\. Jiang, B\. Vidolov, P\. Sajda, Y\. Efroni, and K\. HassaniSelf\-improvement of language models by post\-training on multi\-agent debate\.External Links:2509\.15172,[Link](https://arxiv.org/abs/2509.15172)Cited by:[Table 3](https://arxiv.org/html/2608.24982#S3.T3.2.19.1),[§5\.3](https://arxiv.org/html/2608.24982#S5.SS3.p1.1)\.
- Shafayatet al\.\(2025\)S\. Shafayat, F\. Tajwar, R\. Salakhutdinov, J\. Schneider, and A\. ZanetteCan large reasoning models self\-train?\.External Links:2505\.21444,[Link](https://arxiv.org/abs/2505.21444)Cited by:[§D\.2](https://arxiv.org/html/2608.24982#A4.SS2.p1.1),[Table 3](https://arxiv.org/html/2608.24982#S3.T3.2.10.1),[§5\.2](https://arxiv.org/html/2608.24982#S5.SS2.p1.1),[§7](https://arxiv.org/html/2608.24982#S7.SS0.SSS0.Px3.p1.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\.K\. Li, Y\. Wu, and D\. GuoDeepSeekMath: pushing the limits of mathematical reasoning in open language models\.External Links:2402\.03300,[Link](https://arxiv.org/abs/2402.03300)Cited by:[§1](https://arxiv.org/html/2608.24982#S1.p1.1)\.
- Shenet al\.\(2025\)Z\. Shen, H\. Chen, Y\. Tang, S\. Zhu, W\. Ye, X\. Hu, H\. Wang, G\. Chen, and J\. ZhaoCYCLE\-INSTRUCT: fully seed\-free instruction tuning via dual self\-training and cycle consistency\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 5123–5137\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.258/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.258),ISBN 979\-8\-89176\-332\-6Cited by:[Table 3](https://arxiv.org/html/2608.24982#S3.T3.2.5.1),[§5\.1](https://arxiv.org/html/2608.24982#S5.SS1.p2.1)\.
- Sim \(2025\)G\. SimWhen to ponder: adaptive compute allocation for code generation via test\-time training\.External Links:2601\.00894,[Link](https://arxiv.org/abs/2601.00894)Cited by:[§8](https://arxiv.org/html/2608.24982#S8.SS0.SSS0.Px2.p1.1)\.
- Singhet al\.\(2025\)A\. Singh, S\. Marjit, W\. Lin, P\. Gavrikov, S\. Yeung\-Levy, H\. Kuehne, R\. Feris, S\. Doveh, J\. Glass, and M\. J\. MirzaTTRV: test\-time reinforcement learning for vision language models\.External Links:2510\.06783,[Link](https://arxiv.org/abs/2510.06783)Cited by:[Table 2](https://arxiv.org/html/2608.24982#S2.T2.2.21.1),[§4\.3](https://arxiv.org/html/2608.24982#S4.SS3.p1.1),[§8](https://arxiv.org/html/2608.24982#S8.SS0.SSS0.Px1.p1.1)\.
- Songet al\.\(2026\)C\. Song, M\. Seo, Y\. Seong, D\. Kim, and C\. KimQuery\-conditioned test\-time self\-training for large language models\.External Links:2605\.13369,[Link](https://arxiv.org/abs/2605.13369)Cited by:[Table 3](https://arxiv.org/html/2608.24982#S3.T3.2.23.1),[§5\.2](https://arxiv.org/html/2608.24982#S5.SS2.p2.1)\.
- Srivastavaet al\.\(2025\)G\. Srivastava, Z\. Bi, M\. Lu, and X\. WangDEBATE, TRAIN, EVOLVE: Self\-Evolution of language model reasoning\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 32764–32810\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.1666/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1666),ISBN 979\-8\-89176\-332\-6Cited by:[Table 3](https://arxiv.org/html/2608.24982#S3.T3.2.11.1),[§5\.2](https://arxiv.org/html/2608.24982#S5.SS2.p1.1)\.
- Sui and Hooi \(2026\)Y\. Sui and B\. HooiConversation for non\-verifiable learning: self\-evolving LLMs through meta\-evaluation\.InProceedings of the 43rd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.306\.External Links:2601\.21464,[Link](https://arxiv.org/abs/2601.21464)Cited by:[§6\.2](https://arxiv.org/html/2608.24982#S6.SS2.p1.1),[Table 4](https://arxiv.org/html/2608.24982#S6.T4.2.6.1),[§7](https://arxiv.org/html/2608.24982#S7.SS0.SSS0.Px2.p1.1)\.
- Tanet al\.\(2026\)L\. S\. Tan, J\. Chen, X\. Fu, L\. Ma, J\. Huang, J\. Shi, Y\. Li, and L\. WenMeta\-ttrl: a metacognitive framework for self\-improving test\-time reinforcement learning in unified multimodal models\.External Links:2603\.15724,[Link](https://arxiv.org/abs/2603.15724)Cited by:[§6\.2](https://arxiv.org/html/2608.24982#S6.SS2.p1.1),[Table 4](https://arxiv.org/html/2608.24982#S6.T4.2.8.1)\.
- Taoet al\.\(2024\)Z\. Tao, T\. Lin, X\. Chen, H\. Li, Y\. Wu, Y\. Li, Z\. Jin, F\. Huang, D\. Tao, and J\. ZhouA survey on self\-evolution of large language models\.External Links:2404\.14387,[Link](https://arxiv.org/abs/2404.14387)Cited by:[Appendix E](https://arxiv.org/html/2608.24982#A5.p1.1),[§1](https://arxiv.org/html/2608.24982#S1.p3.1)\.
- Thawakaret al\.\(2025\)O\. Thawakar, S\. Venkatraman, R\. Thawkar, A\. Shaker, H\. Cholakkal, R\. M\. Anwer, S\. Khan, and F\. KhanEvoLMM: self\-evolving large multimodal models with continuous rewards\.External Links:2511\.16672,[Link](https://arxiv.org/abs/2511.16672)Cited by:[Table 2](https://arxiv.org/html/2608.24982#S2.T2.2.8.1),[§4\.3](https://arxiv.org/html/2608.24982#S4.SS3.p1.1)\.
- Tieet al\.\(2025\)G\. Tie, Z\. Zhao, D\. Song, F\. Wei, R\. Zhou, Y\. Dai, W\. Yin, Z\. Yang, J\. Yan, Y\. Su, Z\. Dai, Y\. Xie, Y\. Cao, L\. Sun, P\. Zhou, L\. He, H\. Chen, Y\. Zhang, Q\. Wen, T\. Liu, N\. Z\. Gong, J\. Tang, C\. Xiong, H\. Ji, P\. S\. Yu, and J\. GaoLarge language models post\-training: surveying techniques from alignment to reasoning\.External Links:2503\.06072,[Link](https://arxiv.org/abs/2503.06072)Cited by:[Appendix E](https://arxiv.org/html/2608.24982#A5.p1.1)\.
- van Niekerket al\.\(2025\)C\. van Niekerk, R\. Vukovic, B\. M\. Ruppik, H\. Lin, and M\. GašićPost\-training large language models via reinforcement learning from self\-feedback\.External Links:2507\.21931,[Link](https://arxiv.org/abs/2507.21931)Cited by:[§D\.3](https://arxiv.org/html/2608.24982#A4.SS3.p1.1),[Table 3](https://arxiv.org/html/2608.24982#S3.T3.2.21.1),[§5\.3](https://arxiv.org/html/2608.24982#S5.SS3.p1.1),[§7](https://arxiv.org/html/2608.24982#S7.SS0.SSS0.Px3.p1.1)\.
- Wanget al\.\(2026a\)H\. Wang, Y\. Yang, J\. Hu, M\. Zhu, and W\. ChenV\-zero: self\-improving multimodal reasoning with zero annotation\.External Links:2601\.10094,[Link](https://arxiv.org/abs/2601.10094)Cited by:[Table 3](https://arxiv.org/html/2608.24982#S3.T3.2.24.1),[§5\.2](https://arxiv.org/html/2608.24982#S5.SS2.p2.1)\.
- Wanget al\.\(2026b\)R\. Wang, W\. Huang, Q\. Cao, Y\. Iwasawa, Y\. Matsuo, and J\. GuoSELF\-HARMONY: LEARNING TO HARMONIZE SELF\-SUPERVISION AND SELF\-PLAY IN TEST\-TIME REINFORCEMENT LEARNING\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=ZzG6oJ5ehI)Cited by:[Table 2](https://arxiv.org/html/2608.24982#S2.T2.2.13.1),[§4\.2](https://arxiv.org/html/2608.24982#S4.SS2.p1.1),[§8](https://arxiv.org/html/2608.24982#S8.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2025a\)W\. Wang, Y\. Wang, K\. Chen, and H\. HuangBeyond majority voting: towards fine\-grained and more reliable reward signal for test\-time reinforcement learning\.External Links:2512\.15146,[Link](https://arxiv.org/abs/2512.15146)Cited by:[Table 2](https://arxiv.org/html/2608.24982#S2.T2.2.15.1),[§4\.2](https://arxiv.org/html/2608.24982#S4.SS2.p1.1),[§8](https://arxiv.org/html/2608.24982#S8.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2025b\)Y\. Wang, X\. Wang, C\. Wang, J\. Fang, Q\. Wang, J\. Chu, X\. Meng, S\. Yang, L\. Qin, Y\. Zhang, W\. Ye, and S\. ZhangTemporal self\-rewarding language models: decoupling chosen\-rejected via past\-future\.External Links:2508\.06026,[Link](https://arxiv.org/abs/2508.06026)Cited by:[§6\.1](https://arxiv.org/html/2608.24982#S6.SS1.p1.1),[Table 4](https://arxiv.org/html/2608.24982#S6.T4.2.5.1),[§9](https://arxiv.org/html/2608.24982#S9.SS0.SSS0.Px3.p1.1)\.
- Wanget al\.\(2023\)Y\. Wang, Y\. Kordi, S\. Mishra, A\. Liu, N\. A\. Smith, D\. Khashabi, and H\. HajishirziSelf\-instruct: aligning language models with self\-generated instructions\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 13484–13508\.External Links:[Link](https://aclanthology.org/2023.acl-long.754/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.754)Cited by:[§D\.1](https://arxiv.org/html/2608.24982#A4.SS1.p3.1),[§2](https://arxiv.org/html/2608.24982#S2.SS0.SSS0.Px6.p1.1)\.
- Wanget al\.\(2025c\)Z\. Wang, W\. He, Z\. Liang, X\. Zhang, C\. Bansal, Y\. Wei, W\. Zhang, and H\. YaoCREAM: consistency regularized self\-rewarding language models\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Vf6RDObyEF)Cited by:[§6\.1](https://arxiv.org/html/2608.24982#S6.SS1.p1.1),[Table 4](https://arxiv.org/html/2608.24982#S6.T4.2.3.1),[§9](https://arxiv.org/html/2608.24982#S9.SS0.SSS0.Px3.p1.1)\.
- Weiet al\.\(2021\)C\. Wei, K\. Shen, Y\. Chen, and T\. MaTheoretical analysis of self\-training with deep networks on unlabeled data\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=rC8sJ4i6kaH)Cited by:[§7](https://arxiv.org/html/2608.24982#S7.SS0.SSS0.Px1.p2.1)\.
- Weiet al\.\(2025\)L\. Wei, Y\. Li, C\. Wang, Y\. Wang, L\. Kong, W\. Huang, and L\. SunFirst SFT, second RL, third UPT: continual improving multi\-modal LLM reasoning via unsupervised post\-training\.InAdvances in Neural Information Processing Systems,Vol\.38\.External Links:[Document](https://dx.doi.org/10.52202/085713-2084),[Link](https://proceedings.neurips.cc/paper_files/paper/2025/hash/59ddfff7979b43f54690fa986c0e5138-Abstract-Conference.html)Cited by:[Table 2](https://arxiv.org/html/2608.24982#S2.T2.2.22.1),[§4\.3](https://arxiv.org/html/2608.24982#S4.SS3.p1.1),[§8](https://arxiv.org/html/2608.24982#S8.SS0.SSS0.Px1.p1.1)\.
- Wenet al\.\(2026a\)W\. Wen, tianwu zhi, K\. FAN, Y\. Li, X\. Peng, Y\. ZHANG, Y\. Liao, J\. Li, and L\. zhangSelf\-evolving vision\-language models for image quality assessment via voting and ranking\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=INOi0YqI8p)Cited by:[Table 2](https://arxiv.org/html/2608.24982#S2.T2.2.25.1),[§4\.2](https://arxiv.org/html/2608.24982#S4.SS2.p2.1)\.
- Wenet al\.\(2026b\)X\. Wen, H\. Yu, L\. Zhu, and G\. WangVerifier\-free rl for llms via intrinsic gradient\-norm reward\.External Links:2605\.09920,[Link](https://arxiv.org/abs/2605.09920)Cited by:[Table 1](https://arxiv.org/html/2608.24982#S2.T1.2.25.1),[§3](https://arxiv.org/html/2608.24982#S3.SS0.SSS0.Px3.p1.1)\.
- Wuet al\.\(2025\)J\. Wu, Y\. George, J\. Ye, Y\. Wu, D\. F\. Schmidt, and J\. CaiSPINE: token\-selective test\-time reinforcement learning with entropy\-band regularization\.External Links:2511\.17938,[Link](https://arxiv.org/abs/2511.17938)Cited by:[Table 2](https://arxiv.org/html/2608.24982#S2.T2.2.12.1),[§4\.2](https://arxiv.org/html/2608.24982#S4.SS2.p1.1)\.
- Wuet al\.\(2024\)T\. Wu, W\. Yuan, O\. Golovneva, J\. Xu, Y\. Tian, J\. Jiao, J\. Weston, and S\. SukhbaatarMeta\-rewarding language models: self\-improving alignment with llm\-as\-a\-meta\-judge\.External Links:2407\.19594,[Link](https://arxiv.org/abs/2407.19594)Cited by:[§6\.1](https://arxiv.org/html/2608.24982#S6.SS1.p1.1),[Table 4](https://arxiv.org/html/2608.24982#S6.T4.2.4.1),[§9](https://arxiv.org/html/2608.24982#S9.SS0.SSS0.Px3.p1.1)\.
- Wu \(2025\)X\. WuSailing by the stars: a survey on reward models and learning strategies for learning from rewards\.External Links:2505\.02686,[Link](https://arxiv.org/abs/2505.02686)Cited by:[Appendix E](https://arxiv.org/html/2608.24982#A5.p1.1)\.
- Wuet al\.\(2026\)Z\. Wu, K\. Shi, C\. Zhang, Z\. Liao, J\. Yang, N\. Yang, Q\. Peng, L\. Zhang, H\. Xu, T\. Su, Z\. Yang, H\. Lu, and H\. WangWhen models judge themselves: unsupervised self\-evolution for multimodal reasoning\.External Links:2603\.21289,[Link](https://arxiv.org/abs/2603.21289)Cited by:[§6\.2](https://arxiv.org/html/2608.24982#S6.SS2.p2.1),[Table 4](https://arxiv.org/html/2608.24982#S6.T4.2.10.1)\.
- Xieet al\.\(2026\)J\. Xie, A\. Tonioni, N\. Rauschmayr, F\. Tombari, and B\. SchieleSSL\-r1: self\-supervised visual reinforcement post\-training for multimodal large language models\.External Links:2604\.20705,[Link](https://arxiv.org/abs/2604.20705)Cited by:[Table 1](https://arxiv.org/html/2608.24982#S2.T1.2.27.1),[§3](https://arxiv.org/html/2608.24982#S3.SS0.SSS0.Px3.p1.1)\.
- Xinget al\.\(2025\)J\. Xing, C\. Tang, X\. Liu, D\. Xiong, S\. Huang, W\. Ju, J\. Lv, and Z\. QiaoRewarding the journey, not just the destination: a composite path and answer self\-scoring reward mechanism for test\-time reinforcement learning\.External Links:2510\.17923,[Link](https://arxiv.org/abs/2510.17923)Cited by:[Table 2](https://arxiv.org/html/2608.24982#S2.T2.2.16.1),[§4\.2](https://arxiv.org/html/2608.24982#S4.SS2.p1.1),[§8](https://arxiv.org/html/2608.24982#S8.SS0.SSS0.Px1.p1.1)\.
- Xionget al\.\(2024\)W\. Xiong, J\. Liu, I\. Molybog, H\. Zhang, P\. Bhargava, R\. Hou, L\. Martin, R\. Rungta, K\. A\. Sankararaman, B\. Oguz, M\. Khabsa, H\. Fang, Y\. Mehdad, S\. Narang, K\. Malik, A\. Fan, S\. Bhosale, S\. Edunov, M\. Lewis, S\. Wang, and H\. MaEffective long\-context scaling of foundation models\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 4643–4663\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.260),[Link](https://aclanthology.org/2024.naacl-long.260/)Cited by:[Table 1](https://arxiv.org/html/2608.24982#S2.T1.2.10.1),[§3](https://arxiv.org/html/2608.24982#S3.SS0.SSS0.Px1.p1.1)\.
- Xuet al\.\(2025a\)F\. Xu, H\. Yan, C\. Ma, H\. Zhao, Q\. Sun, K\. Cheng, J\. He, J\. Liu, and Z\. WuGenius: a generalizable and purely unsupervised self\-training framework for advanced reasoning\.External Links:2504\.08672,[Link](https://arxiv.org/abs/2504.08672)Cited by:[Table 3](https://arxiv.org/html/2608.24982#S3.T3.2.9.1),[§5\.2](https://arxiv.org/html/2608.24982#S5.SS2.p1.1)\.
- Xuet al\.\(2025b\)F\. Xu, Q\. Hao, Z\. Zong, J\. Wang, Y\. Zhang, J\. Wang, X\. Lan, J\. Gong, T\. Ouyang, F\. Meng, C\. Shao, Y\. Yan, Q\. Yang, Y\. Song, S\. Ren, X\. Hu, Y\. Li, J\. Feng, C\. Gao, and Y\. LiTowards large reasoning models: a survey of reinforced reasoning with large language models\.External Links:2501\.09686,[Link](https://arxiv.org/abs/2501.09686)Cited by:[Appendix E](https://arxiv.org/html/2608.24982#A5.p1.1),[§1](https://arxiv.org/html/2608.24982#S1.p3.1)\.
- Xuet al\.\(2026\)L\. Xu, C\. Chen, and F\. YinUnsupervised layer\-wise dynamic test time adaptation for llms\.External Links:2602\.09719,[Link](https://arxiv.org/abs/2602.09719)Cited by:[Table 1](https://arxiv.org/html/2608.24982#S2.T1.2.24.1),[§3](https://arxiv.org/html/2608.24982#S3.SS0.SSS0.Px4.p1.1),[§8](https://arxiv.org/html/2608.24982#S8.SS0.SSS0.Px2.p1.1)\.
- Xuet al\.\(2025c\)Y\. Xu, H\. Yao, Z\. Guo, P\. Li, A\. Liu, X\. Hu, W\. Guo, and H\. XiongYou only need 4 extra tokens: synergistic test\-time adaptation for llms\.External Links:2510\.10223,[Link](https://arxiv.org/abs/2510.10223)Cited by:[Table 1](https://arxiv.org/html/2608.24982#S2.T1.2.22.1),[§3](https://arxiv.org/html/2608.24982#S3.SS0.SSS0.Px4.p1.1),[§8](https://arxiv.org/html/2608.24982#S8.SS0.SSS0.Px2.p1.1)\.
- Yanet al\.\(2026\)D\. Yan, J\. Liang, Y\. Wang, S\. Lu, R\. He, and T\. TanWhat if consensus lies? selective\-complementary reinforcement learning at test time\.External Links:2603\.19880,[Link](https://arxiv.org/abs/2603.19880)Cited by:[Table 2](https://arxiv.org/html/2608.24982#S2.T2.2.17.1),[§4\.2](https://arxiv.org/html/2608.24982#S4.SS2.p1.1)\.
- Yanget al\.\(2026a\)C\. Yang, Z\. Xiang, Y\. Tang, Z\. Teng, C\. Huang, F\. Long, Y\. Liu, and J\. SuTTCS: test\-time curriculum synthesis for self\-evolving\.External Links:2601\.22628,[Link](https://arxiv.org/abs/2601.22628)Cited by:[Table 3](https://arxiv.org/html/2608.24982#S3.T3.2.14.1),[§5\.2](https://arxiv.org/html/2608.24982#S5.SS2.p1.1)\.
- Yanget al\.\(2026b\)H\. Yang, M\. Xerri, S\. Park, H\. Zhang, Y\. Feng, S\. A\. Kogilathota, and J\. ZhouSelf\-improvement of large language models: a technical overview and future outlook\.External Links:2603\.25681,[Link](https://arxiv.org/abs/2603.25681)Cited by:[Appendix E](https://arxiv.org/html/2608.24982#A5.p1.1)\.
- Yuet al\.\(2026\)Y\. Yu, Z\. Wu, Z\. Chen, H\. Xu, Z\. Liao, X\. Deng, Z\. Liu, S\. Shi, and H\. WangStabilizing unsupervised self\-evolution of mllms via continuous softened retracing resampling\.External Links:2604\.03647,[Link](https://arxiv.org/abs/2604.03647)Cited by:[Table 2](https://arxiv.org/html/2608.24982#S2.T2.2.24.1),[§4\.2](https://arxiv.org/html/2608.24982#S4.SS2.p2.1),[§9](https://arxiv.org/html/2608.24982#S9.SS0.SSS0.Px2.p1.1)\.
- Yuanet al\.\(2024\)W\. Yuan, R\. Y\. Pang, K\. Cho, X\. Li, S\. Sukhbaatar, J\. Xu, and J\. E\. WestonSelf\-rewarding language models\.InProceedings of the 41st International Conference on Machine Learning,R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research, Vol\.235,pp\. 57905–57923\.External Links:[Link](https://proceedings.mlr.press/v235/yuan24d.html)Cited by:[§1](https://arxiv.org/html/2608.24982#S1.p1.1),[§6\.1](https://arxiv.org/html/2608.24982#S6.SS1.p1.1),[Table 4](https://arxiv.org/html/2608.24982#S6.T4.2.2.1),[§7](https://arxiv.org/html/2608.24982#S7.SS0.SSS0.Px2.p1.1),[§8](https://arxiv.org/html/2608.24982#S8.SS0.SSS0.Px1.p1.1)\.
- Yuanet al\.\(2025\)W\. Yuan, S\. Tang, W\. Lin, J\. Ruan, G\. Cui, B\. Zhang, T\. Chen, T\. Liu, Y\. Fu, P\. Ye, and L\. BaiWisdom of the crowd: reinforcement learning from coevolutionary collective feedback\.External Links:2508\.12338,[Link](https://arxiv.org/abs/2508.12338)Cited by:[Table 2](https://arxiv.org/html/2608.24982#S2.T2.2.18.1),[§4\.2](https://arxiv.org/html/2608.24982#S4.SS2.p1.1)\.
- Zelikmanet al\.\(2024\)E\. Zelikman, G\. Harik, Y\. Shao, V\. Jayasiri, N\. Haber, and N\. D\. GoodmanQuiet\-STaR: language models can teach themselves to think before speaking\.InFirst Conference on Language Modeling,External Links:2403\.09629,[Link](https://openreview.net/forum?id=oRXPiSOGH9)Cited by:[Table 3](https://arxiv.org/html/2608.24982#S3.T3.2.7.1),[§5\.2](https://arxiv.org/html/2608.24982#S5.SS2.p1.1),[§7](https://arxiv.org/html/2608.24982#S7.SS0.SSS0.Px4.p1.1)\.
- Zenget al\.\(2025\)Z\. Zeng, Y\. Chen, S\. Yu, R\. Wang, Y\. Yan, Z\. Liu, S\. Wang, X\. Han, Z\. Liu, and M\. SunKBAlign: efficient self adaptation on specific textual knowledge bases\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 13519–13532\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.728/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.728),ISBN 979\-8\-89176\-335\-7Cited by:[Table 3](https://arxiv.org/html/2608.24982#S3.T3.2.4.1),[§5\.1](https://arxiv.org/html/2608.24982#S5.SS1.p2.1)\.
- Zhanget al\.\(2025a\)K\. Zhang, Q\. Yao, S\. Liu, Y\. Wang, B\. Lai, J\. Ye, M\. Song, and D\. TaoConsistent paths lead to truth: self\-rewarding reinforcement learning for llm reasoning\.External Links:2506\.08745,[Link](https://arxiv.org/abs/2506.08745)Cited by:[Table 2](https://arxiv.org/html/2608.24982#S2.T2.2.6.1),[§4\.1](https://arxiv.org/html/2608.24982#S4.SS1.p1.1)\.
- Zhanget al\.\(2026a\)N\. Zhang, W\. Ma, Z\. Ma, J\. Xu, J\. Gao, J\. Hao, R\. He, and J\. XuSilence the judge: reinforcement learning with self\-verifier via latent geometric clustering\.External Links:2601\.08427,[Link](https://arxiv.org/abs/2601.08427)Cited by:[Table 1](https://arxiv.org/html/2608.24982#S2.T1.2.26.1),[§3](https://arxiv.org/html/2608.24982#S3.SS0.SSS0.Px3.p1.1)\.
- Zhanget al\.\(2025b\)Q\. Zhang, H\. Wu, C\. Zhang, P\. Zhao, and Y\. BianRight question is already half the answer: fully unsupervised llm reasoning incentivization\.Note:Ongoing work\. First released on April 8, 2025External Links:2504\.05812,[Link](https://arxiv.org/abs/2504.05812)Cited by:[Table 2](https://arxiv.org/html/2608.24982#S2.T2.2.4.1),[§4\.1](https://arxiv.org/html/2608.24982#S4.SS1.p1.1)\.
- Zhanget al\.\(2025c\)X\. Zhang, B\. Peng, Y\. Tian, J\. Zhou, Y\. Zhang, H\. Mi, and H\. M\. MengSelf\-tuning: instructing LLMs to effectively acquire new knowledge through self\-teaching\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 5688–5724\.External Links:[Link](https://aclanthology.org/2025.findings-acl.297/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.297),ISBN 979\-8\-89176\-256\-5Cited by:[Table 3](https://arxiv.org/html/2608.24982#S3.T3.2.3.1),[§5\.1](https://arxiv.org/html/2608.24982#S5.SS1.p1.1)\.
- Zhanget al\.\(2025d\)Y\. Zhang, Z\. Zhang, H\. Guan, Y\. Cheng, Y\. Duan, C\. Wang, Y\. Wang, S\. Zheng, and J\. HeNo free lunch: rethinking internal feedback for llm reasoning\.External Links:2506\.17219,[Link](https://arxiv.org/abs/2506.17219)Cited by:[§3](https://arxiv.org/html/2608.24982#S3.SS0.SSS0.Px2.p1.1),[§9](https://arxiv.org/html/2608.24982#S9.SS0.SSS0.Px4.p1.1)\.
- Zhanget al\.\(2026b\)Z\. Zhang, J\. Zhu, X\. Ge, Z\. Zhao, Z\. Zhou, X\. Li, X\. Feng, J\. Yao, and B\. HanCo\-rewarding: stable self\-supervised RL for eliciting reasoning in large language models\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=fDk95XPsCU)Cited by:[Table 2](https://arxiv.org/html/2608.24982#S2.T2.2.7.1),[§4\.1](https://arxiv.org/html/2608.24982#S4.SS1.p1.1)\.
- Zhaoet al\.\(2025\)A\. Zhao, Y\. Wu, Y\. Yue, T\. Wu, Q\. Xu, Y\. Yue, M\. Lin, S\. Wang, Q\. Wu, Z\. Zheng, and G\. HuangAbsolute zero: reinforced self\-play reasoning with zero data\.External Links:2505\.03335,[Link](https://arxiv.org/abs/2505.03335)Cited by:[§D\.1](https://arxiv.org/html/2608.24982#A4.SS1.p2.1),[§1](https://arxiv.org/html/2608.24982#S1.p1.1),[§2](https://arxiv.org/html/2608.24982#S2.SS0.SSS0.Px6.p1.1)\.
- Zhaoet al\.\(2026a\)C\. Zhao, E\. Yang, Y\. Liu, J\. Zhao, and G\. GuoECHO: entropy\-confidence hybrid optimization for test\-time reinforcement learning\.External Links:2602\.02150,[Link](https://arxiv.org/abs/2602.02150)Cited by:[Table 2](https://arxiv.org/html/2608.24982#S2.T2.2.11.1),[§4\.2](https://arxiv.org/html/2608.24982#S4.SS2.p1.1),[§8](https://arxiv.org/html/2608.24982#S8.SS0.SSS0.Px1.p1.1)\.
- Zhaoet al\.\(2026b\)X\. Zhao, Z\. Kang, A\. Feng, S\. Levine, and D\. SongLearning to reason without external rewards\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=OU9nFEYR2M)Cited by:[Table 2](https://arxiv.org/html/2608.24982#S2.T2.2.5.1),[§4\.1](https://arxiv.org/html/2608.24982#S4.SS1.p1.1)\.
- Zhouet al\.\(2024\)Y\. Zhou, Z\. Fan, D\. Cheng, S\. Yang, Z\. Chen, C\. Cui, X\. Wang, Y\. Li, L\. Zhang, and H\. YaoCalibrated self\-rewarding vision language models\.External Links:2405\.14622,[Link](https://arxiv.org/abs/2405.14622)Cited by:[§D\.1](https://arxiv.org/html/2608.24982#A4.SS1.p4.1),[§6\.1](https://arxiv.org/html/2608.24982#S6.SS1.p1.1)\.
- Zhouet al\.\(2025\)Y\. Zhou, Z\. Liang, H\. Liu, W\. Yu, K\. Panaganti, L\. Song, D\. Yu, X\. Zhang, H\. Mi, and D\. YuEvolving language models without labels: majority drives selection, novelty promotes variation\.External Links:2509\.15194,[Link](https://arxiv.org/abs/2509.15194)Cited by:[Table 2](https://arxiv.org/html/2608.24982#S2.T2.2.20.1),[§4\.2](https://arxiv.org/html/2608.24982#S4.SS2.p1.1),[§9](https://arxiv.org/html/2608.24982#S9.SS0.SSS0.Px2.p1.1)\.
- Zuoet al\.\(2025\)Y\. Zuo, K\. Zhang, L\. Sheng, S\. Qu, G\. Cui, X\. Zhu, H\. Li, Y\. Zhang, X\. Long, E\. Hua, B\. Qi, Y\. Sun, Z\. Ma, L\. Yuan, N\. Ding, and B\. ZhouTTRL: test\-time reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.38\.External Links:[Document](https://dx.doi.org/10.52202/085713-4376),[Link](https://proceedings.neurips.cc/paper_files/paper/2025/hash/be690ea16f005c174f6c4102a5970e67-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2608.24982#S1.p1.1),[Table 2](https://arxiv.org/html/2608.24982#S2.T2.2.9.1),[§4\.2](https://arxiv.org/html/2608.24982#S4.SS2.p1.1),[§7](https://arxiv.org/html/2608.24982#S7.SS0.SSS0.Px2.p1.1),[§8](https://arxiv.org/html/2608.24982#S8.SS0.SSS0.Px1.p1.1)\.

## Appendix Contents

## Appendix ASurvey Protocol and Screening Details

### A\.1Search Sources and Query Strings

We searched ACL Anthology, arXiv, Semantic Scholar, and Google Scholar for work dated January 2023–May 2026\. The following frozen templates make the search reconstructable; each quoted mechanism phrase was run separately to avoid engine\-specific Boolean limits\.

- •ACL Anthology search:"<mechanism\>" \("large language model" OR LLM OR multimodal\)\.
- •arXiv API:\(ti:"<mechanism\>" OR abs:"<mechanism\>"\) AND \(all:"large language model" OR all:"multimodal large language model"\), with submitted\-date bounds 2023\-01\-01 and 2026\-05\-31\.
- •Semantic Scholar API:query="<mechanism\> large language model",year=2023\-2026,fieldsOfStudy=Computer Science\.
- •Google Scholar:"<mechanism\>" \("large language model" OR LLM OR MLLM\) \-survey, with a custom 2023–2026 year range\.

The mechanism list was:*unsupervised post\-training*,*self\-improvement*,*self\-rewarding*,*self\-training*,*test\-time training*,*test\-time adaptation*,*test\-time reinforcement learning*,*internal reward*,*self\-consistency*,*majority vote*,*intrinsic reward*, and*evaluator\-driven RL*\. Forward and backward snowballing expanded the four strands in §[2](https://arxiv.org/html/2608.24982#S2)\.

### A\.2Inclusion and Exclusion Criteria

Inclusion: a candidate is included as strict UPT iff it satisfies all four boundary checks[\(B1\)](https://arxiv.org/html/2608.24982#S2.SS0.SSS0.Px4)–[\(B4\)](https://arxiv.org/html/2608.24982#S2.SS0.SSS0.Px4)of §[2](https://arxiv.org/html/2608.24982#S2), has a verifiable algorithmic description in a paper, preprint, or extended technical report, and operates on foundation\-scale text or multimodal models\. Methods on smaller classifiers, image\-only encoders predating LLM/MLLM self\-improvement, or pure decoding\-time tricks without an explicit update are not included as strict UPT\. Tool\-grounded, verifier\-grounded, seed\-supervised, stronger\-teacher, and external\-evaluator methods are kept as adjacent methods\.Exclusion: surveys, benchmarks\-only papers, hardware/system papers, and unrelated domain\-specific applications without a post\-training contribution\.

### A\.3Screening Flow

Each candidate paper was reviewed with a structured note recording: update target \(parameters, adapters, memories, persistent local state, none\); signal source \(model samples, internal aggregates, internal evaluator, external verifier, external label\); whether any seed, tool, or stronger\-teacher signal entered the update loop; the internal update object against which the gradient is actually computed; and the timing regime \(offline corpus, full\-cohort transductive, few\-sample target, streaming continual, test\-time instance, within\-sequence, or no\-update inference\-time\)\. Boundary cases receive a source\-section recheck and a recorded assignment rationale in Appendix[D](https://arxiv.org/html/2608.24982#A4)\.

After deduplication, the inventory contains94 method records from 91 papers:80 strict rows from 78 papers,8 adjacent rows, and6 prose\-only boundary or antecedent records\. Multiple algorithmic variants in one paper are separate method rows\.

## Appendix BRepresentative Evidence and Task\-Structure Audit

Tablesandprovide the empirical and task\-structure detail referenced by the main\-text synthesis\. The first preserves each source paper’s reported setup; the second records structural priors separately from correctness\-bearing supervision\.

Table 7:Cross\-section of the strict inventory by family, primary timing regime, and update target\. Family I includes theSUDERbridge row\. Off\.: offline corpus; Coh\.: full\-cohort transductive; Few: few\-sample target; Str\.: streaming continual; Inst\.: test\-time instance; Seq\.: within\-sequence; Local: sample\-local state\. Each method record contributes once\.\{forest\}

Figure 3:Full update\-object taxonomy\. Each of the 80 strict UPT methods appears once under its primary family and sub\-class;‡marks the Family I/IV bridge cases \(SUDERandGvU\)\. The dashed adjacent branch places eight neighboring methods by the boundary they cross\.

## Appendix CFull Method Inventory

Figure[3](https://arxiv.org/html/2608.24982#A2.F3)maps all 80 strict methods by family, sub\-class, and update object\. Tables[1](https://arxiv.org/html/2608.24982#S2.T1)–[4](https://arxiv.org/html/2608.24982#S6.T4)compare method attributes; the tree provides the complete hierarchy in a single view\.

### C\.1Family, Timing, and Update Target

The[companion inventory](https://github.com/yeahjack/awesome-unsupervised-post-training)provides machine\-readable method records and per\-paper rationales\. Table[7](https://arxiv.org/html/2608.24982#A2.T7)aggregates its 80 strict rows along the timing and update\-target axes used in the main text\.

Offline updates dominate all four families \(59/80\)\. Full\-cohort transduction is concentrated in Family II \(7/11\), whereas all five sample\-local\-state methods occur in Families I and III\. The cross\-tabulation therefore connects the update\-object taxonomy to the deployment regimes in §[8](https://arxiv.org/html/2608.24982#S8)\.

## Appendix DBoundary Decisions

Adjacent methods are organized by the boundary check they fail\. The cases below cover the recurring ambiguities; stronger\-teacher distillation fails[\(B2\)](https://arxiv.org/html/2608.24982#S2.SS0.SSS0.Px4)–[\(B3\)](https://arxiv.org/html/2608.24982#S2.SS0.SSS0.Px4)directly\.

### D\.1Representative Adjacent Cases

No\-update inference\-time optimization\.EM\-INF\([Agarwal et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib2)\)performs inference\-time entropy descent over logits or hidden states without modifying parameters, adapters, memories, or persistent local state, failing[\(B1\)](https://arxiv.org/html/2608.24982#S2.SS0.SSS0.Px4)\. Training\-free multimodal prompting provides the same boundary test:See&Trek\([Li et al\., 2026](https://arxiv.org/html/2608.24982#bib.bib102)\)changes spatial prompt construction without updating model parameters or persistent state\. Likewise,SePer\([Dai et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib101)\)uses semantic\-perplexity reduction to measure retrieval utility; an internal model statistic does not satisfy[\(B1\)](https://arxiv.org/html/2608.24982#S2.SS0.SSS0.Px4)unless it drives an explicit update\.

Verifier\- or tool\-assisted self\-training\.T3RL\([Liao et al\., 2026](https://arxiv.org/html/2608.24982#bib.bib33)\)pairs majority\-vote selection with code\-interpreter verification;Absolute Zero\([Zhao et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib90)\)closes a propose\-solve loop with a code executor as the truth oracle\. Both fail[\(B3\)](https://arxiv.org/html/2608.24982#S2.SS0.SSS0.Px4)\. They are important neighboring evidence for false\-popular collapse fixes \(§[9](https://arxiv.org/html/2608.24982#S9)\)\.

Human\- or seed\-supervised bootstrapping\.Self\-Instruct\([Wang et al\., 2023](https://arxiv.org/html/2608.24982#bib.bib57)\)and instruction\-backtranslation pipelines\([Li et al\., 2023](https://arxiv.org/html/2608.24982#bib.bib30)\)bootstrap from human\-written seeds, failing[\(B3\)](https://arxiv.org/html/2608.24982#S2.SS0.SSS0.Px4)at the seed stage\. They are treated as precursors of self\-generated target bootstrapping rather than strict UPT\.

External reward or evaluator methods\.The fullCSR\([Zhou et al\., 2024](https://arxiv.org/html/2608.24982#bib.bib93)\)system includes a CLIP\-derived visual\-relevance term in its reward\. Because the evaluator is not derived from the same model lineage,CSRfails[\(B4\)](https://arxiv.org/html/2608.24982#S2.SS0.SSS0.Px4)\. It is routed to adjacent unless an internal\-only variant is analyzed separately\.

### D\.2Family II vs\. Family III

LRM Self\-Train\([Shafayat et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib47)\)uses majority vote to filter candidate solutions before SFT on the survivors\. Because the gradient is computed against the kept solutions \(self\-generated targets\), not against the consensus statistic itself, the update\-object rule assigns it to Family III\. The same rule placesTTRL,RoiRL, and methods whose gradient is computed directly againstr=𝟏\[y=maj\]r=\\mathbf\{1\}\[y=\\mathrm\{maj\}\]in Family II\.

### D\.3Family III vs\. Family IV

Confident ST\([Jang et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib60)\)andRLSF\([van Niekerk et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib56)\)use a self\-confidence or self\-rated score on candidate trajectories\. When the score acts as a selection mask before SFT or DPO, the gradient is computed against the kept generations \(Family III\)\. When the score itself appears as the scalar reward in PG, the gradient is computed through the evaluator \(Family IV\)\. The update\-object rule assigns each method according to its actual gradient path; the companion inventory records the mapping\.

### D\.4Strict UPT vs\. Adjacent Methods

ECHOandSPINEuse multi\-sample consensus as a selection mask while a separate intrinsic term shapes advantages or selects gradient\-receiving tokens\. Their consensus reward places them in Family II, with the intrinsic term acting as a within\-family modulation\.EM\-INFand the fullCSRare routed to adjacent for the reasons in Appendix[D](https://arxiv.org/html/2608.24982#A4)\.

## Appendix EComparison with Existing Surveys

Tableplaces this survey against the closest surveys along four scope dimensions\. Prior work surveys self\-feedback and self\-improvement\([Tao et al\., 2024](https://arxiv.org/html/2608.24982#bib.bib54);[Liang et al\., 2024](https://arxiv.org/html/2608.24982#bib.bib96);[Deng et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib99);[Yang et al\., 2026b](https://arxiv.org/html/2608.24982#bib.bib100)\), broad LLM post\-training and reward learning\([Kumar et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib29);[Tie et al\., 2025](https://arxiv.org/html/2608.24982#bib.bib97);[Wu, 2025](https://arxiv.org/html/2608.24982#bib.bib98)\), test\-time adaptation\([Liang et al\., 2023](https://arxiv.org/html/2608.24982#bib.bib32)\), and reinforced reasoning\([Xu et al\., 2025b](https://arxiv.org/html/2608.24982#bib.bib74);[Chen et al\., 2025b](https://arxiv.org/html/2608.24982#bib.bib7)\)\. These scopes overlap with parts of strict UPT, but none makes all four dimensions a joint inclusion rule\. Our distinctive unit of analysis is the internal object consumed by an explicit update under the no\-external\-signal boundary, across text and multimodal models\.