Online State-Representation Adaptation under Synthetic Interaction-Cue Remapping
Summary
This paper studies online state-representation adaptation under synthetic interaction-cue remapping, showing that continuing to update a logistic state predictor yields 93.39% late action accuracy versus 64.63% when frozen, while a contextual bandit and a direct policy achieve even higher performance depending on thresholds and feedback information.
View Cached Full Text
Cached at: 11/19/26, 09:03 AM
# Online State-Representation Adaptation under Synthetic Interaction-Cue Remapping
Source: [https://arxiv.org/html/2611.23836](https://arxiv.org/html/2611.23836)
###### Abstract
When cues change what they predict, continuing to update a learned mapping may help an agent choose the appropriate action\. We tested this in a one\-step synthetic task by comparing a logistic state predictor that continued to learn from state labels with the same predictor frozen at transfer\. We evaluated ten conditions in six held\-out environments, with 1,296 paired runs per condition and 1,296,000 transfer records in total\. Late action accuracy was93\.3993\.39% with online updates and64\.6364\.63% with frozen coefficients, a paired difference of28\.76\[8\.73,53\.19\]28\.76\\,\[8\.73,53\.19\]percentage points \(95% environment\-bootstrap interval\)\. A direct policy reached95\.6595\.65% using the same predictions with a different action threshold\. A contextual bandit reached94\.2394\.23% and earned more cumulative utility than the adaptive predictor\. The bandit used the reward of its chosen action, which in this task reveals the true state when combined with that action\. Weights based on development history lowered cumulative utility relative to neutral weights \(difference−3\.45\[−4\.48,−2\.43\]\-3\.45\\,\[\-4\.48,\-2\.43\]\)\. The adaptive predictor outperformed its frozen counterpart in late accuracy in each of the six test environments\. The other comparisons show how the choice of action threshold and the information available in feedback affect what can be attributed to online adaptation\.
Keywordsonline adaptation, concept drift, cue remapping, state representation, contextual bandit
## 1Introduction
A cue that predicts support during training may predict threat after deployment\. This change in the association between observations and the target state is a form of concept drift\[[1](https://arxiv.org/html/2611.23836#bib.bib1)\]\. An online learner can respond by revising its state predictions as new labels arrive\. In a decision task, however, action performance also depends on how an agent translates its predictions into actions\. Measuring both state prediction and action performance helps identify which part of the learner changed\.
We test this distinction in a synthetic task where an agent chooses whether to approach or avoid after observing six cues\. At transfer, the cues can change their association with the underlying state while the action rule and exploration schedule remain the same for the adaptive and frozen predictors\. The main comparison asks how much continuing the same supervised update improves performance\. A second comparison uses the same predictions across learners and changes the decision threshold\. Two value learners then show how performance varies when algorithms learn from different feedback\.
The task also separates early exposure from the costs of current actions\. Development histories differ in how often the supportive state occurs\. Each history is crossed with each current payoff profile, so a learner’s decision rule can be based on the current payoff, the earlier state frequencies, or a fixed reference\. This design makes it possible to ask whether the tested history rule improves decisions on the same transfer streams\.
## 2Methods
### 2\.1Task and transfer design
In each episode, the simulator samples a binary statess, wheres=0s=0denotes threat ands=1s=1denotes support\. It then generates a vector of six Gaussian cuesxx\. The learner observesxxand chooses either avoidance \(a=0a=0\) or approach \(a=1a=1\)\. The simulator scores the decision using the true state\. An action changes that episode’s payoff\. Later states and cues are drawn independently of the learner’s actions\.
Each run starts with 160 development episodes and continues for 100 transfer episodes\. During development, the probability of support is 0\.8, 0\.5, or 0\.2, corresponding to supportive, ordinary, and adverse exposure histories\. At transfer, the cue loadings and observation noise change\. Depending on the environment, some cue loadings reverse sign or fall to zero, a previously weak cue becomes informative, and the support probability can drift\. Environments also specify the probability of a feedback label flip and the feedback delay\. Appendix[A](https://arxiv.org/html/2611.23836#A1)gives the generator and the six test configurations\.
The payoff weightswsw\_\{s\}andwaw\_\{a\}control the costs and benefits of actions in threat and support, respectively\. Table[1](https://arxiv.org/html/2611.23836#S2.T1)gives the payoffU\(s,a\)U\(s,a\)for a realized state and action\. The three transfer profiles vary these weights independently of exposure history\. Safety priority uses\(ws,wa\)=\(2\.6,1\.2\)\(w\_\{s\},w\_\{a\}\)=\(2\.6,1\.2\), balanced uses\(2\.0,1\.5\)\(2\.0,1\.5\), and affiliation priority uses\(1\.4,1\.8\)\(1\.4,1\.8\)\. Each run uses one profile throughout transfer\. Evaluation uses that profile even when a learner chooses actions using other weights\.
Table 1:Payoff for each true state and action\. The action that matches the state earns a positive payoff\.
### 2\.2Paired evaluation
We crossed six test environments \(IDs 200–205\), three development histories, and three current payoff profiles, using 24 seed indices per environment and history\. For each environment, history, and seed index, one random seed generates the states, cues, and label flips\. A separate seed generates exploration and tie\-breaking draws\. The same streams are reused across the three payoff profiles and all model conditions\.
The 432 distinct seed pairs therefore give432×3=1,296432\\times 3=1\{,\}296units for pairing model outcomes\. One such unit is an environment, history, payoff profile, and seed index\. Across ten model conditions and 100 transfer episodes per run, the archive contains 1,296,000 transfer records\. The six environment configurations are the highest level of aggregation for the reported uncertainty\.
### 2\.3Learners and decision rules
The adaptive state learner estimates the probability of support asq\(x\)=σ\(β⊤ϕ\)q\(x\)=\\sigma\(\\beta^\{\\top\}\\phi\), whereϕ=\(1,x⊤\)⊤\\phi=\(1,x^\{\\top\}\)^\{\\top\}andσ\\sigmais the logistic function\. When an observed state labels~\\tilde\{s\}becomes available, the learner takes an update using the cue vector for that label and the current coefficients:
β←β\+η\[s~−σ\(β⊤ϕ\)\]ϕ\.\\beta\\leftarrow\\beta\+\\eta\\big\[\\tilde\{s\}\-\\sigma\(\\beta^\{\\top\}\\phi\)\\big\]\\phi\.\(1\)The frozen state learner has the same development training and keeps its development coefficients throughout transfer\. Both learners choose the action with higher estimated expected utility under the current weights\. For this rule, approach is preferred whenq\(x\)q\(x\)exceeds the threshold in Appendix[B](https://arxiv.org/html/2611.23836#A2)\.
The direct policy uses the same logistic update and the same label stream as the adaptive state learner\. It usesq\(x\)q\(x\)as a score for approach and applies a threshold of 0\.5\. The four weighting variants also share the adaptive state learner’s predictions\. They compute expected utility with equal weights, fixed balanced weights, weights derived from the realized fraction of true support states during development, or weights deliberately mismatched to the current profile\. These variants change which action the learner takes at a given predicted probability\.
The full\-information value learner fits a linear value for each action\. It calculates a target payoff for both actions from the observed state labels~\\tilde\{s\}and the phase’s payoff weights\. The contextual bandit instead uses the realized payoff of its chosen action to update that action’s value\. This is the chosen\-action feedback interface of a contextual bandit\[[2](https://arxiv.org/html/2611.23836#bib.bib2)\]\. In this simulator, a positive payoff means thata=sa=sand a negative payoff means thata≠sa\\neq s\. Thus the sign and the chosen action identify the true state even when the label available to other learners has been flipped\. The bandit update uses the reward directly\.
At transfer, all nine learned conditions share exploration events with probability 0\.05, the uniformly sampled actions on those events, and the draws used to break ties\. Each action is chosen before using any feedback due in that episode\. The tenth condition is a Bayes utility oracle\. It uses the simulator’s observation model and current support prior to estimateq\(x\)q\(x\), then chooses the action with higher expected utility without exploration\. The reported accuracy and utility of the oracle are realized outcomes on the same streams\.
### 2\.4Validation, outcomes, and uncertainty
We used four separate validation environments \(IDs 100–103\) to select learning rates\. Each candidate used eight seed indices per environment and history across all three payoff profiles, for 288 validation units\. We evaluated three rates for the adaptive mapping and three for each value learner, selecting by mean cumulative utility with late accuracy as the first tie\-breaker\. Validation selected 0\.12 for the adaptive mapping and 0\.05 for each value learner\. The frozen mapper, direct policy, and weighting variants used the adaptive mapper’s rate\.
An action is correct whena=sa=s\. Early accuracy uses transfer episodes 1–20 and late accuracy uses episodes 81–100\. Cumulative utility sums realized payoffs over all 100 transfer episodes\. For the late conditional error rates, we divide approaches in threat by the number of threat episodes and avoidances in support by the number of support episodes\. We compute these ratios for each run before averaging\. For methods that outputq\(x\)q\(x\), the Brier score measures mean squared probability error against true states over the full transfer period\. Appendix[C](https://arxiv.org/html/2611.23836#A3)also defines the accuracy area and the first threshold crossing\.
For a performance estimate, we average the run\-level metric within each environment, then give the six environment means equal weight\. For a contrast, we subtract the two models’ scores within each paired unit before averaging the differences within environments\. Each of 2,000 bootstrap replicates draws six environment means with replacement and averages them\[[3](https://arxiv.org/html/2611.23836#bib.bib3)\]\. The interval gives the 2\.5th and 97\.5th percentiles of those averages\. It describes variation when resampling the six fixed test configurations, with the pairing preserved within each configuration\.
## 3Results
### 3\.1Online updates improve state prediction and late action accuracy
The adaptive and frozen learners began transfer with the same cue\-to\-state mapping\. Over episodes 81–100, the adaptive learner reached93\.3993\.39% accuracy and the frozen learner reached64\.6364\.63% \(Table[2](https://arxiv.org/html/2611.23836#S3.T2)\)\. The paired difference was28\.76\[8\.73,53\.19\]28\.76\\,\[8\.73,53\.19\]percentage points and the cumulative utility difference was80\.71\[13\.23,166\.42\]80\.71\\,\[13\.23,166\.42\]\(Table[3](https://arxiv.org/html/2611.23836#S3.T3)\)\. The adaptive learner had higher late accuracy in all six environments, with gains that varied across configurations \(Table[4](https://arxiv.org/html/2611.23836#A1.T4)\)\.
Mean Brier score was0\.05000\.0500with online updates and0\.24730\.2473with frozen coefficients \(Table[7](https://arxiv.org/html/2611.23836#A3.T7)\)\. This improvement in state prediction accompanies the action gains under the shared payoff rule\. The moving accuracy curves show when the average action performance separated after transfer \(Figure[1](https://arxiv.org/html/2611.23836#S3.F1)\)\.
Table 2:Late action accuracy and cumulative utility across the ten conditions\. Entries are equally weighted environment means with 95% environment\-bootstrap intervals\. Late accuracy uses episodes 81–100 and utility sums episodes 1–100\.Figure 1:Action accuracy during transfer\. Each point summarizes the ten episodes ending at that position\. Accuracy is averaged within each run, then within each environment and across the six environments\. The four variants that change payoff weights are reported in the tables\.
### 3\.2Performance under different action rules and feedback
Using the same state predictions, the direct policy reached95\.6595\.65% late accuracy\. The adaptive\-minus\-direct accuracy difference was−2\.26\[−2\.76,−1\.71\]\-2\.26\\,\[\-2\.76,\-1\.71\]percentage points\. The adaptive state learner approached less often in threat \(3\.093\.09% versus4\.124\.12%\) and avoided more often in support \(10\.0110\.01% versus4\.624\.62%\)\. Both differences follow the higher approach threshold used by the adaptive state learner\. Its cumulative utility difference from the direct policy was0\.92\[−1\.60,4\.15\]0\.92\\,\[\-1\.60,4\.15\]\.
The two value learners fit action payoffs directly\. The full\-information learner, which derives both action targets from the same observed labels as the logistic learners, reached87\.8787\.87% late accuracy\. Its late missed\-support rate was20\.9520\.95%, compared with10\.0110\.01% for the adaptive state learner\. The contextual bandit reached94\.2394\.23% late accuracy and98\.0398\.03cumulative utility, compared with94\.1094\.10cumulative utility for the adaptive state learner\. Adaptive\-minus\-bandit differences were−0\.84\[−1\.45,−0\.37\]\-0\.84\\,\[\-1\.45,\-0\.37\]percentage points and−3\.93\[−7\.17,−0\.89\]\-3\.93\\,\[\-7\.17,\-0\.89\]in cumulative utility \(Table[3](https://arxiv.org/html/2611.23836#S3.T3)\)\.
### 3\.3The tested history weights lowered cumulative utility
The history\-weight variant uses the same state predictions as the neutral and fixed\-weight variants\. Relative to neutral weights, the history variant had a late accuracy difference of−1\.15\[−1\.52,−0\.79\]\-1\.15\\,\[\-1\.52,\-0\.79\]percentage points and a cumulative utility difference of−3\.45\[−4\.48,−2\.43\]\-3\.45\\,\[\-4\.48,\-2\.43\]\. The history variant also earned less cumulative utility than the fixed balanced variant \(difference−2\.55\[−2\.93,−2\.09\]\-2\.55\\,\[\-2\.93,\-2\.09\]\)\.
Using the actual current weights improved cumulative utility over deliberately mismatched weights by2\.22\[0\.20,4\.81\]2\.22\\,\[0\.20,4\.81\], while lowering late accuracy\. The same prediction can thus lead to different outcomes depending on which errors the rule penalizes\. Table[3](https://arxiv.org/html/2611.23836#S3.T3)and Figure[2](https://arxiv.org/html/2611.23836#S3.F2)show these changes alongside the gain from online updating\.
Table 3:Paired differences between model conditions with 95% environment\-bootstrap intervals\. Each entry subtracts the right\-hand condition from the left\-hand condition within each of 1,296 units before averaging by environment\. Accuracy differences are in percentage points\.Figure 2:Paired mean differences and their 95% environment\-bootstrap intervals\. The top row shows the comparison with the frozen learner\. Separate horizontal scales in the lower row make the six smaller contrasts visible\. Each column uses the units given above it\.
## 4Discussion
The decrease in Brier score shows that continued supervised updates reduced probability error under the changed cue distribution\. The task is favorable to a logistic predictor in one respect\. At a given episode, the two state\-conditional cue distributions have opposite means and equal covariance, so the true log odds are linear in the cues\. Transfer changes the coefficients of this relationship and can also change the intercept as the state prior drifts\. The six configurations combine changes to cues, priors, and feedback, so their results do not estimate the effect of each perturbation in isolation\.
The action results show why state predictions and decision rules need separate evaluation\. The direct policy’s higher accuracy came from a lower approach threshold and fewer missed support episodes\. Its lower point estimate for cumulative utility comes with an interval for the difference that spans zero\. Likewise, the history weights changed the decision threshold without changing predictions\. This particular rule carried a summary of past exposure into a setting where the current costs varied independently\. Its lower cumulative utility addresses that fixed weighting rule\. Other ways of using history, such as estimating a changing state prior or learning how costs vary over time, would require their own comparisons\.
The bandit comparison concerns feedback content as well as the update rule\. The reward was computed from the true state, while three of the six environments flipped some observed labels for supervised learners\. The payoff sign gives the bandit an informative, uncorrupted signal about whether its action matched the state\. The present design cannot estimate the separate contributions of that signal, the action\-specific value updates, and the policy\. A comparison designed to isolate feedback access would need to control what can be inferred about the state from each feedback channel\.
The exposure manipulation also defines what the task can say about experience\. Developmental robotics studies how early experience shapes behavior as an agent interacts with its environment\[[4](https://arxiv.org/html/2611.23836#bib.bib4)\]\. Here, early experience varies through the frequency of states during training, and a learner’s actions affect only the current payoff\. Studying adaptation with the mutual agent–environment coupling modeled by Beer\[[5](https://arxiv.org/html/2611.23836#bib.bib5)\]would require evaluating decisions that alter future observations\. The present results provide a comparison for the one\-step case with exogenous cues\.
## Data and code availability
The code and synthetic data are available on Zenodo\[[6](https://arxiv.org/html/2611.23836#bib.bib6)\]at[DOI 10\.5281/zenodo\.24329690](https://doi.org/10.5281/zenodo.24329690)\. The archive contains the configuration, simulator, analysis scripts, and 1,296,000 original transfer outcome records\. These records contain the true state, action, realized utility, and the probability estimate where available\. A separate export regenerates 69,120 development and 43,200 transfer inputs from the 432 seed pairs\. Each input includes the cues and feedback label\. Transfer inputs join to outcomes by environment, history, seed index, and episode\.
The archive’s README gives commands for rerunning the simulation and regenerating the tables and figures\. The archived deterministic fields agree with a full rerun at absolute tolerance10−1210^\{\-12\}using Python 3\.13\.9, NumPy 2\.3\.5, and pandas 2\.3\.3\. The verification reports trace the tables and plot coordinates to the result files\.
## Generative AI disclosure
OpenAI Codex assisted with drafting and editing the English text, preparing the LaTeX manuscript, and building and running packaging and verification scripts for this version\.
## References
- \[1\]J\. Gama, I\. Žliobaitė, A\. Bifet, M\. Pechenizkiy, and A\. Bouchachia\. A survey on concept drift adaptation\.*ACM Computing Surveys*, 46\(4\), Article 44, 2014\.[doi:10\.1145/2523813](https://doi.org/10.1145/2523813)\.
- \[2\]L\. Li, W\. Chu, J\. Langford, and R\. E\. Schapire\. A contextual\-bandit approach to personalized news article recommendation\. In*Proceedings of the 19th International Conference on World Wide Web*, pages 661–670, 2010\.[doi:10\.1145/1772690\.1772758](https://doi.org/10.1145/1772690.1772758)\.
- \[3\]B\. Efron\. Bootstrap methods: Another look at the jackknife\.*The Annals of Statistics*, 7\(1\):1–26, 1979\.[doi:10\.1214/aos/1176344552](https://doi.org/10.1214/aos/1176344552)\.
- \[4\]A\. Cangelosi and M\. Schlesinger\.*Developmental Robotics: From Babies to Robots*\. MIT Press, 2015\.[ISBN 9780262028011](https://mitpress.mit.edu/9780262028011/developmental-robotics/)\.
- \[5\]R\. D\. Beer\. A dynamical systems perspective on agent\-environment interaction\.*Artificial Intelligence*, 72\(1–2\):173–215, 1995\.[doi:10\.1016/0004\-3702\(94\)00005\-L](https://doi.org/10.1016/0004-3702(94)00005-L)\.
- \[6\]B\. Yin\.*Online State\-Representation Adaptation under Synthetic Interaction\-Cue Remapping: Code and Synthetic Data*\. Version 1\.0\.0\. Zenodo, 2026\.[doi:10\.5281/zenodo\.24329690](https://doi.org/10.5281/zenodo.24329690)\.
## Appendix ACue generation and random streams
Letℓj\\ell\_\{j\}andσj\\sigma\_\{j\}denote the loading and noise standard deviation of cuejjin a given phase\. Given the true statests\_\{t\}of episodett, the simulator draws cues independently according to
xtj=\(2st−1\)ℓj\+etj,etj∼𝒩\(0,σj2\)\.x\_\{tj\}=\(2s\_\{t\}\-1\)\\ell\_\{j\}\+e\_\{tj\},\\qquad e\_\{tj\}\\sim\\mathcal\{N\}\(0,\\sigma\_\{j\}^\{2\}\)\.\(2\)The support probability is fixed by history during development\. In transfer it follows a linear schedule fromp0p\_\{0\}top1p\_\{1\}\. The simulator draws states independently across episodes given this schedule\.
The generator is seeded with10,000\+e10\{,\}000\+efor a validation environment and20,000\+e20\{,\}000\+efor a test environment, whereeeis the environment ID\. Development loadings are uniform on\[0\.65,1\.20\]\[0\.65,1\.20\], with the sixth set to 0\.05\. Development noise standard deviations are uniform on\[0\.55,0\.90\]\[0\.55,0\.90\]\. At transfer the generator selects zero, two, or four of the first five cues for sign reversal, then multiplies one loading by 0\.20\. In even\-numbered environments it sets another loading to zero\. A zero loading removes the cue’s dependence on state while retaining observation noise\. Since the generator can zero a selected loading, the selected flip count may exceed the number of sign\-reversed cues that remain informative\. The sixth cue receives a random sign and a loading magnitude drawn uniformly from\[0\.75,1\.15\]\[0\.75,1\.15\]\. Each transfer noise standard deviation is the development value multiplied by a draw from\[0\.90,1\.30\]\[0\.90,1\.30\]\.
The generator drawsp0p\_\{0\}uniformly from\[0\.38,0\.62\]\[0\.38,0\.62\]and applies an ID\-dependent drift of−0\.15\-0\.15,00, or0\.150\.15to determinep1p\_\{1\}, clipped to\[0\.20,0\.80\]\[0\.20,0\.80\]\. The state label available for supervised feedback is independently flipped with probability zero or 0\.05, depending on the environment\. Label flips and an environment\-specific delay of zero or two episodes apply in both development and transfer\. Table[4](https://arxiv.org/html/2611.23836#A1.T4)lists the test configurations and the resulting late accuracy of the adaptive and frozen learners\.
Table 4:Held\-out environments and mean late action accuracy \(%\)\. The missing index is zero\-based\. A dash indicates that all cues retain nonzero state loadings\. The flip count records selections before weakening or zeroing\. Label flip gives the probability of flipping supervised state labels\. Delay is measured in episodes\.Within an environment, leth=0,1,2h=0,1,2index supportive, ordinary, and adverse histories and letiibe the seed index\. The environment seed is20,260,802\+100,000e\+10,000h\+i20\{,\}260\{,\}802\+100\{,\}000e\+10\{,\}000h\+i\. The policy seed is the environment seed plus900,000,000900\{,\}000\{,\}000\. This construction preserves the same observation and exploration streams for all model conditions and payoff profiles within each environment, history, and seed index\.
## Appendix BDecision thresholds and update timing
Given an estimateqqof the probability of support and the payoff in Table[1](https://arxiv.org/html/2611.23836#S2.T1), the expected utility rule prefers approach when
q\>2\.45ws2\.45ws\+1\.35wa\.q\>\\frac\{2\.45w\_\{s\}\}\{2\.45w\_\{s\}\+1\.35w\_\{a\}\}\.\(3\)The matched and frozen conditions use the current utility weights\. Neutral weights are\(1,1\)\(1,1\)and fixed balanced weights are\(2,1\.5\)\(2,1\.5\)\. Equal weights still yield an approach threshold above 0\.5 because the payoffs are asymmetric\. The history rule uses\(1\+2\(1−s¯dev\),1\+s¯dev\)\(1\+2\(1\-\\bar\{s\}\_\{\\mathrm\{dev\}\}\),\\,1\+\\bar\{s\}\_\{\\mathrm\{dev\}\}\), wheres¯dev\\bar\{s\}\_\{\\mathrm\{dev\}\}is the fraction of*true*support states in the 160 development episodes\. This gives the rule access to a simulator quantity that could be unavailable when labels are noisy\. The resulting weights remain fixed during transfer\. Mismatched weights swap the current safety and affiliation weights, including the balanced case\(1\.5,2\.0\)\(1\.5,2\.0\)\.
The value learners use the augmented cue vectorϕ\\phiand one coefficient vectorvav\_\{a\}per action\. On receipt of a targetrar\_\{a\}for actionaa, the update is
va←va\+α\(ra−va⊤ϕ\)ϕ\.v\_\{a\}\\leftarrow v\_\{a\}\+\\alpha\(r\_\{a\}\-v\_\{a\}^\{\\top\}\\phi\)\\phi\.\(4\)The full\-information learner applies this update to both actions withra=U\(s~,a\)r\_\{a\}=U\(\\tilde\{s\},a\)\. The bandit applies it only to the selected action, withra=U\(s,a\)r\_\{a\}=U\(s,a\)\. Both use balanced payoff weights for development targets and the current profile for transfer targets\. The greedy action has the higher predicted valueva⊤ϕv\_\{a\}^\{\\top\}\\phi\.
All learned coefficients start at zero\. At the transfer boundary, remaining development feedback is processed before the first transfer decision\. During transfer, the learner chooses an action with its current coefficients before processing the feedback that is due in that episode\. With delaydd, the feedback for episodettupdates the learner after the decision att\+dt\+d, so it can first affect the action att\+d\+1t\+d\+1\. The frozen learner retains its development coefficients\. Transfer feedback still in the queue at the end of transfer has no effect on the reported decisions\.
The implementation randomizes numerical ties using shared draws\. On exploration episodes the learned conditions use the same draw from the uniform action distribution\. The Bayes oracle instead uses the known observation\-model parameters and support priorptp\_\{t\}to compute the posterior estimate
q∗=σ\(logpt1−pt\+2∑j=16ℓjxtjσj2\)\.q^\{\*\}=\\sigma\\\!\\left\(\\log\\frac\{p\_\{t\}\}\{1\-p\_\{t\}\}\+2\\sum\_\{j=1\}^\{6\}\\frac\{\\ell\_\{j\}x\_\{tj\}\}\{\\sigma\_\{j\}^\{2\}\}\\right\)\.\(5\)It acts on this posterior and the current weights before observing the true state\. Sigmoid evaluations clip the logit to\[−30,30\]\[\-30,30\]\.
Validation tested mapping rates 0\.04, 0\.08, and 0\.12 and value rates 0\.01, 0\.03, and 0\.05\. If two candidates tied on cumulative utility and late accuracy, the lower rate was preferred\. The archived validation grid gives each family’s candidate scores\.
## Appendix CSupplementary outcomes
Letctc\_\{t\}equal 1 whenat=sta\_\{t\}=s\_\{t\}and 0 otherwise\. The normalized accuracy area is the trapezoidal sum ofctc\_\{t\}over all 100 transfer episodes divided by 99\. The first threshold crossing is the earliest ending episode of a complete ten\-episode window with meanctc\_\{t\}at least 0\.8\. It is coded as 101 if the threshold is never reached\. This metric describes the first successful window, including the possibility of a brief crossing in a run with lower accuracy later\.
Table[5](https://arxiv.org/html/2611.23836#A3.T5)reports late conditional action errors\. All archived late windows contain at least one episode of each state\. Table[6](https://arxiv.org/html/2611.23836#A3.T6)reports early accuracy, accuracy area, and first threshold crossing\. Table[7](https://arxiv.org/html/2611.23836#A3.T7)reports mean utility within each true state and the Brier score\. Both use all 100 transfer episodes\. The six online logistic conditions produce identical Brier point estimates because they share predictions\. Their intervals vary slightly because the analysis draws a separate bootstrap resample sequence for each model and metric\.
Table 5:Late conditional action errors \(%\) with 95% environment\-bootstrap intervals\. Within each run, the denominators are the numbers of threat and support episodes, respectively, in episodes 81–100\.Table 6:Early action accuracy, normalized accuracy area, and first ten\-episode threshold crossing\. Entries give means and 95% environment\-bootstrap intervals\. A run that never crosses the threshold contributes 101 to the crossing score\.Table 7:Full\-transfer state\-conditional mean utilities and Brier scores, with 95% environment\-bootstrap intervals\. A dash marks an undefined Brier score for a learner that outputs values without a probability estimate\.Similar Articles
Causal Retention in Interactive Agents: Interface Factorization and Selective Adaptation
This paper introduces causal retention in interactive agents, examining how frozen learned states can answer intervention queries independently of training objectives, supported by theoretical analysis and experiments on finite causal systems and AI models like Qwen.
BiPACE: Bisimulation-Guided Policy Optimization with Action Counterfactual Estimation for LLM Agents
BiPACE introduces a drop-in advantage estimator that fixes state-action credit mismatch in stepwise group-based RL for LLM agents, using bisimulation-guided state clustering and action counterfactual estimation, achieving significant performance gains on ALFWorld, WebShop, and TextCraft with Qwen2.5 models.
Unmask the State: When Does State Adaptation Matter for Masked Diffusion Language Models
This paper investigates when state adaptation during inference matters for masked diffusion language models (MDMs), organizing inference into five axes and showing that selective adaptation (using lightweight detectors to identify high-opportunity states) captures a large share of oracle gains, e.g., 56.9% of opportunity while adapting only the top 10% of states on LLaDA-8B constrained JSON filling.
Adaptive Memory for Online Reinforcement Learning under Sparse Model Changes
A reinforcement learning paper proposing an adaptive memory algorithm that selectively reuses observations from past deployments when only a small fraction of reward and transition entries change, achieving a long-run average gap of Õ(√(ε/H)) using only the most recent O(1/ε) deployments.
When Offline Evaluation Misleads: A Diagnostic Protocol for Reward and Policy Selection in Delayed-Feedback Contextual Bandits
This paper presents a diagnostic protocol for selecting rewards and policies in delayed-feedback contextual bandits, arguing that standard offline evaluation can mislead and validating the approach on benchmarks and a deployed push system.