In-Context Reinforcement Learning under Non-Stationarity: A Survey

arXiv cs.AI 07/15/26, 04:00 AM Papers
Summary
This survey examines in-context reinforcement learning (ICRL) under non-stationary environments, where a pretrained decision model adapts through accumulated context without parameter updates. It organizes the literature around what changes, how it unfolds, and how observable it is, and identifies research gaps such as stale-context stress tests and adaptive forgetting.
arXiv:2607.11906v1 Announce Type: new Abstract: The development of decision-pretrained transformers, algorithm distillation, long-context meta-RL, and retrieval-augmented agents has renewed interest in in-context reinforcement learning (ICRL): the ability of a pretrained or fine-tuned decision model to infer latent task rules and improve future behavior from interaction context, without test-time parameter updates. This line of work asks when trial-and-error evidence, rewards, transitions, demonstrations, feedback, or retrieved experience can make learning-like computation happen inside the context window. However, existing surveys of ICRL mainly organize the field around pretraining objectives, architectures, context formats, evaluation protocols, and theoretical mechanisms, while the non-stationary setting remains comparatively underexamined. In changing environments, accumulated context is not merely more evidence about a fixed task: the reward specification, transition kernel, observation channel, action interface, constraint model, or demonstration and memory distribution can fall out of alignment with the current regime. Previously useful context can therefore become stale, misleading, or useful again when an old regime returns. We survey non-stationary ICRL as the problem of adapting through context while deployed policy parameters remain fixed: the policy must infer both the current decision rule and which parts of its accumulated evidence still support that rule. We define non-stationary ICRL, relate it to meta-RL, decision sequence modeling, retrieval-augmented RL, value- and model-aware ICRL, and reward-feedback agents, and organize the literature along three questions: what changes, how the change unfolds, and how observable the change is to the agent.
Original Article
View Cached Full Text
Cached at: 07/15/26, 04:19 AM
# In-Context Reinforcement Learning under Non-Stationarity: A Survey
Source: [https://arxiv.org/html/2607.11906](https://arxiv.org/html/2607.11906)
A Run Independent ResearcherZiluo Ding Beijing Innovation Center of Humanoid Robotics

###### Abstract

The development of decision\-pretrained transformers, algorithm distillation, long\-context meta\-RL, and retrieval\-augmented agents has renewed interest in in\-context reinforcement learning \(ICRL\): the ability of a pretrained or fine\-tuned decision model to infer latent task rules and improve future behavior from interaction context, without test\-time parameter updates\. This line of work asks when trial\-and\-error evidence, rewards, transitions, demonstrations, feedback, or retrieved experience can make learning\-like computation happen inside the context window\. However, existing surveys of ICRL mainly organize the field around pretraining objectives, architectures, context formats, evaluation protocols, and theoretical mechanisms, while the non\-stationary setting remains comparatively underexamined\. In changing environments, accumulated context is not merely more evidence about a fixed task: the reward specification, transition kernel, observation channel, action interface, constraint model, or demonstration and memory distribution can fall out of alignment with the current regime\. Previously useful context can therefore become stale, misleading, or useful again when an old regime returns\. We survey non\-stationary ICRL as the problem of adapting through context while deployed policy parameters remain fixed: the policy must infer both the current decision rule and which parts of its accumulated evidence still support that rule\. We define non\-stationary ICRL, relate it to meta\-RL, decision sequence modeling, retrieval\-augmented RL, value\- and model\-aware ICRL, and reward\-feedback agents, and organize the literature along three questions: what changes, how the change unfolds, and how observable the change is to the agent\. We further analyze methods through context\-management operations and influence facets, including what is written, retrieved, compressed, trusted, forgotten, or isolated, and argue that average return and held\-out task generalization are insufficient to establish adaptation under drift\. The survey identifies a research agenda around lifecycle evaluation, stale\-context stress tests, validity\-aware retrieval, adaptive forgetting, context poisoning, and finite\-context theory for non\-stationary ICRL\.

## 1Introduction

In\-context reinforcement learning \(ICRL\) asks whether a pretrained or fine\-tuned sequential decision model can infer task rules and improve future actions from the interaction history available in its context, without test\-time parameter updates\(Duanet al\.,[2016](https://arxiv.org/html/2607.11906#bib.bib5)\)\. The context may contain states, actions, rewards, transition outcomes, demonstrations, feedback, retrieved memories, or summaries, but the defining property is learning\-like adaptation through that context rather than through an online optimizer\(Laskinet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib3); Zismanet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib25)\)\. That idea runs through black\-box meta\-reinforcement learning\(Duanet al\.,[2016](https://arxiv.org/html/2607.11906#bib.bib5); Wanget al\.,[2016](https://arxiv.org/html/2607.11906#bib.bib6)\), gradient\-based meta\-learning\(Finnet al\.,[2017](https://arxiv.org/html/2607.11906#bib.bib84)\), offline trajectory modeling\(Janneret al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib97)\), decision transformers\(Chenet al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib4)\), algorithm distillation\(Laskinet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib3)\), supervised ICRL\(Leeet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib13)\), and long\-context meta\-RL\(Melo,[2022](https://arxiv.org/html/2607.11906#bib.bib168); Grigsbyet al\.,[2024a](https://arxiv.org/html/2607.11906#bib.bib17)\)\. It also inherits a broader question from in\-context learning: when does a sequence model merely condition on examples\(Brownet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib132); Minet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib182)\), and when does its forward pass implement something close to inference\(Chanet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib127); Garget al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib128); Akyureket al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib129)\), optimization\(von Oswaldet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib130); Kirschet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib131)\), or learning\(Wieset al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib183); Akyureket al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib184)\)? This capability is especially appealing in reinforcement learning\. If useful adaptation can happen through the input stream, then a system can respond quickly without running an optimizer, changing deployed weights, or hiding the adaptation process inside an online update\.

That promise is much easier to state in stationary settings than in changing ones\. Recent survey work already organizes ICRL broadly around pretraining, context construction, architectures, evaluation, and theory\(Moeiniet al\.,[2025b](https://arxiv.org/html/2607.11906#bib.bib1)\)\. Here we focus on the case where the environment changes during the lifetime over which context is accumulated\. By non\-stationarity, we mean temporal changes in the decision process that alter which accumulated evidence remains valid for action\. In a stationary few\-shot task, more context is usually more evidence about one latent task\. Under non\-stationarity, the same extra trajectory can be useful evidence, obsolete evidence, a misleading demonstration, or a memory that becomes useful again only after a previous regime returns\. The question is no longer simply whether the agent uses context\. It is whether the agent can tell which parts of its context still describe the world it is acting in\.

For ICRL, non\-stationarity is best viewed as a change in the validity conditions of context: past trajectories, rewards, feedback, demonstrations, or retrieved memories may no longer license the same policy choice\. The underlying source can be a changed reward specification\(Besbeset al\.,[2014](https://arxiv.org/html/2607.11906#bib.bib82); Songet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib65)\), transition kernel\(Lecarpentier and Rachelson,[2019](https://arxiv.org/html/2607.11906#bib.bib135); Chandaket al\.,[2020c](https://arxiv.org/html/2607.11906#bib.bib136)\), observation channel\(Cobbeet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib27); Wanget al\.,[2025e](https://arxiv.org/html/2607.11906#bib.bib29)\), action interface\(Chandaket al\.,[2020b](https://arxiv.org/html/2607.11906#bib.bib24); Siniiet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib23)\), preference or safety model\(Moeiniet al\.,[2025a](https://arxiv.org/html/2607.11906#bib.bib57); Songet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib65)\), task distribution\(Al\-Shedivatet al\.,[2017](https://arxiv.org/html/2607.11906#bib.bib142); Wanget al\.,[2025a](https://arxiv.org/html/2607.11906#bib.bib60)\), or behavior policy behind the data\(Donget al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib133); Schmiedet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib20)\)\. These sources impose different adaptation burdens\. Reward switches require reinterpretation of feedback, transition drift changes action–outcome evidence, action remapping changes the meaning of behavior, and behavior\-policy shift changes the trustworthiness of demonstrations or retrieved trajectories\. An abrupt reward switch creates a different adaptation problem\(Adams and MacKay,[2007](https://arxiv.org/html/2607.11906#bib.bib117); Garivier and Moulines,[2011](https://arxiv.org/html/2607.11906#bib.bib118)\)from gradual transition drift\(Lecarpentier and Rachelson,[2019](https://arxiv.org/html/2607.11906#bib.bib135); Fenget al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib81)\); a recurring latent mode creates a different memory problem\(Hallaket al\.,[2015](https://arxiv.org/html/2607.11906#bib.bib115); Killianet al\.,[2017](https://arxiv.org/html/2607.11906#bib.bib116)\)from one\-way degradation\(Khetarpalet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib79); Abelet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib80)\); an observed task descriptor creates a different inference problem\(Hallaket al\.,[2015](https://arxiv.org/html/2607.11906#bib.bib115); Humpliket al\.,[2019](https://arxiv.org/html/2607.11906#bib.bib165)\)from a hidden change point\(Adams and MacKay,[2007](https://arxiv.org/html/2607.11906#bib.bib117); Keplingeret al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib44)\)\. For ICRL, the relevant consequence is that context has a lifetime\. It can be useful when it identifies the current mode, harmful when it anchors the agent to a past mode, and useful again when an old mode recurs\. This framing draws on older Markov decision process\(Puterman,[1994](https://arxiv.org/html/2607.11906#bib.bib113)\), partially observable decision process\(Kaelblinget al\.,[1998](https://arxiv.org/html/2607.11906#bib.bib114)\), contextual MDP\(Hallaket al\.,[2015](https://arxiv.org/html/2607.11906#bib.bib115)\), hidden\-parameter MDP\(Killianet al\.,[2017](https://arxiv.org/html/2607.11906#bib.bib116); Doshi\-Velez and Konidaris,[2016](https://arxiv.org/html/2607.11906#bib.bib141)\), and epistemic\-POMDP\(Ghoshet al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib153)\)formalisms, but shifts the emphasis from solving a known model class to managing evidence inside a fixed context window\.

We use*Non\-Stationary ICRL*for fixed\-parameter, context\-conditioned sequential decision making in which the decision process changes during the policy’s context\-bearing lifetime\. The deployed policy must use recent and stored interaction history to detect regime evidence, revise its latent task belief, and change its action distribution without test\-time parameter updates\. The central difficulty is not only selecting the policy appropriate for the current regime, but also deciding which previous rewards, transitions, demonstrations, feedback, or memories still provide valid evidence for that choice\. It covers algorithm distillation\(Laskinet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib3)\)and decision\-pretrained transformers\(Furutaet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib170); Liu and Abbeel,[2023](https://arxiv.org/html/2607.11906#bib.bib171); Polubarovet al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib63)\)when interaction history drives adaptation; supervised ICRL\(Leeet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib13); Linet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib14)\)and offline\-pretrained agents\(Tarasovet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib50); Wanget al\.,[2024a](https://arxiv.org/html/2607.11906#bib.bib174)\)when task examples, rewards, or trajectories identify the current regime; retrieval\-augmented ICRL\(Schmiedet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib20); Sridharet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib21)\)and retrieval\-based RL\(Goyalet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib199); Humphreyset al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib200)\)when memory selection decides what evidence the model sees; and value\-aware agents\(Liuet al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib62); Berkeset al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib61)\)or model\-aware agents\(Sonet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib51); Xuet al\.,[2026c](https://arxiv.org/html/2607.11906#bib.bib52)\)when values or dynamics are inferred from context\. It leaves out standard offline RL policies that do not adapt to new test\-time evidence, deployment\-time fine\-tuning, and domain\-specific robotics or multi\-agent surveys unless their mechanisms clarify the fixed\-parameter context problem\. We treat LLM agents and broader foundation\-agent systems as bridge cases when they close the loop between context, action, feedback, and future behavior\. Pure conversational memory, retrieval\-augmented question answering, or one\-shot prompting is therefore outside the core claim unless the retained context changes decisions in a repeated environment\. The closest bodies of work are therefore not one lineage but several intersecting ones: lifelong and continual ICRL\(Xuet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib22); Wanget al\.,[2025d](https://arxiv.org/html/2607.11906#bib.bib59)\); non\-stationary MDP analyses\(Lecarpentier and Rachelson,[2019](https://arxiv.org/html/2607.11906#bib.bib135); Chandaket al\.,[2020c](https://arxiv.org/html/2607.11906#bib.bib136); Wei and Luo,[2021](https://arxiv.org/html/2607.11906#bib.bib147)\), dynamic\-regret analyses\(Luoet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib137); Pettetet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib139); Chenet al\.,[2025a](https://arxiv.org/html/2607.11906#bib.bib55)\), and near\-optimal non\-stationary control\(Maoet al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib148)\); value\- and uncertainty\-aware ICRL\(Tarasovet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib50); Liuet al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib62); Berkeset al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib61)\), compositional or suboptimal\-data ICRL\(Xuet al\.,[2026a](https://arxiv.org/html/2607.11906#bib.bib64); Donget al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib133)\), and model\-aware ICRL\(Sonet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib51); Xuet al\.,[2026c](https://arxiv.org/html/2607.11906#bib.bib52)\); retrieval\-augmented ICRL\(Schmiedet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib20); Sridharet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib21)\), history curation\(Chenet al\.,[2025b](https://arxiv.org/html/2607.11906#bib.bib53)\), and retrieval\-based RL\(Goyalet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib199); Humphreyset al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib200)\); large heterogeneous sequence\-agent datasets\(Nikulinet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib42); Wanget al\.,[2025a](https://arxiv.org/html/2607.11906#bib.bib60)\), generalist sequence agents\(Polubarovet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib49);[2026](https://arxiv.org/html/2607.11906#bib.bib63)\), and mixture\-of\-experts routing\(Wuet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib54)\); and reward\-feedback agents\(Moneaet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib103); Songet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib65)\), safety\-feedback agents\(Chandaket al\.,[2020a](https://arxiv.org/html/2607.11906#bib.bib140); Moeiniet al\.,[2025a](https://arxiv.org/html/2607.11906#bib.bib57)\), and barrier\-style safe ICRL\(Kwonet al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib58)\)\.

The non\-stationary lens changes how the field should be evaluated\. Many current ICRL evaluations test held\-out task generalization or average improvement with longer prompts\. Those tests are useful, but they do not by themselves show whether an agent can recover after a shift, reject stale demonstrations, reuse old context when a mode recurs, or operate under a finite context budget\. For non\-stationary ICRL, evaluation should report lifecycle curves, post\-shift recovery, dynamic regret, stale\-context sensitivity, retrieval utility, and adaptation under controlled changes in shift frequency and observability\. Recent work begins to address these issues directly through lifelong ICRL\(Xuet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib22)\), continual ICRL\(Wanget al\.,[2025d](https://arxiv.org/html/2607.11906#bib.bib59)\), classical non\-stationary RL\(Cheunget al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib89); Feiet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib90); Wei and Luo,[2021](https://arxiv.org/html/2607.11906#bib.bib147)\), near\-optimal non\-stationary control\(Maoet al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib148)\), bandit tracking\(Garivier and Moulines,[2011](https://arxiv.org/html/2607.11906#bib.bib118); Russacet al\.,[2019](https://arxiv.org/html/2607.11906#bib.bib143)\), and dynamic\-regret analyses\(Chenet al\.,[2025a](https://arxiv.org/html/2607.11906#bib.bib55)\), butthe area still lacks a shared taxonomy and reporting standard\.

![Refer to caption](https://arxiv.org/html/2607.11906v1/x1.png)Figure 1:Survey frame for non\-stationary ICRL\. A fixed\-parameter policy model must infer and act from interaction trajectories while the decision process and the validity of accumulated evidence change\.Figure[1](https://arxiv.org/html/2607.11906#S1.F1)gives the organizing picture\. The frozen decision model is only one part of the system\. The context pipeline decides which observations, demonstrations, rewards, retrieved memories, and summaries reach that model at each decision point\. Once the environment changes, this pipeline is both the adaptation mechanism and a common source of failure\. ICRL systems should therefore be evaluated by how their policy model, interaction history, memory, and feedback loop behave together under shifts\. Table[1](https://arxiv.org/html/2607.11906#S1.T1)makes this distinction explicit by contrasting the scope of this paper with a general ICRL survey\. TableLABEL:tab:related\-surveysthen positions the paper against adjacent survey literatures, and Figure[2](https://arxiv.org/html/2607.11906#S1.F2)shows the resulting domain boundary\.

Table 1:Scope contrast between general ICRL surveys and this non\-stationary ICRL survey\.Table 2:Related survey landscape\.LiteratureRepresentative Work and EmphasisGap Addressed HereGeneral ICRLICRL surveys organize pretraining, context construction, architectures, evaluation, and theory for ICRL broadly\(Moeiniet al\.,[2025b](https://arxiv.org/html/2607.11906#bib.bib1)\)\.Re\-centers the discussion on context validity after environmental change rather than ICRL as a whole\.Continual RLContinual RL studies lifelong adaptation, plasticity, forgetting, and transfer\(Khetarpalet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib79); Abelet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib80); Hadsellet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib208); Cacciaet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib216)\)\.Imposes the stricter ICRL constraint: the deployed decision model adapts through context, not weight updates\.Non\-Stationary RLNon\-stationary bandits, MDP algorithms, regret analyses, and NS\-Gym benchmarks address changing rewards, dynamics, and latent regimes\(Besbeset al\.,[2014](https://arxiv.org/html/2607.11906#bib.bib82); Padakandla,[2020](https://arxiv.org/html/2607.11906#bib.bib88); Lecarpentier and Rachelson,[2019](https://arxiv.org/html/2607.11906#bib.bib135); Cheunget al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib89); Feiet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib90); Fenget al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib81); Wei and Luo,[2021](https://arxiv.org/html/2607.11906#bib.bib147); Keplingeret al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib44)\)\.Adds learned context construction, retrieval, and stale evidence as first\-class variables\.Change Detection And BanditsChangepoint inference, sliding\-window or discounted bandits, and restart methods discount old samples or react to abrupt shifts\(Adams and MacKay,[2007](https://arxiv.org/html/2607.11906#bib.bib117); Garivier and Moulines,[2011](https://arxiv.org/html/2607.11906#bib.bib118); Russacet al\.,[2019](https://arxiv.org/html/2607.11906#bib.bib143); Trovoet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib151); Aueret al\.,[2002](https://arxiv.org/html/2607.11906#bib.bib119);[2019](https://arxiv.org/html/2607.11906#bib.bib145)\)\.Treats these tools as context\-validity mechanisms inside fixed\-parameter agents\.Meta\-RLBlack\-box, gradient\-based, belief\-based, and non\-stationary meta\-RL learn fast adaptation through hidden state, gradients, or latent task beliefs\(Duanet al\.,[2016](https://arxiv.org/html/2607.11906#bib.bib5); Wanget al\.,[2016](https://arxiv.org/html/2607.11906#bib.bib6); Finnet al\.,[2017](https://arxiv.org/html/2607.11906#bib.bib84); Becket al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib85); Rakellyet al\.,[2019](https://arxiv.org/html/2607.11906#bib.bib8); Zintgrafet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib9); Humpliket al\.,[2019](https://arxiv.org/html/2607.11906#bib.bib165); Mendoncaet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib157); Binget al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib163)\)\.Connects this lineage to modern sequence models and finite context\-management failures\.Offline And Sequence RLOffline RL, transformer\-in\-RL surveys, and trajectory models learn from fixed datasets or tokenize trajectories as supervised sequences\(Levineet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib91); Prudencioet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib95); Liet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib2); Chenet al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib4); Janneret al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib97); Emmonset al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib99)\)\.Separates dataset\-scale sequence modeling from true test\-time context adaptation\.ICL MechanismsWork on formatting, function learning, learnability, and implicit optimization studies how transformers infer new functions from examples\(Chanet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib127); Minet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib182); Garget al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib128); Akyureket al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib129); Wieset al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib183); von Oswaldet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib130); Kirschet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib131); Akyureket al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib184)\)\.Transfers the mechanism question to sequential decisions with rewards, stale evidence, and feedback loops\.RL GeneralizationZero\-shot and procedural generalization work studies held\-out environments, overfitting control, and distribution shift\(Kirket al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib86); Cobbeet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib27)\)\.Requires within\-lifetime context invalidation, not only train\-test diversity\.LLM AgentsAgent surveys and reflection or reward\-feedback agents emphasize planning, memory, tool use, and reward\-guided improvement\(Wanget al\.,[2024b](https://arxiv.org/html/2607.11906#bib.bib83); Yaoet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib72); Shinnet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib73); Songet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib65)\)\.Includes only sequential feedback loops where actions, rewards, tool outcomes, or memories change future decisions; excludes general chatbot memory, one\-shot prompting, and RAG without control feedback\.Retrieval And History CurationRetrieval\-augmented and history\-filtered ICRL systems decide which evidence reaches the policy model\(Schmiedet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib20); Sridharet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib21); Chenet al\.,[2025b](https://arxiv.org/html/2607.11906#bib.bib53)\)\.Asks whether retained evidence is valid under the current regime, not merely similar or high\-return\.Generalist Sequence AgentsCross\-domain agents, large ICRL datasets, and generalist sequence models scale task breadth, action interfaces, and trajectory\-token modeling\(Leeet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib12); Reedet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib10); Nikulinet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib42); Wanget al\.,[2025a](https://arxiv.org/html/2607.11906#bib.bib60); Polubarovet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib49);[2026](https://arxiv.org/html/2607.11906#bib.bib63); Wuet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib54)\)\.Treats scale as useful but insufficient without shift\-aware evaluation and context\-validity tests\.Table 2:Related survey landscape \(continued\)\.![Refer to caption](https://arxiv.org/html/2607.11906v1/x2.png)Figure 2:Scope map for this survey, not a timeline\. The paper studies the intersection where sequential decision agents adapt through context while the data\-generating process changes; adjacent fields enter only when they clarify this mechanism\.The survey builds its argument in four steps\. Figure[2](https://arxiv.org/html/2607.11906#S1.F2)is a domain\-boundary view: it identifies which literatures overlap in the problem studied here\. The survey first casts non\-stationary ICRL as a context\-validity problem\. It then separates non\-stationarity into three questions that are often conflated: what changes, how the change unfolds, and how directly the change can be observed by a frozen context\-conditioned agent\. On that basis, it reviews methods by the mechanism that manages context–raw histories, curated histories, retrieval, compression, value\-aware inference, model\-based planning, reflection, and forgetting–rather than by publication chronology\. Finally, it argues for evaluation protocols that expose adaptation rather than memorization or domain\-randomization\-style generalization, and it highlights problems that stationary benchmarks tend to miss: action\-semantics shift, context poisoning, memory isolation, uncertainty\-calibrated retrieval, and theory for finite, stale, or conflicting context\. The same source/pattern/observability vocabulary is used throughout the survey so that method claims, benchmark claims, and open problems can be stated at the level of shift actually being tested\.

The rest of the survey is organized as follows\. Section[2](https://arxiv.org/html/2607.11906#S2)defines ICRL and its boundaries\. Section[3](https://arxiv.org/html/2607.11906#S3)formalizes non\-stationary ICRL and introduces context validity\. Section[4](https://arxiv.org/html/2607.11906#S4)develops the taxonomy\. Section[5](https://arxiv.org/html/2607.11906#S5)explains why non\-stationarity changes the role of context\. Section[6](https://arxiv.org/html/2607.11906#S6)reviews method families through the lens of context management\. Section[7](https://arxiv.org/html/2607.11906#S7)discusses training data and objectives\. Section[8](https://arxiv.org/html/2607.11906#S8)proposes evaluation protocols and metrics\. Section[9](https://arxiv.org/html/2607.11906#S9)summarizes theoretical links and gaps\. Section[10](https://arxiv.org/html/2607.11906#S10)closes with design principles and open problems\.

## 2Foundations of In\-Context Reinforcement Learning

### 2\.1Fixed\-Parameter Test\-Time Adaptation

We use an operational definition, defining the concept through observable, actionable, and testable criteria\. An agent performs ICRL when a fixed\-parameter decision model uses the current episode or a broader interaction history to infer latent task structure and improve future action selection at test time\. For an observationoto\_\{t\}, actionata\_\{t\}, and contextCtC\_\{t\}, the policy can be written as

πθ\(at∣ot,Ct\),θfixed at test time\.\\pi\_\{\\theta\}\(a\_\{t\}\\mid o\_\{t\},C\_\{t\}\),\\qquad\\theta\\ \\text\{fixed at test time\}\.\(1\)The context is usually a finite serialization of interaction evidence,

Ct=packB⁡\(τ1:k,d1:ℓ,q1:m,zt\),τi=\(o1i,a1i,r1i,…,oHii\),C\_\{t\}=\\operatorname\{pack\}\_\{B\}\\left\(\\tau\_\{1:k\},d\_\{1:\\ell\},q\_\{1:m\},z\_\{t\}\\right\),\\qquad\\tau\_\{i\}=\(o\_\{1\}^\{i\},a\_\{1\}^\{i\},r\_\{1\}^\{i\},\\ldots,o\_\{H\_\{i\}\}^\{i\}\),\(2\)wherepackB\\operatorname\{pack\}\_\{B\}denotes a budget\-limited formatting, retrieval, or compression operator;τ1:k\\tau\_\{1:k\}are trajectories,d1:ℓd\_\{1:\\ell\}are demonstrations or descriptions,q1:mq\_\{1:m\}are retrieved memories or summaries, andztz\_\{t\}denotes other task hints or feedback\. The contextCtC\_\{t\}may contain recent trajectories, older episodes, demonstrations, rewards, return prompts, task descriptions, retrieved memories, compressed summaries, value estimates, natural\-language feedback, or other tokens that influence the model’s action\. The defining constraint is not the token type but the adaptation channel: the agent adapts throughCtC\_\{t\}, not through gradient updates toθ\\theta\. Merely conditioning on a prompt is therefore not enough; the context must carry evidence from which the model can change later decisions in a task\-relevant direction\. Equivalently, deployment\-time adaptation is a forward\-pass map from evidence to behavior,

Ct↦πθ\(⋅∣ot,Ct\),θt\+1=θt=θ,C\_\{t\}\\mapsto\\pi\_\{\\theta\}\(\\cdot\\mid o\_\{t\},C\_\{t\}\),\\qquad\\theta\_\{t\+1\}=\\theta\_\{t\}=\\theta,\(3\)whereas online RL or gradient\-based meta\-learning would allowθt\+1≠θt\\theta\_\{t\+1\}\\neq\\theta\_\{t\}after observing new experience\.

This definition covers several lines of work\. Algorithm distillation trains a model on histories generated by a learning algorithm, so that the trained model can imitate the algorithm’s within\-lifetime improvement through its context\(Laskinet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib3)\)\. Decision\-pretrained transformers and supervised\-pretraining approaches show that trajectory models can learn useful in\-context decision rules from offline data\(Chenet al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib4); Janneret al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib97); Emmonset al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib99)\), prompting and generalized decision\-transformer objectives\(Xuet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib11); Furutaet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib170)\), supervised ICRL theory and pretraining\(Leeet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib13); Linet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib14)\), agentic or variable\-action formulations\(Liu and Abbeel,[2023](https://arxiv.org/html/2607.11906#bib.bib171); Siniiet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib23)\), and large multi\-task decision datasets\(Nikulinet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib42); Polubarovet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib49);[2026](https://arxiv.org/html/2607.11906#bib.bib63)\)\. Long\-context meta\-RL agents connect the same idea to recurrent or transformer policies trained by reinforcement learning over multiple tasks or episodes, including transformer meta\-RL\(Melo,[2022](https://arxiv.org/html/2607.11906#bib.bib168)\), AMAGO\-style agents\(Grigsbyet al\.,[2024a](https://arxiv.org/html/2607.11906#bib.bib17);[b](https://arxiv.org/html/2607.11906#bib.bib18)\), and RELIC\-style long\-context systems\(Elawadyet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib100)\)\. Retrieval\-augmented agents make the context partly external by selecting relevant experience from a memory database, including retrieval\-augmented decision transformers\(Schmiedet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib20)\), REGENT\-style retrieval\(Sridharet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib21)\), and earlier retrieval\-based RL systems\(Goyalet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib199); Humphreyset al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib200)\)\. History curation and value\-aware variants extend this design space by filtering learning histories\(Chenet al\.,[2025b](https://arxiv.org/html/2607.11906#bib.bib53)\), routing heterogeneous decision tokens\(Wuet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib54)\), or replacing pure imitation with value\-sensitive objectives\(Tarasovet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib50); Liuet al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib62); Berkeset al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib61)\)\. Efficient long\-sequence alternatives such as S4\-style state\-space models\(Luet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib123); Davidet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib124)\), recurrent\-memory and S5\-style models\(Bulatovet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib189); Smithet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib190)\), gated or xLSTM\-style recurrent attention\(Becket al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib188)\), and Mamba\-style sequence models\(Ota,[2024](https://arxiv.org/html/2607.11906#bib.bib125); Gu and Dao,[2023](https://arxiv.org/html/2607.11906#bib.bib187)\)are also relevant because non\-stationary ICRL often asks the agent to retain long histories while still reacting quickly to recent evidence\.

ICRL is therefore a behavioral property of the deployment loop, not a single architecture\. A transformer with a trajectory prompt may fail to perform ICRL if the prompt is ignored\. A recurrent policy may perform ICRL if its hidden state or explicit context implements fast adaptation\. A large language model belongs in scope only when it is embedded in a sequential decision process with action, feedback, and reward\-like signals, rather than merely recalling conversational state\. This includes LLM bandit agents\(Moneaet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib103)\), reward\-feedback agents\(Songet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib65)\), ReAct\-style agents that act in an environment\(Yaoet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib72)\), reflection or experience\-library systems whose stored lessons change later trials\(Shinnet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib73); Zhaoet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib106); Wanget al\.,[2023a](https://arxiv.org/html/2607.11906#bib.bib105)\), and tool\-use or computer\-control agents whose future actions depend on previous tool outcomes\(Tanet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib243)\)\. It excludes static prompting\(Brownet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib132); Minet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib182)\), one\-shot retrieval\-augmented question answering\(Lewiset al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib194)\), generic chatbot memory\(Zhonget al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib108); Wanget al\.,[2023b](https://arxiv.org/html/2607.11906#bib.bib109); Packeret al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib110)\), and summaries that do not affect a repeated decision loop\. In latent\-task notation, many ICRL systems can be read as amortizing a Bayes\-adaptive control rule:

bt\(ξ\)=p\(ξ∣Ct\)∝p\(ξ\)∏\(o,a,r,o′\)∈Ctpξ\(o′,r∣o,a\),π⋆\(a∣o,Ct\)=∫πξ⋆\(a∣o\)bt\(ξ\)𝑑ξ,b\_\{t\}\(\\xi\)=p\(\\xi\\mid C\_\{t\}\)\\propto p\(\\xi\)\\prod\_\{\(o,a,r,o^\{\\prime\}\)\\in C\_\{t\}\}p\_\{\\xi\}\(o^\{\\prime\},r\\mid o,a\),\\qquad\\pi^\{\\star\}\(a\\mid o,C\_\{t\}\)=\\int\\pi\_\{\\xi\}^\{\\star\}\(a\\mid o\)\\,b\_\{t\}\(\\xi\)\\,d\\xi,\(4\)whereξ\\xiis a hidden task, dynamics, reward, or goal parameter\. The learned policy need not explicitly representbtb\_\{t\}, but successful ICRL behavior requires some computation with the same effect: evidence in context changes the posterior over which decision rule is currently appropriate\. Recent theory strengthens this operational reading by showing that transformers can represent policy\-improvement\-like computations from in\-context trajectories\(Liang and Lai,[2026](https://arxiv.org/html/2607.11906#bib.bib75)\), complementing chain\-of\-thought and temporal\-difference analyses of ICRL\(Xieet al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib74); Wanget al\.,[2024a](https://arxiv.org/html/2607.11906#bib.bib174)\)\. A fixed\-parameter model can still implement an update\-like decision rule inside the forward pass, so ICRL should not be dismissed as static behavior cloning\. The guarantees usually rely on coherent context drawn from a stable task family; non\-stationary ICRL asks what happens when the same context window also contains obsolete, conflicting, or regime\-mismatched evidence\. This behavioral definition is consistent with broader ICL work showing that in\-context behavior can arise from distributional structure\(Chanet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib127)\), meta\-training and function learning\(Garget al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib128); Akyureket al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib129); Wieset al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib183)\), implicit linear\-model fitting or gradient\-descent\-like computations\(von Oswaldet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib130); Akyureket al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib184)\), and demonstration formatting\(Kirschet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib131); Minet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib182)\)\. For this survey, those results are useful background but not sufficient: reinforcement learning adds actions, rewards, exploration, credit assignment, and the possibility that old examples become actively wrong\.

This behavioral definition also avoids a common ambiguity in the literature\. A method can be transformer\-based, pretrained on trajectories, and evaluated on held\-out tasks without demonstrating ICRL\. The decisive test is counterfactual: if the same observation is paired with different valid contexts, does the policy change in the direction implied by those contexts? One compact diagnostic is a context\-contrast score,

Δctx\(o;C,C′\)\\displaystyle\\Delta\_\{\\mathrm\{ctx\}\}\(o;C,C^\{\\prime\}\)=D\(πθ\(⋅∣o,C\)∥πθ\(⋅∣o,C′\)\),\\displaystyle=D\\\!\\left\(\\pi\_\{\\theta\}\(\\cdot\\mid o,C\)\\,\\middle\\\|\\,\\pi\_\{\\theta\}\(\\cdot\\mid o,C^\{\\prime\}\)\\right\),\(5\)J\(πθ;C\)\\displaystyle J\(\\pi\_\{\\theta\};C\)\>J\(πθ;C′\)whenCcontains better current evidence\.\\displaystyle\>J\(\\pi\_\{\\theta\};C^\{\\prime\}\)\\quad\\text\{when $C$ contains better current evidence\}\.whereDDmay be total variation, KL divergence, or another distance between action distributions\. Large distributional change alone is not enough; the changed action distribution must improve the downstream decision objective under the task implied by the context\. For non\-stationary ICRL, the counterfactual is stronger\. If the prompt contains both old and new evidence, can the policy privilege the evidence that is valid under the current regime? This is the reason the survey treats context construction, retrieval, and forgetting as first\-class parts of the agent\.

### 2\.2Nearby Paradigms

ICRL overlaps with several mature paradigms but is not identical to them\. Black\-box meta\-RL trains an agent over a distribution of tasks such that within\-episode or within\-lifetime computation implements a learning procedure\(Duanet al\.,[2016](https://arxiv.org/html/2607.11906#bib.bib5); Wanget al\.,[2016](https://arxiv.org/html/2607.11906#bib.bib6)\)\. ICRL can be seen as a modern sequence\-modeling version of this idea, with explicit context windows, retrieval, and trajectory tokens\.

Classical RL and planning texts frame the stationary baseline through MDPs and value functions\(Puterman,[1994](https://arxiv.org/html/2607.11906#bib.bib113); Sutton and Barto,[2018](https://arxiv.org/html/2607.11906#bib.bib112)\), while POMDP work shows how hidden state can be converted into belief\-state inference\(Kaelblinget al\.,[1998](https://arxiv.org/html/2607.11906#bib.bib114)\)\. Contextual MDPs\(Hallaket al\.,[2015](https://arxiv.org/html/2607.11906#bib.bib115)\)and hidden\-parameter MDPs\(Killianet al\.,[2017](https://arxiv.org/html/2607.11906#bib.bib116); Doshi\-Velez and Konidaris,[2016](https://arxiv.org/html/2607.11906#bib.bib141); Perezet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib167)\)are especially close to meta\-RL because a latent or hidden parameter controls rewards and dynamics across related tasks\. Epistemic\-POMDP\(Ghoshet al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib153)\)and task\-inference views\(Humpliket al\.,[2019](https://arxiv.org/html/2607.11906#bib.bib165); Renet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib164)\)make the same point from a different angle: what looks like poor generalization can be implicit partial observability over an unobserved task, regime, or data\-collection process\. For non\-stationary ICRL this perspective is useful but incomplete\. The latent task belief is not only uncertain; it can expire, conflict with new evidence, or become useful again when a previous regime returns\. This is why recurrent memory baselines\(Heesset al\.,[2015](https://arxiv.org/html/2607.11906#bib.bib155); Niet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib154)\), trajectory\-contrastive context encoders\(Wanget al\.,[2021a](https://arxiv.org/html/2607.11906#bib.bib166)\), and non\-stationary meta\-RL methods\(Mendoncaet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib157); Binget al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib163)\)are important bridge literatures for the survey\.

The older recurrent meta\-RL line, including RL2\(Duanet al\.,[2016](https://arxiv.org/html/2607.11906#bib.bib5)\)and learning\-to\-reinforcement\-learn systems\(Wanget al\.,[2016](https://arxiv.org/html/2607.11906#bib.bib6)\), already made the key move of representing fast adaptation in hidden state rather than in test\-time weight updates\. Attentive and latent\-belief meta\-RL methods such as SNAIL\(Mishraet al\.,[2018](https://arxiv.org/html/2607.11906#bib.bib7)\), PEARL\(Rakellyet al\.,[2019](https://arxiv.org/html/2607.11906#bib.bib8)\), and VariBAD\(Zintgrafet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib9)\)sharpened the same idea by treating recent experience as evidence for a latent task or Bayes\-adaptive belief\. Gradient\-based meta\-RL, represented by MAML\-style training\(Finnet al\.,[2017](https://arxiv.org/html/2607.11906#bib.bib84)\)and later meta\-RL variants\(Becket al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib85)\), is adjacent but distinct because its adaptation channel is usually an explicit inner\-loop parameter update rather than a fixed\-parameter forward pass\. The operational difference can be summarized as

πθ′\(at∣ot\),θ′=θ−α∇θℒ\(θ;τ1:k\)⏟gradient\-based adaptationversusπθ\(at∣ot,Ct\),θunchanged⏟in\-context adaptation\.\\underbrace\{\\pi\_\{\\theta^\{\\prime\}\}\(a\_\{t\}\\mid o\_\{t\}\),\\quad\\theta^\{\\prime\}=\\theta\-\\alpha\\nabla\_\{\\theta\}\\mathcal\{L\}\(\\theta;\\tau\_\{1:k\}\)\}\_\{\\text\{gradient\-based adaptation\}\}\\qquad\\text\{versus\}\\qquad\\underbrace\{\\pi\_\{\\theta\}\(a\_\{t\}\\mid o\_\{t\},C\_\{t\}\),\\quad\\theta\\ \\text\{unchanged\}\}\_\{\\text\{in\-context adaptation\}\}\.\(6\)Decision Transformer\(Chenet al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib4)\), Trajectory Transformer\(Janneret al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib97)\), RvS\(Emmonset al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib99)\), and Multi\-Game Decision Transformer\(Leeet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib12)\)then made trajectory sequence modeling a scalable substrate for offline decision models\. Offline RL also learns from previously collected data\(Levineet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib91); Prudencioet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib95)\), but standard offline RL usually produces a policy that acts without using new test\-time examples as context; conservative Q\-learning\(Kumaret al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib96)\)and implicit Q\-learning\(Kostrikovet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib98)\)are therefore background for value objectives, not ICRL by themselves\.

Continual RL studies agents that persist across changing experiences and often update parameters, replay buffers, or other long\-lived state\(Khetarpalet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib79); Abelet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib80)\)\. Non\-stationary RL studies algorithms for changing MDPs, often with change\-point tests\(Adams and MacKay,[2007](https://arxiv.org/html/2607.11906#bib.bib117); Chenet al\.,[2019](https://arxiv.org/html/2607.11906#bib.bib146)\), sliding windows or discounting\(Garivier and Moulines,[2011](https://arxiv.org/html/2607.11906#bib.bib118)\), bandit and MDP variation\-budget guarantees\(Besbeset al\.,[2014](https://arxiv.org/html/2607.11906#bib.bib82); Cheunget al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib89); Feiet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib90)\), dynamic or near\-optimal regret bounds\(Fenget al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib81); Wei and Luo,[2021](https://arxiv.org/html/2607.11906#bib.bib147); Maoet al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib148)\), transformer\-based regret analysis\(Chenet al\.,[2025a](https://arxiv.org/html/2607.11906#bib.bib55)\), or restart and policy\-search schedules\(Aueret al\.,[2019](https://arxiv.org/html/2607.11906#bib.bib145); Padakandlaet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib87); Lecarpentier and Rachelson,[2019](https://arxiv.org/html/2607.11906#bib.bib135)\)\. Survey work\(Padakandla,[2020](https://arxiv.org/html/2607.11906#bib.bib88)\)and policy\-search variants\(Chandaket al\.,[2020c](https://arxiv.org/html/2607.11906#bib.bib136); Luoet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib137); Pettetet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib139)\)further broaden this algorithmic view\. ICRL imposes the additional constraint that the main policy parameters are fixed at test time\.

The distinction among continual RL, lifelong RL, and ICRL is therefore operational rather than merely terminological\. Continual RL asks how an agent keeps learning across a stream of tasks or changing environments\(Khetarpalet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib79); Abelet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib80)\); the usual toolbox may include online gradient updates in persistent agents\(Khetarpalet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib79); Abelet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib80)\), replay\(Lopez\-Paz and Ranzato,[2017](https://arxiv.org/html/2607.11906#bib.bib214); Chaudhryet al\.,[2019](https://arxiv.org/html/2607.11906#bib.bib215)\), regularization\(Kirkpatricket al\.,[2017](https://arxiv.org/html/2607.11906#bib.bib210); Zenkeet al\.,[2017](https://arxiv.org/html/2607.11906#bib.bib211); Aljundiet al\.,[2018](https://arxiv.org/html/2607.11906#bib.bib212)\), architectural expansion\(Rusuet al\.,[2016](https://arxiv.org/html/2607.11906#bib.bib213)\), or explicit task\-free segmentation\(Cacciaet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib216)\)\. Lifelong RL is often used more broadly for persistent agents that accumulate and reuse experience over long horizons\(Hadsellet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib208); Lesortet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib209); Chandaket al\.,[2020b](https://arxiv.org/html/2607.11906#bib.bib24)\); it emphasizes retention\(Kirkpatricket al\.,[2017](https://arxiv.org/html/2607.11906#bib.bib210); Zenkeet al\.,[2017](https://arxiv.org/html/2607.11906#bib.bib211); Aljundiet al\.,[2018](https://arxiv.org/html/2607.11906#bib.bib212)\), forward transfer\(Wolczyket al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib217); Powerset al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib38)\), and recovery or reuse when older skills or modes become relevant again\(Steinparzet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib219); Schmiedet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib20); Sridharet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib21)\)\. ICRL, in contrast, asks whether the deployed decision model can adapt with its parameters fixed, by changing only the context, hidden state, retrieval set, or summary supplied to the forward pass\. Non\-stationary ICRL is the intersection of these ideas: it inherits continual and lifelong RL’s changing\-world concern, but it restricts the adaptation mechanism to context management\. For a fixed context budgetBB, the ICRL boundary can be written as the constrained deployment problem

maxg⁡𝔼\[∑t=1Tγt−1rt\]s\.t\.Ct=g\(h1:t−1\),\|Ct\|≤B,θfixed,\\max\_\{g\}\\ \\mathbb\{E\}\\\!\\left\[\\sum\_\{t=1\}^\{T\}\\gamma^\{t\-1\}r\_\{t\}\\right\]\\quad\\text\{s\.t\.\}\\quad C\_\{t\}=g\(h\_\{1:t\-1\}\),\\ \|C\_\{t\}\|\\leq B,\\ \\theta\\ \\text\{fixed\},\(7\)which places the burden of adaptation on what is selected, compressed, retrieved, or forgotten before the policy’s forward pass\. Recent direct work on lifelong ICRL\(Xuet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib22)\)and continual ICRL\(Wanget al\.,[2025d](https://arxiv.org/html/2607.11906#bib.bib59)\), together with adjacent non\-stationary in\-context\-learning theory\(Qinet al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib78)\), makes this intersection increasingly explicit\.

Figure[3](https://arxiv.org/html/2607.11906#S2.F3)gives a compact historical timeline for this lineage\. The important transition is from hidden\-state adaptation in early meta\-RL, to trajectory sequence modeling, to explicit ICRL, and finally to the current non\-stationary turn where retrieval, value awareness, and lifecycle evaluation become central\. Table[3](https://arxiv.org/html/2607.11906#S2.T3)uses this distinction to define the core, bridge, and excluded cases for the rest of the survey\.

12345Meta\-RLSequence modelsExplicit ICRLScale \+ memoryNon\-stationarity2016–2020Hidden\-state adaptationRL2, PEARL, VariBAD\(Duanet al\.,[2016](https://arxiv.org/html/2607.11906#bib.bib5); Rakellyet al\.,[2019](https://arxiv.org/html/2607.11906#bib.bib8); Zintgrafet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib9)\)2021–2022Trajectories as tokensDT, TT, Multi\-Game DT\(Chenet al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib4); Janneret al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib97); Leeet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib12)\)2022–2023Learning in contextAD, DPT, theory\(Laskinet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib3); Xuet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib11); Leeet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib13)\)2024–2025Long\-horizon contextAMAGO, RA\-DT, XLand\(Grigsbyet al\.,[2024a](https://arxiv.org/html/2607.11906#bib.bib17); Schmiedet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib20); Nikulinet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib42)\)2025–2026Drift\-aware ICRLCICRL, SPICE, regret\(Wanget al\.,[2025d](https://arxiv.org/html/2607.11906#bib.bib59); Berkeset al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib61); Chenet al\.,[2025a](https://arxiv.org/html/2607.11906#bib.bib55)\)

Figure 3:Historical lineage of ICRL\-related method families\. The timeline orders recurrent meta\-RL, sequence decision models, retrieval\-augmented agents, and explicit non\-stationary ICRL by development history\.Table 3:Scope boundary for this survey\.
### 2\.3Boundary Choices Under Non\-Stationarity

This survey draws the boundary around the adaptation channel rather than around the mere presence of non\-stationarity: a method is central only when a fixed\-parameter sequential decision agent must adapt to within\-lifetime changes by selecting, compressing, retrieving, or forgetting context\. Boundary choices matter more under non\-stationarity than under stationary task generalization\. If online parameter updates are allowed, adaptation can be attributed to continual learning rather than context use\. If the environment is not sequential, the problem becomes supervised in\-context learning or contextual inference rather than reinforcement learning\. If the setting is purely offline and the agent receives no test\-time context, then stale\-context behavior cannot be studied\. This survey therefore uses a strict core definition and a conservative extended scope\. The aim is not to claim every context\-conditioned model as ICRL, but to isolate the cases where changing context is the mechanism by which the agent adapts to changing decision processes\.

The conservative boundary is also useful for negative results\. If a fixed model fails after a reward switch, the failure should not immediately be attributed to lack of model capacity\. It may be caused by a context policy that supplies the wrong evidence, a training distribution that never invalidates old context, or an evaluation protocol that rewards memorization\. Conversely, if a system succeeds only because it fine\-tunes online, the result may be valuable for continual RL but does not answer the ICRL question\. Maintaining this boundary keeps the survey focused on the mechanisms that make adaptation through context possible or impossible while deployed policy parameters remain fixed\.

## 3Problem Setting for Non\-Stationary ICRL

### 3\.1From Stationary MDPs to Changing Decision Processes

Let the agent interact with a sequence of decision processes

ℳt=\(𝒮t,𝒜t,Pt,Rt,Ωt,Ot,γt\),\\mathcal\{M\}\_\{t\}=\(\\mathcal\{S\}\_\{t\},\\mathcal\{A\}\_\{t\},P\_\{t\},R\_\{t\},\\Omega\_\{t\},O\_\{t\},\\gamma\_\{t\}\),\(8\)where𝒮t\\mathcal\{S\}\_\{t\}is the state space,𝒜t\\mathcal\{A\}\_\{t\}is the action space,PtP\_\{t\}is the transition kernel,RtR\_\{t\}is the reward function,Ωt\\Omega\_\{t\}is the observation space,OtO\_\{t\}is the observation function, andγt\\gamma\_\{t\}is the discount factor or horizon convention\. This notation extends the standard MDP formalism\(Puterman,[1994](https://arxiv.org/html/2607.11906#bib.bib113)\)and allows partial observability\(Kaelblinget al\.,[1998](https://arxiv.org/html/2607.11906#bib.bib114)\), hidden contexts\(Hallaket al\.,[2015](https://arxiv.org/html/2607.11906#bib.bib115)\), or latent task parameters\(Killianet al\.,[2017](https://arxiv.org/html/2607.11906#bib.bib116); Doshi\-Velez and Konidaris,[2016](https://arxiv.org/html/2607.11906#bib.bib141); Ghoshet al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib153)\)to enter throughOtO\_\{t\}and through the unobserved variables that governPtP\_\{t\}andRtR\_\{t\}\. Non\-stationarity means that at least one decision\-relevant component changes over time or across phases of the same context\-bearing lifetime, for examplePt≠Pt\+1P\_\{t\}\\neq P\_\{t\+1\},Rt≠Rt\+1R\_\{t\}\\neq R\_\{t\+1\}, or a change inOtO\_\{t\},𝒜t\\mathcal\{A\}\_\{t\}, goals, constraints, or action semantics\. Not every component must change\. The survey focuses on changes that alter which actions are good, which observations are informative, which memories are relevant, or which previous trajectories should be trusted\.

At timett, the agent acts with a fixed policy classπθ\(at∣ot,Ct\)\\pi\_\{\\theta\}\(a\_\{t\}\\mid o\_\{t\},C\_\{t\}\)\. The contextCtC\_\{t\}is produced by a context\-management process:

Ct=gϕ\(h1:t−1,m1:t−1,bt\),C\_\{t\}=g\_\{\\phi\}\(h\_\{1:t\-1\},m\_\{1:t\-1\},b\_\{t\}\),\(9\)whereh1:t−1h\_\{1:t\-1\}denotes interaction history,m1:t−1m\_\{1:t\-1\}denotes external memories or retrieved items, andbtb\_\{t\}denotes budget constraints such as context length, retrieval count, or latency\. In the strictest ICRL setting, bothθ\\thetaandϕ\\phiare fixed after training\. Some systems use hand\-designed retrieval or summarization rules\(Schmiedet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib20); Sridharet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib21)\); others learn the context manager\(Rakellyet al\.,[2019](https://arxiv.org/html/2607.11906#bib.bib8); Zintgrafet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib9); Chenet al\.,[2025b](https://arxiv.org/html/2607.11906#bib.bib53)\)\. The key distinction from continual RL remains that the adaptation channel at deployment is the constructed context, not online weight modification\. The policy must therefore infer task rules, reward meanings, transition structure, or action effects from history and express adaptation through future actions\.

This formulation separates three objects that are often conflated\. The*decision model*maps observations and context to actions\. The*context manager*chooses what evidence is visible to the decision model\. The*environment sequence*determines whether old evidence remains compatible with the present decision problem\. In stationary ICRL, the second object can often be treated as a formatting detail: include demonstrations, rewards, or previous episodes, and ask whether the model uses them\. In non\-stationary ICRL, the context manager is a policy in its own right\. It must trade off recency, reliability, diversity, budget, and the possibility that a previously encountered regime has returned\. Classical non\-stationary bandit and RL algorithms often encode this tradeoff through sliding windows and discount factors\(Garivier and Moulines,[2011](https://arxiv.org/html/2607.11906#bib.bib118); Russacet al\.,[2019](https://arxiv.org/html/2607.11906#bib.bib143)\), optimism bonuses\(Besbeset al\.,[2014](https://arxiv.org/html/2607.11906#bib.bib82); Cheunget al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib89); Wei and Luo,[2021](https://arxiv.org/html/2607.11906#bib.bib147)\), restart schedules\(Aueret al\.,[2019](https://arxiv.org/html/2607.11906#bib.bib145)\), or explicit changepoint posteriors\(Adams and MacKay,[2007](https://arxiv.org/html/2607.11906#bib.bib117); Chenet al\.,[2019](https://arxiv.org/html/2607.11906#bib.bib146)\)\. The ICRL version differs because those operations may be implicit in the prompt, retrieval index, attention pattern, or memory state rather than in a hand\-coded online estimator\.

Regime ARegime Breward relabelRegime Cdynamics driftRegime A returnstimereturncurrent\-regime oracleshiftshiftrecurrencepost\-shiftdroprecovery windowpre\-shift baselinememory reuseold contextvalid againFigure 4:Lifecycle structure of a non\-stationary ICRL episode\. Average return hides where adaptation succeeds or fails\.Figure[4](https://arxiv.org/html/2607.11906#S3.F4)frames the lifecycle perspective behind the adaptation metrics below, including dynamic regret, recovery time, context efficiency, and stale\-context sensitivity\. A non\-stationary benchmark should therefore reveal pre\-shift competence, immediate post\-shift degradation, recovery speed, and the reuse of context when a previous regime returns\.

### 3\.2Context Validity

We define context validity as a decision\-time property\. A context itemci∈Ctc\_\{i\}\\in C\_\{t\}is valid at timettif conditioning on it improves, calibrates, or preserves decision quality under the current environmentℳt\\mathcal\{M\}\_\{t\}compared with ignoring it\. It is invalid if it systematically worsens decisions, biases value estimates, hides a change point, or consumes scarce context budget without useful information\. One way to express this decision\-relative notion is the marginal context value

νt\(ci\)=𝔼\[Qtπθ\(st,at∣Ct\)−Qtπθ\(st,at∣Ct∖\{ci\}\)\],\\nu\_\{t\}\(c\_\{i\}\)=\\mathbb\{E\}\\\!\\left\[Q\_\{t\}^\{\\pi\_\{\\theta\}\}\(s\_\{t\},a\_\{t\}\\mid C\_\{t\}\)\-Q\_\{t\}^\{\\pi\_\{\\theta\}\}\(s\_\{t\},a\_\{t\}\\mid C\_\{t\}\\setminus\\\{c\_\{i\}\\\}\)\\right\],\(10\)where the expectation is over the current state, action\-selection randomness, and any retrieval randomness\. Positiveνt\(ci\)\\nu\_\{t\}\(c\_\{i\}\)indicates helpful evidence, negativeνt\(ci\)\\nu\_\{t\}\(c\_\{i\}\)indicates harmful or misleading evidence, and values near zero indicate context that may be syntactically relevant but decision\-irrelevant\. This definition is intentionally performance\-relative: old context is not invalid merely because it is old, and recent context is not valid merely because it is recent\. For recurring modes, an old trajectory may become valid again\(Khetarpalet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib79); Abelet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib80); Xuet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib22)\)\. For abrupt reward relabeling, even a high\-return recent trajectory may be misleading\(Besbeset al\.,[2014](https://arxiv.org/html/2607.11906#bib.bib82); Mendoncaet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib157); Songet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib65)\)\.

This definition deliberately differs from syntactic relevance\. A trajectory can be syntactically similar to the current observation stream while being decision\-invalid because the reward changed\(Schmiedet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib20); Berkeset al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib61)\)\. A demonstration can be high quality under its original regime while being invalid under a new action interface\(Chandaket al\.,[2020b](https://arxiv.org/html/2607.11906#bib.bib24); Siniiet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib23)\)\. A recent item can be invalid because it was produced by an exploratory or corrupted behavior policy\(Dorfmanet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib159); Fuet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib120); Levineet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib91)\)\. The survey therefore uses validity as a causal and decision\-theoretic notion: context is valid when it improves the action distribution for the present regime, not when it looks similar, recent, or successful in isolation\.

The validity lens gives a compact statement of the non\-stationary ICRL problem:

Anon\-stationary ICRLagent must adapt through context while estimating which previously inferred rules, strategies, and memories still describe the current data\-generating process\.

This is a stronger requirement than stationary few\-shot generalization\. It requires the model or its context manager to solve a joint problem of inference, memory selection, and control\.

### 3\.3Objectives

The objective can be expressed in several complementary ways\. The standard cumulative\-reward objective is

J\(πθ,g\)=𝔼\[∑t=1Tγtrt\],J\(\\pi\_\{\\theta\},g\)=\\mathbb\{E\}\\left\[\\sum\_\{t=1\}^\{T\}\\gamma\_\{t\}r\_\{t\}\\right\],\(11\)where actions are produced byπθ\(at∣ot,g\(h1:t−1,m1:t−1,bt\)\)\\pi\_\{\\theta\}\(a\_\{t\}\\mid o\_\{t\},g\(h\_\{1:t\-1\},m\_\{1:t\-1\},b\_\{t\}\)\)\. For non\-stationary environments, average return hides important lifecycle behavior\. Two agents can have similar average return while one recovers immediately after shifts and the other fails catastrophically but performs well between shifts\. Ergodicity\-focused work gives a second reason to distrust a single average\-return number: when reward processes are non\-ergodic, ensemble expectations can diverge from the time\-average experience of an individual deployed agent\(Baumannet al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib76); Verbruggenet al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib77)\)\. In such settings, optimizing or reporting expected return can favor a policy that looks good across many hypothetical runs while producing poor long\-run outcomes on the actual trajectory that carries the agent’s context\. This concern is especially acute for ICRL because the context window is itself trajectory\-dependent; early failures, stale memories, or unlucky regime exposure can change the evidence available for later decisions\. Dynamic regret is often more diagnostic:

RegTdyn=∑t=1TVtπt⋆\(st\)−∑t=1TVtπθ\(st\),\\mathrm\{Reg\}\_\{T\}^\{\\mathrm\{dyn\}\}=\\sum\_\{t=1\}^\{T\}V\_\{t\}^\{\\pi\_\{t\}^\{\\star\}\}\(s\_\{t\}\)\-\\sum\_\{t=1\}^\{T\}V\_\{t\}^\{\\pi\_\{\\theta\}\}\(s\_\{t\}\),\(12\)whereπt⋆\\pi\_\{t\}^\{\\star\}is an oracle policy for the current environment or current latent mode\. This objective follows the same motivation as adversarial and non\-stationary bandit work\(Aueret al\.,[2002](https://arxiv.org/html/2607.11906#bib.bib119); Besbeset al\.,[2014](https://arxiv.org/html/2607.11906#bib.bib82); Li and Li,[2019](https://arxiv.org/html/2607.11906#bib.bib150)\), non\-stationary MDP regret analyses\(Cheunget al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib89); Feiet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib90); Wei and Luo,[2021](https://arxiv.org/html/2607.11906#bib.bib147)\), near\-optimal or kernelized variants\(Maoet al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib148); Domingueset al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib149)\), and non\-ergodic RL critiques of expectation\-based objectives\(Baumannet al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib76); Verbruggenet al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib77)\): a static comparator or ensemble average can make an agent look good simply because the environment changes or because failures are averaged away, whereas a dynamic comparator exposes the price paid for delayed detection, adaptation, and context repair along the realized lifetime\. The amount of environmental movement can be summarized by a variation budget, for example

BT=∑t=2T\(‖Pt−Pt−1‖∞\+‖Rt−Rt−1‖∞\),B\_\{T\}=\\sum\_\{t=2\}^\{T\}\\left\(\\\|P\_\{t\}\-P\_\{t\-1\}\\\|\_\{\\infty\}\+\\\|R\_\{t\}\-R\_\{t\-1\}\\\|\_\{\\infty\}\\right\),\(13\)with analogous terms added when observation functions, action semantics, constraints, or task priors change\. This budget is not the only possible formalization, but it makes explicit that a method’s adaptation claim depends on how often and how much the environment changes\. Post\-shift recovery time, post\-shift area under the learning curve, context efficiency, and stale\-context sensitivity provide additional views that are more directly connected to ICRL\. Given a shift atτ\\tau, one useful recovery statistic is

Trec\(τ,α\)=min⁡\{k≥0:1w∑i=0w−1Jτ\+k\+iπθ≥α1w∑i=0w−1Jτ\+k\+iπ⋆\},T\_\{\\mathrm\{rec\}\}\(\\tau,\\alpha\)=\\min\\left\\\{k\\geq 0:\\frac\{1\}\{w\}\\sum\_\{i=0\}^\{w\-1\}J\_\{\\tau\+k\+i\}^\{\\pi\_\{\\theta\}\}\\geq\\alpha\\frac\{1\}\{w\}\\sum\_\{i=0\}^\{w\-1\}J\_\{\\tau\+k\+i\}^\{\\pi^\{\\star\}\}\\right\\\},\(14\)whereα\\alphais a target fraction of oracle performance andwwis a smoothing window\. These objectives are summarized in Table[4](https://arxiv.org/html/2607.11906#S3.T4), which shows why stationary ICRL metrics are insufficient once context validity changes over time\.

Table 4:Stationary ICRL and non\-stationary ICRL emphasize different objectives\.
### 3\.4Non\-Stationarity Is Not Domain Randomization

Held\-out task evaluation and domain randomization both test generalization across tasks or environment parameters, but they do not by themselves test non\-stationary ICRL\. In non\-stationary ICRL, the task may change during the agent’s lifetime, so the agent must adjust its behavior from context rather than from a reset, retraining, or a clean task descriptor\. A model can therefore generalize well to a new stationary task while still failing when the task changes halfway through an episode; conversely, it may handle recurring change points in a narrow environment family without achieving broad held\-out generalization\.

This distinction is a useful diagnostic for benchmarks\. If changes occur only between episodes and the agent receives a clean task descriptor at the start of each episode, the benchmark mainly tests task\-conditioned generalization\. If the task changes within a lifetime but the prompt or history is reset at the change point, it tests fast reconditioning rather than robustness to stale context\(Wanget al\.,[2025d](https://arxiv.org/html/2607.11906#bib.bib59); Chenet al\.,[2025b](https://arxiv.org/html/2607.11906#bib.bib53)\)\. The central non\-stationary ICRL regime arises when old and new regimes coexist in the same context window, forcing the agent to decide which evidence remains valid\(Schmiedet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib20); Berkeset al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib61)\)\. Papers reporting non\-stationary ICRL results should therefore specify which regime they evaluate\.

## 4A Taxonomy of Non\-Stationarity for ICRL

Non\-stationarity is too broad a label to support useful comparisons by itself\. The term can refer to qualitatively different stressors, including announced or inferable reward changes in non\-stationary bandits and MDPs\(Garivier and Moulines,[2011](https://arxiv.org/html/2607.11906#bib.bib118); Besbeset al\.,[2014](https://arxiv.org/html/2607.11906#bib.bib82); Feiet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib90); Songet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib65)\), gradual variation in transition dynamics\(Lecarpentier and Rachelson,[2019](https://arxiv.org/html/2607.11906#bib.bib135); Chandaket al\.,[2020c](https://arxiv.org/html/2607.11906#bib.bib136); Fenget al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib81)\), recurrent latent regimes in contextual or hidden\-parameter MDPs\(Hallaket al\.,[2015](https://arxiv.org/html/2607.11906#bib.bib115); Killianet al\.,[2017](https://arxiv.org/html/2607.11906#bib.bib116); Doshi\-Velez and Konidaris,[2016](https://arxiv.org/html/2607.11906#bib.bib141)\), and changes in the action interface or action\-to\-effect mapping\(Chandaket al\.,[2020b](https://arxiv.org/html/2607.11906#bib.bib24); Siniiet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib23)\)\. These regimes do not exercise the same adaptation mechanism\. An append\-all history may be adequate for gradual reward estimation and harmful after abrupt action remapping\. A retrieval\-augmented model may be strong on recurring modes and still fail under subtle observation drift if its retrieval key is calibrated to the wrong surface features\(Thakuret al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib193); Yoranet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib198)\)\. A value\-aware model may react to reward shifts while remaining brittle when transition dynamics move outside its support\(Fujimotoet al\.,[2019](https://arxiv.org/html/2607.11906#bib.bib92); Kidambiet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib93); Yuet al\.,[2020b](https://arxiv.org/html/2607.11906#bib.bib94)\)\.

We therefore describe a non\-stationary ICRL setting along three axes: the component that changes, the temporal structure of the change, and the evidence by which the agent can observe it\. This is narrower than general RL generalization taxonomies, which mainly ask whether a trained policy transfers to held\-out environments; here the issue is whether a fixed model can keep using, rejecting, or reactivating context during a lifetime\(Kirket al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib86)\)\. It is also more operational than POMDP and contextual\-MDP formalisms\(Kaelblinget al\.,[1998](https://arxiv.org/html/2607.11906#bib.bib114); Hallaket al\.,[2015](https://arxiv.org/html/2607.11906#bib.bib115)\), epistemic\-POMDP views\(Ghoshet al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib153)\), and hidden\-parameter MDP formalisms\(Killianet al\.,[2017](https://arxiv.org/html/2607.11906#bib.bib116); Doshi\-Velez and Konidaris,[2016](https://arxiv.org/html/2607.11906#bib.bib141)\)\. Those formalisms justify treating task identity or regime as latent\. They do not by themselves say how a finite prompt, memory index, or retrieval policy should be managed under drift\. They also do not reduce non\-stationary ICRL to domain randomization, because the relevant change happens inside the interaction trajectory rather than only between sampled training or test instances\.

Figure[5](https://arxiv.org/html/2607.11906#S4.F5)shows a compact map of the axes, with representative labels for each dimension, and Figure[6](https://arxiv.org/html/2607.11906#S4.F6)gives a tree view for placing papers and benchmarks\. The full category definitions appear in TablesLABEL:tab:what\-changes–[7](https://arxiv.org/html/2607.11906#S4.T7)\. A concrete non\-stationary setting can be described along all three independent dimensions simultaneously\. For example, a setting labeled as dynamics/recurring/latent denotes a task in which the transition function changes across recurring regimes, while the current regime is not directly observed and must be inferred from the interaction context\. The main practical point is that these axes should be reported independently\. Explicit reward switches and latent recurring dynamics are both non\-stationary, but they are not interchangeable evidence for the same adaptation claim\.

![Refer to caption](https://arxiv.org/html/2607.11906v1/x3.png)Figure 5:Compact map of three independent axes for describing non\-stationarity in ICRL\. Short labels indicate representative categories; TablesLABEL:tab:what\-changes–[7](https://arxiv.org/html/2607.11906#S4.T7)give the full definitions\.Non\-stationary ICRLWhat changes?How does it change?How visible is it?Reward / Preference reward feedback; safe ICRL \(Songet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib65)\)Dynamics / Observation DICP; non\-stationary RL \(Lecarpentier and Rachelson,[2019](https://arxiv.org/html/2607.11906#bib.bib135); Sonet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib51)\)Action Interface changing action sets \(Chandaket al\.,[2020b](https://arxiv.org/html/2607.11906#bib.bib24)\)Task / Data Source PSBL; AnyMDP; XLand \(Xuet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib22); Nikulinet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib42)\)Abrupt change\-point recovery \(Adams and MacKay,[2007](https://arxiv.org/html/2607.11906#bib.bib117); Wanget al\.,[2025d](https://arxiv.org/html/2607.11906#bib.bib59)\)Gradual variation budget \(Besbeset al\.,[2014](https://arxiv.org/html/2607.11906#bib.bib82)\)Recurring retrieval and reuse \(Schmiedet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib20)\)Adversarial stale\-memory stress tests \(Chenet al\.,[2025a](https://arxiv.org/html/2607.11906#bib.bib55)\)Explicit task or goal token \(Xuet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib11); Leeet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib13)\)Partial reward or transition evidence \(Kaelblinget al\.,[1998](https://arxiv.org/html/2607.11906#bib.bib114)\)Latent ambiguous modes \(Hallaket al\.,[2015](https://arxiv.org/html/2607.11906#bib.bib115); Killianet al\.,[2017](https://arxiv.org/html/2607.11906#bib.bib116)\)Delayed / Noisy long\-horizon evidence \(Grigsbyet al\.,[2024b](https://arxiv.org/html/2607.11906#bib.bib18); Elawadyet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib100)\)

Figure 6:Tree\-structured taxonomy for non\-stationary ICRL\. The leaves are representative problem families rather than mutually exclusive classes\.### 4\.1Changing Components

The first axis characterizes where change enters the decision process\. TableLABEL:tab:what\-changessummarizes the main cases\. The rows are not mutually exclusive, since deployed systems often face multiple forms of change at the same time\. The central question is which part of the accumulated context has become unreliable\.

Table 5:Taxonomy axis 1: sources of non\-stationarity and their context consequences\.Shift SourceChanging ObjectContext Failure ModeTypical Evaluation ProbeReward or PreferenceRtR\_\{t\}; goal; utility; constraint weights; user preference\.Previously good trajectories become wrong demonstrations; reward labels may need reinterpretation\.Reward relabeling and goal switches in non\-stationary bandits or MDPs\(Garivier and Moulines,[2011](https://arxiv.org/html/2607.11906#bib.bib118); Besbeset al\.,[2014](https://arxiv.org/html/2607.11906#bib.bib82); Feiet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib90)\); preference reversal or reward\-feedback changes\(Songet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib65)\); safety\-cost shifts\(Chandaket al\.,[2020a](https://arxiv.org/html/2607.11906#bib.bib140); Moeiniet al\.,[2025a](https://arxiv.org/html/2607.11906#bib.bib57)\)\.DynamicsPtP\_\{t\}; latent physics; transition stochasticity; exogenous process\.Old action–outcome evidence misleads planning, credit assignment, or value estimation\.Abrupt and gradual transition changes with fixed observations\(Padakandlaet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib87); Lecarpentier and Rachelson,[2019](https://arxiv.org/html/2607.11906#bib.bib135); Chandaket al\.,[2020c](https://arxiv.org/html/2607.11906#bib.bib136)\); regret and policy\-search variants\(Cheunget al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib89); Fenget al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib81); Luoet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib137); Pettetet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib139)\); latent and model\-based dynamics inference\(Hallaket al\.,[2015](https://arxiv.org/html/2607.11906#bib.bib115); Killianet al\.,[2017](https://arxiv.org/html/2607.11906#bib.bib116); Doshi\-Velez and Konidaris,[2016](https://arxiv.org/html/2607.11906#bib.bib141); Sonet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib51); Xuet al\.,[2026c](https://arxiv.org/html/2607.11906#bib.bib52)\)\.ObservationOtO\_\{t\}; sensor mapping; feature availability; observability level\.Context describes a mode that is no longer identifiable through the same observed features\.Feature corruption, sensor remapping, and missing observations in POMDP\-style settings\(Kaelblinget al\.,[1998](https://arxiv.org/html/2607.11906#bib.bib114); Ghoshet al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib153)\); recurrent memory probes\(Heesset al\.,[2015](https://arxiv.org/html/2607.11906#bib.bib155); Niet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib154)\); long\-context generalization tests\(Grigsbyet al\.,[2024b](https://arxiv.org/html/2607.11906#bib.bib18); Raparthyet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib15)\)\.Action Semantics𝒜t\\mathcal\{A\}\_\{t\}; available actions; action\-to\-effect map\.Old action tokens no longer denote the same intervention\.Action permutation, action masking, and continuous\-action rescaling\(Chandaket al\.,[2020b](https://arxiv.org/html/2607.11906#bib.bib24); Siniiet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib23)\)\.Instruction or Tool InterfaceSystem instructions; user constraints; tool schemas; website state; feedback format\.Old instructions, reflections, or tool\-use memories no longer describe valid actions\.Hidden instruction changes, preference drift, tool\-schema remapping, website distribution shift, and feedback reliability changes in interactive language environments\(Yaoet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib72); Wanget al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib39); Yaoet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib40); Shridharet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib41); Tanet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib243)\)\.Task DistributionLatent task sequence; support; mixture weights\.Pretraining or retrieval priors overweight outdated task modes\.Changing task families in meta\-RL\(Al\-Shedivatet al\.,[2017](https://arxiv.org/html/2607.11906#bib.bib142); Mendoncaet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib157)\); out\-of\-support lifelong ICRL and generated\-MDP sequences\(Xuet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib22); Wanget al\.,[2025a](https://arxiv.org/html/2607.11906#bib.bib60); Nikulinet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib42)\)\.Behavior PolicyDemonstrator; data collector; exploration or logging policy\.Retrieved or demonstrated trajectories reflect policy mismatch rather than current task evidence\.Mixed\-quality demonstrations and weak\-data histories in offline RL\(Levineet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib91); Kumaret al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib96); Kostrikovet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib98)\); value\-aware ICRL\(Tarasovet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib50)\); Bayesian context use\(Berkeset al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib61)\)\.Horizon or ResourceEpisode length; discount; budget; latency; context length\.A context\-use rule that works with long horizons fails when adaptation opportunities are scarce\.Short\-horizon shift and delayed\-feedback stress tests\(Grigsbyet al\.,[2024a](https://arxiv.org/html/2607.11906#bib.bib17);[b](https://arxiv.org/html/2607.11906#bib.bib18); Elawadyet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib100)\); context\-budget and retrieval ablations\(Schmiedet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib20)\)\.Table 5:Taxonomy axis 1: sources of non\-stationarity and their context consequences \(continued\)\.These sources of change stress different failure modes of context use\. Reward and preference shifts are the cleanest validity test: a trajectory that was optimal before a reward switch can become actively misleading afterward, especially for models trained to imitate high\-return histories\. Dynamics and observation shifts test whether the agent infers latent environment structure from context rather than replaying familiar action sequences or overfitting to superficial tokens\. Action\-semantics shifts are particularly sharp for sequence models, because the same action token or embedding may denote a different intervention after the interface changes\. Task\-distribution, behavior\-policy, and resource shifts instead alter the credibility of accumulated evidence: the agent must decide whether demonstrations, retrieved trajectories, and old priors still describe the current decision problem\.

### 4\.2Temporal Patterns of Change

The second axis is temporal structure\. An abrupt change asks for change\-point detection, gradual drift for tracking, recurring modes for memory indexing, and adversarial or strategically timed changes for robustness\. Table[6](https://arxiv.org/html/2607.11906#S4.T6)links these patterns to the context\-management implication and to metrics that reveal the corresponding failure\.

Table 6:Taxonomy axis 2: temporal patterns of non\-stationarity\.These temporal patterns imply different forms of memory\. Abrupt non\-stationarity favors short\-term evidence and explicit forgetting\. Gradual drift favors smoothing and recency\-weighted estimation\. Recurring modes favor long\-term memory with mode\-specific retrieval\. Expanding support favors exploration and uncertainty calibration\. A single context\-length ablation cannot distinguish these cases\.

### 4\.3Shift Observability

The third axis asks how directly the agent can observe that a shift has occurred\. At one extreme, the change is announced by a task descriptor, instruction, goal token, or other explicit signal\. At the other, it must be inferred from prediction errors, transition mismatches, or delayed outcomes that may appear only after consequential actions have already been taken\. This distinction is central for ICRL because the same context must often serve two roles at once: diagnosing the current regime and choosing actions under that diagnosis\. Table[7](https://arxiv.org/html/2607.11906#S4.T7)separates the resulting observability regimes and the abilities they stress\.

Table 7:Taxonomy axis 3: observability regimes\.Observability also changes what a benchmark can legitimately claim to measure\. When every shift is announced by a clean descriptor, success mostly demonstrates task conditioning, instruction use, or prompt parsing\. When shifts are hidden behind sparse delayed feedback, the same label instead covers exploration, belief updating, and long\-horizon credit assignment\. Both regimes are valuable, but they should not be treated as interchangeable evidence for a single adaptation capability\.

### 4\.4A Compact Naming Scheme

For reporting experiments and organizing the literature, it is useful to name a non\-stationary setting with a compact three\-part notation:

source/temporal pattern/observability\.\\text\{\\fcolorbox\{nsblue\}\{nslight\}\{\\textcolor\{nsblue\}\{\\normalfont\\scshape source\}\}\}\\;\{\\color\[rgb\]\{0\.34375,0\.3671875,0\.40625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.34375,0\.3671875,0\.40625\}\\bm\{/\}\}\\;\\text\{\\fcolorbox\{nsgreen\}\{nslight\}\{\\textcolor\{nsgreen\}\{\\normalfont\\scshape temporal pattern\}\}\}\\;\{\\color\[rgb\]\{0\.34375,0\.3671875,0\.40625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.34375,0\.3671875,0\.40625\}\\bm\{/\}\}\\;\\text\{\\fcolorbox\{nsorange\}\{nslight\}\{\\textcolor\{nsorange\}\{\\normalfont\\scshape observability\}\}\}\.For example, an unannounced reward switch is ‘reward/abrupt/partial’, a recurring hidden dynamics mode is ‘dynamics/recurring/latent’, and an action remapping announced by an interface token is ‘action/abrupt/explicit’\. The notation is not a substitute for a full benchmark description\. Its purpose is more modest: it makes the adaptation claim legible before the details of the environment, dataset, or model architecture are introduced\.

### 4\.5Using the Taxonomy Analytically

The taxonomy is intended to discipline comparisons, not merely to label benchmarks\. Success on ‘reward/abrupt/explicit’ tasks may reflect effective use of a goal token, whereas success on ‘dynamics/gradual/partial’ tasks requires tracking transition evidence over time\. Performance on ‘task/recurring/latent’ tasks may come from memory indexing and mode reuse; performance on ‘behavior\-policy/abrupt/noisy’ tasks instead depends on judging whether demonstrations and retrieved trajectories are trustworthy\. These are distinct capabilities, so they should support distinct claims\.

The same taxonomy suggests a minimal stress\-test rule\. For each claimed source of non\-stationarity, evaluation should include at least one temporal pattern where stale context is harmful and at least one observability setting where trivial task\-ID conditioning is unavailable\. For each claimed memory mechanism, evaluation should include one case where old context should be discarded and one where old context should be recovered\. This two\-sided requirement prevents degenerate solutions from looking general: a recent\-only policy can excel on one\-way drift, and an append\-all policy can excel on stationary or recurring tasks, but neither by itself solves context validity\.

## 5The Role of Context Under Non\-Stationarity

### 5\.1Context Is Evidence, Memory, and Intervention

In stationary ICRL, context is usually treated as evidence about a fixed latent task\. Relevant examples improve inference; irrelevant ones are expected to be ignored\. Non\-stationarity breaks that tidy picture\. The context is now a time\-stamped record of decisions, rewards, and observations that may have been generated under different regimes\. It is also an intervention: the retrieved or summarized context changes what the frozen model attends to, and therefore changes the policy it implements\. This is why external\-memory work in language agents is relevant as a systems analogy even when it is not itself reinforcement learning: MemoryBank\(Zhonget al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib108)\), LongMem\(Wanget al\.,[2023b](https://arxiv.org/html/2607.11906#bib.bib109)\), and MemGPT\(Packeret al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib110)\)make persistent memory explicit\. Retrieval\-augmented language models\(Lewiset al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib194); Guuet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib192); Borgeaudet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib191)\), nearest\-neighbor language models\(Khandelwalet al\.,[2019](https://arxiv.org/html/2607.11906#bib.bib195)\), and in\-context retrieval\(Ramet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib196)\)make retrieval and evidence selection explicit\. The same evidence\-management problem appears in older sequential decision theory under different names: POMDP belief states\(Kaelblinget al\.,[1998](https://arxiv.org/html/2607.11906#bib.bib114)\), contextual MDP latent contexts\(Hallaket al\.,[2015](https://arxiv.org/html/2607.11906#bib.bib115)\), hidden\-parameter MDP embeddings\(Killianet al\.,[2017](https://arxiv.org/html/2607.11906#bib.bib116); Doshi\-Velez and Konidaris,[2016](https://arxiv.org/html/2607.11906#bib.bib141)\), task\-inference variables\(Humpliket al\.,[2019](https://arxiv.org/html/2607.11906#bib.bib165); Renet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib164)\), and Bayesian changepoint posteriors\(Adams and MacKay,[2007](https://arxiv.org/html/2607.11906#bib.bib117)\)all encode the idea that recent observations should update beliefs about the current regime\. Non\-stationary ICRL differs in its engineering constraint: the belief update may be only an implicit computation inside a fixed sequence model, retrieval rule, or prompt summary\.

Context therefore plays three roles at once\. It is evidence about rewards, dynamics, goals, preferences, and uncertainty\. It is memory, preserving information that may become useful again if a mode recurs\. It is also a control input to a frozen policy, because changing the context can change behavior without changing weights\. Non\-stationary ICRL is hard because these roles do not always agree\. The best evidence for the current mode may require discarding long\-term memory; the best long\-term memory may exceed the context budget; the most salient trajectory may be precisely the one that should no longer be trusted\. Thus the core question is not whether the agent remembers history, but whether it can infer when history has stopped supporting the current policy\. In a finite\-budget system, this tradeoff can be written as a context\-selection problem:

Ct⋆=arg⁡maxC⊆ℳt−1,\|C\|≤bt⁡𝔼\[Vtπθ\(st∣C\)\]−λcost\(C\),C\_\{t\}^\{\\star\}=\\arg\\max\_\{C\\subseteq\\mathcal\{M\}\_\{t\-1\},\\ \|C\|\\leq b\_\{t\}\}\\mathbb\{E\}\\\!\\left\[V\_\{t\}^\{\\pi\_\{\\theta\}\}\(s\_\{t\}\\mid C\)\\right\]\-\\lambda\\,\\mathrm\{cost\}\(C\),\(15\)whereℳt−1\\mathcal\{M\}\_\{t\-1\}is the memory available before acting,btb\_\{t\}is the context budget, and the cost term can represent tokens, retrieval calls, latency, or risk of stale evidence\. Most existing systems approximate this objective implicitly; non\-stationary evaluation should make the approximation visible\.

### 5\.2Failure Modes

Table[8](https://arxiv.org/html/2607.11906#S5.T8)summarizes common failure modes that are easy to miss in stationary evaluation\.

Table 8:Failure modes induced by stale or invalid context\.Several of these failures come from context management rather than from the policy class itself\. A sufficiently expressive model can still fail when its prompt is dominated by invalid evidence, when retrieval keys optimize surface similarity, or when compression removes a rare but decisive reward event\. Conversely, a modest model can look robust if it receives clean, recent, mode\-specific evidence\. The policy and the context pipeline therefore have to be evaluated together\.

The failure modes also explain why longer context is an ambiguous intervention\. Increasing the context window can improve adaptation by making more evidence available, but it can also increase the fraction of invalid evidence, make attention more diffuse, and amplify high\-salience pre\-shift trajectories\. The right comparison is therefore not “short context versus long context” in isolation\. It is short context, full history, recent\-only history, oracle\-truncated history, learned retrieval, and adversarially contaminated retrieval under the same shift schedule\. Only this comparison reveals whether scale is solving adaptation or merely hiding the cost of invalid context\. This point is consistent with the broader ICL literature: training distributions\(Chanet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib127)\), transformer function learning\(Garget al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib128)\), gradient\-free or probabilistic ICL mechanisms\(Kirschet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib131); Wieset al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib183)\), demonstration ordering\(Minet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib182)\), and contextual learning dynamics\(Akyureket al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib184)\)can encourage in\-context inference, memorization, or failure depending on how examples are clustered, repeated, ordered, or distributed across tasks\. For ICRL, the analogous distributional feature is not only task diversity but whether context windows contain the right conflicts: pre\-shift versus post\-shift evidence, high\-return stale trajectories versus low\-return current evidence, and recurring modes separated by misleading intervening data\.

Retrieval and long context make the problem sharper rather than eliminating it\. Retrieval\-augmented language systems\(Izacardet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib197)\), retrieval\-based RL systems\(Goyalet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib199); Humphreyset al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib200)\), and retrieval\-augmented ICRL systems\(Schmiedet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib20); Sridharet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib21)\)show that external memories can improve sample efficiency and reuse, but they also introduce a new decision variable: which memory should be treated as evidence for the current regime\. Long\-context studies similarly warn that simply appending more examples can dilute or distort in\-context learning when the relevant evidence is far from the query or mixed with distractors\(Liet al\.,[2024b](https://arxiv.org/html/2607.11906#bib.bib240); Bertschet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib241)\)\. For non\-stationary ICRL, this means retrieval precision and attention allocation should be reported as adaptation mechanisms, not only as engineering choices\.

### 5\.3The Context Lifecycle

A useful system\-level view is to give context a lifecycle:

1. 1\.Write:which interactions, demonstrations, and feedback are stored?
2. 2\.Index:how are stored items keyed for future retrieval?
3. 3\.Retrieve:which items are selected for the current decision?
4. 4\.Compress:how are long histories summarized within a finite context budget?
5. 5\.Trust or weight:after evidence is available to the policy, how much influence should each item have on action selection?
6. 6\.Forget or isolate:when should context be removed, downweighted, or stored in a mode\-specific partition?

These entries should not be read as a mandatory serial architecture\. Write, index, retrieve, compress, and forget or isolate are often discrete context\-manager operations: a system can log an item, create a key, retrieve a subset, summarize a history, or move an item out of the active memory\. Trust is different\. It denotes the*influence assignment*given to available evidence once it reaches the decision computation\. In an explicit system, trust may be implemented by a reliability score, uncertainty gate, value\-fusion weight, safety filter, or regime posterior\. In a plain transformer or recurrent policy, it may be an emergent property of attention, hidden\-state dynamics, value prediction, and decoding rather than a separately exposed module\. The lifecycle is therefore an analytic decomposition of responsibilities, not a claim that every ICRL agent contains six separate processing blocks\.

This distinction clarifies the relation between retrieve and trust\. Retrieval answers a candidate\-selection question: which memories or summaries enter the context window at all? Trust answers an influence question: conditional on evidence being present, how strongly does it shape the action distribution? A retrieval\-augmented agent can retrieve the right item and still fail by over\-weighting a stale high\-return trajectory; conversely, a long\-context agent with no explicit retriever can still succeed if attention or value inference suppresses invalid evidence\. For non\-stationary ICRL, both questions should be reported when possible: selection quality through retrieval or context\-ablation metrics, and influence quality through attention, gating, counterfactual prompting, value sensitivity, or stale\-evidence stress tests\. Under non\-stationarity, every operation or influence facet in this lifecycle can become either an adaptation mechanism or a failure point\. Figure[7](https://arxiv.org/html/2607.11906#S5.F7)visualizes this lifecycle, and Figure[8](https://arxiv.org/html/2607.11906#S5.F8)states the design question and non\-stationary risk at each operation or facet\.

![Refer to caption](https://arxiv.org/html/2607.11906v1/x4.png)Figure 7:Context lifecycle\. Under non\-stationarity, every operation can improve adaptation or inject invalid evidence\.Context Lifecycle Under Non\-Stationarity1WriteQuestion\.Which evidence enters memory? Risk\.Storing low\-quality or shifted behavior can poison future context in offline datasets\(Fuet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib120); Gulcehreet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib121); Levineet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib91)\), value\-aware ICRL\(Tarasovet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib50); Donget al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib133)\), or Bayesian fusion\(Berkeset al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib61)\)\.2IndexQuestion\.What features define similarity? Risk\.Similarity under old rewards or dynamics may be invalid after a shift in language retrieval\(Lewiset al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib194); Khandelwalet al\.,[2019](https://arxiv.org/html/2607.11906#bib.bib195)\)and retrieval\-augmented ICRL\(Schmiedet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib20); Sridharet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib21)\)\.3RetrieveQuestion\.Which memories enter the current context? Risk\.High\-reward old trajectories may outcompete lower\-return but current evidence in retrieval\-based RL\(Goyalet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib199); Humphreyset al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib200)\), RA\-DT\(Schmiedet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib20)\), or REGENT\(Sridharet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib21)\)\.4CompressQuestion\.What information is preserved? Risk\.Rare change\-point evidence can be averaged away in language\-agent memory\(Zhonget al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib108); Wanget al\.,[2023b](https://arxiv.org/html/2607.11906#bib.bib109); Packeret al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib110)\), communicative compression\(Martinez\-Lopezet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib56)\), or history filtering\(Chenet al\.,[2025b](https://arxiv.org/html/2607.11906#bib.bib53)\)\.5TrustWeightQuestion\.Given available evidence, how strongly should each item influence the policy? Risk\.This influence may be implicit: the model may attend to salient but stale evidence under transformer ICL mechanisms\(Akyureket al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib129); von Oswaldet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib130)\), Bayesian value fusion\(Berkeset al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib61)\), or reward\-feedback agents\(Songet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib65)\)\.6ForgetIsolateQuestion\.What should be removed or partitioned? Risk\.Aggressive forgetting hurts recurring modes; weak forgetting hurts abrupt shifts in continual RL surveys and regularization methods\(Khetarpalet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib79); Kirkpatricket al\.,[2017](https://arxiv.org/html/2607.11906#bib.bib210)\), replay\-based methods\(Lopez\-Paz and Ranzato,[2017](https://arxiv.org/html/2607.11906#bib.bib214); Chaudhryet al\.,[2019](https://arxiv.org/html/2607.11906#bib.bib215)\), and continual ICRL\(Wanget al\.,[2025d](https://arxiv.org/html/2607.11906#bib.bib59)\)\.

Figure 8:Context lifecycle operations and their non\-stationary design questions\. The layout follows a taxonomy\-style view: each lifecycle operation is treated as a stage whose design decision determines how stale or shifted evidence can affect the policy\.
### 5\.4Implication for Survey Organization

The rest of the survey therefore organizes methods by how they manage context\. Algorithm distillation learns from histories that contain improvement\(Laskinet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib3)\)\. Supervised ICRL learns mappings from contextual examples to decisions\(Leeet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib13); Linet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib14); Huanget al\.,[2024a](https://arxiv.org/html/2607.11906#bib.bib172)\)\. Retrieval\-augmented methods select external memories\(Schmiedet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib20); Sridharet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib21); Goyalet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib199)\)\. Value\-aware and Bayesian methods combine context with priors\(Tarasovet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib50); Liuet al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib62); Berkeset al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib61)\)\. Model\-based methods infer dynamics or planning computations in context\(Sonet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib51); Xuet al\.,[2026c](https://arxiv.org/html/2607.11906#bib.bib52)\)\. World\-model and communication\-based methods compress experience\(Zhonget al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib108); Wanget al\.,[2023b](https://arxiv.org/html/2607.11906#bib.bib109); Packeret al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib110); Martinez\-Lopezet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib56)\)\. Reflection\-style agents use textual or symbolic summaries of feedback\(Shinnet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib73); Zhaoet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib106); Wanget al\.,[2023a](https://arxiv.org/html/2607.11906#bib.bib105); Songet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib65)\)\. For non\-stationarity, the key question for each family is the same: what makes the context valid, and what happens when it is not?

This lifecycle view suggests a decomposition of responsibility\. The write operation controls what evidence can ever be used\. The index and retrieve operations control which evidence is considered similar or relevant enough to expose\. The compress operation controls which rare events survive a finite budget\. The trust facet controls how strongly available evidence affects the frozen policy, even when that weighting is implicit in the model rather than implemented as a separate module\. The forget or isolate operation controls whether old evidence disappears, is downweighted, or remains available for future recurrence\. A non\-stationary ICRL paper need not solve all parts of the lifecycle, but it should state which responsibility its contribution improves, which parts are held fixed, and whether trust is explicitly modeled or only inferred from behavior\.

## 6Method Families Through the Lens of Context Management

Methods for ICRL differ less by their names than by where they place the burden of adaptation\. Some rely on the sequence model to infer the right update from raw history\. Others move part of the problem into retrieval, summarization, memory partitioning, value estimation, or model inference\. For non\-stationary settings, the useful organizing question is simple: when the world changes, what mechanism decides which context still matters?

Figure[9](https://arxiv.org/html/2607.11906#S6.F9)summarizes the resulting landscape\. The horizontal axis separates implicit history use from explicit context selection through retrieval, summaries, and memory policies\. The vertical axis separates raw behavioral evidence from value\- or model\-aware evidence\. The empty region is as informative as the occupied one: many systems scale context length or memory access without estimating validity, while value\- and model\-aware systems still depend on the evidence selected for them\.

implicit history \+ value/model evidencemanaged memory \+ value/model evidenceimplicit history \+ behavioral evidencemanaged memory \+ behavioral evidenceexplicit contextselectionvalue/modelawarenessimplicit historymanaged memorybehavioral evidencevalue/model evidenceDecision Transformers: trajectory tokens\(Chenet al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib4); Janneret al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib97)\)Algorithm Distillation: learning histories\(Laskinet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib3)\)Long\-context meta\-RL: extended histories\(Grigsbyet al\.,[2024b](https://arxiv.org/html/2607.11906#bib.bib18); Elawadyet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib100)\)Retrieval\-augmented ICRL: external memory\(Schmiedet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib20); Sridharet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib21)\)Filtering / compression: budgeted histories\(Chenet al\.,[2025b](https://arxiv.org/html/2607.11906#bib.bib53); Martinez\-Lopezet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib56)\)Model\-based ICRL: dynamics inference\(Sonet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib51); Xuet al\.,[2026c](https://arxiv.org/html/2607.11906#bib.bib52)\)Value\-aware ICRL: value\-sensitive context\(Tarasovet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib50); Berkeset al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib61)\)Reward\-feedback agents: preference evidence\(Songet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib65)\)Open gap: validity\-aware retrieval and forgetting\(Schmiedet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib20); Wanget al\.,[2025d](https://arxiv.org/html/2607.11906#bib.bib59)\)World\-model memory: selected model evidence\(Sonet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib51); Xuet al\.,[2026c](https://arxiv.org/html/2607.11906#bib.bib52)\)

Figure 9:Method landscape organized by context\-selection explicitness and value/model awareness\.### 6\.1Algorithm Distillation and Learning Histories

Algorithm distillation trains a sequence model on histories generated by a learning algorithm, with the aim that the trained model internalizes the algorithm’s within\-lifetime improvement\(Laskinet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib3)\)\. For non\-stationary ICRL, this family is important because learning histories can include exploration, failure, reward feedback, and policy improvement\. If the pretraining histories include changes in task or reward, the model may learn a pattern of adaptation through context\. If they are stationary, the model may instead learn that older high\-return behavior should remain trusted\. This distinction mirrors broader ICL evidence that models can shift between memorization, system identification, and genuine in\-context learning depending on distributional burstiness\(Chanet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib127)\), probabilistic or gradient\-free ICL structure\(Kirschet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib131); Wieset al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib183)\), example ordering\(Minet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib182)\), and contextual learning dynamics\(Akyureket al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib184)\)\.

The critical design variable is the distribution of histories\. Histories generated by a strong learner on stationary tasks provide clean examples of improvement, but they may not teach change detection\. Histories generated under weak data or noisy policies may better reflect deployment, but they can also create ambiguous supervision\. Cross\-episodic curricula\(Shiet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib16)\), noise distillation\(Zismanet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib25)\), contrastive context encoders\(Wanget al\.,[2021a](https://arxiv.org/html/2607.11906#bib.bib166)\), and filtered learning histories\(Chenet al\.,[2025b](https://arxiv.org/html/2607.11906#bib.bib53)\)all modify the history distribution to make within\-context improvement easier to learn\. From the non\-stationary perspective, filtering should not merely remove low\-return episodes; it should preserve transitions that reveal when old context became invalid\. ICPRL adds a physically grounded variant of this idea by training a vision\-language policy on multi\-episode trial\-and\-error histories, so that later actions can condition on previous physical failures and successes without test\-time weight updates\(Xuet al\.,[2026c](https://arxiv.org/html/2607.11906#bib.bib52)\)\. The adjacent meta\-exploration literature is useful here because it explicitly treats early actions as information\-gathering, not only as return maximization, through exploration objectives\(Stadieet al\.,[2018](https://arxiv.org/html/2607.11906#bib.bib161); Dorfmanet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib159); Kamiennyet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib158)\), decoupled exploration\(Liuet al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib160)\), hyperstate inference\(Zintgrafet al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib162)\), or first\-explore strategies\(Norman and Clune,[2024](https://arxiv.org/html/2607.11906#bib.bib179)\)\.

### 6\.2Supervised ICRL and Decision\-Pretrained Transformers

Supervised pretraining can induce ICRL when the model is trained on datasets where context examples identify a latent decision problem\(Leeet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib13); Linet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib14)\)\. Decision Transformer\(Chenet al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib4)\), Trajectory Transformer\(Janneret al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib97)\), RvS\(Emmonset al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib99)\), prompting variants\(Xuet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib11)\), generalized decision transformers\(Furutaet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib170)\), chain\-of\-hindsight or agentic variants\(Liu and Abbeel,[2023](https://arxiv.org/html/2607.11906#bib.bib171)\), and ICDT\-style models\(Huanget al\.,[2024a](https://arxiv.org/html/2607.11906#bib.bib172)\)provide the broader sequence\-modeling substrate\. These methods are attractive because they can use offline data and scale with standard supervised objectives\. Recent scaling and routing variants, including multi\-game decision transformers\(Leeet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib12)\), online decision transformers\(Zhenget al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib169)\), hierarchical prompt transformers\(Wanget al\.,[2025f](https://arxiv.org/html/2607.11906#bib.bib176)\), Vintix\-style action\-model formulations\(Polubarovet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib49)\), and mixture\-of\-experts formulations\(Wuet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib54)\), further test whether the same substrate can support broader task mixtures\. Theoretical results on in\-context policy improvement clarify why this family is more than sequence imitation: under structured conditions, a transformer can use trajectory context to implement a policy\-improvement procedure rather than simply replaying the behavior policy\(Liang and Lai,[2026](https://arxiv.org/html/2607.11906#bib.bib75)\)\. This gives supervised and decision\-pretrained ICRL a principled bridge to classical policy iteration, but it also exposes the non\-stationary missing assumption\. Policy improvement is well grounded when the trajectories describe the same decision process; after reward, dynamics, or action\-semantics shifts, the model must first infer which trajectories are still valid before any improvement step is meaningful\. Their weakness under non\-stationarity is objective mismatch\. Next\-action prediction can reward copying behavior from context even when that behavior is no longer optimal\. Return conditioning can encode a desired outcome, but it does not by itself solve which past examples are relevant after a shift\.

The non\-stationary version of supervised ICRL should therefore include contrastive context designs\. For example, prompts can contain pre\-shift and post\-shift trajectories with identical observations but different rewards\. Training can require the model to attend to the most recent valid evidence, infer a latent mode, or predict change\-point indicators\. Without such designs, strong held\-out\-task performance may reflect static generalization rather than adaptation to changing environments\. Demonstration\-conditioned sequential decision studies show that task diversity, dataset size, and trajectory structure matter for in\-context generalization, but those results still need explicit stale\-context and shift tests before they establish non\-stationary adaptation\(Raparthyet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib15)\)\. The weak\-label and suboptimal\-data variants are particularly important for this survey\. Offline ICRL rarely has perfect learning histories or optimal labels at scale; offline RL coverage issues\(Levineet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib91)\), conservative and implicit value learning\(Kumaret al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib96); Kostrikovet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib98)\), return\-conditioned value learning\(Brandfonbreneret al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib177)\), Q\-learning\-style ICRL\(Tarasovet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib50)\), suboptimal\-data ICRL\(Donget al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib133)\), random\-policy state\-action distillation\(Chen and Paternain,[2024](https://arxiv.org/html/2607.11906#bib.bib134)\), and TD\-style ICRL\(Wanget al\.,[2024a](https://arxiv.org/html/2607.11906#bib.bib174)\)can turn imperfect historical trajectories into useful adaptation signals, but only if the context window exposes which behavior was suboptimal under the current regime\.

### 6\.3Efficient Sequence Backbones

The default ICRL backbone is a causal transformer, but long non\-stationary lifetimes make the quadratic cost of attention a practical bottleneck\. Structured state\-space models, recurrent\-memory transformers, gated recurrent attention, and Mamba\-style sequence models therefore deserve explicit treatment\. S4\-style ICRL\(Luet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib123)\)and Decision S4\(Davidet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib124)\)show that state\-space sequence models can support long\-horizon memory and sequence\-based RL, while Decision Mamba\(Ota,[2024](https://arxiv.org/html/2607.11906#bib.bib125); Huanget al\.,[2024b](https://arxiv.org/html/2607.11906#bib.bib173)\)and Mamba\-based scaling of algorithm distillation\(Beaussant and Mounsif,[2025](https://arxiv.org/html/2607.11906#bib.bib126)\)test selective state\-space decision models\. Related selective\-state, recurrent\-memory, and xLSTM\-style models\(Gu and Dao,[2023](https://arxiv.org/html/2607.11906#bib.bib187); Smithet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib190); Bulatovet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib189)\)and transformer\-RL memory studies\(Becket al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib188); Niet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib185)\)test whether recurrent state can replace or complement transformer attention in decision modeling\. For non\-stationarity, the architectural question is not only speed\. A compressed recurrent state can make old information cheap to carry, but it can also make invalid information hard to remove\. Thus efficient backbones should be evaluated with the same stale\-context, oracle\-reset, and recurring\-mode tests as transformer systems\. Recent non\-stationary ICL theory\(Qinet al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib78)\)makes this architectural point explicit: adaptive filtering\(Sayed,[2011](https://arxiv.org/html/2607.11906#bib.bib235)\), gating\(Yanget al\.,[2023b](https://arxiv.org/html/2607.11906#bib.bib236); Katsch,[2023](https://arxiv.org/html/2607.11906#bib.bib238)\), retention\(Sunet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib237)\), and recurrent attention\(Penget al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib239)\)can be interpreted as learnable forgetting factors when the target function changes over time\.

### 6\.4Long\-Context Meta\-RL Agents

Long\-context agents such as AdA\-style systems\(Baueret al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib19)\), AMAGO\-style systems\(Grigsbyet al\.,[2024a](https://arxiv.org/html/2607.11906#bib.bib17);[b](https://arxiv.org/html/2607.11906#bib.bib18)\), RELIC\-style systems\(Elawadyet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib100)\), transformer meta\-RL systems\(Melo,[2022](https://arxiv.org/html/2607.11906#bib.bib168)\), and hierarchical meta\-RL agents\(Shalaet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib186)\)use recurrent, transformer, or external\-memory mechanisms to solve adaptive RL problems across long horizons and open\-ended task spaces\. They are natural candidates for non\-stationary ICRL because they can integrate long histories and delayed feedback\. However, long context is not automatically better\. When the environment changes, a longer history increases both evidence and contamination\. The relevant question becomes whether the model has learned attention patterns that separate current\-mode evidence from stale\-mode evidence\. Large context, open\-ended learning\(Stookeet al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib45)\), dataset efforts such as XLand and XLand\-100B\(Nikulinet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib42)\), broader generalist\-agent sequence models\(Reedet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib10); Polubarovet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib49);[2026](https://arxiv.org/html/2607.11906#bib.bib63)\), and episodic or long\-term recall systems\(Ritteret al\.,[2018](https://arxiv.org/html/2607.11906#bib.bib178); Pasukoniset al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib206)\)make this question sharper because they scale learning histories far beyond toy settings while still requiring tests of whether scaled histories teach invalidation rather than only broad task recognition\.

Long\-context meta\-RL also raises a reporting issue\. If a model is trained online with RL across non\-stationary episodes, then test\-time adaptation may be partly due to learned recurrent dynamics rather than explicit context retrieval or prompting\. This is still within our ICRL boundary when parameters are fixed at deployment, but papers should report the lifetime state, context window, reset rules, and whether memory persists across episodes\.

### 6\.5Retrieval\-Augmented ICRL

Retrieval\-augmented methods externalize part of the context problem\. Instead of fitting the entire history into the model’s input, they retrieve trajectories, transitions, or summaries from memory in RA\-DT\(Schmiedet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib20)\), REGENT\(Sridharet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib21)\), in\-context Q\-learning with retrieval\(Xuet al\.,[2026a](https://arxiv.org/html/2607.11906#bib.bib64)\), and earlier retrieval\-based RL systems\(Goyalet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib199); Humphreyset al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib200)\)\. This is one of the most promising families for non\-stationarity because retrieval can, in principle, select mode\-relevant memories and ignore stale ones\. It is also one of the most fragile\. If retrieval is based on surface similarity, it may select trajectories from the wrong reward regime\. If retrieval is based on return, it may prefer high\-return old behavior that is now invalid\. If retrieval is based on recentness, it may fail when an old mode recurs\.

For non\-stationary ICRL, retrieval should be evaluated with validity metrics, not only downstream return\. Precision of retrieved context, counterfactual performance without retrieval, and adversarial stale\-memory tests are needed\. The best retrieval key may combine observations, actions, rewards, inferred latent modes, uncertainty, and temporal metadata\. This creates a bridge between ICRL and memory systems: retrieval is not just a scaling trick, but a learned or designed adaptation policy\. The language\-model retrieval literature is a useful warning because nearest\-neighbor language models\(Khandelwalet al\.,[2019](https://arxiv.org/html/2607.11906#bib.bib195)\), retrieval\-augmented generation and pretraining\(Lewiset al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib194); Guuet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib192); Borgeaudet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib191)\), in\-context retrieval\(Ramet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib196)\), and Atlas\-style retrieval\(Izacardet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib197)\)can improve prediction while leaving relevance and trust as separate design problems\.

### 6\.6Value\-Aware and Bayesian Context Use

Value\-aware methods attempt to make ICRL less dependent on imitation of raw trajectories\. Q\-learning objectives can improve offline ICRL\(Tarasovet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib50)\)by teaching the model value\-sensitive behavior rather than pure behavior cloning, building on the broader offline RL lesson that conservative and implicit value regularization matter under support mismatch\(Kumaret al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib96); Kostrikovet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib98)\)\. Scalable in\-context Q\-learning\(Liuet al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib62)\)and compositional Q\-learning\(Xuet al\.,[2026a](https://arxiv.org/html/2607.11906#bib.bib64)\)extend this idea by conditioning value estimates on compact context or retrieved transitions\. Bayesian\-fusion approaches combine context with value priors, aiming to decide how much to trust contextual evidence under uncertainty\(Berkeset al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib61)\)\.

This family is especially relevant under reward and dynamics shifts\. If the model estimates uncertainty over values, it can downweight context that conflicts with the current reward signal\. If it only predicts next actions, it may reproduce old behavior even after feedback changes\. The limitation is that value\-aware objectives still need the right evidence\. A Q\-function conditioned on invalid retrieved context can still produce invalid actions\. Thus value awareness and context validity are complementary: values help interpret context, while context management determines which evidence values are conditioned on\.

### 6\.7Action\-Space and Interface Adaptation

Action\-space variation deserves separate treatment because it is both a scaling issue and a non\-stationarity issue\. Variable\-action ICRL shows that models can be designed to generalize across discrete action spaces with different size, order, and semantic content\(Siniiet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib23)\)\. For this survey, the non\-stationary version is stricter: the action interface may change during a lifetime while old context still contains action tokens from the previous interface\. This makes action embeddings, action descriptions, and interface metadata part of context validity rather than a mere implementation detail\. Benchmarks that only switch rewards or tasks miss this failure mode\.

### 6\.8Model\-Based In\-Context Planning

Model\-based ICRL distills dynamics inference\(Sonet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib51)\)or planning computation\(Xuet al\.,[2026c](https://arxiv.org/html/2607.11906#bib.bib52)\)into context\-conditioned computation\. Under dynamics shifts, this is a direct fit: the agent must infer how actions change the world from recent evidence and plan accordingly\. Model\-based context also gives a clearer diagnostic than action prediction\. If a model predicts transitions correctly after a shift but still acts poorly, the failure may lie in planning\. If it predicts old transitions after observing new ones, the failure is context adaptation\. ICPRL is useful here because it separates an adaptive policy from a world model used for lookahead search in visual physics tasks\(Xuet al\.,[2026c](https://arxiv.org/html/2607.11906#bib.bib52)\)\. For a non\-stationary benchmark, the same architecture would need to show that the world\-model\-guided search reacts to changed physical parameters or action outcomes, rather than merely exploiting a static simulator prior\. World\-model work\(Hafneret al\.,[2019](https://arxiv.org/html/2607.11906#bib.bib207)\), simulator\-learning work\(Yanget al\.,[2023a](https://arxiv.org/html/2607.11906#bib.bib202)\), Genie\-style interactive generation\(Bruceet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib203)\), and instructable simulators\(Raadet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib204)\)provide adjacent mechanisms for this direction, but the survey treats them as relevant only when the learned model is used in a repeated decision loop with context\-conditioned adaptation\.

The challenge is compounding error\. A model\-based in\-context planner may infer the wrong latent dynamics from stale history and then plan confidently under that wrong model\. Non\-stationary evaluation should therefore report both predictive accuracy and control performance across pre\-shift, immediately post\-shift, and recovered phases\.

### 6\.9Context Compression and Communicative World Models

Finite context makes compression unavoidable\. Communication\- or world\-model\-based approaches can summarize interaction histories into compact language\-agent memory\(Zhonget al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib108); Wanget al\.,[2023b](https://arxiv.org/html/2607.11906#bib.bib109); Packeret al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib110)\), predictive video representations\(Wuet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib111)\), or communicative world\-model representations\(Martinez\-Lopezet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib56)\)\. Compression is useful under long horizons and recurring tasks, but it can erase the rare evidence that marks a shift\. The key question is not compression ratio alone, but preservation of decision\-relevant change evidence\. For example, a summary that preserves average reward may miss a sudden preference reversal; a summary that preserves the most recent transition may lose an older mode that will recur\.

### 6\.10LLM and Foundation Agents as Extended ICRL Cases

LLM\-based and foundation\-agent systems are in scope only when the model participates in a sequential decision loop where context, actions, feedback, and later behavior are coupled\. Recent work frames LLMs as in\-context reinforcement learners under bandit feedback\(Moneaet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib103)\), reward feedback\(Songet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib65)\), or online regret criteria\(Parket al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib181)\)\. In these settings, the frozen model weights are not the adaptation channel\. The adaptation channel is the prompt, transcript, retrieved memory, reflection, executable skill, or summary that changes future actions after previous outcomes\. ReAct\-style agents\(Yaoet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib72)\)connect reasoning traces to environment actions, while Reflexion\(Shinnet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib73)\), ExpeL\(Zhaoet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib106)\), Voyager\(Wanget al\.,[2023a](https://arxiv.org/html/2607.11906#bib.bib105)\), and general computer control agents\(Tanet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib243)\)store verbal reinforcement, experience summaries, executable skills, or interaction memories for later trials\.

Non\-stationarity makes this bridge case more than a systems detail\. User preferences can drift, hidden instructions can change, reward feedback can be reinterpreted, tool schemas and website layouts can move, and old reflections can become stale policy hints\. A memory may be semantically similar to the current situation but invalid because it was produced under a different reward function, tool interface, or user constraint\. Natural\-language summaries are especially risky because they compress many observations into declarative advice; after a shift, the advice can remain fluent and authoritative while becoming wrong\. This is the LLM\-agent analogue of stale trajectories in decision\-transformer systems and stale retrieved transitions in retrieval\-augmented ICRL\.

The survey still treats these systems as extended cases rather than the core of ICRL\. Many LLM\-agent papers do not report scalar rewards, controlled action spaces, lifecycle curves, no\-memory baselines, or context ablations\. Success in an interactive language environment may reflect instruction following, prompt parsing, static task recognition, or retrieval of a near duplicate rather than reinforcement learning through context\. Bandit studies provide an important caution: frozen LLMs often need explicit summaries\(Krishnamurthyet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib102); Daiet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib101)\)or algorithmic scaffolding\(Nieet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib104)\)to explore reliably in context\. Multi\-turn agent training methods such as PAPRIKA\(Tajwaret al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib67)\), ICAL\(Sarchet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib68)\), ArCHer\(Zhouet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib69)\), VAGEN\(Wanget al\.,[2025c](https://arxiv.org/html/2607.11906#bib.bib70)\), SCoRe\-style self\-correction\(Kumaret al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib71)\), and cultural\-accumulation systems\(Cooket al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib180)\)are therefore adjacent unless the experiment shows that repeated feedback changes future decisions through the context channel\.

For a large\-model result to support a non\-stationary ICRL claim, the evidence should show post\-shift behavioral change without parameter updates, compare memory policies, and expose stale\-context failure\. Useful baselines include no memory, last\-kktranscript, full transcript, summary memory, retrieval memory, oracle\-current\-rule context, and stale\-memory adversaries\. Small\-scale reinforcement\-tuned transformer studies that report adaptation to changing gridworld conditions are useful proof\-of\-concept evidence\(Rentschler and Roberts,[2025](https://arxiv.org/html/2607.11906#bib.bib66)\), but a convincing large\-model benchmark must also test whether verbal memories, tool histories, and retrieved examples remain valid under preference, instruction, reward, or interface shifts\. Without environment interaction, scalar or preference feedback, and repeated decisions, LLM memory is better viewed as supervised or conversational in\-context learning\.

Table 9:Large\-model bridge cases for non\-stationary ICRL\.
### 6\.11Method Matrix

Figure[10](https://arxiv.org/html/2607.11906#S6.F10)condenses the preceding method discussion by mapping each family to its context mechanism, non\-stationary strength, and main risk\.

ICRL Method FamiliesLearn From HistoriesManage ContextInfer Values or ModelsAlgorithm Distillation Internalizes improvement patterns\. Risk:over\-trusts stationary histories\. \(Laskinet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib3); Zismanet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib25); Chenet al\.,[2025b](https://arxiv.org/html/2607.11906#bib.bib53)\)Supervised ICRL Scales through offline task families\. Risk:next\-action loss rewards stale imitation\. \(Chenet al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib4); Janneret al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib97); Huanget al\.,[2024a](https://arxiv.org/html/2607.11906#bib.bib172)\)Efficient Backbones Extends memory at lower cost\. Risk:compressed state keeps invalid evidence\. \(Luet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib123); Gu and Dao,[2023](https://arxiv.org/html/2607.11906#bib.bib187); Becket al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib188)\)Long\-Context Meta\-RL Handles delayed feedback\. Risk:long context amplifies contamination\. \(Melo,[2022](https://arxiv.org/html/2607.11906#bib.bib168); Grigsbyet al\.,[2024b](https://arxiv.org/html/2607.11906#bib.bib18); Elawadyet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib100)\)Retrieval\-Augmented ICRL Reuses recurring modes under finite context\. Risk:similarity retrieves invalid old modes\. \(Goyalet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib199); Schmiedet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib20); Sridharet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib21)\)Compression / World Models Preserves long\-horizon evidence under budget\. Risk:rare shift signals disappear\. \(Zhonget al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib108); Packeret al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib110); Martinez\-Lopezet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib56)\)Value\-Aware ICRL Adapts to reward shifts better than imitation\. Risk:invalid context corrupts values\. \(Tarasovet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib50); Liuet al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib62); Berkeset al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib61)\)Model\-Based ICRL Targets dynamics and planning shifts\. Risk:wrong inferred model fails confidently\. \(Sonet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib51); Xuet al\.,[2026c](https://arxiv.org/html/2607.11906#bib.bib52)\)Reflection / LLM Agents Turns feedback into verbal or symbolic context\. Risk:outside RL unless feedback drives action\. \(Yaoet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib72); Shinnet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib73); Songet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib65)\)

Figure 10:Method families organized by context\-management mechanism\. Each leaf states the adaptation advantage and the main non\-stationary failure mode\.
### 6\.12Cross\-Family Lessons

Across these families, architecture is not the cleanest dividing line\. Algorithm distillation and supervised ICRL ask the pretrained model to infer the meaning of context inside its forward pass\. Retrieval and compression make the context pipeline more explicit: the model can adapt only to evidence that was selected and preserved\. Value\-aware and model\-based methods change what the evidence means, treating context as information about values, dynamics, or uncertainty rather than as behavior to imitate\. These choices are complementary under non\-stationarity, but each needs different evidence\. Table[10](https://arxiv.org/html/2607.11906#S6.T10)turns this point into a standard for evaluating claims about each method family\.

The common pattern is that useful systems need three things at once\. Their training histories must contain invalidation events, otherwise old evidence is always presented as trustworthy\. Their context managers need validity cues–time, shift likelihood, reward compatibility, uncertainty, or latent\-mode metadata–instead of relying only on similarity\. Their evaluations must test both forgetting and reuse\. Aggressive forgetting can look good after an abrupt shift and fail when a mode recurs; preserving everything can look good on recurrence and fail after an irreversible preference change\. The target is not more memory in the abstract, but controlled influence of memory\.

Table 10:Required evidence for method families under non\-stationarity\.

## 7Training Data and Objectives

### 7\.1Training Data as an Implicit Non\-Stationarity Assumption

Every ICRL training pipeline encodes an assumption about how context should be interpreted\. Stationary trajectories teach that old evidence remains relevant\. Learning histories teach that behavior can improve across time\. Mixed\-task datasets teach that context identifies a latent task\. Retrieval databases teach that similar past experience can help the present\. Non\-stationary training data must teach something more specific: the validity of context can change\. This point is easy to miss if ICRL is treated only as offline RL at scale: offline RL datasets\(Fuet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib120); Gulcehreet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib121); Agarwalet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib122)\), offline RL surveys\(Levineet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib91); Prudencioet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib95)\), and offline sequence\-modeling substrates\(Chenet al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib4); Janneret al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib97); Emmonset al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib99)\)provide the data substrate, but they do not automatically create within\-context invalidation events\.

This suggests a simple audit question for training data:

Does the dataset contain examples in which previously useful context becomes invalid, and does the supervision reward the model for changing its behavior accordingly?

If the answer is no, then robust non\-stationary adaptation is unlikely to emerge reliably from scale alone\. The model may still generalize, but the benchmark should not interpret this as evidence of context\-validity reasoning\.

This audit question should be applied at the level of individual context windows, not only at the dataset level\. A corpus may contain many tasks and still present each window as internally coherent\. Such a corpus teaches task inference but not conflict resolution\. To train non\-stationary ICRL, windows should sometimes contain pre\-shift and post\-shift evidence together, with supervision or rewards that make the distinction consequential\. The model must experience cases where the correct action is not the action most often seen, not the action seen in the highest\-return old trajectory, and not necessarily the action associated with the nearest retrieved neighbor\. General ICL studies make the same lesson visible in a simpler setting: example arrangement\(Minet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib182)\), task diversity and burstiness\(Chanet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib127)\), transformer function learning\(Garget al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib128); Akyureket al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib129)\), gradient\-free or probabilistic ICL mechanisms\(Kirschet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib131); Wieset al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib183)\), and contextual learning dynamics\(Akyureket al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib184)\)can determine whether a transformer learns from context or memorizes a distributional shortcut\. For ICRL, dataset construction must additionally expose rewards, interventions, and control consequences\.

### 7\.2Stationary Pretraining, Non\-Stationary Testing

A common pattern is to pretrain on stationary tasks and test on related but changing tasks\. This creates a useful stress test, but it should be interpreted carefully\. Failure may indicate that the model never learned change detection\. Success may indicate that the non\-stationary benchmark is solvable through recency heuristics or static generalization\. To separate these cases, evaluations should include context\-order permutations, stale\-context injections, repeated\-mode tests, and ablations that remove post\-shift reward evidence\.

### 7\.3Non\-Stationary Pretraining

Direct non\-stationary pretraining can vary reward functions, dynamics, observations, action mappings, and task distributions across the lifetime\. Work on lifelong ICRL\(Xuet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib22)\)and continual ICRL\(Wanget al\.,[2025d](https://arxiv.org/html/2607.11906#bib.bib59)\)moves in this direction\. The main design choice is whether changes are random, curriculum\-based, or adversarial\. Randomized worlds\(Wanget al\.,[2025a](https://arxiv.org/html/2607.11906#bib.bib60)\), contextual MDP families\(Hallaket al\.,[2015](https://arxiv.org/html/2607.11906#bib.bib115)\), hidden\-parameter MDPs\(Killianet al\.,[2017](https://arxiv.org/html/2607.11906#bib.bib116); Doshi\-Velez and Konidaris,[2016](https://arxiv.org/html/2607.11906#bib.bib141)\), open\-ended or human\-timescale task generators\(Stookeet al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib45); Baueret al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib19)\), unsupervised environment design\(Denniset al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib220); Jianget al\.,[2021b](https://arxiv.org/html/2607.11906#bib.bib221);[a](https://arxiv.org/html/2607.11906#bib.bib222)\), PLR\-style or evolving generators\(Parker\-Holderet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib223)\), and non\-stationary MDP benchmark suites\(Keplingeret al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib44)\)can provide scale, but the generator must expose the shift types the survey cares about\. Otherwise, a large randomized dataset can still underrepresent action\-semantics shifts, hidden preference changes, or recurring modes\. Recent curriculum\-design work makes this point concrete: generators can prioritize levels near the competence frontier\(Rutherfordet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib224)\), improve regret approximations for task discovery\(Garcinet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib225)\), regularize environment design\(Frans and Isola,[2023](https://arxiv.org/html/2607.11906#bib.bib226)\), or evolve the mechanics of the game itself\(Earle and Togelius,[2024](https://arxiv.org/html/2607.11906#bib.bib227)\)\.

### 7\.4Objective Alignment

The training objective should align with the desired adaptation behavior\. Behavior cloning rewards reproducing the data policy\. Return\-conditioned sequence modeling rewards action sequences associated with high returns in Decision Transformer\(Chenet al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib4)\), Trajectory Transformer\(Janneret al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib97)\), and RvS\(Emmonset al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib99)\)\. Q\-learning\-style objectives reward value\-sensitive decisions in ICRL\(Tarasovet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib50); Liuet al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib62); Donget al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib133)\)and TD\-style variants\(Wanget al\.,[2024a](https://arxiv.org/html/2607.11906#bib.bib174)\)\. They can inherit useful lessons from conservative\(Kumaret al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib96)\), optimistic\(Agarwalet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib122)\), implicit\(Kostrikovet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib98)\), or return\-conditioned offline value learning\(Brandfonbreneret al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib177)\)\. Model\-based objectives reward predictive structure in DICP\(Sonet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib51)\)and ICPRL\(Xuet al\.,[2026c](https://arxiv.org/html/2607.11906#bib.bib52)\)\. Contrastive or auxiliary objectives can reward change\-point identification, latent\-mode inference, or retrieval validity\. Table[11](https://arxiv.org/html/2607.11906#S7.T11)summarizes how common training choices shape expected behavior under non\-stationarity\.

Table 11:Training choices and their expected non\-stationary behavior\.
### 7\.5Reporting Requirements

Papers should report the stationarity assumptions of both training and testing\. At minimum, a non\-stationary ICRL paper should specify which components can change, whether changes appear in training, whether changes are signaled, whether context persists across changes, and whether the agent has a finite context or retrieval budget\. Without these details, it is difficult to know whether a method has learned non\-stationary adaptation or has simply exploited a benchmark convention\.

### 7\.6Data Construction Patterns

Several practical data\-construction patterns follow from the survey’s taxonomy\. Contrastive windows pair two contexts that share observations but differ in rewards, dynamics, or action semantics\. Shift\-labeled windows include explicit change\-point metadata, useful for supervised auxiliary losses and for measuring whether a model can use clean signals\. Shift\-unlabeled windows remove metadata and require inference from feedback\. Recurring\-mode windows force the model to store old evidence without letting it dominate every later decision\. Contaminated windows insert invalid demonstrations or retrieved trajectories, making robustness to stale evidence part of the training objective\.

These patterns can be mixed with existing ICRL training objectives\. For behavior cloning, the key is to avoid rewarding stale imitation after a shift\. For return conditioning, the key is to ensure that high desired returns cannot be achieved by repeating pre\-shift actions\. For Q\-learning objectives, the key is to provide enough post\-shift evidence for the value function to disambiguate regimes\. For model\-based objectives, the key is to evaluate predictions separately before and after changes\. The training objective need not explicitly name “context validity”, but it must create a learning signal where validity affects action quality\.

## 8Evaluation of Adaptation Beyond Memorization

### 8\.1Limits of Average Return

Average return is necessary, but it is a poor summary of adaptation\. It collapses the lifetime into one number and can hide the exact interval where non\-stationarity matters\. One agent can score well by performing strongly before a shift and failing afterward\. Another can recover after each shift but have lower steady\-state return\. For non\-stationary ICRL, the curve itself is the evidence: behavior before the shift, damage immediately afterward, recovery, and performance when an old mode returns\. This aligns with non\-stationary bandits’ move from static to dynamic comparators\(Garivier and Moulines,[2011](https://arxiv.org/html/2607.11906#bib.bib118); Besbeset al\.,[2014](https://arxiv.org/html/2607.11906#bib.bib82); Li and Li,[2019](https://arxiv.org/html/2607.11906#bib.bib150)\)and with analogous non\-stationary RL analyses\(Lecarpentier and Rachelson,[2019](https://arxiv.org/html/2607.11906#bib.bib135); Cheunget al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib89); Feiet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib90)\)\. Related MDP regret work\(Wei and Luo,[2021](https://arxiv.org/html/2607.11906#bib.bib147)\)and deep RL generalization studies\(Packeret al\.,[2018](https://arxiv.org/html/2607.11906#bib.bib156); Kirket al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib86)\)make the same warning concrete: held\-out performance alone can hide shortcut learning\. Figure[11](https://arxiv.org/html/2607.11906#S8.F11)illustrates the lifecycle quantities that should be read from this curve\.

timereturnoracleshiftshiftrecovery timepost\-shiftAUCdynamic regret isarea to oracleFigure 11:Lifecycle evaluation quantities for non\-stationary ICRL\. The shape of the curve is more informative than its average\.
### 8\.2Core Metrics

Several metrics have to be read together because each exposes a different failure mode\. For a shift atτ\\tau, post\-shift area under the curve can be reported as

AUCτ:K=∑k=0K−1Jτ\+kπθ,\\mathrm\{AUC\}\_\{\\tau:K\}=\\sum\_\{k=0\}^\{K\-1\}J\_\{\\tau\+k\}^\{\\pi\_\{\\theta\}\},\(16\)while stale\-context sensitivity can be measured by the drop between a valid context and a deliberately contaminated one:

Sstale=J\(πθ,gvalid\)−J\(πθ,gvalid∪stale\)\.S\_\{\\mathrm\{stale\}\}=J\\\!\\left\(\\pi\_\{\\theta\},g\_\{\\mathrm\{valid\}\}\\right\)\-J\\\!\\left\(\\pi\_\{\\theta\},g\_\{\\mathrm\{valid\}\\cup\\mathrm\{stale\}\}\\right\)\.\(17\)To separate memory scale from adaptation quality, context efficiency can be normalized as

ηctx=J\(πθ,g\)−J\(πθ,g∅\)tokens\(C\)\+ρretrievals\(C\),\\eta\_\{\\mathrm\{ctx\}\}=\\frac\{J\(\\pi\_\{\\theta\},g\)\-J\(\\pi\_\{\\theta\},g\_\{\\emptyset\}\)\}\{\\mathrm\{tokens\}\(C\)\+\\rho\\,\\mathrm\{retrievals\}\(C\)\},\(18\)whereρ\\rhoconverts retrieval calls into a token\-equivalent cost\. Table[12](https://arxiv.org/html/2607.11906#S8.T12)lists these metrics and explains what each one measures\.

Table 12:Recommended metrics for non\-stationary ICRL\.
### 8\.3Benchmark Dimensions

A useful benchmark suite has to vary four dimensions at once: source of change, temporal pattern, observability, and context budget\. The first includes reward, dynamics, observation, action semantics, task distribution, and behavior policy\. The second includes abrupt, gradual, recurring, expanding, and adversarial changes\. The third distinguishes explicit, partial, latent, delayed, and noisy evidence\. The fourth covers short context, long context, external retrieval, compressed memory, and no\-memory baselines\. Without this structure, a benchmark can easily support claims that are technically correct but much narrower than the label “non\-stationary ICRL” suggests\. Table[14](https://arxiv.org/html/2607.11906#S8.T14)translates the factorial design into concrete diagnostic benchmark blocks\.

Procedural environment families are especially useful for this purpose\. XLand\-MiniGrid\(Nikulinet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib26)\)provides scalable JAX meta\-RL environments with many task instances, and XLand\-100B\(Nikulinet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib42)\)turns this family into a large\-scale ICRL learning\-history dataset\. Kinetix\(Matthewset al\.,[2024b](https://arxiv.org/html/2607.11906#bib.bib43)\)provides an open\-ended 2D physics\-based control space with procedurally generated tasks\. NS\-Gym\(Keplingeret al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib44)\)is a more direct non\-stationary MDP substrate, while PHYRE\(Bakhtinet al\.,[2019](https://arxiv.org/html/2607.11906#bib.bib46)\), I\-PHYRE\(Liet al\.,[2024a](https://arxiv.org/html/2607.11906#bib.bib47)\), and DeepPHY\(Xuet al\.,[2026b](https://arxiv.org/html/2607.11906#bib.bib48)\)provide interactive physical reasoning tasks where repeated attempts and visual dynamics can be repackaged into context\-adaptation protocols\(Xuet al\.,[2026c](https://arxiv.org/html/2607.11906#bib.bib52)\)\. These environments are not automatically non\-stationary merely because they are procedural; they become non\-stationary ICRL benchmarks when the evaluator constructs within\-lifetime schedules that switch, drift, recur, or corrupt rewards, dynamics, observations, action interfaces, or task distributions\. Offline datasets such as D4RL\(Fuet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib120)\), RL Unplugged\(Gulcehreet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib121)\), and Atari replay\(Agarwalet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib122)\)can also be useful when they are repackaged into contextual windows with controlled policy quality, support mismatch, and post\-shift evidence, but they should not be confused with non\-stationary benchmarks unless the test protocol actually changes the task within the context lifetime\.

### 8\.4Candidate Environment Families

Several existing environment families are useful substrates for non\-stationary ICRL, even if they were not originally introduced under that label\. Their value is not that they automatically solve the benchmark\-design problem, but that they are programmable, procedurally diverse, memory\-demanding, or already tied to continual\-learning metrics\. TableLABEL:tab:candidate\-environmentsgroups these candidates by the kind of non\-stationary ICRL evidence they can support\.

The critical distinction is between an*environment substrate*and a*non\-stationary ICRL protocol*\. Procgen\(Cobbeet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib27)\), MiniGrid\(Chevalier\-Boisvertet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib30)\), Craftax\(Matthewset al\.,[2024a](https://arxiv.org/html/2607.11906#bib.bib34)\), MiniHack\(Samvelyanet al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib32)\), and Jumanji\(Bonnetet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib35)\)are not evidence of non\-stationary ICRL merely because they generate many tasks\. They become relevant when the evaluator specifies a within\-lifetime shift process, keeps old context visible after the shift, prevents online parameter updates, and measures recovery through context\. Continual RL benchmarks such as CRLMaze\(Lomonacoet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib37)\)and CORA\(Powerset al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib38)\)provide useful shift schedules and metrics, but they need the same care because many continual RL baselines assume online learning\. An ICRL protocol should freeze the main policy parameters and compare raw\-history, recent\-only, retrieval, compression, oracle\-change\-point, and stale\-context adversary variants\.

Table 13:Candidate environment families for non\-stationary ICRL evaluation\.Environment FamilyWhy It Is UsefulNon\-Stationary Protocol to AddBest Role in This SurveyAnyMDP / OmniRLRandomized MDPs are generated by varying transitions and rewards, directly matching the formal ICRL setting of adapting to unseen decision processes\(Wanget al\.,[2025a](https://arxiv.org/html/2607.11906#bib.bib60)\)\.Switch or recur latent MDPs within a lifetime; vary reward and transition budgets; keep pre\-shift context in the window; compare oracle restart with learned context filtering\.Controlled mechanism layer for reward, transition, and latent\-mode shift\.Contextual / Hidden\-Parameter MDP GeneratorsContextual MDPs\(Hallaket al\.,[2015](https://arxiv.org/html/2607.11906#bib.bib115)\)and hidden\-parameter MDPs\(Killianet al\.,[2017](https://arxiv.org/html/2607.11906#bib.bib116); Doshi\-Velez and Konidaris,[2016](https://arxiv.org/html/2607.11906#bib.bib141); Perezet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib167)\)use latent variables to define rewards and dynamics while preserving shared structure\.Let the latent parameter drift, switch, or recur within a lifetime; hide the parameter from the agent and require inference from context\.Formal mechanism layer for latent\-regime inference\.MiniGrid / BabyAIMiniGrid\(Chevalier\-Boisvertet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib30)\)and BabyAI\(Chevalier\-Boisvertet al\.,[2019](https://arxiv.org/html/2607.11906#bib.bib31)\)are lightweight grid and instruction environments that are modular and easy to customize\.Change goals, object rules, door\-key dependencies, instructions, observation masks, or action permutations after context has accumulated\.Minimal diagnostic layer for action semantics, instruction shift, and stale demonstrations\.AlchemyLatent causal rules are procedurally resampled and require online inference, hypothesis testing, and action sequencing\(Wanget al\.,[2021b](https://arxiv.org/html/2607.11906#bib.bib36)\)\.Change the hidden chemistry during a lifetime, make old chemistry recur, or insert examples from incompatible chemistries into retrieval memory\.Latent\-rule inference layer for context ambiguity and recurring modes\.XLand\-MiniGrid / XLand\-100BXLand\-MiniGrid\(Nikulinet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib26)\)and XLand\-100B\(Nikulinet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib42)\)make broad ICRL pretraining and evaluation possible through scalable JAX meta\-RL environments and large learning\-history data\.Schedule task, layout, rule, reward, or action\-interface changes across a single lifetime; hold context budget fixed across append\-all and retrieval methods\.Large\-scale meta\-RL and learning\-history layer\.KinetixOpen\-ended 2D physics\-based control tasks provide procedural diversity with controllable physical structure\(Matthewset al\.,[2024b](https://arxiv.org/html/2607.11906#bib.bib43)\)\.Drift gravity, mass, joints, morphology, object layout, goals, or reward weights; test whether old physics evidence is downweighted\.Open\-ended physics layer for dynamics and morphology\-like shift without making robotics central\.PHYRE / I\-PHYRE / DeepPHYPHYRE\(Bakhtinet al\.,[2019](https://arxiv.org/html/2607.11906#bib.bib46)\), I\-PHYRE\(Liet al\.,[2024a](https://arxiv.org/html/2607.11906#bib.bib47)\), and DeepPHY\(Xuet al\.,[2026b](https://arxiv.org/html/2607.11906#bib.bib48)\)expose action\-outcome evidence, visual dynamics, repeated attempts, and VLM\-compatible observations relevant to world\-model\-guided ICRL\(Xuet al\.,[2026c](https://arxiv.org/html/2607.11906#bib.bib52)\)\.Change friction, gravity, object properties, action discretization, visual annotations, or reward rules across attempts; keep failed pre\-shift attempts in context and measure whether the agent updates physical intuition\.Physically grounded dynamics and observation layer for world\-model\-guided ICRL\.ProcgenProcedurally generated games test visual and level generalization across many instances\(Cobbeet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib27)\)\.Change level generators, visual themes, object layouts, difficulty, or reward rules during a lifetime; include recurring generator families\.Visual procedural layer for observation and task\-distribution shift\.CraftaxFast JAX open\-ended crafting tasks require exploration, long\-term planning, memory, and adaptation to newly discovered situations\(Matthewset al\.,[2024a](https://arxiv.org/html/2607.11906#bib.bib34)\)\.Drift resource distributions, recipes, tool effects, biome rules, survival objectives, or inventory constraints; test old\-recipe contamination\.Open\-ended memory and planning layer, complementary to Kinetix\.MiniHack / NetHack Learning EnvironmentMiniHack\(Samvelyanet al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib32)\)is a sandbox for designing rich NetHack\-based tasks, while NLE\(Küttleret al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib33)\)provides a stochastic, procedurally generated roguelike environment\.Change object semantics, dungeon families, monster/item distributions, reward shaping, or action effects; test whether retrieved dungeon memories remain valid\.Rich single\-agent procedural layer for long\-horizon memory and sparse rewards\.JumanjiJAX environments for combinatorial and general decision problems are scalable and customizable in initial\-state distribution and problem complexity\(Bonnetet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib35)\)\.Drift problem size, constraints, cost functions, instance distributions, or resource budgets; measure context efficiency under fixed retrieval cost\.Non\-visual combinatorial layer for budget and constraint shift\.POPGym / POPGym ArcadePOPGym\(Moradet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib28)\)explicitly stresses memory in POMDPs; POPGym Arcade\(Wanget al\.,[2025e](https://arxiv.org/html/2607.11906#bib.bib29)\)provides pixel\-based fully and partially observable variants and studies memory contamination by old observations\.Switch observability regimes, delay change evidence, inject stale observations, or alternate fully observable and partially observable modes\.Memory\-validity layer for partial observability and context poisoning\.NS\-GymNS\-Gym\(Keplingeret al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib44)\)is a benchmark suite explicitly designed for non\-stationary MDPs; it connects naturally to non\-stationary reward and dynamics algorithms\(Lecarpentier and Rachelson,[2019](https://arxiv.org/html/2607.11906#bib.bib135); Chandaket al\.,[2020c](https://arxiv.org/html/2607.11906#bib.bib136)\), latent\-process algorithms\(Luoet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib137)\), and policy\-search settings\(Pettetet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib139)\)\.Freeze the agent after pretraining; expose regime changes only through interaction history; compare append\-all, recent\-only, oracle\-change, retrieval, and learned context filters\.Direct bridge from non\-stationary MDP evaluation to frozen\-parameter ICRL\.CRLMazeA 3D continual RL benchmark explicitly built around non\-stationary environmental changes\(Lomonacoet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib37)\)\.Freeze the policy after pretraining and expose changes only through interaction history; compare ICRL context policies against online continual RL baselines\.Bridge case for explicit non\-stationary RL environments\.CORAA continual RL platform with benchmarks, baselines, and metrics for forgetting, plasticity, generalization, and forward transfer\(Powerset al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib38)\)\.Recast Atari, Procgen, or NetHack task streams as frozen\-parameter context\-adaptation tests; report CORA\-style forgetting and transfer with ICRL context ablations\.Protocol and metric bridge from continual RL to ICRL\.ScienceWorld / WebShop / ALFWorldScienceWorld\(Wanget al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib39)\), WebShop\(Yaoet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib40)\), and ALFWorld\(Shridharet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib41)\)expose feedback, tool/action histories, and task instructions in interactive language environments\.Drift user preferences, reward functions, hidden instructions, website/product distributions, or feedback reliability across trials\.Extended LLM\-agent layer, included only when feedback drives repeated sequential decisions\.Table 13:Candidate environment families for non\-stationary ICRL evaluation \(continued\)\.This table also clarifies what should be excluded from the core claim\. Robotics\-specific suites and controllers such as Meta\-World\(Yuet al\.,[2020a](https://arxiv.org/html/2607.11906#bib.bib244)\), Continual World\(Wolczyket al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib217)\), LIBERO\(Liuet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib245)\), RoboMimic\(Mandlekaret al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib246)\), Open X\-Embodiment\(Open X\-Embodiment Collaborationet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib247)\), RoboMME\(Daiet al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib248)\), and humanoid whole\-body\-control systems\(Dinget al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib242)\)are informative for continual learning and generalist action modeling, but they are outside the main scope of this survey\. They can be mentioned as adjacent benchmarks when discussing evaluation traditions, but the paper should not rely on them for its central non\-stationary ICRL argument\. Similarly, multi\-agent suites should be avoided because they introduce strategic non\-stationarity, for example when concurrently updating agents make the environment non\-stationary from each individual agent’s perspective\(Luoet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib138)\), which is explicitly outside scope here\.

Table 14:A benchmark blueprint for separating adaptation mechanisms\.
### 8\.5Constructing Stale\-Context Stress Tests

A non\-stationary ICRL benchmark should be specified as a protocol over lifetimes, not only as a set of environment IDs\. The core intervention is simple: create a period in which old context is genuinely useful, change the decision process, and then keep some of the old evidence visible so the agent must decide whether to trust it\. This makes the failure modes in Table[8](https://arxiv.org/html/2607.11906#S5.T8)operational\. For example, to test stale imitation, the evaluator can collect high\-return trajectories under regimem0m\_\{0\}, switch to a regimem1m\_\{1\}with the same observations but different rewards or dynamics, and inject them0m\_\{0\}trajectories into the post\-shift context while also providing a small amount of validm1m\_\{1\}evidence\. If performance drops relative to the same post\-shift context without the injected trajectories, the agent is not merely slow to learn; it is being actively pulled by stale evidence\. This protocol connects continual ICRL evaluation\(Wanget al\.,[2025d](https://arxiv.org/html/2607.11906#bib.bib59)\), history filtering and weak\-data ICRL\(Chenet al\.,[2025b](https://arxiv.org/html/2607.11906#bib.bib53); Donget al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib133)\), retrieval\-augmented ICRL\(Schmiedet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib20); Sridharet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib21)\), and non\-stationary MDP or bandit stress tests\(Besbeset al\.,[2014](https://arxiv.org/html/2607.11906#bib.bib82); Cheunget al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib89); Feiet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib90)\)\.

A minimal protocol can be written as follows:

1. 1\.Choose a base environment and shift family\.Specify which component changes: reward, transition, observation, action semantics, constraint, task distribution, or behavior policy\(Lecarpentier and Rachelson,[2019](https://arxiv.org/html/2607.11906#bib.bib135); Chandaket al\.,[2020b](https://arxiv.org/html/2607.11906#bib.bib24); Moeiniet al\.,[2025a](https://arxiv.org/html/2607.11906#bib.bib57)\)\.
2. 2\.Generate a lifetime\.Run a pre\-shift phase underm0m\_\{0\}, a post\-shift phase underm1m\_\{1\}, and optionally a recurrence phase wherem0m\_\{0\}returns\(Wanget al\.,[2025d](https://arxiv.org/html/2607.11906#bib.bib59); Steinparzet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib219)\)\.
3. 3\.Build paired contexts\.For the same post\-shift query state, construct at least four contexts: no\-context, valid\-current context, valid\-current plus stale context, and oracle\-truncated context that removes pre\-shift evidence\.
4. 4\.Freeze adaptation channels\.Keep policy parameters fixed and vary only the context policy: append\-all, last\-kk, retrieval, compression, learned filter, oracle filter, and stale adversary\(Moeiniet al\.,[2025b](https://arxiv.org/html/2607.11906#bib.bib1); Schmiedet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib20); Chenet al\.,[2025b](https://arxiv.org/html/2607.11906#bib.bib53)\)\.
5. 5\.Measure lifecycle outcomes\.Report pre\-shift return, immediate post\-shift drop, recovery time, post\-shift AUC, dynamic regret, stale\-context sensitivity, and recurring\-mode re\-adaptation\(Wanget al\.,[2025d](https://arxiv.org/html/2607.11906#bib.bib59); Chenet al\.,[2025a](https://arxiv.org/html/2607.11906#bib.bib55)\)\.
6. 6\.Attribute the failure\.Use ablations to determine whether the bottleneck is change detection, retrieval, compression, influence weighting, forgetting, or policy capacity\.

The most important design rule is to use paired evaluations\. The valid and contaminated contexts should be identical except for the intervention being tested\. Otherwise, a method can look robust simply because the contaminated run also has more useful post\-shift evidence, or it can look fragile because the contaminated context changes budget, ordering, or formatting\. When possible, the injected stale item should be strong under its original regime: a high\-return demonstration, a confident value estimate, a retrieved near\-neighbor, or a concise verbal rule\. Weak stale evidence is too easy to ignore and underestimates the risk faced by long\-context and retrieval\-augmented agents\. This paired\-design principle is the context analogue of oracle\-change and controlled\-ablation baselines in changepoint, non\-stationary regret, and ICRL reporting\(Adams and MacKay,[2007](https://arxiv.org/html/2607.11906#bib.bib117); Chenet al\.,[2025a](https://arxiv.org/html/2607.11906#bib.bib55); Moeiniet al\.,[2025b](https://arxiv.org/html/2607.11906#bib.bib1)\)\.

Table 15:Concrete context\-window interventions for stale\-context stress tests\.These interventions also define a practical “non\-stationary ICRL\-ready” benchmark checklist\. A benchmark is ready for this survey’s claims only if it can freeze the policy, expose the agent to within\-lifetime shifts, keep old context available after the shift, inject or remove specific context items, control context budget, and report paired lifecycle metrics\. It should also support oracle variants: oracle\-current\-mode context, oracle\-change\-point truncation, oracle retrieval, and oracle summary\. Without these controls, a benchmark may still be useful for generalization or continual learning, but it cannot isolate whether a frozen policy adapts through context under stale evidence\.

Table 16:Minimum reporting checklist for non\-stationary ICRL experiments\.
### 8\.6Baselines That Matter

The most informative baselines are often simple\. A recent\-only context baseline tests whether long\-term memory is helping or hurting\. A full\-history baseline tests whether selective retrieval matters\. A no\-context baseline tests whether performance is static generalization\. An oracle\-change\-point baseline estimates the value of knowing when to forget\. A stale\-context adversary estimates how badly the model can be misled by old evidence\. For retrieval methods, random retrieval and similarity\-only retrieval should be compared with any learned validity\-aware retrieval method\. The minimum reporting items in Table[16](https://arxiv.org/html/2607.11906#S8.T16)are intended to make these baselines and ablations comparable across papers\.

### 8\.7Distinguishing Adaptation from Memorization

Non\-stationary evaluation should rule out three shortcuts\. The first is memorization of task IDs or environment templates\. The second is recency\-only adaptation, where the model ignores long context and simply follows the last few rewards\. The third is retrieval shortcutting, where a memory database contains near\-duplicates of test tasks\. Countermeasures include held\-out generators, changed task order, unseen shift schedules, adversarial stale\-memory insertion, and explicit tests of recurring modes where recency alone is insufficient\.

### 8\.8A Practical Evaluation Stack

A practical stack should include at least three layers\. The first layer is a controlled diagnostic layer\. AnyMDP\(Wanget al\.,[2025a](https://arxiv.org/html/2607.11906#bib.bib60)\), MiniGrid\(Chevalier\-Boisvertet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib30)\), BabyAI\(Chevalier\-Boisvertet al\.,[2019](https://arxiv.org/html/2607.11906#bib.bib31)\), and Alchemy\-style tasks\(Wanget al\.,[2021b](https://arxiv.org/html/2607.11906#bib.bib36)\)are appropriate here because the evaluator can exactly control which part of the decision process changes\. This layer should be used for mechanism claims: reward switches, transition switches, action remapping, hidden rule changes, and context contamination\. The experiments should be small enough that oracle baselines are meaningful, including oracle\-change\-point, oracle\-retrieval, and oracle\-current\-mode policies\. Generalization benchmarks should be used with the same caution: a held\-out level or task family tests extrapolation, whereas non\-stationary ICRL requires within\-lifetime invalidation and recovery\(Packeret al\.,[2018](https://arxiv.org/html/2607.11906#bib.bib156); Ghoshet al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib153); Kirket al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib86)\)\.

The second layer is a scalable procedural layer\. XLand\-MiniGrid and XLand\-100B\(Nikulinet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib26);[2025](https://arxiv.org/html/2607.11906#bib.bib42)\), Kinetix\(Matthewset al\.,[2024b](https://arxiv.org/html/2607.11906#bib.bib43)\), PHYRE/I\-PHYRE/DeepPHY\(Bakhtinet al\.,[2019](https://arxiv.org/html/2607.11906#bib.bib46); Liet al\.,[2024a](https://arxiv.org/html/2607.11906#bib.bib47); Xuet al\.,[2026b](https://arxiv.org/html/2607.11906#bib.bib48)\), Procgen\(Cobbeet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib27)\), Craftax\(Matthewset al\.,[2024a](https://arxiv.org/html/2607.11906#bib.bib34)\), MiniHack/NLE\(Samvelyanet al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib32); Küttleret al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib33)\), and Jumanji\(Bonnetet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib35)\)are natural candidates because they support many task instances while remaining programmable enough to impose explicit non\-stationary schedules\. This layer should test whether a method that works on clean diagnostics still works when observations are richer, horizons are longer, rewards are sparser, and the task distribution has many modes\. It should also test context efficiency: if a method only succeeds by appending all experience, it may not be viable once the environment family becomes large\.

The third layer is a memory and continual\-protocol layer\. POPGym\(Moradet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib28)\), POPGym Arcade\(Wanget al\.,[2025e](https://arxiv.org/html/2607.11906#bib.bib29)\), NS\-Gym\(Keplingeret al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib44)\), CRLMaze\(Lomonacoet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib37)\), and CORA\(Powerset al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib38)\)are useful here because they foreground memory, partial observability, non\-stationary streams, and continual\-learning metrics\. This layer should measure whether the context policy can handle delayed evidence, old\-observation poisoning, forward transfer, forgetting, and re\-adaptation when old modes return\. Unlike the first two layers, this layer is less about task diversity and more about diagnosing the lifetime of information\.

An optional extended layer can include interactive language environments such as ScienceWorld\(Wanget al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib39)\), WebShop\(Yaoet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib40)\), and ALFWorld\(Shridharet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib41)\)when the paper explicitly studies LLM reward\-feedback agents\. These environments are useful for testing preference drift, natural\-language feedback, tool\-use histories, and changing user instructions\. However, they should remain extended cases unless the experiment contains repeated decisions, external feedback, and a clear mechanism by which future behavior improves through context\. Otherwise, the result is better described as conversational memory or prompting rather than ICRL\. For LLM\-agent bridge cases, a non\-stationary protocol should vary user preferences, reward feedback, hidden instructions, tool schemas, website distributions, or feedback reliability across repeated trials\. Baselines should include no\-memory, last\-kkcontext, full transcript, summary memory, retrieval memory, oracle\-current\-rule context, and stale\-memory adversary variants\. Metrics should include post\-shift success rate, correction latency, invalid tool\-call rate, stale\-memory sensitivity, and preference compliance after drift\.

Across all layers, the stack should be reported as a lifecycle evaluation rather than as a single leaderboard\. The paper should state the shift source, temporal pattern, observability, context budget, and reset rule for every environment\. It should also state whether the environment is intrinsically non\-stationary or merely a programmable substrate used to construct a non\-stationary protocol\. This distinction is important for review: success on a broad procedural benchmark is not automatically success on non\-stationary ICRL unless stale context is present and recovery must occur through context\.

### 8\.9Interpreting Evaluation Outcomes

Evaluation should be diagnostic enough to explain why a method succeeds\. High return with high stale\-context sensitivity suggests that the method is powerful but brittle\. Low stale\-context sensitivity with poor recurring\-mode performance suggests over\-forgetting\. Good recovery with poor dynamic regret may indicate that the method eventually adapts but pays a large cost after every change\. Strong oracle\-change\-point performance and weak learned\-change\-point performance indicates that the main bottleneck is detection or context management rather than policy capacity\.

The conclusion is that non\-stationary ICRL evaluation should not end with a single return number\. A convincing result must identify which failure mode was solved: recognizing that the regime changed, selecting evidence from the current regime, retaining useful evidence from older recurring regimes, or acting well after the relevant context has been selected\. The strongest evidence is therefore a consistent pattern across return, dynamic regret, recovery time, stale\-context sensitivity, and oracle gaps\. If a method improves return but remains sensitive to stale context, the claim should be limited to better exploitation under favorable histories\. If it closes the learned\-to\-oracle change\-point gap, the claim can be about change detection or context filtering\. If it improves recurring\-mode performance without hurting post\-shift recovery, the claim can be about useful long\-term memory\. This interpretation discipline keeps benchmark results tied to the central question of the survey: whether a frozen policy can use context to adapt under changing decision processes, not merely whether it performs well on a diverse task suite\.

## 9Theory and Conceptual Links

### 9\.1Existing Theoretical Coverage

Existing theory has begun to make ICRL more than an analogy between prompting and reinforcement learning\. Supervised pretraining analyses show that decision transformers can implement in\-context policies under structured task distributions\(Leeet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib13); Linet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib14)\)\. General ICL theory supplies the broader mechanisms: transformers can learn simple function classes in context\(Garget al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib128); Akyureket al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib129)\), encode implicit models or gradient\-descent\-like computations inside the forward pass\(von Oswaldet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib130); Akyureket al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib184)\), and depend strongly on demonstration structure\(Minet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib182); Wieset al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib183)\)\. Recent work studies chain\-of\-thought or explicit iterative computation as a path toward in\-context reinforcement learning\(Xieet al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib74)\)\. Other work shows that transformers can implement policy\-improvement\-like procedures in context\(Liang and Lai,[2026](https://arxiv.org/html/2607.11906#bib.bib75)\)\. These results are important because they show that ICRL is not merely an empirical analogy: under certain assumptions, the architecture can represent update\-like computations\. Adjacent non\-stationary ICL theory\(Qinet al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib78)\), while not reinforcement learning, also supports the relevance of recency, adaptive filtering\(Sayed,[2011](https://arxiv.org/html/2607.11906#bib.bib235)\), gating\(Yanget al\.,[2023b](https://arxiv.org/html/2607.11906#bib.bib236); Katsch,[2023](https://arxiv.org/html/2607.11906#bib.bib238)\), retention\-style sequence models\(Sunet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib237)\), and recurrent attention mechanisms\(Penget al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib239)\)when the target function changes over time\. Long\-context ICL studies provide a complementary caution: increasing the number of examples can help only when the model can keep the relevant evidence accessible and distinguish it from distractors\(Liet al\.,[2024b](https://arxiv.org/html/2607.11906#bib.bib240); Bertschet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib241)\)\.

### 9\.2Added Challenges Under Non\-Stationarity

Non\-stationarity adds two theoretical problems that are easy to blur together\. The comparator problem asks what the agent should be compared against when the optimal policy changes over time\. Dynamic regret is the natural answer in many settings, including adversarial or bandit settings\(Aueret al\.,[2002](https://arxiv.org/html/2607.11906#bib.bib119); Garivier and Moulines,[2011](https://arxiv.org/html/2607.11906#bib.bib118); Besbeset al\.,[2014](https://arxiv.org/html/2607.11906#bib.bib82)\), adaptive\-window or weighted bandits\(Aueret al\.,[2019](https://arxiv.org/html/2607.11906#bib.bib145); Russacet al\.,[2019](https://arxiv.org/html/2607.11906#bib.bib143); Li and Li,[2019](https://arxiv.org/html/2607.11906#bib.bib150)\), non\-stationary MDPs\(Cheunget al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib89); Feiet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib90); Fenget al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib81)\), refined MDP bounds\(Wei and Luo,[2021](https://arxiv.org/html/2607.11906#bib.bib147); Maoet al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib148); Domingueset al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib149)\), and transformer\-based analyses\(Chenet al\.,[2025a](https://arxiv.org/html/2607.11906#bib.bib55)\)\. The validity problem asks which context items should influence the current policy\. This part is less developed theoretically\. Most in\-context theory assumes that examples are sampled from a coherent latent task or distribution\. Non\-stationary ICRL requires theory for contexts that are finite, ordered, partially invalid, and sometimes contradictory\. POMDP\(Kaelblinget al\.,[1998](https://arxiv.org/html/2607.11906#bib.bib114)\), CMDP\(Hallaket al\.,[2015](https://arxiv.org/html/2607.11906#bib.bib115)\), epistemic\-POMDP\(Ghoshet al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib153)\), task\-inference\(Humpliket al\.,[2019](https://arxiv.org/html/2607.11906#bib.bib165)\), and HiP\-MDP theory\(Killianet al\.,[2017](https://arxiv.org/html/2607.11906#bib.bib116); Doshi\-Velez and Konidaris,[2016](https://arxiv.org/html/2607.11906#bib.bib141)\)suggest one route: treat the current regime as latent and context as evidence about that regime\. The missing piece is that an ICRL agent observes this evidence through a bounded and possibly learned context channel, so inference error and context\-selection error become part of the control problem\. One useful decomposition is to separate environmental movement from context\-management error:

RegTdyn\(πθ,g\)≤\\displaystyle\\mathrm\{Reg\}^\{\\mathrm\{dyn\}\}\_\{T\}\(\\pi\_\{\\theta\},g\)\\leq\{\}𝒜\(T,BT\)⏟move\+∑t=1Tϵctx\(t;g\)⏟ctx\+∑t=1Tϵmodel\(t;θ\)⏟model\.\\displaystyle\\underbrace\{\\mathcal\{A\}\(T,B\_\{T\}\)\}\_\{\\mathrm\{move\}\}\+\\underbrace\{\\sum\_\{t=1\}^\{T\}\\epsilon\_\{\\mathrm\{ctx\}\}\(t;g\)\}\_\{\\mathrm\{ctx\}\}\+\\underbrace\{\\sum\_\{t=1\}^\{T\}\\epsilon\_\{\\mathrm\{model\}\}\(t;\\theta\)\}\_\{\\mathrm\{model\}\}\.\(19\)HereBTB\_\{T\}is a variation budget: the movement term captures adaptation to environmental drift, the context term captures retrieval, compression, or stale\-context mistakes, and the model term captures the approximation limits of the frozen decision model\. This is not a theorem as stated; it is a target form for future theory that would make context\-management assumptions explicit\. The main gap is therefore an interface between ICRL theory, non\-stationary RL regret analysis, and retrieval or memory theory: the target object is not only a changing MDP, but a changing MDP observed through a bounded context channel\.

### 9\.3Open Theoretical Questions

Several questions would make the theory more precise:

- •Which variation budgets or change\-point assumptions make fixed\-parameter context adaptation competitive with online updating?
- •How should regret bounds scale with context length, memory size, retrieval error, and shift observability?
- •When is stale context harmless because attention can suppress it, and when does it induce unavoidable lower bounds?
- •How do value\-aware and Bayesian context mechanisms alter sample complexity under reward or dynamics shift?
- •Can recurring\-mode memory be given a formal advantage over recency\-only adaptation?

Table[17](https://arxiv.org/html/2607.11906#S9.T17)maps existing theory topics to the non\-stationary pieces that remain missing\.

A useful first step is to make the underlying assumptions explicit:

- •Under bounded variation in rewards or dynamics, theory could characterize how regret scales with context length\.
- •Under a finite set of recurring latent regimes, theory could identify when retrieval reduces re\-adaptation latency\.
- •Under adversarial stale context, theory could characterize robustness guarantees for attention or retrieval policies\.

These stylized settings would not capture the full complexity of deployment, but they would turn vague claims about “using context” into testable statements about when context helps and when it provably hurts\.

Table 17:Theory topics and their relevance to non\-stationary ICRL\.

## 10Design Principles and Open Problems

### 10\.1Design Principles

The preceding sections point to several design principles for non\-stationary ICRL systems\.

1. 1\.Treat context as a managed resource\.Longer context is useful only when the system can identify what remains valid, a lesson shared by long\-memory LLM systems\(Zhonget al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib108); Wanget al\.,[2023b](https://arxiv.org/html/2607.11906#bib.bib109); Packeret al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib110)\), long\-context ICL studies\(Liet al\.,[2024b](https://arxiv.org/html/2607.11906#bib.bib240); Bertschet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib241)\), efficient sequence models\(Luet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib123)\), and long\-context ICRL agents\(Elawadyet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib100)\)\.
2. 2\.Train on invalidation, not only adaptation\.Training histories should include moments where old evidence becomes wrong, not only stationary improvement traces from algorithm distillation\(Laskinet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib3)\), noise or history filtering\(Zismanet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib25); Chenet al\.,[2025b](https://arxiv.org/html/2607.11906#bib.bib53)\), continual or suboptimal ICRL\(Wanget al\.,[2025d](https://arxiv.org/html/2607.11906#bib.bib59); Donget al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib133)\), or environment design\(Denniset al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib220); Jianget al\.,[2021b](https://arxiv.org/html/2607.11906#bib.bib221)\)\.
3. 3\.Separate recency from relevance\.Recency is a useful heuristic in sliding\-window and discounted methods\(Garivier and Moulines,[2011](https://arxiv.org/html/2607.11906#bib.bib118); Russacet al\.,[2019](https://arxiv.org/html/2607.11906#bib.bib143); Trovoet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib151)\), but it fails under changepoints and recurring modes\(Adams and MacKay,[2007](https://arxiv.org/html/2607.11906#bib.bib117)\)\.
4. 4\.Expose retrieval to stale\-memory tests\.Retrieval should be evaluated by validity, not only similarity or return, in retrieval\-based RL\(Goyalet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib199); Humphreyset al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib200)\)and retrieval\-augmented ICRL\(Schmiedet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib20); Sridharet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib21)\)\.
5. 5\.Report lifecycle curves\.A single average return hides post\-shift failure and recovery in non\-stationary MDPs\(Cheunget al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib89); Feiet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib90); Wei and Luo,[2021](https://arxiv.org/html/2607.11906#bib.bib147)\)and continual ICRL\(Wanget al\.,[2025d](https://arxiv.org/html/2607.11906#bib.bib59)\)\.
6. 6\.Include oracle context baselines\.Oracle\-change\-point baselines\(Adams and MacKay,[2007](https://arxiv.org/html/2607.11906#bib.bib117)\)and general ICRL reporting baselines\(Moeiniet al\.,[2025b](https://arxiv.org/html/2607.11906#bib.bib1)\)estimate the headroom available from better context management\.
7. 7\.State the adaptation channel\.Papers should make clear whether adaptation occurs through prompt tokens, hidden state, external memory, retrieved trajectories, summaries, or online parameter updates\.

### 10\.2Open Problems

Not every item in the research agenda is open in the same sense\. Some topics already have direct starting points, such as continual ICRL benchmarks and metrics\(Wanget al\.,[2025d](https://arxiv.org/html/2607.11906#bib.bib59)\), dynamic\-regret analyses for non\-stationary transformers\(Chenet al\.,[2025a](https://arxiv.org/html/2607.11906#bib.bib55)\), history filtering\(Chenet al\.,[2025b](https://arxiv.org/html/2607.11906#bib.bib53)\), retrieval\-augmented ICRL\(Schmiedet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib20); Sridharet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib21)\), and safe ICRL mechanisms\(Moeiniet al\.,[2025a](https://arxiv.org/html/2607.11906#bib.bib57); Kwonet al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib58)\)\. Other topics are seeded only by adjacent literatures, such as changepoint detection, continual learning, retrieval, or safe RL, and still need to be reformulated under the fixed\-parameter context\-adaptation constraint\. A third group is more genuinely open: the field lacks clear benchmarks, formal objectives, or accepted measurement protocols\. TableLABEL:tab:open\-problemstherefore distinguishes*seeded*directions from*partly seeded*and*truly open*ones\.

Table 18:A status\-aware roadmap for non\-stationary ICRL\.ProblemCurrent StatusActionable Next StepStandardized Lifecycle BenchmarksSeeded\.Continual ICRL gives direct benchmark and metric examples, including post\-shift recovery and lifecycle reporting\(Wanget al\.,[2025d](https://arxiv.org/html/2607.11906#bib.bib59)\); broader benchmark reporting norms remain fragmented\.Consolidate a shared suite with declared shift source, temporal pattern, observability, context reset policy, stale\-context ablations, and oracle\-context headroom\(Moeiniet al\.,[2025b](https://arxiv.org/html/2607.11906#bib.bib1); Kirket al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib86); Wanget al\.,[2025d](https://arxiv.org/html/2607.11906#bib.bib59)\)\.Change\-Point\-Aware Context PoliciesSeeded but unresolved\.Changepoint methods and continual ICRL show useful templates\(Adams and MacKay,[2007](https://arxiv.org/html/2607.11906#bib.bib117); Wanget al\.,[2025d](https://arxiv.org/html/2607.11906#bib.bib59)\), while transformer dynamic\-regret work studies related adaptation objectives\(Chenet al\.,[2025a](https://arxiv.org/html/2607.11906#bib.bib55)\)\.Learn write/retrieve/forget rules from weakly labeled or unlabeled shifts, then evaluate detection delay, context repair, and post\-shift return together\.Validity\-Aware RetrievalPartly seeded\.Retrieval\-augmented ICRL and retrieval\-based RL exist\(Goyalet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib199); Schmiedet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib20); Sridharet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib21)\), but most retrieval keys still do not estimate decision validity after shifts\.Retrieve by latent mode, uncertainty, reward fit, recency, and time metadata; report precision of valid retrieved items, not only downstream return\.Adaptive Forgetting And Memory IsolationPartly seeded\.Continual learning and lifelong memory offer replay, regularization, and isolation mechanisms\(Khetarpalet al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib79); Kirkpatricket al\.,[2017](https://arxiv.org/html/2607.11906#bib.bib210); Zhonget al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib108)\), but context\-only agents need different guarantees\.Partition memory by inferred regime and test both irreversible shifts and recurring modes, where aggressive forgetting and indiscriminate retention fail in opposite ways\.Finite\-Context Theory Under Stale EvidenceTruly open\.ICL theory, non\-stationary regret, and chain\-of\-thought ICRL theory provide components\(Garget al\.,[2022](https://arxiv.org/html/2607.11906#bib.bib128); Cheunget al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib89); Xieet al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib74)\), but not a theory of bounded context containing invalid examples\.Bound regret or sample complexity as a function of context length, retrieval error, stale\-context mass, shift observability, and model approximation error\.Efficient Long\-Memory ArchitecturesPartly seeded\.S4, Mamba, and long\-context agents address cost and memory length\(Luet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib123); Gu and Dao,[2023](https://arxiv.org/html/2607.11906#bib.bib187); Elawadyet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib100)\), but rarely under matched invalidation tests\.Compare compressed\-memory models under identical abrupt, gradual, and recurring shifts, with oracle\-reset and stale\-state ablations\.Action\-Semantics ShiftPartly seeded\.Variable\-action ICRL and changing\-action RL cover action\-set variation\(Chandaket al\.,[2020b](https://arxiv.org/html/2607.11906#bib.bib24); Siniiet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib23)\), but within\-lifetime interface drift remains under\-tested\.Test permutations, masks, rescaling, action\-description changes, and interface remapping while old context still contains obsolete action tokens\.Context PoisoningTruly open for non\-stationary ICRL\.Safety and Bayesian value\-fusion work are relevant\(Moeiniet al\.,[2025a](https://arxiv.org/html/2607.11906#bib.bib57); Berkeset al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib61)\), but adversarial stale memories are not yet a standard threat model\.Stress\-test stale demonstrations, malicious retrievals, corrupted rewards, and misleading summaries; measure whether the agent can isolate harmful context without online parameter updates\.Verbal Memory Validity In LLM AgentsPartly seeded\.Reflection and experience\-library agents store natural\-language lessons\(Shinnet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib73); Zhaoet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib106); Wanget al\.,[2023a](https://arxiv.org/html/2607.11906#bib.bib105)\), but validity under preference, tool, or reward drift is rarely measured\.Timestamp, regime\-tag, contradict, retire, or reactivate memories using reward and feedback evidence rather than semantic similarity alone\.Constraint And Safety ShiftsSeeded\.Safe ICRL and barrier\-style shielding provide mechanisms\(Altman,[2021](https://arxiv.org/html/2607.11906#bib.bib228); Moeiniet al\.,[2025a](https://arxiv.org/html/2607.11906#bib.bib57); Kwonet al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib58)\), but shifting constraints require separate reward\-validity and safety\-validity tests\.Add safety filters that work without parameter updates and report reward, cost, and violation recovery after constraint changes\.Weak\-Data Non\-StationarityPartly seeded\.Offline RL and value\-aware ICRL address weak or suboptimal data\(Fuet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib120); Kumaret al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib96); Berkeset al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib61)\), but changing data quality is usually not isolated from changing tasks\.Factor benchmarks by behavior\-policy quality, reward shift, support mismatch, and uncertainty, so failures are not attributed only to non\-stationarity\.Exploration Under Changing FeedbackPartly seeded\.In\-context exploration and LLM bandit work study diagnostic action choice\(Krishnamurthyet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib102); Moneaet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib103); Nieet al\.,[2025](https://arxiv.org/html/2607.11906#bib.bib104)\), but feedback drift adds a second inference problem\.Combine in\-context exploration with shift\-aware uncertainty summaries and report how quickly the agent collects disambiguating evidence after feedback changes\.Table 18:A status\-aware roadmap for non\-stationary ICRL \(continued\)\.Safety is a related but distinct pressure on this agenda\. Safe ICRL extends the adaptation problem to constrained decision processes\(Altman,[2021](https://arxiv.org/html/2607.11906#bib.bib228); Achiamet al\.,[2017](https://arxiv.org/html/2607.11906#bib.bib229); Satijaet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib230)\), where context must help the agent improve reward while respecting cost budgets\(Moeiniet al\.,[2025a](https://arxiv.org/html/2607.11906#bib.bib57)\)\. Recent shielding work makes the same point more explicit under OOD deployment shifts: Q\-Barrier\(Kwonet al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib58)\)suggests that a frozen safe\-ICRL policy may need an action\-level latent budget and cost filter, not only a longer history, to preserve a useful reward\-safety tradeoff\. The non\-stationary version is harder because a context item may be reward\-valid but safety\-invalid after a constraint or cost shift\. This reinforces the need to report validity with respect to rewards, dynamics, and constraints separately\. Classical safe exploration\(Moldovan and Abbeel,[2012](https://arxiv.org/html/2607.11906#bib.bib232)\), model\-based safety guarantees\(Berkenkampet al\.,[2017](https://arxiv.org/html/2607.11906#bib.bib233)\), safe policy improvement\(Larocheet al\.,[2019](https://arxiv.org/html/2607.11906#bib.bib231)\), and modern safe\-RL benchmarks\(Jiet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib234)\)provide useful components, but they still need to be recast under the fixed\-parameter, context\-only adaptation constraint\.

### 10\.3A Research Agenda

The next phase of the field should move beyond the broad question of whether context helps\. For non\-stationary ICRL, the sharper questions are which context helps, when it helps, when it hurts, and how a frozen agent can know the difference\. The roadmap should proceed in stages\. The first stage is consolidation: seeded directions such as continual ICRL benchmarks\(Wanget al\.,[2025d](https://arxiv.org/html/2607.11906#bib.bib59)\), safe ICRL\(Moeiniet al\.,[2025a](https://arxiv.org/html/2607.11906#bib.bib57); Kwonet al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib58)\), and retrieval\-augmented ICRL\(Schmiedet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib20); Sridharet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib21)\)should be evaluated under a common lifecycle protocol rather than treated as isolated demonstrations\. The second stage is translation: adjacent tools from changepoint detection, non\-stationary regret, continual learning, retrieval, and safe RL should be rewritten in context\-channel terms, specifying what is stored, retrieved, suppressed, or exposed to the frozen model\. The third stage is genuinely open theory and stress testing: finite\-context bounds, adversarial stale\-context threat models, and validity\-aware memory metrics should become standard targets\.

Claims about non\-stationary adaptation should specify the shift source, temporal pattern, observability regime, context policy, and training exposure\. They should report post\-shift curves and stale\-context ablations, compare against simple context policies before attributing gains to architecture scale, and state whether a result is solving a new open problem or extending an already seeded line\.

The agenda also requires a cleaner separation between the pretrained decision model, the context manager, and the memory store\. Current papers often blur these components together\. In stationary settings, that may be tolerable because most useful context points in the same direction\. In non\-stationary settings, the distinction is central\. A strong policy with a weak context manager can fail, while a modest policy with a good validity filter can recover quickly\.

### 10\.4Reporting Standards for a Mature Literature

A mature non\-stationary ICRL literature would make claims at the right level of specificity\. Instead of saying that a model handles non\-stationarity, a paper would say that it handles reward/abrupt/partial shifts with a finite context budget, or dynamics/recurring/latent shifts with an external memory\. Instead of reporting only average return, it would show lifecycle curves, recovery time, stale\-context sensitivity, and oracle\-context headroom\. Instead of treating retrieval as a black\-box improvement, it would report whether retrieved items are valid under the current regime\. Instead of presenting long context as inherently beneficial, it would show when additional context helps, when it hurts, and what mechanism controls its influence\.

This reporting standard is demanding, but it is aligned with the central promise of ICRL\. If adaptation is supposed to happen without weight updates, then the evidence passed to the frozen model is the adaptation interface\. The field should therefore scrutinize that interface with the same care that reinforcement\-learning papers apply to rewards, dynamics, and exploration\.

## 11Conclusion

This survey has argued that non\-stationary In\-context reinforcement learning is not just ICRL evaluated on a larger or more diverse task distribution\. It is the problem of adapting through context while deployed policy parameters remain fixed, even when accumulated evidence becomes obsolete, misleading, or useful again after a regime returns\. The key difficulty is context validity: a fixed\-parameter policy must infer which accumulated evidence still describes the current decision process and which evidence should be downweighted, retired, or reactivated\. This view connects black\-box meta\-RL\(Duanet al\.,[2016](https://arxiv.org/html/2607.11906#bib.bib5); Wanget al\.,[2016](https://arxiv.org/html/2607.11906#bib.bib6)\), decision sequence modeling\(Chenet al\.,[2021](https://arxiv.org/html/2607.11906#bib.bib4); Leeet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib13)\), algorithm distillation\(Laskinet al\.,[2023](https://arxiv.org/html/2607.11906#bib.bib3)\), retrieval\-augmented agents\(Schmiedet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib20); Sridharet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib21)\), and long\-context adaptive systems\(Elawadyet al\.,[2024](https://arxiv.org/html/2607.11906#bib.bib100)\), but it shifts the organizing question from “does context help?” to “when should context be trusted?”\.

From this perspective, the survey made five contributions to the non\-stationary ICRL problem\.First,it defined the setting and separated it from adjacent notions such as stationary few\-shot adaptation, domain randomization, online continual learning, standard offline RL, robotics\-specific lifelong learning, and multi\-agent strategic non\-stationarity\. The defining constraint is that adaptation occurs through context while the deployed policy parameters remain fixed\.Second,it introduced a taxonomy that describes a non\-stationary ICRL setting along three axes: what changes, how the change unfolds over time, and how observable the change is to the agent\. This source–pattern–observability vocabulary is intended to prevent overbroad claims: reward specifications, transition kernels, action interfaces, observation channels, constraint models, and data sources create different problems, and abrupt, gradual, recurring, expanding, or adversarial changes require different evidence\-management behavior\.Third,the survey recast methods as context\-management mechanisms rather than only as model families\. Raw history conditioning, context filtering, retrieval, compression, recurrent memory, value\-aware inference, model\-based prediction, reflection, and forgetting are all ways of deciding which past evidence reaches the frozen decision policy and how strongly it should influence action\. Under non\-stationarity, these mechanisms are not engineering details; they are the adaptation channel\.Fourth,the survey translated that view into training and evaluation requirements\. Training data should contain invalidation, mixed pre\-shift and post\-shift histories, stale demonstrations, suboptimal or mismatched trajectories, recurring modes, and finite\-context pressure\(Chenet al\.,[2025b](https://arxiv.org/html/2607.11906#bib.bib53); Donget al\.,[2026](https://arxiv.org/html/2607.11906#bib.bib133)\)\. Evaluation should then report lifecycle curves, post\-shift recovery, dynamic regret, stale\-context sensitivity, retrieval utility, oracle\-context gaps, recurring\-mode reuse, and finite\-context degradation rather than only average return\. Benchmarks such as procedural environments, continual RL suites, and memory\-focused tasks become evidence for non\-stationary ICRL only when they impose within\-lifetime shifts and require recovery through context\.Fifth,the survey identified the theoretical gap created by stale and conflicting context\. Existing change\-point methods\(Adams and MacKay,[2007](https://arxiv.org/html/2607.11906#bib.bib117)\), non\-stationary bandit regret\(Besbeset al\.,[2014](https://arxiv.org/html/2607.11906#bib.bib82)\), non\-stationary MDP regret\(Cheunget al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib89); Feiet al\.,[2020](https://arxiv.org/html/2607.11906#bib.bib90)\), and continual ICRL\(Wanget al\.,[2025d](https://arxiv.org/html/2607.11906#bib.bib59)\)provide useful pieces, but they do not yet explain how regret, sample efficiency, or robustness should scale with context length, retrieval error, memory size, shift observability, and context poisoning\. A mature theory of non\-stationary ICRL should treat the context manager as part of the control system, because the agent acts through a bounded and imperfect evidence channel\.

The practical conclusion is that future work should make its claim at the level of the shift it actually solves\. A method that improves return under clean histories is not yet a non\-stationary ICRL solution if it fails under stale context\. A method that recovers after abrupt reward changes has not necessarily solved gradual dynamics drift or hidden action remapping\. A method that forgets quickly may adapt after one\-way change but fail when old regimes recur\. Progress in non\-stationary ICRL will come from systems that can detect change, select currently valid evidence, preserve reusable memories, reject obsolete context, and act well under a finite context budget\. The central benchmark for the field is therefore not larger context alone, but disciplined evidence use across the lifetime of a changing decision process\.

## References

- D\. Abel, A\. Barreto, B\. Van Roy, D\. Precup, H\. van Hasselt, and S\. Singh \(2023\)A definition of continual reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 50377–50407\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.4.2.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p4.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p5.1),[§3\.2](https://arxiv.org/html/2607.11906#S3.SS2.p1.5),[Table 6](https://arxiv.org/html/2607.11906#S4.T6.1.6.5.3.1.1),[Table 12](https://arxiv.org/html/2607.11906#S8.T12.1.8.7.3.1.1)\.
- J\. Achiam, D\. Held, A\. Tamar, and P\. Abbeel \(2017\)Constrained policy optimization\.InInternational Conference on Machine Learning,Cited by:[§10\.2](https://arxiv.org/html/2607.11906#S10.SS2.p2.1)\.
- R\. P\. Adams and D\. J\. C\. MacKay \(2007\)Bayesian online changepoint detection\.arXiv preprint arXiv:0710\.3742\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.6.4.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p3.1),[item 3](https://arxiv.org/html/2607.11906#S10.I1.i3.p1.1),[item 6](https://arxiv.org/html/2607.11906#S10.I1.i6.p1.1),[Table 18](https://arxiv.org/html/2607.11906#S10.T18.5.4.2.2.1.1),[§11](https://arxiv.org/html/2607.11906#S11.p2.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p4.1),[§3\.1](https://arxiv.org/html/2607.11906#S3.SS1.p3.1),[Table 4](https://arxiv.org/html/2607.11906#S3.T4.1.6.5.3.1.1),[Figure 6](https://arxiv.org/html/2607.11906#S4.F6.1.pic1.5.5.5.1.1.2.1.1),[Table 6](https://arxiv.org/html/2607.11906#S4.T6.1.2.1.3.1.1),[§5\.1](https://arxiv.org/html/2607.11906#S5.SS1.p1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.8.7.2.1.1),[§8\.5](https://arxiv.org/html/2607.11906#S8.SS5.p3.1),[Table 12](https://arxiv.org/html/2607.11906#S8.T12.1.3.2.3.1.1),[Table 16](https://arxiv.org/html/2607.11906#S8.T16.1.2.1.2.1.1)\.
- R\. Agarwal, D\. Schuurmans, and M\. Norouzi \(2020\)An optimistic perspective on offline reinforcement learning\.InInternational Conference on Machine Learning,Cited by:[§7\.1](https://arxiv.org/html/2607.11906#S7.SS1.p1.1),[§7\.4](https://arxiv.org/html/2607.11906#S7.SS4.p1.1),[§8\.3](https://arxiv.org/html/2607.11906#S8.SS3.p2.1)\.
- E\. Akyurek, D\. Schuurmans, J\. Andreas, T\. Ma, and D\. Zhou \(2023\)What learning algorithm is in\-context learning? investigations with linear models\.InInternational Conference on Learning Representations,Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.9.7.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p3.2),[Figure 8](https://arxiv.org/html/2607.11906#S5.F8.1.pic1.5.5.5.5.1.1.2.2),[§7\.1](https://arxiv.org/html/2607.11906#S7.SS1.p5.1),[§9\.1](https://arxiv.org/html/2607.11906#S9.SS1.p1.1),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.2.1.2.1.1)\.
- E\. Akyurek, B\. Wang, Y\. Kim, and J\. Andreas \(2024\)In\-context language learning: architectures and algorithms\.arXiv preprint arXiv:2401\.12973\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.9.7.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p3.2),[§5\.2](https://arxiv.org/html/2607.11906#S5.SS2.p3.1),[§6\.1](https://arxiv.org/html/2607.11906#S6.SS1.p1.1),[§7\.1](https://arxiv.org/html/2607.11906#S7.SS1.p5.1),[§9\.1](https://arxiv.org/html/2607.11906#S9.SS1.p1.1)\.
- M\. Al\-Shedivat, T\. Bansal, Y\. Burda, I\. Sutskever, I\. Mordatch, and P\. Abbeel \(2017\)Continuous adaptation via meta\-learning in nonstationary and competitive environments\.arXiv preprint arXiv:1710\.03641\.Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p3.1),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.4.8.2.4.1.1)\.
- R\. Aljundi, F\. Babiloni, M\. Elhoseiny, M\. Rohrbach, and T\. Tuytelaars \(2018\)Memory aware synapses: learning what not to forget\.InEuropean Conference on Computer Vision,pp\. 139–154\.Cited by:[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p5.1)\.
- E\. Altman \(2021\)Constrained markov decision processes\.Routledge\.Cited by:[§10\.2](https://arxiv.org/html/2607.11906#S10.SS2.p2.1),[Table 18](https://arxiv.org/html/2607.11906#S10.T18.5.12.10.2.1.1)\.
- P\. Auer, N\. Cesa\-Bianchi, Y\. Freund, and R\. E\. Schapire \(2002\)The nonstochastic multiarmed bandit problem\.InSIAM Journal on Computing,pp\. 48–77\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.6.4.2.1.1),[§3\.3](https://arxiv.org/html/2607.11906#S3.SS3.p1.2),[§9\.2](https://arxiv.org/html/2607.11906#S9.SS2.p1.2),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.5.4.2.1.1)\.
- P\. Auer, P\. Gajane, and R\. Ortner \(2019\)Adaptively tracking the best bandit arm with an unknown number of distribution changes\.InConference on Learning Theory,pp\. 138–158\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.6.4.2.1.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p4.1),[§3\.1](https://arxiv.org/html/2607.11906#S3.SS1.p3.1),[Table 6](https://arxiv.org/html/2607.11906#S4.T6.1.2.1.3.1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.8.7.2.1.1),[§9\.2](https://arxiv.org/html/2607.11906#S9.SS2.p1.2)\.
- A\. Bakhtin, L\. van der Maaten, J\. Johnson, L\. Gustafson, and R\. Girshick \(2019\)PHYRE: a new benchmark for physical reasoning\.InAdvances in Neural Information Processing Systems,Cited by:[§8\.3](https://arxiv.org/html/2607.11906#S8.SS3.p2.1),[§8\.8](https://arxiv.org/html/2607.11906#S8.SS8.p2.1),[Table 13](https://arxiv.org/html/2607.11906#S8.T13.5.9.7.2.1.1)\.
- J\. Bauer, K\. Baumli, F\. M\. P\. Behbahani, A\. Bhoopchand, N\. Bradley\-Schmieg, M\. Chang, N\. Clay, A\. Collister, V\. Dasagi, L\. Gonzalez, K\. Gregor, E\. Hughes, S\. Kashem, M\. Loks\-Thompson, H\. Openshaw, J\. Parker\-Holder, S\. Pathak, N\. Perez Nieves, N\. Rakicevic, T\. Rocktäschel, Y\. Schroecker, S\. Singh, J\. Sygnowski, K\. Tuyls, S\. York, A\. Zacherl, and L\. M\. Zhang \(2023\)Human\-timescale adaptation in an open\-ended task space\.InInternational Conference on Machine Learning,Cited by:[§6\.4](https://arxiv.org/html/2607.11906#S6.SS4.p1.1),[§7\.3](https://arxiv.org/html/2607.11906#S7.SS3.p1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.5.4.2.1.1)\.
- D\. Baumann, E\. Noorani, A\. Mustafin, X\. Sheng, B\. Verbruggen, A\. Vanhoyweghen, V\. Ginis, and T\. B\. Schön \(2026\)Ergodicity in reinforcement learning\.arXiv preprint arXiv:2603\.10895\.Cited by:[§3\.3](https://arxiv.org/html/2607.11906#S3.SS3.p1.1),[§3\.3](https://arxiv.org/html/2607.11906#S3.SS3.p1.2)\.
- S\. Beaussant and M\. Mounsif \(2025\)Scaling algorithm distillation for continuous control with mamba\.arXiv preprint arXiv:2506\.13892\.Cited by:[§6\.3](https://arxiv.org/html/2607.11906#S6.SS3.p1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.9.8.2.1.1)\.
- J\. Beck, R\. Vuorio, E\. Z\. Liu, Z\. Xiong, L\. Zintgraf, C\. Finn, and S\. Whiteson \(2023\)A survey of meta\-reinforcement learning\.arXiv preprint arXiv:2301\.08028\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.7.5.2.1.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p3.1)\.
- M\. Beck, K\. Poppel, M\. Spanring, A\. Auer, O\. Prudnikova, M\. Kopp, G\. Klambauer, J\. Brandstetter, and S\. Hochreiter \(2024\)XLSTM: extended long short\-term memory\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 107547–107603\.Cited by:[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p2.1),[Figure 10](https://arxiv.org/html/2607.11906#S6.F10.1.pic1.3.3.3.1.1.2.1.1),[§6\.3](https://arxiv.org/html/2607.11906#S6.SS3.p1.1)\.
- F\. Berkenkamp, M\. Turchetta, A\. P\. Schoellig, and A\. Krause \(2017\)Safe model\-based reinforcement learning with stability guarantees\.InAdvances in Neural Information Processing Systems,Cited by:[§10\.2](https://arxiv.org/html/2607.11906#S10.SS2.p2.1)\.
- A\. Berkes, V\. Taboga, D\. Vakalis, D\. Rolnick, and Y\. Bengio \(2026\)In\-context reinforcement learning through bayesian fusion of context and value prior\.arXiv preprint arXiv:2601\.03015\.Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[Table 18](https://arxiv.org/html/2607.11906#S10.T18.5.10.8.2.1.1),[Table 18](https://arxiv.org/html/2607.11906#S10.T18.5.13.11.2.1.1),[Figure 3](https://arxiv.org/html/2607.11906#S2.F3.1.pic1.10.10.10.1.1.5),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p2.1),[§3\.2](https://arxiv.org/html/2607.11906#S3.SS2.p2.1),[§3\.4](https://arxiv.org/html/2607.11906#S3.SS4.p2.1),[Table 4](https://arxiv.org/html/2607.11906#S3.T4.1.7.6.3.1.1),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.4.9.3.4.1.1),[Table 7](https://arxiv.org/html/2607.11906#S4.T7.1.4.3.3.1.1),[Figure 8](https://arxiv.org/html/2607.11906#S5.F8.1.pic1.1.1.1.1.1.1.2.2),[Figure 8](https://arxiv.org/html/2607.11906#S5.F8.1.pic1.5.5.5.5.1.1.2.2),[§5\.4](https://arxiv.org/html/2607.11906#S5.SS4.p1.1),[Figure 10](https://arxiv.org/html/2607.11906#S6.F10.1.pic1.7.7.7.1.1.2.1.1),[Figure 9](https://arxiv.org/html/2607.11906#S6.F9.1.pic1.7.7.7.1.1.2.1.1),[§6\.6](https://arxiv.org/html/2607.11906#S6.SS6.p1.1),[Table 10](https://arxiv.org/html/2607.11906#S6.T10.1.7.6.2.1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.4.3.2.1.1),[Table 12](https://arxiv.org/html/2607.11906#S8.T12.1.5.4.3.1.1),[Table 14](https://arxiv.org/html/2607.11906#S8.T14.1.6.5.2.1.1),[Table 16](https://arxiv.org/html/2607.11906#S8.T16.1.3.2.2.1.1),[Table 16](https://arxiv.org/html/2607.11906#S8.T16.1.9.8.2.1.1),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.6.5.2.1.1)\.
- A\. Bertsch, M\. Ivgi, E\. Xiao, U\. Alon, J\. Berant, M\. R\. Gormley, and G\. Neubig \(2025\)In\-context learning with long\-context models: an in\-depth exploration\.InAssociation for Computational Linguistics,pp\. 12119–12149\.Cited by:[item 1](https://arxiv.org/html/2607.11906#S10.I1.i1.p1.1),[§5\.2](https://arxiv.org/html/2607.11906#S5.SS2.p4.1),[§9\.1](https://arxiv.org/html/2607.11906#S9.SS1.p1.1)\.
- O\. Besbes, Y\. Gur, and A\. Zeevi \(2014\)Stochastic multi\-armed\-bandit problem with non\-stationary rewards\.InAdvances in Neural Information Processing Systems,Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.5.3.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p3.1),[§11](https://arxiv.org/html/2607.11906#S11.p2.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p4.1),[§3\.1](https://arxiv.org/html/2607.11906#S3.SS1.p3.1),[§3\.2](https://arxiv.org/html/2607.11906#S3.SS2.p1.5),[§3\.3](https://arxiv.org/html/2607.11906#S3.SS3.p1.2),[Table 4](https://arxiv.org/html/2607.11906#S3.T4.1.5.4.3.1.1),[Figure 6](https://arxiv.org/html/2607.11906#S4.F6.1.pic1.6.6.6.1.1.2.1.1),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.1.1.4.1.1),[Table 6](https://arxiv.org/html/2607.11906#S4.T6.1.3.2.3.1.1),[§4](https://arxiv.org/html/2607.11906#S4.p1.1),[§8\.1](https://arxiv.org/html/2607.11906#S8.SS1.p1.1),[§8\.5](https://arxiv.org/html/2607.11906#S8.SS5.p1.4),[Table 12](https://arxiv.org/html/2607.11906#S8.T12.1.4.3.3.1.1),[Table 14](https://arxiv.org/html/2607.11906#S8.T14.1.2.1.2.1.1),[Table 16](https://arxiv.org/html/2607.11906#S8.T16.1.10.9.2.1.1),[§9\.2](https://arxiv.org/html/2607.11906#S9.SS2.p1.2),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.5.4.2.1.1)\.
- Z\. Bing, D\. Lerch, K\. Huang, and A\. Knoll \(2022\)Meta\-reinforcement learning in non\-stationary and dynamic environments\.IEEE Transactions on Pattern Analysis and Machine Intelligence45\(3\),pp\. 3476–3491\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.7.5.2.1.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p2.1)\.
- C\. Bonnet, D\. Luo, D\. Byrne, S\. Surana, S\. Abramowitz, P\. Duckworth, V\. Coyette, L\. I\. Midgley, E\. Tegegn, T\. Kalloniatis, O\. Mahjoub, M\. Macfarlane, A\. P\. Smit, N\. Grinsztajn, R\. Boige, C\. N\. Waters, M\. A\. Mimouni, U\. A\. Mbou Sob, R\. de Kock, S\. Singh, D\. Furelos\-Blanco, V\. Le, A\. Pretorius, and A\. Laterre \(2024\)Jumanji: a diverse suite of scalable reinforcement learning environments in JAX\.InInternational Conference on Learning Representations,Cited by:[§8\.4](https://arxiv.org/html/2607.11906#S8.SS4.p2.1),[§8\.8](https://arxiv.org/html/2607.11906#S8.SS8.p2.1),[Table 13](https://arxiv.org/html/2607.11906#S8.T13.5.13.11.2.1.1),[Table 14](https://arxiv.org/html/2607.11906#S8.T14.1.8.7.2.1.1)\.
- S\. Borgeaud, A\. Mensch, J\. Hoffmann, T\. Cai, E\. Rutherford, K\. Millican, G\. B\. M\. van den Driessche, J\. Lespiau, B\. Damoc, A\. Clark,et al\.\(2022\)Improving language models by retrieving from trillions of tokens\.InInternational Conference on Machine Learning,pp\. 2206–2240\.Cited by:[§5\.1](https://arxiv.org/html/2607.11906#S5.SS1.p1.1),[§6\.5](https://arxiv.org/html/2607.11906#S6.SS5.p2.1)\.
- D\. Brandfonbrener, A\. Bietti, J\. Buckman, R\. Laroche, and J\. Bruna \(2022\)When does return\-conditioned supervised learning work for offline reinforcement learning?\.Advances in Neural Information Processing Systems35,pp\. 1542–1553\.Cited by:[§6\.2](https://arxiv.org/html/2607.11906#S6.SS2.p2.1),[§7\.4](https://arxiv.org/html/2607.11906#S7.SS4.p1.1)\.
- T\. B\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.\(2020\)Language models are few\-shot learners\.Advances in Neural Information Processing Systems\.Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p3.3)\.
- J\. Bruce, M\. Dennis, A\. Edwards, J\. Parker\-Holder, Y\. Shi, E\. Hughes, M\. Lai, A\. Mavalankar, R\. Steigerwald, C\. Apps,et al\.\(2024\)Genie: generative interactive environments\.arXiv preprint arXiv:2402\.15391\.Cited by:[§6\.8](https://arxiv.org/html/2607.11906#S6.SS8.p1.1)\.
- A\. Bulatov, Y\. Kuratov, and M\. Burtsev \(2022\)Recurrent memory transformer\.Advances in Neural Information Processing Systems35,pp\. 11079–11091\.Cited by:[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p2.1),[§6\.3](https://arxiv.org/html/2607.11906#S6.SS3.p1.1)\.
- M\. Caccia, J\. Mueller, T\. Kim, L\. Charlin, and R\. Fakoor \(2022\)Task\-agnostic continual reinforcement learning: in praise of a simple baseline\.arXiv preprint arXiv:2205\.14495\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.4.2.2.1.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p5.1)\.
- S\. C\. Y\. Chan, A\. Santoro, A\. K\. Lampinen, J\. X\. Wang, A\. Singh, P\. H\. Richemond, J\. L\. McClelland, and F\. Hill \(2022\)Data distributional properties drive emergent in\-context learning in transformers\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 18878–18891\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.9.7.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p3.2),[§5\.2](https://arxiv.org/html/2607.11906#S5.SS2.p3.1),[§6\.1](https://arxiv.org/html/2607.11906#S6.SS1.p1.1),[§7\.1](https://arxiv.org/html/2607.11906#S7.SS1.p5.1)\.
- Y\. Chandak, S\. Jordan, G\. Theocharous, M\. White, and P\. S\. Thomas \(2020a\)Towards safe policy improvement for non\-stationary mdps\.Advances in Neural Information Processing Systems33,pp\. 9156–9168\.Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.1.1.4.1.1),[Table 14](https://arxiv.org/html/2607.11906#S8.T14.1.2.1.2.1.1)\.
- Y\. Chandak, G\. Theocharous, J\. Kostas, S\. Jordan, and P\. S\. Thomas \(2020b\)Lifelong learning with a changing action set\.InAAAI Conference on Artificial Intelligence,Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p3.1),[Table 18](https://arxiv.org/html/2607.11906#S10.T18.5.9.7.2.1.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p5.1),[§3\.2](https://arxiv.org/html/2607.11906#S3.SS2.p2.1),[Figure 6](https://arxiv.org/html/2607.11906#S4.F6.1.pic1.3.3.3.1.1.2.1.1),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.4.4.4.1.1),[§4](https://arxiv.org/html/2607.11906#S4.p1.1),[Table 8](https://arxiv.org/html/2607.11906#S5.T8.1.7.6.4.1.1),[item 1](https://arxiv.org/html/2607.11906#S8.I1.i1.p1.1),[Table 14](https://arxiv.org/html/2607.11906#S8.T14.1.4.3.2.1.1),[Table 15](https://arxiv.org/html/2607.11906#S8.T15.4.9.4.2.1.1)\.
- Y\. Chandak, G\. Theocharous, S\. Shankar, S\. Mahadevan, M\. White, and P\. S\. Thomas \(2020c\)Optimizing for the future in non\-stationary mdps\.InInternational Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p3.1),[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p4.1),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.2.2.4.1.1),[§4](https://arxiv.org/html/2607.11906#S4.p1.1),[Table 13](https://arxiv.org/html/2607.11906#S8.T13.5.15.13.2.1.1),[Table 14](https://arxiv.org/html/2607.11906#S8.T14.1.3.2.2.1.1)\.
- A\. Chaudhry, M\. Ranzato, M\. Rohrbach, and M\. Elhoseiny \(2019\)Efficient lifelong learning with a\-gem\.InInternational Conference on Learning Representations,Cited by:[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p5.1),[Figure 8](https://arxiv.org/html/2607.11906#S5.F8.1.pic1.6.6.6.6.1.1.2.2)\.
- B\. Chen, S\. Ito, and M\. Imaizumi \(2025a\)Optimal dynamic regret by transformers for non\-stationary reinforcement learning\.arXiv preprint arXiv:2508\.16027\.Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[§1](https://arxiv.org/html/2607.11906#S1.p5.1),[§10\.2](https://arxiv.org/html/2607.11906#S10.SS2.p1.1),[Table 18](https://arxiv.org/html/2607.11906#S10.T18.5.4.2.2.1.1),[Figure 3](https://arxiv.org/html/2607.11906#S2.F3.1.pic1.10.10.10.1.1.5),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p4.1),[Table 4](https://arxiv.org/html/2607.11906#S3.T4.1.5.4.3.1.1),[Figure 6](https://arxiv.org/html/2607.11906#S4.F6.1.pic1.8.8.8.1.1.2.1.1),[Table 6](https://arxiv.org/html/2607.11906#S4.T6.1.7.6.3.1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.8.7.2.1.1),[item 5](https://arxiv.org/html/2607.11906#S8.I1.i5.p1.1),[§8\.5](https://arxiv.org/html/2607.11906#S8.SS5.p3.1),[Table 12](https://arxiv.org/html/2607.11906#S8.T12.1.4.3.3.1.1),[Table 12](https://arxiv.org/html/2607.11906#S8.T12.1.9.8.3.1.1),[Table 15](https://arxiv.org/html/2607.11906#S8.T15.4.6.1.4.1.1),[Table 16](https://arxiv.org/html/2607.11906#S8.T16.1.8.7.2.1.1),[§9\.2](https://arxiv.org/html/2607.11906#S9.SS2.p1.2),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.5.4.2.1.1)\.
- L\. Chen, K\. Lu, A\. Rajeswaran, K\. Lee, A\. Grover, M\. Laskin, P\. Abbeel, A\. Srinivas, and I\. Mordatch \(2021\)Decision transformer: reinforcement learning via sequence modeling\.InAdvances in Neural Information Processing Systems,Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.8.6.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p1.1),[§11](https://arxiv.org/html/2607.11906#S11.p1.1),[Figure 3](https://arxiv.org/html/2607.11906#S2.F3.1.pic1.7.7.7.1.1.5),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p2.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p3.2),[Table 8](https://arxiv.org/html/2607.11906#S5.T8.1.2.1.4.1.1),[Figure 10](https://arxiv.org/html/2607.11906#S6.F10.1.pic1.2.2.2.1.1.2.1.1),[Figure 9](https://arxiv.org/html/2607.11906#S6.F9.1.pic1.1.1.1.1.1.2.1.1),[§6\.2](https://arxiv.org/html/2607.11906#S6.SS2.p1.1),[§7\.1](https://arxiv.org/html/2607.11906#S7.SS1.p1.1),[§7\.4](https://arxiv.org/html/2607.11906#S7.SS4.p1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.2.1.2.1.1),[Table 15](https://arxiv.org/html/2607.11906#S8.T15.4.6.1.2.1.1)\.
- W\. Chen and S\. Paternain \(2024\)Random policy enables in\-context reinforcement learning within trust horizons\.arXiv preprint arXiv:2410\.19982\.Cited by:[§6\.2](https://arxiv.org/html/2607.11906#S6.SS2.p2.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.4.3.2.1.1)\.
- W\. Chen, X\. Zhang, D\. Subramanian, and S\. Paternain \(2025b\)Filtering learning histories enhances in\-context reinforcement learning\.arXiv preprint arXiv:2505\.15143\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.12.10.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[item 2](https://arxiv.org/html/2607.11906#S10.I1.i2.p1.1),[§10\.2](https://arxiv.org/html/2607.11906#S10.SS2.p1.1),[§11](https://arxiv.org/html/2607.11906#S11.p2.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p2.1),[§3\.1](https://arxiv.org/html/2607.11906#S3.SS1.p2.8),[§3\.4](https://arxiv.org/html/2607.11906#S3.SS4.p2.1),[Table 4](https://arxiv.org/html/2607.11906#S3.T4.1.4.3.3.1.1),[Table 7](https://arxiv.org/html/2607.11906#S4.T7.1.6.5.3.1.1),[Figure 8](https://arxiv.org/html/2607.11906#S5.F8.1.pic1.4.4.4.4.1.1.2.2),[Table 8](https://arxiv.org/html/2607.11906#S5.T8.1.6.5.4.1.1),[Figure 10](https://arxiv.org/html/2607.11906#S6.F10.1.pic1.1.1.1.1.1.2.1.1),[Figure 9](https://arxiv.org/html/2607.11906#S6.F9.1.pic1.5.5.5.1.1.2.1.1),[§6\.1](https://arxiv.org/html/2607.11906#S6.SS1.p2.1),[Table 10](https://arxiv.org/html/2607.11906#S6.T10.1.9.8.2.1.1),[item 4](https://arxiv.org/html/2607.11906#S8.I1.i4.p1.1),[§8\.5](https://arxiv.org/html/2607.11906#S8.SS5.p1.4),[Table 14](https://arxiv.org/html/2607.11906#S8.T14.1.6.5.2.1.1),[Table 15](https://arxiv.org/html/2607.11906#S8.T15.4.8.3.2.1.1),[Table 16](https://arxiv.org/html/2607.11906#S8.T16.1.5.4.2.1.1),[Table 16](https://arxiv.org/html/2607.11906#S8.T16.1.9.8.2.1.1)\.
- Y\. Chen, C\. Lee, H\. Luo, and C\. Wei \(2019\)A new algorithm for non\-stationary contextual bandits: efficient, optimal and parameter\-free\.InConference on Learning Theory,pp\. 696–726\.Cited by:[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p4.1),[§3\.1](https://arxiv.org/html/2607.11906#S3.SS1.p3.1),[Table 6](https://arxiv.org/html/2607.11906#S4.T6.1.2.1.3.1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.8.7.2.1.1)\.
- W\. C\. Cheung, D\. Simchi\-Levi, and R\. Zhu \(2020\)Reinforcement learning for non\-stationary markov decision processes: the blessing of \(more\) optimism\.InInternational Conference on Machine Learning,Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.5.3.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p5.1),[item 5](https://arxiv.org/html/2607.11906#S10.I1.i5.p1.1),[Table 18](https://arxiv.org/html/2607.11906#S10.T18.5.7.5.2.1.1),[§11](https://arxiv.org/html/2607.11906#S11.p2.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p4.1),[§3\.1](https://arxiv.org/html/2607.11906#S3.SS1.p3.1),[§3\.3](https://arxiv.org/html/2607.11906#S3.SS3.p1.2),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.2.2.4.1.1),[Table 6](https://arxiv.org/html/2607.11906#S4.T6.1.3.2.3.1.1),[§8\.1](https://arxiv.org/html/2607.11906#S8.SS1.p1.1),[§8\.5](https://arxiv.org/html/2607.11906#S8.SS5.p1.4),[Table 12](https://arxiv.org/html/2607.11906#S8.T12.1.4.3.3.1.1),[Table 16](https://arxiv.org/html/2607.11906#S8.T16.1.10.9.2.1.1),[§9\.2](https://arxiv.org/html/2607.11906#S9.SS2.p1.2),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.5.4.2.1.1)\.
- M\. Chevalier\-Boisvert, D\. Bahdanau, S\. Lahlou, L\. Willems, C\. Saharia, T\. H\. Nguyen, and Y\. Bengio \(2019\)BabyAI: a platform to study the sample efficiency of grounded language learning\.InInternational Conference on Learning Representations,Cited by:[§8\.8](https://arxiv.org/html/2607.11906#S8.SS8.p1.1),[Table 13](https://arxiv.org/html/2607.11906#S8.T13.5.5.3.2.1.1)\.
- M\. Chevalier\-Boisvert, B\. Dai, M\. Towers, R\. de Lazcano, L\. Willems, S\. Lahlou, S\. Pal, P\. S\. Castro, and J\. Terry \(2023\)Minigrid & Miniworld: modular & customizable reinforcement learning environments for goal\-oriented tasks\.arXiv preprint arXiv:2306\.13831\.Cited by:[§8\.4](https://arxiv.org/html/2607.11906#S8.SS4.p2.1),[§8\.8](https://arxiv.org/html/2607.11906#S8.SS8.p1.1),[Table 13](https://arxiv.org/html/2607.11906#S8.T13.5.5.3.2.1.1)\.
- K\. Cobbe, C\. Hesse, J\. Hilton, and J\. Schulman \(2020\)Leveraging procedural generation to benchmark reinforcement learning\.InInternational Conference on Machine Learning,Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.10.8.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p3.1),[§8\.4](https://arxiv.org/html/2607.11906#S8.SS4.p2.1),[§8\.8](https://arxiv.org/html/2607.11906#S8.SS8.p2.1),[Table 13](https://arxiv.org/html/2607.11906#S8.T13.5.10.8.2.1.1),[Table 14](https://arxiv.org/html/2607.11906#S8.T14.1.8.7.2.1.1)\.
- J\. Cook, C\. Lu, E\. Hughes, J\. Z\. Leibo, and J\. Foerster \(2024\)Artificial generational intelligence: cultural accumulation in reinforcement learning\.InAdvances in Neural Information Processing Systems,Cited by:[§6\.10](https://arxiv.org/html/2607.11906#S6.SS10.p3.1)\.
- Y\. Dai, H\. Fu, J\. Lee, Y\. Liu, H\. Zhang, J\. Yang, C\. Finn, N\. Fazeli, and J\. Chai \(2026\)RoboMME: benchmarking and understanding memory for robotic generalist policies\.arXiv preprint arXiv:2603\.04639\.Cited by:[§8\.4](https://arxiv.org/html/2607.11906#S8.SS4.p3.1)\.
- Z\. Dai, F\. Tomasi, and S\. Ghiassian \(2024\)In\-context exploration\-exploitation for reinforcement learning\.InInternational Conference on Learning Representations,Cited by:[Table 3](https://arxiv.org/html/2607.11906#S2.T3.1.3.2.3.1.1),[Table 7](https://arxiv.org/html/2607.11906#S4.T7.1.3.2.3.1.1),[§6\.10](https://arxiv.org/html/2607.11906#S6.SS10.p3.1)\.
- S\. B\. David, I\. Zimerman, E\. Nachmani, and L\. Wolf \(2023\)Decision S4: efficient sequence\-based RL via state spaces layers\.InInternational Conference on Learning Representations,Cited by:[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p2.1),[§6\.3](https://arxiv.org/html/2607.11906#S6.SS3.p1.1),[Table 10](https://arxiv.org/html/2607.11906#S6.T10.1.4.3.2.1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.9.8.2.1.1)\.
- M\. Dennis, N\. Jaques, E\. Vinitsky, A\. M\. Bayen, S\. Russell, A\. Critch, and S\. Levine \(2020\)Emergent complexity and zero\-shot transfer via unsupervised environment design\.InAdvances in Neural Information Processing Systems,Cited by:[item 2](https://arxiv.org/html/2607.11906#S10.I1.i2.p1.1),[§7\.3](https://arxiv.org/html/2607.11906#S7.SS3.p1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.5.4.2.1.1)\.
- Z\. Ding, H\. Jiang, Y\. Wang, Z\. Sun, Y\. Zhang, X\. Niu, M\. Yang, W\. Zeng, X\. Xu, and Z\. Lu \(2025\)JAEGER: dual\-level humanoid whole\-body controller\.arXiv preprint arXiv:2505\.06584\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2505.06584)Cited by:[§8\.4](https://arxiv.org/html/2607.11906#S8.SS4.p3.1)\.
- O\. D\. Domingues, P\. Menard, M\. Pirotta, E\. Kaufmann, and M\. Valko \(2021\)A kernel\-based approach to non\-stationary reinforcement learning in metric spaces\.InInternational Conference on Artificial Intelligence and Statistics,pp\. 3538–3546\.Cited by:[§3\.3](https://arxiv.org/html/2607.11906#S3.SS3.p1.2),[Table 6](https://arxiv.org/html/2607.11906#S4.T6.1.3.2.3.1.1),[§9\.2](https://arxiv.org/html/2607.11906#S9.SS2.p1.2)\.
- J\. Dong, M\. Guo, E\. X\. Fang, Z\. Yang, and V\. Tarokh \(2026\)In\-context reinforcement learning from suboptimal historical data\.arXiv preprint arXiv:2601\.20116\.Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p3.1),[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[item 2](https://arxiv.org/html/2607.11906#S10.I1.i2.p1.1),[§11](https://arxiv.org/html/2607.11906#S11.p2.1),[Figure 8](https://arxiv.org/html/2607.11906#S5.F8.1.pic1.1.1.1.1.1.1.2.2),[Table 8](https://arxiv.org/html/2607.11906#S5.T8.1.2.1.4.1.1),[§6\.2](https://arxiv.org/html/2607.11906#S6.SS2.p2.1),[Table 10](https://arxiv.org/html/2607.11906#S6.T10.1.3.2.2.1.1),[§7\.4](https://arxiv.org/html/2607.11906#S7.SS4.p1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.4.3.2.1.1),[§8\.5](https://arxiv.org/html/2607.11906#S8.SS5.p1.4),[Table 14](https://arxiv.org/html/2607.11906#S8.T14.1.6.5.2.1.1),[Table 15](https://arxiv.org/html/2607.11906#S8.T15.4.6.1.2.1.1)\.
- R\. Dorfman, I\. Shenfeld, and A\. Tamar \(2020\)Offline meta learning of exploration\.arXiv preprint arXiv:2008\.02598\.Cited by:[§3\.2](https://arxiv.org/html/2607.11906#S3.SS2.p2.1),[§6\.1](https://arxiv.org/html/2607.11906#S6.SS1.p2.1)\.
- F\. Doshi\-Velez and G\. Konidaris \(2016\)Hidden parameter markov decision processes: a semiparametric regression approach for discovering latent task parametrizations\.InInternational Joint Conference on Artificial Intelligence,pp\. 1432–1440\.Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p2.1),[§3\.1](https://arxiv.org/html/2607.11906#S3.SS1.p1.14),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.2.2.4.1.1),[Table 7](https://arxiv.org/html/2607.11906#S4.T7.1.4.3.3.1.1),[§4](https://arxiv.org/html/2607.11906#S4.p1.1),[§4](https://arxiv.org/html/2607.11906#S4.p2.1),[§5\.1](https://arxiv.org/html/2607.11906#S5.SS1.p1.1),[§7\.3](https://arxiv.org/html/2607.11906#S7.SS3.p1.1),[Table 13](https://arxiv.org/html/2607.11906#S8.T13.5.4.2.2.1.1),[§9\.2](https://arxiv.org/html/2607.11906#S9.SS2.p1.2),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.6.5.2.1.1)\.
- Y\. Duan, J\. Schulman, X\. Chen, P\. L\. Bartlett, I\. Sutskever, and P\. Abbeel \(2016\)RL2: fast reinforcement learning via slow reinforcement learning\.arXiv preprint arXiv:1611\.02779\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.7.5.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p1.1),[§11](https://arxiv.org/html/2607.11906#S11.p1.1),[Figure 3](https://arxiv.org/html/2607.11906#S2.F3.1.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.5),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p1.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p3.1)\.
- S\. Earle and J\. Togelius \(2024\)Autoverse: an evolvable game language for learning robust embodied agents\.arXiv preprint arXiv:2407\.04221\.Cited by:[§7\.3](https://arxiv.org/html/2607.11906#S7.SS3.p1.1)\.
- A\. Elawady, G\. Chhablani, R\. Ramrakhya, K\. Yadav, D\. Batra, Z\. Kira, and A\. Szot \(2024\)ReLIC: a recipe for 64k steps of in\-context reinforcement learning for embodied AI\.arXiv preprint arXiv:2410\.02751\.Cited by:[item 1](https://arxiv.org/html/2607.11906#S10.I1.i1.p1.1),[Table 18](https://arxiv.org/html/2607.11906#S10.T18.5.8.6.2.1.1),[§11](https://arxiv.org/html/2607.11906#S11.p1.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p2.1),[Figure 6](https://arxiv.org/html/2607.11906#S4.F6.1.pic1.12.12.12.1.1.2.1.1),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.4.10.4.4.1.1),[Table 8](https://arxiv.org/html/2607.11906#S5.T8.1.3.2.4.1.1),[Figure 10](https://arxiv.org/html/2607.11906#S6.F10.1.pic1.4.4.4.1.1.2.1.1),[Figure 9](https://arxiv.org/html/2607.11906#S6.F9.1.pic1.3.3.3.1.1.2.1.1),[§6\.4](https://arxiv.org/html/2607.11906#S6.SS4.p1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.3.2.2.1.1),[Table 14](https://arxiv.org/html/2607.11906#S8.T14.1.7.6.2.1.1),[Table 15](https://arxiv.org/html/2607.11906#S8.T15.1.1.3.1.1)\.
- S\. Emmons, B\. Eysenbach, I\. Kostrikov, and S\. Levine \(2022\)RVS: what is essential for offline RL via supervised learning?\.InInternational Conference on Learning Representations,Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.8.6.2.1.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p2.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p3.2),[§6\.2](https://arxiv.org/html/2607.11906#S6.SS2.p1.1),[§7\.1](https://arxiv.org/html/2607.11906#S7.SS1.p1.1),[§7\.4](https://arxiv.org/html/2607.11906#S7.SS4.p1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.2.1.2.1.1)\.
- Y\. Fei, Z\. Yang, Z\. Wang, and Q\. Xie \(2020\)Dynamic regret of policy optimization in non\-stationary environments\.InAdvances in Neural Information Processing Systems,Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.5.3.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p5.1),[item 5](https://arxiv.org/html/2607.11906#S10.I1.i5.p1.1),[§11](https://arxiv.org/html/2607.11906#S11.p2.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p4.1),[§3\.3](https://arxiv.org/html/2607.11906#S3.SS3.p1.2),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.1.1.4.1.1),[Table 6](https://arxiv.org/html/2607.11906#S4.T6.1.3.2.3.1.1),[§4](https://arxiv.org/html/2607.11906#S4.p1.1),[§8\.1](https://arxiv.org/html/2607.11906#S8.SS1.p1.1),[§8\.5](https://arxiv.org/html/2607.11906#S8.SS5.p1.4),[Table 16](https://arxiv.org/html/2607.11906#S8.T16.1.10.9.2.1.1),[§9\.2](https://arxiv.org/html/2607.11906#S9.SS2.p1.2),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.5.4.2.1.1)\.
- S\. Feng, M\. Yin, and L\. F\. Yang \(2023\)Non\-stationary reinforcement learning under general function approximation\.arXiv preprint arXiv:2306\.00861\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.5.3.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p4.1),[Table 4](https://arxiv.org/html/2607.11906#S3.T4.1.5.4.3.1.1),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.2.2.4.1.1),[Table 6](https://arxiv.org/html/2607.11906#S4.T6.1.3.2.3.1.1),[§4](https://arxiv.org/html/2607.11906#S4.p1.1),[Table 14](https://arxiv.org/html/2607.11906#S8.T14.1.3.2.2.1.1),[Table 16](https://arxiv.org/html/2607.11906#S8.T16.1.10.9.2.1.1),[Table 16](https://arxiv.org/html/2607.11906#S8.T16.1.2.1.2.1.1),[§9\.2](https://arxiv.org/html/2607.11906#S9.SS2.p1.2),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.5.4.2.1.1)\.
- C\. Finn, P\. Abbeel, and S\. Levine \(2017\)Model\-agnostic meta\-learning for fast adaptation of deep networks\.InInternational Conference on Machine Learning,Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.7.5.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p1.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p3.1)\.
- K\. Frans and P\. Isola \(2023\)Powderworld: a platform for understanding generalization via rich task distributions\.InInternational Conference on Learning Representations,Cited by:[§7\.3](https://arxiv.org/html/2607.11906#S7.SS3.p1.1)\.
- J\. Fu, A\. Kumar, O\. Nachum, G\. Tucker, and S\. Levine \(2020\)D4RL: datasets for deep data\-driven reinforcement learning\.InAdvances in Neural Information Processing Systems Datasets and Benchmarks,Cited by:[Table 18](https://arxiv.org/html/2607.11906#S10.T18.5.13.11.2.1.1),[§3\.2](https://arxiv.org/html/2607.11906#S3.SS2.p2.1),[Figure 8](https://arxiv.org/html/2607.11906#S5.F8.1.pic1.1.1.1.1.1.1.2.2),[§7\.1](https://arxiv.org/html/2607.11906#S7.SS1.p1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.4.3.2.1.1),[§8\.3](https://arxiv.org/html/2607.11906#S8.SS3.p2.1)\.
- S\. Fujimoto, D\. Meger, and D\. Precup \(2019\)Off\-policy deep reinforcement learning without exploration\.InInternational Conference on Machine Learning,pp\. 2052–2062\.Cited by:[§4](https://arxiv.org/html/2607.11906#S4.p1.1)\.
- H\. Furuta, Y\. Matsuo, and S\. S\. Gu \(2022\)Generalized decision transformer for offline hindsight information matching\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p2.1),[§6\.2](https://arxiv.org/html/2607.11906#S6.SS2.p1.1),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.4.3.2.1.1)\.
- S\. Garcin, J\. Doran, S\. Guo, C\. G\. Lucas, and S\. V\. Albrecht \(2024\)DRED: zero\-shot transfer in reinforcement learning via data\-regularised environment design\.InInternational Conference on Machine Learning,Cited by:[§7\.3](https://arxiv.org/html/2607.11906#S7.SS3.p1.1)\.
- S\. Garg, D\. Tsipras, P\. Liang, and G\. Valiant \(2022\)What can transformers learn in\-context? a case study of simple function classes\.InAdvances in Neural Information Processing Systems,Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.9.7.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p1.1),[Table 18](https://arxiv.org/html/2607.11906#S10.T18.5.7.5.2.1.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p3.2),[§5\.2](https://arxiv.org/html/2607.11906#S5.SS2.p3.1),[§7\.1](https://arxiv.org/html/2607.11906#S7.SS1.p5.1),[§9\.1](https://arxiv.org/html/2607.11906#S9.SS1.p1.1),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.2.1.2.1.1)\.
- A\. Garivier and E\. Moulines \(2011\)On upper\-confidence bound policies for non\-stationary bandit problems\.InInternational Conference on Algorithmic Learning Theory,Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.6.4.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p3.1),[§1](https://arxiv.org/html/2607.11906#S1.p5.1),[item 3](https://arxiv.org/html/2607.11906#S10.I1.i3.p1.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p4.1),[§3\.1](https://arxiv.org/html/2607.11906#S3.SS1.p3.1),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.1.1.4.1.1),[Table 6](https://arxiv.org/html/2607.11906#S4.T6.1.3.2.3.1.1),[§4](https://arxiv.org/html/2607.11906#S4.p1.1),[§8\.1](https://arxiv.org/html/2607.11906#S8.SS1.p1.1),[Table 14](https://arxiv.org/html/2607.11906#S8.T14.1.2.1.2.1.1),[Table 16](https://arxiv.org/html/2607.11906#S8.T16.1.7.6.2.1.1),[§9\.2](https://arxiv.org/html/2607.11906#S9.SS2.p1.2),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.5.4.2.1.1)\.
- D\. Ghosh, J\. Rahme, A\. Kumar, A\. Zhang, R\. P\. Adams, and S\. Levine \(2021\)Why generalization in rl is difficult: epistemic pomdps and implicit partial observability\.arXiv preprint arXiv:2107\.06277\.Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p2.1),[§3\.1](https://arxiv.org/html/2607.11906#S3.SS1.p1.14),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.3.3.4.1.1),[Table 7](https://arxiv.org/html/2607.11906#S4.T7.1.4.3.3.1.1),[§4](https://arxiv.org/html/2607.11906#S4.p2.1),[§8\.8](https://arxiv.org/html/2607.11906#S8.SS8.p1.1),[Table 16](https://arxiv.org/html/2607.11906#S8.T16.1.3.2.2.1.1),[§9\.2](https://arxiv.org/html/2607.11906#S9.SS2.p1.2)\.
- A\. Goyal, A\. Friesen, A\. Banino, T\. Weber, N\. R\. Ke, A\. P\. Badia, A\. Guez, M\. Mirza, P\. C\. Humphreys, K\. Konyushova,et al\.\(2022\)Retrieval\-augmented reinforcement learning\.InInternational Conference on Machine Learning,pp\. 7740–7765\.Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[item 4](https://arxiv.org/html/2607.11906#S10.I1.i4.p1.1),[Table 18](https://arxiv.org/html/2607.11906#S10.T18.5.5.3.2.1.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p2.1),[Table 6](https://arxiv.org/html/2607.11906#S4.T6.1.4.3.3.1.1),[Figure 8](https://arxiv.org/html/2607.11906#S5.F8.1.pic1.3.3.3.3.1.1.2.2),[§5\.2](https://arxiv.org/html/2607.11906#S5.SS2.p4.1),[§5\.4](https://arxiv.org/html/2607.11906#S5.SS4.p1.1),[Figure 10](https://arxiv.org/html/2607.11906#S6.F10.1.pic1.5.5.5.1.1.2.1.1),[§6\.5](https://arxiv.org/html/2607.11906#S6.SS5.p1.1),[Table 10](https://arxiv.org/html/2607.11906#S6.T10.1.6.5.2.1.1),[Table 12](https://arxiv.org/html/2607.11906#S8.T12.1.7.6.3.1.1),[Table 14](https://arxiv.org/html/2607.11906#S8.T14.1.5.4.2.1.1),[Table 15](https://arxiv.org/html/2607.11906#S8.T15.4.7.2.2.1.1),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.7.6.2.1.1)\.
- J\. Grigsby, L\. Fan, and Y\. Zhu \(2024a\)AMAGO: scalable in\-context reinforcement learning for adaptive agents\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p1.1),[Figure 3](https://arxiv.org/html/2607.11906#S2.F3.1.pic1.9.9.9.1.1.5),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p2.1),[Table 4](https://arxiv.org/html/2607.11906#S3.T4.1.4.3.2.1.1),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.4.10.4.4.1.1),[Table 7](https://arxiv.org/html/2607.11906#S4.T7.1.5.4.3.1.1),[Table 8](https://arxiv.org/html/2607.11906#S5.T8.1.3.2.4.1.1),[§6\.4](https://arxiv.org/html/2607.11906#S6.SS4.p1.1),[Table 10](https://arxiv.org/html/2607.11906#S6.T10.1.5.4.2.1.1),[Table 14](https://arxiv.org/html/2607.11906#S8.T14.1.7.6.2.1.1)\.
- J\. Grigsby, J\. Sasek, S\. Parajuli, D\. Adebi, A\. Zhang, and Y\. Zhu \(2024b\)AMAGO\-2: breaking the multi\-task barrier in meta\-reinforcement learning with transformers\.InAdvances in Neural Information Processing Systems,Cited by:[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p2.1),[Table 4](https://arxiv.org/html/2607.11906#S3.T4.1.4.3.2.1.1),[Figure 6](https://arxiv.org/html/2607.11906#S4.F6.1.pic1.12.12.12.1.1.2.1.1),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.3.3.4.1.1),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.4.10.4.4.1.1),[Table 7](https://arxiv.org/html/2607.11906#S4.T7.1.5.4.3.1.1),[Table 8](https://arxiv.org/html/2607.11906#S5.T8.1.3.2.4.1.1),[Figure 10](https://arxiv.org/html/2607.11906#S6.F10.1.pic1.4.4.4.1.1.2.1.1),[Figure 9](https://arxiv.org/html/2607.11906#S6.F9.1.pic1.3.3.3.1.1.2.1.1),[§6\.4](https://arxiv.org/html/2607.11906#S6.SS4.p1.1),[Table 10](https://arxiv.org/html/2607.11906#S6.T10.1.5.4.2.1.1),[Table 12](https://arxiv.org/html/2607.11906#S8.T12.1.6.5.3.1.1),[Table 14](https://arxiv.org/html/2607.11906#S8.T14.1.7.6.2.1.1),[Table 15](https://arxiv.org/html/2607.11906#S8.T15.1.1.3.1.1),[Table 16](https://arxiv.org/html/2607.11906#S8.T16.1.6.5.2.1.1)\.
- A\. Gu and T\. Dao \(2023\)Mamba: linear\-time sequence modeling with selective state spaces\.arXiv preprint arXiv:2312\.00752\.Cited by:[Table 18](https://arxiv.org/html/2607.11906#S10.T18.5.8.6.2.1.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p2.1),[Figure 10](https://arxiv.org/html/2607.11906#S6.F10.1.pic1.3.3.3.1.1.2.1.1),[§6\.3](https://arxiv.org/html/2607.11906#S6.SS3.p1.1),[Table 10](https://arxiv.org/html/2607.11906#S6.T10.1.4.3.2.1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.9.8.2.1.1)\.
- C\. Gulcehre, Z\. Wang, A\. Novikov, T\. Paine, S\. Gomez, K\. Zolna, R\. Agarwal, J\. Merel, D\. Mankowitz, C\. Paduraru, G\. Dulac\-Arnold, J\. Li, M\. Norouzi, M\. Hoffman, O\. Nachum, G\. Tucker, N\. Heess, and N\. de Freitas \(2020\)RL unplugged: benchmarks for offline reinforcement learning\.InAdvances in Neural Information Processing Systems,Cited by:[Figure 8](https://arxiv.org/html/2607.11906#S5.F8.1.pic1.1.1.1.1.1.1.2.2),[§7\.1](https://arxiv.org/html/2607.11906#S7.SS1.p1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.4.3.2.1.1),[§8\.3](https://arxiv.org/html/2607.11906#S8.SS3.p2.1)\.
- K\. Guu, K\. Lee, Z\. Tung, P\. Pasupat, and M\. Chang \(2020\)Retrieval augmented language model pre\-training\.InInternational Conference on Machine Learning,pp\. 3929–3938\.Cited by:[§5\.1](https://arxiv.org/html/2607.11906#S5.SS1.p1.1),[§6\.5](https://arxiv.org/html/2607.11906#S6.SS5.p2.1)\.
- R\. Hadsell, D\. Rao, A\. A\. Rusu, and R\. Pascanu \(2020\)Embracing change: continual learning in deep neural networks\.Trends in Cognitive Sciences24\(12\),pp\. 1028–1040\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.4.2.2.1.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p5.1)\.
- D\. Hafner, T\. Lillicrap, I\. Fischer, R\. Villegas, D\. Ha, H\. Lee, and J\. Davidson \(2019\)Learning latent dynamics for planning from pixels\.InInternational Conference on Machine Learning,pp\. 2555–2565\.Cited by:[§6\.8](https://arxiv.org/html/2607.11906#S6.SS8.p1.1)\.
- A\. Hallak, D\. Di Castro, and S\. Mannor \(2015\)Contextual markov decision processes\.arXiv preprint arXiv:1502\.02259\.Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p2.1),[§3\.1](https://arxiv.org/html/2607.11906#S3.SS1.p1.14),[Figure 6](https://arxiv.org/html/2607.11906#S4.F6.1.pic1.11.11.11.1.1.2.1.1),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.2.2.4.1.1),[Table 7](https://arxiv.org/html/2607.11906#S4.T7.1.4.3.3.1.1),[§4](https://arxiv.org/html/2607.11906#S4.p1.1),[§4](https://arxiv.org/html/2607.11906#S4.p2.1),[§5\.1](https://arxiv.org/html/2607.11906#S5.SS1.p1.1),[§7\.3](https://arxiv.org/html/2607.11906#S7.SS3.p1.1),[Table 13](https://arxiv.org/html/2607.11906#S8.T13.5.4.2.2.1.1),[Table 16](https://arxiv.org/html/2607.11906#S8.T16.1.3.2.2.1.1),[§9\.2](https://arxiv.org/html/2607.11906#S9.SS2.p1.2),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.6.5.2.1.1)\.
- N\. Heess, J\. J\. Hunt, T\. P\. Lillicrap, and D\. Silver \(2015\)Memory\-based control with recurrent neural networks\.arXiv preprint arXiv:1512\.04455\.Cited by:[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p2.1),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.3.3.4.1.1),[Table 7](https://arxiv.org/html/2607.11906#S4.T7.1.3.2.3.1.1)\.
- S\. Huang, J\. Hu, H\. Chen, L\. Sun, and B\. Yang \(2024a\)In\-context decision transformer: reinforcement learning via hierarchical chain\-of\-thought\.arXiv preprint arXiv:2405\.20692\.Cited by:[§5\.4](https://arxiv.org/html/2607.11906#S5.SS4.p1.1),[Figure 10](https://arxiv.org/html/2607.11906#S6.F10.1.pic1.2.2.2.1.1.2.1.1),[§6\.2](https://arxiv.org/html/2607.11906#S6.SS2.p1.1)\.
- S\. Huang, J\. Hu, Z\. Yang, L\. Yang, T\. Luo, H\. Chen, L\. Sun, and B\. Yang \(2024b\)Decision mamba: reinforcement learning via hybrid selective sequence modeling\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 72688–72709\.Cited by:[§6\.3](https://arxiv.org/html/2607.11906#S6.SS3.p1.1)\.
- P\. Humphreys, A\. Guez, O\. Tieleman, L\. Sifre, T\. Weber, and T\. Lillicrap \(2022\)Large\-scale retrieval for reinforcement learning\.Advances in Neural Information Processing Systems35,pp\. 20092–20104\.Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[item 4](https://arxiv.org/html/2607.11906#S10.I1.i4.p1.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p2.1),[Table 6](https://arxiv.org/html/2607.11906#S4.T6.1.4.3.3.1.1),[Figure 8](https://arxiv.org/html/2607.11906#S5.F8.1.pic1.3.3.3.3.1.1.2.2),[§5\.2](https://arxiv.org/html/2607.11906#S5.SS2.p4.1),[§6\.5](https://arxiv.org/html/2607.11906#S6.SS5.p1.1),[Table 10](https://arxiv.org/html/2607.11906#S6.T10.1.6.5.2.1.1),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.7.6.2.1.1)\.
- J\. Humplik, A\. Galashov, L\. Hasenclever, P\. A\. Ortega, Y\. W\. Teh, and N\. Heess \(2019\)Meta reinforcement learning as task inference\.arXiv preprint arXiv:1905\.06424\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.7.5.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p2.1),[Table 7](https://arxiv.org/html/2607.11906#S4.T7.1.4.3.3.1.1),[§5\.1](https://arxiv.org/html/2607.11906#S5.SS1.p1.1),[§9\.2](https://arxiv.org/html/2607.11906#S9.SS2.p1.2),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.6.5.2.1.1)\.
- G\. Izacard, P\. Lewis, M\. Lomeli, L\. Hosseini, F\. Petroni, T\. Schick, J\. Dwivedi\-Yu, A\. Joulin, S\. Riedel, and E\. Grave \(2022\)Few\-shot learning with retrieval augmented language models\.arXiv preprint arXiv:2208\.03299\.Cited by:[§5\.2](https://arxiv.org/html/2607.11906#S5.SS2.p4.1),[§6\.5](https://arxiv.org/html/2607.11906#S6.SS5.p2.1)\.
- M\. Janner, Q\. Li, and S\. Levine \(2021\)Offline reinforcement learning as one big sequence modeling problem\.InAdvances in Neural Information Processing Systems,Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.8.6.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p1.1),[Figure 3](https://arxiv.org/html/2607.11906#S2.F3.1.pic1.7.7.7.1.1.5),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p2.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p3.2),[Figure 10](https://arxiv.org/html/2607.11906#S6.F10.1.pic1.2.2.2.1.1.2.1.1),[Figure 9](https://arxiv.org/html/2607.11906#S6.F9.1.pic1.1.1.1.1.1.2.1.1),[§6\.2](https://arxiv.org/html/2607.11906#S6.SS2.p1.1),[§7\.1](https://arxiv.org/html/2607.11906#S7.SS1.p1.1),[§7\.4](https://arxiv.org/html/2607.11906#S7.SS4.p1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.2.1.2.1.1)\.
- J\. Ji, B\. Zhang, J\. Zhou, X\. Pan, W\. Huang, R\. Sun, Y\. Geng, Y\. Zhong, J\. Dai, and Y\. Yang \(2023\)Safety gymnasium: a unified safe reinforcement learning benchmark\.InAdvances in Neural Information Processing Systems,Cited by:[§10\.2](https://arxiv.org/html/2607.11906#S10.SS2.p2.1)\.
- M\. Jiang, M\. Dennis, J\. Parker\-Holder, J\. N\. Foerster, E\. Grefenstette, and T\. Rocktaschel \(2021a\)Replay\-guided adversarial environment design\.InAdvances in Neural Information Processing Systems,Cited by:[§7\.3](https://arxiv.org/html/2607.11906#S7.SS3.p1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.5.4.2.1.1)\.
- M\. Jiang, E\. Grefenstette, and T\. Rocktaschel \(2021b\)Prioritized level replay\.InInternational Conference on Machine Learning,Cited by:[item 2](https://arxiv.org/html/2607.11906#S10.I1.i2.p1.1),[§7\.3](https://arxiv.org/html/2607.11906#S7.SS3.p1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.5.4.2.1.1)\.
- L\. P\. Kaelbling, M\. L\. Littman, and A\. R\. Cassandra \(1998\)Planning and acting in partially observable stochastic domains\.Artificial Intelligence101\(1–2\),pp\. 99–134\.Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p2.1),[§3\.1](https://arxiv.org/html/2607.11906#S3.SS1.p1.14),[Figure 6](https://arxiv.org/html/2607.11906#S4.F6.1.pic1.10.10.10.1.1.2.1.1),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.3.3.4.1.1),[Table 7](https://arxiv.org/html/2607.11906#S4.T7.1.3.2.3.1.1),[§4](https://arxiv.org/html/2607.11906#S4.p2.1),[§5\.1](https://arxiv.org/html/2607.11906#S5.SS1.p1.1),[Table 16](https://arxiv.org/html/2607.11906#S8.T16.1.3.2.2.1.1),[§9\.2](https://arxiv.org/html/2607.11906#S9.SS2.p1.2),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.6.5.2.1.1)\.
- P\. Kamienny, M\. Pirotta, A\. Lazaric, T\. Lavril, N\. Usunier, and L\. Denoyer \(2020\)Learning adaptive exploration strategies in dynamic environments through informed policy regularization\.arXiv preprint arXiv:2005\.02934\.Cited by:[§6\.1](https://arxiv.org/html/2607.11906#S6.SS1.p2.1)\.
- T\. Katsch \(2023\)GateLoop: fully data\-controlled linear recurrence for sequence modeling\.arXiv preprint arXiv:2311\.01927\.Cited by:[§6\.3](https://arxiv.org/html/2607.11906#S6.SS3.p1.1),[§9\.1](https://arxiv.org/html/2607.11906#S9.SS1.p1.1)\.
- N\. S\. Keplinger, B\. Luo, I\. Bektas, Y\. Zhang, K\. H\. Wray, A\. Laszka, A\. Dubey, and A\. Mukhopadhyay \(2025\)NS\-Gym: open\-source simulation environments and benchmarks for non\-stationary markov decision processes\.arXiv preprint arXiv:2501\.09646\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.5.3.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p3.1),[§7\.3](https://arxiv.org/html/2607.11906#S7.SS3.p1.1),[§8\.3](https://arxiv.org/html/2607.11906#S8.SS3.p2.1),[§8\.8](https://arxiv.org/html/2607.11906#S8.SS8.p3.1),[Table 13](https://arxiv.org/html/2607.11906#S8.T13.5.15.13.2.1.1)\.
- U\. Khandelwal, O\. Levy, D\. Jurafsky, L\. Zettlemoyer, and M\. Lewis \(2019\)Generalization through memorization: nearest neighbor language models\.arXiv preprint arXiv:1911\.00172\.Cited by:[Figure 8](https://arxiv.org/html/2607.11906#S5.F8.1.pic1.2.2.2.2.1.1.2.2),[§5\.1](https://arxiv.org/html/2607.11906#S5.SS1.p1.1),[§6\.5](https://arxiv.org/html/2607.11906#S6.SS5.p2.1)\.
- K\. Khetarpal, M\. Riemer, I\. Rish, and D\. Precup \(2022\)Towards continual reinforcement learning: a review and perspectives\.Journal of Artificial Intelligence Research75,pp\. 1401–1476\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.4.2.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p3.1),[Table 18](https://arxiv.org/html/2607.11906#S10.T18.5.6.4.2.1.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p4.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p5.1),[§3\.2](https://arxiv.org/html/2607.11906#S3.SS2.p1.5),[Table 6](https://arxiv.org/html/2607.11906#S4.T6.1.6.5.3.1.1),[Figure 8](https://arxiv.org/html/2607.11906#S5.F8.1.pic1.6.6.6.6.1.1.2.2),[Table 8](https://arxiv.org/html/2607.11906#S5.T8.1.8.7.4.1.1),[Table 12](https://arxiv.org/html/2607.11906#S8.T12.1.8.7.3.1.1),[Table 15](https://arxiv.org/html/2607.11906#S8.T15.3.3.2.2.2)\.
- R\. Kidambi, A\. Rajeswaran, P\. Netrapalli, and T\. Joachims \(2020\)MOReL: model\-based offline reinforcement learning\.InAdvances in Neural Information Processing Systems,Cited by:[§4](https://arxiv.org/html/2607.11906#S4.p1.1)\.
- T\. W\. Killian, S\. Daulton, G\. Konidaris, and F\. Doshi\-Velez \(2017\)Robust and efficient transfer learning with hidden\-parameter markov decision processes\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p2.1),[§3\.1](https://arxiv.org/html/2607.11906#S3.SS1.p1.14),[Figure 6](https://arxiv.org/html/2607.11906#S4.F6.1.pic1.11.11.11.1.1.2.1.1),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.2.2.4.1.1),[Table 7](https://arxiv.org/html/2607.11906#S4.T7.1.4.3.3.1.1),[§4](https://arxiv.org/html/2607.11906#S4.p1.1),[§4](https://arxiv.org/html/2607.11906#S4.p2.1),[§5\.1](https://arxiv.org/html/2607.11906#S5.SS1.p1.1),[§7\.3](https://arxiv.org/html/2607.11906#S7.SS3.p1.1),[Table 13](https://arxiv.org/html/2607.11906#S8.T13.5.4.2.2.1.1),[§9\.2](https://arxiv.org/html/2607.11906#S9.SS2.p1.2),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.6.5.2.1.1)\.
- B\. Kim and A\. Tewari \(2020\)Randomized exploration for non\-stationary stochastic linear bandits\.InConference on Uncertainty in Artificial Intelligence,pp\. 71–80\.Cited by:[Table 16](https://arxiv.org/html/2607.11906#S8.T16.1.10.9.2.1.1)\.
- R\. Kirk, A\. Zhang, E\. Grefenstette, and T\. Rocktäschel \(2023\)A survey of zero\-shot generalisation in deep reinforcement learning\.Journal of Artificial Intelligence Research76,pp\. 201–264\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.10.8.2.1.1),[Table 18](https://arxiv.org/html/2607.11906#S10.T18.5.3.1.3.1.1),[§4](https://arxiv.org/html/2607.11906#S4.p2.1),[§8\.1](https://arxiv.org/html/2607.11906#S8.SS1.p1.1),[§8\.8](https://arxiv.org/html/2607.11906#S8.SS8.p1.1)\.
- J\. Kirkpatrick, R\. Pascanu, N\. Rabinowitz, J\. Veness, G\. Desjardins, A\. A\. Rusu, K\. Milan, J\. Quan, T\. Ramalho, A\. Grabska\-Barwinska,et al\.\(2017\)Overcoming catastrophic forgetting in neural networks\.Proceedings of the National Academy of Sciences114\(13\),pp\. 3521–3526\.Cited by:[Table 18](https://arxiv.org/html/2607.11906#S10.T18.5.6.4.2.1.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p5.1),[Figure 8](https://arxiv.org/html/2607.11906#S5.F8.1.pic1.6.6.6.6.1.1.2.2)\.
- L\. Kirsch, J\. Harrison, J\. Sohl\-Dickstein, and L\. Metz \(2022\)General\-purpose in\-context learning by meta\-learning transformers\.InNeurIPS Workshop on Meta\-Learning,Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.9.7.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p3.2),[§5\.2](https://arxiv.org/html/2607.11906#S5.SS2.p3.1),[§6\.1](https://arxiv.org/html/2607.11906#S6.SS1.p1.1),[§7\.1](https://arxiv.org/html/2607.11906#S7.SS1.p5.1),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.2.1.2.1.1)\.
- I\. Kostrikov, A\. Nair, and S\. Levine \(2022\)Offline reinforcement learning with implicit Q\-learning\.InInternational Conference on Learning Representations,Cited by:[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p3.2),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.4.9.3.4.1.1),[§6\.2](https://arxiv.org/html/2607.11906#S6.SS2.p2.1),[§6\.6](https://arxiv.org/html/2607.11906#S6.SS6.p1.1),[§7\.4](https://arxiv.org/html/2607.11906#S7.SS4.p1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.4.3.2.1.1)\.
- A\. Krishnamurthy, K\. Harris, D\. J\. Foster, C\. Zhang, and A\. Slivkins \(2024\)Can large language models explore in\-context?\.InAdvances in Neural Information Processing Systems,Cited by:[Table 18](https://arxiv.org/html/2607.11906#S10.T18.5.14.12.2.1.1),[Table 3](https://arxiv.org/html/2607.11906#S2.T3.1.3.2.3.1.1),[Table 7](https://arxiv.org/html/2607.11906#S4.T7.1.3.2.3.1.1),[§6\.10](https://arxiv.org/html/2607.11906#S6.SS10.p3.1)\.
- A\. Kumar, A\. Zhou, G\. Tucker, and S\. Levine \(2020\)Conservative Q\-learning for offline reinforcement learning\.InAdvances in Neural Information Processing Systems,Cited by:[Table 18](https://arxiv.org/html/2607.11906#S10.T18.5.13.11.2.1.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p3.2),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.4.9.3.4.1.1),[§6\.2](https://arxiv.org/html/2607.11906#S6.SS2.p2.1),[§6\.6](https://arxiv.org/html/2607.11906#S6.SS6.p1.1),[§7\.4](https://arxiv.org/html/2607.11906#S7.SS4.p1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.4.3.2.1.1)\.
- A\. Kumar, V\. Zhuang, R\. Agarwal, Y\. Su, J\. D\. Co\-Reyes, A\. Singh, K\. Baumli, S\. Iqbal, C\. Bishop, R\. Roelofs, L\. M\. Zhang, K\. McKinney, D\. Shrivastava, C\. Paduraru, G\. Tucker, D\. Precup, F\. Behbahani, and A\. Faust \(2025\)Training language models to self\-correct via reinforcement learning\.InInternational Conference on Learning Representations,Cited by:[§6\.10](https://arxiv.org/html/2607.11906#S6.SS10.p3.1)\.
- H\. Küttler, N\. Nardelli, A\. H\. Miller, R\. Raileanu, M\. Selvatici, E\. Grefenstette, and T\. Rocktäschel \(2020\)The NetHack learning environment\.InAdvances in Neural Information Processing Systems,Cited by:[§8\.8](https://arxiv.org/html/2607.11906#S8.SS8.p2.1),[Table 13](https://arxiv.org/html/2607.11906#S8.T13.5.12.10.2.1.1)\.
- M\. Kwon, A\. Moeini, S\. Zhang, and L\. Feng \(2026\)Latent Q\-barrier shielding for safe in\-context reinforcement learning\.arXiv preprint arXiv:2605\.25267\.Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[§10\.2](https://arxiv.org/html/2607.11906#S10.SS2.p1.1),[§10\.2](https://arxiv.org/html/2607.11906#S10.SS2.p2.1),[§10\.3](https://arxiv.org/html/2607.11906#S10.SS3.p1.1),[Table 18](https://arxiv.org/html/2607.11906#S10.T18.5.12.10.2.1.1)\.
- R\. Laroche, P\. Trichelair, and R\. Tachet des Combes \(2019\)Safe policy improvement with baseline bootstrapping\.InInternational Conference on Machine Learning,Cited by:[§10\.2](https://arxiv.org/html/2607.11906#S10.SS2.p2.1)\.
- M\. Laskin, L\. Wang, J\. Oh, E\. Parisotto, S\. Spencer, R\. Steigerwald, D\. Strouse, S\. S\. Hansen, A\. Filos, E\. Brooks, M\. Gazeau, H\. Sahni, S\. Singh, and V\. Mnih \(2023\)In\-context reinforcement learning with algorithm distillation\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p1.1),[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[item 2](https://arxiv.org/html/2607.11906#S10.I1.i2.p1.1),[§11](https://arxiv.org/html/2607.11906#S11.p1.1),[Figure 3](https://arxiv.org/html/2607.11906#S2.F3.1.pic1.8.8.8.1.1.5),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p2.1),[Table 7](https://arxiv.org/html/2607.11906#S4.T7.1.3.2.3.1.1),[§5\.4](https://arxiv.org/html/2607.11906#S5.SS4.p1.1),[Table 8](https://arxiv.org/html/2607.11906#S5.T8.1.2.1.4.1.1),[Figure 10](https://arxiv.org/html/2607.11906#S6.F10.1.pic1.1.1.1.1.1.2.1.1),[Figure 9](https://arxiv.org/html/2607.11906#S6.F9.1.pic1.2.2.2.1.1.2.1.1),[§6\.1](https://arxiv.org/html/2607.11906#S6.SS1.p1.1),[Table 10](https://arxiv.org/html/2607.11906#S6.T10.1.2.1.2.1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.3.2.2.1.1),[Table 15](https://arxiv.org/html/2607.11906#S8.T15.4.6.1.2.1.1)\.
- E\. Lecarpentier and E\. Rachelson \(2019\)Non\-stationary markov decision processes, a worst\-case approach using model\-based reinforcement learning\.Advances in Neural Information Processing Systems32\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.5.3.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p3.1),[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p4.1),[Figure 6](https://arxiv.org/html/2607.11906#S4.F6.1.pic1.2.2.2.1.1.2.1.1),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.2.2.4.1.1),[§4](https://arxiv.org/html/2607.11906#S4.p1.1),[item 1](https://arxiv.org/html/2607.11906#S8.I1.i1.p1.1),[§8\.1](https://arxiv.org/html/2607.11906#S8.SS1.p1.1),[Table 13](https://arxiv.org/html/2607.11906#S8.T13.5.15.13.2.1.1),[Table 14](https://arxiv.org/html/2607.11906#S8.T14.1.3.2.2.1.1),[Table 16](https://arxiv.org/html/2607.11906#S8.T16.1.2.1.2.1.1)\.
- J\. N\. Lee, A\. Xie, A\. Pacchiano, Y\. Chandak, C\. Finn, O\. Nachum, and E\. Brunskill \(2023\)Supervised pretraining can learn in\-context reinforcement learning\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p1.1),[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[§11](https://arxiv.org/html/2607.11906#S11.p1.1),[Figure 3](https://arxiv.org/html/2607.11906#S2.F3.1.pic1.8.8.8.1.1.5),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p2.1),[Table 4](https://arxiv.org/html/2607.11906#S3.T4.1.2.1.2.1.1),[Figure 6](https://arxiv.org/html/2607.11906#S4.F6.1.pic1.9.9.9.1.1.2.1.1),[Table 7](https://arxiv.org/html/2607.11906#S4.T7.1.3.2.3.1.1),[§5\.4](https://arxiv.org/html/2607.11906#S5.SS4.p1.1),[§6\.2](https://arxiv.org/html/2607.11906#S6.SS2.p1.1),[Table 10](https://arxiv.org/html/2607.11906#S6.T10.1.3.2.2.1.1),[Table 16](https://arxiv.org/html/2607.11906#S8.T16.1.3.2.2.1.1),[§9\.1](https://arxiv.org/html/2607.11906#S9.SS1.p1.1),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.3.2.2.1.1)\.
- K\. Lee, O\. Nachum, M\. Yang, L\. Lee, D\. Freeman, S\. Guadarrama, I\. Fischer, W\. Xu, E\. Jang, H\. Michalewski, and I\. Mordatch \(2022\)Multi\-game decision transformers\.InAdvances in Neural Information Processing Systems,Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.13.11.2.1.1),[Figure 3](https://arxiv.org/html/2607.11906#S2.F3.1.pic1.7.7.7.1.1.5),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p3.2),[§6\.2](https://arxiv.org/html/2607.11906#S6.SS2.p1.1)\.
- T\. Lesort, V\. Lomonaco, A\. Stoian, D\. Maltoni, D\. Filliat, and N\. Diaz\-Rodriguez \(2020\)Continual learning for robotics: definition, framework, learning strategies, opportunities and challenges\.Information Fusion58,pp\. 52–68\.Cited by:[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p5.1)\.
- S\. Levine, A\. Kumar, G\. Tucker, and J\. Fu \(2020\)Offline reinforcement learning: tutorial, review, and perspectives on open problems\.arXiv preprint arXiv:2005\.01643\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.8.6.2.1.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p3.2),[Table 3](https://arxiv.org/html/2607.11906#S2.T3.1.4.3.3.1.1),[§3\.2](https://arxiv.org/html/2607.11906#S3.SS2.p2.1),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.4.9.3.4.1.1),[Figure 8](https://arxiv.org/html/2607.11906#S5.F8.1.pic1.1.1.1.1.1.1.2.2),[§6\.2](https://arxiv.org/html/2607.11906#S6.SS2.p2.1),[§7\.1](https://arxiv.org/html/2607.11906#S7.SS1.p1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.4.3.2.1.1)\.
- P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Kuttler, M\. Lewis, W\. Yih, T\. Rocktaschel,et al\.\(2020\)Retrieval\-augmented generation for knowledge\-intensive nlp tasks\.Advances in Neural Information Processing Systems33,pp\. 9459–9474\.Cited by:[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p3.3),[Figure 8](https://arxiv.org/html/2607.11906#S5.F8.1.pic1.2.2.2.2.1.1.2.2),[§5\.1](https://arxiv.org/html/2607.11906#S5.SS1.p1.1),[§6\.5](https://arxiv.org/html/2607.11906#S6.SS5.p2.1)\.
- S\. Li, K\. Wu, C\. Zhang, and Y\. Zhu \(2024a\)I\-PHYRE: interactive physical reasoning\.InInternational Conference on Learning Representations,Cited by:[§8\.3](https://arxiv.org/html/2607.11906#S8.SS3.p2.1),[§8\.8](https://arxiv.org/html/2607.11906#S8.SS8.p2.1),[Table 13](https://arxiv.org/html/2607.11906#S8.T13.5.9.7.2.1.1)\.
- T\. Li, G\. Zhang, Q\. D\. Do, X\. Yue, and W\. Chen \(2024b\)Long\-context llms struggle with long in\-context learning\.arXiv preprint arXiv:2404\.02060\.Cited by:[item 1](https://arxiv.org/html/2607.11906#S10.I1.i1.p1.1),[§5\.2](https://arxiv.org/html/2607.11906#S5.SS2.p4.1),[Table 15](https://arxiv.org/html/2607.11906#S8.T15.1.1.3.1.1),[§9\.1](https://arxiv.org/html/2607.11906#S9.SS1.p1.1)\.
- W\. Li, H\. Luo, Z\. Lin, C\. Zhang, Z\. Lu, and D\. Ye \(2023\)A survey on transformers in reinforcement learning\.Transactions on Machine Learning Research\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.8.6.2.1.1)\.
- Y\. Li and N\. Li \(2019\)Online learning for markov decision processes in nonstationary environments: a dynamic regret analysis\.InAmerican Control Conference,pp\. 1232–1237\.Cited by:[§3\.3](https://arxiv.org/html/2607.11906#S3.SS3.p1.2),[§8\.1](https://arxiv.org/html/2607.11906#S8.SS1.p1.1),[§9\.2](https://arxiv.org/html/2607.11906#S9.SS2.p1.2)\.
- H\. Liang and L\. Lai \(2026\)Transformers provably implement in\-context reinforcement learning with policy improvement\.arXiv preprint arXiv:2605\.05755\.Cited by:[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p3.2),[§6\.2](https://arxiv.org/html/2607.11906#S6.SS2.p1.1),[§9\.1](https://arxiv.org/html/2607.11906#S9.SS1.p1.1),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.4.3.2.1.1)\.
- L\. Lin, Y\. Bai, and S\. Mei \(2024\)Transformers as decision makers: provable in\-context reinforcement learning via supervised pretraining\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p2.1),[Table 4](https://arxiv.org/html/2607.11906#S3.T4.1.2.1.2.1.1),[§5\.4](https://arxiv.org/html/2607.11906#S5.SS4.p1.1),[§6\.2](https://arxiv.org/html/2607.11906#S6.SS2.p1.1),[Table 10](https://arxiv.org/html/2607.11906#S6.T10.1.3.2.2.1.1),[§9\.1](https://arxiv.org/html/2607.11906#S9.SS1.p1.1),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.3.2.2.1.1)\.
- B\. Liu, Y\. Zhu, C\. Gao, Y\. Feng, Q\. Liu, Y\. Zhu, and P\. Stone \(2023\)LIBERO: benchmarking knowledge transfer for lifelong robot learning\.arXiv preprint arXiv:2306\.03310\.Cited by:[§8\.4](https://arxiv.org/html/2607.11906#S8.SS4.p3.1)\.
- E\. Z\. Liu, A\. Raghunathan, P\. Liang, and C\. Finn \(2021\)Decoupling exploration and exploitation for meta\-reinforcement learning without sacrifices\.InInternational Conference on Machine Learning,pp\. 6925–6935\.Cited by:[§6\.1](https://arxiv.org/html/2607.11906#S6.SS1.p2.1)\.
- H\. Liu and P\. Abbeel \(2023\)Emergent agentic transformer from chain of hindsight experience\.InInternational Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p2.1),[§6\.2](https://arxiv.org/html/2607.11906#S6.SS2.p1.1),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.4.3.2.1.1)\.
- J\. Liu, F\. Liu, Z\. Sun, J\. Hao, H\. Li, B\. Wang, D\. Dong, C\. Chen, and Z\. Wang \(2026\)Scalable in\-context Q\-learning\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p2.1),[§5\.4](https://arxiv.org/html/2607.11906#S5.SS4.p1.1),[Figure 10](https://arxiv.org/html/2607.11906#S6.F10.1.pic1.7.7.7.1.1.2.1.1),[§6\.6](https://arxiv.org/html/2607.11906#S6.SS6.p1.1),[Table 10](https://arxiv.org/html/2607.11906#S6.T10.1.7.6.2.1.1),[§7\.4](https://arxiv.org/html/2607.11906#S7.SS4.p1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.7.6.2.1.1)\.
- V\. Lomonaco, K\. Desai, E\. Culurciello, and D\. Maltoni \(2020\)Continual reinforcement learning in 3d non\-stationary environments\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops,pp\. 999–1008\.Cited by:[§8\.4](https://arxiv.org/html/2607.11906#S8.SS4.p2.1),[§8\.8](https://arxiv.org/html/2607.11906#S8.SS8.p3.1),[Table 13](https://arxiv.org/html/2607.11906#S8.T13.5.16.14.2.1.1)\.
- D\. Lopez\-Paz and M\. Ranzato \(2017\)Gradient episodic memory for continual learning\.Advances in Neural Information Processing Systems30\.Cited by:[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p5.1),[Figure 8](https://arxiv.org/html/2607.11906#S5.F8.1.pic1.6.6.6.6.1.1.2.2)\.
- C\. Lu, Y\. Schroecker, A\. Gu, E\. Parisotto, J\. Foerster, S\. Singh, and F\. Behbahani \(2023\)Structured state space models for in\-context reinforcement learning\.InAdvances in Neural Information Processing Systems,Cited by:[item 1](https://arxiv.org/html/2607.11906#S10.I1.i1.p1.1),[Table 18](https://arxiv.org/html/2607.11906#S10.T18.5.8.6.2.1.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p2.1),[Table 7](https://arxiv.org/html/2607.11906#S4.T7.1.5.4.3.1.1),[Table 8](https://arxiv.org/html/2607.11906#S5.T8.1.3.2.4.1.1),[Figure 10](https://arxiv.org/html/2607.11906#S6.F10.1.pic1.3.3.3.1.1.2.1.1),[§6\.3](https://arxiv.org/html/2607.11906#S6.SS3.p1.1),[Table 10](https://arxiv.org/html/2607.11906#S6.T10.1.4.3.2.1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.9.8.2.1.1),[Table 12](https://arxiv.org/html/2607.11906#S8.T12.1.6.5.3.1.1),[Table 14](https://arxiv.org/html/2607.11906#S8.T14.1.7.6.2.1.1)\.
- B\. Luo, Y\. Zhang, A\. Dubey, and A\. Mukhopadhyay \(2024\)Act as you learn: adaptive decision\-making in non\-stationary markov decision processes\.InInternational Joint Conference on Autonomous Agents and Multiagent Systems,pp\. 1301–1309\.Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p4.1),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.2.2.4.1.1),[Table 13](https://arxiv.org/html/2607.11906#S8.T13.5.15.13.2.1.1)\.
- H\. Luo, J\. Jiang, and Z\. Lu \(2023\)Model\-based decentralized policy optimization\.arXiv preprint arXiv:2302\.08139\.Cited by:[Table 3](https://arxiv.org/html/2607.11906#S2.T3.1.4.3.3.1.1),[§8\.4](https://arxiv.org/html/2607.11906#S8.SS4.p3.1)\.
- A\. Mandlekar, D\. Xu, J\. Wong, S\. Nasiriany, C\. Wang, R\. Kulkarni, L\. Fei\-Fei, S\. Savarese, Y\. Zhu, and R\. Martín\-Martín \(2021\)What matters in learning from offline human demonstrations for robot manipulation\.InConference on Robot Learning,Cited by:[§8\.4](https://arxiv.org/html/2607.11906#S8.SS4.p3.1)\.
- W\. Mao, K\. Zhang, R\. Zhu, D\. Simchi\-Levi, and T\. Basar \(2021\)Near\-optimal model\-free reinforcement learning in non\-stationary episodic mdps\.InInternational Conference on Machine Learning,pp\. 7447–7458\.Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[§1](https://arxiv.org/html/2607.11906#S1.p5.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p4.1),[§3\.3](https://arxiv.org/html/2607.11906#S3.SS3.p1.2),[Table 6](https://arxiv.org/html/2607.11906#S4.T6.1.3.2.3.1.1),[§9\.2](https://arxiv.org/html/2607.11906#S9.SS2.p1.2)\.
- F\. Martinez\-Lopez, T\. Li, Y\. Lu, and J\. Chen \(2025\)In\-context reinforcement learning via communicative world models\.arXiv preprint arXiv:2508\.06659\.Cited by:[Figure 8](https://arxiv.org/html/2607.11906#S5.F8.1.pic1.4.4.4.4.1.1.2.2),[§5\.4](https://arxiv.org/html/2607.11906#S5.SS4.p1.1),[Table 8](https://arxiv.org/html/2607.11906#S5.T8.1.6.5.4.1.1),[Figure 10](https://arxiv.org/html/2607.11906#S6.F10.1.pic1.6.6.6.1.1.2.1.1),[Figure 9](https://arxiv.org/html/2607.11906#S6.F9.1.pic1.5.5.5.1.1.2.1.1),[§6\.9](https://arxiv.org/html/2607.11906#S6.SS9.p1.1),[Table 10](https://arxiv.org/html/2607.11906#S6.T10.1.9.8.2.1.1),[Table 9](https://arxiv.org/html/2607.11906#S6.T9.1.5.4.2.1.1)\.
- M\. Matthews, M\. Beukman, B\. Ellis, M\. Samvelyan, M\. T\. Jackson, S\. Coward, and J\. N\. Foerster \(2024a\)Craftax: a lightning\-fast benchmark for open\-ended reinforcement learning\.InInternational Conference on Machine Learning,Cited by:[§8\.4](https://arxiv.org/html/2607.11906#S8.SS4.p2.1),[§8\.8](https://arxiv.org/html/2607.11906#S8.SS8.p2.1),[Table 13](https://arxiv.org/html/2607.11906#S8.T13.5.11.9.2.1.1),[Table 14](https://arxiv.org/html/2607.11906#S8.T14.1.8.7.2.1.1)\.
- M\. Matthews, M\. Beukman, C\. Lu, and J\. Foerster \(2024b\)Kinetix: investigating the training of general agents through open\-ended physics\-based control tasks\.arXiv preprint arXiv:2410\.23208\.Cited by:[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.5.4.2.1.1),[§8\.3](https://arxiv.org/html/2607.11906#S8.SS3.p2.1),[§8\.8](https://arxiv.org/html/2607.11906#S8.SS8.p2.1),[Table 13](https://arxiv.org/html/2607.11906#S8.T13.5.8.6.2.1.1),[Table 14](https://arxiv.org/html/2607.11906#S8.T14.1.8.7.2.1.1)\.
- L\. C\. Melo \(2022\)Transformers are meta\-reinforcement learners\.InInternational Conference on Machine Learning,pp\. 15340–15359\.Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p2.1),[Figure 10](https://arxiv.org/html/2607.11906#S6.F10.1.pic1.4.4.4.1.1.2.1.1),[§6\.4](https://arxiv.org/html/2607.11906#S6.SS4.p1.1)\.
- R\. Mendonca, X\. Geng, C\. Finn, and S\. Levine \(2020\)Meta\-reinforcement learning robust to distributional shift via model identification and experience relabeling\.arXiv preprint arXiv:2006\.07178\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.7.5.2.1.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p2.1),[§3\.2](https://arxiv.org/html/2607.11906#S3.SS2.p1.5),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.4.8.2.4.1.1)\.
- S\. Min, X\. Lyu, A\. Holtzman, M\. Artetxe, M\. Lewis, H\. Hajishirzi, and L\. Zettlemoyer \(2022\)Rethinking the role of demonstrations: what makes in\-context learning work?\.InConference on Empirical Methods in Natural Language Processing,pp\. 11048–11064\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.759)Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.9.7.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p3.2),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p3.3),[§5\.2](https://arxiv.org/html/2607.11906#S5.SS2.p3.1),[§6\.1](https://arxiv.org/html/2607.11906#S6.SS1.p1.1),[§7\.1](https://arxiv.org/html/2607.11906#S7.SS1.p5.1),[§9\.1](https://arxiv.org/html/2607.11906#S9.SS1.p1.1)\.
- N\. Mishra, M\. Rohaninejad, X\. Chen, and P\. Abbeel \(2018\)A simple neural attentive meta\-learner\.InInternational Conference on Learning Representations,Cited by:[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p3.1)\.
- A\. Moeini, M\. Kwon, A\. K\. Bozkurt, Y\. Motai, R\. Chandra, L\. Feng, and S\. Zhang \(2025a\)Safe in\-context reinforcement learning\.arXiv preprint arXiv:2509\.25582\.Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p3.1),[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[§10\.2](https://arxiv.org/html/2607.11906#S10.SS2.p1.1),[§10\.2](https://arxiv.org/html/2607.11906#S10.SS2.p2.1),[§10\.3](https://arxiv.org/html/2607.11906#S10.SS3.p1.1),[Table 18](https://arxiv.org/html/2607.11906#S10.T18.5.10.8.2.1.1),[Table 18](https://arxiv.org/html/2607.11906#S10.T18.5.12.10.2.1.1),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.1.1.4.1.1),[Table 6](https://arxiv.org/html/2607.11906#S4.T6.1.7.6.3.1.1),[item 1](https://arxiv.org/html/2607.11906#S8.I1.i1.p1.1),[Table 14](https://arxiv.org/html/2607.11906#S8.T14.1.2.1.2.1.1)\.
- A\. Moeini, J\. Wang, J\. Beck, E\. Blaser, S\. Whiteson, R\. Chandra, and S\. Zhang \(2025b\)A survey of in\-context reinforcement learning\.arXiv preprint arXiv:2502\.07978\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.3.1.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p2.1),[item 6](https://arxiv.org/html/2607.11906#S10.I1.i6.p1.1),[Table 18](https://arxiv.org/html/2607.11906#S10.T18.5.3.1.3.1.1),[item 4](https://arxiv.org/html/2607.11906#S8.I1.i4.p1.1),[§8\.5](https://arxiv.org/html/2607.11906#S8.SS5.p3.1),[Table 16](https://arxiv.org/html/2607.11906#S8.T16.1.7.6.2.1.1)\.
- T\. M\. Moldovan and P\. Abbeel \(2012\)Safe exploration in markov decision processes\.arXiv preprint\.Cited by:[§10\.2](https://arxiv.org/html/2607.11906#S10.SS2.p2.1)\.
- G\. Monea, A\. Bosselut, K\. Brantley, and Y\. Artzi \(2024\)LLMs are in\-context bandit reinforcement learners\.arXiv preprint arXiv:2410\.05362\.Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[Table 18](https://arxiv.org/html/2607.11906#S10.T18.5.14.12.2.1.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p3.3),[Table 3](https://arxiv.org/html/2607.11906#S2.T3.1.3.2.3.1.1),[§6\.10](https://arxiv.org/html/2607.11906#S6.SS10.p1.1),[Table 9](https://arxiv.org/html/2607.11906#S6.T9.1.3.2.2.1.1)\.
- S\. Morad, R\. Kortvelesy, M\. Bettini, S\. Liwicki, and A\. Prorok \(2023\)POPGym: benchmarking partially observable reinforcement learning\.InInternational Conference on Learning Representations,Cited by:[§8\.8](https://arxiv.org/html/2607.11906#S8.SS8.p3.1),[Table 13](https://arxiv.org/html/2607.11906#S8.T13.5.14.12.2.1.1),[Table 14](https://arxiv.org/html/2607.11906#S8.T14.1.6.5.2.1.1),[Table 14](https://arxiv.org/html/2607.11906#S8.T14.1.7.6.2.1.1)\.
- T\. Ni, B\. Eysenbach, and R\. Salakhutdinov \(2022\)Recurrent model\-free rl can be a strong baseline for many pomdps\.arXiv preprint arXiv:2110\.05038\.Cited by:[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p2.1),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.3.3.4.1.1),[Table 7](https://arxiv.org/html/2607.11906#S4.T7.1.3.2.3.1.1)\.
- T\. Ni, M\. Ma, B\. Eysenbach, and P\. Bacon \(2024\)When do transformers shine in rl? decoupling memory from credit assignment\.Advances in Neural Information Processing Systems36\.Cited by:[§6\.3](https://arxiv.org/html/2607.11906#S6.SS3.p1.1)\.
- A\. Nie, Y\. Su, B\. Chang, J\. N\. Lee, E\. H\. Chi, Q\. V\. Le, and M\. Chen \(2025\)EVOLvE: evaluating and optimizing LLMs for in\-context exploration\.InInternational Conference on Machine Learning,Cited by:[Table 18](https://arxiv.org/html/2607.11906#S10.T18.5.14.12.2.1.1),[Table 3](https://arxiv.org/html/2607.11906#S2.T3.1.3.2.3.1.1),[§6\.10](https://arxiv.org/html/2607.11906#S6.SS10.p3.1)\.
- A\. Nikulin, V\. Kurenkov, I\. Zisman, A\. Agarkov, V\. Sinii, and S\. Kolesnikov \(2024\)XLand\-MiniGrid: scalable meta\-reinforcement learning environments in JAX\.InAdvances in Neural Information Processing Systems,Cited by:[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.5.4.2.1.1),[§8\.3](https://arxiv.org/html/2607.11906#S8.SS3.p2.1),[§8\.8](https://arxiv.org/html/2607.11906#S8.SS8.p2.1),[Table 13](https://arxiv.org/html/2607.11906#S8.T13.5.7.5.2.1.1),[Table 14](https://arxiv.org/html/2607.11906#S8.T14.1.8.7.2.1.1)\.
- A\. Nikulin, I\. Zisman, A\. Zemtsov, V\. Sinii, V\. Kurenkov, and S\. Kolesnikov \(2025\)XLand\-100B: a large\-scale multi\-task dataset for in\-context reinforcement learning\.InInternational Conference on Learning Representations,Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.13.11.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[Figure 3](https://arxiv.org/html/2607.11906#S2.F3.1.pic1.9.9.9.1.1.5),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p2.1),[Figure 6](https://arxiv.org/html/2607.11906#S4.F6.1.pic1.4.4.4.1.1.2.1.1),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.4.8.2.4.1.1),[§6\.4](https://arxiv.org/html/2607.11906#S6.SS4.p1.1),[§8\.3](https://arxiv.org/html/2607.11906#S8.SS3.p2.1),[§8\.8](https://arxiv.org/html/2607.11906#S8.SS8.p2.1),[Table 13](https://arxiv.org/html/2607.11906#S8.T13.5.7.5.2.1.1),[Table 14](https://arxiv.org/html/2607.11906#S8.T14.1.8.7.2.1.1)\.
- B\. Norman and J\. Clune \(2024\)First\-explore, then exploit: meta\-learning to solve hard exploration\-exploitation trade\-offs\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 27490–27528\.Cited by:[§6\.1](https://arxiv.org/html/2607.11906#S6.SS1.p2.1)\.
- Open X\-Embodiment Collaboration, A\. O’Neill, A\. Rehman, A\. Gupta, A\. Maddukuri, A\. Gupta, A\. Padalkar,et al\.\(2024\)Open X\-embodiment: robotic learning datasets and RT\-X models\.InIEEE International Conference on Robotics and Automation,Cited by:[§8\.4](https://arxiv.org/html/2607.11906#S8.SS4.p3.1)\.
- T\. Ota \(2024\)Decision mamba: reinforcement learning via sequence modeling with selective state spaces\.arXiv preprint arXiv:2403\.19925\.Cited by:[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p2.1),[§6\.3](https://arxiv.org/html/2607.11906#S6.SS3.p1.1),[Table 10](https://arxiv.org/html/2607.11906#S6.T10.1.4.3.2.1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.9.8.2.1.1)\.
- C\. Packer, K\. Gao, J\. Kos, P\. Krahenbuhl, V\. Koltun, and D\. Song \(2018\)Assessing generalization in deep reinforcement learning\.arXiv preprint arXiv:1810\.12282\.Cited by:[§8\.1](https://arxiv.org/html/2607.11906#S8.SS1.p1.1),[§8\.8](https://arxiv.org/html/2607.11906#S8.SS8.p1.1)\.
- C\. Packer, S\. Wooders, K\. Lin, V\. Fang, S\. G\. Patil, I\. Stoica, and J\. E\. Gonzalez \(2023\)MemGPT: towards LLMs as operating systems\.arXiv preprint arXiv:2310\.08560\.Cited by:[item 1](https://arxiv.org/html/2607.11906#S10.I1.i1.p1.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p3.3),[Figure 8](https://arxiv.org/html/2607.11906#S5.F8.1.pic1.4.4.4.4.1.1.2.2),[§5\.1](https://arxiv.org/html/2607.11906#S5.SS1.p1.1),[§5\.4](https://arxiv.org/html/2607.11906#S5.SS4.p1.1),[Table 8](https://arxiv.org/html/2607.11906#S5.T8.1.6.5.4.1.1),[Figure 10](https://arxiv.org/html/2607.11906#S6.F10.1.pic1.6.6.6.1.1.2.1.1),[§6\.9](https://arxiv.org/html/2607.11906#S6.SS9.p1.1),[Table 9](https://arxiv.org/html/2607.11906#S6.T9.1.6.5.2.1.1),[Table 15](https://arxiv.org/html/2607.11906#S8.T15.4.8.3.2.1.1),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.7.6.2.1.1)\.
- S\. Padakandla, P\. K\. J\., and S\. Bhatnagar \(2020\)Reinforcement learning in non\-stationary environments\.Applied Intelligence\.Cited by:[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p4.1),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.2.2.4.1.1)\.
- S\. Padakandla \(2020\)A survey of reinforcement learning algorithms for dynamically varying environments\.ACM Computing Surveys\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.5.3.2.1.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p4.1)\.
- C\. Park, X\. Liu, A\. Ozdaglar, and K\. Zhang \(2024\)Do llm agents have regret? a case study in online learning and games\.arXiv preprint\.Cited by:[§6\.10](https://arxiv.org/html/2607.11906#S6.SS10.p1.1),[Table 9](https://arxiv.org/html/2607.11906#S6.T9.1.3.2.2.1.1)\.
- J\. Parker\-Holder, M\. Jiang, M\. Dennis, M\. Samvelyan, J\. N\. Foerster, E\. Grefenstette, and T\. Rocktaschel \(2022\)Evolving curricula with regret\-based environment design\.InInternational Conference on Machine Learning,Cited by:[§7\.3](https://arxiv.org/html/2607.11906#S7.SS3.p1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.5.4.2.1.1)\.
- J\. Pasukonis, T\. Lillicrap, and D\. Hafner \(2022\)Evaluating long\-term memory in 3d mazes\.arXiv preprint arXiv:2210\.13383\.Cited by:[Table 7](https://arxiv.org/html/2607.11906#S4.T7.1.5.4.3.1.1),[§6\.4](https://arxiv.org/html/2607.11906#S6.SS4.p1.1),[Table 10](https://arxiv.org/html/2607.11906#S6.T10.1.5.4.2.1.1)\.
- B\. Peng, D\. Goldstein, Q\. Anthony, A\. Albalak, E\. Alcaide, S\. Biderman, E\. Cheah, X\. Du, T\. Ferdinan, H\. Hou,et al\.\(2024\)Eagle and finch: rwkv with matrix\-valued states and dynamic recurrence\.arXiv preprint arXiv:2404\.05892\.Cited by:[§6\.3](https://arxiv.org/html/2607.11906#S6.SS3.p1.1),[§9\.1](https://arxiv.org/html/2607.11906#S9.SS1.p1.1)\.
- C\. Perez, F\. P\. Such, and T\. Karaletsos \(2020\)Generalized hidden parameter mdps: transferable model\-based rl in a handful of trials\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.34,pp\. 5403–5411\.Cited by:[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p2.1),[Table 13](https://arxiv.org/html/2607.11906#S8.T13.5.4.2.2.1.1)\.
- A\. Pettet, Y\. Zhang, B\. Luo, K\. Wray, H\. Baier, A\. Laszka, A\. Dubey, and A\. Mukhopadhyay \(2024\)Decision making in non\-stationary environments with policy\-augmented search\.InInternational Joint Conference on Autonomous Agents and Multiagent Systems,pp\. 2417–2419\.Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p4.1),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.2.2.4.1.1),[Table 13](https://arxiv.org/html/2607.11906#S8.T13.5.15.13.2.1.1)\.
- A\. Polubarov, N\. Lyubaykin, A\. Derevyagin, A\. Grishin, I\. Saprygin, A\. Serkov, M\. Averchenko, D\. Tikhonov, M\. Zhdanov, A\. Nikulin, I\. Zisman, A\. Klepach, A\. Zemtsov, and V\. Kurenkov \(2026\)Vintix II: decision pre\-trained transformer is a scalable in\-context reinforcement learner\.InInternational Conference on Learning Representations,Note:Poster; arXiv:2604\.05112Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.13.11.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p2.1),[§6\.4](https://arxiv.org/html/2607.11906#S6.SS4.p1.1),[Table 9](https://arxiv.org/html/2607.11906#S6.T9.1.2.1.2.1.1)\.
- A\. Polubarov, N\. Lyubaykin, A\. Derevyagin, I\. Zisman, D\. Tarasov, A\. Nikulin, and V\. Kurenkov \(2025\)Vintix: action model via in\-context reinforcement learning\.arXiv preprint arXiv:2501\.19400\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.13.11.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p2.1),[§6\.2](https://arxiv.org/html/2607.11906#S6.SS2.p1.1),[§6\.4](https://arxiv.org/html/2607.11906#S6.SS4.p1.1),[Table 9](https://arxiv.org/html/2607.11906#S6.T9.1.2.1.2.1.1)\.
- S\. Powers, E\. Xing, E\. Kolve, R\. Mottaghi, and A\. Gupta \(2021\)CORA: benchmarks, baselines, and metrics as a platform for continual reinforcement learning agents\.arXiv preprint arXiv:2110\.10067\.Cited by:[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p5.1),[§8\.4](https://arxiv.org/html/2607.11906#S8.SS4.p2.1),[§8\.8](https://arxiv.org/html/2607.11906#S8.SS8.p3.1),[Table 13](https://arxiv.org/html/2607.11906#S8.T13.5.17.15.2.1.1)\.
- R\. F\. Prudencio, M\. R\. O\. A\. Maximo, and E\. L\. Colombini \(2023\)A survey on offline reinforcement learning: taxonomy, review, and open problems\.IEEE Transactions on Neural Networks and Learning Systems35\(8\),pp\. 10237–10257\.External Links:[Document](https://dx.doi.org/10.1109/TNNLS.2023.3250269)Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.8.6.2.1.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p3.2),[Table 3](https://arxiv.org/html/2607.11906#S2.T3.1.4.3.3.1.1),[§7\.1](https://arxiv.org/html/2607.11906#S7.SS1.p1.1)\.
- M\. L\. Puterman \(1994\)Markov decision processes: discrete stochastic dynamic programming\.Wiley\.Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p2.1),[§3\.1](https://arxiv.org/html/2607.11906#S3.SS1.p1.14)\.
- Z\. Qin, J\. Jiang, and Z\. Zhu \(2026\)Learning to adapt: in\-context learning beyond stationarity\.arXiv preprint arXiv:2604\.10946\.Cited by:[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p5.2),[§6\.3](https://arxiv.org/html/2607.11906#S6.SS3.p1.1),[§9\.1](https://arxiv.org/html/2607.11906#S9.SS1.p1.1)\.
- M\. A\. Raad, A\. Ahuja, C\. Barros, F\. Besse, A\. Bolt, A\. Bolton, B\. Brownfield, G\. Buttimore, M\. Cant, S\. Chakera,et al\.\(2024\)Scaling instructable agents across many simulated worlds\.arXiv preprint arXiv:2404\.10179\.Cited by:[§6\.8](https://arxiv.org/html/2607.11906#S6.SS8.p1.1)\.
- K\. Rakelly, A\. Zhou, D\. Quillen, C\. Finn, and S\. Levine \(2019\)Efficient off\-policy meta\-reinforcement learning via probabilistic context variables\.InInternational Conference on Machine Learning,Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.7.5.2.1.1),[Figure 3](https://arxiv.org/html/2607.11906#S2.F3.1.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.5),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p3.1),[§3\.1](https://arxiv.org/html/2607.11906#S3.SS1.p2.8),[Table 7](https://arxiv.org/html/2607.11906#S4.T7.1.4.3.3.1.1),[Table 8](https://arxiv.org/html/2607.11906#S5.T8.1.4.3.4.1.1),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.6.5.2.1.1)\.
- O\. Ram, Y\. Levine, I\. Dalmedigos, D\. Muhlgay, A\. Shashua, K\. Leyton\-Brown, and Y\. Shoham \(2023\)In\-context retrieval\-augmented language models\.Transactions of the Association for Computational Linguistics11,pp\. 1316–1331\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00605)Cited by:[§5\.1](https://arxiv.org/html/2607.11906#S5.SS1.p1.1),[§6\.5](https://arxiv.org/html/2607.11906#S6.SS5.p2.1)\.
- S\. C\. Raparthy, E\. Hambro, R\. Kirk, M\. Henaff, and R\. Raileanu \(2024\)Generalization to new sequential decision making tasks with in\-context learning\.InInternational Conference on Machine Learning,pp\. 42138–42158\.Cited by:[Table 4](https://arxiv.org/html/2607.11906#S3.T4.1.3.2.2.1.1),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.3.3.4.1.1),[§6\.2](https://arxiv.org/html/2607.11906#S6.SS2.p2.1)\.
- S\. Reed, K\. Zolna, E\. Parisotto, S\. G\. Colmenarejo, A\. Novikov, G\. Barth\-Maron, M\. Gimenez, Y\. Sulsky, J\. Kay, J\. T\. Springenberg, T\. Eccles, J\. Bruce, A\. Razavi, A\. Edwards, N\. Heess, Y\. Chen, R\. Hadsell, O\. Vinyals, M\. Bordbar, and N\. de Freitas \(2022\)A generalist agent\.Transactions on Machine Learning Research\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.13.11.2.1.1),[§6\.4](https://arxiv.org/html/2607.11906#S6.SS4.p1.1),[Table 9](https://arxiv.org/html/2607.11906#S6.T9.1.2.1.2.1.1)\.
- H\. Ren, Y\. Zhu, J\. Leskovec, A\. Anandkumar, and A\. Garg \(2020\)OCEAN: online task inference for compositional tasks with context adaptation\.InConference on Uncertainty in Artificial Intelligence,pp\. 1378–1387\.Cited by:[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p2.1),[Table 7](https://arxiv.org/html/2607.11906#S4.T7.1.4.3.3.1.1),[§5\.1](https://arxiv.org/html/2607.11906#S5.SS1.p1.1)\.
- M\. Rentschler and J\. Roberts \(2025\)RL \+ transformer = a general\-purpose problem solver\.InProceedings of the 1st Workshop for Research on Agent Language Models,pp\. 401–410\.Cited by:[§6\.10](https://arxiv.org/html/2607.11906#S6.SS10.p4.1)\.
- S\. Ritter, J\. X\. Wang, Z\. Kurth\-Nelson, S\. M\. Jayakumar, C\. Blundell, R\. Pascanu, and M\. Botvinick \(2018\)Been there, done that: meta\-learning with episodic recall\.InInternational Conference on Machine Learning,Cited by:[§6\.4](https://arxiv.org/html/2607.11906#S6.SS4.p1.1)\.
- Y\. Russac, C\. Vernade, and O\. Cappe \(2019\)Weighted linear bandits for non\-stationary environments\.arXiv preprint arXiv:1909\.09146\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.6.4.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p5.1),[item 3](https://arxiv.org/html/2607.11906#S10.I1.i3.p1.1),[§3\.1](https://arxiv.org/html/2607.11906#S3.SS1.p3.1),[Table 6](https://arxiv.org/html/2607.11906#S4.T6.1.3.2.3.1.1),[Table 16](https://arxiv.org/html/2607.11906#S8.T16.1.10.9.2.1.1),[§9\.2](https://arxiv.org/html/2607.11906#S9.SS2.p1.2),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.5.4.2.1.1)\.
- A\. A\. Rusu, N\. C\. Rabinowitz, G\. Desjardins, H\. Soyer, J\. Kirkpatrick, K\. Kavukcuoglu, R\. Pascanu, and R\. Hadsell \(2016\)Progressive neural networks\.arXiv preprint arXiv:1606\.04671\.Cited by:[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p5.1)\.
- A\. Rutherford, M\. Beukman, T\. Willi, B\. Lacerda, N\. Hawes, and J\. N\. Foerster \(2024\)No regrets: investigating and improving regret approximations for curriculum discovery\.InAdvances in Neural Information Processing Systems,Cited by:[§7\.3](https://arxiv.org/html/2607.11906#S7.SS3.p1.1)\.
- M\. Samvelyan, R\. Kirk, V\. Kurin, J\. Parker\-Holder, M\. Jiang, E\. Hambro, F\. Petroni, H\. Küttler, E\. Grefenstette, and T\. Rocktäschel \(2021\)MiniHack the planet: a sandbox for open\-ended reinforcement learning research\.InAdvances in Neural Information Processing Systems Datasets and Benchmarks Track,Cited by:[§8\.4](https://arxiv.org/html/2607.11906#S8.SS4.p2.1),[§8\.8](https://arxiv.org/html/2607.11906#S8.SS8.p2.1),[Table 13](https://arxiv.org/html/2607.11906#S8.T13.5.12.10.2.1.1),[Table 14](https://arxiv.org/html/2607.11906#S8.T14.1.8.7.2.1.1)\.
- G\. Sarch, L\. Jang, M\. J\. Tarr, W\. W\. Cohen, K\. Marino, and K\. Fragkiadaki \(2024\)VLM agents generate their own memories: distilling experience into embodied programs of thought\.InAdvances in Neural Information Processing Systems,Cited by:[§6\.10](https://arxiv.org/html/2607.11906#S6.SS10.p3.1)\.
- H\. Satija, P\. Amortila, and J\. Pineau \(2020\)Constrained markov decision processes via backward value functions\.InInternational Conference on Machine Learning,Cited by:[§10\.2](https://arxiv.org/html/2607.11906#S10.SS2.p2.1)\.
- A\. H\. Sayed \(2011\)Adaptive filters\.John Wiley and Sons\.Cited by:[§6\.3](https://arxiv.org/html/2607.11906#S6.SS3.p1.1),[§9\.1](https://arxiv.org/html/2607.11906#S9.SS1.p1.1)\.
- T\. Schmied, F\. Paischer, V\. Patil, M\. Hofmarcher, R\. Pascanu, and S\. Hochreiter \(2024\)Retrieval\-augmented decision transformer: external memory for in\-context RL\.arXiv preprint arXiv:2410\.07071\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.12.10.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p3.1),[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[item 4](https://arxiv.org/html/2607.11906#S10.I1.i4.p1.1),[§10\.2](https://arxiv.org/html/2607.11906#S10.SS2.p1.1),[§10\.3](https://arxiv.org/html/2607.11906#S10.SS3.p1.1),[Table 18](https://arxiv.org/html/2607.11906#S10.T18.5.5.3.2.1.1),[§11](https://arxiv.org/html/2607.11906#S11.p1.1),[Figure 3](https://arxiv.org/html/2607.11906#S2.F3.1.pic1.9.9.9.1.1.5),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p2.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p5.1),[§3\.1](https://arxiv.org/html/2607.11906#S3.SS1.p2.8),[§3\.2](https://arxiv.org/html/2607.11906#S3.SS2.p2.1),[§3\.4](https://arxiv.org/html/2607.11906#S3.SS4.p2.1),[Table 4](https://arxiv.org/html/2607.11906#S3.T4.1.4.3.3.1.1),[Table 4](https://arxiv.org/html/2607.11906#S3.T4.1.7.6.3.1.1),[Figure 6](https://arxiv.org/html/2607.11906#S4.F6.1.pic1.7.7.7.1.1.2.1.1),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.4.10.4.4.1.1),[Table 6](https://arxiv.org/html/2607.11906#S4.T6.1.4.3.3.1.1),[Figure 8](https://arxiv.org/html/2607.11906#S5.F8.1.pic1.2.2.2.2.1.1.2.2),[Figure 8](https://arxiv.org/html/2607.11906#S5.F8.1.pic1.3.3.3.3.1.1.2.2),[§5\.2](https://arxiv.org/html/2607.11906#S5.SS2.p4.1),[§5\.4](https://arxiv.org/html/2607.11906#S5.SS4.p1.1),[Table 8](https://arxiv.org/html/2607.11906#S5.T8.1.5.4.4.1.1),[Table 8](https://arxiv.org/html/2607.11906#S5.T8.1.8.7.4.1.1),[Figure 10](https://arxiv.org/html/2607.11906#S6.F10.1.pic1.5.5.5.1.1.2.1.1),[Figure 9](https://arxiv.org/html/2607.11906#S6.F9.1.pic1.4.4.4.1.1.2.1.1),[Figure 9](https://arxiv.org/html/2607.11906#S6.F9.1.pic1.9.9.9.1.1.2.1.1),[§6\.5](https://arxiv.org/html/2607.11906#S6.SS5.p1.1),[Table 10](https://arxiv.org/html/2607.11906#S6.T10.1.6.5.2.1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.6.5.2.1.1),[item 4](https://arxiv.org/html/2607.11906#S8.I1.i4.p1.1),[§8\.5](https://arxiv.org/html/2607.11906#S8.SS5.p1.4),[Table 12](https://arxiv.org/html/2607.11906#S8.T12.1.5.4.3.1.1),[Table 12](https://arxiv.org/html/2607.11906#S8.T12.1.7.6.3.1.1),[Table 14](https://arxiv.org/html/2607.11906#S8.T14.1.5.4.2.1.1),[Table 15](https://arxiv.org/html/2607.11906#S8.T15.4.7.2.2.1.1),[Table 16](https://arxiv.org/html/2607.11906#S8.T16.1.5.4.2.1.1),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.7.6.2.1.1)\.
- G\. Shala, A\. Biedenkapp, and J\. Grabocka \(2024\)Hierarchical transformers are efficient meta\-reinforcement learners\.arXiv preprint arXiv:2402\.06402\.Cited by:[§6\.4](https://arxiv.org/html/2607.11906#S6.SS4.p1.1)\.
- L\. X\. Shi, Y\. Jiang, J\. Grigsby, L\. Fan, and Y\. Zhu \(2023\)Cross\-episodic curriculum for transformer agents\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 13–34\.Cited by:[§6\.1](https://arxiv.org/html/2607.11906#S6.SS1.p2.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.3.2.2.1.1)\.
- N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. Yao \(2023\)Reflexion: language agents with verbal reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 8634–8652\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.11.9.2.1.1),[Table 18](https://arxiv.org/html/2607.11906#S10.T18.5.11.9.2.1.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p3.3),[§5\.4](https://arxiv.org/html/2607.11906#S5.SS4.p1.1),[Figure 10](https://arxiv.org/html/2607.11906#S6.F10.1.pic1.9.9.9.1.1.2.1.1),[§6\.10](https://arxiv.org/html/2607.11906#S6.SS10.p1.1),[Table 15](https://arxiv.org/html/2607.11906#S8.T15.4.10.5.2.1.1)\.
- M\. Shridhar, X\. Yuan, M\. Côté, Y\. Bisk, A\. Trischler, and M\. Hausknecht \(2020\)ALFWorld: aligning text and embodied environments for interactive learning\.arXiv preprint arXiv:2010\.03768\.Cited by:[Table 5](https://arxiv.org/html/2607.11906#S4.T5.4.7.1.4.1.1),[§8\.8](https://arxiv.org/html/2607.11906#S8.SS8.p4.1),[Table 13](https://arxiv.org/html/2607.11906#S8.T13.5.18.16.2.1.1)\.
- V\. Sinii, A\. Nikulin, V\. Kurenkov, I\. Zisman, and S\. Kolesnikov \(2024\)In\-context reinforcement learning for variable action spaces\.InInternational Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p3.1),[Table 18](https://arxiv.org/html/2607.11906#S10.T18.5.9.7.2.1.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p2.1),[§3\.2](https://arxiv.org/html/2607.11906#S3.SS2.p2.1),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.4.4.4.1.1),[§4](https://arxiv.org/html/2607.11906#S4.p1.1),[Table 8](https://arxiv.org/html/2607.11906#S5.T8.1.7.6.4.1.1),[§6\.7](https://arxiv.org/html/2607.11906#S6.SS7.p1.1),[Table 14](https://arxiv.org/html/2607.11906#S8.T14.1.4.3.2.1.1),[Table 15](https://arxiv.org/html/2607.11906#S8.T15.4.9.4.2.1.1)\.
- J\. T\. H\. Smith, A\. Warrington, and S\. W\. Linderman \(2023\)Simplified state space layers for sequence modeling\.InInternational Conference on Learning Representations,Cited by:[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p2.1),[§6\.3](https://arxiv.org/html/2607.11906#S6.SS3.p1.1)\.
- J\. Son, S\. Lee, and G\. Kim \(2025\)Distilling reinforcement learning algorithms for in\-context model\-based planning\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[Figure 6](https://arxiv.org/html/2607.11906#S4.F6.1.pic1.2.2.2.1.1.2.1.1),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.2.2.4.1.1),[§5\.4](https://arxiv.org/html/2607.11906#S5.SS4.p1.1),[Figure 10](https://arxiv.org/html/2607.11906#S6.F10.1.pic1.8.8.8.1.1.2.1.1),[Figure 9](https://arxiv.org/html/2607.11906#S6.F9.1.pic1.10.10.10.1.1.2.1.1),[Figure 9](https://arxiv.org/html/2607.11906#S6.F9.1.pic1.6.6.6.1.1.2.1.1),[§6\.8](https://arxiv.org/html/2607.11906#S6.SS8.p1.1),[Table 10](https://arxiv.org/html/2607.11906#S6.T10.1.8.7.2.1.1),[§7\.4](https://arxiv.org/html/2607.11906#S7.SS4.p1.1),[Table 14](https://arxiv.org/html/2607.11906#S8.T14.1.3.2.2.1.1)\.
- K\. Song, A\. Moeini, P\. Wang, L\. Gong, R\. Chandra, S\. Zhang, and Y\. Qi \(2025\)Reward is enough: LLMs are in\-context reinforcement learners\.arXiv preprint arXiv:2506\.06303\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.11.9.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p3.1),[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p3.3),[§3\.2](https://arxiv.org/html/2607.11906#S3.SS2.p1.5),[Figure 6](https://arxiv.org/html/2607.11906#S4.F6.1.pic1.1.1.1.1.1.2.1.1),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.1.1.4.1.1),[§4](https://arxiv.org/html/2607.11906#S4.p1.1),[Figure 8](https://arxiv.org/html/2607.11906#S5.F8.1.pic1.5.5.5.5.1.1.2.2),[§5\.4](https://arxiv.org/html/2607.11906#S5.SS4.p1.1),[Figure 10](https://arxiv.org/html/2607.11906#S6.F10.1.pic1.9.9.9.1.1.2.1.1),[Figure 9](https://arxiv.org/html/2607.11906#S6.F9.1.pic1.8.8.8.1.1.2.1.1),[§6\.10](https://arxiv.org/html/2607.11906#S6.SS10.p1.1),[Table 9](https://arxiv.org/html/2607.11906#S6.T9.1.3.2.2.1.1),[Table 14](https://arxiv.org/html/2607.11906#S8.T14.1.2.1.2.1.1),[Table 15](https://arxiv.org/html/2607.11906#S8.T15.4.10.5.2.1.1)\.
- K\. Sridhar, S\. Dutta, D\. Jayaraman, and I\. Lee \(2024\)REGENT: a retrieval\-augmented generalist agent that can act in\-context in new environments\.arXiv preprint arXiv:2412\.04759\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.12.10.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[item 4](https://arxiv.org/html/2607.11906#S10.I1.i4.p1.1),[§10\.2](https://arxiv.org/html/2607.11906#S10.SS2.p1.1),[§10\.3](https://arxiv.org/html/2607.11906#S10.SS3.p1.1),[Table 18](https://arxiv.org/html/2607.11906#S10.T18.5.5.3.2.1.1),[§11](https://arxiv.org/html/2607.11906#S11.p1.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p2.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p5.1),[§3\.1](https://arxiv.org/html/2607.11906#S3.SS1.p2.8),[Table 6](https://arxiv.org/html/2607.11906#S4.T6.1.4.3.3.1.1),[Figure 8](https://arxiv.org/html/2607.11906#S5.F8.1.pic1.2.2.2.2.1.1.2.2),[Figure 8](https://arxiv.org/html/2607.11906#S5.F8.1.pic1.3.3.3.3.1.1.2.2),[§5\.2](https://arxiv.org/html/2607.11906#S5.SS2.p4.1),[§5\.4](https://arxiv.org/html/2607.11906#S5.SS4.p1.1),[Table 8](https://arxiv.org/html/2607.11906#S5.T8.1.5.4.4.1.1),[Figure 10](https://arxiv.org/html/2607.11906#S6.F10.1.pic1.5.5.5.1.1.2.1.1),[Figure 9](https://arxiv.org/html/2607.11906#S6.F9.1.pic1.4.4.4.1.1.2.1.1),[§6\.5](https://arxiv.org/html/2607.11906#S6.SS5.p1.1),[Table 10](https://arxiv.org/html/2607.11906#S6.T10.1.6.5.2.1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.6.5.2.1.1),[§8\.5](https://arxiv.org/html/2607.11906#S8.SS5.p1.4),[Table 12](https://arxiv.org/html/2607.11906#S8.T12.1.6.5.3.1.1),[Table 12](https://arxiv.org/html/2607.11906#S8.T12.1.7.6.3.1.1),[Table 14](https://arxiv.org/html/2607.11906#S8.T14.1.5.4.2.1.1),[Table 15](https://arxiv.org/html/2607.11906#S8.T15.4.7.2.2.1.1),[Table 16](https://arxiv.org/html/2607.11906#S8.T16.1.6.5.2.1.1),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.7.6.2.1.1)\.
- B\. C\. Stadie, G\. Yang, R\. Houthooft, X\. Chen, Y\. Duan, Y\. Wu, P\. Abbeel, and I\. Sutskever \(2018\)Some considerations on learning to explore via meta\-reinforcement learning\.arXiv preprint arXiv:1803\.01118\.Cited by:[§6\.1](https://arxiv.org/html/2607.11906#S6.SS1.p2.1)\.
- C\. A\. Steinparz, T\. Schmied, F\. Paischer, M\. Dinu, V\. P\. Patil, A\. Bitto\-Nemling, H\. Eghbal\-zadeh, and S\. Hochreiter \(2022\)Reactive exploration to cope with non\-stationarity in lifelong reinforcement learning\.InConference on Lifelong Learning Agents,pp\. 441–469\.Cited by:[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p5.1),[Table 6](https://arxiv.org/html/2607.11906#S4.T6.1.4.3.3.1.1),[Table 8](https://arxiv.org/html/2607.11906#S5.T8.1.8.7.4.1.1),[item 2](https://arxiv.org/html/2607.11906#S8.I1.i2.p1.3),[Table 14](https://arxiv.org/html/2607.11906#S8.T14.1.5.4.2.1.1),[Table 15](https://arxiv.org/html/2607.11906#S8.T15.3.3.2.2.2)\.
- A\. Stooke, A\. Mahajan, C\. Barros, C\. Deck, J\. Bauer, J\. Sygnowski, M\. Trebacz, M\. Jaderberg, M\. Mathieu, N\. McAleese, N\. Bradley\-Schmieg, N\. Wong, N\. Porcel, R\. Raileanu, S\. Hughes\-Fitt, V\. Dalibard, and W\. M\. Czarnecki \(2021\)Open\-ended learning leads to generally capable agents\.arXiv preprint arXiv:2107\.12808\.Cited by:[§6\.4](https://arxiv.org/html/2607.11906#S6.SS4.p1.1),[§7\.3](https://arxiv.org/html/2607.11906#S7.SS3.p1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.5.4.2.1.1)\.
- Y\. Sun, L\. Dong, S\. Huang, S\. Ma, Y\. Xia, J\. Xue, J\. Wang, and F\. Wei \(2023\)Retentive network: a successor to transformer for large language models\.arXiv preprint arXiv:2307\.08621\.Cited by:[§6\.3](https://arxiv.org/html/2607.11906#S6.SS3.p1.1),[§9\.1](https://arxiv.org/html/2607.11906#S9.SS1.p1.1)\.
- R\. S\. Sutton and A\. G\. Barto \(2018\)Reinforcement learning: an introduction\.2 edition,MIT Press\.Cited by:[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p2.1)\.
- F\. Tajwar, Y\. Jiang, A\. Thankaraj, S\. S\. Rahman, J\. Z\. Kolter, J\. Schneider, and R\. Salakhutdinov \(2025\)Training a generally curious agent\.arXiv preprint arXiv:2502\.17543\.Cited by:[§6\.10](https://arxiv.org/html/2607.11906#S6.SS10.p3.1)\.
- W\. Tan, W\. Zhang, X\. Xu, H\. Xia, Z\. Ding, B\. Li, B\. Zhou, J\. Yue, J\. Jiang, Y\. Li, R\. An, M\. Qin, C\. Zong, L\. Zheng, Y\. Wu, X\. Chai, Y\. Bi, T\. Xie, P\. Gu, X\. Li, C\. Zhang, L\. Tian, C\. Wang, X\. Wang, B\. F\. Karlsson, B\. An, S\. Yan, and Z\. Lu \(2025\)Cradle: empowering foundation agents towards general computer control\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 58658–58725\.External Links:[Link](https://proceedings.mlr.press/v267/tan25h.html)Cited by:[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p3.3),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.4.7.1.4.1.1),[§6\.10](https://arxiv.org/html/2607.11906#S6.SS10.p1.1),[Table 9](https://arxiv.org/html/2607.11906#S6.T9.1.4.3.2.1.1)\.
- D\. Tarasov, A\. Nikulin, I\. Zisman, A\. Klepach, A\. Polubarov, N\. Lyubaykin, A\. Derevyagin, I\. Kiselev, and V\. Kurenkov \(2025\)Yes, Q\-learning helps offline in\-context RL\.arXiv preprint arXiv:2502\.17666\.Note:Version 4, revised 25 May 2026Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p2.1),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.4.9.3.4.1.1),[Figure 8](https://arxiv.org/html/2607.11906#S5.F8.1.pic1.1.1.1.1.1.1.2.2),[§5\.4](https://arxiv.org/html/2607.11906#S5.SS4.p1.1),[Figure 10](https://arxiv.org/html/2607.11906#S6.F10.1.pic1.7.7.7.1.1.2.1.1),[Figure 9](https://arxiv.org/html/2607.11906#S6.F9.1.pic1.7.7.7.1.1.2.1.1),[§6\.2](https://arxiv.org/html/2607.11906#S6.SS2.p2.1),[§6\.6](https://arxiv.org/html/2607.11906#S6.SS6.p1.1),[Table 10](https://arxiv.org/html/2607.11906#S6.T10.1.7.6.2.1.1),[§7\.4](https://arxiv.org/html/2607.11906#S7.SS4.p1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.4.3.2.1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.7.6.2.1.1)\.
- N\. Thakur, N\. Reimers, A\. Rücklé, A\. Srivastava, and I\. Gurevych \(2021\)BEIR: a heterogeneous benchmark for zero\-shot evaluation of information retrieval models\.Advances in Neural Information Processing Systems34,pp\. 26980–26992\.Cited by:[§4](https://arxiv.org/html/2607.11906#S4.p1.1)\.
- F\. Trovo, S\. Paladino, M\. Restelli, and N\. Gatti \(2020\)Sliding\-window thompson sampling for non\-stationary settings\.Journal of Artificial Intelligence Research68,pp\. 311–364\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.6.4.2.1.1),[item 3](https://arxiv.org/html/2607.11906#S10.I1.i3.p1.1),[Table 6](https://arxiv.org/html/2607.11906#S4.T6.1.3.2.3.1.1)\.
- B\. Verbruggen, A\. Vanhoyweghen, and V\. Ginis \(2026\)Model\-agnostic solutions for deep reinforcement learning in non\-ergodic contexts\.arXiv preprint arXiv:2601\.08726\.Cited by:[§3\.3](https://arxiv.org/html/2607.11906#S3.SS3.p1.1),[§3\.3](https://arxiv.org/html/2607.11906#S3.SS3.p1.2)\.
- J\. von Oswald, E\. Niklasson, E\. Randazzo, J\. Sacramento, A\. Mordvintsev, A\. Zhmoginov, and M\. Vladymyrov \(2023\)Transformers learn in\-context by gradient descent\.InInternational Conference on Machine Learning,Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.9.7.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p3.2),[Figure 8](https://arxiv.org/html/2607.11906#S5.F8.1.pic1.5.5.5.5.1.1.2.2),[§9\.1](https://arxiv.org/html/2607.11906#S9.SS1.p1.1),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.2.1.2.1.1)\.
- B\. Wang, S\. Xu, K\. Keutzer, Y\. Gao, and B\. Wu \(2021a\)Improving context\-based meta\-reinforcement learning with self\-supervised trajectory contrastive learning\.arXiv preprint arXiv:2103\.06386\.Cited by:[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p2.1),[§6\.1](https://arxiv.org/html/2607.11906#S6.SS1.p2.1),[Table 10](https://arxiv.org/html/2607.11906#S6.T10.1.2.1.2.1.1)\.
- F\. Wang, P\. Shao, Y\. Zhang, B\. Yu, S\. Liu, N\. Ding, Y\. Cao, Y\. Kang, and H\. Wang \(2025a\)Towards large\-scale in\-context reinforcement learning by meta\-training in randomized worlds\.InAdvances in Neural Information Processing Systems,Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.13.11.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p3.1),[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.4.8.2.4.1.1),[Table 6](https://arxiv.org/html/2607.11906#S4.T6.1.5.4.3.1.1),[§7\.3](https://arxiv.org/html/2607.11906#S7.SS3.p1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.5.4.2.1.1),[§8\.8](https://arxiv.org/html/2607.11906#S8.SS8.p1.1),[Table 13](https://arxiv.org/html/2607.11906#S8.T13.5.3.1.2.1.1)\.
- G\. Wang, Y\. Xie, Y\. Jiang, A\. Mandlekar, C\. Xiao, Y\. Zhu, L\. Fan, and A\. Anandkumar \(2023a\)Voyager: an open\-ended embodied agent with large language models\.arXiv preprint arXiv:2305\.16291\.Cited by:[Table 18](https://arxiv.org/html/2607.11906#S10.T18.5.11.9.2.1.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p3.3),[§5\.4](https://arxiv.org/html/2607.11906#S5.SS4.p1.1),[§6\.10](https://arxiv.org/html/2607.11906#S6.SS10.p1.1),[Table 9](https://arxiv.org/html/2607.11906#S6.T9.1.6.5.2.1.1),[Table 15](https://arxiv.org/html/2607.11906#S8.T15.4.10.5.2.1.1)\.
- J\. X\. Wang, M\. King, N\. Porcel, Z\. Kurth\-Nelson, T\. Zhu, C\. Deck, P\. Choy, M\. Cassin, M\. Reynolds, F\. Song, G\. Buttimore, D\. P\. Reichert, N\. Rabinowitz, L\. Matthey, D\. Hassabis, A\. Lerchner, and M\. Botvinick \(2021b\)Alchemy: a benchmark and analysis toolkit for meta\-reinforcement learning agents\.arXiv preprint arXiv:2102\.02926\.Cited by:[§8\.8](https://arxiv.org/html/2607.11906#S8.SS8.p1.1),[Table 13](https://arxiv.org/html/2607.11906#S8.T13.5.6.4.2.1.1),[Table 14](https://arxiv.org/html/2607.11906#S8.T14.1.5.4.2.1.1)\.
- J\. X\. Wang, Z\. Kurth\-Nelson, D\. Tirumala, H\. Soyer, J\. Z\. Leibo, R\. Munos, C\. Blundell, D\. Kumaran, and M\. Botvinick \(2016\)Learning to reinforcement learn\.arXiv preprint arXiv:1611\.05763\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.7.5.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p1.1),[§11](https://arxiv.org/html/2607.11906#S11.p1.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p1.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p3.1)\.
- J\. Wang, E\. Blaser, H\. Daneshmand, and S\. Zhang \(2024a\)Transformers learn temporal difference methods for in\-context reinforcement learning\.arXiv preprint arXiv:2405\.13861\.Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p3.2),[§6\.2](https://arxiv.org/html/2607.11906#S6.SS2.p2.1),[Table 10](https://arxiv.org/html/2607.11906#S6.T10.1.3.2.2.1.1),[§7\.4](https://arxiv.org/html/2607.11906#S7.SS4.p1.1),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.3.2.2.1.1)\.
- J\. Wang, R\. Chandra, and S\. Zhang \(2025b\)Towards provable emergence of in\-context reinforcement learning\.InAdvances in Neural Information Processing Systems,Cited by:[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.3.2.2.1.1)\.
- K\. Wang, P\. Zhang, Z\. Wang, Y\. Gao, L\. Li, Q\. Wang, H\. Chen, C\. Wan, Y\. Lu, Z\. Yang, L\. Wang, R\. Krishna, J\. Wu, L\. Fei\-Fei, Y\. Choi, and M\. Li \(2025c\)VAGEN: reinforcing world model reasoning for multi\-turn VLM agents\.InAdvances in Neural Information Processing Systems,Cited by:[§6\.10](https://arxiv.org/html/2607.11906#S6.SS10.p3.1)\.
- L\. Wang, C\. Ma, X\. Feng, Z\. Zhang, H\. Yang, J\. Zhang, Z\. Chen, J\. Tang, X\. Chen, Y\. Lin, W\. X\. Zhao, Z\. Wei, and J\. Wen \(2024b\)A survey on large language model based autonomous agents\.Frontiers of Computer Science\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.11.9.2.1.1)\.
- R\. Wang, P\. Jansen, M\. Côté, and P\. Ammanabrolu \(2022\)ScienceWorld: is your agent smarter than a 5th grader?\.InConference on Empirical Methods in Natural Language Processing,Cited by:[Table 5](https://arxiv.org/html/2607.11906#S4.T5.4.7.1.4.1.1),[§8\.8](https://arxiv.org/html/2607.11906#S8.SS8.p4.1),[Table 13](https://arxiv.org/html/2607.11906#S8.T13.5.18.16.2.1.1)\.
- W\. Wang, L\. Dong, H\. Cheng, X\. Liu, X\. Yan, J\. Gao, and F\. Wei \(2023b\)Augmenting language models with long\-term memory\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 74530–74543\.Cited by:[item 1](https://arxiv.org/html/2607.11906#S10.I1.i1.p1.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p3.3),[Figure 8](https://arxiv.org/html/2607.11906#S5.F8.1.pic1.4.4.4.4.1.1.2.2),[§5\.1](https://arxiv.org/html/2607.11906#S5.SS1.p1.1),[§5\.4](https://arxiv.org/html/2607.11906#S5.SS4.p1.1),[Table 8](https://arxiv.org/html/2607.11906#S5.T8.1.6.5.4.1.1),[§6\.9](https://arxiv.org/html/2607.11906#S6.SS9.p1.1),[Table 9](https://arxiv.org/html/2607.11906#S6.T9.1.6.5.2.1.1),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.7.6.2.1.1)\.
- Y\. Wang, K\. Cui, T\. Zhang, and Q\. Jin \(2025d\)Towards unpredictable worlds: continual in\-context reinforcement learning in non\-stationary environments\.OpenReview submission to ICLR 2026\.Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[§1](https://arxiv.org/html/2607.11906#S1.p5.1),[item 2](https://arxiv.org/html/2607.11906#S10.I1.i2.p1.1),[item 5](https://arxiv.org/html/2607.11906#S10.I1.i5.p1.1),[§10\.2](https://arxiv.org/html/2607.11906#S10.SS2.p1.1),[§10\.3](https://arxiv.org/html/2607.11906#S10.SS3.p1.1),[Table 18](https://arxiv.org/html/2607.11906#S10.T18.5.3.1.2.1.1),[Table 18](https://arxiv.org/html/2607.11906#S10.T18.5.3.1.3.1.1),[Table 18](https://arxiv.org/html/2607.11906#S10.T18.5.4.2.2.1.1),[§11](https://arxiv.org/html/2607.11906#S11.p2.1),[Figure 3](https://arxiv.org/html/2607.11906#S2.F3.1.pic1.10.10.10.1.1.5),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p5.2),[§3\.4](https://arxiv.org/html/2607.11906#S3.SS4.p2.1),[Table 4](https://arxiv.org/html/2607.11906#S3.T4.1.2.1.3.1.1),[Table 4](https://arxiv.org/html/2607.11906#S3.T4.1.6.5.3.1.1),[Figure 6](https://arxiv.org/html/2607.11906#S4.F6.1.pic1.5.5.5.1.1.2.1.1),[Table 6](https://arxiv.org/html/2607.11906#S4.T6.1.2.1.3.1.1),[Figure 8](https://arxiv.org/html/2607.11906#S5.F8.1.pic1.6.6.6.6.1.1.2.2),[Figure 9](https://arxiv.org/html/2607.11906#S6.F9.1.pic1.9.9.9.1.1.2.1.1),[Table 10](https://arxiv.org/html/2607.11906#S6.T10.1.2.1.2.1.1),[§7\.3](https://arxiv.org/html/2607.11906#S7.SS3.p1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.8.7.2.1.1),[item 2](https://arxiv.org/html/2607.11906#S8.I1.i2.p1.3),[item 5](https://arxiv.org/html/2607.11906#S8.I1.i5.p1.1),[§8\.5](https://arxiv.org/html/2607.11906#S8.SS5.p1.4),[Table 12](https://arxiv.org/html/2607.11906#S8.T12.1.2.1.3.1.1),[Table 12](https://arxiv.org/html/2607.11906#S8.T12.1.3.2.3.1.1),[Table 12](https://arxiv.org/html/2607.11906#S8.T12.1.9.8.3.1.1),[Table 15](https://arxiv.org/html/2607.11906#S8.T15.3.3.2.2.2),[Table 15](https://arxiv.org/html/2607.11906#S8.T15.4.6.1.4.1.1),[Table 16](https://arxiv.org/html/2607.11906#S8.T16.1.2.1.2.1.1),[Table 16](https://arxiv.org/html/2607.11906#S8.T16.1.4.3.2.1.1),[Table 16](https://arxiv.org/html/2607.11906#S8.T16.1.8.7.2.1.1)\.
- Z\. Wang, Z\. He, B\. Zhang, E\. Toledo, and S\. Morad \(2025e\)Investigating memory in RL with POPGym arcade\.arXiv preprint arXiv:2503\.01450\.Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p3.1),[§8\.8](https://arxiv.org/html/2607.11906#S8.SS8.p3.1),[Table 13](https://arxiv.org/html/2607.11906#S8.T13.5.14.12.2.1.1),[Table 14](https://arxiv.org/html/2607.11906#S8.T14.1.6.5.2.1.1)\.
- Z\. Wang, H\. Wang, and Y\. Qi \(2025f\)Hierarchical prompt decision transformer: improving few\-shot policy generalization with global and adaptive guidance\.InCompanion Proceedings of the ACM Web Conference,pp\. 520–529\.External Links:[Document](https://dx.doi.org/10.1145/3701716.3715233)Cited by:[§6\.2](https://arxiv.org/html/2607.11906#S6.SS2.p1.1)\.
- C\. Wei and H\. Luo \(2021\)Non\-stationary reinforcement learning without prior knowledge: an optimal black\-box approach\.InConference on Learning Theory,pp\. 4300–4354\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.5.3.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[§1](https://arxiv.org/html/2607.11906#S1.p5.1),[item 5](https://arxiv.org/html/2607.11906#S10.I1.i5.p1.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p4.1),[§3\.1](https://arxiv.org/html/2607.11906#S3.SS1.p3.1),[§3\.3](https://arxiv.org/html/2607.11906#S3.SS3.p1.2),[Table 6](https://arxiv.org/html/2607.11906#S4.T6.1.3.2.3.1.1),[§8\.1](https://arxiv.org/html/2607.11906#S8.SS1.p1.1),[§9\.2](https://arxiv.org/html/2607.11906#S9.SS2.p1.2),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.5.4.2.1.1)\.
- N\. Wies, Y\. Levine, and A\. Shashua \(2023\)The learnability of in\-context learning\.Advances in Neural Information Processing Systems36\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.9.7.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p3.2),[§5\.2](https://arxiv.org/html/2607.11906#S5.SS2.p3.1),[§6\.1](https://arxiv.org/html/2607.11906#S6.SS1.p1.1),[§7\.1](https://arxiv.org/html/2607.11906#S7.SS1.p5.1),[§9\.1](https://arxiv.org/html/2607.11906#S9.SS1.p1.1),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.2.1.2.1.1)\.
- M\. Wolczyk, M\. Zajac, R\. Pascanu, L\. Kucinski, and P\. Milos \(2022\)Disentangling transfer in continual reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 6304–6317\.Cited by:[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p5.1),[Table 6](https://arxiv.org/html/2607.11906#S4.T6.1.6.5.3.1.1),[§8\.4](https://arxiv.org/html/2607.11906#S8.SS4.p3.1)\.
- J\. Wu, S\. Yin, N\. Feng, X\. He, D\. Li, J\. Hao, and M\. Long \(2024\)iVideoGPT: interactive VideoGPTs are scalable world models\.Advances in Neural Information Processing Systems37,pp\. 68082–68119\.Cited by:[§6\.9](https://arxiv.org/html/2607.11906#S6.SS9.p1.1)\.
- W\. Wu, F\. Liu, H\. Li, Z\. Hu, D\. Dong, C\. Chen, and Z\. Wang \(2025\)Mixture\-of\-experts meets in\-context reinforcement learning\.arXiv preprint arXiv:2506\.05426\.Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.13.11.2.1.1),[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p2.1),[§6\.2](https://arxiv.org/html/2607.11906#S6.SS2.p1.1)\.
- Z\. Xie, X\. Liu, R\. Chandra, and S\. Zhang \(2026\)Convergence and emergence of in\-context reinforcement learning with chain of thought\.arXiv preprint arXiv:2605\.07123\.Cited by:[Table 18](https://arxiv.org/html/2607.11906#S10.T18.5.7.5.2.1.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p3.2),[§9\.1](https://arxiv.org/html/2607.11906#S9.SS1.p1.1),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.4.3.2.1.1)\.
- M\. Xu, Y\. Shen, S\. Zhang, Y\. Lu, D\. Zhao, J\. B\. Tenenbaum, and C\. Gan \(2022\)Prompting decision transformer for few\-shot policy generalization\.InInternational Conference on Machine Learning,Cited by:[Figure 3](https://arxiv.org/html/2607.11906#S2.F3.1.pic1.8.8.8.1.1.5),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p2.1),[Table 4](https://arxiv.org/html/2607.11906#S3.T4.1.3.2.2.1.1),[Figure 6](https://arxiv.org/html/2607.11906#S4.F6.1.pic1.9.9.9.1.1.2.1.1),[Table 7](https://arxiv.org/html/2607.11906#S4.T7.1.2.1.3.1.1),[§6\.2](https://arxiv.org/html/2607.11906#S6.SS2.p1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.2.1.2.1.1)\.
- Q\. Xu, Y\. Huang, Y\. Jiang, L\. Song, J\. Wang, W\. Zheng, and J\. Bian \(2026a\)In\-context compositional Q\-learning for offline reinforcement learning\.InInternational Conference on Learning Representations,Note:arXiv:2509\.24067Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[§6\.5](https://arxiv.org/html/2607.11906#S6.SS5.p1.1),[§6\.6](https://arxiv.org/html/2607.11906#S6.SS6.p1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.7.6.2.1.1)\.
- T\. Xu, Z\. Li, and Q\. Ren \(2024\)Meta\-reinforcement learning robust to distributional shift via performing lifelong in\-context learning\.InInternational Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[§1](https://arxiv.org/html/2607.11906#S1.p5.1),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p5.2),[§3\.2](https://arxiv.org/html/2607.11906#S3.SS2.p1.5),[Table 4](https://arxiv.org/html/2607.11906#S3.T4.1.2.1.3.1.1),[Figure 6](https://arxiv.org/html/2607.11906#S4.F6.1.pic1.4.4.4.1.1.2.1.1),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.4.8.2.4.1.1),[Table 6](https://arxiv.org/html/2607.11906#S4.T6.1.5.4.3.1.1),[§7\.3](https://arxiv.org/html/2607.11906#S7.SS3.p1.1),[Table 16](https://arxiv.org/html/2607.11906#S8.T16.1.4.3.2.1.1)\.
- X\. Xu, P\. Bu, Y\. Wang, B\. F\. Karlsson, Z\. Wang, T\. Song, Q\. Zhu, J\. Song, Z\. Ding, and B\. Zheng \(2026b\)DeepPHY: benchmarking agentic VLMs on physical reasoning\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 34160–34168\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v40i40.40711)Cited by:[§8\.3](https://arxiv.org/html/2607.11906#S8.SS3.p2.1),[§8\.8](https://arxiv.org/html/2607.11906#S8.SS8.p2.1),[Table 13](https://arxiv.org/html/2607.11906#S8.T13.5.9.7.2.1.1)\.
- X\. Xu, P\. Bu, Y\. Wang, B\. F\. Karlsson, Z\. Wang, T\. Song, Q\. Zhu, J\. Song, S\. Zhang, Z\. Ding, and B\. Zheng \(2026c\)ICPRL: acquiring physical intuition from interactive control\.arXiv preprint arXiv:2603\.13295\.Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p4.1),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.2.2.4.1.1),[§5\.4](https://arxiv.org/html/2607.11906#S5.SS4.p1.1),[Figure 10](https://arxiv.org/html/2607.11906#S6.F10.1.pic1.8.8.8.1.1.2.1.1),[Figure 9](https://arxiv.org/html/2607.11906#S6.F9.1.pic1.10.10.10.1.1.2.1.1),[Figure 9](https://arxiv.org/html/2607.11906#S6.F9.1.pic1.6.6.6.1.1.2.1.1),[§6\.1](https://arxiv.org/html/2607.11906#S6.SS1.p2.1),[§6\.8](https://arxiv.org/html/2607.11906#S6.SS8.p1.1),[Table 10](https://arxiv.org/html/2607.11906#S6.T10.1.8.7.2.1.1),[Table 9](https://arxiv.org/html/2607.11906#S6.T9.1.5.4.2.1.1),[§7\.4](https://arxiv.org/html/2607.11906#S7.SS4.p1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.3.2.2.1.1),[§8\.3](https://arxiv.org/html/2607.11906#S8.SS3.p2.1),[Table 13](https://arxiv.org/html/2607.11906#S8.T13.5.9.7.2.1.1),[Table 14](https://arxiv.org/html/2607.11906#S8.T14.1.3.2.2.1.1)\.
- M\. Yang, Y\. Du, K\. Ghasemipour, J\. Tompson, D\. Schuurmans, and P\. Abbeel \(2023a\)Learning interactive real\-world simulators\.arXiv preprint arXiv:2310\.06114\.Cited by:[§6\.8](https://arxiv.org/html/2607.11906#S6.SS8.p1.1)\.
- S\. Yang, B\. Wang, Y\. Shen, R\. Panda, and Y\. Kim \(2023b\)Gated linear attention transformers with hardware\-efficient training\.arXiv preprint arXiv:2312\.06635\.Cited by:[§6\.3](https://arxiv.org/html/2607.11906#S6.SS3.p1.1),[Table 10](https://arxiv.org/html/2607.11906#S6.T10.1.4.3.2.1.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.9.8.2.1.1),[§9\.1](https://arxiv.org/html/2607.11906#S9.SS1.p1.1)\.
- S\. Yao, H\. Chen, J\. Yang, and K\. Narasimhan \(2022\)WebShop: towards scalable real\-world web interaction with grounded language agents\.InAdvances in Neural Information Processing Systems,Cited by:[Table 5](https://arxiv.org/html/2607.11906#S4.T5.4.7.1.4.1.1),[§8\.8](https://arxiv.org/html/2607.11906#S8.SS8.p4.1),[Table 13](https://arxiv.org/html/2607.11906#S8.T13.5.18.16.2.1.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao \(2023\)ReAct: synergizing reasoning and acting in language models\.InInternational Conference on Learning Representations,Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.11.9.2.1.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p3.3),[Table 5](https://arxiv.org/html/2607.11906#S4.T5.4.7.1.4.1.1),[Table 7](https://arxiv.org/html/2607.11906#S4.T7.1.2.1.3.1.1),[Figure 10](https://arxiv.org/html/2607.11906#S6.F10.1.pic1.9.9.9.1.1.2.1.1),[§6\.10](https://arxiv.org/html/2607.11906#S6.SS10.p1.1),[Table 9](https://arxiv.org/html/2607.11906#S6.T9.1.4.3.2.1.1)\.
- O\. Yoran, T\. Wolfson, O\. Ram, and J\. Berant \(2023\)Making retrieval\-augmented language models robust to irrelevant context\.arXiv preprint arXiv:2310\.01558\.Cited by:[§4](https://arxiv.org/html/2607.11906#S4.p1.1)\.
- T\. Yu, D\. Quillen, Z\. He, R\. Julian, A\. Narayan, H\. Shively, A\. Bellathur, K\. Hausman, C\. Finn, and S\. Levine \(2020a\)Meta\-World: a benchmark and evaluation for multi\-task and meta reinforcement learning\.InConference on Robot Learning,Cited by:[§8\.4](https://arxiv.org/html/2607.11906#S8.SS4.p3.1)\.
- T\. Yu, G\. Thomas, L\. Yu, S\. Ermon, J\. Zou, S\. Levine, C\. Finn, and T\. Ma \(2020b\)MOPO: model\-based offline policy optimization\.InAdvances in Neural Information Processing Systems,Cited by:[§4](https://arxiv.org/html/2607.11906#S4.p1.1)\.
- F\. Zenke, B\. Poole, and S\. Ganguli \(2017\)Continual learning through synaptic intelligence\.InInternational Conference on Machine Learning,pp\. 3987–3995\.Cited by:[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p5.1)\.
- A\. Zhao, D\. Huang, Q\. Xu, M\. Lin, Y\. Liu, and G\. Huang \(2024\)ExpeL: LLM agents are experiential learners\.InAAAI Conference on Artificial Intelligence,Cited by:[Table 18](https://arxiv.org/html/2607.11906#S10.T18.5.11.9.2.1.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p3.3),[§5\.4](https://arxiv.org/html/2607.11906#S5.SS4.p1.1),[§6\.10](https://arxiv.org/html/2607.11906#S6.SS10.p1.1),[Table 15](https://arxiv.org/html/2607.11906#S8.T15.4.10.5.2.1.1)\.
- Q\. Zheng, A\. Zhang, and A\. Grover \(2022\)Online decision transformer\.InInternational Conference on Machine Learning,pp\. 27042–27059\.Cited by:[§6\.2](https://arxiv.org/html/2607.11906#S6.SS2.p1.1)\.
- W\. Zhong, L\. Guo, Q\. Gao, H\. Ye, and Y\. Wang \(2024\)MemoryBank: enhancing large language models with long\-term memory\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 19724–19731\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v38i17.29946)Cited by:[item 1](https://arxiv.org/html/2607.11906#S10.I1.i1.p1.1),[Table 18](https://arxiv.org/html/2607.11906#S10.T18.5.6.4.2.1.1),[§2\.1](https://arxiv.org/html/2607.11906#S2.SS1.p3.3),[Figure 8](https://arxiv.org/html/2607.11906#S5.F8.1.pic1.4.4.4.4.1.1.2.2),[§5\.1](https://arxiv.org/html/2607.11906#S5.SS1.p1.1),[§5\.4](https://arxiv.org/html/2607.11906#S5.SS4.p1.1),[Table 8](https://arxiv.org/html/2607.11906#S5.T8.1.6.5.4.1.1),[Figure 10](https://arxiv.org/html/2607.11906#S6.F10.1.pic1.6.6.6.1.1.2.1.1),[§6\.9](https://arxiv.org/html/2607.11906#S6.SS9.p1.1),[Table 9](https://arxiv.org/html/2607.11906#S6.T9.1.6.5.2.1.1),[Table 15](https://arxiv.org/html/2607.11906#S8.T15.4.8.3.2.1.1),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.7.6.2.1.1)\.
- Y\. Zhou, A\. Zanette, J\. Pan, S\. Levine, and A\. Kumar \(2024\)ArCHer: training language model agents via hierarchical multi\-turn RL\.InInternational Conference on Machine Learning,Cited by:[§6\.10](https://arxiv.org/html/2607.11906#S6.SS10.p3.1)\.
- L\. M\. Zintgraf, L\. Feng, C\. Lu, M\. Igl, K\. Hartikainen, K\. Hofmann, and S\. Whiteson \(2021\)Exploration in approximate hyper\-state space for meta reinforcement learning\.InInternational Conference on Machine Learning,pp\. 12991–13001\.Cited by:[§6\.1](https://arxiv.org/html/2607.11906#S6.SS1.p2.1)\.
- L\. M\. Zintgraf, K\. Shiarlis, M\. Igl, S\. Schulze, Y\. Gal, K\. Hofmann, and S\. Whiteson \(2020\)VariBAD: a very good method for bayes\-adaptive deep RL via meta\-learning\.InInternational Conference on Learning Representations,Cited by:[Table 2](https://arxiv.org/html/2607.11906#S1.T2.5.7.5.2.1.1),[Figure 3](https://arxiv.org/html/2607.11906#S2.F3.1.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.5),[§2\.2](https://arxiv.org/html/2607.11906#S2.SS2.p3.1),[§3\.1](https://arxiv.org/html/2607.11906#S3.SS1.p2.8),[Table 7](https://arxiv.org/html/2607.11906#S4.T7.1.4.3.3.1.1),[Table 8](https://arxiv.org/html/2607.11906#S5.T8.1.4.3.4.1.1),[Table 17](https://arxiv.org/html/2607.11906#S9.T17.1.6.5.2.1.1)\.
- I\. Zisman, V\. Kurenkov, A\. Nikulin, V\. Sinii, and S\. Kolesnikov \(2024\)Emergence of in\-context reinforcement learning from noise distillation\.InInternational Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2607.11906#S1.p1.1),[item 2](https://arxiv.org/html/2607.11906#S10.I1.i2.p1.1),[Table 7](https://arxiv.org/html/2607.11906#S4.T7.1.6.5.3.1.1),[Figure 10](https://arxiv.org/html/2607.11906#S6.F10.1.pic1.1.1.1.1.1.2.1.1),[§6\.1](https://arxiv.org/html/2607.11906#S6.SS1.p2.1),[Table 11](https://arxiv.org/html/2607.11906#S7.T11.1.3.2.2.1.1)\.
In-Context Reinforcement Learning under Non-Stationarity: A Survey

Similar Articles

Context-Aware RL for Agentic and Multimodal LLMs

Safe Continual Reinforcement Learning under Nonstationarity via Adaptive Safety Constraints

Position: Deployed Reinforcement Learning should be Continual

Reinforcement Learning Elicits Contextual Learning of Unseen Language Translation

Self-Consolidating Language Models: Continual Knowledge Incorporation from Context

Submit Feedback

Similar Articles

Context-Aware RL for Agentic and Multimodal LLMs
Safe Continual Reinforcement Learning under Nonstationarity via Adaptive Safety Constraints
Position: Deployed Reinforcement Learning should be Continual
Reinforcement Learning Elicits Contextual Learning of Unseen Language Translation
Self-Consolidating Language Models: Continual Knowledge Incorporation from Context