Dynamic Context Scheduling: Learning Beyond the Static Universe

# Learning Beyond the Static Universe
Source: [https://arxiv.org/html/2608.20799](https://arxiv.org/html/2608.20799)
## Dynamic Context Scheduling: Learning Beyond the Static Universe

###### Abstract

We study*dynamic context scheduling*as a training instrument for contextual reinforcement learning\. Rather than treating intra\-episode context variation as a deployment reality, we treat it as a controlled shaping mechanism\. Thereby, context evolves within each training episode according to a predetermined schedule, exposing the policy to a richer and more temporally structured region of the environment parameter space\. We introduceDynamicCARLEnv, a framework that wraps contextual environments with pluggable schedule families, such as sinusoidal offsets or cosine annealing\. Across CartPole, BipedalWalker and VehicleRacing withCarlcontextualization, we show that dynamic schedules match or outperform static context baselines in the out\-of\-distribution \(OOD\) regimes\. Interestingly, for the more complex BipedalWalker and VehicleRacing environments we also achieve higher in\-distribution \(ID\) evaluation performance\. Preliminary findings indicate that automatic search for multi\-stage curricula can successfully discover schedules that improve generalization, performing comparably to extensive grid search over single\-stage schedulers\.

## 1Introduction

Reinforcement learning \(RL\) achieves strong performance in simulation, yet policies trained under narrow, stationary conditions often fail when environment parameters shift at deployment\. Small changes in friction, payload, actuator gain, or gravity can cause large drops in return\([58](https://arxiv.org/html/2608.20799#bib.bib5);[66](https://arxiv.org/html/2608.20799#bib.bib8);[15](https://arxiv.org/html/2608.20799#bib.bib16);[63](https://arxiv.org/html/2608.20799#bib.bib53);[13](https://arxiv.org/html/2608.20799#bib.bib43);[60](https://arxiv.org/html/2608.20799#bib.bib21);[6](https://arxiv.org/html/2608.20799#bib.bib41);[27](https://arxiv.org/html/2608.20799#bib.bib46);[24](https://arxiv.org/html/2608.20799#bib.bib15);[43](https://arxiv.org/html/2608.20799#bib.bib51);[55](https://arxiv.org/html/2608.20799#bib.bib18);[26](https://arxiv.org/html/2608.20799#bib.bib17), see, e\.g\., \)\. This brittleness reflects a clear mismatch between the single\-dynamics regime encountered during training and the potentially different dynamics faced at test\-time\.

Several established paradigms address this issue\. Predominantly, they aim to expose agents to a broader variety of experiences during training, such that transfer to novel experiences does not pose such a drastic, and potentially catastrophic shift at test\-time\.*Domain randomization*\(DR\) improves robustness by exposing the agent to many environment instances\([58](https://arxiv.org/html/2608.20799#bib.bib5);[39](https://arxiv.org/html/2608.20799#bib.bib6)\)in an unstructured manner\. DR typically samples related environments from a distribution without explicitly informing learning agents about the changes in environments\.[13](https://arxiv.org/html/2608.20799#bib.bib43)proposed a suite of environments that leverage*procedural content generation*\(PCG\) to vary level structure and visuals of video games\. This broadly enables learning of behaviors that are robust to changes in an environment but is highly dependent on the choice of distribution\. A too broad distribution might even cause agents to unlearn desirable behavior as sampled environments might require diametrically opposed solutions\. Thus, DR and PCG are often coupled with*curriculum learning*techniques to guide learning to more and more complex scenarios\([38](https://arxiv.org/html/2608.20799#bib.bib50);[28](https://arxiv.org/html/2608.20799#bib.bib47)\)\.*Robust RL*optimizes worst\-case objectives\([41](https://arxiv.org/html/2608.20799#bib.bib7)\)\. For example, in real\-world systems noisy sensor readings are to be expected\([64](https://arxiv.org/html/2608.20799#bib.bib19)\)\. To increase robustness of RL policies the robustness objective is typically modeled as a max\-min problem\. In this setting the goal is to learn a policy that maximizes the reward under the worst possible adversarial setting\([40](https://arxiv.org/html/2608.20799#bib.bib20)\)\. This style of learning can mitigate worst\-case outcomes but largely sacrifices performance in average or best case scenarios as the learned policies act conservatively\.*Meta\-RL*enables online adaptation to new environments\([16](https://arxiv.org/html/2608.20799#bib.bib9);[20](https://arxiv.org/html/2608.20799#bib.bib44);[51](https://arxiv.org/html/2608.20799#bib.bib10);[34](https://arxiv.org/html/2608.20799#bib.bib25);[22](https://arxiv.org/html/2608.20799#bib.bib24);[3](https://arxiv.org/html/2608.20799#bib.bib23);[54](https://arxiv.org/html/2608.20799#bib.bib22), see, e\.g\., \)\. Such approaches, however, often require complex architectures and expensive bi\-level optimization\. Furthermore, Meta\-RL often builds on system identification approaches\([61](https://arxiv.org/html/2608.20799#bib.bib26);[66](https://arxiv.org/html/2608.20799#bib.bib8);[19](https://arxiv.org/html/2608.20799#bib.bib27)\)\. Thereby agents attempt to estimate or recognize environment dynamics from a history of observations\. While enabling online adaptation, such approaches require further environment interactions at deployment to continue to learn\([3](https://arxiv.org/html/2608.20799#bib.bib23);[23](https://arxiv.org/html/2608.20799#bib.bib29)\)\.

Counter to the prior examples,*Contextual RL*\(cRL\) aims to explicitly provide agents with the knowledge of how environments are related to each other\. To this end, cRL assumes that transitions and rewards depend on explicit context variablescc, and the agent learns a policy conditioned on the current context\([25](https://arxiv.org/html/2608.20799#bib.bib45);[35](https://arxiv.org/html/2608.20799#bib.bib48);[6](https://arxiv.org/html/2608.20799#bib.bib41);[65](https://arxiv.org/html/2608.20799#bib.bib28)\)\. While naïvely treating context as another observable can aide in learning more general behavior, more dedicated architectures have been explored in which context is injected into the latent\-representations\([7](https://arxiv.org/html/2608.20799#bib.bib30);[43](https://arxiv.org/html/2608.20799#bib.bib51);[5](https://arxiv.org/html/2608.20799#bib.bib31);[18](https://arxiv.org/html/2608.20799#bib.bib32)\)\.

A key assumption shared by most approaches listed above is that context is treated as a monolithic, static element\([8](https://arxiv.org/html/2608.20799#bib.bib33)\)\. The corresponding contextccis sampled once at the beginning of an episode and held fixed until the episode ends\. This design choice is convenient for training stability and credit assignment, but it is also highly restrictive\. Real\-world physics do not reset between timesteps; payloads shift, actuators fatigue, and terrain variations unfold continuously\. More importantly for training, per\-episode static sampling discards a rich source of structured signal, i\.e\., the temporal variation of context*within*an episode\.

We propose to exploit intra\-episode context variation not as a model of deployment conditions but as a*training instrument*\. Concretely, we define a family of*dynamic context schedules*that govern how the contextccevolves within each training episode, including sinusoidal sweeps, piecewise constant regimes, linear drift, random walks, cosine annealing, and hybrid compositions, and study their effect on policy robustness and generalization\. However, our evaluation protocol always considers policies under static context\. Thus, the temporal variation serves the training\-time purposes

\(i\)*regularization*;\(ii\)and*exploration*\.

The former prevents policies from overfitting while the latter ensures that agents experience a broader, temporally structured set of state\-context pairs\. This training strategy is related to the*dynamic contextual MDP*\(dcMDP\) formalism of[57](https://arxiv.org/html/2608.20799#bib.bib1), which allows context to evolve exogenously via a processΩ⁡\(ct\+1∣ct\)\\Omega\(c\_\{t\+1\}\\mid c\_\{t\}\)\. Here, we instantiateΩ\\Omegawith deterministic, parametric schedules rather than a learned or stochastic model, keeping the framework simple and interpretable while retaining full cMDP semantics\. Thereby, context remains outside the agent’s control, transitions and rewards depend onctc\_\{t\}, and the agent may or may not observectc\_\{t\}\.

Our work provides the following contributions:

1. 1\.We provide a novel training paradigm for \(contextual\) reinforcement learning to facilitate better generalizability of learned policies;
2. 2\.We empirically evaluate a broad suite of scheduling families to study how and which dynamic context changes facilitate better generalization;
3. 3\.We present an open\-source extension of the CARL\([6](https://arxiv.org/html/2608.20799#bib.bib41)\)benchmark to facilitate easy use of dynamic context schedules;
4. 4\.We conduct a state\-space coverage analysis, revealing the counter\-intuitive result that dynamic schedules improve generalization without expanding the agent’s state\-space footprint\.

## 2Related Work

A contextual Markov Decision Process \(cMDP\)\([25](https://arxiv.org/html/2608.20799#bib.bib45)\)augments a standard MDP\([4](https://arxiv.org/html/2608.20799#bib.bib34)\)with the notion of context\. An MDPM=\(𝒮,𝒜,𝒯,ℛ,ρ\)M=\\left\(\\mathcal\{S\},\\mathcal\{A\},\\mathcal\{T\},\\mathcal\{R\},\\rho\\right\)entails a state space𝒮\\mathcal\{S\}, an action space𝒜\\mathcal\{A\}, transition dynamics𝒯:𝒮×𝒜×𝒮→\[0,1\]\\mathcal\{T\}\\colon\\mathcal\{S\}\\times\\mathcal\{A\}\\times\\mathcal\{S\}\\to\\left\[0,1\\right\], a reward functionℛ:𝒮×𝒜→ℝ\\mathcal\{R\}\\colon\\mathcal\{S\}\\times\\mathcal\{A\}\\to\\mathbb\{R\}and an initial state distributionρ\\rho\. Contextual MDPs introduce the notion of contextc∈𝒞c\\in\\mathcal\{C\}that parametrizes transition function𝒯c\\mathcal\{T\}\_\{c\}, the reward functionℛc\\mathcal\{R\}\_\{c\}as well as the initial state distributionρc\\rho\_\{c\}while leaving the state and action spaces unchanged\. Context spaces can be either discrete or described by a distributionp𝒞p\_\{\\mathcal\{C\}\}\([6](https://arxiv.org/html/2608.20799#bib.bib41)\)\. Consequently, a cMDPℳ\\mathcal\{M\}represents a family of related MDPsℳ=\{Mc\}c∼𝒞\\mathcal\{M\}=\\left\\\{M\_\{c\}\\right\\\}\_\{c\\sim\\mathcal\{C\}\}and can be seen as a sub\-class of partially observable MDPs\([27](https://arxiv.org/html/2608.20799#bib.bib46)\)\.[57](https://arxiv.org/html/2608.20799#bib.bib1)presents a special case of cMDPs where contexts are history\-dependent and are allowed to evolve over time, which they dubbed dynamic cMDPs \(dcMDPs\)\. Recently[8](https://arxiv.org/html/2608.20799#bib.bib33)proposes a novel taxonomy of context that distinguishes between allogenic \(environment\-imposed\) and autogenic \(agent\-driven\) contexts\. They further discuss how context might evolve over time\. The notion of autogenic context thus relates to the dcMDP setting, as autogenic contexts may be directly influenced by an agent’s behaviour \(e\.g\. battery power\), whereas allogenic context is independent of an agent’s decisions\.

[27](https://arxiv.org/html/2608.20799#bib.bib46)highlight the utility of the cMDP setting for assessing the zero\-shot generalizability of learned policies and propose a novel evaluation protocol\.[6](https://arxiv.org/html/2608.20799#bib.bib41)implement this protocol in their study on the effect of context on training various deep RL agents on their novel CARL benchmark\. CARL extends common RL benchmarks and environments\([9](https://arxiv.org/html/2608.20799#bib.bib42);[56](https://arxiv.org/html/2608.20799#bib.bib36);[21](https://arxiv.org/html/2608.20799#bib.bib35), e\.g\., \)with physical contexts, such as gravity, friction or masses of robots\. Their study shows that agents that are simply trained on a distribution of contexts without having explicit access to the true context value tend to learn robust behavior but do not necessarily solve every environment optimally\. On the other hand, context aware agents that naïvely concatenate the context to the state\-observation, might be able to perfectly adapt to the changes in environments but may require changes to the RL pipeline \(such as choice of hyperparameters\([17](https://arxiv.org/html/2608.20799#bib.bib37)\)\) to be able to do so\. Beyond naïve concatenation, multiple works explore how to employ hypernetworks to facilitate better adaptability of learned policy by learning adapter modules or the weights of a policy directly\([7](https://arxiv.org/html/2608.20799#bib.bib30);[5](https://arxiv.org/html/2608.20799#bib.bib31);[18](https://arxiv.org/html/2608.20799#bib.bib32)\)\. Opposite to these lines of work,[43](https://arxiv.org/html/2608.20799#bib.bib51);[24](https://arxiv.org/html/2608.20799#bib.bib15)try to exploit contextual information by injecting contextual information into latent representations of a world model\.

Most commonly in contextual RL research, context is treated as a monolithic, static quantity and assumed to be mostly static throughout an episode\([8](https://arxiv.org/html/2608.20799#bib.bib33)\)\. To the best of our knowledge, few works explore dynamic changes of context\.[24](https://arxiv.org/html/2608.20799#bib.bib15)for example, aims to learn when shifts in context, such as the opening or closing of a door, occur\. Whenever works aim to learn to estimate context on\-the\-fly, however, policies might operate under the assumption that context changes between states\([31](https://arxiv.org/html/2608.20799#bib.bib11);[52](https://arxiv.org/html/2608.20799#bib.bib3);[36](https://arxiv.org/html/2608.20799#bib.bib49)\)\. Relatedly,[10](https://arxiv.org/html/2608.20799#bib.bib2)aims to learn policies that are not only working well with the context, but simultaneously aim to estimate how the MDP changes between episodes, such that the policy is setup well for solving this future task as well\. Counter to these approaches, our work proposes to leverage the fact that most of the training occurs in simulation and that it is possible to explicitly adapt context throughout an episode with the goal to push the generalization capabilities of RL agents\.

## 3Dynamic Context Scheduling

We introduce a training framework that replaces the standard static per\-episode context with a parametric intra\-episode schedule\. The contextctc\_\{t\}evolves according to a chosen schedule family throughout each training episode, while evaluation always uses a predefined set of static contexts so that the policy is judged on its generalization to fixed dynamics, not on its ability to track change\.

##### Dynamic Contextual Environments

We open\-source our[DynamicCARLEnv](https://github.com/mrazmartin/dynamicCARL)as a lightweight wrapper that sits on top of anyCarlenvironment\([6](https://arxiv.org/html/2608.20799#bib.bib41)\)\. We intercept each environment step to

\(i\)advance a parametric context schedule,\(ii\)pass the updatedctc\_\{t\}to the underlying simulator, and\(iii\)deliver the chosen context signal to the policy observation\.

The transitionst→at,ctst\+1s\_\{t\}\\xrightarrow\{a\_\{t\},\\,c\_\{t\}\}s\_\{t\+1\}is therefore governed by the live scheduled context rather than the static episode draw\. Context is read from and written to the simulator via user\-supplied getter/setter callables, which isolates the scheduling logic from environment internals and makes the wrapper applicable to anyCarl\-compatible physics backend\.

Figure 1:Dynamic context wrapper around CARLEnv\. Blue components are user\-controlled\. The*Context Scheduler*produces the per\-step contextctc\_\{t\}given the*Schedule Parameters*\(e\.g\., amplitude, period, drift rate, dwell times, change points\)\. The wrapper appliesctc\_\{t\}through a*Physics Adapter*, controls what the policy observes through an*Observation Adapter*\(none,c0c\_\{0\}, orctc\_\{t\}\), and logs\(st,at,rt\+1,ct,cvis\)\(s\_\{t\},a\_\{t\},r\_\{t\+1\},c\_\{t\},c\_\{\\mathrm\{vis\}\}\)via the*Context and State Tracker*\. The base contextualized environment still receivesc0c\_\{0\}at reset\.We study two context observability modes \(Table[1](https://arxiv.org/html/2608.20799#S3.T1)\)\.*Live*mode allows the context to evolve within an episode and can be explicitly observed by an agent, similar to a dcMDP transition model\.*None*mode recovers domain randomization with dynamic changes within an episode\. However*None*does not make the context observable and lets us isolate the effect of temporal structure from explicit context conditioning\.111Note, in settings where only static contexts have been considered,*None*is commonly referred to as “hidden” and live as “concatenation” or “naïve”\.

Table 1:Context observability modes\.When context is observable by an agent, we normalizectc\_\{t\}to lie in\[−1,\+1\]\[\-1,\+1\]before appending it to the state\. We motivate this choice as some context might be obtainable with special sensors\. Such sensors would need to be calibrated and provide sensor\-limit bounds\. Crucially, these bounds are set to a physically plausible range that extends*beyond*the evaluation contexts, rather than being fit to the training or evaluation splits\. This ensures that OOD contexts remain well within the normalized range at test time, so the agent is never exposed to saturated inputs\. Further, this ensures that the evaluation grid is not inadvertently encoded into the policy’s input representation\.

##### Schedule Families

We implement six canonical schedule families, each inducing qualitatively different temporal structure on the context trajectory within an episode\. Three additional composite families \(Sinusoidal Jump, Ornstein–Uhlenbeck, Phased OU\) are described in Appendix[A](https://arxiv.org/html/2608.20799#A1)\.

- •Sinusoidal\.ct=c0\+A​sin⁡\(ω​t\)c\_\{t\}=c\_\{0\}\+A\\sin\(\\omega t\), wherec0c\_\{0\}is the episode\-initial context drawn from the training pool,AAis the amplitude, andω=2​π/T\\omega=2\\pi/Tthe angular frequency\. The direction is randomized per episode; values are reflected at context bounds\. Repeatedly sweeps the policy across a wide context range within a single episode\.
- •Cosine Annealing\.ct=cend\+12​\(cstart−cend\)​\(1\+cos⁡\(π​t/T0\)\)c\_\{t\}=c\_\{\\text\{end\}\}\+\\tfrac\{1\}\{2\}\(c\_\{\\text\{start\}\}\-c\_\{\\text\{end\}\}\)\(1\+\\cos\(\\pi t/T\_\{0\}\)\), with optional periodic restarts\. Produces a smooth monotone drift per cycle; with restarts the policy must repeatedly re\-adapt from diverse starting points, inspired by\([33](https://arxiv.org/html/2608.20799#bib.bib38)\)\.
- •Continuous Incrementer\.ct\+1=ct±δc\_\{t\+1\}=c\_\{t\}\\pm\\delta, reflected at bounds\. The drift direction is fixed per episode \(randomized at reset\)\. Requires generalization across the full context range as a single monotone sweep per episode\.
- •Piecewise Constant\.Context is held fixed within segments of random duration and jumps abruptly to a new value sampled from a discrete set at each change point\. Simulates discrete regime shifts \(e\.g\., sudden payload drops or actuator failure\)\.
- •Random Walk\.ct\+1=clip⁡\(ct\+ϵt,cmin,cmax\)c\_\{t\+1\}=\\operatorname\{clip\}\(c\_\{t\}\+\\epsilon\_\{t\},\\;c\_\{\\text\{min\}\},\\;c\_\{\\text\{max\}\}\),ϵt∼𝒩⁡\(0,σ2\)\\epsilon\_\{t\}\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}\)\. Bounded stochastic drift; tests robustness under slow, persistent perturbations\.
- •Stochastic Jumps\.Large discrete displacements of magnitudeΔ∼Uniform⁡\(mlo,mhi\)\\Delta\\sim\\operatorname\{Uniform\}\(m\_\{\\text\{lo\}\},m\_\{\\text\{hi\}\}\)occur at random timesteps, with direction sampled independently\. Unlike piecewise constant, the magnitude and timing are fully random, producing heavy\-tailed context trajectories\.

Composite families can be formed by combining some of the above\. For example, a*sinusoidal jump*layers a sinusoidal oscillation on top of an intermittently jumping baseline; a*Levy walk*combines stochastic jumps with additive Gaussian noise to obtain heavy\-tailed drift\([62](https://arxiv.org/html/2608.20799#bib.bib39)\)\. Amplitude and step\-size parameters are expressed as fractions of the full context range so that a single hyperparameter setting transfers across context variables with different physical scales\.

Appendix[A](https://arxiv.org/html/2608.20799#A1)shows trajectory examples for all schedule families included in our experiments\.

##### Context Pool Design

Each training episode draws its initial contextc0c\_\{0\}from a finite*context pool*\. We construct pools hierarchically by recursive midpoint insertion:*Pool\-1*contains the single midpoint of the context range;*Pool\-3*adds the two range extremes;*Pool\-5*inserts midpoints between each adjacent pair; and so on\. This construction keeps the global range constant across pool sizes, isolating the effect of context density from boundary effects\. The episode\-initial context is resampled from the pool at the start of each episode, providing inter\-episode diversity independently of intra\-episode schedule variation\.

##### Multi\-Stage Curriculum Scheduling

A single schedule family applied for the full training run may not be optimal\. In the early phases the agent might benefit from aggressive exploration to seed the policy with a broad behavioral repertoire, while late training calls for a more focused refinement pass\. We therefore support*multi\-stage*schedules that divide training intoKKfixed\-length stages, each governed by an independently configured scheduler from any of the families above\. At each stage boundary the scheduler is re\-initialized without interrupting policy training, allowing exploration pressure to be varied throughout the learning lifecycle\.

## 4Experiments

Figure 2:IQM episode reward \(bars: Q1–Q3\) at the final checkpoint across all 16 evaluation contexts for CartPole\. Dashed lines show the random and oracle policy baselines\. For dynamic schedulers we show the two best representatives by combined IQM: one observing only states, one also observing the context value\.We empirically investigate whether dynamic context schedules improve policy robustness and generalization compared to static context training\. Our core ablation on CartPole systematically compares schedule families, pool sizes, and observability modes\. We then validate the findings on the more demanding BipedalWalker and VehicleRacing environments\. We additionally analyze the state\-space coverage induced by different schedules or other multi\-context configurations and explore automated multi\-stage curriculum search via Optuna\([2](https://arxiv.org/html/2608.20799#bib.bib13)\)\.

### 4\.1Experimental Setup

##### Environments\.

We empirically evaluate the impact of dynamic context variation during training on three contextualized environments fromCarl\([6](https://arxiv.org/html/2608.20799#bib.bib41)\)\. We use our dynamic context scheduler as introduced in Section[3](https://arxiv.org/html/2608.20799#S3)\. We thoroughly focus our experiment on CartPole, varying*pole length*as the context\. This axis is challenging at both OOD extremes: short poles and long poles each present distinct failure modes \(see Figure[2](https://arxiv.org/html/2608.20799#S4.F2)\), unlike contexts such as gravity or force magnitude where only one extreme is typically difficult\. Its fast simulation enables a thorough ablation across scheduler families and configurations\. We further validate the methodology onCarlBipedalWalker, where we vary a single*payload x\-axis offset*by attaching a rigid payload to the torso \(see Figure[6\(a\)](https://arxiv.org/html/2608.20799#A3.F6.sf1)\)\. Finally, we extend the evaluation to CarRacing, a vision\-based environment with pixel observations and more complex dynamics, testing whether the methodology scales to image\-based settings\. CarRacing is derived fromCarlVehicleRacing: we fix the vehicle type to a single car and replace the original discrete vehicle\-type context with a continuous payload offset to support our scheduling framework, hence the rename\. We control the offset along the car’s longitudinal and lateral axes \(see Figure[6\(b\)](https://arxiv.org/html/2608.20799#A3.F6.sf2)\)\.222We provide an interactive notebook in which users can drive the car themselves under both static and dynamic payload contexts; see[https://github\.com/mrazmartin/dynamicCARL](https://github.com/mrazmartin/dynamicCARL)\.

##### Evaluation protocol\.

Following[27](https://arxiv.org/html/2608.20799#bib.bib46);[6](https://arxiv.org/html/2608.20799#bib.bib41), we split evaluation contexts into three regimes:*in\-distribution*\(ID, within the training context pool boundaries\),*OOD\-low*\(below the minimum training context value\), and*OOD\-high*\(above the maximum training context value\)\. Full context ranges and eval grids are listed in Appendix[C](https://arxiv.org/html/2608.20799#A3)\. All agents are trained with PPO\([53](https://arxiv.org/html/2608.20799#bib.bib52)\)as implemented in Stable\-Baselines3\([50](https://arxiv.org/html/2608.20799#bib.bib14)\)and evaluated deterministically over3030episodes per evaluation context at the final training checkpoint\. Following[1](https://arxiv.org/html/2608.20799#bib.bib4), we report the interquartile mean \(IQM\) of episode rewards aggregated across evaluation contexts, which provides a robust estimate of central tendency in the presence of outlier seeds\. Due to computational restrictions we report experiments using1010seeds for CartPole,88for BipedalWalker and55for the CarRacing experiments\. Full hyperparameter settings are provided in Appendix[B](https://arxiv.org/html/2608.20799#A2)\. Finally, when context is observable by the agent, we normalize the context input to stabilize PPO training across both static and dynamic conditions \(see Appendix[D\.3\.6](https://arxiv.org/html/2608.20799#A4.SS3.SSS6)\)\.

### 4\.2Scheduler Search Results

Table[2](https://arxiv.org/html/2608.20799#S4.T2)reports the combined IQM at the last training checkpoint for all three environments\. We opt to report final scores throughout to preserve evaluation fairness: selecting the best\-achieved checkpoint would require knowledge of generalisation performance during training, which is information we do not assume access to in practice\. Best\-anytime results are reported in Appendix[D](https://arxiv.org/html/2608.20799#A4)for reference\.

Table 2:Combined IQM score at thelast checkpoint, computed from the per\-seedavg\_combinedmetric \(average over all ID and OOD contexts\)\. Values: IQM \[Q1, Q3\]\. Bold marks the best scheduler per environment\.Across all environments and observability modes, the best dynamic scheduler outperforms its static counterpart \(Table[2](https://arxiv.org/html/2608.20799#S4.T2)\)\. The gains are modest but consistent for CartPole \(\+39\+39IQM observed,\+53\+53blind\), where the ceiling effect of the task limits headroom\. Walker shows the most dramatic improvement under the observed condition \(\+120\+120IQM\)\. The static observed baseline collapses to an IQM of only31\.031\.0, likely due to overfitting to the three fixed training contexts when the context value is directly visible, while dynamic schedules force broader generalization\. CarRacing presents the sharpest contrast between observability modes as the best blind dynamic scheduler reaches592\.3592\.3\(\+489\+489IQM over static blind\), while both observed conditions result in negative IQM, representing failed policies\. We report the lateral axis \(COM\_Y\) as the primary CarRacing result, as it was the main focus of our scheduler search; longitudinal \(COM\_X\) results are provided in Table[17](https://arxiv.org/html/2608.20799#A6.T17)\. The best\-anytime results in Appendix[D\.2](https://arxiv.org/html/2608.20799#A4.SS2)show that observed policies do learn during training but ultimately degrade, suggesting late\-stage instability rather than a fundamental inability to train on this environment\. This reversal, where blind outperforms observed by a wide margin in CarRacing but not in simpler environments, is consistent with the context\-observability analysis in Appendix[D\.1](https://arxiv.org/html/2608.20799#A4.SS1)\.

Figure[2](https://arxiv.org/html/2608.20799#S4.F2)shows per\-context IQM episode return for CartPole across all 16 evaluation contexts, separated out into OOD\-low, ID, and OOD\-high regions\. The regional breakdown reveals that aggregate IQM improvements can mask meaningfully different dynamics across regimes\. In the ID and OOD\-low regions, dynamic and static schedulers perform comparably, with dynamic schedulers providing modest but consistent gains\. The OOD\-high region tells a different story\. Static baselines tend to exhibit a characteristic generalization drop around the midpoint of training \(see Figure[37](https://arxiv.org/html/2608.20799#A7.F37)in the Appendix\), where OOD\-high performance peaks early before degrading and struggles to recover as the policy specializes to the training distribution\. Dynamic schedulers substantially limit this effect\. Continuously varying the context during training prevents a policy from overfitting to a fixed regime, so several families either never exhibit the drop or recover more quickly\. Looking at only the best\-anytime scores would obscure the degradation that static training accumulates over time\. Analogous regional figures for Walker and CarRacing appear in Appendix[D](https://arxiv.org/html/2608.20799#A4); these show the same behaviour to be much less pronounced\. Results per scheduler families are detailed in Appendix[F](https://arxiv.org/html/2608.20799#A6)\.

### 4\.3Multi\-Stage Schedulers

We use Optuna\([2](https://arxiv.org/html/2608.20799#bib.bib13)\)with a TPE sampler to automatically search for effective multi\-stage schedule curricula on CartPole and BipedalWalker\. Each trial independently selects a scheduler mode from\{constant,sinusoidal,cosine annealing\}\\\{\\text\{constant\},\\,\\text\{sinusoidal\},\\,\\text\{cosine annealing\}\\\}and its associated parameters for each stage\. The trial objective is the mean across seeds of each seed’s best combined evaluation score observed at any stage boundary during training\. For CartPole we search overK=4K=4stages \(30 k / 30 k / 45 k / 45 k steps, 150 k total\) on the pool\-7 distribution across 240 trials with 4 seeds each; the stage lengths were chosen to align with the OOD\-high generalization loss pattern observed in the single\-stage experiments, where performance peaks early and degrades thereafter\. For BipedalWalker we useK=3K=3equal stages \(500 k each, 1\.5 M total\) on the pool\-3 distribution across 120 trials with 3 seeds each; the 1\.5 M budget is half the full single\-stage run to keep search tractable, andK=3K=3equal stages was a pragmatic choice to keep the search space manageable\.

The top\-20 CartPole configurations are retrained with 10 seeds to obtain reliable IQM estimates\. For BipedalWalker, 16 top configurations that clearly outperformed the static baselines during the 3\-seed search are retrained with 8 seeds each\. All comparisons are at the last training checkpoint; single\-scheduler baselines are evaluated at the same training budget to ensure a fair comparison\.

Table 3:Multi\-stage Optuna retrain results at thelast checkpoint\. Walker values are evaluated at the matched 1\.5 M\-step budget \(half the full single\-scheduler training run\); single\-scheduler baselines are re\-evaluated at the same budget for a fair comparison\. Values: IQM \[Q1, Q3\]\. Top\-3 retrained trials shown per environment\.Table[3](https://arxiv.org/html/2608.20799#S4.T3)summarises the last\-checkpoint IQM for the top\-3 retrained trials alongside the static and best single\-stage baselines; best\-anytime results are in Appendix[E\.1](https://arxiv.org/html/2608.20799#A5.SS1)\. On both environments the top retrained configurations largely outperform the static baseline\. However, the best multi\-stage trials do not consistently exceed the best single\-stage dynamic schedulers found by the grid search over the full set of schedule families\.

This comparison comes with important caveats: the Optuna search is restricted to two active scheduler families \(sinusoidal and cosine annealing\) and the context\-observed mode only, whereas the single\-stage grid search covers the full diversity of families and both modes\. The 3–4 seeds used to score each trial during search are insufficient to reliably rank configurations: after retraining with more seeds, rankings shift substantially and some configurations that appeared promising fall below the static baseline \(Figures[28](https://arxiv.org/html/2608.20799#A5.F28)and[31](https://arxiv.org/html/2608.20799#A5.F31)\)\. Given these constraints, we cannot conclude whether the multi\-stage structure itself is beneficial beyond what a single well\-chosen stage already provides\.

Nevertheless, inspecting the top CartPole configurations \(Figure[3\(a\)](https://arxiv.org/html/2608.20799#S4.F3.sf1)\) reveals a consistent structural tendency: the first stage is often idle while an active stage follows later, suggesting a*warm\-up then diversify*pattern\. Sinusoidal schedules dominate the active stages, and the number of active stages is essentially uncorrelated with final performance, indicating that a single well\-placed active stage captures most of the benefit\. Equivalent plots for BipedalWalker are provided in Appendix[E\.3](https://arxiv.org/html/2608.20799#A5.SS3)\(Figures[29\(a\)](https://arxiv.org/html/2608.20799#A5.F29.sf1)and[29\(b\)](https://arxiv.org/html/2608.20799#A5.F29.sf2)\)\.

![Refer to caption](https://arxiv.org/html/2608.20799v1/C1_scheduler_heatmap.png)\(a\)Scheduler mode per stage for the top\-5 retrained CartPole trials, sorted by retrain IQM\. Grey: constant; blue: sinusoidal; orange: cosine annealing\.
\(b\)Final\-checkpoint IQM±\\pmIQR for the top\-5 retrained CartPole trials \(10 seeds\)\. The dashed line marks the static pool\-7 baseline IQM\.
Figure 3:CartPole Optuna multi\-stage retrain results\.The total number of Optuna trials is comparable to the size of the single\-stage grid search over schedule families and their hyperparameters, so both searches operate on a roughly equal compute budget\. For BipedalWalker, the per\-trial training cost further restricted the search to a subset of the context range and a training run of 1\.5 M steps rather than the full 3 M used in the single\-stage ablation, which additionally limits what can be concluded from those results specifically\.

### 4\.4State Space Coverage

A natural hypothesis is that dynamic schedulers improve OOD robustness by steering the agent through a broader region of the combined state–context space during training\. We test this with a discretised coverage diagnostic on both CartPole and BipedalWalker\.

For CartPole, we run a dedicated coverage diagnostic, pooling pool\-3 and pool\-7 conditions across 20 seeds each, and discretise the four observation dimensions into1212bins each \(124=20,73612^\{4\}=20\{,\}736hypercells\), measuring the fraction of cells visited over150150k training steps\. Figure[4](https://arxiv.org/html/2608.20799#S4.F4)shows final IQM score and 4D coverage per scheduler\. While evaluation scores vary visibly across conditions, with Lévy walk and sudden jump outperforming the static baseline, coverage does not follow the same pattern: all conditions cluster between6\.54%6\.54\\%and6\.83%6\.83\\%\. A one\-way Analysis of Variance \(ANOVA\) calculated via SciPy\([59](https://arxiv.org/html/2608.20799#bib.bib40)\)confirmed that these coverage differences are not statistically significant \(p\>0\.05p\>0\.05\)\. This holds even for the parallel condition that doubles the effective sampling rate, ruling out that raw state throughput is the missing ingredient \(see Appendix[D\.3\.4](https://arxiv.org/html/2608.20799#A4.SS3.SSS4)for the full parallel vs\. sequential analysis\)\. We repeat the analysis on BipedalWalker using four independent projections of the 24\-dimensional observation space \(joint angles, posture, and per\-leg phase portraits\) and find the same null result across all projections \(ANOVAp\>0\.05p\>0\.05in every case; Appendix[D\.3\.5](https://arxiv.org/html/2608.20799#A4.SS3.SSS5)\)\.

Taken together, these results suggest that dynamic scheduling improves robustness by*restructuring*the training signal within the visited state space, i\.e\., exposing the policy to richer temporal sequences of\(st,ct\)\(s\_\{t\},c\_\{t\}\)pairs within each episode rather than by expanding the set of states visited\. The pool\-size scaling study in Appendix[D\.3\.3](https://arxiv.org/html/2608.20799#A4.SS3.SSS3)further shows that coverage is flat regardless of how many training contexts are used, and that virtually all cells are discovered within the first 50 k training steps for any scheduler, after which the policy converges to a narrow behavioral manifold\.

Figure 4:IQM score \(left\) and 4D state\-space coverage \(right\) per scheduler for CartPole pole\-length at pool sizes 3 and 7 \(20 seeds each\)\. Points show individual seeds; bars show mean±\\pmSEM\. ANOVA on coverage:p\>0\.05p\>0\.05\.
### 4\.5Experiments Summary

Across CartPole, BipedalWalker, and CarRacing, dynamic context schedules consistently match or outperform static context baselines, with the largest gains on BipedalWalker \(observed\) and CarRacing \(blind\)\. Multi\-stage curriculum search via Optuna reliably improves over the static baseline but does not consistently exceed the best single\-stage scheduler found by grid search\. State\-space coverage analysis on CartPole and BipedalWalker shows that dynamic and static schedulers visit indistinguishable fractions of the observation space; the benefit of dynamic scheduling therefore derives from the temporal structure of the training signal rather than from broader state exploration\.

## 5Discussion and Conclusion

We proposed dynamic context scheduling as a training instrument to improve zero\-shot generalization and out\-of\-distribution \(OOD\) robustness in reinforcement learning\. By continuously evolving physical parameters within training episodes, agents learn behaviors resilient to temporal shifts, consistently matching or outperforming static baselines across CartPole, BipedalWalker, and CarRacing\. Crucially, our coverage analysis revealed that these gains do not stem from broader state\-space exploration\. Instead, dynamic scheduling restructures the temporal sequence of the training signal, acting as a powerful regularizer that prevents convergence to narrow, over\-specialized behavioral manifolds\. We also found that context observability requires careful consideration: while explicit observation benefits simpler tasks, "blind" dynamic scheduling proves superior in complex environments like CarRacing\.

Our approach is currently limited by the introduction of schedule hyperparameters and the need for manually defined normalization bounds in observable modes\. While automated multi\-stage search via Optuna mitigates manual tuning, it remains computationally expensive\. Future work will explore integrating dynamic schedules with automated curriculum learning\([42](https://arxiv.org/html/2608.20799#bib.bib12)\)to adapt temporal structures to an agent’s real\-time progress\. Furthermore, rather than naïvely appending observable context to the state, we plan to investigate more advanced integration methods, such as injecting contextual information directly into the latent representations of world models\([43](https://arxiv.org/html/2608.20799#bib.bib51);[24](https://arxiv.org/html/2608.20799#bib.bib15), e\.g\.,\)\.

## Acknowledgments

The authors are funded by the Deutsche Forschungsgemeinschaft \(DFG, German Research Foundation\) – 572775489\. The authors acknowledge support by the state of Baden\-Württemberg through bwHPC and the German Research Foundation \(DFG\) through grant INST 35/1597\-1 FUGG\.

## References

- Agarwalet al\.\(2021\)R\. Agarwal, M\. Schwarzer, P\. S\. Castro, A\. C\. Courville, and M\. G\. BellemareDeep reinforcement learning at the edge of the statistical precipice\.CoRRabs/2108\.13264\.External Links:[Link](https://arxiv.org/abs/2108.13264),2108\.13264Cited by:[§4\.1](https://arxiv.org/html/2608.20799#S4.SS1.SSS0.Px2.p1.1)\.
- Akibaet al\.\(2019\)T\. Akiba, S\. Sano, T\. Yanase, T\. Ohta, and M\. KoyamaOptuna: A next\-generation hyperparameter optimization framework\.InProceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, \(KDD 2019\),A\. Teredesai, V\. Kumar, Y\. Li, R\. Rosales, E\. Terzi, and G\. Karypis \(Eds\.\),pp\. 2623–2631\.Cited by:[§4\.3](https://arxiv.org/html/2608.20799#S4.SS3.p1.1),[§4](https://arxiv.org/html/2608.20799#S4.p1.1)\.
- Becket al\.\(2025\)J\. Beck, R\. Vuorio, E\. Z\. Liu, Z\. Xiong, L\. M\. Zintgraf, C\. Finn, and S\. WhitesonA tutorial on meta\-reinforcement learning\.Found\. Trends Mach\. Learn\.18\(2\-3\),pp\. 224–384\.External Links:[Document](https://dx.doi.org/10.1561/2200000080)Cited by:[§1](https://arxiv.org/html/2608.20799#S1.p2.1)\.
- Bellman \(1957\)R\. BellmanA markovian decision process\.Journal of Mathematics and Mechanics,pp\. 679–684\.Cited by:[§2](https://arxiv.org/html/2608.20799#S2.p1.1)\.
- Benadet al\.\(2025\)J\. Benad, F\. Röder, M\. V\. Butz, and M\. EppeShared dynamic model aligned hypernetworks for contextual reinforcement learning\.InEighteenth European Workshop on Reinforcement Learning,External Links:[Link](https://openreview.net/forum?id=6gdvQqkFKT)Cited by:[§1](https://arxiv.org/html/2608.20799#S1.p3.1),[§2](https://arxiv.org/html/2608.20799#S2.p2.1)\.
- Benjaminset al\.\(2023\)C\. Benjamins, T\. Eimer, F\. Schubert, A\. Mohan, S\. Döhler, A\. Biedenkapp, B\. Rosenhan, F\. Hutter, and M\. LindauerContextualize me – the case for context in reinforcement learning\.Transactions on Machine Learning Research\.Cited by:[item 3](https://arxiv.org/html/2608.20799#S1.I2.i3.p1.1),[§1](https://arxiv.org/html/2608.20799#S1.p1.1),[§1](https://arxiv.org/html/2608.20799#S1.p3.1),[§2](https://arxiv.org/html/2608.20799#S2.p1.1),[§2](https://arxiv.org/html/2608.20799#S2.p2.1),[§3](https://arxiv.org/html/2608.20799#S3.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2608.20799#S4.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2608.20799#S4.SS1.SSS0.Px2.p1.1)\.
- \[7\]M\. Beukman, D\. Jarvis, R\. Klein, S\. James, and B\. RosmanDynamics generalisation in reinforcement learning via adaptive context\-aware policies\.See[37](https://arxiv.org/html/2608.20799#bib.bib66),Cited by:[§1](https://arxiv.org/html/2608.20799#S1.p3.1),[§2](https://arxiv.org/html/2608.20799#S2.p2.1)\.
- Biedenkapp \(2026\)A\. BiedenkappContextual intelligence: the next leap for reinforcement learning: blue sky ideas track\.InProceedings of the 25th International Conference on Autonomous Agents and Multiagent Systems \(AAMAS 2026\),External Links:[Document](https://dx.doi.org/10.65109/QNKH4630),[Link](https://doi.org/10.65109/QNKH4630)Cited by:[§1](https://arxiv.org/html/2608.20799#S1.p4.1),[§2](https://arxiv.org/html/2608.20799#S2.p1.1),[§2](https://arxiv.org/html/2608.20799#S2.p3.1)\.
- Brockmanet al\.\(2016\)G\. Brockman, V\. Cheung, L\. Pettersson, J\. Schneider, J\. Schulman, J\. Tang, and W\. ZarembaOpenAI gym\.arXiv:1606\.01540 \[cs\.LG\]\.Cited by:[§2](https://arxiv.org/html/2608.20799#S2.p2.1)\.
- \[10\]Y\. Chandak, G\. Theocharous, S\. Shankar, M\. White, S\. Mahadevan, and P\. S\. ThomasOptimizing for the future in non\-stationary MDPs\.See[14](https://arxiv.org/html/2608.20799#bib.bib61),pp\. 1414–1425\.Cited by:[§2](https://arxiv.org/html/2608.20799#S2.p3.1)\.
- K\. Chaudhuri, S\. Jegelka, L\. Song, C\. Szepesvári, G\. Niu, and S\. Sabato \(Eds\.\) \(2022\)K\. Chaudhuri, S\. Jegelka, L\. Song, C\. Szepesvári, G\. Niu, and S\. Sabato \(Eds\.\)Proceedings of the 39th international conference on machine learning \(ICML’22\)\.Proceedings of Machine Learning Research, Vol\.162,PMLR\.Cited by:[34](https://arxiv.org/html/2608.20799#bib.bib25)\.
- K\. Chaudhuri and R\. Salakhutdinov \(Eds\.\) \(2019\)K\. Chaudhuri and R\. Salakhutdinov \(Eds\.\)Proceedings of the 36th international conference on machine learning \(ICML’19\)\.Vol\.97,Proceedings of Machine Learning Research\.Cited by:[51](https://arxiv.org/html/2608.20799#bib.bib10)\.
- \[13\]K\. Cobbe, C\. Hesse, J\. Hilton, and J\. SchulmanLeveraging procedural generation to benchmark reinforcement learning\.See[14](https://arxiv.org/html/2608.20799#bib.bib61),Cited by:[§1](https://arxiv.org/html/2608.20799#S1.p1.1),[§1](https://arxiv.org/html/2608.20799#S1.p2.1)\.
- H\. Daume III and A\. Singh \(Eds\.\) \(2020\)H\. Daume III and A\. Singh \(Eds\.\)Proceedings of the 37th international conference on machine learning \(ICML’20\)\.Vol\.98,Proceedings of Machine Learning Research\.Cited by:[10](https://arxiv.org/html/2608.20799#bib.bib2),[13](https://arxiv.org/html/2608.20799#bib.bib43)\.
- Dinget al\.\(2020\)C\. Ding, L\. Zhou, Y\. Li, and X\. RongLocomotion control of quadruped robots with online center of mass adaptation and payload identification\.IEEE Access8\(\),pp\. 224578–224587\.External Links:[Document](https://dx.doi.org/10.1109/ACCESS.2020.3044933)Cited by:[§1](https://arxiv.org/html/2608.20799#S1.p1.1)\.
- Duanet al\.\(2016\)Y\. Duan, J\. Schulman, X\. Chen, P\. L\. Bartlett, I\. Sutskever, and P\. AbbeelRL2\{\}^\{2\}: fast reinforcement learning via slow reinforcement learning\.arXiv:1611\.02779 \[cs\.AI\]\.Cited by:[§1](https://arxiv.org/html/2608.20799#S1.p2.1)\.
- Eimeret al\.\(2021\)T\. Eimer, C\. Benjamins, and M\. LindauerHyperparameters in contextual rl are highly situational\.InEcological Theory of RL Workshop NeurIPS,Cited by:[§2](https://arxiv.org/html/2608.20799#S2.p2.1)\.
- Engwegenet al\.\(2025\)L\. Engwegen, D\. Brinks, and W\. BoehmerModular recurrence in contextual MDPs for universal morphology control\.InEighteenth European Workshop on Reinforcement Learning,External Links:[Link](https://openreview.net/forum?id=0fn0ii1njp)Cited by:[§1](https://arxiv.org/html/2608.20799#S1.p3.1),[§2](https://arxiv.org/html/2608.20799#S2.p2.1)\.
- Evanset al\.\(2022\)B\. Evans, A\. Thankaraj, and L\. PintoContext is everything: implicit identification for dynamics adaptation\.InProceedings of the International Conference on Robotics and Automation, \(ICRA 2022\),pp\. 2642–2648\.External Links:[Document](https://dx.doi.org/10.1109/ICRA46639.2022.9812119)Cited by:[§1](https://arxiv.org/html/2608.20799#S1.p2.1)\.
- \[20\]C\. Finn, P\. Abbeel, and S\. LevineModel\-agnostic meta\-learning for fast adaptation of deep networks\.See[44](https://arxiv.org/html/2608.20799#bib.bib59),pp\. 1126–1135\.Cited by:[§1](https://arxiv.org/html/2608.20799#S1.p2.1)\.
- Freemanet al\.\(2021\)C\. D\. Freeman, E\. Frey, A\. Raichuk, S\. Girgin, I\. Mordatch, and O\. BachemBrax \- A differentiable physics engine for large scale rigid body simulation\.InProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks \(NeurIPS DBT’21\),J\. Vanschoren and S\.\-K\. Yeung \(Eds\.\),Cited by:[§2](https://arxiv.org/html/2608.20799#S2.p2.1)\.
- Grigsbyet al\.\(2024\)J\. Grigsby, L\. Fan, and Y\. ZhuAMAGO: scalable in\-context reinforcement learning for adaptive agents\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=M6XWoEdmwf)Cited by:[§1](https://arxiv.org/html/2608.20799#S1.p2.1)\.
- Grootenet al\.\(2026\)B\. Grooten, P\. MacAlpine, K\. Subramanian, P\. R\. Wurman, and P\. StoneOut\-of\-distribution generalization with a sparc: racing 100 unseen vehicles with a single policy\.InProceedings of the Fourtieth AAAI Conference on Artificial Intelligence,Cited by:[§1](https://arxiv.org/html/2608.20799#S1.p2.1)\.
- Gumbschet al\.\(2024\)C\. Gumbsch, N\. Sajid, G\. Martius, and M\. V\. ButzLearning hierarchical world models with adaptive temporal abstractions from discrete latent dynamics\.See[48](https://arxiv.org/html/2608.20799#bib.bib57),External Links:[Link](https://openreview.net/forum?id=TjCDNssXKU)Cited by:[§1](https://arxiv.org/html/2608.20799#S1.p1.1),[§2](https://arxiv.org/html/2608.20799#S2.p2.1),[§2](https://arxiv.org/html/2608.20799#S2.p3.1),[§5](https://arxiv.org/html/2608.20799#S5.p2.1)\.
- Hallaket al\.\(2015\)A\. Hallak, D\. D\. Castro, and S\. MannorContextual markov decision processes\.arXiv:1502\.02259 \[stat\.ML\]\.Cited by:[§1](https://arxiv.org/html/2608.20799#S1.p3.1),[§2](https://arxiv.org/html/2608.20799#S2.p1.1)\.
- Iannottaet al\.\(2025\)M\. Iannotta, Y\. Yang, J\. A\. Stork, E\. Schaffernicht, and T\. StoyanovCan context bridge the reality gap? sim\-to\-real transfer of context\-aware policies\.arXiv:2511\.04249 \[cs\.RO\]\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2511.04249)Cited by:[§1](https://arxiv.org/html/2608.20799#S1.p1.1)\.
- Kirket al\.\(2023\)R\. Kirk, A\. Zhang, E\. Grefenstette, and T\. RocktäschelA survey of zero\-shot generalisation in deep reinforcement learning\.Journal of Artificial Intelligence Research \(JAIR\)76,pp\. 201–264\.Cited by:[§1](https://arxiv.org/html/2608.20799#S1.p1.1),[§2](https://arxiv.org/html/2608.20799#S2.p1.1),[§2](https://arxiv.org/html/2608.20799#S2.p2.1),[§4\.1](https://arxiv.org/html/2608.20799#S4.SS1.SSS0.Px2.p1.1)\.
- \[28\]P\. Klink, C\. D’Eramo, J\. Peters, and J\. PajarinenSelf\-paced deep reinforcement learning\.See[32](https://arxiv.org/html/2608.20799#bib.bib64),pp\. 9216–9227\.Cited by:[§1](https://arxiv.org/html/2608.20799#S1.p2.1)\.
- S\. Koyejo, S\. Mohamed, A\. Agarwal, D\. Belgrave, K\. Cho, and A\. Oh \(Eds\.\) \(2022\)S\. Koyejo, S\. Mohamed, A\. Agarwal, D\. Belgrave, K\. Cho, and A\. Oh \(Eds\.\)Proceedings of the 35th international conference on advances in neural information processing systems \(NeurIPS’22\)\.Curran Associates\.Cited by:[40](https://arxiv.org/html/2608.20799#bib.bib20)\.
- A\. Krause, E\. Brunskill, K\. Cho, B\. Engelhardt, S\. Sabato, and J\. Scarlett \(Eds\.\) \(2023\)A\. Krause, E\. Brunskill, K\. Cho, B\. Engelhardt, S\. Sabato, and J\. Scarlett \(Eds\.\)Proceedings of the 40th international conference on machine learning \(ICML’23\)\.Proceedings of Machine Learning Research, Vol\.202,PMLR\.Cited by:[57](https://arxiv.org/html/2608.20799#bib.bib1)\.
- Kumaret al\.\(2021\)A\. Kumar, Z\. Fu, D\. Pathak, and J\. MalikRMA: rapid motor adaptation for legged robots\.InRobotics: Science and Systems XVII, \(RSS 2021\),D\. A\. Shell, M\. Toussaint, and M\. A\. Hsieh \(Eds\.\),Cited by:[§2](https://arxiv.org/html/2608.20799#S2.p3.1)\.
- H\. Larochelle, M\. Ranzato, R\. Hadsell, M\.\-F\. Balcan, and H\. Lin \(Eds\.\) \(2020\)H\. Larochelle, M\. Ranzato, R\. Hadsell, M\.\-F\. Balcan, and H\. Lin \(Eds\.\)Proceedings of the 33rd international conference on advances in neural information processing systems \(NeurIPS’20\)\.Curran Associates\.Cited by:[28](https://arxiv.org/html/2608.20799#bib.bib47),[64](https://arxiv.org/html/2608.20799#bib.bib19)\.
- Loshchilov and Hutter \(2017\)I\. Loshchilov and F\. HutterSGDR: stochastic gradient descent with warm restarts\.External Links:1608\.03983,[Link](https://arxiv.org/abs/1608.03983)Cited by:[2nd item](https://arxiv.org/html/2608.20799#S3.I2.i2.p1.1)\.
- Melo \(2022\)L\. C\. MeloTransformers are meta\-reinforcement learners\.See[11](https://arxiv.org/html/2608.20799#bib.bib62),pp\. 15340–15359\.Cited by:[§1](https://arxiv.org/html/2608.20799#S1.p2.1)\.
- Modiet al\.\(2018\)A\. Modi, N\. Jiang, S\. Singh, and A\. TewariMarkov decision processes with continuous side information\.InAlgorithmic Learning Theory \(ALT’18\),Vol\.83,pp\. 597–618\.Cited by:[§1](https://arxiv.org/html/2608.20799#S1.p3.1)\.
- Ndiret al\.\(2024\)T\. C\. Ndir, A\. Biedenkapp, and N\. AwadInferring behavior\-specific context improves zero\-shot generalization in reinforcement learning\.InSeventeenth European Workshop on Reinforcement Learning,External Links:[Link](https://openreview.net/forum?id=51XSWH0mgN)Cited by:[§2](https://arxiv.org/html/2608.20799#S2.p3.1)\.
- A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\) \(2023\)A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\)Proceedings of the 36th international conference on advances in neural information processing systems \(NeurIPS’23\)\.Curran Associates\.Cited by:[7](https://arxiv.org/html/2608.20799#bib.bib30)\.
- OpenAIet al\.\(2019\)OpenAI, I\. Akkaya, M\. Andrychowicz, M\. Chociej, M\. Litwin, B\. McGrew, A\. Petron, A\. Paino, M\. Plappert, G\. Powell, R\. Ribas, J\. Schneider, N\. Tezak, J\. Tworek, P\. Welinder, L\. Weng, Q\. Yuan, W\. Zaremba, and L\. ZhangSolving rubik’s cube with a robot hand\.arXiv:1910\.07113 \[cs\.LG\]\.Cited by:[§1](https://arxiv.org/html/2608.20799#S1.p2.1)\.
- Packeret al\.\(2019\)C\. Packer, K\. Gao, J\. Kos, P\. Krähenbühl, V\. Koltun, and D\. SongAssessing generalization in deep reinforcement learning\.arXiv:1810\.12282 \[cs\.LG\]\.Cited by:[§1](https://arxiv.org/html/2608.20799#S1.p2.1)\.
- Panagantiet al\.\(2022\)K\. Panaganti, Z\. Xu, D\. Kalathil, and M\. GhavamzadehRobust reinforcement learning using offline data\.See[29](https://arxiv.org/html/2608.20799#bib.bib65),Cited by:[§1](https://arxiv.org/html/2608.20799#S1.p2.1)\.
- Pintoet al\.\(2017\)L\. Pinto, J\. Davidson, R\. Sukthankar, and A\. GuptaRobust adversarial reinforcement learning\.See[44](https://arxiv.org/html/2608.20799#bib.bib59),D\. Precup and Y\. W\. Teh \(Eds\.\),pp\. 2817–2826\.Cited by:[§1](https://arxiv.org/html/2608.20799#S1.p2.1)\.
- Portelaset al\.\(2019\)R\. Portelas, C\. Colas, K\. Hofmann, and P\.\-Y\. OudeyerTeacher algorithms for curriculum learning of deep RL in continuously parameterized environments\.InProceedings of the 3rd Annual Conference on Robot Learning, \(CoRL 2019\),L\. P\. Kaelbling, D\. Kragic, and K\. Sugiura \(Eds\.\),pp\. 835–853\.Cited by:[§5](https://arxiv.org/html/2608.20799#S5.p2.1)\.
- Prasannaet al\.\(2024\)S\. Prasanna, K\. Farid, R\. Rajan, and A\. BiedenkappDreaming of many worlds: learning contextual world models aids zero\-shot generalization\.Reinforcement Learning Journal1\.Cited by:[§1](https://arxiv.org/html/2608.20799#S1.p1.1),[§1](https://arxiv.org/html/2608.20799#S1.p3.1),[§2](https://arxiv.org/html/2608.20799#S2.p2.1),[§5](https://arxiv.org/html/2608.20799#S5.p2.1)\.
- D\. Precup and Y\. Teh \(Eds\.\) \(2017\)D\. Precup and Y\. Teh \(Eds\.\)Proceedings of the 34th international conference on machine learning \(ICML’17\)\.Vol\.70,Proceedings of Machine Learning Research\.Cited by:[20](https://arxiv.org/html/2608.20799#bib.bib44),[41](https://arxiv.org/html/2608.20799#bib.bib7)\.
- \[45\]\(2019\)Proceedings of the international conference on learning representations \(ICLR’19\)\.ICLR\.Cited by:[66](https://arxiv.org/html/2608.20799#bib.bib8)\.
- \[46\]\(2021\)Proceedings of the international conference on learning representations \(ICLR’21\)\.ICLR\.Cited by:[63](https://arxiv.org/html/2608.20799#bib.bib53)\.
- \[47\]\(2023\)Proceedings of the international conference on learning representations \(ICLR’23\)\.ICLR\.Cited by:[52](https://arxiv.org/html/2608.20799#bib.bib3)\.
- \[48\]\(2024\)Proceedings of the international conference on learning representations \(ICLR’24\)\.ICLR\.Cited by:[24](https://arxiv.org/html/2608.20799#bib.bib15)\.
- \[49\]\(2025\)Proceedings of the international conference on learning representations \(ICLR’25\)\.ICLR\.Cited by:[54](https://arxiv.org/html/2608.20799#bib.bib22)\.
- Raffinet al\.\(2021\)A\. Raffin, A\. Hill, A\. Gleave, A\. Kanervisto, M\. Ernestus, and N\. DormannStable\-baselines3: reliable reinforcement learning implementations\.J\. Mach\. Learn\. Res\.22,pp\. 268:1–268:8\.Cited by:[Appendix B](https://arxiv.org/html/2608.20799#A2.p1.1),[§4\.1](https://arxiv.org/html/2608.20799#S4.SS1.SSS0.Px2.p1.1)\.
- \[51\]K\. Rakelly, A\. Zhou, C\. Finn, S\. Levine, and D\. QuillenEfficient off\-policy meta\-reinforcement learning via probabilistic context variables\.See[12](https://arxiv.org/html/2608.20799#bib.bib60),pp\. 5331–5340\.Cited by:[§1](https://arxiv.org/html/2608.20799#S1.p2.1)\.
- \[52\]T\. Ren, C\. Xiao, T\. Zhang, N\. Li, Z\. Wang, S\. Sanghavi, D\. Schuurmans, and B\. DaiLatent variable representation for reinforcement learning\.See[47](https://arxiv.org/html/2608.20799#bib.bib56),Cited by:[§2](https://arxiv.org/html/2608.20799#S2.p3.1)\.
- Schulmanet al\.\(2017\)J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. KlimovProximal policy optimization algorithms\.arXiv:1707\.06347 \[cs\.LG\]\.Cited by:[Appendix B](https://arxiv.org/html/2608.20799#A2.p1.1),[§4\.1](https://arxiv.org/html/2608.20799#S4.SS1.SSS0.Px2.p1.1)\.
- Shalaet al\.\(2025\)G\. Shala, A\. Biedenkapp, P\. Krack, F\. Walter, and J\. GrabockaEfficient cross\-episode meta\-rl\.See[49](https://arxiv.org/html/2608.20799#bib.bib58),Note:Published online:[iclr\.cc](https://iclr.cc/)Cited by:[§1](https://arxiv.org/html/2608.20799#S1.p2.1)\.
- Suauet al\.\(2024\)M\. Suau, M\. T\. J\. Spaan, and F\. A\. OliehoekBad habits: policy confounding and out\-of\-trajectory generalization in RL\.RLJ4,pp\. 1711–1732\.Cited by:[§1](https://arxiv.org/html/2608.20799#S1.p1.1)\.
- Tassaet al\.\(2018\)Y\. Tassa, Y\. Doron, A\. Muldal, T\. Erez, Y\. Li, D\. de Las Casas, D\. Budden, A\. Abdolmaleki, J\. Merel, A\. Lefrancq, T\. P\. Lillicrap, and M\. A\. RiedmillerDeepMind control suite\.arXiv:1801\.00690 \[cs\.AI\]\.Cited by:[§2](https://arxiv.org/html/2608.20799#S2.p2.1)\.
- Tennenholtzet al\.\(2023\)G\. Tennenholtz, N\. Merlis, L\. Shani, M\. Mladenov, and C\. BoutilierReinforcement learning with history dependent dynamic contexts\.See[30](https://arxiv.org/html/2608.20799#bib.bib63),pp\. 34011–34053\.Cited by:[§1](https://arxiv.org/html/2608.20799#S1.p5.2),[§2](https://arxiv.org/html/2608.20799#S2.p1.1)\.
- Tobinet al\.\(2017\)J\. Tobin, R\. Fong, A\. Ray, J\. Schneider, W\. Zaremba, and P\. AbbeelDomain randomization for transferring deep neural networks from simulation to the real world\.InProceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems \(IROS’17\),pp\. 23–30\.Cited by:[§1](https://arxiv.org/html/2608.20799#S1.p1.1),[§1](https://arxiv.org/html/2608.20799#S1.p2.1)\.
- Virtanenet al\.\(2020\)P\. Virtanen, R\. Gommers, T\. E\. Oliphant, M\. Haberland, T\. Reddy, D\. Cournapeau, E\. Burovski, P\. Peterson, W\. Weckesser, J\. Bright, S\. J\. van der Walt, M\. Brett, J\. Wilson, K\. J\. Millman, N\. Mayorov, A\. R\. J\. Nelson, E\. Jones, R\. Kern, E\. Larson, C\. J\. Carey, İ\. Polat, Y\. Feng, E\. W\. Moore, J\. VanderPlas, D\. Laxalde, J\. Perktold, R\. Cimrman, I\. Henriksen, E\. A\. Quintero, C\. R\. Harris, A\. M\. Archibald, A\. H\. Ribeiro, F\. Pedregosa, P\. van Mulbregt, A\. Vijaykumar, A\. P\. Bardelli, A\. Rothberg, A\. Hilboll, A\. Kloeckner, A\. Scopatz, A\. Lee, A\. Rokem, C\. N\. Woods, C\. Fulton, C\. Masson, C\. Häggström, C\. Fitzgerald, D\. A\. Nicholson, D\. R\. Hagen, D\. V\. Pasechnik, E\. Olivetti, E\. Martin, E\. Wieser, F\. Silva, F\. Lenders, F\. Wilhelm, G\. Young, G\. A\. Price, G\. Ingold, G\. E\. Allen, G\. R\. Lee, H\. Audren, I\. Probst, J\. P\. Dietrich, J\. Silterra, J\. T\. Webber, J\. Slavič, J\. Nothman, J\. Buchner, J\. Kulick, J\. L\. Schönberger, J\. V\. de Miranda Cardoso, J\. Reimer, J\. Harrington, J\. L\. C\. Rodríguez, J\. Nunez\-Iglesias, J\. Kuczynski, K\. Tritz, M\. Thoma, M\. Newville, M\. Kümmerer, M\. Bolingbroke, M\. Tartre, M\. Pak, N\. J\. Smith, N\. Nowaczyk, N\. Shebanov, O\. Pavlyk, P\. A\. Brodtkorb, P\. Lee, R\. T\. McGibbon, R\. Feldbauer, S\. Lewis, S\. Tygier, S\. Sievert, S\. Vigna, S\. Peterson, S\. More, T\. Pudlik, T\. Oshima, T\. J\. Pingel, T\. P\. Robitaille, T\. Spura, T\. R\. Jones, T\. Cera, T\. Leslie, T\. Zito, T\. Krauss, U\. Upadhyay, Y\. O\. Halchenko, and Y\. Vázquez\-BaezaSciPy 1\.0: fundamental algorithms for scientific computing in python\.Nature Methods17\(3\),pp\. 261–272\.External Links:ISSN 1548\-7105,[Link](http://dx.doi.org/10.1038/s41592-019-0686-2),[Document](https://dx.doi.org/10.1038/s41592-019-0686-2)Cited by:[§4\.4](https://arxiv.org/html/2608.20799#S4.SS4.p2.1)\.
- Wanget al\.\(2021\)J\. Wang, M\. King, N\. Porcel, Z\. Kurth\-Nelson, T\. Zhu, C\. Deck, P\. Choy, M\. Cassin, M\. Reynolds, H\. F\. Song, G\. Buttimore, D\. P\. Reichert, N\. C\. Rabinowitz, L\. Matthey, D\. Hassabis, A\. Lerchner, and M\. M\. BotvinickAlchemy: A benchmark and analysis toolkit for meta\-reinforcement learning agents\.InProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, \(NeurIPS DBT 2021\),Cited by:[§1](https://arxiv.org/html/2608.20799#S1.p1.1)\.
- Yuet al\.\(2017\)W\. Yu, J\. Tan, C\. K\. Liu, and G\. TurkPreparing for the unknown: learning a universal policy with online system identification\.InRobotics: Science and Systems XIII, \(RSS 2017\),External Links:[Document](https://dx.doi.org/10.15607/RSS.2017.XIII.048)Cited by:[§1](https://arxiv.org/html/2608.20799#S1.p2.1)\.
- Zaburdaevet al\.\(2015\)V\. Zaburdaev, S\. Denisov, and J\. KlafterLévy walks\.Reviews of Modern Physics87\(2\),pp\. 483–530\.External Links:ISSN 1539\-0756,[Link](http://dx.doi.org/10.1103/RevModPhys.87.483),[Document](https://dx.doi.org/10.1103/revmodphys.87.483)Cited by:[§3](https://arxiv.org/html/2608.20799#S3.SS0.SSS0.Px2.p3.1)\.
- \[63\]A\. Zhang, S\. Sodhani, K\. Khetarpal, and J\. PineauLearning robust state abstractions for hidden\-parameter block MDPs\.See[46](https://arxiv.org/html/2608.20799#bib.bib55),Cited by:[§1](https://arxiv.org/html/2608.20799#S1.p1.1)\.
- Zhanget al\.\(2020\)H\. Zhang, H\. Chen, C\. Xiao, B\. Li, M\. Liu, D\. S\. Boning, and C\.\-J\. HsiehRobust deep reinforcement learning against adversarial perturbations on state observations\.See[32](https://arxiv.org/html/2608.20799#bib.bib64),Cited by:[§1](https://arxiv.org/html/2608.20799#S1.p2.1)\.
- Zhouet al\.\(2026\)T\. Zhou, J\. Cho, and C\. WuStructure detection for contextual reinforcement learning\.InFortieth AAAI Conference on Artificial Intelligence, AAAI 2026,S\. Koenig, C\. Jenkins, and M\. E\. Taylor \(Eds\.\),pp\. 29009–29016\.Cited by:[§1](https://arxiv.org/html/2608.20799#S1.p3.1)\.
- Zhouet al\.\(2019\)W\. Zhou, L\. Pinto, and A\. GuptaEnvironment probing interaction policies\.See[45](https://arxiv.org/html/2608.20799#bib.bib54),Cited by:[§1](https://arxiv.org/html/2608.20799#S1.p1.1),[§1](https://arxiv.org/html/2608.20799#S1.p2.1)\.

## Appendix ASchedulers Details

Figure[5](https://arxiv.org/html/2608.20799#A1.F5)illustrates trajectory examples for all eleven schedule families over three episodes\.

Figure 5:Trajectory examples for all eleven schedule families, simulated over three episodes of 500 steps each \(episode boundaries marked by dashed vertical lines\)\. All values are normalised to\[−1,\+1\]\[\-1,\+1\]relative to the training context range so that qualitative structure can be compared across context variables with different physical scales\. Dotted horizontal lines mark the range boundaries \(±1\\pm 1\) and midpoint \(00\)\. Multiple curves per panel show how the qualitative behaviour changes with the key hyperparameters of each family \(amplitude, step size, drift rate, etc\.\)\.In addition to the six canonical families described in Section[3](https://arxiv.org/html/2608.20799#S3.SS0.SSS0.Px2), three further families appear in the figure above\.

Sinusoidal Jump\.Superimposes the sinusoidal oscillation \(Section[3](https://arxiv.org/html/2608.20799#S3.SS0.SSS0.Px2)\) onto an intermittently jumping baseline: at random timesteps drawn from a uniform interval range, the current context receives an additional signed displacement of magnitudeΔ∼Uniform⁡\(mlo,mhi\)\\Delta\\sim\\operatorname\{Uniform\}\(m\_\{\\text\{lo\}\},m\_\{\\text\{hi\}\}\)\. The result is a smooth oscillation punctuated by abrupt level shifts, combining the dense in\-episode sweep of the sinusoidal family with the distributional tail coverage of stochastic jumps\.

Ornstein\-Uhlenbeck \(OU\)\.d​ct=θ⁡\(μ−ct\)​d​t\+σ​d​Wtdc\_\{t\}=\\theta\(\\mu\-c\_\{t\}\)\\,dt\+\\sigma\\,dW\_\{t\}, discretised asct\+1=ct\+θ⁡\(μ−ct\)\+σ​ϵtc\_\{t\+1\}=c\_\{t\}\+\\theta\(\\mu\-c\_\{t\}\)\+\\sigma\\,\\epsilon\_\{t\},ϵt∼𝒩⁡\(0,1\)\\epsilon\_\{t\}\\sim\\mathcal\{N\}\(0,1\)\. The reversion targetμ\\muis the episode\-initial context;θ\\thetacontrols mean\-reversion speed andσ\\sigmathe noise level\. OU produces temporally correlated drift that stays statistically anchored to a reference value, unlike the unbounded random walk\.

Phased OU\.Extends OU by periodically resampling the reversion targetμ\\mufrom a discrete set of anchor values \(constructed identically to the context pool, Section[3](https://arxiv.org/html/2608.20799#S3.SS0.SSS0.Px3)\)\. The retarget interval is drawn uniformly from a specified range, producing a piece\-wise\-drifting trajectory where each phase smoothly relaxes toward a new anchor\.

Identity \(baseline\)\.Context is held constant at its episode\-initial value throughout the episode\. This is the degenerate special case of a dynamic schedule and serves as the within\-experiment reference against which all schedule\-induced variation is measured\.

## Appendix BHyperparameters and Training Details

All agents are trained with PPO\[[53](https://arxiv.org/html/2608.20799#bib.bib52)\]via Stable\-Baselines3\[[50](https://arxiv.org/html/2608.20799#bib.bib14)\]\. Table[4](https://arxiv.org/html/2608.20799#A2.T4)lists all hyperparameters; all values are the SB3 defaults and were not tuned — PPO is regarded as robust to hyperparameter variation and fixing the optimizer ensures that observed differences are attributable to the scheduling strategy rather than incidental tuning\. The policy is an MlpPolicy with two fully\-connected hidden layers of 64 units each for CartPole and BipedalWalker\. CarRacing uses a MultiInputPolicy with a shared CNN feature extractor \(default SB3 NatureCNN\) followed by a fully\-connected head; context is fed through a separate MLP branch and fused with the CNN output\. The main experiments use a single parallel training environment \(N\_TRAIN\_ENVS= 1\); a small auxiliary set of runs explored the effect of increasing the number of parallel environments within PPO but are not part of the main comparison\. Evaluation runs in four parallel workers for all conditions\.

Table 4:PPO hyperparameters across environments\.
## Appendix CEnvironment and Context Details

### CartPole

Table[5](https://arxiv.org/html/2608.20799#A3.T5)summarises the training and evaluation context grids\. The normalization bounds used for live context observation \(Section[3](https://arxiv.org/html/2608.20799#S3.SS0.SSS0.Px1)\) are manually specified conservative intervals chosen to cover a physically plausible operating range for each variable, in this case\(0\.05,2\.0\)\(0\.05,2\.0\)m for the pole length\. This normalization is motivated by PPO training stability; its effect is documented in Appendix[D\.3\.6](https://arxiv.org/html/2608.20799#A4.SS3.SSS6)\.

Table 5:CartPole context ranges and evaluation splits\.Table[6](https://arxiv.org/html/2608.20799#A3.T6)lists the CartPole observation space bounds used to set the bin boundaries in the 4D coverage diagnostic\.

Table 6:Observation Space Bounds of CartPoleThe pool\-7 used in the main CartPole experiments contains the 7 midpoint\-inserted values\{0\.35,0\.41,0\.48,0\.55,0\.62,0\.68,0\.75\}\\\{0\.35,\\,0\.41,\\,0\.48,\\,0\.55,\\,0\.62,\\,0\.68,\\,0\.75\\\}for pole length and\{5\.0,7\.5,10\.0,12\.5,15\.0\}\\\{5\.0,\\,7\.5,\\,10\.0,\\,12\.5,\\,15\.0\\\}for force magnitude and gravity \(pool\-5\)\.

### BipedalWalker

A rigid payload of mass2\.02\.0kg and radius0\.50\.5m is attached to the torso\. The context COM\_X shifts this payload along the body’s longitudinal axis\. Training range is\[−0\.6,\+0\.6\]\[\-0\.6,\\,\+0\.6\]m; pool\-3 contains\{−0\.6,0\.0,\+0\.6\}\\\{\-0\.6,\\,0\.0,\\,\+0\.6\\\}\. The normalization bounds are\(−2\.0,\+2\.0\)\(\-2\.0,\\,\+2\.0\)m, a conservatively chosen physical operating range for the payload offset\.

Table 7:BipedalWalker evaluation splits \(COM\_X, m\)\.Table[8](https://arxiv.org/html/2608.20799#A3.T8)lists the BipedalWalker observation space bounds used to define the coverage projections\.

Table 8:BipedalWalker Observation Space Bounds
### CarRacing

Two context axes: COM\_X and COM\_Y \(vehicle center\-of\-mass offsets\), each in\[−0\.8,\+0\.8\]\[\-0\.8,\\,\+0\.8\]m, with pool\-3 containing\{−0\.8,0\.0,\+0\.8\}\\\{\-0\.8,\\,0\.0,\\,\+0\.8\\\}per axis\. The normalization bounds are\(−2\.0,\+2\.0\)\(\-2\.0,\\,\+2\.0\)m per axis, conservatively covering the physical operating range\. The payload has mass1\.01\.0kg and radius0\.50\.5m\. Table[9](https://arxiv.org/html/2608.20799#A3.T9)summarises the evaluation splits; both axes use identical ranges\.

Table 9:CarRacing evaluation splits \(COM\_X and COM\_Y, m\)\. Both axes share the same ranges\.![Refer to caption](https://arxiv.org/html/2608.20799v1/walker_payload.png)\(a\)BipedalWalker with attached payload \(dark mass on torso\)\.
![Refer to caption](https://arxiv.org/html/2608.20799v1/racer_payload.png)\(b\)CarRacing with attached longitudinally offset payload \(cyan marker\)\.

Figure 6:Visualisation of the payload\-extended environments\. Both payloads are novel additions to theCarlbenchmark introduced in this work\. Shifting the payload position changes the center\-of\-mass of the agent, altering its dynamics in a physically grounded and continuously parameterisable way\.

## Appendix DAdditional Results

### D\.1Cross\-Environment Scheduler Analysis

We present three complementary views of the scheduler search results across all environments, aggregating over the full set of scheduler families evaluated\.

##### Fraction of schedulers outperforming the static baseline\.

Figure[7](https://arxiv.org/html/2608.20799#A4.F7)shows, for each environment and context\-observability mode, the fraction of scheduler families whose last\-checkpoint IQM exceeds the corresponding static baseline\. Context\-observed dynamic schedulers beat the static observed baseline in nearly all cases \(89–100% across environments\), confirming that any dynamic curriculum helps when the agent can observe the context\. The blind mode is more nuanced: in CartPole and Walker roughly 70–78% of blind schedulers improve over the static blind baseline, whereas in CarRacing the picture splits sharply — the longitudinal \(COM\_X\) static blind baseline is exceptionally strong, with only 3 out of 9 schedulers surpassing it, while the lateral \(COM\_Y\) static blind baseline is weak and all 9 schedulers beat it\.

Figure 7:Fraction of scheduler families \(last checkpoint IQM\) that outperform the static baseline per environment and mode\. Numbers above bars indicate the exact count over the total number of families evaluated\.
##### Cross\-environment performance ranking\.

Figure[8](https://arxiv.org/html/2608.20799#A4.F8)shows a heatmap of each scheduler family’s performance relative to the static baseline \(score=0=0\) and the best scheduler in that column \(score=1=1\), with families sorted by their mean score across all six columns\. No single family dominates across all environments and modes: Sinusoidal, Lévy Walk, and Sudden Jump rank most consistently above the static baseline on average \(Piecewise Constant ranks highest overall but was only evaluated on CartPole, leaving its Walker and CarRacing cells empty\), while Phased OU and Cosine Annealing are the weakest overall\. Dynamic schedulers reliably outperform the static baseline in the context\-observed columns \(zero red cells for CartPole and Walker observed; only one marginal exception for CarRacing observed\), and the CarRacing \(lat\.\) blind column is entirely green\. The most notable failures occur in the Walker blind column, where Cosine Annealing and Random Walk fall clearly below the static blind baseline, suggesting these families are particularly sensitive to the absence of context information in that environment\.

![Refer to caption](https://arxiv.org/html/2608.20799v1/cross_env_ranking.png)Figure 8:Heatmap of scheduler family performance relative to the static baseline\. Colour encodes the static\-anchored score: yellow \(=0=0\) matches the static baseline, green \(=1=1\) is the best scheduler in that column, red indicates worse than static\. Raw IQM values \(last checkpoint\) are shown in each cell\. Families are sorted by mean score across all columns \(right margin\)\.
##### Context\-observed vs\. context\-blind gap\.

Figure[9](https://arxiv.org/html/2608.20799#A4.F9)quantifies the blind−\-observed IQM gap per scheduler family, normalised by each environment’s own IQM range so that results are comparable across environments\. The pattern is consistent across nearly all scheduler families: CartPole benefits from observing the context \(negative gap, observed wins\), CarRacing benefits strongly from*not*observing it \(large positive gap, blind wins\), and Walker sits in between with mixed results depending on the family\. This environment\-level reversal holds independently of which scheduler is used, suggesting the effect is driven by the environment rather than the scheduler choice\. Notably, Cosine Annealing, Random Walk, and Sinusoidal are the only families that consistently prefer the observed mode in both CartPole and Walker, while Sudden Jump and Cosine Annealing show the largest blind advantage in CarRacing\.

Figure 9:Blind−\-observed IQM gap per scheduler family and environment, normalised by the environment’s IQM range \(max−\-min across all schedulers\)\. Positive values indicate blind schedulers outperform observed ones; negative values indicate the reverse\. Top panel: last checkpoint; bottom panel: best anytime\.

### D\.2Best\-Anytime Checkpoint Results for Scheduler Search

Table[10](https://arxiv.org/html/2608.20799#A4.T10)reports the best\-anytime IQM — the highestavg\_combinedscore achieved at any evaluation checkpoint during training — as a complement to the last\-checkpoint results in Table[2](https://arxiv.org/html/2608.20799#S4.T2)\. Dynamic schedulers improve over the static baseline across all environments and modes at peak performance, and the margins are generally larger than at the final checkpoint, indicating that the best policies are learned earlier in training and partially lost to instability or overfitting by the end\.

The gains are most pronounced for Walker, where the best observed scheduler reaches193\.0193\.0vs\. static observed134\.2134\.2\(\+59\+59IQM\), and for CarRacing, where the best blind scheduler reaches778\.9778\.9vs\. static blind617\.7617\.7\(\+161\+161IQM\)\. CartPole improvements are modest in comparison, consistent with the ceiling effect noted in the main text\. Notably, the best CarRacing observed scheduler \(634\.1634\.1\) now substantially exceeds the static observed baseline \(358\.5358\.5\) at peak, though it still falls short of the static blind baseline \(617\.7617\.7\), confirming that context observability remains a net negative in that environment even at best\-achieved performance\.

Table 10:Combined IQM score of thebest anytime checkpoint, computed from the per\-seedavg\_combinedmetric \(average over all ID and OOD contexts\)\. Values: IQM \[Q1, Q3\]\. Bold marks the best scheduler per environment\.
### D\.3CartPole

#### D\.3\.1Scheduler Search

Figure[10](https://arxiv.org/html/2608.20799#A4.F10)shows the full combined IQM training progression for all scheduler families on CartPole pole length \(pool\-7\), with each family represented by its best\-performing configuration selected by final checkpoint combined IQM\.

Figure 10:CartPole / pole length / pool\-7 — combined IQM training progression per scheduler family \(10 seeds\)\. Each line shows the best representative configuration for that family, selected by final combined IQM\. Shaded bands indicate Q1–Q3 across seeds\. The dashed line marks the static pool\-7 baseline\.
#### D\.3\.2Dynamic vs\. Static: Coverage and Score

To directly compare dynamic and static schedulers on coverage independently of pool size effects, we use all sequential tracking runs that include representative dynamic families: identity \(static baseline\), sinusoidal, Lévy walk, and sudden jump, across all available pool sizes \(10 seeds per condition\)\. This selection deliberately spans scheduler families with qualitatively different temporal structure — smooth oscillation, heavy\-tailed jumps, and abrupt discrete displacements — while avoiding any configuration tied to the scheduler search or the pool size study\.

The 4D coverage metric discretises CartPole’s four observation dimensions into1212bins each \(124=20,73612^\{4\}=20\{,\}736cells\), with boundaries set to physically plausible ranges: cart position\[−2\.5,2\.5\]\[\-2\.5,2\.5\]m, cart velocity\[−3\.5,3\.5\]\[\-3\.5,3\.5\]m/s, pole angle\[−0\.30,0\.30\]\[\-0\.30,0\.30\]rad, and pole angular velocity\[−4\.0,4\.0\]\[\-4\.0,4\.0\]rad/s\.

Figure[11](https://arxiv.org/html/2608.20799#A4.F11)plots coverage against final eval score for each seed\. No scheduler family occupies a systematically higher or lower region of the coverage axis: dynamic and static seeds are interleaved throughout\. The Pearson correlation isr=0\.21r=0\.21\(p=0\.003p=0\.003\), indicating a negligible linear relationship between coverage and score — knowing how much of the state space a scheduler visits tells you almost nothing about how well it generalises\.

Figure 11:4D state\-space coverage vs\. final eval score across all sequential seeds \(pool sizes 3 and 7 for dynamic schedulers, pool sizes 1–61 for identity\), coloured by scheduler\. Dynamic and static seeds are interleaved throughout the coverage axis\. The dashed line shows the least\-squares fit \(r=0\.21r=0\.21\)\.
#### D\.3\.3Pool Size Scaling

We vary the number of training contexts from 1 to 61 for the static identity baseline and two dynamic schedulers \(sinusoidal, cosine annealing\) on CartPole pole\-length, using 10 seeds per condition\. The scheduler hyperparameters used here are fixed reference configurations chosen for the tracking runs and are not the result of the scheduler search described in Section[4\.2](https://arxiv.org/html/2608.20799#S4.SS2); absolute scores for the dynamic conditions should therefore not be compared directly to the best\-found schedulers reported elsewhere\. Figure[12](https://arxiv.org/html/2608.20799#A4.F12)shows the last\-checkpoint IQM score and final 4D state\-space coverage\. The static baseline is sensitive to pool size: it peaks around pool\-3 and degrades at both extremes, with pool\-1 providing too little diversity and pool\-61 diluting training across too many contexts\. Dynamic schedulers are largely unaffected by pool size, remaining competitive from pool\-1 upwards\. The training curves in Figure[13](https://arxiv.org/html/2608.20799#A4.F13)confirm this: static pool\-3 converges fastest among static conditions, while pools 31 and 61 plateau noticeably lower, and sinusoidal pool\-7 tracks or exceeds the best static condition throughout\.

Crucially, 4D state\-space coverage is flat at≈6\.5%\\approx\\\!6\.5\\%across all pool sizes and all schedulers, and Table[11](https://arxiv.org/html/2608.20799#A4.T11)shows that this coverage is concentrated almost entirely in the first 50 k training steps: stages 2 and 3 contribute less than1%1\\%, and virtually no cells are newly discovered after stage 1 \(≤0\.03%\\leq\\\!0\.03\\%\)\. Pool size therefore modulates the*distribution*of the training signal across context values, not the breadth of states visited\.

Figure 12:Last\-checkpoint IQM score \(A\) and final 4D coverage \(B\) vs\. pool size for CartPole pole\-length \(10 seeds\)\. Error bars show SEM\.Figure 13:IQM training curves \(shading: Q1–Q3\) for identity pools 1–61; sinusoidal pool\-7 shown dashed for reference\.Table 11:Fraction of the12412^\{4\}CartPole hypergrid visited per 50 k\-step training window \(mean over 10 seeds, %\)\. Coverage in stages 2–3 is below1%1\\%for all conditions; virtually no cells are newly discovered after stage 1\.
#### D\.3\.4Parallel vs\. Sequential Training

We compare running 8 parallel environments against a single sequential environment for identity, sinusoidal, and cosine annealing schedulers at pool sizes 5 and 15 \(and identity at pool sizes 3 and 7\), using 10 seeds per condition on CartPole pole\-length\. Figure[14](https://arxiv.org/html/2608.20799#A4.F14)shows the last\-checkpoint IQM score and final 4D coverage for each condition, with the diagonal marking equal performance\.

Both metrics scatter tightly around the diagonal with no consistent direction \(Table[12](https://arxiv.org/html/2608.20799#A4.T12)\): score differences range from−21\-21to\+28\+28IQM with no pattern across schedulers or pool sizes, and coverage differences are below0\.5%0\.5\\%in all cases\. The summed deltas across all eight conditions are\+13\.1\+13\.1IQM and−0\.56%\-0\.56\\%coverage, both negligible\. Parallel training neither expands state\-space coverage nor reliably improves generalisation\.

This result is informative beyond the numerical statement\. Parallel environments, typical for PPO, introduce context diversity*across*simultaneous rollouts, enriching each policy gradient update with transitions from multiple dynamics regimes at once\. Dynamic scheduling, by contrast, introduces context variation*within*a single episode, exposing the policy to a structured temporal sequence of dynamics changes during one rollout\. The fact that the former does not replicate the gains of the latter suggests that the benefit of dynamic scheduling is not simply a consequence of seeing more diverse contexts per update, but is tied to the intra\-episode temporal structure of the training signal itself\.

Table 12:Score and coverage difference \(parallel−\-sequential\) per condition\. Positive = parallel preferred; negative = sequential preferred\.Figure 14:Sequential vs\. parallel IQM score \(A\) and 4D coverage \(B\) for CartPole\. Points on the dashed diagonal indicate no difference\. Colour encodes scheduler family; marker shape encodes pool size\.
#### D\.3\.5BipedalWalker State\-Space Coverage

We extend the coverage analysis to BipedalWalker \(COM\_X context, 4 seeds per pool size, pool sizes 1–15, sequential only\) using four complementary projections of the 24\-dimensional observation space\. For each projection we compute the fraction of discretised cells visited over 1 M training steps and compare across scheduler families \(identity, sinusoidal, cosine annealing, Lévy walk\)\. No projection yields a significant difference across schedulers \(Figure[15](https://arxiv.org/html/2608.20799#A4.F15)\), consistent with the CartPole finding\.

Table[13](https://arxiv.org/html/2608.20799#A4.T13)reports full\-run coverage across all four projections alongside mean training return \(averaged over pool sizes; note this is rollout return, not the IQM evaluation metric used in the main text\)\. None of the per\-projection ANOVAs are significant\. Across projections, the static identity baseline consistently ranks highest on coverage and lowest on return, while sinusoidal ranks lowest on coverage and highest on return\. Given the limited number of seeds, this trend is suggestive rather than conclusive; the broader state\-space spread of the static baseline may reflect slower convergence rather than active exploration\.

Figure[16](https://arxiv.org/html/2608.20799#A4.F16)breaks 4D joint coverage into three equal training stages \(0–333k, 333–667k, 667k–1 M steps\) with a sharedyy\-axis\. All schedulers drop from≈10\\approx\\\!10–13%13\\%in stage 1 to≈3\\approx\\\!3–5%5\\%by stage 3, confirming that BipedalWalker policies also converge to a narrow behavioral manifold well before the end of training, regardless of scheduler\.

Figure 15:BipedalWalker full\-run coverage per scheduler across four state\-space projections \(4 seeds×\\times4 context ranges, aggregated\)\.A: inter\-leg phase \(30230^\{2\},p=0\.12p=0\.12\)\.B: 4D joint angles \(12412^\{4\},p=0\.34p=0\.34\)\.C: 3D posture \(15315^\{3\},p=0\.68p=0\.68\)\.D: leg\-1 phase \(30230^\{2\},p=0\.31p=0\.31\)\.Table 13:Full\-run coverage \(%\) per scheduler across four projections and mean rollout return, averaged over pool sizes 1, 3, 7, 15 \(4 seeds each\)\. Bold marks the highest value per column\.Coverage \(%\)SchedulerA: inter\-legB: 4D jointsC: postureD: leg phaseReturnIdentity75\.112\.548\.725\.7142Cosine annealing71\.711\.647\.025\.8150Lévy walk74\.311\.447\.725\.4163Sinusoidal73\.511\.045\.726\.2177ANOVApp0\.120\.340\.680\.31—A: hip1vs hip2\(30230^\{2\}\); B: hip1, knee1, hip2, knee2\(12412^\{4\}\); C: hull, hip1, hip2\(15315^\{3\}\); D: leg\-1 phase \(30230^\{2\}\)\.Figure 16:BipedalWalker 4D joint\-space coverage per scheduler across three equal training stages \(≈333\\approx\\\!333k steps each\)\. Sharedyy\-axis shows the progressive contraction of the visited state space\. ANOVAp\>0\.05p\>0\.05for all stages\.
#### D\.3\.6Context Normalization Ablation

The figure below support the normalization design choice discussed in Section[3](https://arxiv.org/html/2608.20799#S3.SS0.SSS0.Px1)\. Results are from a preliminary subset of schedule families; the effect is consistent across conditions\.

\(a\)Static training — low pole length\.\(b\)Dynamic training — low pole length\.\(c\)Static training — high pole length\.\(d\)Dynamic training — high pole length\.
Figure 17:Effect of context normalization on static and dynamic PPO training \(CartPole, pole length context\)\. The dynamic curves aggregate performance across a preliminary set of non\-stationary schedules\. Lines show mean evaluation score across seeds; shaded regions indicate standard deviation\.

### D\.4Additional Bar Plots per Eval Context

##### CartPole\.

The last\-checkpoint bar plot appears in the main paper \(Figure[2](https://arxiv.org/html/2608.20799#S4.F2)\); here we include the best\-anytime companion\. All conditions perform strongly as the ID region saturates at the maximum reward of 500\. The meaningful spread is in OOD\-low, where static observed is the weakest \(IQM 313 at last checkpoint\) and dynamic blind the strongest \(IQM 456\), with static blind and dynamic observed in between\. Dynamic observed leads at the last checkpoint \(combined IQM 464\) while dynamic blind leads at best\-anytime \(488\); the two are close throughout, suggesting that even a blind scheduler captures most of the benefit of temporal context variation on this environment\.

Figure 18:CartPole combined IQM \(best anytime, pool\-7\)\. Per\-scheduler values in Table[15](https://arxiv.org/html/2608.20799#A6.T15)\.
##### BipedalWalker\.

Figure[19](https://arxiv.org/html/2608.20799#A4.F19)shows per\-context IQM at the last checkpoint \(pool\-3\)\. The static observed baseline achieves strong in\-distribution performance \(IQM 114\) but collapses on OOD contexts \(IQM−48\-48high,−10\-10low\), pulling its combined IQM to only 31: with three fixed training contexts the policy overfits to those specific dynamics and fails to generalise\. Dynamic schedulers substantially outperform both static conditions; even dynamic blind comfortably beats static blind, confirming that intra\-episode temporal structure adds value beyond naïve domain randomisation\.

Figure 19:BipedalWalker combined IQM \(last checkpoint, pool\-3\)\. Per\-scheduler values in Table[16](https://arxiv.org/html/2608.20799#A6.T16)\.Figure 20:BipedalWalker combined IQM \(best anytime, pool\-3\)\. The gap relative to the last\-checkpoint figure above is most pronounced for the static observed baseline, which collapses late in training\.
##### CarRacing\.

Context\-observed conditions exhibit significant late\-training policy degradation on this environment; we therefore show both the last\-checkpoint and best\-anytime figures side by side for each axis\. At the last checkpoint, dynamic blind is the strongest condition on both axes\. At best\-anytime, static blind leads on COM\_X \(IQM 842\.9 vs\. 803\.7 for dynamic blind\) while dynamic blind leads on COM\_Y \(760\.7 vs\. 636\.4 for static blind\); all observed conditions struggle to stably exploit the live context signal through the CNN pipeline\. Per\-scheduler IQM values are in Tables[17](https://arxiv.org/html/2608.20799#A6.T17)and[18](https://arxiv.org/html/2608.20799#A6.T18)\.

Figure 21:CarRacing COM\_X last\-checkpoint IQM \(pool\-3\)\.Figure 22:CarRacing COM\_X best\-anytime IQM \(pool\-3\)\.Figure 23:CarRacing COM\_Y last\-checkpoint IQM \(pool\-3\)\.Figure 24:CarRacing COM\_Y best\-anytime IQM \(pool\-3\)\.

## Appendix EMulti\-stage Schedulers Additional Results

### E\.1Best\-Anytime Results

Table[14](https://arxiv.org/html/2608.20799#A5.T14)complements Table[3](https://arxiv.org/html/2608.20799#S4.T3)with best\-anytime IQM scores — the highest combined evaluation score achieved at any stage boundary during training\. Walker values remain at the matched 1\.5 M\-step budget\.

Table 14:Multi\-stage Optuna retrain results at thebest\-anytime checkpoint\. Walker values at the matched 1\.5 M\-step budget\. Values: IQM \[Q1, Q3\]\. Top\-3 retrained trials shown per environment\.
### E\.2Optuna Results

The figures below detail the CartPole Optuna retrain: training curves \(Figure[25](https://arxiv.org/html/2608.20799#A5.F25)\), scheduler mode frequency per stage \(Figure[26](https://arxiv.org/html/2608.20799#A5.F26)\), hyperparameter distributions for active stages \(Figure[27](https://arxiv.org/html/2608.20799#A5.F27)\), and rank shift after retraining with more seeds \(Figure[28](https://arxiv.org/html/2608.20799#A5.F28)\)\.

Figure 25:Learning curves \(IQM±\\pmIQR\) for the top\-5 retrained CartPole trials vs\. static baseline\.Figure 26:Frequency of each scheduler mode \(constant, sinusoidal, cosine annealing\) at each stage position across all 20 retrained CartPole trials\. Stage 2 is active in nearly all configurations; Stage 1 is most often constant\.Figure 27:Hyperparameter distributions for active stages across all 20 retrained CartPole trials\. Top row: sinusoidal amplitude fraction vs\. period; bottom row: cosine annealing neighbourhood radius vs\.T0T\_\{0\}\. Only trials where the respective mode is active are shown\.Figure 28:Rank shift between the 4\-seed Optuna search ranking and the 10\-seed retrain ranking for CartPole\. Left: scatter of Optuna score vs\. retrain IQM±\\pmIQR; the dashed line marks the static pool\-7 baseline\. Right: bump chart showing the rank change for each trial after retraining with more seeds\.
### E\.3Walker

The figures below mirror the CartPole analysis for BipedalWalker: scheduler patterns and final scores \(Figure[29\(a\)](https://arxiv.org/html/2608.20799#A5.F29.sf1)\), training curves \(Figure[30](https://arxiv.org/html/2608.20799#A5.F30)\), rank shift \(Figure[31](https://arxiv.org/html/2608.20799#A5.F31)\), mode frequency \(Figure[32](https://arxiv.org/html/2608.20799#A5.F32)\), and hyperparameter distributions \(Figure[33](https://arxiv.org/html/2608.20799#A5.F33)\)\.

![Refer to caption](https://arxiv.org/html/2608.20799v1/C1_scheduler_heatmap.png)\(a\)Scheduler mode per stage for the top\-5 retrained trials, sorted by retrain IQM\.
\(b\)Final\-checkpoint IQM±\\pmIQR for top\-5 retrained trials vs\. static baseline \(1\.5 M steps\)\.
Figure 29:BipedalWalker Optuna retrain: scheduler patterns and final scores\.Figure 30:Learning curves \(IQM±\\pmIQR\) for the top\-5 retrained Walker trials vs\. static baseline\.Figure 31:Rank shift between the 3\-seed Optuna ranking and the 8\-seed retrain ranking for BipedalWalker\.Figure 32:Scheduler mode frequency per stage across all retrained Walker trials\.Figure 33:Scheduler parameter distributions per stage for all retrained Walker trials \(active schedulers only\)\.

## Appendix FScheduler Performance

Tables[15](https://arxiv.org/html/2608.20799#A6.T15)–[18](https://arxiv.org/html/2608.20799#A6.T18)report the per\-scheduler\-family IQM \[Q1, Q3\] ofavg\_combinedfor both context\-observability modes across all three environments\. For each family and mode, we show the single best\-performing configuration found during the scheduler search, selected by IQM of the last\-checkpointavg\_combinedscore\. Both last\-checkpoint and best\-anytime IQM are reported\. Bold marks the highest IQM in each column; in Table[17](https://arxiv.org/html/2608.20799#A6.T17)the best\-anytime blind column is an exception where the static baseline exceeds all dynamic schedulers, with the best dynamic result marked in italic bold for reference\.

Table 15:CartPole / length / pool\-7\. Per\-scheduler\-type IQM \[Q1, Q3\] ofavg\_combinedover 10 seeds\.Table 16:BipedalWalker / COM\_X / pool\-3\. Per\-scheduler\-type IQM \[Q1, Q3\] ofavg\_combinedover 8 seeds\.Table 17:CarRacing / longitudinal / pool\-3\. Per\-scheduler\-type IQM \[Q1, Q3\] ofavg\_combinedover 5 seeds\.Table 18:CarRacing / lateral / pool\-3\. Per\-scheduler\-type IQM \[Q1, Q3\] ofavg\_combinedover 5 seeds\.
## Appendix GScheduler Training Progression

Each figure shows the IQM episode return averaged over training checkpoints for all scheduler families, with one line per family using its best representative run\. Left panel: dynamicobservedschedulers\. Right panel: dynamicblindschedulers\. The dashed line is the static pool baseline\. Shaded bands indicate Q1–Q3 across seeds\.

Figure 34:CartPole / pole length / pool\-7 —combinedIQM \(all contexts, 10 seeds\)\. Each family’s best representative is selected by final combined IQM\.Figure 35:CartPole / pole length / pool\-7 —IDIQM \(in\-distribution contexts only, 10 seeds\)\.Figure 36:CartPole / pole length / pool\-7 —OOD\-lowIQM \(short\-pole contexts below training range, 10 seeds\)\.Figure 37:CartPole / pole length / pool\-7 —OOD\-highIQM \(long\-pole contexts above training range, 10 seeds\)\.Figure 38:BipedalWalker / COM\_X / pool\-3 —combinedIQM \(all contexts, 8 seeds\)\.Figure 39:BipedalWalker / COM\_X / pool\-3 —IDIQM \(in\-distribution contexts only, 8 seeds\)\.Figure 40:BipedalWalker / COM\_X / pool\-3 —OOD\-lowIQM \(centre\-of\-mass shifted below training range, 8 seeds\)\.Figure 41:BipedalWalker / COM\_X / pool\-3 —OOD\-highIQM \(centre\-of\-mass shifted above training range, 8 seeds\)\.Figure 42:CarRacing / COM\_X \(longitudinal CoM\) / pool\-3 —combinedIQM \(all contexts, 5 seeds\)\.Figure 43:CarRacing / COM\_X / pool\-3 —IDIQM \(in\-distribution contexts only, 5 seeds\)\.Figure 44:CarRacing / COM\_X / pool\-3 —OOD\-lowIQM \(longitudinal CoM shifted below training range, 5 seeds\)\.Figure 45:CarRacing / COM\_X / pool\-3 —OOD\-highIQM \(longitudinal CoM shifted above training range, 5 seeds\)\.Figure 46:CarRacing / COM\_Y \(lateral CoM\) / pool\-3 —combinedIQM \(all contexts, 5 seeds\)\.Figure 47:CarRacing / COM\_Y / pool\-3 —IDIQM \(in\-distribution contexts only, 5 seeds\)\.Figure 48:CarRacing / COM\_Y / pool\-3 —OOD\-lowIQM \(lateral CoM shifted below training range, 5 seeds\)\.Figure 49:CarRacing / COM\_Y / pool\-3 —OOD\-highIQM \(lateral CoM shifted above training range, 5 seeds\)\.