When to Plan: Learning to Select Between Reactive Control and Deliberative Planning

# Learning to Select Between Reactive Control and Deliberative Planning
Source: [https://arxiv.org/html/2607.16421](https://arxiv.org/html/2607.16421)
When to Plan: Learning to Select Between Reactive Control and Deliberative Planning

Adam Labiosa, Josiah P\. Hanna

Keywords:Meta\-reasoning, reinforcement learning, adaptive computation\.

SummaryIt has long been recognized that humans have the ability to switch between fast, reactive decision\-making and slower, deliberative planning\. In this paper, we study the question of how to learn this ability, known as meta\-reasoning, in artificial agents\. We model reactive decision\-making as a policy that directly maps state observations to actions\. Such policies can be trained with reinforcement learning \(RL\) or imitation learning, but may generalize poorly outside of their training distribution\. Alternatively, model\-based decision\-time planning is more likely to produce good actions across a broader set of states but requires additional computation time, which delays acting\. In this work, we introduce an RL method for training a meta\-reasoning policy that allocates computation by conditioning on a reactive\-policy uncertainty score\. This score enables it to predict when the reactive policy is likely to perform poorly and when planning is needed\. We conduct an empirical study on motion planning and navigation environments, showing that this design enables the meta\-reasoning policy to learn when the reactive policy provides a good\-enough action versus when decision\-time planning is needed\. Additionally, we show that our design enables the meta\-agent to shift toward fully reactive control as the reactive policy improves\.

Contribution\(s\)1\.We introduce a reinforcement learning approach to learning a meta\-policy for selecting between a fast, reactive policy, which may perform poorly out of its training distribution, and a planner, which requires time but reliably produces near\-optimal actions\. Context:Prior work has used RL to minimize compute, either for a single decision by selecting internal computations\(callaway\_learning\_2018\)or across an episode by using action repetitions\(orenstein\_toward\_2025\)\. Our work differs by addressing the meta\-reasoning problem as needing to make a sequence of meta\-level selections between a reactive policy and a planner across a full task episode\. Related ideas have been studied in cognitive science\(keramati\_speedaccuracy\_2011;kool\_planning\_2018\), but these works aim to describe human behavior rather than provide mechanisms for artificial agents\.2\.We show that our method produces meta\-policies which balance deliberate planning and using a reactive policy as a function of environment characteristics such as planning delay, reactive policy competence, and environment stochasticity\. Context:We conduct an empirical study which shows our method obtains higher return than baselines that always commit to one type of decision\-rule \([Table1](https://arxiv.org/html/2607.16421#S6.T1)\)\. By varying planning cost, reactive policy competence, and task stochasticity, we analyze how each factor changes the balance of reactive action compared to deliberative planning \([Figure3](https://arxiv.org/html/2607.16421#S6.F3)\)\.3\.We show that an uncertainty\-conditioned meta\-policy can track reactive policy competence even if the reactive policy improves over time, shifting toward more reactive control as reactive competence increases\. Context:We conduct an experiment which uses behavior cloning on planner rollouts to fine\-tune the reactive policy during training \([Section6\.6](https://arxiv.org/html/2607.16421#S6.SS6),[Figure4](https://arxiv.org/html/2607.16421#S6.F4)\)\. The key result is that uncertainty\-conditioned observations are sufficient for the meta\-policy to learn to call the reactive policy more as its training distribution widens\.

###### Abstract

It has long been recognized that humans have the ability to switch between fast, reactive decision\-making and slower, deliberative planning\. In this paper, we study the question of how to learn this ability, known as meta\-reasoning, in artificial agents\. We model reactive decision\-making as a policy that directly maps state observations to actions\. Such policies can be trained with reinforcement learning \(RL\) or imitation learning, but may generalize poorly outside of their training distribution\. Alternatively, model\-based decision\-time planning is more likely to produce good actions across a broader set of states but requires additional computation time, which delays acting\. In this work, we introduce an RL method for training a meta\-reasoning policy that allocates computation by conditioning on a reactive\-policy uncertainty score\. This score enables it to predict when the reactive policy is likely to perform poorly and when planning is needed\. We conduct an empirical study on motion planning and navigation environments, showing that this design enables the meta\-reasoning policy to learn when the reactive policy provides a good\-enough action versus when decision\-time planning is needed\. Additionally, we show that our design enables the meta\-agent to shift toward fully reactive control as the reactive policy improves\.

## 1Introduction

Humans naturally allocate cognitive effort depending on the situation\. For example, drivers often navigate familiar routes with minimal cognitive effort, whereas in unfamiliar environments they shift from following intuitive reactions to deliberate decision\-making\(charlton\_driving\_2013\)\. Acting reactively as opposed to thinking before acting has been termed System 1 and System 2 thinking in cognitive science\(kahneman2011thinking;evans1984heuristic\)and a similar distinction can be seen in AI systems\. Reactive, pattern\-matching agents trained with model\-free RL or imitation learning enable quick decision\-making but may generalize poorly to states outside of their training distribution whereas agents that always invoke a decision\-time planner produce high\-quality actions but must spend time creating a plan before acting\. Most current systems commit to a fixed strategy at test time like, for example, always using model\-free reactive actions\(bansal\_chauffeurnet\_2018;haarnoja\_learning\_2023\), always planning before acting\(xia\_survey\_2024\), or always reasoning before responding\(openai\_openai\_2024;deepseek\-ai\_deepseek\-r1\_2025\)\.

The goal of this work is to enable*adaptive computation*by learningwhento invoke more computationally\-intensive planning routines as opposed to taking a reactive action\. Our work contributes to the area of AI research known as meta\-reasoning\(horvitz\_computation\_nodate;russell\_principles\_1991\), the study of how an agent should allocate its computation, and specifically the use of RL to learn policies for meta\-cognition\.

We study the meta\-reasoning problem in a setting with two primary action\-generation mechanisms: a reactive policy and a decision\-time planner\. The reactive policy produces an action from a single forward pass of a neural network but only performs well within a limited training distribution\. In contrast, the planner produces a good sequence of actions from any environment state albeit at the cost of additional computation compared to the reactive policy\. While learned world models used for planning are similarly limited to their training distribution, prior work has shown that they generalize more robustly than reactive policies\(young\_benefits\_2023\)\. Planning is therefore most valuable in states outside of the reactive policy’s training distribution\. These observations motivate the need to arbitrate between a distribution\-constrained reactive policy and a planner that is effective across the full state space\.

Within this problem setting, we ask the following question:

*How can an RL agent learn to select between a distribution\-constrained, reactive policy and a slower decision\-time planning routine?*

To answer this question, we develop a method to train a meta\-policy that selects between reactive action and variable\-depth planning to maximize task return under a time\-based cost\. Critically, the meta\-policy is conditioned on uncertainty signals from the reactive policy which allow it to estimate reactive competence in the current state\.

We evaluate our approach in a suite of motion planning and navigation tasks\. First, we show our adaptive meta\-policy takes less time to reach goal states in comparison to fixed\-compute baselines including always react and always plan\. Second, we ablate key design choices of the meta\-policy observation space and show that reactive uncertainty and observation history are the most critical components for effective compute allocation\. Third, we demonstrate the meta\-policy learns environment characteristic\-dependent behavior by varying the reactive competence, planning cost and environment stochasticity\. Lastly, we apply our method to a joint\-training setting where the reactive policy improves alongside the meta\-policy and show that uncertainty\-conditioned observations enable the meta\-policy to track increasing reactive competence and shift toward fully reactive control\.

In summary, our work makes the following contributions:

1. 1\.We introduce a novel reinforcement learning approach to learning a meta\-policy for selecting between a fast, reactive policy, which may perform poorly out of its training distribution, and a planner, which requires time but reliably produces near\-optimal actions\.
2. 2\.We show that our method produces meta\-policies which balance deliberate planning and using a reactive policy as a function of environment characteristics such as planning delay, reactive policy competence, and environment stochasticity\.
3. 3\.We show that an uncertainty\-conditioned meta\-policy can track reactive policy competence even if the reactive policy improves over time, shifting toward more reactive control as reactive competence increases\.

## 2Markov Decision Processes

We consider a Markov decision process \(MDP\) defined by the tupleℳ=\(𝒮,𝒜,P,r,γ\)\\mathcal\{M\}=\(\\mathcal\{S\},\\mathcal\{A\},P,r,\\gamma\), where𝒮\\mathcal\{S\}is the state space,𝒜\\mathcal\{A\}is the action space,P​\(s′∣s,a\)P\(s^\{\\prime\}\\mid s,a\)denotes the transition dynamics,r​\(s,a\)r\(s,a\)is the reward function, andγ∈\[0,1\)\\gamma\\in\[0,1\)is the discount factor\. At each time steptt, the agent selects an actionat∼π\(⋅∣st\)a\_\{t\}\\sim\\pi\(\\cdot\\mid s\_\{t\}\)according to a policyπ\\pi, inducing a trajectoryτ=\(s0,a0,s1,a1,…\)\\tau=\(s\_\{0\},a\_\{0\},s\_\{1\},a\_\{1\},\\dots\)\. The RL objective is to learn a policy that maximizes the expected discounted return:J​\(π\)=𝔼τ∼π​\[∑t=0∞γt​r​\(st,at\)\]J\(\\pi\)=\\mathbb\{E\}\_\{\\tau\\sim\\pi\}\\left\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}r\(s\_\{t\},a\_\{t\}\)\\right\]\.

![Refer to caption](https://arxiv.org/html/2607.16421v1/figs/box_push.png)\(a\)Box Push
![Refer to caption](https://arxiv.org/html/2607.16421v1/figs/doorkey.png)\(b\)Doorkey
![Refer to caption](https://arxiv.org/html/2607.16421v1/figs/fetch.png)\(c\)Fetch
![Refer to caption](https://arxiv.org/html/2607.16421v1/figs/maze.png)\(d\)Maze
![Refer to caption](https://arxiv.org/html/2607.16421v1/figs/navigation.png)\(e\)Navigation

Figure 1:Example images of our environment suite\. Environments vary in state space type \(discrete versus continuous\), task structure, and reactive policy difficulty\. In all environments the objective is to reach a goal state from a start state\.
## 3Related Work

##### Learning to use computation\.

Several works use RL to learn meta\-reasoning\. Similar to our work,callaway\_learning\_2018uses model\-free RL to select computations\. However, their work focuses on allocating compute before taking an action in the real world while our work frames computation allocation as a sequential decision problem that persists across a full task episode\. Other works adapt the amount of compute within a planner\(budd\_stop\_2024;wang\_dynamic\_2025\), whereas our agent decides between executing a reactive policy instantly or spending time to generate a plan\. Event\-triggered and adaptive replanning methods\(hashimoto\_learning\-based\_2025;chen\_reinforcement\_2022;honda\_when\_2024\)learn when a plan is no longer valid, but assume an agent that always plans and does not consider minimizing computation to achieve faster task completion\. Other related works consider variable\-length options to minimize the number of actions\(karimi\_dynamic\_2023;lakshminarayanan\_dynamic\_2017;metelli\_control\_2020;orenstein\_toward\_2025;sharma\_learning\_2020\)but these works optimize action efficiency rather than computation allocation, and do not consider an explicit tradeoff between reactive execution and planning cost\.

##### Arbitration between model\-free and model\-based control\.

The question of when to use model\-free versus model\-based control has been studied in both AI and neuroscience\(dolle\_path\_2010;lee\_neural\_2014\)\. Existing AI approaches typically rely on hand\-designed switching rules or uncertainty heuristics derived from prediction error signals\(sheikhnezhad\_fard\_novel\_2019;fard\_mixing\_2018;sharma\_hybrid\_2025;raj\_rethinking\_2024\), and ignore the computation required for model\-based planning\. Other methods learn to select among multiple controllers\(kastner\_all\-\-one\_2021;dey\_learning\_2023;hamrick\_metacontrol\_2017\)but assume planning and reactive inference take equal time\. Our work differs by explicitly modeling the time\-based cost of different controllers and learning a meta\-reasoning policy to arbitrate between them\.

##### Adaptive computation in LLMs\.

Recent work on LLMs addresses when to generate reasoning tokens\(paglieri\_learning\_2025;fang\_thinkless\_2025;sabbata\_rational\_2025;manvi\_zero\-overhead\_2025;chung\_thinker\_2025\)\. While structurally similar, these works do not use meta\-level agents and instead aim to encode the decision into the LLMs themselves, which constrains the adaptability of the methods\.ong\_routellm\_2025uses a controller to route between a larger and smaller model but is trained on preference data instead of learning the decision online as ours does\.hanna\_when\_2025introduce a theoretical formalism for when an RL agent should learn to take abstract thinking actions that can be interpreted as choosing additional computation\. In contrast, we focus on developing agents that choose from among a set of concrete decision\-rules that use varying amounts of computation\.

##### Cognitive science\.

The tradeoff between fast reactive behavior and slow deliberation is well\-studied in cognitive science under bounded rationality and dual\-process theories\(griffiths\_doing\_2019;keramati\_speedaccuracy\_2011;kelly\_best\_2022\)\. Several works theorize that humans deliberate when the expected value exceeds its cost\(lieder\_strategy\_2017;lieder\_rational\_2018\), which directly motivates our formulation of planning as a time\-costly option\. Similar work applies RL as a model of meta\-cognitive control\(krueger\_enhancing\_2017;jain\_how\_2019\), but these works provide descriptive models of human behavior whereas we focus on operationalizing this ability in artificial agents\.

![Refer to caption](https://arxiv.org/html/2607.16421v1/x1.png)Figure 2:The meta reasoning agent\. The reactive policy \(ensemble of neural networks\), outputs an actionara\_\{r\}and an uncertainty scoreut~u\_\{\\tilde\{t\}\}for the current state\.ara\_\{r\}andut~u\_\{\\tilde\{t\}\}along with task features of distance to goal and previous action are input to the meta\-agent\. The meta\-agent chooses between a reactive action and a set of plans with different horizons\. The actions are executed in the environment\.

## 4Problem Formulation

We consider the problem of training, with RL, a meta\-reasoning policy that selects between controllers with different time\-based costs\. Specifically, we are interested in switching between a reactive policy, which produces correct actions within its training distribution but costs little time, and a planner, which produces correct actions across the entire state space but costs more time\. We formalize this meta\-reasoning problem using the options framework\(sutton\_between\_1999\)\. The meta\-MDP is defined asℳ~=\(𝒮~,𝒪,𝒫~,r~,γ\)\\tilde\{\\mathcal\{M\}\}=\(\\tilde\{\\mathcal\{S\}\},\\mathcal\{O\},\\tilde\{\\mathcal\{P\}\},\\tilde\{r\},\\gamma\)where𝒮~\\tilde\{\\mathcal\{S\}\}is the meta\-state space,𝒪\\mathcal\{O\}is the options space, where an optiono∈𝒪o\\in\\mathcal\{O\}is a possibly temporally\-extended choice action that executes in the world environment,𝒫~\\tilde\{\\mathcal\{P\}\}is the meta\-transition function,r~\\tilde\{r\}is the meta\-reward andγ\\gammais the discount factor\.

The meta\-state space𝒮~\\tilde\{\\mathcal\{S\}\}is the space of the agent’s possible internal states, which need not correspond directly to the world\-environment state space𝒮\\mathcal\{S\}\. There are two types of options𝒪\\mathcal\{O\}available to the meta\-agent: planning options and reactive control options\. We are particularly interested in understanding how the tradeoff between decision speed and decision quality affects task performance\. With this in mind, we restrict ourselves to goal\-reaching domains where optimal behavior minimizes time to reach the goal state\.

Planning options correspond to selecting a plannerπk\\pi\_\{k\}constrained to a fixed planning horizonkk\. For a plan option of horizonkk, the planner computes a sequencep=\[a0,a1,…,ak−1\]p=\[a\_\{0\},a\_\{1\},\\dots,a\_\{k\-1\}\]which is then executed to completion in the world environment\. We unify the cost to plan and cost to act under a time\-based scale to enable optimizing under the time\-to\-goal objective\. We define the plan option cost ascp​\(k\)=−α​kc\_\{p\}\(k\)=\-\\alpha kwhereα\\alphascales the time\-cost per search depth\. Each execution of an action incurs a costca=−1c\_\{a\}=\-1\(i\.e\. takes one unit of time\), so the total reward of executing a planning option isr~t~=cp​\(k\)\+k⋅ca=−α​k−k\\tilde\{r\}\_\{\\tilde\{t\}\}=c\_\{p\}\(k\)\+k\\cdot c\_\{a\}=\-\\alpha k\-k\.

Reactive control is modeled as a one\-step option where the policy is a learned reactive policyπr​\(a∣s\)\\pi\_\{\\mathrm\{r\}\}\(a\\mid s\)\. This option incurs no planning cost \(cp=0c\_\{p\}=0\) and executes a single environment action, thereforer~t~=ca=−1\\tilde\{r\}\_\{\\tilde\{t\}\}=c\_\{a\}=\-1\. The reactive policy is trained, through imitation learning, so that it produces reasonable actions on a limited amount of the full state space\. Formally, the reactive policy is trained on a subset of states𝒮I​D⊂𝒮\\mathcal\{S\}\_\{ID\}\\subset\\mathcal\{S\}, with the remaining states𝒮O​O​D=𝒮/𝒮I​D\\mathcal\{S\}\_\{OOD\}=\\mathcal\{S\}/\\mathcal\{S\}\_\{ID\}considered out\-of\-distribution\. For in\-distribution states, the reactive policy performs near\-optimally such thatVπr​\(s\)≈V∗​\(s\)V^\{\\pi\_\{\\mathrm\{r\}\}\}\(s\)\\approx V^\{\*\}\(s\)fors∈𝒮I​Ds\\in\\mathcal\{S\}\_\{ID\}\. For out\-of\-distribution statess∈𝒮O​O​Ds\\in\\mathcal\{S\}\_\{OOD\}, the reactive policy may generalize poorly such thatVπr​\(s\)≪V∗​\(s\)V^\{\\pi\_\{\\mathrm\{r\}\}\}\(s\)\\ll V^\{\*\}\(s\), motivating the use of planning despite the incurred computational cost\.

In the meta\-transition dynamics𝒫~\\tilde\{\\mathcal\{P\}\}, transitions between states are induced by the underlying world\-dynamics and the meta\-action chosen\. When a planning option of horizonkkis taken, the meta\-state transition occurs afterkkworld\-environment steps\. When a reactive control option is chosen, the transition occurs after one world\-environment step\. The only time akkstep planning option terminates early is when the underlying world\-environment terminates partially through an option\. At each meta\-decision timestept~\\tilde\{t\}, the meta\-policyπΩ​\(o∣s~t~\)\\pi\_\{\\Omega\}\(o\\mid\\tilde\{s\}\_\{\\tilde\{t\}\}\)selects an optionoobased on the agent’s internal information states~t~\\tilde\{s\}\_\{\\tilde\{t\}\}to control the agent’s use of computation\. When an option terminates, the meta\-policy observes the updated internal state and selects the next option\.

## 5Reinforcement Learning\-Based Meta\-Agent for When to Plan

In this section, we describe how to learn a meta\-policy for allocating computation between reactive control and decision\-time planning\. Our meta\-observation design is motivated by three properties of agents operating in large, open\-ended environments\. First, as the reactive policy will encounter both in\-distribution and out\-of\-distribution states, the meta\-observation cannot use raw environment state, since the meta\-policy would then face the same in\-distribution/out\-of\-distribution problem\. Second, the reactive policy may improve over time through experience\(anthony2017thinking\), so the meta\-observation must reflect current reactive competence\. Third, since the primary benefit of the reactive policy is that it requires no planning time, the meta\-observation must itself be computable without search\.

With these observations in mind, we instantiate the meta\-MDP as follows\. The meta\-policy operates on an internal states~\\tilde\{s\}which can be computed without search\. The observation consists of an uncertainty score from the reactive policyut~u\_\{\\tilde\{t\}\}, the current distance to the goaldt~d\_\{\\tilde\{t\}\}, and the previous meta\-action takena~t~−1\\tilde\{a\}\_\{\\tilde\{t\}\-1\}\. These values are concatenated as a single observation:

s^t~=\[ut~,dt~,a~t~−1\]\\hat\{s\}\_\{\\tilde\{t\}\}=\[u\_\{\\tilde\{t\}\},d\_\{\\tilde\{t\}\},\\tilde\{a\}\_\{\\tilde\{t\}\-1\}\]Additionally, we provide the meta\-agent with a history ofnnmeta\-observationss^t~\\hat\{s\}\_\{\\tilde\{t\}\}, giving it access to trends in uncertainty and progress toward the goal:

s~t~=\[s^t~,s^t~−1,…,s^t~−\(n−1\)\]\\tilde\{s\}\_\{\\tilde\{t\}\}=\[\\hat\{s\}\_\{\\tilde\{t\}\},\\hat\{s\}\_\{\\tilde\{t\}\-1\},\\dots,\\hat\{s\}\_\{\\tilde\{t\}\-\(n\-1\)\}\]
The action space for our meta\-agent includes a reactive option andNNplanning policy options:

𝒪=\{πr,πk1,πk2,…,πkN\}\\mathcal\{O\}=\\\{\\pi\_\{\\mathrm\{r\}\},\\pi\_\{k\_\{1\}\},\\pi\_\{k\_\{2\}\},\\dots,\\pi\_\{k\_\{N\}\}\\\}Each planning option corresponds to a fixed planning horizon, with longer horizons producing longer plans at a greater time cost\.

A meta\-decision is made whenever an option terminates\. At each meta\-step \([Figure2](https://arxiv.org/html/2607.16421#S3.F2)\), the reactive policy produces an action and uncertainty value, the meta\-state is constructed, and the meta\-policy selects an option\. If the reactive optionπr\\pi\_\{\\mathrm\{r\}\}is chosen, the reactive policy’s action is executed for one world step\. If a planning optionπk\\pi\_\{k\}is chosen, a plan of horizonkkis computed using the planner and then executed open\-loop in the world environment for its full duration\. The meta\-policyπΩ\\pi\_\{\\Omega\}is trained with model\-free RL to minimize time to the goal\.

To compute the uncertainty score of the reactive policy for the meta\-state, we implement our reactive policy as an ensemble ofMMneural networks\(lakshminarayanan\_simple\_2017\)\. The networks directly map the environment statessto an actionaa, and using the logits of the output, we compute the uncertainty score for the meta\-state\. This uncertainty score is a task\-agnostic estimate of in\-distribution competence based on ensemble disagreement\. For continuous control, each network outputs an action meanμi∈ℝD\\mu\_\{i\}\\in\\mathbb\{R\}^\{D\}\. The ensemble action and uncertainty are computed as:

a=1M​∑i=1Mμi,u​\(s\)=∑j=1DVari​\[μi,j\]\.a=\\frac\{1\}\{M\}\\sum\_\{i=1\}^\{M\}\\mu\_\{i\},\\qquad u\(s\)=\\sum\_\{j=1\}^\{D\}\\mathrm\{Var\}\_\{i\}\\\!\\left\[\\mu\_\{i,j\}\\right\]\.For discrete action spaces, each network outputs a categorical distribution𝐩i\\mathbf\{p\}\_\{i\}and the averaged distribution is𝐏¯=1M​∑i=1M𝐩i\\bar\{\\mathbf\{P\}\}=\\frac\{1\}\{M\}\\sum\_\{i=1\}^\{M\}\\mathbf\{p\}\_\{i\}\. The ensemble action and uncertainty are:

a=arg⁡maxj⁡P¯​\(j\),u​\(s\)=−∑j=1JP¯​\(j\)​log⁡P¯​\(j\)\.a=\\arg\\max\_\{j\}\\,\\bar\{P\}\(j\),\\qquad u\(s\)=\-\\sum\_\{j=1\}^\{J\}\\bar\{P\}\(j\)\\log\\bar\{P\}\(j\)\.In both cases,ut~=u​\(st~\)u\_\{\\tilde\{t\}\}=u\(s\_\{\\tilde\{t\}\}\), and high uncertainty corresponds to out\-of\-distribution inputs where planning may be beneficial, as the reactive policy is more likely to produce worse actions and therefore take more time to reach the goal\.

## 6Experiments

We evaluate our method on a suite of environments in which the goal is to navigate from a starting location to a goal location \([Figure1](https://arxiv.org/html/2607.16421#S2.F1)\)\. First, we show that our meta\-policy reaches goal states in fewer timesteps than fixed\-compute baselines including always\-reactive and always\-planning policies\. Second, we ablate the design of the meta\-agent’s observation space, finding that reactive uncertainty and observation history are the most critical components\. Third, we characterize how environment factors including planning cost and reactive competence change the learned allocation strategy\. Finally, we show the meta\-policy tracks improving reactive competence during joint training, shifting to fully reactive control as both the meta\-policy and reactive policy converge\.

### 6\.1Environments

Our environment suite \([Figure1](https://arxiv.org/html/2607.16421#S2.F1)\) has five environments with varying characteristics\. For each environment, we generate distinct sets of object configurations along with a start and goal location which we consider to be a task\. Unless otherwise stated, we use 20 tasks and assign half to reactive in\-distribution and half to reactive out\-of\-distribution\. For all environments except Maze, each task has a single start and goal location\. For Maze, each task has 5 start and goal configurations\.

Here we provide a summary of each environment:

1. 1\.Box Push: A simplified version of Sokoban, a standard planning benchmark where the goal is to push boxes onto goal locations\(botea\_using\_2003\)\. In our implementation the available gridspace is 6x6 and there are three boxes\. In this domain, an incorrect move \(e\.g\., moving a box into an incorrect corner\) can be irrecoverable and cause task failure\. The reactive actors receive a symbolic representation of the world\.
2. 2\.DoorKey: An environment from the MiniGrid library\(MinigridMiniworld23\)\. DoorKey is a discrete, multi\-stage task where the objective is to navigate to a goal position by picking up a key, unlocking a door and walking to the goal\. We modified the environment to use symbolic observations instead of the original image observations\.
3. 3\.Fetch: An environment modified from the Gymnasium Robotics Fetch Reach environment\(gymnasium\_robotics2023github\)\. We added in obstacles to the table which the gripper must navigate around\. The reactive actors receive two images of the state of the world from different angles\. The planner for this domain operates in Cartesian space while the reactive actor operates in 7 DOF joint space to make planner easier while preserving the difficulty of multi\-joint arm control\. We convert from Cartesian space to joint space through MuJoCo inverse kinematic control for reactive policy training\.
4. 4\.Maze: A maze\-like gridworld environment where the goal is to navigate to a goal location\. Each obstacle configuration defines 5 start/goal pairs to test whether the meta\-policy generalizes across targets within the same layout\. The reactive actors receive the full maze as an image input\.
5. 5\.Navigation: A continuous navigation task\. The goal of this environment is to navigate from a start position to a goal position through continuous control\. The agent navigates by controlling continuous displacement at each timestep up to a small maximum distance in all directions\. The reactive actors receive the full environment as an image input\.

EnvironmentMeta\-policyReactiveShortMediumLong\\rowcolorshade Box Push−24\.3\\mathbf\{\-24\.3\}−133\.0\-133\.0−306\.1\-306\.1−44\.7\-44\.7−59\.8\-59\.8\\rowcolorshade\[0\.4,0\.4\]\[0\.4,0\.4\]\[3\.1,3\.1\]\[3\.1,3\.1\]\[3\.6,3\.5\]\[3\.6,3\.5\]\[2\.0,1\.9\]\[2\.0,1\.9\]\[3\.7,3\.6\]\[3\.7,3\.6\]Doorkey−25\.1\\mathbf\{\-25\.1\}−135\.2\-135\.2−126\.7\-126\.7−27\.2\-27\.2−32\.0\-32\.0\[0\.3,0\.3\]\[0\.3,0\.3\]\[2\.5,2\.4\]\[2\.5,2\.4\]\[2\.9,2\.9\]\[2\.9,2\.9\]\[0\.1,0\.1\]\[0\.1,0\.1\]\[0\.1,0\.1\]\[0\.1,0\.1\]\\rowcolorshade Fetch−12\.2\\mathbf\{\-12\.2\}−132\.4\-132\.4−51\.0\-51\.0−70\.7\-70\.7−30\.0\-30\.0\\rowcolorshade\[0\.1,0\.1\]\[0\.1,0\.1\]\[2\.4,2\.4\]\[2\.4,2\.4\]\[2\.5,2\.4\]\[2\.5,2\.4\]\[3\.9,3\.7\]\[3\.9,3\.7\]\[0\.3,0\.3\]\[0\.3,0\.3\]Maze−19\.3\\mathbf\{\-19\.3\}−56\.2\-56\.2−89\.6\-89\.6−37\.8\-37\.8−24\.5\-24\.5\[0\.3,0\.3\]\[0\.3,0\.3\]\[0\.9,0\.9\]\[0\.9,0\.9\]\[1\.4,1\.4\]\[1\.4,1\.4\]\[1\.0,1\.0\]\[1\.0,1\.0\]\[0\.4,0\.4\]\[0\.4,0\.4\]\\rowcolorshade Navigation−39\.6\\mathbf\{\-39\.6\}−140\.8\-140\.8−41\.2\-41\.2−45\.4\-45\.4−46\.9\-46\.9\\rowcolorshade\[0\.1,0\.1\]\[0\.1,0\.1\]\[2\.3,2\.3\]\[2\.3,2\.3\]\[0\.1,0\.1\]\[0\.1,0\.1\]\[0\.1,0\.1\]\[0\.1,0\.1\]\[0\.2,0\.2\]\[0\.2,0\.2\]Table 1:Average episode return which includes both cost of acting and planning\. Baselines are fixed policies which exclusively call the reactive, short plan, medium plan, and long plan option\. 95% bootstrap confidence intervals \(margins shown below the mean\)\. Bold entries overlap in CI with the best performer per environment\. We run 30 seeds per experiment with 300 evaluation episodes per seed\.
### 6\.2Experimental Setup

##### Meta\-Policy\.

We train the meta\-policy with PPO\(schulman\_proximal\_2017\)implemented in Stable Baselines3\(stable\-baselines3\)with default hyperparameters\. We chose PPO for its stability and well\-understood training dynamics\. We use a history of 4 observations to capture the short\-term trends in reactive uncertainty and task progress\.

##### Reactive Policy\.

The reactive policy is pretrained by behavior cloning from trajectories generated by the planner on in\-distribution tasks\. All experiments useM=4M=4networks\. For pretraining parameters see[Appendix4](https://arxiv.org/html/2607.16421#A1.T4)\.

##### Planning\.

Planning is performed using a given accurate world model which isolates the compute allocation problem from the model learning problem\. In all environments we use an A\* planner which we constrain to a maximum search depth to approximate a time limit111Several proxies for a time\-based budget are possible\. In this work, we assign cost based on search depth\. An alternative is an effort\-based approximation, such as assigning the cost to be the number of A\* nodes expanded\. Both are proxies for the same underlying time\-based cost\.\([Table4](https://arxiv.org/html/2607.16421#A1.T4)\)\.

For continuous control domains, we use a lattice\-based A\* over a discretized Euclidean state space — end\-effector space for Fetch and agent location space for Navigation\. We use short, medium, and long planning horizons of 15%, 30%, and 50% of the task horizon to span a low, medium and high amount of compute\. We leave the sensitivity of the meta\-policy to these values for future work\.

We use a plan cost ofα=0\.5\\alpha=0\.5as the default value\. This value was chosen to avoid planning that is prohibitively expensive or trivially cheap\. A value ofα=0\.5\\alpha=0\.5intuitively means that each planning step costs 50% more time units than a reactive step and avoids the two non\-interesting regimes\. Asα→0\\alpha\\rightarrow 0, planning is effectively free so an optimal policy always plans and whenα\\alphais large planning is dominated by reactive control in all but the most extreme out\-of\-distribution cases\. We analyze the sensitivity of the meta\-policy toα\\alphain[Section6\.5](https://arxiv.org/html/2607.16421#S6.SS5)\.

##### Evaluation\.

We define return to be the total cost of time to plan and act over an episode as stated in[Section4](https://arxiv.org/html/2607.16421#S4)\. We run all experiments over 30 independent seeds and results are aggregated across seeds\. All methods are evaluated for 300 episodes using deterministic evaluation after training and we fix the hyperparameters across all ablations\.

EnvironmentReactiveShortMediumLong\\rowcolorshade Box Push44\.444\.40\.00\.00\.00\.055\.655\.6\\rowcolorshade\[0\.8,0\.8\]\[0\.8,\\,0\.8\]\[0\.0,0\.0\]\[0\.0,\\,0\.0\]\[0\.0,0\.0\]\[0\.0,\\,0\.0\]\[0\.8,0\.8\]\[0\.8,\\,0\.8\]DoorKey23\.823\.835\.435\.440\.840\.80\.00\.0\[0\.6,0\.7\]\[0\.6,\\,0\.7\]\[0\.3,0\.3\]\[0\.3,\\,0\.3\]\[0\.4,0\.4\]\[0\.4,\\,0\.4\]\[0\.0,0\.0\]\[0\.0,\\,0\.0\]\\rowcolorshade Fetch77\.077\.020\.720\.70\.00\.02\.32\.3\\rowcolorshade\[0\.5,0\.5\]\[0\.5,\\,0\.5\]\[0\.5,0\.5\]\[0\.5,\\,0\.5\]\[0\.0,0\.0\]\[0\.0,\\,0\.0\]\[0\.2,0\.2\]\[0\.2,\\,0\.2\]Maze51\.151\.10\.30\.30\.70\.747\.947\.9\[1\.0,1\.0\]\[1\.0,\\,1\.0\]\[0\.1,0\.1\]\[0\.1,\\,0\.1\]\[0\.1,0\.1\]\[0\.1,\\,0\.1\]\[1\.0,1\.0\]\[1\.0,\\,1\.0\]\\rowcolorshade Navigation20\.020\.080\.080\.00\.00\.00\.00\.0\\rowcolorshade\[0\.7,0\.7\]\[0\.7,\\,0\.7\]\[0\.7,0\.7\]\[0\.7,\\,0\.7\]\[0\.0,0\.0\]\[0\.0,\\,0\.0\]\[0\.0,0\.0\]\[0\.0,\\,0\.0\]Table 2:Meta\-action percentages for the trained meta\-policy with 95% bootstrap confidence intervals \(margins shown below each mean\)\. We run 30 seeds per experiment with 300 evaluation episodes per seed\.

### 6\.3Learning When to Plan

In this experiment, we verify that our meta\-policy learns to effectively allocate compute across our suite of environments\.[Table1](https://arxiv.org/html/2607.16421#S6.T1)shows the mean evaluation return for our meta\-agent in comparison to fixed\-compute baselines and[Table2](https://arxiv.org/html/2607.16421#S6.T2)shows the meta\-action distribution of the learned meta\-policy\. In all five domains, our meta\-agent performs better than all fixed compute strategies, showing our adaptive computation method is able to learn effective planning\-reacting behavior\. The closest heuristic to our meta\-agent is the Medium Plan in the Doorkey environment\. Notably, Doorkey is also the only domain where the meta\-policy does not converge to a two\-action strategy\. Since Doorkey is a staged task, these results suggest that the medium and short planning horizons align with the sub\-goal structure of the task and a fixed strategy can achieve close performance to an adaptive one\.

The two tasks where the meta\-agent achieves the largest percent improvement over baselines are Box Push with almost 2x improvement and Fetch with about a 2\.5x improvement\. In Box Push, short\-horizon planning is suboptimal in most situations since low\-depth search is inadequate to find the long\-horizon action sequences required to reach the goal state, as seen in the large performance gap between short and medium planning results\. Furthermore, Box Push is the only domain where mistakes are unrecoverable, meaning imprecise use of the reactive policy risks failure\. The meta\-agent’s performance suggests it learns to avoid reactive control and short plans in states where mistakes would cause failure\. In Fetch, the complexity stems from the action space of the 7 DOF robot arm the reactive policy must control\. Since the meta\-agent converges to a predominately reactive strategy with over 75% reactive control, it suggests that the reactive policy partially generalizes to out\-of\-distribution tasks, but cannot complete them without planning\. The strong performance in both domains suggests that task complexity does not make the meta\-reasoning problem harder\.

In three of the five tasks, the meta\-policy converges to a near\-equal split between reactive and planning actions, suggesting the agent learns to distinguish between in\-distribution and out\-of\-distribution states\. Navigation and Fetch are two exceptions, which likely reflect the reactive policy’s out\-of\-distribution performance\. Qualitatively, the navigation reactive policy sometimes gets "stuck" on objects in the task, and the results suggest that the meta\-policy can distinguish when that happened and switch to planning\.

EnvironmentMeta\-policyDistanceHistoryPrevious ActionUncertainty\\rowcolorshade Box Push−24\.3\-24\.3−20\.9\\mathbf\{\-20\.9\}−110\.7\-110\.7−36\.3\-36\.3−73\.9\-73\.9\\rowcolorshade\[0\.4,0\.4\]\[0\.4,\\,0\.4\]\[0\.3,0\.2\]\[0\.3,\\,0\.2\]\[2\.5,2\.5\]\[2\.5,\\,2\.5\]\[1\.2,1\.1\]\[1\.2,\\,1\.1\]\[2\.4,2\.2\]\[2\.4,\\,2\.2\]DoorKey−25\.1\\mathbf\{\-25\.1\}−34\.3\-34\.3−26\.1\-26\.1−29\.1\-29\.1−27\.9\-27\.9\[0\.3,0\.3\]\[0\.3,\\,0\.3\]\[1\.0,0\.9\]\[1\.0,\\,0\.9\]\[0\.1,0\.1\]\[0\.1,\\,0\.1\]\[0\.6,0\.6\]\[0\.6,\\,0\.6\]\[0\.6,0\.6\]\[0\.6,\\,0\.6\]\\rowcolorshade Fetch−12\.2\\mathbf\{\-12\.2\}−12\.5\-12\.5−21\.1\-21\.1−12\.1\\mathbf\{\-12\.1\}−16\.1\-16\.1\\rowcolorshade\[0\.1,0\.1\]\[0\.1,\\,0\.1\]\[0\.1,0\.1\]\[0\.1,\\,0\.1\]\[0\.8,0\.8\]\[0\.8,\\,0\.8\]\[0\.1,0\.1\]\[0\.1,\\,0\.1\]\[0\.2,0\.2\]\[0\.2,\\,0\.2\]Maze−19\.3\\mathbf\{\-19\.3\}−19\.7\\mathbf\{\-19\.7\}−19\.1\\mathbf\{\-19\.1\}−19\.3\\mathbf\{\-19\.3\}−25\.0\-25\.0\[0\.4,0\.3\]\[0\.4,\\,0\.3\]\[0\.4,0\.4\]\[0\.4,\\,0\.4\]\[0\.3,0\.3\]\[0\.3,\\,0\.3\]\[0\.4,0\.4\]\[0\.4,\\,0\.4\]\[0\.4,0\.4\]\[0\.4,\\,0\.4\]\\rowcolorshade Navigation−39\.6\-39\.6−41\.2\-41\.2−40\.7\-40\.7−37\.9\\mathbf\{\-37\.9\}−41\.2\-41\.2\\rowcolorshade\[0\.1,0\.1\]\[0\.1,\\,0\.1\]\[0\.1,0\.1\]\[0\.1,\\,0\.1\]\[0\.1,0\.1\]\[0\.1,\\,0\.1\]\[0\.2,0\.2\]\[0\.2,\\,0\.2\]\[0\.1,0\.1\]\[0\.1,\\,0\.1\]Average−24\.1\\mathbf\{\-24\.1\}−25\.7\-25\.7−43\.5\-43\.5−27\.0\-27\.0−36\.8\-36\.8\[0\.1,0\.1\]\[0\.1,\\,0\.1\]\[0\.2,0\.2\]\[0\.2,\\,0\.2\]\[0\.5,0\.5\]\[0\.5,\\,0\.5\]\[0\.3,0\.3\]\[0\.3,\\,0\.3\]\[0\.5,0\.5\]\[0\.5,\\,0\.5\]Table 3:Average episode return per method ablation with 95% bootstrap confidence intervals \(margins below each mean\)\. Bold entries overlap in CI with the best performer per environment\. Each header is the component of the ablation we remove from the observation space \(e\.g\.,Distanceremoves the distance to goal\)\. We run 30 seeds per experiment with 300 evaluation episodes per seed\.
### 6\.4Ablation Study

In this section we investigate which components of our meta\-agent’s observation space are most critical for learning adaptive computation\. We ablate each component by removing one feature at a time:

1. 1\.Distance: Removes the distance component\.
2. 2\.History: Meta\-agent only receives current state observations\.
3. 3\.Previous Action: Removes the previous action taken\.
4. 4\.Uncertainty: Removes the model\-free uncertainty\.

The results in[Table3](https://arxiv.org/html/2607.16421#S6.T3)show that the two most important components to our method are the history and uncertainty scores\. Poor performance without History is most clearly seen in the Box Push \(4\.5x worse\) and Fetch \(1\.7x worse\) environments\. We once again note that these tasks have challenges in irreversibility of actions and high\-dimensional action space respectively\. By removing history, it seems as if the agent is no longer able to track progress, and therefore it is unclear if the reactive agent is moving toward the goal\. A similar drop\-off is seen without the uncertainty score in these two environments\. This suggests that without directly being able to tell if the task is in\-distribution or out\-of\-distribution for the reactive policy, the meta\-agent is not able to perform optimally\. However, the lack of substantial drop\-off without the uncertainty score in the other three environments suggests that the combination of history and distance enables the agent to learn a proxy for uncertainty in these environments, but that learned metric is not as effective as directly accessing the uncertainty\.

### 6\.5Environment Characteristics

![Refer to caption](https://arxiv.org/html/2607.16421v1/x2.png)Figure 3:Average meta\-action taken after training across all environments for different environment characteristics\. We vary the percentage of in\-distribution tasks \(left\), the computational cost of planning \(center\), and environmental stochasticity \(right\)\. Results reflect 95% bootstrap confidence intervals across 30 seeds per experiment \(300 evaluation episodes per seed\)\.In this section we investigate how different environment characteristics affect the learned meta\-policy behavior\. First, we vary the proportion of in\-distribution tasks by changing the number of out\-of\-distribution tasks\. Results shown in[Figure3](https://arxiv.org/html/2607.16421#S6.F3)in the leftmost section\. As the proportion of in\-distribution tasks increases, the meta\-policy shifts toward reactive control, with the proportion of reactive actions roughly tracking the proportion of in\-distribution tasks\. This behavior is expected since the reactive policy is competent in a higher proportion of tasks so the meta\-policy correctly identifies that planning is less frequently necessary\. At the extreme where all tasks are in\-distribution, the meta\-policy converges to fully reactive control, confirming the meta\-policy correctly interprets the reactive policy’s competence across the distribution of tasks\.

Second, we vary the planning cost parameterα\\alpha, where a value ofα=0\.1\\alpha=0\.1means a plan of length 15 costs 1\.5 timesteps to compute\. Results are shown in[Figure3](https://arxiv.org/html/2607.16421#S6.F3)in the middle section\. As planning cost increases, the meta\-policy relies increasingly on reactive control\. Notably, even though the reactive policy produces suboptimal actions in out\-of\-distribution states, the time saved by avoiding planning is sufficient to achieve better returns than planning in some cases\. A second finding is that the ratio of short, medium and long planning horizons remains roughly constant asα\\alphaincreases, suggesting the meta\-policy reduces planning frequency uniformly rather than exclusively dropping longer plans\. This implies the meta\-policy learns when to plan, but not how long to plan, as the distribution of plan lengths remains consistent across environments\.

Third, we vary the stochasticity of the environment by introducing action noiseϵ\\epsiloninto the transition dynamics\. For discrete environments,ϵ\\epsiloncontrols the probability of a random action being taken and for continuous environments, we add Gaussian noise with varianceϵ\\epsilonto the action\. Results are shown in[Figure3](https://arxiv.org/html/2607.16421#S6.F3)in the rightmost section\. The most significant shift occurs between no noise and low noise \(ϵ=0\.1\\epsilon=0\.1\), where the reactive action percentage increases from approximately 45% to 70%\. The increase in reactive action percentage is driven mostly by a reduction in long planning, which drops from roughly 20% to 5%\. This result is consistent with the intuition that longer plans are most affected by increased noise since fewer replanning actions means errors compound more over the plan duration\. As noise increases further, this trend continues with long planning decreasing with each increase inϵ\\epsilon\. Interestingly, as noise increases fromϵ=0\.1\\epsilon=0\.1toϵ=0\.3\\epsilon=0\.3, the reactive percentage decreases slightly while short and medium planning increase slightly\. This result suggests that while the reactive policy handles high noise better than long planning, short and medium plans retain utility in out\-of\-distribution states since half of the tasks remain out\-of\-distribution for the reactive policy\. The meta\-policy therefore converges to a mixed strategy that favors reactive control more than the baselineϵ=0\\epsilon=0but sometimes uses short planning for out\-of\-distribution states under high stochasticity\.

![Refer to caption](https://arxiv.org/html/2607.16421v1/x3.png)Figure 4:Average return, reactive action percent and reactive actor uncertainty value as the meta\-policy is trained\. The reactive policy is trained on a history buffer of planner actions after every rollout\. 30 seeds per environment\. Shaded regions indicate the 95% confidence interval\. Box Push, DoorKey, and Maze only; Fetch and Navigation excluded due to compute constraints\.
### 6\.6Joint\-Training Setting

In this section we empirically evaluate the ability of our method to adapt in a joint\-training setting where the reactive actor improves alongside the meta\-agent\. To train the reactive policy, we record the state\-action pairs produced by planning options and add these to a replay buffer\. After each meta\-agent rollout, all reactive ensemble members are trained by behavior cloning on this buffer\. As joint training requires significantly longer training runs, we restrict this experiment to Box Push, DoorKey, and Maze due to compute constraints\. Training hyperparameters are reported in[Table4](https://arxiv.org/html/2607.16421#A1.T4)\.

Results are shown in[Figure4](https://arxiv.org/html/2607.16421#S6.F4)\. As training progresses, the meta\-policy shifts toward reactive control across all environments, converging to fully reactive execution as the reactive actor improves\. This is also seen in the reactive uncertainty signal, which converges to low disagreement across environments and confirms the ensemble uncertainty score reflects the strength of the reactive policy\. Together these results show the meta\-policy is able to adapt as the reactive policy learns and converge to the optimal strategy\.

An interesting property of this training setup is the reactive policy is trained on data generated from when a planning option is called\. This structure creates a learning curriculum where the meta\-agent must call planning in out\-of\-distribution states, which simultaneously produces the best immediate reward and generates the training data for the reactive policy\. Once the reactive policy has been trained on out\-of\-distribution states, the meta\-policy then recognizes the reactive policy uncertainty has decreased, revealing that the reactive policy is now competent in more states and stops calling planning options\. This shift maximizes the return and avoids overtraining the reactive policy on already in\-distribution states\.

## 7Discussion and Future Work

Our empirical results demonstrate that RL\-based meta\-policies can successfully learn adaptive computation allocation from uncertainty signals and high\-level state features across a suite of motion planning and navigation tasks\. However, several assumptions we make point toward natural future work\. First, our method assumes access to a perfect world model for planning which does not hold in most real\-world settings\. Cognitive science work on meta\-reasoning suggests that humans account for model uncertainty when deciding whether to plan\(glascher\_states\_2010\)so extending our method to incorporate learned world models with state uncertainty is a natural next step\.

Second, we constrain our environments to be single\-agent and static during plan creation\. Widening the environment suite to include tasks without such constraints and enabling the meta\-agent to interrupt executing plans would test whether the reactive\-planning tradeoff we study can generalize beyond the synchronous and single\-agent setting\.

Third, our meta\-observation relies on hand\-designed features\. While our ablations confirm these are effective, more expressive uncertainty estimates \(e\.g\., Bayesian or flow\-based methods\) or learned observation representations could improve meta\-policy generalization, particularly in high\-dimensional observation spaces such as images\.

Our joint\-training setup also suggests a promising direction for future work\. Our meta\-agent was able to track an improving reactive policy over the course of training so an interesting question is whether it can also track a*worsening*reactive policy\. This is a setting that may occur when the reactive policy has insufficient capacity to represent optimal actions in all states across a big world\(javed\_big\_nodate\)or suffers from catastrophic forgetting\.

More broadly, the reactive\-planning tradeoff is applicable to many adaptive\-computation problems\. In LLM settings, the tradeoff between fast single\-pass generation and slower chain\-of\-thought reasoning is structurally similar to the reactive\-planning tradeoff we study in this work\. Some VLA works\(team\_gemini\_2025\)adopt a similar structure with a fast and lightweight model on a robot and a larger but slower model in the cloud\. Our uncertainty\-conditioned meta\-policy provides a method for learning these allocations\.

## 8Conclusion

In this work, we presented an RL method for learning adaptive computation allocation through a meta\-reasoning policy\. Our approach conditions on reactive policy uncertainty and high\-level state features to select between a fast reactive option and planning options, where planning improves action quality at the cost of time\. Across a suite of motion planning and navigation tasks, our meta\-policy outperforms all fixed\-compute heuristic baselines when optimizing for a time\-to\-goal objective\. Ablations confirm that reactive uncertainty and observation history are the most critical components of the meta\-state\. We further showed that the learned meta\-policy adapts to environment characteristics: increasing planning cost shifts allocation toward reactive control, and increasing reactive in\-distribution coverage shifts allocation away from planning\. Finally, we demonstrated that conditioning on uncertainty signals enables the meta\-policy to converge to exclusively reactive control in a joint\-training setting where the reactive policy continues to learn\. Together, these results show that high\-level state and reactive competence signals are sufficient for a meta\-reasoning policy to effectively allocate computation across environments and training regimes\.

#### Acknowledgments

We thank Brahma Pavse and the RLC reviewers for providing helpful feedback which improved this work before and during the review process\. This work took place in the Prediction and Action Lab \(PAL\) at the University of Wisconsin–Madison\. PAL research is supported by NSF \(IIS\-2410981\) and the Wisconsin Alumni Research Foundation\. Additionally, Adam Labiosa is supported by an NSF Research Traineeship award, No\. 2152163\.

## References

Supplementary Materials

*The following content was not necessarily subject to peer review\.*

## Appendix AAppendix 1: Hyperparameters

Box PushDoorKeyFetchMazeNavigationReactive PretrainingSteps collected10,00010,0002,00010,00010,000Epochs3030303030LossCECEMSECEMSEPlannerTask horizon3025301230Short / Med / Long plan5/10/154/8/125/10/152/4/65/10/15Table 4:Environment\-specific hyperparameters for reactive pretraining and planner configuration\. CE = Cross\-Entropy, MSE = Mean Squared Error\. Batch size 64 across all environments\.HyperparameterValueJoint\-Training Reactive PolicyTrain frequency2048 stepsBuffer size10,000Epochs / Batch size3 / 64Learning rate1e\-4Table 5:Joint\-training hyperparameters\.
## Appendix BReactive Policy Architecture

Here we detail the architectures used for the reactive policy\.

Actor TypeObs\. TypeActionArchitectureImageImageDiscreteCNN:Conv\(C→\\to16, 3×\\times3\)→\\toReLU→\\toMP\(2\)MLP:FC\(n→n\\to512\)→\\toReLU→\\toFC\(512→d\\to d\)VectorTensorDiscreteMLP:FC\(din→d\_\{\\text\{in\}\}\\to512\)→\\toReLU→\\toFC\(512→\\to512\)→\\toReLU→\\toFC\(512→d\\to d\)Image\-ContImageContinuousCNN:\[Conv→\\toReLU→\\toMP\(2\)\]×\\times3→\\toAAP\(8×\\times8\)Shared:FC\(n→n\\to1024\)→\\toReLU→\\toFC\(1024→\\to512\)Head:FC\(512→a\\to a\)→\\toTanhTable 6:Model\-free actor network architectures\. Image is used for Maze\. Vector is used for BoxPush and DoorKey\. Image\-Cont is used for Fetch and Navigation\. MP = MaxPool2d, AAP = AdaptiveAvgPool2d, FC = fully\-connected,dd=features\_dim,aa=action\_dim,nn= CNN output size\.
## Appendix CSeparated Environment Ablations

Here we show the results in[Figure3](https://arxiv.org/html/2607.16421#S6.F3)non\-averaged over environments\.

![Refer to caption](https://arxiv.org/html/2607.16421v1/x4.png)Figure 5:Reactive policy in\-distribution percentage experiment\.Individual environment results when varying the percent of in\-distribution tasks for the reactive policy\.![Refer to caption](https://arxiv.org/html/2607.16421v1/x5.png)Figure 6:Plan cost experiment\.Individual environment results when varying the planning costα\\alpha\.![Refer to caption](https://arxiv.org/html/2607.16421v1/x6.png)Figure 7:Action stochasticity experiments\.Individual environment results when varying the environment stochasticity\.