Let it Cook: Learning to Wait in Sequential Decision Making

# Let it Cook:Learning to Wait in Sequential Decision Making
Source: [https://arxiv.org/html/2608.11511](https://arxiv.org/html/2608.11511)
###### Abstract

In sequential decision making, an agent typically observes its environment and acts at every timestep\. However, such active participation may not always be necessary; tasks such as brewing coffee include periods that are served equally well by letting the environment evolve without constant monitoring and control\. During such periods, the agent could simply wait to conserve its resources, or redirect its attention to another task\. We capitalize on these opportunities by training a “waiting policy” that decideswhere and how long to wait\. This involves forgoing sensing to commit to awaitaction, representing a deliberate pause for a set number of timesteps\. We formalize “learning to wait” as minimizing the frequency of sensing and decision making without sacrificing task performance \(e\.g\., the total amount of time to complete a task\)\. To train a waiting policy, we propose an approach that employs reinforcement learning with lexicographically ordered objectives\. In experiments across44discrete\-state household tasks and33continuous\-state environments, we show that our approach successfully learns waiting behaviors, and can adapt pre\-trained policies to wait where appropriate\. While different tasks permit different amounts of waiting without sacrificing task performance, our approach consistently finds solutions with significant waiting, sometimes waiting for over50%50\\%of the task duration\.

## 1Introduction

The prevailing paradigm in sequential decision making treats agents as active, closed\-loop systems that continually monitor and react to their environment\. This approach assumes that making meaningful progress towards a goal requires constant engagement from the agent—resulting in active utilization of its sensory, computational, and motor resources\. However, many real\-world tasks inherently possess passive dynamics that can be exploited to achieve goals more efficiently\([34](https://arxiv.org/html/2608.11511#bib.bib19)\)\. Consider the mundane but complex choreography of a busy kitchen: a kettle takes time to boil, a soup takes time to cook, and a coffee machine operates independently once set to brew\. In each of these scenarios, the most effective action for a human, or a robot, is to simply wait\. An agent that identifies where and how long to wait can switch to a timed “standby mode” to conserve cognitive and motor resources, or alternatively redirect these resources to an auxiliary task\. Given the benefits of strategically waiting, the key questions that arise are: How would one \(1\) formalize the objective of waiting? and \(2\) develop learning approaches that train an agent to optimize this waiting objective?

For each environment, we define a distinguishedwaitaction, e\.g\., standing still while coffee brews, or holding the paddle still in the video game ofPong\. Then "waiting" is simply the act of committing to perform thiswaitaction for multiple consecutive timesteps\. We aim to discover and exploit opportunities where the agent can wait for an extended period\. While our approach is technically agnostic to the effect thewaitaction has on the environment, we limit our experiments to settings where “waiting” corresponds to letting the environment evolve according to an intuitive notion of passive dynamics, as in the aforementioned examples\.

Learning to wait balances two potentially competing objectives: maximizing task performance and minimizing the number of times the agent has to sense and respond to the environment \(which corresponds to maximizing the amount of time the agent spends waiting\)\. We consider agents that strictly prioritize task performance \(measured by cumulative reward from a Markovian environment\) and only seek to maximize waiting when doing so does not degrade task performance\. Formally, this preference transforms the problem into a multi\-objective optimization problem with a lexicographic ordering: the agent seeks to findmax⁡\[J0,J1\]\\max\\\>\\\>\[J^\{0\},J^\{1\}\], where thecumulative task rewardJ0J^\{0\}is represents expected task performance andcumulative waiting rewardJ1=−DJ^\{1\}=\-Ddecreases with the number of timesDDthe agent is called upon to sense and make a decision \(to either wait or act\)\. For us, waiting periods involve no sensing, no computation, and no task\-oriented behavior\. MinimizingDDthus combines two desiderata: \(1\) reducing sensing and decision computation, and \(2\) reducing the need for the agent’s active interventions\. In a multi\-tasking setting, waiting may correspond to periods during which the agent could redirect its resources towards another task\. More generally, waiting periods permit switching to a timed “standby mode” that conserves sensory, computational, and motor resources\.

To exploit our objective’s structure, we apply alexicographicMulti\-Objective Reinforcement Learning \(MORL\) algorithm across several discrete\- and continuous\-state environments\. The amount of waiting possible depends on the environment; in some environments, our method yields policies that wait \(in some environments, for over 50% of the task duration\) without sacrificing task performance compared to a vanilla RL, non\-waiting baseline\. We further compare our method against a standard MORL technique that uses scalarized rewards to explore the Pareto frontier between task performance and increased waiting, mediated by a reward\-weighting scalarization coefficent\. Under a strict preference for task performance, our lexicographic MORL approach achieves near\-identical task and waiting performance to the scalarized approach, without needing a hyperparameter sweep to discover the best reward\-weighting coefficient for each environment\. We also show that our approach can be used towrapa pretrained policy to add waiting behavior without sacrificing task performance\. Finally, we demonstrate how learned waiting behavior may enable policy interleaving in a multi\-task scenario\.

## 2Waiting in Sequential Decision Making

We define awaiting Markov Decision Process \(WMDP\)to be a tupleℳ=\(𝒮,𝒜,𝑤𝑎𝑖𝑡,𝒲,𝒫,R,μ\)\\mathcal\{M\}=\(\\mathcal\{S\},\\mathcal\{A\},\\mathit\{wait\},\\mathcal\{W\},\\mathcal\{P\},R,\\mu\)where𝒮\\mathcal\{S\}is the \(discrete or continuous\) set of states,𝒜\\mathcal\{A\}is the \(discrete\) set of primitive actions,𝑤𝑎𝑖𝑡∈𝒜\\mathit\{wait\}\\in\\mathcal\{A\}is the distinguishedwaitaction,𝒲⊆ℕ\+\\mathcal\{W\}\\subseteq\\mathbb\{N\}^\{\+\}is the \(finite\) set of waiting durations,𝒫⁡\(st\+1\|st,at\)\\mathcal\{P\}\(s\_\{t\+1\}\\;\|\\;s\_\{t\},a\_\{t\}\)is the probability of transitioning fromsts\_\{t\}tost\+1s\_\{t\+1\}upon taking primitive actionata\_\{t\},R:𝒮×𝒜×𝒮→ℝR:\\mathcal\{S\}\\times\\mathcal\{A\}\\times\\mathcal\{S\}\\rightarrow\\mathbb\{R\}is the reward function, andμ\\muis the initial state distribution\. What differentiates a WMDP from a standard Markov decision process is the distinguished𝑤𝑎𝑖𝑡\\mathit\{wait\}action and the set𝒲\\mathcal\{W\}of waiting durations\. Each natural numberw∈𝒲w\\in\\mathcal\{W\}characterizes awaiting macro\-actionthat corresponds to waiting forwwconsecutive timesteps—that is, forgoing sensing and executing the primitive𝑤𝑎𝑖𝑡\\mathit\{wait\}action forwwconsecutive timesteps\.111Awaitingmacro\-action is a special kind ofsemi\-Markov optionin the terminology of[33](https://arxiv.org/html/2608.11511#bib.bib1)\.We will denote the duration\-wwwaiting macro\-action by its natural numberww\. We do not make any special requirements of the dynamics𝒫\\mathcal\{P\}under thewaitaction, however in applications we will choose thewaitaction in a way that corresponds to a natural notion of non\-intervention \(e\.g\. standing idly while soup boils on the stove, or not moving the paddle in a game ofPong\)\. We aim to learn policies that can commit to “wait” for an extended period, as illustrated in[Figure1](https://arxiv.org/html/2608.11511#S2.F1)\.

![Refer to caption](https://arxiv.org/html/2608.11511v1/concept_flat_simple.png)Figure 1:In ourCookenvironment, the agent commits to a duration\-5 waiting macro\-action near the end of the soup cooking process\. No sensing, computation, or active motion occurs while waiting\.The𝑤𝑎𝑖𝑡\\mathit\{wait\}action and set of waiting durations𝒲\\mathcal\{W\}let us define the notion of awaiting policyπ:𝒮→Δ⁡\(𝒜∪𝒲\)\{\\pi:\\mathcal\{S\}\\rightarrow\\Delta\(\\mathcal\{A\}\\cup\\mathcal\{W\}\)\}that interacts with the WMDP to receive bothtask rewardsandwaiting rewards\.222For a measurable space𝒳\\mathcal\{X\}, we writeΔ⁡\(𝒳\)\\Delta\(\\mathcal\{X\}\)to denote the set of probability measures over𝒳\\mathcal\{X\}\.When queried at a statess, the waiting policy yields either a primitive actiona∈𝒜a\\in\\mathcal\{A\}, which has its usual effect according to the dynamics𝒫\\mathcal\{P\}, or a waiting macro\-actionw∈𝒲w\\in\\mathcal\{W\}\. A WMDPℳ=\(𝒮,𝒜,𝑤𝑎𝑖𝑡,𝒲,𝒫,R,μ\)\\mathcal\{M\}=\(\\mathcal\{S\},\\mathcal\{A\},\\mathit\{wait\},\\mathcal\{W\},\\mathcal\{P\},R,\\mu\), a waiting policyπ:𝒮→Δ⁡\(𝒜∪𝒲\)\\pi:\\mathcal\{S\}\\rightarrow\\Delta\(\\mathcal\{A\}\\cup\\mathcal\{W\}\), and a finite time horizonH∈ℕH\\in\\mathbb\{N\}induce a distribution over length\-HHtrajectoriess0​a0​r10​r11​s1​…​aH−1​rH0​rH1​sHs\_\{0\}a\_\{0\}r^\{0\}\_\{1\}r^\{1\}\_\{1\}s\_\{1\}\\ldots a\_\{H\-1\}r^\{0\}\_\{H\}r^\{1\}\_\{H\}s\_\{H\}generated by drawing an initial states0∼μs\_\{0\}\\sim\\muand querying the policyπ\\piat timestep00to obtain either a waiting macro\-action or a primitive action\. At each timesteptt, if the policy is queried and yields a waiting macro\-actionwwthen the nextwwactionsat,at\+1,…,at\+w−1a\_\{t\},a\_\{t\+1\},\\ldots,a\_\{t\+w\-1\}are each the waiting action𝑤𝑎𝑖𝑡\\mathit\{wait\}and the policy is not queried again until timestept\+wt\+w\. If instead the the queried policy yields a primitive actiona∈𝒜a\\in\\mathcal\{A\}thenat=aa\_\{t\}=aand the policy is queried at the next timestept\+1t\+1\. For each timestep1≤t≤H1\\leq t\\leq H, the statests\_\{t\}is sampled asst∼𝒫\(⋅\|st−1,at−1\)s\_\{t\}\{\\sim\}\\mathcal\{P\}\(\\cdot\\;\|\\;s\_\{t\-1\},a\_\{t\-1\}\)and thetask rewardrt0r^\{0\}\_\{t\}is defined asrt0=R⁡\(st−1,at−1,st\)r^\{0\}\_\{t\}=R\(s\_\{t\-1\},a\_\{t\-1\},s\_\{t\}\), which matches the usual notion of reward in an MDP\. Thewaiting rewardrt1r^\{1\}\_\{t\}is−1\-1if the policy is queried at timestepttand00otherwise\. When the maximum episode horizonHHis reached, the episode is abruptly terminated, regardless of whether the agent is in the middle of a waiting macro\-action\.

Given a trajectoryτ=s0​a0​r10​r11​s1​…​aH−1​rH0​rH1​sH\\tau=s\_\{0\}a\_\{0\}r^\{0\}\_\{1\}r^\{1\}\_\{1\}s\_\{1\}\\ldots a\_\{H\-1\}r^\{0\}\_\{H\}r^\{1\}\_\{H\}s\_\{H\}we define the \(undiscounted\)cumulative task rewardto beJ0​\(τ\)=∑1≤i≤Hri0J^\{0\}\(\\tau\)=\\sum\_\{1\\leq i\\leq H\}r^\{0\}\_\{i\}\. We define the \(undiscounted\)cumulative waiting rewardto beJ1​\(τ\)=∑1≤i≤Hri1J^\{1\}\(\\tau\)=\\sum\_\{1\\leq i\\leq H\}r^\{1\}\_\{i\}, i\.e\.,−1\-1times the number of times the policyπ\\piis queried during the trajectory\.

##### Lexicographic waiting objective\.

Intuitively, a good waiting policy is one that obtains good task performance \(as measured by cumulative task reward\) while also waiting as much as possible\. Since we consider trajectories with ana priorifixed horizonHHin which the only extended\-duration actions are thewaiting macro\-actions, querying the waiting policy fewer times corresponds to more waiting\. Given a WMDPℳ\\mathcal\{M\}and a finite time horizonHH, we define the vector\-valued objective of a waiting policyπ\\piasJ⁡\(π\)∈ℝ2J\(\\pi\)\\in\\mathbb\{R\}^\{2\}as:

J⁡\(π\)=𝔼τ∼ℳ,π​\[J0​\(τ\),J1​\(τ\)\]J\(\\pi\)=\\mathbb\{E\}\_\{\\tau\\sim\\mathcal\{M\},\\pi\}\[J^\{0\}\(\\tau\),J^\{1\}\(\\tau\)\]\(1\)We adopt the standard lexicographic ordering≤\\leqover objective values defined such that\[J0,J1\]≤\[J0′,J1′\]⇔J0<J0′​OR​\(J0=J0′​AND​J1≤J1′\)\[\{J^\{0\}\},\{J^\{1\}\}\]\\leq\[\{J^\{0\}\}^\{\\prime\},\{J^\{1\}\}^\{\\prime\}\]\\Leftrightarrow\{J^\{0\}\}<\{J^\{0\}\}^\{\\prime\}\\,\\text\{OR\}\\,\(\{J^\{0\}\}=\{J^\{0\}\}^\{\\prime\}\\,\\text\{AND\}\\,\{J^\{1\}\}\\leq\{J^\{1\}\}^\{\\prime\}\)\. In other words, a policy is “better” than another policy if and only if it either \(1\) achieves higher cumulative task rewardJ0J^\{0\}, or \(2\) it achieves equalJ0J^\{0\}and higher cumulative waiting rewardJ1J^\{1\}\. In the following section we will describe our RL\-based approach that seeks to learn a policyπ∗\\pi^\{\*\}that achieves maximalJ⁡\(π∗\)J\(\\pi^\{\*\}\)\.

Our choice of waiting objective—to minimize expected number of times the waiting policy is queried—is a natural fit for settings where active decision making incurs a computational or sensory cost\. This idea was explored by[36](https://arxiv.org/html/2608.11511#bib.bib21), which uses scalar rewards that include aninteraction costto encourage action repetition, and[13](https://arxiv.org/html/2608.11511#bib.bib2), which uses the analogousdeliberation costto reduce the number of times a high\-level policy is queried in hierarchical reinforcement learning\. In the aforementioned works, the magnitude of the \(interaction or deliberation\) cost implicitly defines a tradeoff between task performance and policy query frequency\. In contrast, our lexicographic objective \([1](https://arxiv.org/html/2608.11511#S2.E1)\) renders the relative magnitudes of the task rewards and the waiting rewards inconsequential\. In other words, if we were to define WMDP trajectories such that eachrt1r^\{1\}\_\{t\}were scaled by an arbitrary positive factor, the objective in \([1](https://arxiv.org/html/2608.11511#S2.E1)\) would induce the same ordering over optimal policies and our LQ\-Learning learning algorithm \([Algorithm1](https://arxiv.org/html/2608.11511#algorithm1), which we introduce in[Section3\.1](https://arxiv.org/html/2608.11511#S3.SS1)\) would behave similarly\. Practically, this alleviates the need to perform a hyperparameter sweep to find an appropriate weighting of task reward vs\. waiting reward\.

Another important distinguishing characteristic of our formalization is that the only way to avoid querying the policy is by selecting the temporally extended waiting macro\-actions\. Thus, our objective naturally encourages the agent to pick those actions\. Further, since “wait” actions in our settings cede control to the environment’s passive dynamics, more waiting translates to conserving not only sensing and computation, but also motor resources\. Penalizing all decisions, including the choice to commit to waiting, encourages the agent to takelonger durationwaiting actions when possible\. Preferring long contiguous durations of waiting can enable downstream multitasking applications where a long waiting period provides more time for the agent to “fill in” the waiting period by executing an auxiliary policy to make progress towards another task; we explore a simple instantiation of this idea in the context of theCoffeetask in[Section4](https://arxiv.org/html/2608.11511#S4)\.

## 3Learning to Wait

### 3\.1Lexicographic Q\-Learning

Our goal, to maximize the vector\-valued objective \([1](https://arxiv.org/html/2608.11511#S2.E1)\), lends itself naturally to alexicographicMORL approach\([7](https://arxiv.org/html/2608.11511#bib.bib5);[32](https://arxiv.org/html/2608.11511#bib.bib4)\)that seeks a policy that maximizes a vector\-valued reward signal\. We adapt the lexicographic version of Q\-Learning described by[32](https://arxiv.org/html/2608.11511#bib.bib4)to handle waiting macro\-actions\. Our resulting LQ\-Learning algorithm, which we detail in[Algorithm1](https://arxiv.org/html/2608.11511#algorithm1), maintains two Q\-estimates: the task Q\-estimateQ0Q^\{0\}and waiting Q\-estimateQ1Q^\{1\}\. Each Q\-estimate considers the “full action set”𝒜∪𝒲\\mathcal\{A\}\\cup\\mathcal\{W\}, which contains both primitive actions and waiting macro\-actions\. Greedy policy inference selects an action that isσ\\sigma\-close to optimal with respect toQ0Q^\{0\}’s future cumulative task reward estimate, breaking ties according toQ1Q^\{1\}’s future cumulative waiting reward estimate, whereσ∈ℝ\+\\sigma\\in\\mathbb\{R\}^\{\+\}is a small tolerance parameter that softens the lexicographic comparison\. The tolerance parameter prevents a slightly inaccurate empiricalQ0Q^\{0\}estimate from erroneously ruling out the lexicographically optimal action during theQ1Q^\{1\}update in[Algorithm1](https://arxiv.org/html/2608.11511#algorithm1)of[Algorithm1](https://arxiv.org/html/2608.11511#algorithm1)\. We only update the Q estimates in response to the output \(an element of𝒜∪𝒲\\mathcal\{A\}\\cup\\mathcal\{W\}\) made by the policy when the policy is queried\. We do not perform updates based on the primitive𝑤𝑎𝑖𝑡\\mathit\{wait\}actions applied in the middle of a waiting macro\-action, because these do not directly correspond to decisions made by the policy\. This is analogous to how[33](https://arxiv.org/html/2608.11511#bib.bib1)definesSMDP Q\-Learning\.

1

Input:WMDP

ℳ=\(𝒮,𝒜,𝑤𝑎𝑖𝑡,𝒲,𝒫,R,μ\)\\mathcal\{M\}=\(\\mathcal\{S\},\\mathcal\{A\},\\mathit\{wait\},\\mathcal\{W\},\\mathcal\{P\},R,\\mu\), episode horizon

HH, learning rate

α\\alpha, lexicographic tolerance

σ\\sigma, initial Q tables

Q0,Q1∈ℝ\|𝒮\|×\|𝒜∪𝒲\|Q^\{0\},Q^\{1\}\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\\times\|\\mathcal\{A\}\\cup\\mathcal\{W\}\|\}\.

2

for each*episode*do3

t←0,s←ℳ\.reset\(\)t\\leftarrow 0,\\;s\\leftarrow\\mathcal\{M\}\.\\texttt\{reset\(\)\}4

5

while*t<Ht<H*do6

u←\{Lex\-Argmax​\(s,Q0,Q1,σ\)w\.p\.​1−ϵUniform⁡\(𝒜∪𝒲\)w\.p\.​ϵu\\leftarrow\\begin\{cases\}\\texttt\{Lex\-Argmax\}\(s,Q^\{0\},Q^\{1\},\\sigma\)&\\text\{w\.p\. \}1\-\\epsilon\\\\ \\mathrm\{Uniform\}\(\\mathcal\{A\}\\cup\\mathcal\{W\}\)&\\text\{w\.p\. \}\\epsilon\\end\{cases\}//epsilon\-greedy exploration7

8

if*u∈𝒲u\\in\\mathcal\{W\}*then//Waiting macro\-action9

G0←0G^\{0\}\\leftarrow 0//Task reward during macro\-action10

for*uutimes*do//Take u primitive wait steps11

s′,r0←ℳ\.step​\(𝑤𝑎𝑖𝑡\)s^\{\\prime\},r^\{0\}\\leftarrow\\mathcal\{M\}\.\\texttt\{step\}\(\\mathit\{wait\}\)12

G0←G0\+r0G^\{0\}\\leftarrow G^\{0\}\+r^\{0\}13

else//Primitive action14

15

s′,r0←ℳ\.step​\(u\)s^\{\\prime\},r^\{0\}\\leftarrow\\mathcal\{M\}\.\\texttt\{step\}\(u\)16

G0←r0G^\{0\}\\leftarrow r^\{0\}17

18

G1←−1G^\{1\}\\leftarrow\-1//Waiting reward: \-1 per policy query19

20

//Update Q estimates

Q0​\[s,u\]←\(1−α\)​Q0​\[s,u\]\+α⁡\(G0\+maxv∈𝒜∪𝒲⁡Q0​\[s′,v\]\)Q^\{0\}\[s,u\]\\leftarrow\(1\-\\alpha\)\\,Q^\{0\}\[s,u\]\+\\alpha\\Bigl\(G^\{0\}\+\\max\_\{v\\in\\mathcal\{A\}\\cup\\mathcal\{W\}\}Q^\{0\}\[s^\{\\prime\},v\]\\Bigr\)21

22

Q1​\[s,u\]←\(1−α\)​Q1​\[s,u\]\+α⁡\(G1\+Q1​\[s′,Lex\-Argmax​\(s′,Q0,Q1,σ\)\]\)Q^\{1\}\[s,u\]\\leftarrow\(1\-\\alpha\)\\,Q^\{1\}\[s,u\]\+\\alpha\\Bigl\(G^\{1\}\+Q^\{1\}\[s^\{\\prime\},\\texttt\{Lex\-Argmax\}\(s^\{\\prime\},Q^\{0\},Q^\{1\},\\sigma\)\]\\Bigr\)23

s←s′,t←t\+1s\\leftarrow s^\{\\prime\},\\;t\\leftarrow t\+124

Function*Lex\-Argmax\(s,Q0,Q1,σ\)\(s,Q^\{0\},Q^\{1\},\\sigma\)*:25

U←\{u∈𝒜∪𝒲\|Q0​\[s,u\]≥maxu′∈𝒜∪𝒲⁡Q0​\[s,u′\]−σ\}U\\leftarrow\\Bigl\\\{u\\in\\mathcal\{A\}\\cup\\mathcal\{W\}\\ \\Big\|\\ Q^\{0\}\[s,u\]\\geq\\max\_\{u^\{\\prime\}\\in\\mathcal\{A\}\\cup\\mathcal\{W\}\}Q^\{0\}\[s,u^\{\\prime\}\]\-\\sigma\\Bigr\\\}26

return

arg​maxu∈U⁡Q1​\[s,u\]\\argmax\_\{u\\in U\}Q^\{1\}\[s,u\]27

28

Algorithm 1LQ\-Learning
### 3\.2Learning A Waiting Wrapper For a Pretrained Policy

Thus far, we have focused on training a policyfrom scratch\. In some settings, there may already exist a policyπ𝑏𝑎𝑠𝑒:𝒮→Δ⁡\(𝒜\)\\pi\_\{\\mathit\{base\}\}:\\mathcal\{S\}\\rightarrow\\Delta\(\\mathcal\{A\}\)that achieves satisfactory cumulative \(task\) reward in a non\-waiting MDP environment\. Such a policy may be handwritten, or trained via e\.g\. imitation learning or RL\. Direct deployment ofπ𝑏𝑎𝑠𝑒\\pi\_\{\\mathit\{base\}\}in a WMDP333Deployingπ𝑏𝑎𝑠𝑒:𝒮→Δ⁡\(𝒜\)\\pi\_\{\\mathit\{base\}\}:\\mathcal\{S\}\{\\rightarrow\}\\Delta\(\\mathcal\{A\}\)in a WMDP with actions𝒜\\mathcal\{A\}and wait durations𝒲\\mathcal\{W\}assumes the tacit lift toπ𝑏𝑎𝑠𝑒:𝒮→Δ⁡\(𝒜∪𝒲\)\\pi\_\{\\mathit\{base\}\}:\\mathcal\{S\}\\rightarrow\\Delta\(\\mathcal\{A\}\\cup\\mathcal\{W\}\)\.would garner expected cumulative task rewardJ0​\(π𝑏𝑎𝑠𝑒\)J^\{0\}\(\\pi\_\{\\mathit\{base\}\}\)equal to expected cumulative reward in a non\-waiting MDP, but extremely low cumulative waiting reward\. We will now describe how to train a waiting policy that serves as a wrapper aroundπ𝑏𝑎𝑠𝑒\\pi\_\{\\mathit\{base\}\}\. This approach enjoys efficient exploration because the wrapper policy has a small set of possible actions: either executeπ𝑏𝑎𝑠𝑒\\pi\_\{\\mathit\{base\}\}to evolve the system under its controlled dynamics, or wait and let it evolve passively\.

Given a WMDPℳ=\(𝒮,𝒜,𝑤𝑎𝑖𝑡,𝒫,R,μ\)\\mathcal\{M\}=\(\\mathcal\{S\},\\mathcal\{A\},\\mathit\{wait\},\\mathcal\{P\},R,\\mu\)and a base policyπ𝑏𝑎𝑠𝑒:𝒮→Δ⁡\(𝒜\)\\pi\_\{\\mathit\{base\}\}:\\mathcal\{S\}\\rightarrow\\Delta\(\\mathcal\{A\}\)we define the induced two\-action WMDPℳ′=\(𝒮,\{a𝑏𝑎𝑠𝑒,𝑤𝑎𝑖𝑡\},𝒫′,R,μ\)\\mathcal\{M\}^\{\\prime\}=\(\\mathcal\{S\},\\\{a\_\{\\mathit\{base\}\},\\mathit\{wait\}\\\},\\mathcal\{P\}^\{\\prime\},R,\\mu\)wherea𝑏𝑎𝑠𝑒∉𝒜a\_\{\\mathit\{base\}\}\\notin\\mathcal\{A\}is a new primitive action that “simulates”π𝑏𝑎𝑠𝑒\\pi\_\{\\mathit\{base\}\}\. The dynamics are defined such that∀s,s′∈𝒮\\forall s,s^\{\\prime\}\\in\\mathcal\{S\}

𝒫′​\(s′\|s,𝑤𝑎𝑖𝑡\)=𝒫⁡\(s′\|s,𝑤𝑎𝑖𝑡\)𝒫′​\(s′\|s,a𝑏𝑎𝑠𝑒\)=∑a∈𝒜\(𝒫⁡\(s′\|s,a\)​π𝑏𝑎𝑠𝑒​\(a\|s\)\)\\mathcal\{P\}^\{\\prime\}\(s^\{\\prime\}\\;\|\\;s,\\mathit\{wait\}\)=\\mathcal\{P\}\(s^\{\\prime\}\\;\|\\;s,\\mathit\{wait\}\)\\qquad\\qquad\\mathcal\{P\}^\{\\prime\}\(s^\{\\prime\}\\;\|\\;s,a\_\{\\mathit\{base\}\}\)=\\sum\_\{a\\in\\mathcal\{A\}\}\\left\(\\mathcal\{P\}\(s^\{\\prime\}\\;\|\\;s,a\)\\pi\_\{\\mathit\{base\}\}\(a\\;\|\\;s\)\\right\)A waiting policyπ:𝒮→Δ⁡\(\{a𝑏𝑎𝑠𝑒,𝑤𝑎𝑖𝑡\}∪𝒲\)\\pi:\\mathcal\{S\}\\rightarrow\\Delta\(\\\{a\_\{\\mathit\{base\}\},\\mathit\{wait\}\\\}\\cup\\mathcal\{W\}\)can choose to simulateπ𝑏𝑎𝑠𝑒\\pi\_\{\\mathit\{base\}\}by taking actiona𝑏𝑎𝑠𝑒a\_\{\\mathit\{base\}\}or wait\. Learning a waiting wrapper around a pre\-trained policyπ𝑏𝑎𝑠𝑒\\pi\_\{\\mathit\{base\}\}may be useful in several settings\. When training waiting policies from scratch requires extensive environmental interaction then taking advantage of an existing base policyπ𝑏𝑎𝑠𝑒\\pi\_\{\\mathit\{base\}\}\(trained without waiting considerations\) may significantly reduce the amount of environmental interaction needed to obtain a policy that achieves satisfactory value of[Equation1](https://arxiv.org/html/2608.11511#S2.E1)\. For example, in domains where large pre\-trained policies exist, such as in robotic manipulation, \(e\.g\.[25](https://arxiv.org/html/2608.11511#bib.bib26);[4](https://arxiv.org/html/2608.11511#bib.bib28);[3](https://arxiv.org/html/2608.11511#bib.bib25);[24](https://arxiv.org/html/2608.11511#bib.bib27)\) it may be unnecessary to train a policy from scratch\. Using the method described in this section, we may exploit the pre\-trained base policy’s task performance while also exploring its waiting affordances for improved sensory, computational, and motor resource efficiency during task execution\. Of course, it is possible that there exists no wrapper waiting policyπ\\piwith expected objective value that exceedsJ⁡\(s↦δa𝑏𝑎𝑠𝑒\)J\(s\\mapsto\\delta\_\{a\_\{\\mathit\{base\}\}\}\)\. In our experimental results, we observed non\-trivial waiting behaviors even when the base policyπ𝑏𝑎𝑠𝑒\\pi\_\{\\mathit\{base\}\}was trained through standard RL \(with no additional care to make it robust in states that may be reached via extended\-duration waiting\)\.

## 4Experiments

We compare the effectiveness of our LQ\-Learning approach \([Section3\.1](https://arxiv.org/html/2608.11511#S3.SS1)\) and a scalarized reward baseline \(which we will describe in[Section4\.2](https://arxiv.org/html/2608.11511#S4.SS2)\), both in the context of learning a waiting policy from scratch and wrapping a pre\-existing policy to wait as much as possible without sacrificing expected cumulative task reward\. Finally, we show a case study of how learning to wait can be used as a first step toward interleaving policies to complete multiple tasks efficiently\.

### 4\.1Environments

Each of our environments \(with the exception ofCartPole\) is a goal\-reaching environment: we consider the task reward to be−1\-1per timestep, which encourages the agent to reach the goal as fast as possible\. We also includeearly episode terminationif the agent reaches the goal before thea prioriepisode horizon is reached\. While our choice of task reward allows us to visualize cumulative task reward and the number of timesteps in which the policy is queried \(i\.e\., zero minus the cumulative waiting reward\) on a single set of axes, we note that our approach also applies to settings where cumulative task reward does not enjoy any special correlation to a number of elapsed timesteps\. Our environments are described below, with additional details in[AppendixA](https://arxiv.org/html/2608.11511#A1)\. In all of our environments the agent observes the complete state of the environment; in the cooking and coffee preparation tasks this includes the amount of timesteps remaining for a soup to finish cooking, the coffee machine to finish heating, or the coffee to finish brewing\.

##### Cook\.

We study three variations of a cooking task in theovercooked\-ai\([5](https://arxiv.org/html/2608.11511#bib.bib10)\), which simulates the popular video gameOvercooked\([9](https://arxiv.org/html/2608.11511#bib.bib11)\)in which the agent must cook and serve dishes\. In ourCookenvironment the agent must gather and place 3 onions in the pot, which starts an 18 timestep countdown until the soup is cooked and ready\. The agent must also pick up a dish, use the dish to pick up the soup, and then deliver the soup to the goal location\. InCookLongerthe soup takes 36 timesteps to cook, and inCookTwice, the agent must cook and deliver two soups, each of which takes 18 timesteps to cook\. The𝑤𝑎𝑖𝑡\\mathit\{wait\}action makes the agent stand still and we consider𝒲=\{5,15\}\\mathcal\{W\}=\\\{5,15\\\}\.

##### Coffee\.

We implement aCoffeebrewing task in a deterministic 2D MiniGrid gridworld\([6](https://arxiv.org/html/2608.11511#bib.bib12)\)\. The agent must go to the coffee machine andtoggleit to start it heating\. After an 8 timestep countdown the agent must toggle the machine again to start coffee brewing, which triggers an 18 timestep countdown, after which the agent must toggle the coffee machine again to collect the coffee\. In addition to the multi\-phase interaction with the coffee machine, the agent must go to andtogglethe cream and sugar to pick them up\. The episode terminates \(success\) when the coffee, cream, and sugar are collected \(in any order\)\. The𝑤𝑎𝑖𝑡\\mathit\{wait\}action makes the agent stand still and we consider𝒲=\{5,15\}\\mathcal\{W\}=\\\{5,15\\\}\.

##### Pong\.

OurPongenvironment is a modified version of the Atari game Pong, rewritten with inspiration from the Gymnax\([19](https://arxiv.org/html/2608.11511#bib.bib22)\)implementation\. The agent plays against a scripted opponent, and its objective is to win one point as quickly as possible\. If the ego agent loses a point, the episode does not terminate but the ball restarts in the middle of the arena\. The agent observes a continuous\-valued low\-dimensional vector observation comprising each paddle’s position, the ball’s current and last timestep positions, and the ball’s current velocity\. The𝑤𝑎𝑖𝑡\\mathit\{wait\}action does not move the paddle and we consider𝒲=\{2,4,8,16,32,64,128\}\\mathcal\{W\}=\\\{2,4,8,16,32,64,128\\\}\.

##### MountainCar\.

We adopt Gymnasium’s\([35](https://arxiv.org/html/2608.11511#bib.bib23)\)MountainCar, in which a car must climb up from a valley to reach the top of a mountain as fast as possible\. This is only possible by gradually building up momentum over multiple back\-and\-forth passes\. Unlike our other environments, even small amounts of waiting hinder task performance\. The agent observes the car’s position and velocity\. The𝑤𝑎𝑖𝑡\\mathit\{wait\}action applies zero force to the car \(which may still roll freely\) and we consider𝒲=\{2,4,6,8,10,12,14,16,18,20\}\\mathcal\{W\}=\\\{2,4,6,8,10,12,14,16,18,20\\\}\.

##### Cartpole\.

We adapt Gymnasium’s\([35](https://arxiv.org/html/2608.11511#bib.bib23)\)Cartpoleenvironment\. We add an additional primitive action𝑤𝑎𝑖𝑡\\mathit\{wait\}to standard CartPole’s 2\-action set \(“left” and “right”\)\. Unlike our other environments, the goal is to balance the pole as long as possible, to the maximum horizonH=200H=200; we use a task reward of\+1\+1per timestep\. The agent observes the cart’s position and velocity, and the pole’s angle and angular velocity\. We consider𝒲=\{2,4,6,8,10,12,14,16,18,20\}\\mathcal\{W\}=\\\{2,4,6,8,10,12,14,16,18,20\\\}\.

### 4\.2Scalar reward baseline

A simpler alternative to LQ\-Learning uses a coefficientλ\\lambdato scalarize each task rewardr0r^\{0\}and waiting rewardr1r^\{1\}into a combined rewardr0\+λ​r1r^\{0\}\+\\lambda r^\{1\}\. To train a policy that optimizes[Equation1](https://arxiv.org/html/2608.11511#S2.E1), one must perform a hyperparameter sweep over values ofλ\\lambdain each new environment\. Our LQ\-Learning approach does not require such hyperparameter tuning, however lexicographic MORL approaches \(such as LQ\-Learning\) require specific policy architectures and algorithms \(e\.g\. those presented in[32](https://arxiv.org/html/2608.11511#bib.bib4)\) that are less mature than scalar\-reward RL approaches\. For a scalarization coefficientλ∈ℝ≥0\\lambda\\in\\mathbb\{R\}^\{\\geq 0\}we define theλ\\lambda\-scalarized expected cumulative reward of a policy \(for a fixed waiting MDP and horizon\) to beJλ​\(π\)=J0​\(π\)\+λ​J1​\(π\)J\_\{\\lambda\}\(\\pi\)=J^\{0\}\(\\pi\)\+\\lambda J^\{1\}\(\\pi\)In order to train a policy that optimizesJλJ\_\{\\lambda\}we apply the non\-lexicographic version of the SMDP Q\-Learning algorithm\([33](https://arxiv.org/html/2608.11511#bib.bib1)\)defined similarly to our LQ\-Learning \([Section3\.1](https://arxiv.org/html/2608.11511#S3.SS1)\) except there is only one Q\-estimate and the \(scalarized\) reward at timestepttisrt=rt0\+λ​rt1r\_\{t\}=r^\{0\}\_\{t\}\+\\lambda r^\{1\}\_\{t\}\. Whenλ=0\{\\lambda=0\}we recover the original task reward’s objective that does not incentivize waiting\. Whenλ\\lambdais extremely large, there will always be an optimal policy that waits as much as possible, regardless of the task rewards\.

### 4\.3Implementation details

We apply LQ\-Learning and Q\-Learning with scalar rewards to the discreteCook,CookTwice,CookLongerandCoffeetasks\. We apply their deep RL analogs LDQN \(details in[SectionB\.1](https://arxiv.org/html/2608.11511#A2.SS1)\) and DQN\([21](https://arxiv.org/html/2608.11511#bib.bib16)\)to the continuous\-state environments\. We adapt the Q\-Learning and DQN implementations found in SKRL\([29](https://arxiv.org/html/2608.11511#bib.bib24)\)to LQ\-Learning and LDQN following[32](https://arxiv.org/html/2608.11511#bib.bib4)\. Our DQN\-based experiments use Q\-networks that are MLPs with ReLU activations and 2 hidden 256 neurons per layer\. Further hyperparameter details can be found in[AppendixB](https://arxiv.org/html/2608.11511#A2)\. For the scalar reward baseline, we sweep over theλ\\lambdavalues0,0\.01,0\.1,0\.5,1,5,10,150,0\.01,0\.1,0\.5,1,5,10,15\. We report the result of the bestλ\\lambdavalue with respect to[Equation1](https://arxiv.org/html/2608.11511#S2.E1), additional results are found in[SectionC\.1](https://arxiv.org/html/2608.11511#A3.SS1)\.

### 4\.4Results

##### To what extent can an agent wait?

We first study the extent to which an agent can wait without sacrificing task performance across our suite of tasks\. We apply our LQ\-Learning/LDQN approach \([Section3\.1](https://arxiv.org/html/2608.11511#S3.SS1)\)\. Our comparison in[Figure3](https://arxiv.org/html/2608.11511#S4.F3)shows that LQ\-Learning and LDQN consistently yield policies that achieve highcumulative task reward\(close to that obtained by vanilla RL that maximizes cumulative task reward as its sole objective without the option to wait\), so a quick summary of the waiting affordances of the task exploited by our approach is possible through[Figure2](https://arxiv.org/html/2608.11511#S4.F2), which plots the proportion of episode timesteps spent in different duration waiting macro\-action by our LQ\-Learning and LDQN policies\.

Intuitively,CookandCoffee\(visualized in[Figure3](https://arxiv.org/html/2608.11511#S4.F3)\) would present opportunities for the agent to wait \(e\.g\. during the time that soup takes to cook or coffee takes to brew\) without sacrificing task performance \(here, time to goal\)\. Indeed, inCookthe learned policy chooses the duration55wait action twice during the 33 timesteps taken to fetch ingredients, cook, and serve the soup\. At first it seems counterintuitive that the agent does not take the duration1515waiting action given that the soup takes1818timesteps to cook, however our LQ\-Learning\-learned policy is in fact optimal with respect to[Equation1](https://arxiv.org/html/2608.11511#S2.E1)\. Close inspection of the learned policy’s behavior reveals that our LQ\-Learning approach discovers a policy that fetches the dish during the time the soup is cooking: this permits optimal cumulative task reward, and an optimal value of[Equation1](https://arxiv.org/html/2608.11511#S2.E1), yet precludes the opportunity to wait for 15 uninterrupted timesteps\.444Any policy that performs a duration\-15 waiting macro action would take at least 37 timesteps to completeCook\.Similarly, inCoffee, LQ\-Learning learns a clever policy that achieves a higher value of[Equation1](https://arxiv.org/html/2608.11511#S2.E1)than possible by naively waiting: the learned policy fetches sugar as the coffee brews\.

Training a waiting wrapper policy around a pre\-existing policy \([Section3\.2](https://arxiv.org/html/2608.11511#S3.SS2)\) can expose different opportunities to wait without sacrificing task reward than when learning a waiting policy from scratch\. We see this when we train a waiting wrapper policy to wrap a \(suboptimal\) handwritten policy in our tabular environments: Our naive handwrittenCookbase policy stands idly by as the soup cooks; our wrapper waiting policy trained via LQ\-Learning successfully learns to take the duration\-15 wait during this idle period\. Similarly, our wrapper waiting policy forCoffeeperforms a duration\-5 and a duration\-15 wait as the machine heats and brews, respectively\. In our continuous\-state environments, we wrap a policy trained using vanilla DQN\. Surprisingly, even these natural RL\-trained afford the ability to exercise waiting behaviors\. For example, our learned waiting wrapper policies forPongandCartpolepreserve the DQN\-trained base policy’s perfect cumulative task reward while spending well over half of the timesteps committed to waiting\. This suggests that learning to wait, conveniently applied as a wrapperoverpre\-trained policies, could query the base policy more selectively without reducing task performance\.

![Refer to caption](https://arxiv.org/html/2608.11511v1/April_scratch_vs_wrap_bettermc.png)Figure 2:Empirical performance averaged over 1000 trajectories from the best trained policy of 10 random training seeds using LQ\-Learning or LDQN\. The bar segments show the proportion of timesteps spent waiting vs\. actively engaging with the environment\. Bar heights are normalized; the numbers within each segment denote the absolute number of timesteps and the number at the top denotes absolute episode length\.![Refer to caption](https://arxiv.org/html/2608.11511v1/Figure3_ArXiV.png)Figure 3:\(Left\) Environment visualizations\. In ourCoffeeinterleaving case study, the duration\-15 wait taken while coffee brews permits us to interleave a “fetch sugar” policy that follows the dashed green path\. \(Right\) Empirical mean episode length \(height of bar\) and number of decisions made \(height of green bar segment\) of our approach \(“Lex” denotes LQ\-Learning for discrete environments, LDQN for others\) and the best scalarized approach from our sweep overλ\\lambdavalues\. Episode length is negative cumulative task reward for most environments; forCartPoleepisode length is cumulative task reward\. We also include results from vanilla RL \(Q\-Learning for discrete environments, DQN for others\) that is trained on task reward without the option to wait\. Within each environment, heights are normalized with respect to the vanilla RL baseline\. Labels denote absolute numbers of timesteps\. Error bars represents±\\pm1 standard deviation, computed over 10 random initial policy training seeds\.
##### LQ\-Learning is a good way to learn waiting \(compared to the scalar reward baseline\)\.

We now investigate the extent to which our LQ\-Learning/LDQN approach accurately finds a policy that toes the fine line between increasing waiting and sacrificing task performance\.[Figure3](https://arxiv.org/html/2608.11511#S4.F3)visualizes episode duration \(which directly reflects cumulative task reward\) and number of decisions made \(zero minus cumulative waiting reward\) for three approaches for each task: \(1\.\) Vanilla RL that maximizes task performance with no access to waiting options, \(2\.\) Our LQ\-Learning/LDQN approach, and \(3\.\) The scalarized objective approach with the best \(as determined by[Equation1](https://arxiv.org/html/2608.11511#S2.E1)\) weighting coefficient found in our sweep overλ\\lambdavalues\. A policy that maximizes[Equation1](https://arxiv.org/html/2608.11511#S2.E1)should yield episode length approximately equal to that achieved by Vanilla RL, and should wait as much or more than the scalarized approach\. Our LQ\-Learning and LDQN policies in[Figure3](https://arxiv.org/html/2608.11511#S4.F3)indeed approximate this expected behavior\.

##### Learning a waiting policy wrapper is sample efficient\.

We find it often takes fewer training steps to saturate the performance of a wrapper waiting policy than a waiting policy trained from scratch\. For example, inPongtraining a waiting wrapper policy via LDQN converges after∼250​k\{\\sim\}250\\text\{k\}training environment steps, while training a waiting policy from scratch with LDQN typically takes more than600​k600\\text\{k\}steps \([Figure4](https://arxiv.org/html/2608.11511#S4.F4)\)\. This gain in sample efficiency may be due to \(1\) fewer actions being exposed to the wrapper policy and \(2\) a stark dichotomy between states from which the base policy can perform well vs\. poorly, which means the waiting wrapper consistently receives low \(relative to not waiting\) future cumulative task reward when it waits inappropriately\. Such a gain in sample efficiency need not exist in general, however learning curves in[AppendixC](https://arxiv.org/html/2608.11511#A3)show efficiency gains in each environment exceptMountainCar\.

![Refer to caption](https://arxiv.org/html/2608.11511v1/April_pong_curves_improved_clipped.png)Figure 4:\(Left\)Environment visualizations\.\(Right\)Learning curve comparison for our LDQN applied to train a waiting policy forPongfrom scratch vs\. as a wrapper around a frozen pretrained policy\. Each curve is the mean of 10 random policy training seeds, shaded regions denote 95% confidence interval\.
##### Waiting enables downstream interleaving\.

We now describe a simple form ofpolicy interleavingas a downstream application of learning to wait\. Recall from[Figure2](https://arxiv.org/html/2608.11511#S4.F2)that our approach adapts a handwritten base policy that completes theCoffeetask \(by first turning on the coffee machine, then brewing coffee, then collecting sugar and cream\) in 54 timesteps into a waiting policy that completes the task in the same amount of timesteps, but which commits to waiting\. We now assume access to two additional handwritten policies, one for fetching sugar and another for fetching cream\. Each of these auxiliary policies, when started from a location on the grid, brings the agent to collect its respective item, and returns to the grid position from which the auxiliary policy was invoked\. We also assume access to accurate predictions of how long the auxiliary task will take to execute\.

We then interleave as follows: if the waiting policy selects a duration\-NNwait, we check the time\-to\-completion estimates for each not\-yet\-completed auxiliary task and execute the longest\-duration auxiliary task that can be completed inNNtimesteps\. Our waiting wrapper policy’s duration\-15 waiting gap is long enough to fetch the sugar; performing this interleaved behavior results allows the agent to completeCoffeein a total of 45 timesteps, matching the amount of time taken to complete the task by vanilla Q\-Learning applied to learning how to maximize task performance from scratch \([Figure3](https://arxiv.org/html/2608.11511#S4.F3)\)\.

## 5Related Work

##### Cost of Agent Intervention\.

Prior research\([12](https://arxiv.org/html/2608.11511#bib.bib8);[15](https://arxiv.org/html/2608.11511#bib.bib14);[23](https://arxiv.org/html/2608.11511#bib.bib13);[36](https://arxiv.org/html/2608.11511#bib.bib21);[17](https://arxiv.org/html/2608.11511#bib.bib9)\)demonstrates how imposing costs on sensing and intervention compel the agent to adopt open\-loop strategies in which the agent commits to a sequence of actions without intermediate feedback, typically by relying on internal model of the environment until uncertainty necessitates sensing\. Closest in spirit to our work is[36](https://arxiv.org/html/2608.11511#bib.bib21), in which the agent selects both an action and a number of times to repeat the action, although they do not explore how adjusting the magnitude of their scalarinteraction costtrades off between task performance and policy query frequency\. While we also encourage the agent to sense and deliberate infrequently, we focus on a stricter notion of non\-intervention formalized by the distinguishedwaitaction\. Committing to taking the wait action, i\.e\., surrendering control to the system’s passive dynamics, is qualitatively different than committing to take a predetermined sequence of arbitrary actions because \(1\.\) waiting may conserve motor resources \(e\.g\. the energy needed to walk through a kitchen\) and \(2\.\) periods of waiting can be “filled in” with a policy that pursues an auxiliary goal, under appropriate conditions on the dynamics under the waiting action and the auxiliary goal policy\.

##### Temporal Abstractions\.

RL has long utilized temporal abstractions to improve efficiency in complex domains\([33](https://arxiv.org/html/2608.11511#bib.bib1);[26](https://arxiv.org/html/2608.11511#bib.bib18);[16](https://arxiv.org/html/2608.11511#bib.bib17)\)\. Grounded in bounded rationality\([31](https://arxiv.org/html/2608.11511#bib.bib20)\),[13](https://arxiv.org/html/2608.11511#bib.bib2)argue that agents should identify temporally extended options that effectively minimize thedeliberation costassociated with option selection\. This is similar in spirit to[Equation1](https://arxiv.org/html/2608.11511#S2.E1)\. In particular,[13](https://arxiv.org/html/2608.11511#bib.bib2)seek to limit the number of times a high\-level policy over options may be queried and propose a scalarized objective based on the Lagrangian formulation \(analogous to our scalarized RL baseline[Section4\.2](https://arxiv.org/html/2608.11511#S4.SS2)\) of this constrained optimization problem\. However, a drawback compared to our proposed approach based on lexicographic RL is the need for extensive hyperparameter sweeps to identify appropriate weighting of deliberation cost \(which corresponds to theλ\\lambdavalue of our scalar approach\) relative to the cumulative task reward\.Hierarchical RL with Timed Subgoals\([11](https://arxiv.org/html/2608.11511#bib.bib3)\)introduces fine\-grained temporal abstraction by letting a high\-level policy callspecify when each call to a low\-level policy should terminate\. This is similar in spirit to dynamically\-determined action repeat\([18](https://arxiv.org/html/2608.11511#bib.bib15);[30](https://arxiv.org/html/2608.11511#bib.bib32);[2](https://arxiv.org/html/2608.11511#bib.bib33)\)where, in addition to choosing a primitive action, the agent chooses a number timesteps to repeat that primitive action\. While these approaches reduce decision frequency in practice, their focus is reducing the amount of environmental interaction \(sample complexity\) needed to train a performant policy\. This contrasts with our objective, which explicitly seeks to minimize the number of decisions the agent makes during deployment and privileges extendedwaitingas opposed to extended application of arbitrary primitive actions\.

##### Multi\-objective RL\.

Multi\-objective MDPs \(MOMDPs\) are often employed when a single scalar reward cannot capture the trade\-offs between competing desiderata\([28](https://arxiv.org/html/2608.11511#bib.bib6);[14](https://arxiv.org/html/2608.11511#bib.bib7)\)\. While optimizing a scalarized objective can make use of standard RL techniques, the objective itself requires precise tuning of weights to navigate the Pareto front\. In contrast, lexicographic approaches\([7](https://arxiv.org/html/2608.11511#bib.bib5);[32](https://arxiv.org/html/2608.11511#bib.bib4)\)allow for a strict prioritization of objectives, which is well\-suited for our setting in which we wish to wait as much as possible without sacrificing expected cumulative task reward\. This is closely related to the motivation of constrained policy optimization in safe RL\([8](https://arxiv.org/html/2608.11511#bib.bib31);[10](https://arxiv.org/html/2608.11511#bib.bib30)\); however, rather than choose ana prioriconstraint on the minimal acceptable task performance the lexicographic approach allows the agent to adaptively discover what the best possible task performance and optimize waiting subject to this implicit constraint\.

## 6Discussion

##### Summary\.

In this paper we formulated learning to wait as maximizing[Equation1](https://arxiv.org/html/2608.11511#S2.E1), which encourages the agent to spend long durations committed to apply thewaitaction without sensing, to the extent possible without sacrificing expected cumulative task reward\. Lexicographic MORL allows the agent to discover the optimal expected cumulative task reward, alleviating the need for a human to perform hyperparameter sweeps over a reward weighting coefficient\. We can thus train a policy, or a wrapper around an existing policy, that waits as much as possible without sacrificing task performance\. Waiting inherently saves sensory, computational, and motor resources and may be a useful first step towards multi\-task policy interleaving\.

##### Future work\.

Currently, our approach does not leverage the inherentstructureof extended duration waiting\. Future work could improve sample efficiency during training by e\.g\. fitting a single\-step dynamics model to multi\-timestep waiting experiences, taking inspiration from[36](https://arxiv.org/html/2608.11511#bib.bib21)\. Secondly, we only expect our techniques to work well infully observedenvironments\. If, for example, the agent’s observations did not differentiate when the soup in theCooktask was just starting to cook vs\. when it was almost ready, the agent would not be able to learn appropriate waiting behavior\. An exciting direction of future work would be to learnmemoryfulwaiting policies\. In many practical settings \(e\.g\. cooking\) the task\-relevant latent dynamics are highly correlated with the passage of time; which may permit accurate deterministic belief updates\. Finally, we presented only an initial proof\-of\-concept case study of policy interleaving\. To truly expose opportunities to perform auxiliary tasks would require a stringent definition of what actions cause the aspects of the environment’s state that are relevant to the primary task to evolve as they would under the waiting action\. Without such restrictions, the progress made towards the auxiliary task could impede the completion of the primary task\.

#### Acknowledgments

This project was funded by NSF SLES 2331783, NSF CAREER 2239301, ONR N00014\-22\-1\-2677, DARPA TIAMAT HR00112490421, and a gift from Amazon AWS to the ASSET center at Penn\.

## References

- Bartoet al\.\(1983\)A\. G\. Barto, R\. S\. Sutton, and C\. W\. AndersonNeuronlike adaptive elements that can solve difficult learning control problems\.IEEE Transactions on Systems, Man, and CyberneticsSMC\-13\(5\),pp\. 834–846\.External Links:[Document](https://dx.doi.org/10.1109/TSMC.1983.6313077)Cited by:[Appendix A](https://arxiv.org/html/2608.11511#A1.SS0.SSS0.Px4.p1.1)\.
- Biedenkappet al\.\(2021\)A\. Biedenkapp, R\. Rajan, F\. Hutter, and M\. LindauerTempoRL: learning when to act\.InProceedings of the 38th International Conference on Machine Learning,M\. Meila and T\. Zhang \(Eds\.\),Proceedings of Machine Learning Research, Vol\.139,pp\. 914–924\.Cited by:[§5](https://arxiv.org/html/2608.11511#S5.SS0.SSS0.Px2.p1.1)\.
- Blacket al\.\(2026\)K\. Black, N\. Brown, D\. Driess, A\. Esmail, M\. Equi, C\. Finn, N\. Fusai, L\. Groom, K\. Hausman, B\. Ichter, S\. Jakubczak, T\. Jones, L\. Ke, S\. Levine, A\. Li\-Bell, M\. Mothukuri, S\. Nair, K\. Pertsch, L\. X\. Shi, J\. Tanner, Q\. Vuong, A\. Walling, H\. Wang, and U\. Zhilinskyπ0\\pi\_\{0\}: A vision\-language\-action flow model for general robot control\.External Links:2410\.24164Cited by:[§3\.2](https://arxiv.org/html/2608.11511#S3.SS2.p2.2)\.
- Brohanet al\.\(2023\)A\. Brohan, N\. Brown, J\. Carbajal, Y\. Chebotar, X\. Chen, K\. Choromanski, T\. Ding, D\. Driess, A\. Dubey, C\. Finn, P\. Florence, C\. Fu, M\. G\. Arenas, K\. Gopalakrishnan, K\. Han, K\. Hausman, A\. Herzog, J\. Hsu, B\. Ichter, A\. Irpan, N\. Joshi, R\. Julian, D\. Kalashnikov, Y\. Kuang, I\. Leal, L\. Lee, T\. E\. Lee, S\. Levine, Y\. Lu, H\. Michalewski, I\. Mordatch, K\. Pertsch, K\. Rao, K\. Reymann, M\. Ryoo, G\. Salazar, P\. Sanketi, P\. Sermanet, J\. Singh, A\. Singh, R\. Soricut, H\. Tran, V\. Vanhoucke, Q\. Vuong, A\. Wahid, S\. Welker, P\. Wohlhart, J\. Wu, F\. Xia, T\. Xiao, P\. Xu, S\. Xu, T\. Yu, and B\. ZitkovichRT\-2: vision\-language\-action models transfer web knowledge to robotic control\.External Links:2307\.15818Cited by:[§3\.2](https://arxiv.org/html/2608.11511#S3.SS2.p2.2)\.
- Carrollet al\.\(2019\)M\. Carroll, R\. Shah, M\. K\. Ho, T\. Griffiths, S\. Seshia, P\. Abbeel, and A\. DraganOn the utility of learning about humans for human\-ai coordination\.InAdvances in Neural Information Processing Systems,H\. Wallach, H\. Larochelle, A\. Beygelzimer, F\. d'Alché\-Buc, E\. Fox, and R\. Garnett \(Eds\.\),Vol\.32,pp\.\.Cited by:[Appendix A](https://arxiv.org/html/2608.11511#A1.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2608.11511#S4.SS1.SSS0.Px1.p1.1)\.
- Chevalier\-Boisvertet al\.\(2023\)M\. Chevalier\-Boisvert, B\. Dai, M\. Towers, R\. de Lazcano, L\. Willems, S\. Lahlou, S\. Pal, P\. S\. Castro, and J\. TerryMinigrid & miniworld: modular & customizable reinforcement learning environments for goal\-oriented tasks\.CoRRabs/2306\.13831\.Cited by:[Appendix A](https://arxiv.org/html/2608.11511#A1.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2608.11511#S4.SS1.SSS0.Px2.p1.1)\.
- Gáboret al\.\(1998\)Z\. Gábor, Z\. Kalmár, and C\. SzepesváriMulti\-criteria reinforcement learning\.InProceedings of the Fifteenth International Conference on Machine Learning,ICML ’98,San Francisco, CA, USA,pp\. 197–205\.External Links:ISBN 1558605568Cited by:[§3\.1](https://arxiv.org/html/2608.11511#S3.SS1.p1.1),[§5](https://arxiv.org/html/2608.11511#S5.SS0.SSS0.Px3.p1.1)\.
- García and Fernández \(2015\)J\. García and F\. FernándezA comprehensive survey on safe reinforcement learning\.Journal of Machine Learning Research16\(42\),pp\. 1437–1480\.Cited by:[§5](https://arxiv.org/html/2608.11511#S5.SS0.SSS0.Px3.p1.1)\.
- Ghost Town Games \(2016\)Ghost Town GamesOvercooked\.Cited by:[§4\.1](https://arxiv.org/html/2608.11511#S4.SS1.SSS0.Px1.p1.1)\.
- Guet al\.\(2024\)S\. Gu, L\. Yang, Y\. Du, G\. Chen, F\. Walter, J\. Wang, and A\. KnollA review of safe reinforcement learning: methods, theories, and applications\.IEEE Transactions on Pattern Analysis and Machine Intelligence46\(12\),pp\. 11216–11235\.External Links:[Document](https://dx.doi.org/10.1109/TPAMI.2024.3457538)Cited by:[§5](https://arxiv.org/html/2608.11511#S5.SS0.SSS0.Px3.p1.1)\.
- Gürtleret al\.\(2021\)N\. Gürtler, D\. Büchler, and G\. MartiusHierarchical reinforcement learning with timed subgoals\.InAdvances in Neural Information Processing Systems,A\. Beygelzimer, Y\. Dauphin, P\. Liang, and J\. W\. Vaughan \(Eds\.\),Cited by:[§5](https://arxiv.org/html/2608.11511#S5.SS0.SSS0.Px2.p1.1)\.
- Hansenet al\.\(1996\)E\. Hansen, A\. Barto, and S\. ZilbersteinReinforcement learning for mixed open\-loop and closed\-loop control\.InAdvances in Neural Information Processing Systems,M\.C\. Mozer, M\. Jordan, and T\. Petsche \(Eds\.\),Vol\.9,pp\.\.Cited by:[§5](https://arxiv.org/html/2608.11511#S5.SS0.SSS0.Px1.p1.1)\.
- Harbet al\.\(2018\)J\. Harb, P\. Bacon, M\. Klissarov, and D\. PrecupWhen waiting is not an option: learning options with a deliberation cost\.Proceedings of the AAAI Conference on Artificial Intelligence32\(1\)\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v32i1.11831)Cited by:[§2](https://arxiv.org/html/2608.11511#S2.SS0.SSS0.Px1.p2.1),[§5](https://arxiv.org/html/2608.11511#S5.SS0.SSS0.Px2.p1.1)\.
- Hayeset al\.\(2022\)C\. F\. Hayes, R\. Rădulescu, E\. Bargiacchi, J\. Källström, M\. Macfarlane, M\. Reymond, T\. Verstraeten, L\. M\. Zintgraf, R\. Dazeley, F\. Heintz, E\. Howley, A\. A\. Irissappane, P\. Mannion, A\. Nowé, G\. Ramos, M\. Restelli, P\. Vamplew, and D\. M\. RoijersA practical guide to multi\-objective reinforcement learning and planning\.Autonomous Agents and Multi\-Agent Systems36\(1\)\.External Links:ISSN 1387\-2532,[Document](https://dx.doi.org/10.1007/s10458-022-09552-y)Cited by:[§5](https://arxiv.org/html/2608.11511#S5.SS0.SSS0.Px3.p1.1)\.
- Heemelset al\.\(2012\)W\.P\.M\.H\. Heemels, K\.H\. Johansson, and P\. TabuadaAn introduction to event\-triggered and self\-triggered control\.In2012 IEEE 51st IEEE Conference on Decision and Control \(CDC\),Vol\.,pp\. 3270–3285\.External Links:[Document](https://dx.doi.org/10.1109/CDC.2012.6425820)Cited by:[§5](https://arxiv.org/html/2608.11511#S5.SS0.SSS0.Px1.p1.1)\.
- Hutsebaut\-Buysseet al\.\(2022\)M\. Hutsebaut\-Buysse, K\. Mets, and S\. LatréHierarchical reinforcement learning: a survey and open research challenges\.Machine Learning and Knowledge Extraction4\(1\),pp\. 172–221\.External Links:ISSN 2504\-4990,[Document](https://dx.doi.org/10.3390/make4010009)Cited by:[§5](https://arxiv.org/html/2608.11511#S5.SS0.SSS0.Px2.p1.1)\.
- Krishnaet al\.\(2025\)A\. Krishna, E\. S\. Hu, and D\. JayaramanThe value of sensory information to a robot\.ICLR\.Cited by:[§5](https://arxiv.org/html/2608.11511#S5.SS0.SSS0.Px1.p1.1)\.
- Lakshminarayananet al\.\(2017\)A\. S\. Lakshminarayanan, S\. Sharma, and B\. RavindranDynamic action repetition for deep reinforcement learning\.InProceedings of the Thirty\-First AAAI Conference on Artificial Intelligence,AAAI’17,pp\. 2133–2139\.Cited by:[§5](https://arxiv.org/html/2608.11511#S5.SS0.SSS0.Px2.p1.1)\.
- Lange \(2022\)gymnax: a JAX\-based reinforcement learning environment libraryCited by:[Appendix A](https://arxiv.org/html/2608.11511#A1.SS0.SSS0.Px3.p1.1),[§4\.1](https://arxiv.org/html/2608.11511#S4.SS1.SSS0.Px3.p1.1)\.
- Machadoet al\.\(2018\)M\. C\. Machado, M\. G\. Bellemare, E\. Talvitie, J\. Veness, M\. Hausknecht, and M\. BowlingRevisiting the arcade learning environment: evaluation protocols and open problems for general agents \(extended abstract\)\.InProceedings of the Twenty\-Seventh International Joint Conference on Artificial Intelligence, IJCAI\-18,pp\. 5573–5577\.External Links:[Document](https://dx.doi.org/10.24963/ijcai.2018/787)Cited by:[Appendix A](https://arxiv.org/html/2608.11511#A1.SS0.SSS0.Px3.p1.1)\.
- Mnihet al\.\(2015\)V\. Mnih, K\. Kavukcuoglu, D\. Silver, A\. A\. Rusu, J\. Veness, M\. G\. Bellemare, A\. Graves, M\. A\. Riedmiller, A\. K\. Fidjeland, G\. Ostrovski, S\. Petersen, C\. Beattie, A\. Sadik, I\. Antonoglou, H\. King, D\. Kumaran, D\. Wierstra, S\. Legg, and D\. HassabisHuman\-level control through deep reinforcement learning\.Nature518,pp\. 529–533\.Cited by:[§4\.3](https://arxiv.org/html/2608.11511#S4.SS3.p1.1)\.
- Moore \(1990\)A\. W\. MooreEfficient memory\-based learning for robot control\.Technical reportUniversity of Cambridge\.Cited by:[Appendix A](https://arxiv.org/html/2608.11511#A1.SS0.SSS0.Px5.p1.1)\.
- Nagaharaet al\.\(2016\)M\. Nagahara, D\. E\. Quevedo, and D\. NešićMaximum hands\-off control: a paradigm of control effort minimization\.IEEE Transactions on Automatic Control61\(3\),pp\. 735–747\.External Links:[Document](https://dx.doi.org/10.1109/TAC.2015.2452831)Cited by:[§5](https://arxiv.org/html/2608.11511#S5.SS0.SSS0.Px1.p1.1)\.
- NVIDIAet al\.\(2025\)NVIDIA, J\. Bjorck, N\. C\. Fernando Castañeda, X\. Da, R\. Ding, L\. "\. Fan, Y\. Fang, D\. Fox, F\. Hu, S\. Huang, J\. Jang, Z\. Jiang, J\. Kautz, K\. Kundalia, L\. Lao, Z\. Li, Z\. Lin, K\. Lin, G\. Liu, E\. Llontop, L\. Magne, A\. Mandlekar, A\. Narayan, S\. Nasiriany, S\. Reed, Y\. L\. Tan, G\. Wang, Z\. Wang, J\. Wang, Q\. Wang, J\. Xiang, Y\. Xie, Y\. Xu, Z\. Xu, S\. Ye, Z\. Yu, A\. Zhang, H\. Zhang, Y\. Zhao, R\. Zheng, and Y\. ZhuGR00T N1: an open foundation model for generalist humanoid robots\.InArXiv Preprint,External Links:2503\.14734Cited by:[§3\.2](https://arxiv.org/html/2608.11511#S3.SS2.p2.2)\.
- Octo Model Teamet al\.\(2024\)Octo Model Team, D\. Ghosh, H\. Walke, K\. Pertsch, K\. Black, O\. Mees, S\. Dasari, J\. Hejna, C\. Xu, J\. Luo, T\. Kreiman, Y\. L\. Tan, L\. Y\. Chen, P\. Sanketi, Q\. Vuong, T\. Xiao, D\. Sadigh, C\. Finn, and S\. LevineOcto: an open\-source generalist robot policy\.InProceedings of Robotics: Science and Systems,Delft, Netherlands\.Cited by:[§3\.2](https://arxiv.org/html/2608.11511#S3.SS2.p2.2)\.
- Pateriaet al\.\(2021\)S\. Pateria, B\. Subagdja, A\. Tan, and C\. QuekHierarchical reinforcement learning: a comprehensive survey\.ACM Comput\. Surv\.54\(5\)\.External Links:ISSN 0360\-0300,[Document](https://dx.doi.org/10.1145/3453160)Cited by:[§5](https://arxiv.org/html/2608.11511#S5.SS0.SSS0.Px2.p1.1)\.
- Raffin \(2020\)A\. RaffinRL baselines3 zoo\.GitHub\.Note:[https://github\.com/DLR\-RM/rl\-baselines3\-zoo](https://github.com/DLR-RM/rl-baselines3-zoo)Cited by:[Appendix B](https://arxiv.org/html/2608.11511#A2.p4.1)\.
- Roijerset al\.\(2013\)D\. M\. Roijers, P\. Vamplew, S\. Whiteson, and R\. DazeleyA survey of multi\-objective sequential decision\-making\.J\. Artif\. Int\. Res\.48\(1\),pp\. 67–113\.External Links:ISSN 1076\-9757Cited by:[§5](https://arxiv.org/html/2608.11511#S5.SS0.SSS0.Px3.p1.1)\.
- Serrano\-Muñozet al\.\(2023\)A\. Serrano\-Muñoz, D\. Chrysostomou, S\. Bøgh, and N\. Arana\-ArexolaleibaSkrl: modular and flexible library for reinforcement learning\.Journal of Machine Learning Research24\(254\),pp\. 1–9\.Cited by:[§B\.1](https://arxiv.org/html/2608.11511#A2.SS1.p1.1),[§4\.3](https://arxiv.org/html/2608.11511#S4.SS3.p1.1)\.
- Sharmaet al\.\(2017\)S\. Sharma, A\. S\. Lakshminarayanan, and B\. RavindranLearning to repeat: fine grained action repetition for deep reinforcement learning\.InInternational Conference on Learning Representations,Cited by:[§5](https://arxiv.org/html/2608.11511#S5.SS0.SSS0.Px2.p1.1)\.
- Simon \(1957\)H\. SimonA behavioral model of rational choice\.Models of man, social and rational: Mathematical essays on rational human behavior in a social setting6\(1\),pp\. 241–260\.Cited by:[§5](https://arxiv.org/html/2608.11511#S5.SS0.SSS0.Px2.p1.1)\.
- Skalseet al\.\(2022\)J\. Skalse, L\. Hammond, C\. Griffin, and A\. AbateLexicographic multi\-objective reinforcement learning\.InProceedings of the Thirty\-First International Joint Conference on Artificial Intelligence, IJCAI\-22,L\. D\. Raedt \(Ed\.\),pp\. 3430–3436\.Note:Main TrackExternal Links:[Document](https://dx.doi.org/10.24963/ijcai.2022/476)Cited by:[§B\.1](https://arxiv.org/html/2608.11511#A2.SS1.p1.1),[§3\.1](https://arxiv.org/html/2608.11511#S3.SS1.p1.1),[§4\.2](https://arxiv.org/html/2608.11511#S4.SS2.p1.1),[§4\.3](https://arxiv.org/html/2608.11511#S4.SS3.p1.1),[§5](https://arxiv.org/html/2608.11511#S5.SS0.SSS0.Px3.p1.1)\.
- Suttonet al\.\(1999\)R\. S\. Sutton, D\. Precup, and S\. SinghBetween mdps and semi\-mdps: a framework for temporal abstraction in reinforcement learning\.Artificial Intelligence112\(1\),pp\. 181–211\.External Links:ISSN 0004\-3702Cited by:[§B\.1](https://arxiv.org/html/2608.11511#A2.SS1.p1.1),[§3\.1](https://arxiv.org/html/2608.11511#S3.SS1.p1.1),[§4\.2](https://arxiv.org/html/2608.11511#S4.SS2.p1.1),[§5](https://arxiv.org/html/2608.11511#S5.SS0.SSS0.Px2.p1.1),[footnote 1](https://arxiv.org/html/2608.11511#footnote1)\.
- Todorov \(2009\)E\. TodorovEfficient computation of optimal actions\.Proceedings of the national academy of sciences106\(28\),pp\. 11478–11483\.Cited by:[§1](https://arxiv.org/html/2608.11511#S1.p1.1)\.
- Towerset al\.\(2024\)M\. Towers, A\. Kwiatkowski, J\. Terry, J\. U\. Balis, G\. De Cola, T\. Deleu, M\. Goulão, A\. Kallinteris, M\. Krimmel, A\. KG,et al\.Gymnasium: a standard interface for reinforcement learning environments\.arXiv preprint arXiv:2407\.17032\.Cited by:[Appendix A](https://arxiv.org/html/2608.11511#A1.SS0.SSS0.Px4.p1.1),[§4\.1](https://arxiv.org/html/2608.11511#S4.SS1.SSS0.Px4.p1.1),[§4\.1](https://arxiv.org/html/2608.11511#S4.SS1.SSS0.Px5.p1.1)\.
- Zhouet al\.\(2024\)H\. Zhou, A\. Huang, K\. Azizzadenesheli, D\. Childers, and Z\. LiptonTiming as an action: learning when to observe and act\.InInternational Conference on Artificial Intelligence and Statistics,pp\. 3979–3987\.Cited by:[§2](https://arxiv.org/html/2608.11511#S2.SS0.SSS0.Px1.p2.1),[§5](https://arxiv.org/html/2608.11511#S5.SS0.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2608.11511#S6.SS0.SSS0.Px2.p1.1)\.

Supplementary Materials

*The following content was not necessarily subject to peer review\.*

## Appendix AAdditional Environment Details

##### Cook\.

TheCook,CookLonger, andCookTwice, environments are implemented using[5](https://arxiv.org/html/2608.11511#bib.bib10)\. All three variants use the same kitchen layout shown in[Figure3](https://arxiv.org/html/2608.11511#S4.F3)\. The environment has deterministic 2D grid dynamics: the agent has 6 actions available: go left, go right, go up, go down, stay=𝑤𝑎𝑖𝑡\\mathit\{wait\}, or interact\. Interact will pick up items \(onion, bowl, finished soup\) or put down the object the agent is holding\. In[Figure1](https://arxiv.org/html/2608.11511#S2.F1)we write “turn\-left” to mean go left, and “take\-soup” to mean interact\. The agent’s discrete observation space includes the agent’s 2D position, orientation, whether the agent is holding an \{onion, dish, ready soup, uncooked soup\}, the pot phase \(“empty”, “has some but not all ingredients”, “is cooking”, or “is done”\), how many ingredients are in the pot, and the time remaining for a currently cooking soup to be done\. The maximum episode horizonHHis 200 for each variant\.

##### Coffee\.

TheCoffeeenvironment is a 2D gridworld implemented using Minigrid\([6](https://arxiv.org/html/2608.11511#bib.bib12)\)\. The discrete state is defined by the agent’s position, orientation \(north, south, east, or west\), whether the sugar has been collected or not, whether the cream has been collected or not, the progress of the coffee machine towards heating up \(not toggled yet, the number of timesteps remaining to heat, or that it is fully heated\), the progress of the coffee machine towards brewing \(not toggled yet, the number of timesteps remaining to brew, or that the coffee is brewed and ready to collect\), and whether the coffee has been collected or not\. The agent observes the complete state at each timestep\. We use the standard set of 7 discrete actions available to a Minigrid agent but define custom behavior \(indicated in parentheses where appropriate\): “turn left”, “turn right”, “go forward”, “pickup” \(no\-op\), “drop” \(no\-op\), “toggle”, and “done”=𝑤𝑎𝑖𝑡\\mathit\{wait\}\(we redefine “done” to make the agent stay in place with no special effect on episode termination\)\. The “toggle” action is used for all interactions with the world \(starting coffee machine to warm up, starting coffee machine to brew, collecting coffee, collecting cream, collecting sugar\)\. The maximum episode horizon isH=200H=200\. A truly optimal policy completesCoffeein 41 timesteps, but none of our learning based approaches learned the optimal behavior during our allotted training duration \(10m timesteps\); our best policies need 45 timesteps\.

In our interleaving case study, the “fetch sugar” and “fetch cream” sub\-policies are handwritten optimal policies that, when called for a particular agent position and orientation, make the agent go to the sugar \(resp\. cream\) location, “toggle” to pick up the sugar \(resp\. cream\), and return the agent to the position and orientation from which the subpolicy was called\. It might seem counterintuitive that our simple scheduler does not choose to execute the “fetch cream” policy during the duration\-5 wait that the wrapper waiting policy \(wrapped around our handwritten policy for the entireCoffeetask\) executes while waiting for the coffee machine to heat up\. This is because it would take more than 5 timesteps to collect the creamand return to the position and orientation from the start of the duration\-5 wait\.Overall task completion would be faster if the agent were able to fetch the cream while the coffee machine heats up; future work could explore more sophisticated forms of policy interleaving to enable such behavior\.

##### Pong\.

We adapt Gymnax’s\([19](https://arxiv.org/html/2608.11511#bib.bib22)\)implementation of the well\-known Pong environment\. The agent’s primitive action space𝒜\\mathcal\{A\}comprises four actions:move\_paddle\_upmove\_paddle\_down,𝑤𝑎𝑖𝑡\\mathit\{wait\}\(which leaves the paddle in place\), andhitwhich leaves the paddle in place and returns the ball if contact would be made\. We include the nonstandardhitaction to make the connection between waiting and not actively affecting the environment thematically clear; our techniques would work equally well with standard pong dynamics\. The goal is to score one point as fast as possible against a scripted opponent; the maximum episode horizon isH=1000H=1000\. OurPongenvironment includes the standard “sticky actions” introduced by[20](https://arxiv.org/html/2608.11511#bib.bib29)to introduce a small amount of stochasticity into transitions\. This makes the environment very slightly non\-Markovian with respect to the observations supplied to the agent\.

##### CartPole\.

We adapt the standardCartPole\-v0\([1](https://arxiv.org/html/2608.11511#bib.bib34)\)environment from Gymnasium\([35](https://arxiv.org/html/2608.11511#bib.bib23)\)to have an additional𝑤𝑎𝑖𝑡\\mathit\{wait\}action that applies no force to the cart\. Thus the action set is \{“left”, “right”,𝑤𝑎𝑖𝑡\\mathit\{wait\}\}\. The maximum episode horizon isH=200H=200and the task reward is\+1\+1for each step\. We retain the typical termination logic from the original implementation\.

##### MountainCar\.

We use the standardMountainCar\-v0\([22](https://arxiv.org/html/2608.11511#bib.bib35)\)environment implementation from Gymnasium\. There are three primitive actions: “accelerate to the left”, “don’t accelerate”=𝑤𝑎𝑖𝑡\\mathit\{wait\}, and “accelerate to the right\.” The task reward is \-1 per timestep and the maximum episode horizon isH=200\.H=200\.

## Appendix BTraining Details

During training, we terminate training episodes once thea prioriepisode horizon is reached or \(in the case of goal\-reaching tasks\) when the goal is reached\. ForCartPolewe inherit the usual termination behavior when the pole’s angle becomes excessively low\.

For each ofCook,CookLonger,CookTwice, andCoffeewe train each policy for 10 million environment steps\. For each of our continuous state tasks we train each policy for 1 million timesteps\.

When selecting the best model checkpoint of a training run, we use the environment’s cumulative reward \(for LQ\-Learning and LDQN we use the lexicographic ordering over the task and waiting rewards, for the other approaches we use the usual ordering over the \(scalarized or vanilla\) reward\)\. We evaluate an intermediate checkpoint every 10k training environment steps, and evaluate 20 eval episodes \(using greedy, not epsilon\-greedy policy inference\) to obtain an empirical estimate of the expected cumulative reward\.

We report the hyperparameters used for all experiments\. Lexicographic Q\-learning parameters are shown in Table[1](https://arxiv.org/html/2608.11511#A2.T1), and lexicographic DQN parameters are shown in Table[2](https://arxiv.org/html/2608.11511#A2.T2)\. DQN hyperparameters were adapted from the RL Baselines3 Zoo\([27](https://arxiv.org/html/2608.11511#bib.bib36)\)and manually tuned for each environment\.

### B\.1Lexicographic RL Implementation

We adapt the Q\-Learning and DQN implementations from SKRL\([29](https://arxiv.org/html/2608.11511#bib.bib24)\)into the LQ\-Learning and LDQN algorithms following[32](https://arxiv.org/html/2608.11511#bib.bib4)\. We also add support for durative macro\-actions as described for theSMDP Q\-Learningalgorithm in[33](https://arxiv.org/html/2608.11511#bib.bib1)\. Durative macro\-actions were not part of the standard MOMDP environments treated by[32](https://arxiv.org/html/2608.11511#bib.bib4)or the base implementation of SKRL\.

Our Q\-Learning and LQ\-Learning implementations do not make use of a replay buffer\. Our DQN and LDQN implementations do use a replay buffer: we insert durative\-action transitions into the replay buffer in the straightforward way, that is, we include the state from which the action was taken, the \(durative\) action, the state reached after finishing \(or reaching episode termination\) the durative action, and the cumulative reward accrued during the low\-level transitions within the durative action\.

Table 1:LQ\-Learning hyperparameters\.Table 2:Lexicographic DQN hyperparameters by environment\.

## Appendix CLearning Curves

We now present learning curves for the training runs reported in[Figure2](https://arxiv.org/html/2608.11511#S4.F2)and[Figure3](https://arxiv.org/html/2608.11511#S4.F3)\. Note that[Figure3](https://arxiv.org/html/2608.11511#S4.F3)presents just the best \(in terms[Equation1](https://arxiv.org/html/2608.11511#S2.E1)for LQ\-Learning and LDQN and in terms of scalarized return for scalarized approaches\) policy checkpoint, which may appear before the max number of training environment steps is reached\. In the learning curves, the bold line shows the average of ten training seeds\. The shaded region shows 1 standard deviation\. For some environments, notablyCookTwice, policy optimization is unstable, meaning that averaging performance across ten random seeds does not clearly reflect the performance of each individual seed’s policy\.

For all approaches, we provide learning curves for episode length \(negativeJ0J^\{0\}for most environments; positiveJ0J^\{0\}forCartPole\)\. This characterizes the cumulative task reward for our lexicographic MORL and the cumulative reward for vanilla RL\. For all approaches except vanilla RL, we also include a curve for number of decisions taken \(negativeJ1J^\{1\}\)\. This characterizes the cumulative waiting reward for LQ\-Learning and LDQN approaches\. For the scalarized approaches we also include a learning curve for the scalarized return, which is always a linear combination ofJ0J^\{0\}andJ1J^\{1\}weighted by the coefficientλ\\lambdaas described in[Section4\.2](https://arxiv.org/html/2608.11511#S4.SS2)\.

Figure 5:Episode length vs\. training timesteps for learning from scratch experiments \(see[Figure3](https://arxiv.org/html/2608.11511#S4.F3)\)\. Episode length equals negative cumulative task return for most environments and equals positive task return forCartPole\. Each curve is the mean of 10 random policy training seeds; shaded regions show±\\pm1 standard deviation\. Each point along the curve is the empirical mean over 20 evaluation trajectories \(with greedy policy inference\); evaluation occurs once every 10,000 environment steps during training\.Figure 6:Decisions per episode vs\. training timesteps for learning from scratch experiments \(see[Figure3](https://arxiv.org/html/2608.11511#S4.F3)\)\. Decisions per episode is negative cumulative waiting return for all environments\. Each curve is the mean of 10 random policy training seeds; shaded regions show±\\pm1 standard deviation\. Each point along the curve is the empirical mean over 20 evaluation trajectories \(with greedy policy inference\); evaluation occurs once every 10,000 environment steps during training\.![Refer to caption](https://arxiv.org/html/2608.11511v1/figures_final/camera_ready_learning_curves/july_fixed_scalar_curves_pong.png)Figure 7:Per\-episode cumulative scalar reward vs\. training timesteps for learning from scratch experiments \(see[Figure3](https://arxiv.org/html/2608.11511#S4.F3)\)\. Each curve is the mean of 10 random policy training seeds; shaded regions represents±\\pm1 standard deviation\. Each point along the curve is the empirical mean over 20 evaluation trajectories \(with greedy policy inference\); evaluation occurs once every 10,000 environment steps during training\. We only report curves for the best scalarization coefficientλ\\lambda, as described in[Section4](https://arxiv.org/html/2608.11511#S4)\.Figure 8:Episode length vs\. training timesteps for learning from wrapper waiting policy experiments \(see[Figure2](https://arxiv.org/html/2608.11511#S4.F2)\)\. Episode length equals negative task return for most environments and equals positive task return forCartPole\. Each curve is the mean of 10 random policy training seeds; shaded regions represents±\\pm1 standard deviation\. Each point along the curve is the empirical mean over 20 evaluation trajectories \(with greedy policy inference\); evaluation occurs once every 10,000 environment steps during training\.Figure 9:Decisions per episode vs\. training timesteps for learning from wrapper waiting policy experiments \(see[Figure2](https://arxiv.org/html/2608.11511#S4.F2)\)\. Decisions per episode is negative waiting return for all environments\. Each curve is the mean of 10 random policy training seeds; shaded regions represents±\\pm1 standard deviation\. Each point along the curve is the empirical mean over 20 evaluation trajectories \(with greedy policy inference\); evaluation occurs once every 10,000 environment steps during training\.### C\.1Extended Tabular results

In the following results tables, we report the mean and standard deviation for all experiments performed\. This includes scalarized training values ofλ\\lambdathat do not appear in[Figure3](https://arxiv.org/html/2608.11511#S4.F3)\. For some environments, we extended our sweep to additional values ofλ\\lambda\. Unless otherwise specified, we report mean and standard deviation over 10 training runs\. Our results for learning a waiting wrapper policy using scalarized rewards are reported over 5 training runs\.

Table 3:Cookresults\. “Wrap” refers to wrapping a base policy that is handwritten and which stands idly by as soup cooks\.Table 4:CookLongerresults\. “Wrap” refers to wrapping a base policy that is handwritten and which stands idly by as soup cooks\.Table 5:CookTwiceresults\. “Wrap” refers to wrapping a base policy that is handwritten and which stands idly by as soup cooks\.Table 6:CoffeeResults\. “Wrap” refers to wrapping a base policy that is handwritten and which stands idly by as the coffe machine heats and as coffee brews\.Table 7:Pongresults\. “Wrap” refers to wrapping a base policy that is trained using vanilla DQN\.Table 8:CartpoleResults\. “Wrap” refers to wrapping a base policy that was trained using vanilla DQN\.Table 9:Results forMountainCarenvironment\. “Wrap” refers to wrapping a base policy that is trained using vanilla DQN\.