When Do Options Help? Policy Necrosis and Redundant Coverage in Option-Critic

# When Do Options Help? Policy Necrosis andRedundant Coverage in Option-Critic
Source: [https://arxiv.org/html/2609.05508](https://arxiv.org/html/2609.05508)
Bingyun Liu††thanks:Equal contribution\.†Corresponding author\.Affiliation:Institute of AutomationAffiliation:Chinese Academy of SciencesEmail:[liubingyun2021@ia\.ac\.cn](mailto:)Yuheng Jing\*†Affiliation:Institute of AutomationAffiliation:Chinese Academy of SciencesEmail:[jingyuheng2022@ia\.ac\.cn](mailto:)

###### Abstract

Option\-critic learns options: sub\-policies together with a learned rule for when each one hands control back\. Its headline result is that performance improves as options are added\. We explain that result, with theory and experiment\. First, the termination rule option\-critic learns by maximising return contributes nothing\. When the termination test and the policy that picks options read the same values, the test fires at every step, so the learned rule is identical to always terminating\. When that policy explores and the test does not, as in option\-critic itself, the rule can block the exploration; there are instances where it suffersΩ⁡\(T\)\\Omega\(T\)regret while always terminating holds toO⁡\(log⁡T\)O\(\\log T\)\. Forcing termination at every step leaves the option\-count curve intact\. Second, the policy inside an option barely explores at all, so a state locks onto the first action that looked good and never updates again\. We name this policy necrosis, give a state\-level test for it, and find three fifths of states necrotic in a typical option\. Restoring exploration repairs those states, and one option then solves the task\. Third, extra options improve no option; what falls is the chance that all of them fail in the same state, from59%59\\%to4%4\\%, and performance follows that joint quantity\.

## 1Introduction

The options framework gives reinforcement learning agents temporally extended actions: an option is an intra\-option policy, an initiation set, and a termination condition\([Sutton et al\., 1999](https://arxiv.org/html/2609.05508#bib.bib1);[Precup, 2000](https://arxiv.org/html/2609.05508#bib.bib2)\)\. Option\-critic made the framework practical by differentiating the discounted return with respect to both the intra\-option policy parameters and the termination parameters, so that the control objective itself shapes the options instead of a designer handing them to the agent\([Bacon et al\., 2017](https://arxiv.org/html/2609.05508#bib.bib3)\)\. Its tabular demonstration is FourRooms, and the demonstration is a scaling result: with more options the agent reaches the goal faster, and the effect grows with the difficulty of the dynamics\. That result has been reproduced, extended to continuous control, and built upon many times over\([Klissarov et al\., 2017](https://arxiv.org/html/2609.05508#bib.bib6);[Harb et al\., 2018](https://arxiv.org/html/2609.05508#bib.bib4);[Riemer et al\., 2018](https://arxiv.org/html/2609.05508#bib.bib7);[Zhang and Whiteson, 2019](https://arxiv.org/html/2609.05508#bib.bib8);[Khetarpal et al\., 2020](https://arxiv.org/html/2609.05508#bib.bib9)\)\.

What the scaling result licenses is less clear than it looks\. Three things move together when the number of optionsKKgrows\. Options are temporally extended, so a larger option set changes the temporal structure of behaviour\. Options are separate function approximators, so a larger option set changes the capacity and the optimisation geometry of the lower level\. And options are chosen by a learned router, so a larger option set changes how much of the decision burden sits above the primitive actions\. A curve of performance againstKKcannot separate them, and reading it as evidence for temporal abstraction assumes the other two do nothing\.

This paper separates them, and the answer is not temporal abstraction\. We manipulate one factor at a time — the termination rule, the lower\-level exploration rate, andKK— across several transition kernels, two reward definitions, and a tabular and a neural implementation, and we read performance off frozen checkpoints under a protocol that gives every condition the same zero\-exploration test policy\. Figure[1](https://arxiv.org/html/2609.05508#S1.F1)lays out the environment, the agent and the hypothesis; three results follow, and together they form a single account of what option\-critic is doing\.

The termination function that the termination gradient learns is not a source of performance\.Our target is specific: the termination rule that option\-critic obtains by maximising return, whose fixed point terminates exactly when continuing is worth less than handing control back\. We prove that this rule is either vacuous or harmful\. When the termination test and the option selector read the same values, the test fires at every state, so the learned rule is pathwise identical toβ≡1\\beta\\equiv 1and adds nothing\. When the selector explores while the critic feeding the test does not, the rule can veto the exploration the selector asks for; we exhibit instances on which the return\-maximising rule suffersΩ⁡\(T\)\\Omega\(T\)regret, whileβ≡1\\beta\\equiv 1holds toO⁡\(log⁡T\)O\(\\log T\)on every instance of the class\. The experiments match the theory\. Forcingβ≡1\\beta\\equiv 1, which destroys temporal extension outright, leaves the whole dependence onKKuntouched, and the measured effect of learned termination reverses sign asKKgrows\.

Option\-critic’s lower level suffers a state\-local learning failure, and we can detect it\.The original algorithm runs its intra\-option policies as a softmax at temperature10−310^\{\-3\}with no lower\-level exploration, and with learning rate0\.250\.25that multiplies every advantage by250250before it reaches the logits\. A state commits to whichever action was first sampled with a positive advantage; the score function then vanishes, and the state never updates again\. We call this*policy necrosis*, and we give a state\-level test for it that needs only the set of actions an optimal policy would accept\. Roughly half of all states are necrotic in a typical option\. Restarting lower\-level exploration from a shared checkpoint repairs them: a single option goes from306\.7306\.7to55\.455\.4expected steps under identical zero\-exploration evaluation, and improves on all350350seeds\. Two different exploration mechanisms reproduce the repair in the neural implementation\.

What extra options provide is redundant coverage\.Adding options repairs no individual option — the mean single\-option necrosis rate holds near60%60\\%at everyKK— but the chance that*every*option is necrotic in the same state falls from59%59\\%to4%4\\%, and performance follows that joint quantity and not the individual one\. Necrotic sets overlap more than independence predicts and far less than a common cause would, so a pool ofKKoptions is worth aboutK/2K/2independent attempts at covering the state space\. Once joint coverage exists the residual error moves upstairs, to a router that picks a necrotic option while a viable one is available\. And once exploration has repaired the lower level, further options stop returning anything and start costing\.

The three results compose\. Necrosis is the failure, option multiplicity is a workaround that hides it, lower\-level exploration is a repair that removes it, and the termination gradient plays no part\. Beyond the specific verdict on option\-critic, the paper contributes a diagnostic that is cheap wherever an acceptable\-action set is available, a theoretical account of why a return\-maximising termination rule cannot be a source of gains, and an evaluation protocol — freeze, disable exploration everywhere, solve exactly — under which claims about what a hierarchical agent has learned can be settled instead of argued\.

![Refer to caption](https://arxiv.org/html/2609.05508v1/fig1_framework.png)Figure 1:The benchmark, the agent, and the hypothesis\.\(a\) FourRooms with the goal in the east doorway \(star\) and, for every non\-goal cell, the complete set of actions that reduce the wall\-respecting distance to the goal;6565of the103103cells admit two\. \(b\) The option\-critic loop and the three quantities we manipulate; everything else stays at the values of the original work\. \(c\) An option is necrotic in a state when its policy has committed there to an action no optimal policy would take, so option multiplicity acts on⋂oDo\\bigcap\_\{o\}D\_\{o\}and not on any individualDoD\_\{o\}\.
## 2Setup

#### Option\-critic\.

An optionooconsists of an intra\-option policyπo​\(a\|s\)\\pi\_\{o\}\(a\\,\|\\,s\)and a termination functionβo​\(s\)\\beta\_\{o\}\(s\); a policy over optionsπΩ​\(o\|s\)\\pi\_\{\\Omega\}\(o\\,\|\\,s\)picks the next option whenever the current one terminates\([Sutton et al\., 1999](https://arxiv.org/html/2609.05508#bib.bib1)\)\. Option\-critic learns all three from the return\([Bacon et al\., 2017](https://arxiv.org/html/2609.05508#bib.bib3)\)\. In the tabular implementation we study,πo\\pi\_\{o\}is a softmax over preference weights,πo​\(a\|s\)∝exp⁡\(wo​\(s,a\)/τ\)\\pi\_\{o\}\(a\\,\|\\,s\)\\propto\\exp\(w\_\{o\}\(s,a\)/\\tau\), and receives the update

wo\(s,⋅\)←wo\(s,⋅\)\+α\(QU\(s,o,a\)−QΩ\(s,o\)\)\(ea−πo\(⋅\|s\)\),w\_\{o\}\(s,\\cdot\)\\;\\leftarrow\\;w\_\{o\}\(s,\\cdot\)\+\\alpha\\,\\big\(Q\_\{U\}\(s,o,a\)\-Q\_\{\\Omega\}\(s,o\)\\big\)\\,\\big\(e\_\{a\}\-\\pi\_\{o\}\(\\cdot\\,\|\\,s\)\\big\),\(1\)whereQU​\(s,o,a\)Q\_\{U\}\(s,o,a\)is the critic’s value for taking primitive actionaaunder optionooatssandQΩ​\(s,o\)Q\_\{\\Omega\}\(s,o\)its value for running optionoofromss, so their difference is the intra\-option advantage, andeae\_\{a\}is the one\-hot vector on the executed action, makingea−πo\(⋅\|s\)e\_\{a\}\-\\pi\_\{o\}\(\\cdot\\,\|\\,s\)the softmax score function\. The termination functionβo\\beta\_\{o\}is a sigmoid updated along the option advantageAΩ​\(s,o\)=QΩ​\(s,o\)−maxo′⁡QΩ​\(s,o′\)A\_\{\\Omega\}\(s,o\)=Q\_\{\\Omega\}\(s,o\)\-\\max\_\{o^\{\\prime\}\}Q\_\{\\Omega\}\(s,o^\{\\prime\}\), andπΩ\\pi\_\{\\Omega\}isεup\\varepsilon\_\{\\mathrm\{up\}\}\-greedy on the tabularQΩQ\_\{\\Omega\}\. The original hyperparameters areτ=10−3\\tau=10^\{\-3\},α=0\.25\\alpha=0\.25for both the intra\-option and termination updates,0\.50\.5for the critic,εup=0\.1\\varepsilon\_\{\\mathrm\{up\}\}=0\.1, and discount0\.990\.99\. Every reported configuration keeps them\.

#### Environments\.

FourRooms is the benchmark option\-critic was introduced on, and it is also an unusually good instrument\. Its state space is finite and small enough to enumerate, so for any transition kernel we can solve the MDP exactly, read off the set of actions an optimal policy would accept in every state, and turn that set into a per\-state test of whether an option is usable there; nothing in Section[3](https://arxiv.org/html/2609.05508#S3)is available without it\. The grid is13×1313\\times 13with104104free cells and the goal in the east doorway, leaving103103non\-goal states\. Under*local slip*the agent moves to a uniformly random neighbouring free cell with probabilitypp; under*global teleport*it jumps to a uniformly random free cell with probabilityppand slips locally with probability\(1−p\)​p\(1\-p\)p, withp∈\{0,13,12\}p\\in\\\{0,\\tfrac\{1\}\{3\},\\tfrac\{1\}\{2\}\\\}\. Two rewards are used: the original*sparse*reward of11on entering the goal, and a*constant cost*of−1\-1on every transition, which makes the optimal policy the shortest path and the acceptable action set purely combinatorial\. Episodes are capped at10001000steps and training runs for10001000episodes\.

Two continuous variants test whether the mechanism survives function approximation, where no state can be enumerated and the diagnostics do not apply\. In*Direct*the actions are unit displacements with uniform noise and the observation is the position\.*Inertia*changes both the state and the action spaces: actions are accelerations, the observation carries a velocity, and a collision zeroes it, so the agent controls a second\-order system on the same floor plan\. All four option\-critic components become two\-layer tanh networks trained with Adam, target networks and a replay buffer; Appendix[A](https://arxiv.org/html/2609.05508#A1)lists every hyperparameter\.

#### Evaluation\.

Training curves conflate what a policy has learned with how much it randomises while acting, the exact confound at stake here, so we report performance from frozen checkpoints\. Parameters are copied out and never updated; explicit lower\-level exploration is set to zero in every condition, so a branch trained withεlo=0\.1\\varepsilon\_\{\\mathrm\{lo\}\}=0\.1is evaluated withεlo=0\\varepsilon\_\{\\mathrm\{lo\}\}=0;εup=0\.1\\varepsilon\_\{\\mathrm\{up\}\}=0\.1is retained and marginalised exactly; and the expected number of capped transitions to the goal is computed by finite\-horizon survival dynamic programming on the true kernel with horizon10001000, starting uniformly over the103103non\-goal states\. This evaluation has no sampling error, so all reported uncertainty comes from the350350matched training seeds, and every contrast is paired within seed\. The neural experiments and the tabular termination grid instead use the mean number of steps over the last100100training episodes, the metric the original scaling result was reported on, and every figure and table that uses it says so\. Appendix[B](https://arxiv.org/html/2609.05508#A2)gives the resampling scheme behind every interval\.

## 3Detecting policy necrosis

The measurements below need a notion of what it means for one option to be usable in one state, and a way to attribute a failure either to the options or to the router that picks among them\.

#### Necrotic options\.

Fix a frozen checkpoint and a statess, and letA⋆​\(s\)A^\{\\star\}\(s\)be the set of primitive actions an optimal policy for the true kernel would be willing to take atss\. Optionoois*necrotic*atsswhen its intra\-option policy has committed there to something outside that set\. Under the constant cost this needs no threshold:A⋆​\(s\)A^\{\\star\}\(s\)is exactly the set of actions that reduce the wall\-respecting distance to the goal, all ties kept \(6565of103103states have two\), andoois viable atsswhenarg⁡maxa​wo​\(s,a\)∈A⋆​\(s\)\\arg\\max\_\{a\}w\_\{o\}\(s,a\)\\in A^\{\\star\}\(s\)\. Under the sparse reward we takeA⋆​\(s\)A^\{\\star\}\(s\)from value iteration on the exact kernel, keeping all numerically tied maximisers, and calloonecrotic when∑a∈A⋆​\(s\)πo​\(a\|s\)<0\.01\\sum\_\{a\\in A^\{\\star\}\(s\)\}\\pi\_\{o\}\(a\\,\|\\,s\)<0\.01; the deterministic case agrees state by state with a breadth\-first search\. Both criteria weight all states equally and are evaluated with exploration switched off\.

#### Where the failure sits\.

WithKKoptions, every \(state, seed\) pair falls in exactly one of three classes\. It isDwhen allKKoptions are necrotic, so no choice the router could make would help;Mwhen at least one option is viable but the greedyarg⁡maxo​QΩ​\(s,o\)\\arg\\max\_\{o\}Q\_\{\\Omega\}\(s,o\)selects a necrotic one, so usable coverage exists and is missed; andRwhen the router selects a viable option\. Multiplicity can act onPr⁡\(D\)=Pr⁡\(⋂oDo\)\\Pr\(\\mathrm\{D\}\)=\\Pr\(\\bigcap\_\{o\}D\_\{o\}\), while the mean single\-option necrosis rate1K​∑oPr⁡\(Do\)\\tfrac\{1\}\{K\}\\sum\_\{o\}\\Pr\(D\_\{o\}\)says whether individual options improved\. Separating them is the point: if extra options make each option better, both fall together; if extra options supply redundancy, only the first falls\.

## 4The learned termination function is not a source of performance

### 4\.1What the termination gradient is heading for

Option\-critic updatesβo\\beta\_\{o\}along the option advantageAΩ​\(s,o\)=QΩ​\(s,o\)−maxo′⁡QΩ​\(s,o′\)A\_\{\\Omega\}\(s,o\)=Q\_\{\\Omega\}\(s,o\)\-\\max\_\{o^\{\\prime\}\}Q\_\{\\Omega\}\(s,o^\{\\prime\}\), which is never positive, so the update only ever raisesβo​\(s\)\\beta\_\{o\}\(s\), wherever the current option is not the greedy one\. The rule it heads for is greedy interruption,

β⁡\(s,o\)=\{0,QΩ​\(s,o\)\>VΩ​\(s\),1,otherwise,\\beta\(s,o\)=\\begin\{cases\}0,&Q\_\{\\Omega\}\(s,o\)\>V\_\{\\Omega\}\(s\),\\\\ 1,&\\text\{otherwise,\}\\end\{cases\}\(2\)and the collapse of learnedβ\\betatowards11that the literature reports\([Harb et al\., 2018](https://arxiv.org/html/2609.05508#bib.bib4);[Harutyunyan et al\., 2019](https://arxiv.org/html/2609.05508#bib.bib5)\)is that drift\. We analyse \([2](https://arxiv.org/html/2609.05508#S4.E2)\) in a two\-level setting where an upper level picks an option and a lower level picks an action inside it, both driven by value estimates\. Appendix[C](https://arxiv.org/html/2609.05508#A3)gives the model, the algorithms and the proofs\.

###### Proposition 1\(Vacuity under value consistency\)\.

If the selector and the termination test read the same estimatesQtQ\_\{t\}and the state value obeysVt​\(s\)=maxo⁡Qt​\(s,o\)V\_\{t\}\(s\)=\\max\_\{o\}Q\_\{t\}\(s,o\), thenQt​\(s,C\)≤Vt​\(s\)Q\_\{t\}\(s,C\)\\leq V\_\{t\}\(s\)for every current optionCC, so \([2](https://arxiv.org/html/2609.05508#S4.E2)\) terminates at every step and produces trajectories and regret identical toβ≡1\\beta\\equiv 1\.

###### Theorem 2\(Worst\-case separation under the standard definitions\)\.

If instead the selector uses optimistic indices while the test reads the critic’s raw values, which is option\-critic’s own definitions, then for two\-level UCB there is an instance with two options and two actions each on which, writinggi\\mathrm\{gi\}for rule \([2](https://arxiv.org/html/2609.05508#S4.E2)\), for everyT≥4T\\geq 4,

ℛTgi≥T−264,ℛTβ≡1≤64​ln⁡T\+12​\(1\+π23\)\.\\mathcal\{R\}\_\{T\}^\{\\mathrm\{gi\}\}\\;\\geq\\;\\frac\{T\-2\}\{64\},\\qquad\\mathcal\{R\}\_\{T\}^\{\\beta\\equiv 1\}\\;\\leq\\;64\\ln T\+\\tfrac\{1\}\{2\}\\bigl\(1\+\\tfrac\{\\pi^\{2\}\}\{3\}\\bigr\)\.Over the class the worst case isΘ⁡\(T\)\\Theta\(T\)forgi\\mathrm\{gi\}againstO⁡\(M​T​ln⁡T\)O\(\\sqrt\{MT\\ln T\}\)forβ≡1\\beta\\equiv 1, withMMthe number of option–action pairs, and a two\-step tabular MDP that reselects the option every episode still forcesΩ⁡\(T\)\\Omega\(T\)\.

###### Theorem 3\(Removing termination restores a guarantee\)\.

Underβ≡1\\beta\\equiv 1the two\-level agent is isomorphic to a flat agent on the MDP whose actions are the pairs\(o,a\)\(o,a\), so UCBVI\-BF gives, with probability at least1−δ1\-\\deltaandL=ln⁡\(5​H​S​D​T/δ\)L=\\ln\(5HSDT/\\delta\),RegN≤30​L​H​S​D​T\+2500​H2​S2​D​L2\+4​H​T​L=O~​\(H​S​D​T\)\\operatorname\{Reg\}\_\{N\}\\leq 30L\\sqrt\{HSDT\}\+2500H^\{2\}S^\{2\}DL^\{2\}\+4H\\sqrt\{TL\}=\\widetilde\{O\}\(\\sqrt\{HSDT\}\), whereSScounts states,DDoption–action pairs per state,HHthe horizon andT=N​HT=NH\.

A termination function trained to maximise return therefore has no room in which to help\. Consistent values make it an identity operation; inconsistent values make it an exploration veto with no uniform sublinear guarantee, and the veto costs most exactly when the option set grows and the selector has more to explore\. The separation is worst case and not pointwise — the veto can also suppress exploration of a genuinely worse option, and Appendix[C\.6](https://arxiv.org/html/2609.05508#A3.SS6)gives an instance where it wins — but a component whose sign depends on the instance is not a source of reliable gains either way\.

### 4\.2What the experiments measure

Table 1:The measured effect of learned termination reverses as the option set grows\.Sparse\-reward FourRooms, the setting option\-critic was introduced on\. Each entry is100×\(Term−β≡1\)/β≡1100\\times\(\\text\{Term\}\-\\beta\{\\equiv\}1\)/\\beta\{\\equiv\}1on the mean step count over the last100100training episodes, so negative favours learned termination, with95%95\\%paired bootstrap intervals\. The tabular grid resamples1010rank blocks of3535seeds per cell, the neural grid5050individual seeds\.K=1K\{=\}1is omitted because a single option cannot change identity on termination\.Forcingβ≡1\\beta\\equiv 1removes temporal extension outright: the agent draws a new option after every primitive step, so an option is nothing more than a latent index refreshed at every state\. Temporal extension is the only thing the control takes away, so a scaling result that rested on it should collapse here\.

It does not\. Table[1](https://arxiv.org/html/2609.05508#S4.T1)gives the measured effect in the sparse\-reward setting option\-critic was introduced on, on the metric the original scaling result was reported on\. The two conditions do differ, and they differ in the way Theorem[2](https://arxiv.org/html/2609.05508#Thmtheorem2)predicts: the sign tracks how much exploration the selector has to do\. In the tabular grid learned termination wins all five kernels atK=2K\{=\}2by11\.6%11\.6\\%to23\.8%23\.8\\%, with every interval below zero; it is within±5%\\pm 5\\%in all five atK=4K\{=\}4, with three intervals covering zero; and it loses all five atK=8K\{=\}8by11\.1%11\.1\\%to37\.0%37\.0\\%, with every interval above zero\. With two options there is little for the veto to block; with eight, the router has a large candidate set and blocking it costs\. The neural grid gives no consistent direction, with a29\.0%29\.0\\%advantage in Direct atK=8K\{=\}8against a16\.4%16\.4\\%disadvantage in Inertia, and intervals wide enough atK=4K\{=\}4andK=8K\{=\}8to admit either sign\. A component whose measured contribution reverses with a hyperparameter it is meant to be orthogonal to is not a stable source of gains, and the option\-count scaling it is supposed to explain survives its removal\.

The shortest\-path variant lets us make the comparison without relying on training curves at all, because both arms can be evaluated by exact frozen dynamic programming\. There the paired difference between learned option\-critic and itsβ≡1\\beta\\equiv 1match contains zero at every option count, and the two agents trace the same eightfold improvement asKKgrows,962\.5→821\.4→377\.2→117\.4962\.5\\rightarrow 821\.4\\rightarrow 377\.2\\rightarrow 117\.4expected steps against963\.2→822\.3→384\.4→122\.5963\.2\\rightarrow 822\.3\\rightarrow 384\.4\\rightarrow 122\.5\. Whatever produces that improvement is present when options have no temporal extent at all\. Table[4](https://arxiv.org/html/2609.05508#A4.T4)in Appendix[D](https://arxiv.org/html/2609.05508#A4)gives those numbers, and Figure[5](https://arxiv.org/html/2609.05508#A4.F5)there redraws Table[1](https://arxiv.org/html/2609.05508#S4.T1)as a heat map\.

## 5Policy necrosis, and what repairs it

#### Why states stop learning\.

Look again at the intra\-option update in Eq\.[1](https://arxiv.org/html/2609.05508#S2.E1)\. The quantity that enters the policy iswo​\(s,a\)/τw\_\{o\}\(s,a\)/\\tau, so withτ=10−3\\tau=10^\{\-3\}andα=0\.25\\alpha=0\.25an advantage is multiplied byα/τ=250\\alpha/\\tau=250before it reaches the logits\. From a uniform policy over four actions, an advantage of0\.0080\.008already leaves the sampled action with probability above0\.70\.7, and0\.0320\.032pushes it past0\.9990\.999\. The commitment is self\-sealing: the score functionea−πo\(⋅\|s\)e\_\{a\}\-\\pi\_\{o\}\(\\cdot\\,\|\\,s\)that multiplies the advantage goes to zero for the committed action, and the alternatives that carry a non\-vanishing score are sampled with probability close to zero\. Under a sparse reward the first advantage a state sees comes from an essentially uninformed critic, so what it commits to is close to arbitrary, and further training on the same option does not repair it\. This is the softmax gravity well described by[Mei et al\. \(2020a\)](https://arxiv.org/html/2609.05508#bib.bib26), with the temperature turning a slow drift into a one\-update event\.

#### A matched fork isolates the repair\.

To test the account we train a common500500\-episode prefix withεlo=0\\varepsilon\_\{\\mathrm\{lo\}\}=0, snapshot the entire learner including the random number generator, and continue along two branches: a control that keepsεlo=0\\varepsilon\_\{\\mathrm\{lo\}\}=0and a rescue branch that mixes in uniform random actions with probabilityεlo=0\.1\\varepsilon\_\{\\mathrm\{lo\}\}=0\.1\. The rescue branch changes only which actions are executed, so nothing about the learning rule changes\. Both branches are evaluated withεlo=0\\varepsilon\_\{\\mathrm\{lo\}\}=0, so any difference comes from parameters written during training and not from randomness at test time; the episode\-500 checkpoints of the two branches are bit\-for\-bit identical\.

Figure 2:Restoring lower\-level exploration replaces the entire benefit of extra options\.\(a–d\) Frozen exact evaluation of every checkpoint, five\-kernel average, per option count\. Training shares aεlo=0\\varepsilon\_\{\\mathrm\{lo\}\}=0prefix through episode500500and then forks; both branches are evaluated withεlo=0\\varepsilon\_\{\\mathrm\{lo\}\}=0\. Bands are95%95\\%bootstrap intervals over350350matched seeds\. \(e\) Paired step reduction at episode10001000, per kernel and averaged\.Table 2:Repairing the lower level removes the work that extra options were doing\.Left: sparse\-reward tabular FourRooms, five\-kernel average of expected capped transitions at the episode\-1000 checkpoint, paired over350350seeds\. Right: constant\-cost FourRooms,β≡1\\beta\\equiv 1agents trained with and without lower\-level exploration, evaluated identically\. All evaluations useεlo=0\\varepsilon\_\{\\mathrm\{lo\}\}=0\.Sparse reward: matched fork at episode 500Constant costKKcontrolrescuereduction95% CIseeds helpedtrainεlo=0\\varepsilon\_\{\\mathrm\{lo\}\}=0trainεlo=0\.1\\varepsilon\_\{\\mathrm\{lo\}\}=0\.11306\.6655\.42251\.23251\.23\[242\.59,259\.99\]\[242\.59,\\ 259\.99\]350/350963\.2136\.502152\.9367\.1885\.7585\.75\[79\.68,91\.98\]\[79\.68,\\ 91\.98\]344/350822\.3145\.21479\.1469\.289\.869\.86\[6\.74,13\.03\]\[6\.74,\\ 13\.03\]211/350384\.3755\.09856\.4260\.35−3\.94\-3\.94\[−5\.73,−2\.14\]\[\-5\.73,\\ \-2\.14\]138/350122\.4685\.03
#### One option is enough\.

Figure[2](https://arxiv.org/html/2609.05508#S5.F2)a shows what happens with a single option\. The control drifts from318\.7318\.7at the fork to306\.7306\.7expected steps by the end of training; the rescue branch is at156\.1156\.1ten episodes later,76\.676\.6by episode600600, and55\.455\.4at the end\. All350350seeds improve, and the paired reduction is251\.2251\.2steps\[242\.59,259\.99\]\[242\.59,\\,259\.99\], running from119\.6119\.6under no perturbation to352\.3352\.3under teleport withp=12p=\\tfrac\{1\}\{2\}\(Figure[2](https://arxiv.org/html/2609.05508#S5.F2)e\)\. Under the constant\-cost reward the same intervention takes a singleβ≡1\\beta\\equiv 1option from963\.2963\.2to36\.536\.5expected steps and its frozen success probability from0\.0370\.037to0\.9990\.999\. One option, evaluated with no exploration at all, solves the task\.

#### Why the benefit shrinks asKKgrows\.

Panels b–d of Figure[2](https://arxiv.org/html/2609.05508#S5.F2)repeat the fork atK=2,4,8K=2,4,8, and the gap the rescue branch opens shrinks each time\. The paired reduction falls from251\.2251\.2steps atK=1K\{=\}1to85\.785\.7, then9\.99\.9, and atK=8K\{=\}8it turns negative at−3\.9\-3\.9\[−5\.73,−2\.14\]\[\-5\.73,\\,\-2\.14\], with only138138of350350seeds improving \(Table[2](https://arxiv.org/html/2609.05508#S5.T2)\)\. Under the constant\-cost reward the exploring agent gets steadily worse as options are added, from36\.536\.5to85\.085\.0expected steps\. The two interventions are not interchangeable and do not act on the same object\. Exploration acts on the cause, restoring the sampling that lets a committed state recover, and Section[6](https://arxiv.org/html/2609.05508#S6)shows it lowers the single\-option necrosis rate directly; extra options leave every individual policy where it was and act on the aggregate, by making it unlikely that all of them fail in the same place\. The shrinking gap records the interaction: multiplicity hides most of the cost of necrosis, leaving a repair little headroom, and once the repair has removed the necrosis multiplicity has nothing to compensate for\.

#### The neural implementation agrees, through two different mechanisms\.

Figure[3](https://arxiv.org/html/2609.05508#S5.F3)repeats the comparison in continuous FourRooms withβ≡1\\beta\\equiv 1throughout, contrasting the base softmax against explicitε\\varepsilon\-greedy behaviour and against entropy\-regularised sampling\. With one option the base softmax sits at the10001000\-step cap in both environments \(977977and970970\),ε\\varepsilon\-greedy brings it to299299and250250, and entropy regularisation to207207and156156\. Adding options helps the base softmax a great deal, helpsε\\varepsilon\-greedy much less, and makes the entropy configuration monotonically worse:207→225→244→260207\\rightarrow 225\\rightarrow 244\\rightarrow 260in Direct and156→158→185→221156\\rightarrow 158\\rightarrow 185\\rightarrow 221in Inertia\. Inertia runs atτ=0\.5\\tau=0\.5, five hundred times the tabular temperature, and its one\-option base softmax still fails, so the failure is no artefact of the extreme tabular setting\.

Figure 3:Two exploration mechanisms make one option sufficient in the neural implementation\.Mean steps over the last100100training episodes withβ≡1\\beta\\equiv 1throughout,5050seeds per point,95%95\\%bootstrap intervals, log scale\. Numbers mark the one\-option values\.

## 6What extra options provide is redundant coverage

If the lower level is necrotic and extra options carry the load instead, what exactly is the load they carry? Measuring that calls for the constant\-cost reward, where the optimal policy is the shortest path, so the acceptable action set at a state is exactly the set of distance\-reducing actions and the necrosis test is combinatorial, with no threshold and no dependence on a value scale\. Under the sparse reward the value function encodes reachability and the numerical spread between distinct reachable routes is small, which makes a shortest\-path yardstick a coarser instrument there\. We therefore read the coverage story off the constant\-cost grid, and Figure[6](https://arxiv.org/html/2609.05508#A4.F6)in Appendix[D](https://arxiv.org/html/2609.05508#A4)reports the sparse\-reward version, which agrees\.

Figure 4:More options make simultaneous failure rare without making any option better\.Constant\-cost FourRooms under teleportp=12p=\\tfrac\{1\}\{2\}, episode\-1000 checkpoints,350350matched seeds, threshold\-free necrosis test\. \(a\) Every \(state, seed\) pair in exactly one class\. \(b\) The rate at which every option is necrotic \(solid\) against the mean single\-option rate \(faint\)\. \(c\) Exact frozen performance of the same checkpoints\. Learned option\-critic and itsβ≡1\\beta\\equiv 1match coincide throughout\.Table 3:Going from one option to eight, within seed\.Constant\-cost FourRooms, teleportp=12p=\\tfrac\{1\}\{2\}, episode\-1000 checkpoints,350350matched seeds\. Necrosis rates are in percentage points and steps are expected capped transitions; negative means the eight\-option agent is better\. Joint necrosis collapses in every row while single\-option necrosis does not improve in any of them\.For an agent that never explored, the mean single\-option necrosis rate is59\.04%59\.04\\%atK=1K\{=\}1and59\.35%59\.35\\%atK=8K\{=\}8, and it never leaves that band\. Close to three fifths of the state space is locked onto a wrong action in a typical option, and that is as true of the eighth option as of the first\. The joint rate behaves completely differently: the probability that a state is necrotic for every option falls59\.0→36\.5→15\.5→3\.959\.0\\rightarrow 36\.5\\rightarrow 15\.5\\rightarrow 3\.9percent, and the expected steps of those same frozen policies fall959\.5→875\.8→509\.4→220\.5959\.5\\rightarrow 875\.8\\rightarrow 509\.4\\rightarrow 220\.5\. Extra options do not produce better options; they produce a set whose members fail in different places\. Table[3](https://arxiv.org/html/2609.05508#S6.T3)says the same as a within\-seed contrast, and the learned\-termination agent and itsβ≡1\\beta\\equiv 1match agree to within their intervals, which is Section[4](https://arxiv.org/html/2609.05508#S4)in coverage terms\.

#### How independent are the failures?

If theKKnecrotic sets were independent, the joint rate atK=8K\{=\}8would be0\.5948=1\.5%0\.594^\{8\}=1\.5\\%against the measured3\.9%3\.9\\%; if they were identical, it would be59%59\\%\. SolvingPr⁡\(D\)=Pr⁡\(Do\)κ\\Pr\(\\mathrm\{D\}\)=\\Pr\(D\_\{o\}\)^\{\\,\\kappa\}forκ\\kappagives an effective number of independent options:κ=1\.92\\kappa=1\.92,3\.553\.55and6\.206\.20atK=2,4,8K=2,4,8\. Options trained side by side, on shared trajectories with a shared critic, fail together somewhat more than chance predicts and far less than a common cause would, so a pool ofKKoptions is worth roughlyK/2K/2independent attempts at covering the state space\. That also says what limits the account: the return on adding options is set by how decorrelated their failures are, a property of the training procedure and not of the option count\.

#### The bottleneck moves upstairs\.

Coverage that exists is not coverage that gets used\. As D shrinks, the M class grows in step: the router picks a necrotic option while a viable one is available in0%0\\%,17\.6%17\.6\\%,32\.9%32\.9\\%and42\.7%42\.7\\%of states atK=1,2,4,8K=1,2,4,8\(Figure[4](https://arxiv.org/html/2609.05508#S6.F4)a\)\. ByK=8K\{=\}8the dominant residual error is a routing error and not a coverage error, which is the failure mode that interest functions and attention over options were introduced to attack\([Khetarpal et al\., 2020](https://arxiv.org/html/2609.05508#bib.bib9);[Chunduru and Precup, 2022](https://arxiv.org/html/2609.05508#bib.bib36)\)\.

#### Redundancy has a price once the lower level works\.

The third row of Table[3](https://arxiv.org/html/2609.05508#S6.T3)is the informative boundary case\. An agent trained withεlo=0\.1\\varepsilon\_\{\\mathrm\{lo\}\}=0\.1still gets a coverage benefit from more options, with joint necrosis falling by35\.535\.5points, but its single\-option rate rises by21\.421\.4points and its performance gets*worse*by134\.9134\.9steps: a fixed budget spread over eight options degrades each one, and the router must distinguish eight policies instead of one\. Redundancy earns its keep when there is a failure to be redundant against, and charges rent when there is not\.

Figure[7](https://arxiv.org/html/2609.05508#A4.F7)in Appendix[D](https://arxiv.org/html/2609.05508#A4)shows the transition on the map itself\. For the agent that never explored, joint necrosis occupies6262of103103cells atK=1K\{=\}1and none atK=8K\{=\}8, and four of every five cells it vacates turn into routing misses instead of correct routes\. Itsβ≡1\\beta\\equiv 1match is indistinguishable from it, cell by cell\. The agent that explored starts at3737necrotic cells with one option and ends with44atK=8K\{=\}8, while its correctly routed cells fall from6666to5050along the way\.

## 7Related work

#### Options and option\-critic\.

The options framework formalises temporally extended action\([Sutton et al\., 1999](https://arxiv.org/html/2609.05508#bib.bib1);[Precup, 2000](https://arxiv.org/html/2609.05508#bib.bib2)\), alongside earlier hierarchical formulations\([Parr and Russell, 1998](https://arxiv.org/html/2609.05508#bib.bib21);[Dietterich, 2000](https://arxiv.org/html/2609.05508#bib.bib20)\)\. Option\-critic learns intra\-option policies and terminations from the task return\([Bacon et al\., 2017](https://arxiv.org/html/2609.05508#bib.bib3)\); extensions cover continuous control\([Klissarov et al\., 2017](https://arxiv.org/html/2609.05508#bib.bib6)\), deeper hierarchies\([Riemer et al\., 2018](https://arxiv.org/html/2609.05508#bib.bib7)\), off\-the\-shelf policy optimisation\([Zhang and Whiteson, 2019](https://arxiv.org/html/2609.05508#bib.bib8)\)and learned initiation\([Khetarpal et al\., 2020](https://arxiv.org/html/2609.05508#bib.bib9);[Chunduru and Precup, 2022](https://arxiv.org/html/2609.05508#bib.bib36)\), while other lines build options from environment structure or intrinsic objectives\([Şimşek and Barto, 2009](https://arxiv.org/html/2609.05508#bib.bib35);[Konidaris and Barto, 2009](https://arxiv.org/html/2609.05508#bib.bib14);[Machado et al\., 2017](https://arxiv.org/html/2609.05508#bib.bib10);[Machado et al\., 2018](https://arxiv.org/html/2609.05508#bib.bib11);[Eysenbach et al\., 2019](https://arxiv.org/html/2609.05508#bib.bib13);[Jinnai et al\., 2019](https://arxiv.org/html/2609.05508#bib.bib12);[Bagaria and Konidaris, 2020](https://arxiv.org/html/2609.05508#bib.bib15)\)or hierarchy over goals\([Kulkarni et al\., 2016](https://arxiv.org/html/2609.05508#bib.bib19);[Vezhnevets et al\., 2017](https://arxiv.org/html/2609.05508#bib.bib18);[Nachum et al\., 2018](https://arxiv.org/html/2609.05508#bib.bib16)\)\. Regret guarantees for option\-based learning exist under structural assumptions\([Drappo et al\., 2024](https://arxiv.org/html/2609.05508#bib.bib40)\); Section[4\.1](https://arxiv.org/html/2609.05508#S4.SS1)asks instead what a return\-trained termination rule can guarantee\.

#### Termination collapse and its remedies\.

That option\-critic terminations drift towards11is documented and has motivated direct remedies: a deliberation cost that charges for switching\([Harb et al\., 2018](https://arxiv.org/html/2609.05508#bib.bib4)\), and a termination objective built on option models instead of the control objective\([Harutyunyan et al\., 2019](https://arxiv.org/html/2609.05508#bib.bib5)\)\. Those works treat collapse as an empirical nuisance to engineer around\. Section[4\.1](https://arxiv.org/html/2609.05508#S4.SS1)shows the drift is a property of the objective itself, so those remedies act as stopgaps: they holdβ\\betaaway from the boundary without removing the reason it moves there\.

#### Where hierarchy’s benefits come from\.

[Nachum et al\. \(2019\)](https://arxiv.org/html/2609.05508#bib.bib17)isolate the claimed benefits of goal\-conditioned hierarchies on continuous control and attribute most of the gain to improved exploration in the high\-level action space\. Our result points the same way and localises the mechanism one level lower, inside the option’s own policy parameterisation, where it is visible state by state\.

#### Softmax policy gradient\.

[Mei et al\. \(2020b\)](https://arxiv.org/html/2609.05508#bib.bib25)and[Mei et al\. \(2020a\)](https://arxiv.org/html/2609.05508#bib.bib26)show that softmax\-parameterised policy gradient is initialisation\-sensitive and slows near the corners of the simplex, and[Agarwal et al\. \(2021a\)](https://arxiv.org/html/2609.05508#bib.bib27)show that exploration or regularisation is needed for global convergence; entropy regularisation is the standard practical response\([Williams, 1992](https://arxiv.org/html/2609.05508#bib.bib22);[Mnih et al\., 2016](https://arxiv.org/html/2609.05508#bib.bib23);[Haarnoja et al\., 2018](https://arxiv.org/html/2609.05508#bib.bib24)\)\. Policy necrosis is that phenomenon atτ=10−3\\tau=10^\{\-3\}, where the committal step takes one update, and what we add is that option\-critic covers for it with options\.

#### Empirical practice\.

Reported gains often survive poorly once controls and seed counts are taken seriously\([Henderson et al\., 2018](https://arxiv.org/html/2609.05508#bib.bib28);[Engstrom et al\., 2020](https://arxiv.org/html/2609.05508#bib.bib30);[Andrychowicz et al\., 2021](https://arxiv.org/html/2609.05508#bib.bib31);[Agarwal et al\., 2021b](https://arxiv.org/html/2609.05508#bib.bib29);[Patterson et al\., 2024](https://arxiv.org/html/2609.05508#bib.bib32)\)\. This paper is an instance in the hierarchical setting\.

## 8Discussion

The three results compose into one account\. A near\-deterministic softmax with a large effective step size commits states to arbitrary actions and stops updating them, leaving most of the state space necrotic for any given option; the diagnostic of Section[3](https://arxiv.org/html/2609.05508#S3)locates that failure state by state, and restoring lower\-level exploration repairs it, at which point one option solves the task\. Extra options repair nothing individually\. They supply policies whose necrotic sets only partly overlap, so joint failure falls while individual failure does not, performance follows the joint quantity, and the benefit goes with the failure it was covering\. Learned termination plays no part in any of this, and Section[4\.1](https://arxiv.org/html/2609.05508#S4.SS1)explains why it never could: the rule the termination gradient converges to does nothing at all when its values agree with the selector’s, and blocks the selector’s exploration when they do not\. The option count measures redundancy against a repairable defect, and the case for learned temporal abstraction has to be made somewhere else\.

#### A structural prior can be credited for a defect elsewhere\.

The broader reading is not about options\. A structural component sits on a base learner, and if the base learner has a repairable failure, the component is credited with whatever that failure was costing\. The option count is an unusually legible example: individual policies stay broken, their failures decorrelate, and the aggregate improves\. The same accounting applies to any architectural prior evaluated by a curve over its own capacity, and it is settled the same way, by repairing the base learner first and asking whether the prior still earns its keep\. Redundancy also deserves naming as a mechanism: ensembles obtain it deliberately\([Osband et al\., 2016](https://arxiv.org/html/2609.05508#bib.bib34)\)and option\-critic stumbles into it, so the lever is not the number of options but how decorrelated their failures are\.

#### Limitations\.

Two boundaries are worth stating\. The diagnostic needs a notion of which actions are acceptable in a state, which an exactly solvable environment supplies and a large one does not, so elsewhere it requires a surrogate whose quality any conclusion inherits\. And the exchange rate between options and exploration is set byκ\\kappa, a property of how the options were trained together, so the point at which multiplicity stops helping moves with the training procedure even though the mechanism does not\.

#### Future work\.

Make redundancy deliberate: diversity objectives should be judged by how much they raiseκ\\kappaat fixedKK, a cheaper and sharper target than end\-to\-end return\. Carry the necrosis test past exactly solvable environments, either against a reference policy or learned critic, or by dropping the oracle and tracking the two quantities that make necrosis self\-sealing, policy saturation and the vanishing score\-function norm\. And ask which termination objectives escape Section[4\.1](https://arxiv.org/html/2609.05508#S4.SS1): the obstruction applies to rules trained on return, while deliberation costs and predictability\-based termination\([Harb et al\., 2018](https://arxiv.org/html/2609.05508#bib.bib4);[Harutyunyan et al\., 2019](https://arxiv.org/html/2609.05508#bib.bib5)\)change the objective, so the same analysis should say whether either can be shown to help\.

### Reproducibility statement

Section[2](https://arxiv.org/html/2609.05508#S2)states the evaluation contract and the hyperparameters, Appendix[A](https://arxiv.org/html/2609.05508#A1)lists the remaining settings for the tabular and neural experiments, and Appendix[B](https://arxiv.org/html/2609.05508#A2)gives the resampling scheme behind every interval\. Section[3](https://arxiv.org/html/2609.05508#S3)gives both necrosis criteria in full, including the tie\-handling rule and the audit of the value\-iteration action sets against breadth\-first search\. Appendix[C](https://arxiv.org/html/2609.05508#A3)contains the model, the algorithms and complete proofs for Proposition[1](https://arxiv.org/html/2609.05508#Thmtheorem1), Theorem[2](https://arxiv.org/html/2609.05508#Thmtheorem2)and Theorem[3](https://arxiv.org/html/2609.05508#Thmtheorem3)\. The accompanying code release contains the training, checkpoint\-evaluation, and analysis code for all three experiment families, the derived per\-seed and summary tables behind every figure and table, a single entry point that rebuilds every figure in this paper from those tables without touching a raw checkpoint, and a validator that checks file hashes and the figure contract\.

### Ethics statement

This work is a methodological study of a reinforcement learning algorithm\. It involves no human subjects, no personal data, and no deployment\. We see no ethical concerns beyond the general question of how empirical claims in the field are supported, which the paper addresses directly\.

### AI use statement

We used generative AI tools for readability editing of the manuscript\. We have not used generative AI tools for proposing or refining the hypotheses, designing or critiquing the methodology or experiments, implementing the methods, interpreting the results, formulating the mathematical claims, or writing the proofs; generating synthetic datasets and qualitative data analysis are not applicable to this work\. We have reviewed all AI\-assisted work: every number in the paper was read back from the released derived data files by an author, the figure\-generation code was checked against those files, and every theorem statement and proof was written and verified by the authors\. We take responsibility for the final content of this work, including text, claims, and artifacts produced with the aid of generative AI\.

## References

- A\. Agarwal, S\. M\. Kakade, J\. D\. Lee, and G\. MahajanOn the theory of policy gradient methods: optimality, approximation, and distribution shift\.Journal of Machine Learning Research22\(98\),pp\. 1–76\.Cited by:[§7](https://arxiv.org/html/2609.05508#S7.SS0.SSS0.Px4.p1.1)\.
- Agarwalet al\.\(2021b\)R\. Agarwal, M\. Schwarzer, P\. S\. Castro, A\. Courville, and M\. G\. BellemareDeep reinforcement learning at the edge of the statistical precipice\.InAdvances in Neural Information Processing Systems 34 \(NeurIPS\),pp\. 29304–29320\.Cited by:[§7](https://arxiv.org/html/2609.05508#S7.SS0.SSS0.Px5.p1.1)\.
- Andrychowiczet al\.\(2021\)M\. Andrychowicz, A\. Raichuk, P\. Stańczyk, M\. Orsini, S\. Girgin, R\. Marinier, L\. Hussenot, M\. Geist, O\. Pietquin, M\. Michalski, S\. Gelly, and O\. BachemWhat matters for on\-policy deep actor\-critic methods? a large\-scale study\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§7](https://arxiv.org/html/2609.05508#S7.SS0.SSS0.Px5.p1.1)\.
- Aueret al\.\(2002\)P\. Auer, N\. Cesa\-Bianchi, and P\. FischerFinite\-time analysis of the multiarmed bandit problem\.Machine Learning47\(2–3\),pp\. 235–256\.Cited by:[§C\.3](https://arxiv.org/html/2609.05508#A3.SS3.p2.1.1)\.
- Azaret al\.\(2017\)M\. G\. Azar, I\. Osband, and R\. MunosMinimax regret bounds for reinforcement learning\.InProceedings of the 34th International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.70,pp\. 263–272\.Cited by:[§C\.8](https://arxiv.org/html/2609.05508#A3.SS8.p4.1.1)\.
- Baconet al\.\(2017\)P\. Bacon, J\. Harb, and D\. PrecupThe option\-critic architecture\.InProceedings of the Thirty\-First AAAI Conference on Artificial Intelligence,pp\. 1726–1734\.Cited by:[§1](https://arxiv.org/html/2609.05508#S1.p1.1),[§2](https://arxiv.org/html/2609.05508#S2.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2609.05508#S7.SS0.SSS0.Px1.p1.1)\.
- Bagaria and Konidaris \(2020\)A\. Bagaria and G\. KonidarisOption discovery using deep skill chaining\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§7](https://arxiv.org/html/2609.05508#S7.SS0.SSS0.Px1.p1.1)\.
- Chunduru and Precup \(2022\)R\. Chunduru and D\. PrecupAttention option\-critic\.arXiv preprint arXiv:2201\.02628\.Cited by:[§6](https://arxiv.org/html/2609.05508#S6.SS0.SSS0.Px2.p1.1),[§7](https://arxiv.org/html/2609.05508#S7.SS0.SSS0.Px1.p1.1)\.
- Colaset al\.\(2018\)C\. Colas, O\. Sigaud, and P\. OudeyerHow many random seeds? statistical power analysis in deep reinforcement learning experiments\.arXiv preprint arXiv:1806\.08295\.Cited by:[Appendix B](https://arxiv.org/html/2609.05508#A2.p1.1)\.
- Dietterich \(2000\)T\. G\. DietterichHierarchical reinforcement learning with the MAXQ value function decomposition\.Journal of Artificial Intelligence Research13,pp\. 227–303\.Cited by:[§7](https://arxiv.org/html/2609.05508#S7.SS0.SSS0.Px1.p1.1)\.
- Drappoet al\.\(2024\)G\. Drappo, A\. M\. Metelli, and M\. RestelliA provably efficient option\-based algorithm for both high\-level and low\-level learning\.Reinforcement Learning Journal2,pp\. 819–839\.Cited by:[§7](https://arxiv.org/html/2609.05508#S7.SS0.SSS0.Px1.p1.1)\.
- Engstromet al\.\(2020\)L\. Engstrom, A\. Ilyas, S\. Santurkar, D\. Tsipras, F\. Janoos, L\. Rudolph, and A\. MadryImplementation matters in deep RL: a case study on PPO and TRPO\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§7](https://arxiv.org/html/2609.05508#S7.SS0.SSS0.Px5.p1.1)\.
- Eysenbachet al\.\(2019\)B\. Eysenbach, A\. Gupta, J\. Ibarz, and S\. LevineDiversity is all you need: learning skills without a reward function\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§7](https://arxiv.org/html/2609.05508#S7.SS0.SSS0.Px1.p1.1)\.
- Haarnojaet al\.\(2018\)T\. Haarnoja, A\. Zhou, P\. Abbeel, and S\. LevineSoft actor\-critic: off\-policy maximum entropy deep reinforcement learning with a stochastic actor\.InProceedings of the 35th International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.80,pp\. 1861–1870\.Cited by:[§7](https://arxiv.org/html/2609.05508#S7.SS0.SSS0.Px4.p1.1)\.
- Harbet al\.\(2018\)J\. Harb, P\. Bacon, M\. Klissarov, and D\. PrecupWhen waiting is not an option: learning options with a deliberation cost\.InProceedings of the Thirty\-Second AAAI Conference on Artificial Intelligence,pp\. 3165–3172\.Cited by:[§1](https://arxiv.org/html/2609.05508#S1.p1.1),[§4\.1](https://arxiv.org/html/2609.05508#S4.SS1.p1.2),[§7](https://arxiv.org/html/2609.05508#S7.SS0.SSS0.Px2.p1.1),[§8](https://arxiv.org/html/2609.05508#S8.SS0.SSS0.Px3.p1.1)\.
- Harutyunyanet al\.\(2019\)A\. Harutyunyan, W\. Dabney, D\. Borsa, N\. Heess, R\. Munos, and D\. PrecupThe termination critic\.InProceedings of the 22nd International Conference on Artificial Intelligence and Statistics \(AISTATS\),Proceedings of Machine Learning Research, Vol\.89,pp\. 2231–2240\.Cited by:[§4\.1](https://arxiv.org/html/2609.05508#S4.SS1.p1.2),[§7](https://arxiv.org/html/2609.05508#S7.SS0.SSS0.Px2.p1.1),[§8](https://arxiv.org/html/2609.05508#S8.SS0.SSS0.Px3.p1.1)\.
- Hendersonet al\.\(2018\)P\. Henderson, R\. Islam, P\. Bachman, J\. Pineau, D\. Precup, and D\. MegerDeep reinforcement learning that matters\.InProceedings of the Thirty\-Second AAAI Conference on Artificial Intelligence,pp\. 3207–3214\.Cited by:[§7](https://arxiv.org/html/2609.05508#S7.SS0.SSS0.Px5.p1.1)\.
- Jinnaiet al\.\(2019\)Y\. Jinnai, J\. W\. Park, D\. Abel, and G\. KonidarisDiscovering options for exploration by minimizing cover time\.InProceedings of the 36th International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.97,pp\. 3130–3139\.Cited by:[§7](https://arxiv.org/html/2609.05508#S7.SS0.SSS0.Px1.p1.1)\.
- Khetarpalet al\.\(2020\)K\. Khetarpal, M\. Klissarov, M\. Chevalier\-Boisvert, P\. Bacon, and D\. PrecupOptions of interest: temporal abstraction with interest functions\.InProceedings of the Thirty\-Fourth AAAI Conference on Artificial Intelligence,Vol\.34,pp\. 4444–4451\.Cited by:[§1](https://arxiv.org/html/2609.05508#S1.p1.1),[§6](https://arxiv.org/html/2609.05508#S6.SS0.SSS0.Px2.p1.1),[§7](https://arxiv.org/html/2609.05508#S7.SS0.SSS0.Px1.p1.1)\.
- Klissarovet al\.\(2017\)M\. Klissarov, P\. Bacon, J\. Harb, and D\. PrecupLearnings options end\-to\-end for continuous action tasks\.arXiv preprint arXiv:1712\.00004\.Note:Hierarchical Reinforcement Learning Workshop, NIPS 2017Cited by:[§1](https://arxiv.org/html/2609.05508#S1.p1.1),[§7](https://arxiv.org/html/2609.05508#S7.SS0.SSS0.Px1.p1.1)\.
- Konidaris and Barto \(2009\)G\. Konidaris and A\. G\. BartoSkill discovery in continuous reinforcement learning domains using skill chaining\.InAdvances in Neural Information Processing Systems 22 \(NeurIPS\),pp\. 1015–1023\.Cited by:[§7](https://arxiv.org/html/2609.05508#S7.SS0.SSS0.Px1.p1.1)\.
- Kulkarniet al\.\(2016\)T\. D\. Kulkarni, K\. R\. Narasimhan, A\. Saeedi, and J\. B\. TenenbaumHierarchical deep reinforcement learning: integrating temporal abstraction and intrinsic motivation\.InAdvances in Neural Information Processing Systems 29 \(NeurIPS\),pp\. 3675–3683\.Cited by:[§7](https://arxiv.org/html/2609.05508#S7.SS0.SSS0.Px1.p1.1)\.
- Lai and Robbins \(1985\)T\. L\. Lai and H\. RobbinsAsymptotically efficient adaptive allocation rules\.Advances in Applied Mathematics6\(1\),pp\. 4–22\.Cited by:[§C\.3](https://arxiv.org/html/2609.05508#A3.SS3.p4.1)\.
- Machadoet al\.\(2017\)M\. C\. Machado, M\. G\. Bellemare, and M\. BowlingA Laplacian framework for option discovery in reinforcement learning\.InProceedings of the 34th International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.70,pp\. 2295–2304\.Cited by:[§7](https://arxiv.org/html/2609.05508#S7.SS0.SSS0.Px1.p1.1)\.
- Machadoet al\.\(2018\)M\. C\. Machado, C\. Rosenbaum, X\. Guo, M\. Liu, G\. Tesauro, and M\. CampbellEigenoption discovery through the deep successor representation\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§7](https://arxiv.org/html/2609.05508#S7.SS0.SSS0.Px1.p1.1)\.
- Meiet al\.\(2020a\)J\. Mei, C\. Xiao, B\. Dai, L\. Li, C\. Szepesvári, and D\. SchuurmansEscaping the gravitational pull of softmax\.InAdvances in Neural Information Processing Systems 33 \(NeurIPS\),pp\. 21130–21140\.Cited by:[§5](https://arxiv.org/html/2609.05508#S5.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2609.05508#S7.SS0.SSS0.Px4.p1.1)\.
- Meiet al\.\(2020b\)J\. Mei, C\. Xiao, C\. Szepesvári, and D\. SchuurmansOn the global convergence rates of softmax policy gradient methods\.InProceedings of the 37th International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.119,pp\. 6820–6829\.Cited by:[§7](https://arxiv.org/html/2609.05508#S7.SS0.SSS0.Px4.p1.1)\.
- Mnihet al\.\(2016\)V\. Mnih, A\. P\. Badia, M\. Mirza, A\. Graves, T\. Lillicrap, T\. Harley, D\. Silver, and K\. KavukcuogluAsynchronous methods for deep reinforcement learning\.InProceedings of the 33rd International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.48,pp\. 1928–1937\.Cited by:[§7](https://arxiv.org/html/2609.05508#S7.SS0.SSS0.Px4.p1.1)\.
- Nachumet al\.\(2018\)O\. Nachum, S\. Gu, H\. Lee, and S\. LevineData\-efficient hierarchical reinforcement learning\.InAdvances in Neural Information Processing Systems 31 \(NeurIPS\),pp\. 3303–3313\.Cited by:[§7](https://arxiv.org/html/2609.05508#S7.SS0.SSS0.Px1.p1.1)\.
- Nachumet al\.\(2019\)O\. Nachum, H\. Tang, X\. Lu, S\. Gu, H\. Lee, and S\. LevineWhy does hierarchy \(sometimes\) work so well in reinforcement learning?\.arXiv preprint arXiv:1909\.10618\.Note:NeurIPS 2019 Deep Reinforcement Learning WorkshopCited by:[§7](https://arxiv.org/html/2609.05508#S7.SS0.SSS0.Px3.p1.1)\.
- Osbandet al\.\(2016\)I\. Osband, C\. Blundell, A\. Pritzel, and B\. Van RoyDeep exploration via bootstrapped DQN\.InAdvances in Neural Information Processing Systems 29 \(NeurIPS\),pp\. 4026–4034\.Cited by:[§8](https://arxiv.org/html/2609.05508#S8.SS0.SSS0.Px1.p1.1)\.
- Parr and Russell \(1998\)R\. Parr and S\. RussellReinforcement learning with hierarchies of machines\.InAdvances in Neural Information Processing Systems 10 \(NeurIPS\),pp\. 1043–1049\.Cited by:[§7](https://arxiv.org/html/2609.05508#S7.SS0.SSS0.Px1.p1.1)\.
- Pattersonet al\.\(2024\)A\. Patterson, S\. Neumann, M\. White, and A\. WhiteEmpirical design in reinforcement learning\.Journal of Machine Learning Research25\(318\),pp\. 1–63\.Cited by:[Appendix B](https://arxiv.org/html/2609.05508#A2.p1.1),[§7](https://arxiv.org/html/2609.05508#S7.SS0.SSS0.Px5.p1.1)\.
- Precup \(2000\)D\. PrecupTemporal abstraction in reinforcement learning\.Ph\.D\. Thesis,University of Massachusetts Amherst\.Cited by:[§1](https://arxiv.org/html/2609.05508#S1.p1.1),[§7](https://arxiv.org/html/2609.05508#S7.SS0.SSS0.Px1.p1.1)\.
- Riemeret al\.\(2018\)M\. Riemer, M\. Liu, and G\. TesauroLearning abstract options\.InAdvances in Neural Information Processing Systems 31 \(NeurIPS\),pp\. 10445–10455\.Cited by:[§1](https://arxiv.org/html/2609.05508#S1.p1.1),[§7](https://arxiv.org/html/2609.05508#S7.SS0.SSS0.Px1.p1.1)\.
- Şimşek and Barto \(2009\)Ö\. Şimşek and A\. G\. BartoSkill characterization based on betweenness\.InAdvances in Neural Information Processing Systems 21 \(NeurIPS\),pp\. 1497–1504\.Cited by:[§7](https://arxiv.org/html/2609.05508#S7.SS0.SSS0.Px1.p1.1)\.
- Suttonet al\.\(1999\)R\. S\. Sutton, D\. Precup, and S\. SinghBetween MDPs and semi\-MDPs: a framework for temporal abstraction in reinforcement learning\.Artificial Intelligence112\(1–2\),pp\. 181–211\.Cited by:[§C\.7](https://arxiv.org/html/2609.05508#A3.SS7.p1.1),[§1](https://arxiv.org/html/2609.05508#S1.p1.1),[§2](https://arxiv.org/html/2609.05508#S2.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2609.05508#S7.SS0.SSS0.Px1.p1.1)\.
- Vezhnevetset al\.\(2017\)A\. S\. Vezhnevets, S\. Osindero, T\. Schaul, N\. Heess, M\. Jaderberg, D\. Silver, and K\. KavukcuogluFeUdal networks for hierarchical reinforcement learning\.InProceedings of the 34th International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.70,pp\. 3540–3549\.Cited by:[§7](https://arxiv.org/html/2609.05508#S7.SS0.SSS0.Px1.p1.1)\.
- Williams \(1992\)R\. J\. WilliamsSimple statistical gradient\-following algorithms for connectionist reinforcement learning\.Machine Learning8\(3–4\),pp\. 229–256\.Cited by:[§7](https://arxiv.org/html/2609.05508#S7.SS0.SSS0.Px4.p1.1)\.
- Zhang and Whiteson \(2019\)S\. Zhang and S\. WhitesonDAC: the double actor\-critic architecture for learning options\.InAdvances in Neural Information Processing Systems 32 \(NeurIPS\),pp\. 2010–2020\.Cited by:[§1](https://arxiv.org/html/2609.05508#S1.p1.1),[§7](https://arxiv.org/html/2609.05508#S7.SS0.SSS0.Px1.p1.1)\.

## Appendix AExperimental settings

#### Tabular experiments\.

FourRooms is a13×1313\\times 13grid with104104free cells; the goal is the east doorway, and the103103remaining cells form both the initial\-state distribution and the domain of the state\-uniform diagnostics\. Actions are the four cardinal moves; a move into a wall leaves the agent in place\. The intra\-option policy is a softmax over a104×4104\\times 4preference table with temperatureτ=10−3\\tau=10^\{\-3\}and learning rate0\.250\.25; the termination function is a sigmoid over a104104\-vector with learning rate0\.250\.25; the critic maintains tabularQΩQ\_\{\\Omega\}andQUQ\_\{U\}over the same104104states with learning rate0\.50\.5; the policy over options isεup\\varepsilon\_\{\\mathrm\{up\}\}\-greedy onQΩQ\_\{\\Omega\}withεup=0\.1\\varepsilon\_\{\\mathrm\{up\}\}=0\.1\. The discount is0\.990\.99\. Each cell of every grid runs seeds00through349349for10001000episodes with a10001000\-step cap\. Five distinct kernels are obtained from local slip and global teleport atp∈\{0,13,12\}p\\in\\\{0,\\tfrac\{1\}\{3\},\\tfrac\{1\}\{2\}\\\}, withp=0p=0shared by the two families\. The matched\-fork experiment saves a checkpoint every1010episodes, giving5151prefix checkpoints and5151per branch, and snapshots the Mersenne Twister state so that the fork is exact\.

#### Neural experiments\.

The continuous environments place the agent at cell centres and use the same13×1313\\times 13layout\. In Direct, an action displaces the agent by one unit in a cardinal direction with uniform noise in\[−0\.1,0\.1\]\[\-0\.1,0\.1\]per coordinate, and the observation is\(x,y\)\(x,y\)\. In Inertia, an action changes the corresponding velocity component by0\.1±0\.010\.1\\pm 0\.01, speed is clipped to0\.990\.99per axis, position integrates velocity, a collision zeroes it, and the observation is\(x,y,vx,vy\)\(x,y,v\_\{x\},v\_\{y\}\)\. The goal tolerance is half a cell\.QΩQ\_\{\\Omega\},QUQ\_\{U\}, the policy, and the termination function are two\-hidden\-layer tanh networks of width6464, trained with Adam at learning rate10−310^\{\-3\}, Polyak\-averaged target networks with coefficient0\.0050\.005, a replay buffer of10,00010\{,\}000transitions, batches of6464, and one update every44environment steps after10001000steps of warm\-up\. Output biases are initialised so that the mean output over a grid of states is zero\. Each configuration runs5050seeds for10001000episodes with a10001000\-step cap\. Direct usesτ=10−3\\tau=10^\{\-3\}for the base andε\\varepsilon\-greedy configurations withεlo=0\.1\\varepsilon\_\{\\mathrm\{lo\}\}=0\.1, andτ=1\\tau=1with entropy coefficient0\.50\.5for the entropy configuration; Inertia usesτ=0\.5\\tau=0\.5throughout, withεlo=0\.5\\varepsilon\_\{\\mathrm\{lo\}\}=0\.5and entropy coefficient0\.10\.1\.

## Appendix BStatistics

The tabular grids use350350training seeds per cell, matched across every condition being contrasted, so all contrasts are paired within seed\. Confidence intervals are95%95\\%percentile intervals from100,000100\{,\}000paired bootstrap resamples over seeds\([Colas et al\., 2018](https://arxiv.org/html/2609.05508#bib.bib33);[Patterson et al\., 2024](https://arxiv.org/html/2609.05508#bib.bib32)\), with fixed bootstrap seeds recorded alongside the derived tables\. The termination grid retains block\-level aggregates, so its resampling unit is a3535\-seed rank block, ten per cell; the neural experiments resample5050individual seeds\. Where the five kernels are pooled we average within a seed first and bootstrap over seeds, counting the duplicatedp=0p=0cell once\. The frozen evaluation is an exact dynamic program and contributes no sampling error, so every reported interval reflects training\-seed variability alone\.

## Appendix CTermination rules and regret

This appendix supplies the model, the algorithms and the proofs behind Section[4\.1](https://arxiv.org/html/2609.05508#S4.SS1)\. Two remarks frame what is and is not claimed\. First, “both levels use UCB” does not by itself pin down an algorithm, so the analysis fixes a minimal but non\-trivial two\-level model in which the interaction between termination and exploration is exposed without any other moving part\. Second, the negative results are constructions inside that model, which is a special case of the option setting, so they suffice to rule out a general guarantee\.

### C\.1A two\-level stochastic bandit

LetTTbe the number of atomic steps\. The environment has a single recurring statess\. Let𝒪=\{1,…,K\}\\mathcal\{O\}=\\\{1,\\dots,K\\\}be the option set and𝒜o\\mathcal\{A\}\_\{o\}the actions available inside optionoo, and writeℒ=\{\(o,a\):o∈𝒪,a∈𝒜o\}\\mathcal\{L\}=\\\{\(o,a\):o\\in\\mathcal\{O\},\\ a\\in\\mathcal\{A\}\_\{o\}\\\}for the set of option–action leaves, withM=\|ℒ\|M=\|\\mathcal\{L\}\|\. Pulling leafi∈ℒi\\in\\mathcal\{L\}returns an independent reward in\[0,1\]\[0,1\]with meanμi\\mu\_\{i\}\. Putμo⋆=maxa⁡μo,a\\mu\_\{o\}^\{\\star\}=\\max\_\{a\}\\mu\_\{o,a\},μ⋆=maxo⁡μo⋆\\mu^\{\\star\}=\\max\_\{o\}\\mu\_\{o\}^\{\\star\}, and define the upper, lower and total gaps

Δoup=μ⋆−μo⋆,Δo,alo=μo⋆−μo,a,Δo,a=μ⋆−μo,a=Δoup\+Δo,alo\.\\Delta\_\{o\}^\{\\mathrm\{up\}\}=\\mu^\{\\star\}\-\\mu\_\{o\}^\{\\star\},\\qquad\\Delta\_\{o,a\}^\{\\mathrm\{lo\}\}=\\mu\_\{o\}^\{\\star\}\-\\mu\_\{o,a\},\\qquad\\Delta\_\{o,a\}=\\mu^\{\\star\}\-\\mu\_\{o,a\}=\\Delta\_\{o\}^\{\\mathrm\{up\}\}\+\\Delta\_\{o,a\}^\{\\mathrm\{lo\}\}\.WithIt=\(Ot,At\)I\_\{t\}=\(O\_\{t\},A\_\{t\}\)the leaf executed at stepttandNi​\(T\)N\_\{i\}\(T\)its visit count, the pseudo\-regret isℛT=T​μ⋆−𝔼⁡\[∑t≤TμIt\]=∑iΔi​𝔼​\[Ni​\(T\)\]\\mathcal\{R\}\_\{T\}=T\\mu^\{\\star\}\-\\mathbb\{E\}\[\\sum\_\{t\\leq T\}\\mu\_\{I\_\{t\}\}\]=\\sum\_\{i\}\\Delta\_\{i\}\\,\\mathbb\{E\}\[N\_\{i\}\(T\)\]\. Splitting by level, withNo​\(T\)=∑aNo,a​\(T\)N\_\{o\}\(T\)=\\sum\_\{a\}N\_\{o,a\}\(T\),

ℛTup=∑oΔoup​𝔼​\[No​\(T\)\],ℛTlo=∑o,aΔo,alo​𝔼​\[No,a​\(T\)\]\.\\mathcal\{R\}\_\{T\}^\{\\mathrm\{up\}\}=\\sum\_\{o\}\\Delta\_\{o\}^\{\\mathrm\{up\}\}\\mathbb\{E\}\[N\_\{o\}\(T\)\],\\qquad\\mathcal\{R\}\_\{T\}^\{\\mathrm\{lo\}\}=\\sum\_\{o,a\}\\Delta\_\{o,a\}^\{\\mathrm\{lo\}\}\\mathbb\{E\}\[N\_\{o,a\}\(T\)\]\.
###### Lemma 4\(Exact decomposition\)\.

For any selection algorithm and anyTT,ℛT=ℛTup\+ℛTlo\\mathcal\{R\}\_\{T\}=\\mathcal\{R\}\_\{T\}^\{\\mathrm\{up\}\}\+\\mathcal\{R\}\_\{T\}^\{\\mathrm\{lo\}\}\.

###### Proof\.

SubstituteΔo,a=Δoup\+Δo,alo\\Delta\_\{o,a\}=\\Delta\_\{o\}^\{\\mathrm\{up\}\}\+\\Delta\_\{o,a\}^\{\\mathrm\{lo\}\}intoℛT=∑o,aΔo,a​𝔼​\[No,a​\(T\)\]\\mathcal\{R\}\_\{T\}=\\sum\_\{o,a\}\\Delta\_\{o,a\}\\mathbb\{E\}\[N\_\{o,a\}\(T\)\]and useNo​\(T\)=∑aNo,a​\(T\)N\_\{o\}\(T\)=\\sum\_\{a\}N\_\{o,a\}\(T\)on the first summand\. ∎

### C\.2Compatible two\-level UCB, and the two termination rules

Each leaf is pulled once during initialisation\. Afterwards, writebi​\(t\)=μ^i​\(t−1\)\+2​ln⁡t/Ni​\(t−1\)b\_\{i\}\(t\)=\\widehat\{\\mu\}\_\{i\}\(t\-1\)\+\\sqrt\{2\\ln t/N\_\{i\}\(t\-1\)\}for the UCB1 index of leafii\. The lower level proposesao\+​\(t\)∈arg⁡maxa∈𝒜o​bo,a​\(t\)a\_\{o\}^\{\+\}\(t\)\\in\\arg\\max\_\{a\\in\\mathcal\{A\}\_\{o\}\}b\_\{o,a\}\(t\)inside each option, the upper level scores options byBo​\(t\)=bo,ao\+​\(t\)​\(t\)=maxa⁡bo,a​\(t\)B\_\{o\}\(t\)=b\_\{o,a\_\{o\}^\{\+\}\(t\)\}\(t\)=\\max\_\{a\}b\_\{o,a\}\(t\)and proposesJt∈arg⁡maxo​Bo​\(t\)J\_\{t\}\\in\\arg\\max\_\{o\}B\_\{o\}\(t\)\. Ties are broken by a fixed deterministic rule\. The two levels are compatible in the sense that the option score is the maximum of its own leaf indices, which is what makes UCB1’s assumptions applicable\.

Two algorithms are compared\.𝖠fix\\mathsf\{A\}\_\{\\mathrm\{fix\}\}fixesβ≡1\\beta\\equiv 1: at every step it setsOt=JtO\_\{t\}=J\_\{t\}and executesAt=aOt\+​\(t\)A\_\{t\}=a\_\{O\_\{t\}\}^\{\+\}\(t\)\.𝖠gi\\mathsf\{A\}\_\{\\mathrm\{gi\}\}carries a current optionCtC\_\{t\}into the step and applies the greedy\-interruption rule \([2](https://arxiv.org/html/2609.05508#S4.E2)\): ifβt=1\\beta\_\{t\}=1it setsOt=JtO\_\{t\}=J\_\{t\}, and ifβt=0\\beta\_\{t\}=0it setsOt=CtO\_\{t\}=C\_\{t\}; either way it then executesAt=aOt\+​\(t\)A\_\{t\}=a\_\{O\_\{t\}\}^\{\+\}\(t\)and passesCt\+1=OtC\_\{t\+1\}=O\_\{t\}to the next step\. WriteℛTfix\\mathcal\{R\}\_\{T\}^\{\\mathrm\{fix\}\}andℛTgi\\mathcal\{R\}\_\{T\}^\{\\mathrm\{gi\}\}for their pseudo\-regrets;ℛTfix\\mathcal\{R\}\_\{T\}^\{\\mathrm\{fix\}\}is theℛTβ≡1\\mathcal\{R\}\_\{T\}^\{\\beta\\equiv 1\}of the main text\.

### C\.3Fixedβ≡1\\beta\\equiv 1: finite\-time bounds

###### Theorem 5\(Stepwise equivalence to leaf\-level UCB1\)\.

Under𝖠fix\\mathsf\{A\}\_\{\\mathrm\{fix\}\},It∈arg⁡maxi∈ℒ​bi​\(t\)I\_\{t\}\\in\\arg\\max\_\{i\\in\\mathcal\{L\}\}b\_\{i\}\(t\)at every step, so𝖠fix\\mathsf\{A\}\_\{\\mathrm\{fix\}\}is UCB1 run directly on theMMleaves\.

###### Proof\.

bIt​\(t\)=BJt​\(t\)=maxo⁡Bo​\(t\)=maxo⁡maxa​bo,a​\(t\)=maxi⁡bi​\(t\)b\_\{I\_\{t\}\}\(t\)=B\_\{J\_\{t\}\}\(t\)=\\max\_\{o\}B\_\{o\}\(t\)=\\max\_\{o\}\\max\_\{a\}b\_\{o,a\}\(t\)=\\max\_\{i\}b\_\{i\}\(t\), and the fixed tie\-breaking rule selects the same leaf in both formulations\. ∎

###### Theorem 6\(Instance\-dependent bound\)\.

LetC0=1\+π2/3C\_\{0\}=1\+\\pi^\{2\}/3\. For rewards in\[0,1\]\[0,1\]and anyT≥MT\\geq M,

ℛTfix≤∑\(o,a\):Δo,a\>0\(8​ln⁡TΔo,a\+C0Δo,a\),\\mathcal\{R\}\_\{T\}^\{\\mathrm\{fix\}\}\\;\\leq\\;\\sum\_\{\(o,a\):\\Delta\_\{o,a\}\>0\}\\Bigl\(\\frac\{8\\ln T\}\{\\Delta\_\{o,a\}\}\+C\_\{0\}\\Delta\_\{o,a\}\\Bigr\),and, level by level,ℛTup,fix≤∑\(o,a\):Δo,a\>0Δoup\(8lnT/Δo,a2\+C0\)\\mathcal\{R\}\_\{T\}^\{\\mathrm\{up,fix\}\}\\leq\\sum\_\{\(o,a\):\\Delta\_\{o,a\}\>0\}\\Delta\_\{o\}^\{\\mathrm\{up\}\}\(8\\ln T/\\Delta\_\{o,a\}^\{2\}\+C\_\{0\}\)andℛTlo,fix≤∑\(o,a\):Δo,a\>0Δo,alo\(8lnT/Δo,a2\+C0\)\\mathcal\{R\}\_\{T\}^\{\\mathrm\{lo,fix\}\}\\leq\\sum\_\{\(o,a\):\\Delta\_\{o,a\}\>0\}\\Delta\_\{o,a\}^\{\\mathrm\{lo\}\}\(8\\ln T/\\Delta\_\{o,a\}^\{2\}\+C\_\{0\}\)\.

###### Proof\.

UCB1 guarantees𝔼⁡\[Ni​\(T\)\]≤8​ln⁡T/Δi2\+C0\\mathbb\{E\}\[N\_\{i\}\(T\)\]\\leq 8\\ln T/\\Delta\_\{i\}^\{2\}\+C\_\{0\}for every suboptimal arm\([Auer et al\., 2002](https://arxiv.org/html/2609.05508#bib.bib37)\)\. By Theorem[5](https://arxiv.org/html/2609.05508#Thmtheorem5)the arms are the leaves, so multiplying byΔi\\Delta\_\{i\}and summing gives the first display\. Multiplying the same visit\-count bound byΔoup\\Delta\_\{o\}^\{\\mathrm\{up\}\}and byΔo,alo\\Delta\_\{o,a\}^\{\\mathrm\{lo\}\}and applying Lemma[4](https://arxiv.org/html/2609.05508#Thmtheorem4)gives the other two\. ∎

###### Corollary 7\(Distribution\-free bound\)\.

ForT≥max⁡\{M,2\}T\\geq\\max\\\{M,2\\\},ℛTfix≤min⁡\{T,4​2​M​T​ln⁡T\+C0​M\}\\mathcal\{R\}\_\{T\}^\{\\mathrm\{fix\}\}\\leq\\min\\\{T,\\;4\\sqrt\{2MT\\ln T\}\+C\_\{0\}M\\\}\.

###### Proof\.

Split the leaves at a thresholdε∈\(0,1\]\\varepsilon\\in\(0,1\]\. Leaves withΔi≤ε\\Delta\_\{i\}\\leq\\varepsiloncontribute at mostT​εT\\varepsilon; the rest contribute at most8​M​ln⁡T/ε\+C0​M8M\\ln T/\\varepsilon\+C\_\{0\}Mby Theorem[6](https://arxiv.org/html/2609.05508#Thmtheorem6)\. Choosingε=8​M​ln⁡T/T\\varepsilon=\\sqrt\{8M\\ln T/T\}makes the first two terms4​2​M​T​ln⁡T4\\sqrt\{2MT\\ln T\}\. If thatε\\varepsilonexceeds11the bound already exceeds the trivial boundTT, which always holds because single\-step gaps are at most one\. ∎

A matching lower bound is classical: for Bernoulli leaves with a unique optimum and any consistent algorithm,lim infTℛT/lnT≥∑i:Δi\>0Δi/kl\(μi,μ⋆\)\\liminf\_\{T\}\\mathcal\{R\}\_\{T\}/\\ln T\\geq\\sum\_\{i:\\Delta\_\{i\}\>0\}\\Delta\_\{i\}/\\operatorname\{kl\}\(\\mu\_\{i\},\\mu^\{\\star\}\)\([Lai and Robbins, 1985](https://arxiv.org/html/2609.05508#bib.bib38)\), so theO⁡\(ln⁡T\)O\(\\ln T\)rate of Theorem[6](https://arxiv.org/html/2609.05508#Thmtheorem6)cannot be improved too⁡\(ln⁡T\)o\(\\ln T\)on a fixed non\-degenerate instance\.

### C\.4Two value settings for the return\-maximising rule

Rule \([2](https://arxiv.org/html/2609.05508#S4.E2)\) compares the value of continuing with the value of handing control back, and everything turns on which estimates those two are\.

#### Optimistic indices\.

SetQt\+​\(s,o\)=Bo​\(t\)Q\_\{t\}^\{\+\}\(s,o\)=B\_\{o\}\(t\)and let the upper policy be the point mass onJtJ\_\{t\}, soVt\+​\(s\)=BJt​\(t\)=maxo⁡Bo​\(t\)V\_\{t\}^\{\+\}\(s\)=B\_\{J\_\{t\}\}\(t\)=\\max\_\{o\}B\_\{o\}\(t\)\. Proposition[1](https://arxiv.org/html/2609.05508#Thmtheorem1)of the main text is then immediate\.

###### Proof of Proposition[1](https://arxiv.org/html/2609.05508#Thmtheorem1)\.

For any current optionCtC\_\{t\}we haveQt\+​\(s,Ct\)=BCt​\(t\)≤maxo⁡Bo​\(t\)=Vt\+​\(s\)Q\_\{t\}^\{\+\}\(s,C\_\{t\}\)=B\_\{C\_\{t\}\}\(t\)\\leq\\max\_\{o\}B\_\{o\}\(t\)=V\_\{t\}^\{\+\}\(s\), so \([2](https://arxiv.org/html/2609.05508#S4.E2)\) selectsβt=1\\beta\_\{t\}=1and the step executesJtJ\_\{t\}\. Induct ontt: the two algorithms start identically; if the histories before stepttagree then all counts, sample means and indices agree, henceJtJ\_\{t\}agrees, and both algorithms executeJtJ\_\{t\}with the same lower\-level action, so the histories after stepttagree\. Action sequences, and therefore regret, coincide on every sample path\. ∎

#### Raw\-value indices\.

Option\-critic’s state value is the upper policy’s expectation of raw critic values, while a confidence bonus is a device the exploration rule adds on top\. In the present model that setting is

q^o​\(t\)=μ^o,ao\+​\(t\)​\(t−1\),Qt​\(s,o\)=q^o​\(t\),Vt​\(s\)=q^Jt​\(t\),\\widehat\{q\}\_\{o\}\(t\)=\\widehat\{\\mu\}\_\{o,a\_\{o\}^\{\+\}\(t\)\}\(t\-1\),\\qquad Q\_\{t\}\(s,o\)=\\widehat\{q\}\_\{o\}\(t\),\\qquad V\_\{t\}\(s\)=\\widehat\{q\}\_\{J\_\{t\}\}\(t\),so the option the selector proposes need not be the one with the largest raw value\. WhenBJt​\(t\)\>BCt​\(t\)B\_\{J\_\{t\}\}\(t\)\>B\_\{C\_\{t\}\}\(t\)andQt​\(s,Jt\)<Qt​\(s,Ct\)Q\_\{t\}\(s,J\_\{t\}\)<Q\_\{t\}\(s,C\_\{t\}\)hold at once, the selector asks to exploreJtJ\_\{t\}and the termination test refuses\.

### C\.5Proof of Theorem[2](https://arxiv.org/html/2609.05508#Thmtheorem2)

#### Linear lower bound\.

Take two options\. Both actions of the good optionGGreturnBer⁡\(3/4\)\\operatorname\{Ber\}\(3/4\); both actions of the bad optionBBreturn exactly1/21/2\. ThenμG⋆=3/4\\mu\_\{G\}^\{\\star\}=3/4,μB⋆=1/2\\mu\_\{B\}^\{\\star\}=1/2,ΔBup=1/4\\Delta\_\{B\}^\{\\mathrm\{up\}\}=1/4, and every lower\-level gap is zero, so any regret is caused by the option choice alone\. Initialise in the order\(G,1\),\(G,2\),\(B,1\),\(B,2\)\(G,1\),\(G,2\),\(B,1\),\(B,2\)without applying the termination test, so the current option after step four isBB\. LetEEbe the event that both initialisation draws fromGGreturn00, which has probability\(1/4\)2=1/16\(1/4\)^\{2\}=1/16\.

OnEEboth sample means ofGGare00while both ofBBare1/21/2\. We show by induction that the algorithm executesBBforever\. Assume the current option before stept≥5t\\geq 5isBBandGGhas received no further samples\. IfJt=BJ\_\{t\}=BthenQt​\(s,Ct\)=1/2=Vt​\(s\)Q\_\{t\}\(s,C\_\{t\}\)=1/2=V\_\{t\}\(s\), and the convention that ties terminate givesβt=1\\beta\_\{t\}=1, after which the reselection returnsBB\. IfJt=GJ\_\{t\}=GthenQt​\(s,Ct\)=1/2\>0=Qt​\(s,G\)=Vt​\(s\)Q\_\{t\}\(s,C\_\{t\}\)=1/2\>0=Q\_\{t\}\(s,G\)=V\_\{t\}\(s\), soβt=0\\beta\_\{t\}=0and the step continues withBB\. Either wayBBexecutes and the hypothesis is preserved\. OnEEthe bad option therefore runs on allT−2T\-2steps from step three onwards, each at pseudo\-regret1/41/4, soℛTgi≥116⋅14​\(T−2\)=\(T−2\)/64\\mathcal\{R\}\_\{T\}^\{\\mathrm\{gi\}\}\\geq\\frac\{1\}\{16\}\\cdot\\frac\{1\}\{4\}\(T\-2\)=\(T\-2\)/64\. Since single\-step gaps are at most one,ℛTgi≤T\\mathcal\{R\}\_\{T\}^\{\\mathrm\{gi\}\}\\leq Talways, and the worst case over the class isΘ⁡\(T\)\\Theta\(T\)\.

#### Logarithmic upper bound on the same instance\.

That instance has two optimal leaves at3/43/4and two suboptimal leaves at1/21/2, each with total gap1/41/4\. Theorem[6](https://arxiv.org/html/2609.05508#Thmtheorem6)givesℛTfix≤2​\(8​ln⁡T/\(1/4\)\+C0/4\)=64​ln⁡T\+C0/2\\mathcal\{R\}\_\{T\}^\{\\mathrm\{fix\}\}\\leq 2\(8\\ln T/\(1/4\)\+C\_\{0\}/4\)=64\\ln T\+C\_\{0\}/2\. HenceℛTfix/ℛTgi→0\\mathcal\{R\}\_\{T\}^\{\\mathrm\{fix\}\}/\\mathcal\{R\}\_\{T\}^\{\\mathrm\{gi\}\}\\to 0\. The class\-level statement combines this with Corollary[7](https://arxiv.org/html/2609.05508#Thmtheorem7)\.

#### Episode resets do not repair it\.

The continuing counterexample is broken by an unconditional reset of the option at every episode boundary, but resets do not restore a sublinear rate in general\. Take a two\-step episodic MDP\. At the first step, states0s\_\{0\}admits only the bad optionBB; its single action returns00and moves deterministically tos1s\_\{1\}, so the current option on arrival ats1s\_\{1\}is alwaysBB\. Ats1s\_\{1\}both options are available with two actions each; both actions ofBBreturn exactly1/41/4, both actions ofGGreturnBer⁡\(3/4\)\\operatorname\{Ber\}\(3/4\), and the episode ends\. The optimal episode value is3/43/4and choosingBBats1s\_\{1\}costs1/21/2\. Initialise the four leaves ats1s\_\{1\}over the first four episodes without applying the test, which already costs11, and letEEbe as before\. OnEEthe critic values ats1s\_\{1\}are00for both actions ofGGand1/41/4for both actions ofBB; since the fixed prefix rebuildsC=BC=Bats1s\_\{1\}in every episode, the argument above applies unchanged andBBexecutes ats1s\_\{1\}in all remainingN−4N\-4episodes\. Hence𝔼​RegNgi≥1\+\(N−4\)/32=1\+\(T−8\)/64\\mathbb\{E\}\\operatorname\{Reg\}\_\{N\}^\{\\mathrm\{gi\}\}\\geq 1\+\(N\-4\)/32=1\+\(T\-8\)/64withT=2​NT=2N, whileβ≡1\\beta\\equiv 1ats1s\_\{1\}is UCB1 over four leaves with gaps1/21/2and satisfies𝔼​RegNβ≡1≤32​ln⁡N\+C0\\mathbb\{E\}\\operatorname\{Reg\}\_\{N\}^\{\\beta\\equiv 1\}\\leq 32\\ln N\+C\_\{0\}\. Removing this family of counterexamples needs a condition stronger than resetting the option, for instance a guarantee that every available state–option pair is actually executed and not merely proposed\.

### C\.6No pointwise dominance

Theorem[2](https://arxiv.org/html/2609.05508#Thmtheorem2)is a worst\-case separation, and the closing paragraph of Section[4\.1](https://arxiv.org/html/2609.05508#S4.SS1)records that it cannot be upgraded\. Consider a deterministic instance withμG,1=μG,2=1\\mu\_\{G,1\}=\\mu\_\{G,2\}=1andμB,1=μB,2=0\\mu\_\{B,1\}=\\mu\_\{B,2\}=0, initialised in the order\(B,1\),\(B,2\),\(G,1\),\(G,2\)\(B,1\),\(B,2\),\(G,1\),\(G,2\), so the current option after initialisation isGGwithq^G=1\\widehat\{q\}\_\{G\}=1andq^B=0\\widehat\{q\}\_\{B\}=0\. Under the raw\-value indices, a proposal ofGGgivesQt​\(s,Ct\)=Vt​\(s\)Q\_\{t\}\(s,C\_\{t\}\)=V\_\{t\}\(s\)and terminates intoGGagain, and a proposal ofBBgivesQt​\(s,Ct\)=1\>0=Vt​\(s\)Q\_\{t\}\(s,C\_\{t\}\)=1\>0=V\_\{t\}\(s\)and continues withGG; so𝖠gi\\mathsf\{A\}\_\{\\mathrm\{gi\}\}executesGGfor ever andℛTgi=2\\mathcal\{R\}\_\{T\}^\{\\mathrm\{gi\}\}=2for allT≥4T\\geq 4\. Underβ≡1\\beta\\equiv 1the leaf indices force a return toBB: att=11t=11,BB​\(11\)=2​ln⁡11\>1\+2​ln⁡11/4=BG​\(11\)B\_\{B\}\(11\)=\\sqrt\{2\\ln 11\}\>1\+\\sqrt\{2\\ln 11/4\}=B\_\{G\}\(11\), soℛ11fix=3\>2=ℛ11gi\\mathcal\{R\}\_\{11\}^\{\\mathrm\{fix\}\}=3\>2=\\mathcal\{R\}\_\{11\}^\{\\mathrm\{gi\}\}\. The excursions recur, because ifβ≡1\\beta\\equiv 1pulledBBonly finitely often its index would grow like2​ln⁡t/m→∞\\sqrt\{2\\ln t/m\}\\to\\inftywhileBG​\(t\)→1B\_\{G\}\(t\)\\to 1\. The veto in \([2](https://arxiv.org/html/2609.05508#S4.E2)\) therefore has two faces: it locks in an error when the current option is suboptimal and the unexplored option is not, and it saves exploration cost when the current option is already optimal\.

### C\.7Relation to the interruption theorem

[Sutton et al\. \(1999\)](https://arxiv.org/html/2609.05508#bib.bib1)show that interrupting an option whenever continuing is worth less than reselecting under a*fixed*Markov policyμ\\muimproves the value everywhere and strictly somewhere\. Two hypotheses of that theorem fail for the online process analysed here\. The quantitiesQt,VtQ\_\{t\},V\_\{t\}are finite\-sample estimates and not exact values of a fixed policy, and the upper policy changes with the counts, so there is no fixedμ\\muto improve upon\. In the counterexample,GGis genuinely worth3/43/4, an unlucky initialisation makes its estimate00, and the termination rule reads that estimation error as permanent evidence for continuing withBBwhile simultaneously preventing the samples that would correct it\. The classical theorem concerns policy improvement once values are known; it makes no claim about this estimation\-visitation feedback loop\.

### C\.8Episodic tabular RL

Let𝒮\\mathcal\{S\}be a finite state set withS=\|𝒮\|S=\|\\mathcal\{S\}\|, letHHbe the episode length,NNthe number of episodes andT=N​HT=NH\. Write𝒰\(s\)=\{\(o,a\):o∈𝒪\(s\),a∈𝒜o\(s\)\}\\mathcal\{U\}\(s\)=\\\{\(o,a\):o\\in\\mathcal\{O\}\(s\),a\\in\\mathcal\{A\}\_\{o\}\(s\)\\\}for the augmented actions atssandD=maxs⁡\|𝒰⁡\(s\)\|D=\\max\_\{s\}\|\\mathcal\{U\}\(s\)\|\. A Bellman\-consistent optimistic algorithm maintainsQ¯k,h​\(s,o,a\)\\overline\{Q\}\_\{k,h\}\(s,o,a\)from the optimistic Bellman recursion of a single augmented MDP, and selectsak,h\+​\(s,o\)∈arg⁡maxa​Q¯k,h​\(s,o,a\)a\_\{k,h\}^\{\+\}\(s,o\)\\in\\arg\\max\_\{a\}\\overline\{Q\}\_\{k,h\}\(s,o,a\)below andok,h\+​\(s\)∈arg⁡maxo​Q¯k,h​\(s,o,ak,h\+​\(s,o\)\)o\_\{k,h\}^\{\+\}\(s\)\\in\\arg\\max\_\{o\}\\overline\{Q\}\_\{k,h\}\(s,o,a\_\{k,h\}^\{\+\}\(s,o\)\)above\.

###### Theorem 8\(Augmented isomorphism\)\.

Underβ≡1\\beta\\equiv 1the executed pair satisfiesuk,h∈arg⁡maxu∈𝒰⁡\(sk,h\)​Q¯k,h​\(sk,h,u\)u\_\{k,h\}\\in\\arg\\max\_\{u\\in\\mathcal\{U\}\(s\_\{k,h\}\)\}\\overline\{Q\}\_\{k,h\}\(s\_\{k,h\},u\), and the hierarchical agent produces exactly the same states, actions, rewards and successor states as the same optimistic algorithm run onℳ\+=\(𝒮,𝒰,P,r,H\)\\mathcal\{M\}^\{\+\}=\(\\mathcal\{S\},\\mathcal\{U\},P,r,H\)\.

###### Proof\.

Nested maximisation is joint maximisation:Q¯​\(s,o\+,a\+​\(s,o\+\)\)=maxo⁡maxa​Q¯​\(s,o,a\)=maxu⁡Q¯​\(s,u\)\\overline\{Q\}\(s,o^\{\+\},a^\{\+\}\(s,o^\{\+\}\)\)=\\max\_\{o\}\\max\_\{a\}\\overline\{Q\}\(s,o,a\)=\\max\_\{u\}\\overline\{Q\}\(s,u\)\. Withβ≡1\\beta\\equiv 1the option is resampled at every step, so no constraint carries over from the previous step and the pairuuis free\. Executinguuyields rewardr⁡\(s,u\)r\(s,u\)and successorP\(⋅\|s,u\)P\(\\cdot\\,\|\\,s,u\)by definition ofℳ\+\\mathcal\{M\}^\{\+\}, so the observation histories agree, and induction propagates the agreement to all counts and optimistic values\. ∎

Theorem[3](https://arxiv.org/html/2609.05508#Thmtheorem3)follows by instantiating the optimistic algorithm as UCBVI\-BF and quoting its guarantee\.

###### Proof of Theorem[3](https://arxiv.org/html/2609.05508#Thmtheorem3)\.

Assume known deterministic rewards in\[0,1\]\[0,1\], unknown transitions, a common augmented action set of sizeDD, and failure probabilityδ\\delta\. By Theorem[8](https://arxiv.org/html/2609.05508#Thmtheorem8)the hierarchical agent and UCBVI\-BF onℳ\+\\mathcal\{M\}^\{\+\}have identical episode regret\. Theorem 2 of[Azar et al\. \(2017\)](https://arxiv.org/html/2609.05508#bib.bib39)gives, with probability at least1−δ1\-\\deltaandL=ln⁡\(5​H​S​D​T/δ\)L=\\ln\(5HSDT/\\delta\),RegN≤30​H​L​S​D​N\+2500​H2​S2​D​L2\+4​H3/2​N​L\\operatorname\{Reg\}\_\{N\}\\leq 30HL\\sqrt\{SDN\}\+2500H^\{2\}S^\{2\}DL^\{2\}\+4H^\{3/2\}\\sqrt\{NL\}\. SubstitutingN=T/HN=T/Hturns the first term into30​L​H​S​D​T30L\\sqrt\{HSDT\}and the third into4​H​T​L4H\\sqrt\{TL\}\. ∎

Two caveats belong with this bound\. If rewards and transitions depend only on the primitive action, distinct pairs\(o,a\)\(o,a\)duplicate one another, and takingD≤\|𝒪\|​\|𝒜\|D\\leq\|\\mathcal\{O\}\|\|\\mathcal\{A\}\|is correct but loose unless the algorithm pools statistics across options\. And UCBVI is model\-based: a bonus at each level is not enough to invoke the theorem, since the optimistic values must satisfy the corresponding Bellman recursion and confidence event\.

### C\.9Summary

Consider any tabular scheme whose two\-level selection is equivalent to maximisation over augmented actions\. If the behaviour selector and the termination test share one value function and the state value is the maximum over option values, thenβ≡1\\beta\\equiv 1holds pathwise, the two algorithms coincide, and both inherit whichever guarantee the underlying optimistic algorithm carries\. If the selector uses optimistic values while the termination test uses raw values, no uniform sublinear guarantee survives, even atH=2H=2with per\-episode option resets\. The two rules are not comparable instance by instance\. What can be asserted, and what Section[4\.1](https://arxiv.org/html/2609.05508#S4.SS1)asserts, is that a termination rule trained to maximise return is redundant in the consistent case and unguaranteed in the standard case\.

## Appendix DAdditional results

Table 4:Learned termination and forced per\-step reselection are indistinguishable under exact frozen evaluation\.Shortest\-path FourRooms, episode\-1000 checkpoints, expected capped transitions and success probability under horizon\-1000 exact dynamic programming withεlo=0\\varepsilon\_\{\\mathrm\{lo\}\}=0, averaged over the five kernels within seed\. Differences are paired over350350matched seeds with95%95\\%bootstrap intervals; negative favours learned termination\. AtK=1K\{=\}1termination cannot change which option runs, so that row measures the noise floor of the seed matching\.![Refer to caption](https://arxiv.org/html/2609.05508v1/fig2_termination.png)Figure 5:Table[1](https://arxiv.org/html/2609.05508#S4.T1)as a heat map\.Last\-100\-episode step count of learned termination minus that of a matchedβ≡1\\beta\\equiv 1agent, in absolute steps \(top of each cell\) and relative to the control \(bottom\)\. Blue favours learned termination and red favoursβ≡1\\beta\\equiv 1\. The tabular grid reverses sign completely betweenK=2K\{=\}2andK=8K\{=\}8; the neural grid has no consistent direction\.Figure 6:The sparse\-reward replication of Figure[4](https://arxiv.org/html/2609.05508#S6.F4)\.Episode\-1000 control checkpoints of the matched\-fork grid, five\-kernel average over350350matched seeds, with necrosis defined by the1%1\\%optimal\-action mass criterion of Section[3](https://arxiv.org/html/2609.05508#S3)\. Joint necrosis falls47\.5→31\.8→14\.6→3\.347\.5\\rightarrow 31\.8\\rightarrow 14\.6\\rightarrow 3\.3percent while the mean single\-option rate stays near50%50\\%, and the router’s miss rate rises to39\.2%39\.2\\%; expected steps fall306\.7→152\.9→79\.1→56\.4306\.7\\rightarrow 152\.9\\rightarrow 79\.1\\rightarrow 56\.4\. The effective independent option count isκ=1\.82\\kappa=1\.82,2\.852\.85and4\.144\.14atK=2,4,8K=2,4,8\.![Refer to caption](https://arxiv.org/html/2609.05508v1/figA_statemaps_full.png)Figure 7:Where coverage appears, and where the router loses it\.Every non\-goal cell of constant\-cost FourRooms under teleportp=12p=\\tfrac\{1\}\{2\}, at the episode\-1000 checkpoint of the spatial\-medoid seed, chosen by minimum Hamming distance across all twelve panels without reference to performance\. Dark grey is wall and yellow is the goal\. The middle row is aβ≡1\\beta\\equiv 1agent trained without lower\-level exploration; its maps are visually and numerically close to the learned\-termination agent above it at every option count, which is the coverage\-level statement of Section[4](https://arxiv.org/html/2609.05508#S4)\.