Completion vs Optimality: Policy Gradient in Long-Horizon Cumulative-Damage Problems

# Policy Gradient in Long-Horizon Cumulative-Damage Problems
Source: [https://arxiv.org/html/2605.26657](https://arxiv.org/html/2605.26657)
## Completion vs Optimality: Policy Gradient in Long\-Horizon Cumulative\-Damage Problems

Wolfgang Maass Saarland University & German Research Center for Artificial Intelligence \(DFKI\) wolfgang\.maass@iss\.uni\-saarland\.de&Sabine Janzen German Research Center for Artificial Intelligence \(DFKI\) sabine\.janzen@dfki\.de

###### Abstract

Long\-horizon decision problems with cumulative damage couple locally attractive actions to globally adverse outcomes\. We identify two orthogonal failure modes for policy\-gradient methods on this class and propose a decomposition that separates them:*completion*\(reaching the terminal horizon rather than exiting via an implicit terminal constraint\) and*optimality*\(matching the dynamic\-programming reference given completion\)\. Under PPO with a linear soft penalty, granting horizon access alone reduces the completion rate: the penalty’s equilibrium drives the dominant\-activity share to zero, while action\-space restriction combined with horizon access achieves completion but leaves an optimality gap \(Δ​Mfinal=0\.271\\Delta M\_\{\\text\{final\}\}=0\.271\) that we trace to first\-phase greedy commitment at the damage origin\. We derive four testable predictions and evaluate them in two separately calibrated environments that share the same abstract structure but differ in domain, horizon, activity set, and calibration data: a 49\-step bricklayer career and a 20\-season NBA power\-forward career\. All four predictions replicate qualitatively\. The horizon\-invariance prediction is met at three of four tested horizons, with the exception atH=15H=15consistent with theH∗H^\{\*\}boundary \(H∗∈\[6,14\]H^\{\*\}\\in\[6,14\]under the NBA parameters\)\.

## 1Introduction

Many sequential decision problems accumulate state over long horizons in a way that couples locally attractive actions to globally adverse outcomes\. A clinician dosing a chronic medication, an engineer scheduling heavy\-duty use of a wearing component, and a worker allocating physical effort across a decades\-long career all face the same structural challenge: the action that maximises immediate reward increments a cumulative state whose eventual level terminates the episode\. Whether a policy\-gradient agent\(Schulmanet al\.,[2017](https://arxiv.org/html/2605.26657#bib.bib1)\)can learn the prudent policy in such problems is a question about credit assignment over horizons that exceed any rollout\-based learner’s experience\. Recent long\-horizon credit\-assignment work\(Arjona\-Medinaet al\.,[2019](https://arxiv.org/html/2605.26657#bib.bib16); Harutyunyanet al\.,[2019](https://arxiv.org/html/2605.26657#bib.bib17)\)has identified the structural difficulty but not the specific failure mode that greedy credit assignment induces when the damage signal is implicit in the terminal condition\.

#### The diagnostic gap\.

A policy with low return on a long\-horizon cumulative\-damage problem may be failing for either of two unrelated reasons\. A policy that terminates early through the implicit terminal constraint has low return because it never reaches late\-horizon states\. A policy that reaches the terminal horizon but commits to locally greedy actions has low return because it fails to preserve the cumulative state\. Each failure mode calls for a different intervention: horizon access addresses the former, while credit assignment targets the latter\. A return\-based evaluation, however, conflates both onto a single scalar, forcing a choice between diagnosing*why*the policy exits early and diagnosing*how suboptimal*it is given that the episode completes, even when both failures are present simultaneously and require separate remedies\.

#### This paper\.

We evaluate policies on two independent axes\.*Completion*is whether the policy reaches the terminal horizon\.*Optimality*is whether the policy matches the fixed\-share dynamic\-programming reference\(Puterman,[1994](https://arxiv.org/html/2605.26657#bib.bib2)\)given completion\. When the constraint is*implicit*in the terminal condition, no per\-step cost is available, unlike in standard CMDPs\(Altman,[1999](https://arxiv.org/html/2605.26657#bib.bib5); Achiamet al\.,[2017](https://arxiv.org/html/2605.26657#bib.bib6); Chowet al\.,[2018](https://arxiv.org/html/2605.26657#bib.bib7); García and Fernández,[2015](https://arxiv.org/html/2605.26657#bib.bib9); Rayet al\.,[2019](https://arxiv.org/html/2605.26657#bib.bib8)\), and the standard decomposition does not apply\. A principled evaluation requires making both failure modes separately visible\. For the optimality axis we provide an analytical account: in Appendix[B](https://arxiv.org/html/2605.26657#A2)we derive, on a minimal cumulative\-damage MDP, a sufficient condition under which the*expected*first policy\-gradient update at the damage origin points toward the greedy action \(Proposition[1](https://arxiv.org/html/2605.26657#Thmproposition1)\)\. The proposition governs a binary\-action skeleton\. Its connection to the multi\-activity testbeds is structural analogy and motivating context, not formal derivation\. We evaluate in two environments \(a 49\-step bricklayer career and a 20\-season NBA power\-forward career\) sharing the abstract structure of §[3](https://arxiv.org/html/2605.26657#S3)but differing in activity set, horizon, load model, and calibration data\. In both, PPO on the real environment fails completion\. A Dyna\(Sutton,[1990](https://arxiv.org/html/2605.26657#bib.bib33)\)variant with horizon access and restricted action space achieves completion but fails optimality; backward induction achieves both\.

#### Contributions\.

- •A formalisation of the horizon\-mismatched cumulative\-damage problem class \(§[3](https://arxiv.org/html/2605.26657#S3)\), covering latent damage state, delayed proxy signal, dominant\-activity structure, and implicit role\-viability constraint, together with the completion\-vs\-optimality decomposition \(§[3\.2](https://arxiv.org/html/2605.26657#S3.SS2)\)\.
- •An empirical three\-way comparison \(PPO\-real, fixed\-share Dyna, fixed\-share dynamic programming\) that separately quantifies completion and optimality gaps in two environments \(Δ​Mfinal=0\.271\\Delta M\_\{\\text\{final\}\}=0\.271bricklayer,0\.1500\.150NBA; 95% bootstrap CI\[0\.148,0\.151\]\[0\.148,0\.151\]\)\.
- •A causal analysis showing that, under PPO with a linear soft penalty, horizon access alone is harmful \(mean exit age24\.724\.7vs27\.827\.8,p=0\.028p=0\.028\) and action\-space restriction is the intervention that unlocks completion \(§[5](https://arxiv.org/html/2605.26657#S5)\)\.
- •A basin\-of\-attraction analysis identifying first\-phase greedy commitment as the optimality\-failure mechanism, with basin entry within the first 1% of training regardless of horizon length \(§[6](https://arxiv.org/html/2605.26657#S6)\)\.

## 2Related Work

#### Constrained MDPs and safe RL\.

The constrained\-MDP formulation\(Altman,[1999](https://arxiv.org/html/2605.26657#bib.bib5)\)and its variants\(Achiamet al\.,[2017](https://arxiv.org/html/2605.26657#bib.bib6); Chowet al\.,[2018](https://arxiv.org/html/2605.26657#bib.bib7)\)treat the constraint as an explicit, per\-step observed quantity\. Safe\-exploration methods\(García and Fernández,[2015](https://arxiv.org/html/2605.26657#bib.bib9); Rayet al\.,[2019](https://arxiv.org/html/2605.26657#bib.bib8)\)treat safety violations as continuous costs throughout the rollout\. Our setting differs structurally: the terminal condition functions as the constraint, discovered from rollouts rather than given as a running cost, so standard CMDP methods cannot diagnose the completion failure we identify\.

#### Policy\-gradient convergence\.

A recent theoretical line\(Meiet al\.,[2020](https://arxiv.org/html/2605.26657#bib.bib12); Agarwalet al\.,[2021](https://arxiv.org/html/2605.26657#bib.bib13); Bhandari and Russo,[2024](https://arxiv.org/html/2605.26657#bib.bib14)\)characterises non\-convex basins in policy space\. Our step\-0 basin entry instantiates this regime at the damage\-state origin, where immediate reward dominates the Lipschitz tail of the continuation value\. Proposition[1](https://arxiv.org/html/2605.26657#Thmproposition1)is derived from first principles on a minimal binary\-action MDP\. Its connection to the general convergence literature is structural analogy, not formal derivation\. This literature speaks to our optimality axis but is silent on completion, typically assuming fixed horizons\.

#### Long\-horizon credit assignment, reward shaping, and offline methods\.

Methods that rebind credit to earlier actions\(Harutyunyanet al\.,[2019](https://arxiv.org/html/2605.26657#bib.bib17); Arjona\-Medinaet al\.,[2019](https://arxiv.org/html/2605.26657#bib.bib16)\)target a mechanism that can bind either axis\. Our decomposition isolates*which*axis is the binding constraint\. Potential\-based reward shaping\(Nget al\.,[1999](https://arxiv.org/html/2605.26657#bib.bib4)\)can alter the gradient landscape near initialisation\. Return\-conditioned methods such as Decision Transformer\(Chenet al\.,[2021](https://arxiv.org/html/2605.26657#bib.bib18)\)might select trajectories avoiding early termination but require demonstrations and inherit distribution\-shift problems in the sparse late\-horizon regime\. Neither class is evaluated here\.

#### Afferent signal design\.

The proxy signalStS\_\{t\}used in the testbeds is a fixed sensitisation function calibrated to epidemiological data\. An alternative not pursued here is to learn or evolve the afferent signal end\-to\-end rather than fixing it\. The proxy irrelevance finding \(§[5](https://arxiv.org/html/2605.26657#S5)\) is a diagnostic of what fixed afferents cannot provide, and whether adaptive afferent architectures close this gap without action\-space restriction is an open question\.

## 3Problem Class, Decomposition, and Algorithmic Analysis

### 3\.1Problem class

We study finite\-horizon MDPs with cumulative damage, specified by\(𝒮,𝒜,T,r,H,ℰ\)\(\\mathcal\{S\},\\mathcal\{A\},T,r,H,\\mathcal\{E\}\), where𝒮\\mathcal\{S\}contains a*cumulative damage variable*Dt∈\[0,1\]D\_\{t\}\\in\[0,1\]\(Dt\+1=Dt\+f​\(st,at\)D\_\{t\+1\}=D\_\{t\}\+f\(s\_\{t\},a\_\{t\}\),f≥0f\\geq 0\) and a*secondary preservation variable*Mt∈\[0,1\]M\_\{t\}\\in\[0,1\],M0=1M\_\{0\}=1; reward is locally maximised by actions withf​\(st,at\)\>0f\(s\_\{t\},a\_\{t\}\)\>0; andℰ\\mathcal\{E\}is a set of terminal conditions that exit the episode when cumulative damage crosses a state\-dependent threshold or when role\-viability is violated\. The key distinction from standard CMDPs\(Guin and Bhatnagar,[2022](https://arxiv.org/html/2605.26657#bib.bib10); Wachi and others,[2024](https://arxiv.org/html/2605.26657#bib.bib11)\)is that the constraint inℰ\\mathcal\{E\}is*never observed as a running cost*: the learner discovers the terminal boundary only by reaching it\. We call such a problem*horizon\-mismatched*when the greedy policy’s effective episode length is strictly less thanHH\.

###### Assumption 1\(Cumulative\-damage structure\)\.

A cumulative\-damage MDPℳ\\mathcal\{M\}satisfies: \(i\)Monotone damage:f​\(s,a\)≥0f\(s,a\)\\geq 0for all\(s,a\)\(s,a\), soDtD\_\{t\}is non\-decreasing along any trajectory\. \(ii\)Greedy\-damage alignment: the reward\-maximising action satisfiesf​\(s,arg⁡maxa⁡r​\(s,a\)\)\>0f\(s,\\arg\\max\_\{a\}r\(s,a\)\)\>0\. \(iii\)Implicit terminal boundary:ℰ\\mathcal\{E\}is non\-empty and is*avoided*with positive probability only by policies under which the greedy action is not always selected\.

Beyond Assumption[1](https://arxiv.org/html/2605.26657#Thmassumption1), the instances we study share five structural features\. The first two concern what the agent observes and the structure of its action space: \(a\)latent damage state:DtD\_\{t\}and the secondary preservation variableMtM\_\{t\}are unobserved; the agent receives only a proxy signalSt=g​\(Dt\)S\_\{t\}=g\(D\_\{t\}\)that is uninformative at the low\-damage levels the greedy policy visits, and therefore no observation\-based warning reaches the agent before the implicit boundary\. We use a fixed sensitisation function calibrated to epidemiological data as a diagnostic baseline\. \(b\)dominant\-activity structure: one activity is simultaneously the unique reward maximiser and the unique damage maximiser, withk−1k\-1lower\-reward, lower\-damage alternatives creating a substitution structure whose unconstrained optimum conflicts with the role\-viability constraint \(c\)\. The remaining three govern how the environment enforces and penalises damage: \(c\)role\-viability constraint: the episode terminates if the trailing\-window dominant\-activity share falls belowα\\alpha, operating in the share dimension of the action space; \(d\)capacity\-damage feedback: per\-step reward is suppressed byh​\(Dt,St\)h\(D\_\{t\},S\_\{t\}\), silent belowDclinD\_\{\\text\{clin\}\}and progressive above it; and \(e\)self\-amplifying secondary channel:MtM\_\{t\}degrades at an accelerating rate belowMamp=0\.6M\_\{\\text\{amp\}\}=0\.6, making early maximum\-effort trajectories irreversibly worse than proactive low\-effort ones\. Without \(e\) the optimality gap vanishes: DP and the greedy policy produce similarMfinalM\_\{\\text\{final\}\}at matched career lengths; both testbeds useMamp=0\.6M\_\{\\text\{amp\}\}=0\.6\.

### 3\.2The completion\-vs\-optimality decomposition

###### Definition 1\(Completion\)\.

A policyπ\\piachieves*completion*if the probability of reaching any element ofℰ\\mathcal\{E\}before stepHHis zero: the expected episode length equalsHH\.

###### Definition 2\(Optimality\)\.

Given completion, a policyπ\\piachieves*optimality*ifVπ​\(s0\)=Vπ∗​\(s0\)V^\{\\pi\}\(s\_\{0\}\)=V^\{\\pi^\{\\ast\}\}\(s\_\{0\}\), whereπ∗\\pi^\{\\ast\}is the backward\-induction reference optimised over the same action subspace asπ\\pi\. Optimality is undefined for policies that fail completion\.

These definitions induce a three\-outcome ordinal scale \(Table[1](https://arxiv.org/html/2605.26657#S3.T1)\)\. The*optimality gap*isΔ​Mfinal=MfinalDP−Mfinalπ\\Delta M\_\{\\text\{final\}\}=M^\{\\text\{DP\}\}\_\{\\text\{final\}\}\-M^\{\\pi\}\_\{\\text\{final\}\}\. Definition[2](https://arxiv.org/html/2605.26657#Thmdefinition2)is stated in terms ofVπV^\{\\pi\}\.Δ​Mfinal\\Delta M\_\{\\text\{final\}\}serves as a domain\-specific surrogate for the value gap, justified by feature \(e\): under the self\-amplifying secondary channel and the fixed\-share constraint,MfinalM\_\{\\text\{final\}\}is a monotone function ofVπV^\{\\pi\}given completion, soVπ<Vπ∗⇔Mfinalπ<Mfinalπ∗V^\{\\pi\}<V^\{\\pi^\{\*\}\}\\Leftrightarrow M^\{\\pi\}\_\{\\text\{final\}\}<M^\{\\pi^\{\*\}\}\_\{\\text\{final\}\}\. Accordingly,Δ​Mfinal=0\\Delta M\_\{\\text\{final\}\}=0corresponds to Cell C andΔ​Mfinal\>0\\Delta M\_\{\\text\{final\}\}\>0to Cell B\.

Table 1:Three\-outcome scale induced by Definitions[1](https://arxiv.org/html/2605.26657#Thmdefinition1)and[2](https://arxiv.org/html/2605.26657#Thmdefinition2)\. The scale is strictly ordered Cell A≺\\precCell B≺\\precCell C\. Optimality is only meaningful once completion is achieved; Cell A vs\. Cell B/C comparisons onMfinalM\_\{\\text\{final\}\}confound truncation with suboptimality and are not made in this paper\.\(A\) PPO\-real\(B\) Fixed\-share Dyna\(C\) DP\-optimalEnvironmentlatentDt,MtD\_\{t\},M\_\{t\}EnvironmentlatentDt,MtD\_\{t\},M\_\{t\}Environmentfull state\(D,M,t\)\(D,M,t\)ObservationproxySt=g​\(Dt\)S\_\{t\}\{=\}g\(D\_\{t\}\)ObservationproxyStS\_\{t\}Direct access\(D,M,t\)\(D,M,t\)on gridShare headlearned via PGShare headfixed\_share: DP\-clampedShare allocationbackward inductionEffort headlearned via PGEffort headlearned via PGEffort policybackward inductionRole\-viability gateactive: terminates earlyRole\-viability gateno\_exit: bypassedBackward passfull horizonHHCell Afails completion \(27\.827\.8yr\)Cell Bcompletion;Δ​Mfinal=0\.271\\Delta M\_\{\\text\{final\}\}\{=\}0\.271Cell Ccompletion\+\+optimalityTrainable \(PG\)Frozen / clampedEnvironment / fixedOutcome

Figure 1:Three\-way method\-comparison architecture\. Columns share the pipeline state→\\toobservation→\\toshare/effort heads→\\torole\-viability gate→\\tooutcome and differ only in which components are trainable \(orange\) vs\. clamped \(cyan\)\.fixed\_sharereplaces the trainable share head;no\_exitbypasses the role\-viability gate\.
### 3\.3Mechanism analysis: backward induction vs\. policy gradient

#### Backward induction achieves both axes\.

DP evaluates every\(D,M,t\)\(D,M,t\)grid point backward fromHH, and thereforeℰ\\mathcal\{E\}is avoided*a priori*and the capacity\-damage feedback \(d\) is integrated over the full remaining horizon\. At\(D0=0,M0=1,t=0\)\(D\_\{0\}\\\!=\\\!0,M\_\{0\}\\\!=\\\!1,t\\\!=\\\!0\), sustained low effort is identified as the unique trajectory keepingDDbelowDclinD\_\{\\text\{clin\}\}\. Latent state \(a\) is moot under direct grid access \(Figure[1](https://arxiv.org/html/2605.26657#S3.F1), column C\)\.

#### Policy gradient fails completion\.

Under Assumption[1](https://arxiv.org/html/2605.26657#Thmassumption1)\(ii\)–\(iii\), the greedy policy reachesℰ\\mathcal\{E\}beforeHH\. When horizon access is granted viano\_exit, the linear soft penalty’s per\-step cost outweighs the dominant activity’s reward advantage, and therefore the share head’s unconstrained optimum places mass at zero on the dominant activity, violating the constraint the hard exit would have enforced\. Completion failure arises from two interacting causes: the implicit terminal boundary and the uninformative afferent in the greedy policy’s operating damage range, addressable via action\-space restriction \(fixed\_share, studied here\) or by improving the afferent’s informativeness in the policy’s operating damage range\. Clamping shares to a constraint\-satisfying allocation structurally prevents role\-viability violations regardless of whether horizon access is granted\.

#### Policy gradient fails optimality\.

Even with completion achieved, the dominant\-activity structure \(b\) creates a first\-phase commitment atD=0D=0: the reward gapeHβ−eLβe\_\{H\}^\{\\beta\}\-e\_\{L\}^\{\\beta\}is large while damage cost is zero, and therefore the first policy\-gradient update \(in expectation\) commits to maximum effort on the dominant activity\. Proposition[1](https://arxiv.org/html/2605.26657#Thmproposition1)\(Appendix[B](https://arxiv.org/html/2605.26657#A2)\) formalises this as a sufficient condition on the*expected*gradient direction in a binary\-action minimal MDP\. The self\-amplifying channel \(e\) then ensures that onceMMfalls belowMampM\_\{\\text\{amp\}\}, the degradation cascade is irreversible\.

### 3\.4Testable predictions

The analysis of §[3\.3](https://arxiv.org/html/2605.26657#S3.SS3)yields four testable predictions about any cumulative\-damage MDP satisfying Assumption[1](https://arxiv.org/html/2605.26657#Thmassumption1)with features \(a\)–\(e\), under PPO with a linear soft penalty for the completion\-axis predictions \(P1–P2\)\. The proxy signal \(a\) contributes nothing detectable to completion because its informative range \(D\>DclinD\>D\_\{\\text\{clin\}\}\) is disjoint from the greedy policy’s operating damage range \(confirmed empirically in §[5](https://arxiv.org/html/2605.26657#S5)\)\. Backward induction achieves both axes by construction and occupies Cell C; it serves as the upper\-bound reference throughout\.

1. P1PG\-real fails completion\.Policy gradient trained on real rollouts withℰ\\mathcal\{E\}enforced exits early, occupying Cell A\.
2. P2Horizon access alone is harmful\(under PPO with linear soft penalty\): granting horizon access without action\-space restriction shortens careers relative to PG\-real, because the penalty’s equilibrium under \(c\) drives the dominant\-activity share belowα\\alpha\.
3. P3Reactive\-decline attractor\.Under completion and given the self\-amplifying structure \(e\), the dominant attractor is maximum effort att=0t=0, monotone decline,MMirreversibly belowMampM\_\{\\text\{amp\}\}, and positiveΔ​Mfinal\\Delta M\_\{\\text\{final\}\}\.
4. P4Horizon invariance\.The reactive attractor’s dominance rate andΔ​Mfinal\\Delta M\_\{\\text\{final\}\}are invariant across training horizonsH≤H∗H\\leq H^\{\*\}, because basin entry occurs at step 0\.

## 4Testbeds and Headline Results

We test P1–P4 in two cumulative\-damage environments\. Both satisfy Assumption[1](https://arxiv.org/html/2605.26657#Thmassumption1)and features \(a\)–\(e\) but differ in activity set, horizon, load model, biological calibration data, and role\-viability parameters\. The two environments share the same simulation engine, the HMS regulation pathway, and the Baratz amplification function\. The replication therefore tests whether the findings generalise across parameter regimes of the same problem class, not across independently developed simulators, a limitation we return to in §[8](https://arxiv.org/html/2605.26657#S8)\.

Thebricklayertestbed models a 49\-year career overk=7k=7construction activities, calibrated to a bio\-physical model of the human knee\(Coggonet al\.,[2000](https://arxiv.org/html/2605.26657#bib.bib19); Jensen,[2008](https://arxiv.org/html/2605.26657#bib.bib20); Rytteret al\.,[2009](https://arxiv.org/html/2605.26657#bib.bib21); Baratzet al\.,[1986](https://arxiv.org/html/2605.26657#bib.bib22); Arendt\-Nielsenet al\.,[2010](https://arxiv.org/html/2605.26657#bib.bib23)\)\(Appendix[C](https://arxiv.org/html/2605.26657#A3)\)\. TheNBA power\-forwardtestbed models a 20\-season career over 6 activities, calibrated to the sports\-medicine literature\(Drakoset al\.,[2010](https://arxiv.org/html/2605.26657#bib.bib28); Wigginset al\.,[2016](https://arxiv.org/html/2605.26657#bib.bib29)\)\(Appendix[G](https://arxiv.org/html/2605.26657#A7)\)\.

Figure[2](https://arxiv.org/html/2605.26657#S4.F2)and Table[2](https://arxiv.org/html/2605.26657#S4.T2)summarise the headline three\-way contrast\. The DP reference is computed in under two hours on a single CPU \(Appendix[D](https://arxiv.org/html/2605.26657#A4)\), and the gap cannot therefore be attributed to the cost of computing the DP reference\.

![Refer to caption](https://arxiv.org/html/2605.26657v1/figures/fig1_three_way_effort.png)Figure 2:Three\-way dominant\-activity effort \(bricklayer,H=49H=49\)\. DP\-optimal \(green, Cell C,Mfinal=0\.79M\_\{\\text\{final\}\}\{=\}0\.79\); dominant\-mode Dyna \(purple, Cell B, 8/10 seeds,0\.520\.52\); PPO\-real \(red, Cell A, exit≈27\.8\\approx 27\.8yr\)\. Headline gapΔ​Mfinal=0\.271\\Delta M\_\{\\text\{final\}\}=0\.271\.Table 2:Bricklayer instantiation of Table[1](https://arxiv.org/html/2605.26657#S3.T1)\. Cell AMfinalM\_\{\\text\{final\}\}omitted \(truncation confounds with suboptimality\)\.†\\daggercareer ends at age 65 \(H=49H=49steps\)\.
## 5Completion: Causal Analysis

#### PPO\-real fails completion \(P1\)\.

PPO withγ=1\\gamma=1\(undiscounted episodic return; the rbar exponential moving\-average baseline ofMahadevan[1996](https://arxiv.org/html/2605.26657#bib.bib3)is used for variance reduction\) terminates at a mean age of27\.8±4\.027\.8\\pm 4\.0years, with 100% of 2,000 eval episodes ending in role\-violation \(20 seeds, 10 curriculum\-on and 10 curriculum\-off\)\. The role\-viability rule fires when

1W​∑τ=t−W\+1tsdom\(τ\)<α,\\frac\{1\}\{W\}\\sum\_\{\\tau=t\-W\+1\}^\{t\}s^\{\(\\tau\)\}\_\{\\text\{dom\}\}<\\alpha,withW=5W=5years andα=0\.15\\alpha=0\.15in the bricklayer\. The agent observes only the proxy signalStS\_\{t\}\. The trailing share historysdom\(τ\),τ∈\[t−W\+1,t−1\]s^\{\(\\tau\)\}\_\{\\text\{dom\}\},\\,\\tau\\in\[t\{\-\}W\{\+\}1,t\{\-\}1\]that determines the role\-viability boundary is not provided in the state observation, making the constraint boundary non\-Markovian from the agent’s perspective\. This partial\-observability challenge is a contributing factor to PPO\-real’s failure, separate from and additional to the long\-horizon credit\-assignment difficulty\. Each is acknowledged in the Limitations of §[8](https://arxiv.org/html/2605.26657#S8)\. Including the proxy signalSSin the state yieldsΔ=−0\.65\\Delta=\-0\.65yr relative to zeroing it \(20 seeds per condition\), within seed variance and in the wrong direction:SSbecomes informative only atD\>0\.3D\>0\.3, whereas PPO\-real exits atDfinal=0\.16D\_\{\\text\{final\}\}=0\.16\. The proxy is not uninformative in principle\. It is informative about states the learner never reaches, a diagnostic of the fixed afferent’s design, which covers the clinical damage range rather than the early\-career range where the greedy policy operates\.

#### Horizon access alone is harmful \(P2\)\.

Across 20 seeds of the best unrestricted Dyna configuration, trained with rollouts extended past role\-violation \(no\_exitflag\) and a linear soft role penalty with weightw=1\.0w=1\.0\(applied as−w⋅𝟏​\[sdom,t<α\]\-w\\cdot\\mathbf\{1\}\[s\_\{\\text\{dom\},t\}<\\alpha\]per step\), the mean exit age under real evaluation is24\.724\.7years \(95% CI\[23\.0,26\.8\]\[23\.0,26\.8\]\),*earlier*than PPO\-real’s27\.827\.8\(Welch’st​\(37\.8\)=−2\.29t\(37\.8\)=\-2\.29,p=0\.028p=0\.028; Mann\-Whitneyp=0\.009p=0\.009\)\. The exit\-age distribution is bimodal: 10/20 seeds exit at age 21 via role\-violation \(the earliest age the trailing\-window check can fire\); the remaining 10 seeds survive to ages 26–38\. The linear soft penalty does not induce an interior basin around any positive dominant\-activity allocation, and therefore the share head’s unconstrained optimum places mass at zero once curriculum scaffolding lifts\. Whether convex or entropy\-regularised alternatives escape this pattern is open \(§[8](https://arxiv.org/html/2605.26657#S8)\)\.

#### Action\-space restriction unlocks completion\.

Restricting the policy to the DP\-optimal share allocation \(fixed\_share, learning only effort\) achieves completion in all cases: all 10 seeds reach the terminus \(65\.0±0\.065\.0\\pm 0\.0yr, zero role\-violations over 1,000 eval episodes\) acrossH∈\{13,20,30,49\}H\\in\\\{13,20,30,49\\\}\(Appendix[A](https://arxiv.org/html/2605.26657#A1)\)\. Completion is a function of action\-space structure, not horizon length\.

#### Fixed\-share PPO\-real ablation\.

To isolate whether action\-space restriction alone \(withoutno\_exit\) suffices, we runfixed\_sharePPO\-real with the role\-viability gate active during training \(10 seeds\)\. All 10 seeds achieve 100% completion \(exit age65\.0±0\.065\.0\\pm 0\.0\)\. Completion is therefore attributable to the structural share constraint, which keepssdom≥αs\_\{\\text\{dom\}\}\\geq\\alphaby construction\.no\_exitis not independently necessary for completion but is required for the Dyna agent to accumulate long\-horizon experience\.

A\(W,α\)\(W,\\alpha\)sensitivity sweep \(90 runs, Appendix[F](https://arxiv.org/html/2605.26657#A6)\) confirms threshold robustness\.

## 6Optimality: Basin Analysis

### 6\.1Reactive decline is the dominant attractor \(P3\)

Across 10 seeds of the fixed\-share Dyna sweep, 8 converge to a nearly identical effort profile: maximum effort att=0t=0, monotone decline to0\.220\.22by age 60 \(Figure[3](https://arxiv.org/html/2605.26657#S6.F3)b\)\. Seed\-to\-seed variance in dominant\-modeMfinalM\_\{\\text\{final\}\}is0\.0010\.001, three orders of magnitude tighter than the gap to DP\-optimal, confirming this is a distinct fixed point, not optimisation noise\.

The mechanism is feature \(e\): early maximum effort drivesMMbelowMamp=0\.6M\_\{\\text\{amp\}\}=0\.6irreversibly, triggering a degradation cascade the DP avoids by sustaining low effort fromt=0t=0\. Two seeds escape the reactive basin: seed 4 via an inverted\-U effort profile \(Mfinal=0\.64M\_\{\\text\{final\}\}=0\.64\) and seed 7 via a mid\-training reward collapse to a DP\-like low\-effort mode \(Mfinal=0\.825M\_\{\\text\{final\}\}=0\.825\), demonstrating that the DP\-competitive region is stochastically reachable, though not via a reliable convergence path\. The basin\-dominance claim is accordingly about 8 of 10 seeds \(95% Clopper–Pearson CI\[0\.44,0\.97\]\[0\.44,0\.97\]\), not 10 of 10\.

### 6\.2First\-phase basin entry as the mechanism \(P4\)

Three pieces of evidence support first\-phase basin entry:

1. 1\.All 8 dominant\-cluster seeds reach maximum effort at the initial state within the first 10,000 training steps\(≈\\approx1% of training\), read from periodic training logs\. The reactive trajectory is established before any long\-horizon signal has accumulated\.
2. 2\.The two escaping seeds did not commit at step 0\.Seed 4 has initial\-state effort0\.1830\.183and seed 7 has0\.3460\.346; both escaped because their stochastic first gradient update did not push toward the maximum\. Proposition[1](https://arxiv.org/html/2605.26657#Thmproposition1)governs the*expected*gradient direction\. Individual sample gradients can differ in sign, which is consistent with these two escaping seeds\.
3. 3\.The basin structure is invariant to training horizonacrossH∈\{13,20,30,49\}H\\in\\\{13,20,30,49\\\}: dominant\-modeMfinalM\_\{\\text\{final\}\}is constant at0\.523±0\.0010\.523\\pm 0\.001regardless of horizon \(Appendix[A](https://arxiv.org/html/2605.26657#A1), Figure[4](https://arxiv.org/html/2605.26657#A1.F4)\)\. A basin entered by aggregating long\-horizon signals would shift withHH\. This one does not\.

Proposition[1](https://arxiv.org/html/2605.26657#Thmproposition1)\(Appendix[B](https://arxiv.org/html/2605.26657#A2)\) formalises the*direction of the expected gradient*in a binary\-action minimal MDP\. The empirical evidence above establishes saturation within the first 1% of training for all 8 reactive\-cluster seeds\.

### 6\.3Initialization intervention: a falsifiability test

The first\-phase account predicts that biasing the effort head’s initial output toward the DP\-optimal value weakens the basin’s dominance\. Setting the effort head’s final\-layer bias to−1\.386\-1\.386\(σ​\(−1\.386\)≈0\.20\\sigma\(\-1\.386\)\\approx 0\.20, matching DP\-optimal sustained effort\), with all other settings identical to the fixed\-share sweep, produces4/10 seeds escaping\(vs\. 2/10 baseline\) and raises meanMfinalM\_\{\\text\{final\}\}from0\.5640\.564to0\.6090\.609\.

Pre\-registered binding thresholds \(prompts/e1\_prereg\.md, frozen 2026\-04\-20\):≥6\\geq 6confirms,44–55is*nuanced*,≤3\\leq 3falsifies\. The result falls in the nuanced band\. Six seeds revert to the reactive basin within the first10410^\{4\}gradient updates\. The four escapers split into three attractor types: seeds 4 and 7 reach DP\-competitiveMfinal∈\[0\.79,0\.85\]M\_\{\\text\{final\}\}\{\\in\}\[0\.79,0\.85\]; seed 9 converges to near\-zero late\-career effort \(et\>40≈0e\_\{t\>40\}\\approx 0\); seed 0 oscillates\. Per\-seed results are in Appendix[D](https://arxiv.org/html/2605.26657#A4), Table[5](https://arxiv.org/html/2605.26657#A4.T5)\)\. Seeds 4 and 7 represent qualitatively distinct effort profiles: seed 4 converges to a gradually increasing schedule near the DP\-optimal level, while seed 7 settles below the DP grid minimum \(e<0\.20e<0\.20\) and achievesMfinal=0\.846M\_\{\\text\{final\}\}=0\.846, exceeding the DP reference\. Both escape the reactive basin not by resolving the long\-horizon credit\-assignment problem but by avoiding step\-0 commitment, a pattern consistent with the occupational health finding that conservative early\-career effort profiles produce better long\-run outcomes than maximum early effort\(McLellanet al\.,[2022](https://arxiv.org/html/2605.26657#bib.bib31)\)\. The attraction is strong but not absolute: closing the gap requires weakening the basin’s attraction, not correcting the initialisation\.

![Refer to caption](https://arxiv.org/html/2605.26657v1/figures/fig5_basin_of_attraction.png)Figure 3:Basin\-of\-attraction structure \(10 seeds, bricklayer\)\. \(a\) 8 seeds converge to the reactive\-decline attractor \(∼\\sim100k steps\); seed 4 reaches a stable alternative; seed 7 collapses mid\-training and partially recovers, demonstrating that the DP\-competitive region is stochastically reachable\. \(b\) Final effort profiles; all 8 reactive seeds output maximum effort after only 10k steps\.

## 7Replication: NBA Career Environment

To distinguish the bricklayer findings from parameter\-specific artefacts, we replicate the four\-condition experiment in the NBA power\-forward testbed \(§[4](https://arxiv.org/html/2605.26657#S4)\)\.

#### Completion axis \(P1–P2\) replicates\.

PPO\-real fails across all 20 seeds \(exit22\.6±1\.822\.6\\pm 1\.8, role\-violation100%100\\%\)\. Unrestricted Dyna across 60 seeds \(w∈\{0\.5,1\.0,2\.0\}w\\in\\\{0\.5,1\.0,2\.0\\\}\) exits at21\.221\.2yr,1\.41\.4yr*earlier*than PPO\-real \(p=0\.003p=0\.003Welch’stt;p=0\.0003p=0\.0003Mann\-Whitney\)\. The same direction of harm confirms the mechanism is not domain\-specific\. Fixed\-share Dyna achieves100%100\\%completion \(10/10 reach age 38\)\.

#### Optimality axis \(P3–P4\) replicates with compression\.

Δ​Mfinal=0\.150\\Delta M\_\{\\text\{final\}\}=0\.150\(95% CI\[0\.148,0\.151\]\[0\.148,0\.151\]\), above the pre\-registered0\.100\.10threshold\. The NBA testbed falls outside the analytical guarantee \(H∗∈\[6,14\]H^\{\*\}\\in\[6,14\], Appendix[B](https://arxiv.org/html/2605.26657#A2)\), providing a direct test of whether the commitment condition is sufficient but not necessary: the reactive basin is the plurality attractor \(4/104/10\) rather than the bricklayer’s majority \(8/108/10\), consistent with theH∗H^\{\*\}compression prediction\. Three seeds settle in a mid\-cluster \(Mfinal=0\.71M\_\{\\text\{final\}\}\{=\}0\.71\) and three converge near DP \(0\.850\.85\)\. Fisher’s exactp=0\.17p=0\.17atn=10n=10is non\-significant per\-environment, but the decomposition generalises across domains\.

#### Horizon invariance \(P4\) replicates at 3 of 4 horizons\.

Dominant\-modeMfinalM\_\{\\text\{final\}\}is0\.614,0\.614,0\.648,0\.6150\.614,0\.614,0\.648,0\.615forH=7,10,15,20H=7,10,15,20\. Three horizons agree to within0\.0010\.001; theH=15H=15outlier is consistent withH∗H^\{\*\}in\[6,14\]\[6,14\]\(Appendix[B](https://arxiv.org/html/2605.26657#A2)\):H=15H=15lies outside the analytical guarantee under both damage channels\. The basin\-dominance rate decreases asHHexceedsH∗H^\{\*\}\. The step\-0 mechanism remains universal\.

Table 3:Bricklayer \(H=49H=49\) vs\. NBA \(H=20H=20\) replication summary\.

## 8Discussion

#### The decomposition as diagnostic\.

The decomposition separates two failure modes that return\-based evaluation conflates, and shows that interventions targeting one axis can worsen the other: a paper reporting only exit age would claim fixed\-share Dyna is solved, while one reporting onlyMfinalM\_\{\\text\{final\}\}would be misled by PPO\-real’s artificially high \(early\-exit\) score\. The decomposition also maps existing interventions to the axis each addresses\. Action\-space restriction \(fixed\_share\) targets the completion axis by enforcing a constraint\-satisfying share allocation\. Improving the afferent’s informativeness in the policy’s operating damage range targets the completion axis from the information side\. The optimality gap \(Δ​Mfinal=0\.271\\Delta M\_\{\\text\{final\}\}=0\.271\) remains open to both interventions and requires a separate credit\-assignment solution\. The reactive\-decline policy is qualitatively consistent with occupational epidemiology\(Palmer,[2012](https://arxiv.org/html/2605.26657#bib.bib25); Coggonet al\.,[2000](https://arxiv.org/html/2605.26657#bib.bib19); Jensen,[2008](https://arxiv.org/html/2605.26657#bib.bib20); Rytteret al\.,[2009](https://arxiv.org/html/2605.26657#bib.bib21)\): workers maintain high effort until tissue damage forces reactive regulation\(Woolf and Salter,[2000](https://arxiv.org/html/2605.26657#bib.bib24); Arendt\-Nielsenet al\.,[2010](https://arxiv.org/html/2605.26657#bib.bib23)\), though real workers face financial and social constraints we do not model\. We conjecture the failure modes generalise to other cumulative\-damage domains satisfying Assumption[1](https://arxiv.org/html/2605.26657#Thmassumption1)with features \(a\)–\(e\)\.

#### Connection to early sport specialisation\.

The reactive\-decline attractor has a direct empirical counterpart in the early sport specialisation literature\(DiFioriet al\.,[2014](https://arxiv.org/html/2605.26657#bib.bib30); McLellanet al\.,[2022](https://arxiv.org/html/2605.26657#bib.bib31); Jildeh,[2024](https://arxiv.org/html/2605.26657#bib.bib32)\): maximum dominant\-activity effort from career outset, accelerating secondary degradation once a tissue threshold is crossed, and premature exit\. The structural parallels are precise: latent damage accumulation before pain signals become informative \(feature \(a\)\), self\-amplifying degradation belowMampM\_\{\\text\{amp\}\}\(feature \(e\)\), and implicit role\-viability violation as the terminal condition\. Empirically, NBA players who specialised in a single sport during adolescence play in fewer career games and suffer more serious injuries than multi\-sport athletes\(McLellanet al\.,[2022](https://arxiv.org/html/2605.26657#bib.bib31)\), consistent with the model’s prediction that distributing effort across activities preservesMfinalM\_\{\\text\{final\}\}\. The step\-0 commitment mechanism provides a formal account of why agents in this class converge to this pattern even when proactive load management would be optimal\.

#### Limitations\.

Same engine: replication tests parameter\-regime generality within one codebase, not across independently developed simulators\.Proposition–testbed gap: Proposition[1](https://arxiv.org/html/2605.26657#Thmproposition1)covers a binary\-action minimal MDP; it motivates but does not formally derive the empirical findings in the multi\-activity continuous\-effort testbeds\.PPO and penalty specificity: whether convex penalties or maximum\-entropy methods \(SAC,Haarnojaet al\.,[2018](https://arxiv.org/html/2605.26657#bib.bib15)\) escape harmful horizon access is open\.Ground\-truth model: Dyna uses the exact simulator transition; practical model\-based RL incurs additional misspecification error\.Partial observability: the role\-viability boundary depends on the trailingWW\-step share history, which is absent from the state observation, making the constraint non\-Markovian from the agent’s perspective\.Shared trunk: the share and effort heads share a common network trunk; a two\-trunk architecture would provide stricter causal isolation of the two failure modes\.

#### Broader impact and future directions\.

The framework can inform ergonomic interventions extending productive working life; risks of discriminatory reuse are mitigated by population\-level synthetic agents and no individual\-prediction API\. Priority experiments: convex share penalty \(−λ​log⁡\(sdom−α\)\-\\lambda\\log\(s\_\{\\text\{dom\}\}\-\\alpha\)\), non\-DP\-optimal share control, SAC comparison, a sweep over initial effort\-head bias values to test whether the escape rate increases monotonically, replacing the fixed proxy signal with a learned or evolved internal\-signal architecture, to test whether adaptive afferents restore completion without action\-space restriction, and a two\-trunk architecture to provide a stricter causal isolation of the share and effort failure modes\.

## References

- Constrained policy optimization\.InProceedings of the 34th International Conference on Machine Learning \(ICML\),pp\. 22–31\.Cited by:[§1](https://arxiv.org/html/2605.26657#S1.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2605.26657#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Agarwal, S\. M\. Kakade, J\. D\. Lee, and G\. Mahajan \(2021\)On the theory of policy gradient methods: optimality, approximation, and distribution shift\.Journal of Machine Learning Research22\(98\),pp\. 1–76\.Cited by:[§2](https://arxiv.org/html/2605.26657#S2.SS0.SSS0.Px2.p1.1)\.
- B\. E\. Ainsworth, W\. L\. Haskell, S\. D\. Herrmann, N\. Meckes, D\. R\. Bassett, C\. Tudor\-Locke, J\. L\. Greer, J\. Vezina, M\. C\. Whitt\-Glover, and A\. S\. Leon \(2011\)2011 compendium of physical activities: a second update of codes and met values\.Medicine & Science in Sports & Exercise43\(8\),pp\. 1575–1581\.Cited by:[Table 4](https://arxiv.org/html/2605.26657#A3.T4.3.4.3.5),[Table 4](https://arxiv.org/html/2605.26657#A3.T4.3.5.4.5),[Table 4](https://arxiv.org/html/2605.26657#A3.T4.3.6.5.5),[Table 4](https://arxiv.org/html/2605.26657#A3.T4.3.7.6.5),[Table 4](https://arxiv.org/html/2605.26657#A3.T4.3.8.7.5)\.
- E\. Altman \(1999\)Constrained Markov decision processes\.Chapman & Hall / CRC,Boca Raton\.Cited by:[§1](https://arxiv.org/html/2605.26657#S1.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2605.26657#S2.SS0.SSS0.Px1.p1.1)\.
- L\. Arendt\-Nielsen, H\. Nie, M\. B\. Laursen, B\. S\. Laursen, P\. Madeleine, O\. H\. Simonsen, and T\. Graven\-Nielsen \(2010\)Sensitization in patients with painful knee osteoarthritis\.Pain149\(3\),pp\. 573–581\.Cited by:[Appendix C](https://arxiv.org/html/2605.26657#A3.p3.3),[§4](https://arxiv.org/html/2605.26657#S4.p2.1),[§8](https://arxiv.org/html/2605.26657#S8.SS0.SSS0.Px1.p1.2)\.
- J\. A\. Arjona\-Medina, M\. Gillhofer, M\. Widrich, T\. Unterthiner, J\. Brandstetter, and S\. Hochreiter \(2019\)RUDDER: return decomposition for delayed rewards\.InAdvances in Neural Information Processing Systems 32 \(NeurIPS\),pp\. 13544–13555\.Cited by:[§1](https://arxiv.org/html/2605.26657#S1.p1.1),[§2](https://arxiv.org/html/2605.26657#S2.SS0.SSS0.Px3.p1.1)\.
- M\. E\. Baratz, F\. H\. Fu, and R\. Mengato \(1986\)Meniscal tears: the effect of meniscectomy and of repair on intraarticular contact areas and stress in the human knee\.The American Journal of Sports Medicine14\(4\),pp\. 270–275\.Cited by:[Appendix C](https://arxiv.org/html/2605.26657#A3.p2.3),[§4](https://arxiv.org/html/2605.26657#S4.p2.1)\.
- J\. Bhandari and D\. Russo \(2024\)Global optimality guarantees for policy gradient methods\.Operations Research72\(5\),pp\. 1906–1927\.Cited by:[§2](https://arxiv.org/html/2605.26657#S2.SS0.SSS0.Px2.p1.1)\.
- L\. Chen, K\. Lu, A\. Rajeswaran, K\. Lee, A\. Grover, M\. Laskin, P\. Abbeel, A\. Srinivas, and I\. Mordatch \(2021\)Decision transformer: reinforcement learning via sequence modeling\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.34\.Cited by:[§2](https://arxiv.org/html/2605.26657#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Chow, O\. Nachum, E\. Duenez\-Guzman, and M\. Ghavamzadeh \(2018\)A Lyapunov\-based approach to safe reinforcement learning\.InAdvances in Neural Information Processing Systems 31 \(NeurIPS\),pp\. 8103–8112\.Cited by:[§1](https://arxiv.org/html/2605.26657#S1.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2605.26657#S2.SS0.SSS0.Px1.p1.1)\.
- D\. Coggon, P\. Croft, S\. Kellingray, D\. Barrett, M\. McLaren, and C\. Cooper \(2000\)Occupational physical activities and osteoarthritis of the knee\.Arthritis & Rheumatism43\(7\),pp\. 1443–1449\.Cited by:[Table 4](https://arxiv.org/html/2605.26657#A3.T4.3.2.1.5),[Appendix C](https://arxiv.org/html/2605.26657#A3.p1.12),[§4](https://arxiv.org/html/2605.26657#S4.p2.1),[§8](https://arxiv.org/html/2605.26657#S8.SS0.SSS0.Px1.p1.2)\.
- J\. P\. DiFiori, H\. J\. Benjamin, J\. S\. Brenner, A\. Gregory, N\. Jayanthi, G\. L\. Landry, and A\. Luke \(2014\)Overuse injuries and burnout in youth sports: a position statement from the american medical society for sports medicine\.British Journal of Sports Medicine48\(4\),pp\. 287–288\.Cited by:[§8](https://arxiv.org/html/2605.26657#S8.SS0.SSS0.Px2.p1.2)\.
- M\. C\. Drakos, B\. Domb, C\. Starkey, L\. Callahan, and A\. A\. Allen \(2010\)Injury in the national basketball association: a 17\-year overview\.Sports Health2\(4\),pp\. 284–290\.Cited by:[§4](https://arxiv.org/html/2605.26657#S4.p2.1)\.
- J\. García and F\. Fernández \(2015\)A comprehensive survey on safe reinforcement learning\.Journal of Machine Learning Research16\(1\),pp\. 1437–1480\.Cited by:[§1](https://arxiv.org/html/2605.26657#S1.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2605.26657#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Guin and S\. Bhatnagar \(2022\)A policy gradient approach for finite horizon constrained Markov decision processes\.arXiv preprint arXiv:2210\.04527\.Cited by:[§3\.1](https://arxiv.org/html/2605.26657#S3.SS1.p1.11)\.
- T\. Haarnoja, A\. Zhou, P\. Abbeel, and S\. Levine \(2018\)Soft actor\-critic: off\-policy maximum entropy deep reinforcement learning with a stochastic actor\.InProceedings of the 35th International Conference on Machine Learning \(ICML\),pp\. 1861–1870\.Cited by:[§8](https://arxiv.org/html/2605.26657#S8.SS0.SSS0.Px3.p1.1)\.
- A\. Harutyunyan, W\. Dabney, T\. Mesnard, M\. G\. Azar, B\. Piot, N\. Heess, H\. van Hasselt, G\. Wayne, S\. Singh, D\. Precup, and R\. Munos \(2019\)Hindsight credit assignment\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.32\.Cited by:[§1](https://arxiv.org/html/2605.26657#S1.p1.1),[§2](https://arxiv.org/html/2605.26657#S2.SS0.SSS0.Px3.p1.1)\.
- L\. K\. Jensen \(2008\)Knee osteoarthritis: influence of work involving heavy lifting, kneeling, climbing stairs or ladders, or kneeling/squatting combined with heavy lifting\.Occupational and Environmental Medicine65\(2\),pp\. 72–89\.Cited by:[Table 4](https://arxiv.org/html/2605.26657#A3.T4.3.2.1.5),[Appendix C](https://arxiv.org/html/2605.26657#A3.p1.12),[§4](https://arxiv.org/html/2605.26657#S4.p2.1),[§8](https://arxiv.org/html/2605.26657#S8.SS0.SSS0.Px1.p1.2)\.
- T\. R\. Jildeh \(2024\)Load management is essential to prevent season\-ending injuries in the national basketball association\.Arthroscopy: The Journal of Arthroscopic and Related Surgery40\(6\),pp\. 1680–1681\.Cited by:[§8](https://arxiv.org/html/2605.26657#S8.SS0.SSS0.Px2.p1.2)\.
- S\. Mahadevan \(1996\)Average reward reinforcement learning: foundations, algorithms, and empirical results\.Machine Learning22,pp\. 159–195\.Cited by:[Appendix D](https://arxiv.org/html/2605.26657#A4.SS0.SSS0.Px1.p1.9),[§5](https://arxiv.org/html/2605.26657#S5.SS0.SSS0.Px1.p1.2)\.
- M\. McLellan, S\. Allahabadi, and N\. K\. Pandya \(2022\)Youth sports specialization and its effect on professional, elite, and olympic athlete performance, career longevity, and injury rates: a systematic review\.Orthopaedic Journal of Sports Medicine10\(11\),pp\. 23259671221129594\.Cited by:[§6\.3](https://arxiv.org/html/2605.26657#S6.SS3.p2.9),[§8](https://arxiv.org/html/2605.26657#S8.SS0.SSS0.Px2.p1.2)\.
- J\. Mei, C\. Xiao, C\. Szepesvári, and D\. Schuurmans \(2020\)On the global convergence rates of softmax policy gradient methods\.InProceedings of the 37th International Conference on Machine Learning \(ICML\),pp\. 6820–6829\.Cited by:[§2](https://arxiv.org/html/2605.26657#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Y\. Ng, D\. Harada, and S\. Russell \(1999\)Policy invariance under reward transformations: theory and application to reward shaping\.InProceedings of the 16th International Conference on Machine Learning \(ICML\),pp\. 278–287\.Cited by:[§2](https://arxiv.org/html/2605.26657#S2.SS0.SSS0.Px3.p1.1)\.
- K\. T\. Palmer \(2012\)Occupational activities and osteoarthritis of the knee\.British Medical Bulletin102\(1\),pp\. 147–170\.Cited by:[§8](https://arxiv.org/html/2605.26657#S8.SS0.SSS0.Px1.p1.2)\.
- M\. L\. Puterman \(1994\)Markov decision processes: discrete stochastic dynamic programming\.John Wiley & Sons, Inc\.,New York\.Cited by:[§1](https://arxiv.org/html/2605.26657#S1.SS0.SSS0.Px2.p1.1)\.
- A\. Ray, J\. Achiam, and D\. Amodei \(2019\)Benchmarking safe exploration in deep reinforcement learning\.Technical reportOpenAI\.Note:Safety Gym benchmarkCited by:[§1](https://arxiv.org/html/2605.26657#S1.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2605.26657#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Rytter, L\. K\. Jensen, J\. P\. Bonde, A\. G\. Jurik, and N\. Egund \(2009\)Occupational kneeling and meniscal tears: a magnetic resonance imaging study in floor layers\.The Journal of Rheumatology36\(7\),pp\. 1512–1519\.Cited by:[Appendix C](https://arxiv.org/html/2605.26657#A3.p2.2),[§4](https://arxiv.org/html/2605.26657#S4.p2.1),[§8](https://arxiv.org/html/2605.26657#S8.SS0.SSS0.Px1.p1.2)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§1](https://arxiv.org/html/2605.26657#S1.p1.1)\.
- R\. S\. Sutton \(1990\)Integrated architectures for learning, planning, and reacting based on approximating dynamic programming\.InProceedings of the 7th International Conference on Machine Learning \(ICML\),pp\. 216–224\.Cited by:[Appendix D](https://arxiv.org/html/2605.26657#A4.SS0.SSS0.Px2),[§1](https://arxiv.org/html/2605.26657#S1.SS0.SSS0.Px2.p1.1)\.
- H\. F\. van der Molen, J\. K\. Sluiter, C\. T\. J\. Hulshof, P\. Vink, and M\. H\. W\. Frings\-Dresen \(2005\)Effectiveness of measures and implementation strategies in reducing physical work demands due to manual handling at work\.Scandinavian Journal of Work, Environment & Health31\(supplement 2\),pp\. 75–87\.Cited by:[Table 4](https://arxiv.org/html/2605.26657#A3.T4.3.3.2.5)\.
- A\. Wachiet al\.\(2024\)A survey of constraint formulations in safe reinforcement learning\.arXiv preprint arXiv:2402\.02025\.Cited by:[§3\.1](https://arxiv.org/html/2605.26657#S3.SS1.p1.11)\.
- A\. J\. Wiggins, R\. K\. Grandhi, D\. K\. Schneider, D\. Stanfield, K\. E\. Webster, and G\. D\. Myer \(2016\)Risk of secondary injury in younger athletes after anterior cruciate ligament reconstruction: a systematic review and meta\-analysis\.The American Journal of Sports Medicine44\(7\),pp\. 1861–1876\.Cited by:[§4](https://arxiv.org/html/2605.26657#S4.p2.1)\.
- C\. J\. Woolf and M\. W\. Salter \(2000\)Neuronal plasticity: increasing the gain in pain\.Science288\(5472\),pp\. 1765–1769\.Cited by:[§8](https://arxiv.org/html/2605.26657#S8.SS0.SSS0.Px1.p1.2)\.

## Appendix AHorizon Invariance of the Basin Structure

This appendix reports the horizon\-invariance experiment supporting prediction P4 \(§[3\.4](https://arxiv.org/html/2605.26657#S3.SS4)\) and the first\-phase basin\-entry claim of §[6\.2](https://arxiv.org/html/2605.26657#S6.SS2)\. The bricklayer fixed\-share Dyna sweep was run at four training horizons,H∈\{13,20,30,49\}H\\in\\\{13,20,30,49\\\}, with 10 seeds per horizon and all other hyperparameters held fixed \(Appendix[D](https://arxiv.org/html/2605.26657#A4)\)\.

#### Evaluation protocol\.

All policies, regardless of training horizon, are evaluated on the fullH=49H=49environment\. A policy trained atH=13H=13is therefore evaluated zero\-shot on the career up to age 65; itsMfinalM\_\{\\text\{final\}\}is measured at step 48 \(age 65\)\. This ensures all four training horizons produce comparableMfinalM\_\{\\text\{final\}\}values\. The horizon\-invariance ofMfinal=0\.523±0\.001M\_\{\\text\{final\}\}=0\.523\\pm 0\.001across all four conditions therefore cannot be attributed to differences in evaluation length\.

#### Findings\.

\(i\) The reactive\-decline basin holds 6–8 of 10 seeds at everyHH\(Figure[4](https://arxiv.org/html/2605.26657#A1.F4)a\); the dominance rate does not increase with longer training\. \(ii\) Dominant\-mode effort schedules overlap to within seed noise across the four horizons \(panel b\): the policy learned from 13 training years is indistinguishable from the policy learned from 49\. \(iii\) Dominant\-modeMfinalM\_\{\\text\{final\}\}is invariant at0\.523±0\.0010\.523\\pm 0\.001\(panel c\); the gap to DP\-optimal \(0\.2710\.271\) does not shrink withHH, confirming first\-phase basin entry \(§[6\.2](https://arxiv.org/html/2605.26657#S6.SS2)\)\. NBA replication of P4 atH∈\{7,10,15,20\}H\\in\\\{7,10,15,20\\\}is reported in §[7](https://arxiv.org/html/2605.26657#S7)\.

![Refer to caption](https://arxiv.org/html/2605.26657v1/figures/figA1_horizon_invariance.png)Figure 4:Horizon invariance of the basin\-of\-attraction structure \(bricklayer testbed\)\. All conditions evaluated on the fullH=49H=49environment\. \(a\) Reactive\-decline basin holds 6–8 of 10 seeds across training horizonsH∈\{13,20,30,49\}H\\in\\\{13,20,30,49\\\}\. \(b\) Dominant\-mode effort profiles overlap nearly perfectly acrossHH: the policy learned from 13 years of training rollouts is indistinguishable in effort schedule from the policy learned from 49 years\. \(c\) Dominant\-modeMfinalM\_\{\\text\{final\}\}is invariant acrossHHat0\.523±0\.0010\.523\\pm 0\.001; the gap to DP\-optimal \(0\.2710\.271\) does not shrink with longer training\. This is direct evidence for first\-phase basin entry: the reactive attractor is determined by the first policy\-gradient update, not by the long\-horizon signal that longer rollouts provide\. Data: cluster 152494 \(H=49H=49\) and cluster 152523 \(H∈\{13,20,30\}H\\in\\\{13,20,30\\\}\)\.

## Appendix BStep\-0 Basin Entry in a Minimal Cumulative\-Damage MDP

This appendix derives Proposition[1](https://arxiv.org/html/2605.26657#Thmproposition1)on a minimal MDP that retains only the features essential to the step\-0 mechanism\. The minimal MDP omits the multi\-activity structure, secondary\-channel amplification, and age\-dependent capacity of the testbeds; it therefore speaks to the*direction of the expected first update*and theH∗H^\{\*\}boundary, not to the magnitude ofΔ​Mfinal\\Delta M\_\{\\text\{final\}\}or the full attractor structure\.

#### Setup\.

Finite\-horizon MDP with scalar damageDt∈\[0,1\]D\_\{t\}\\in\[0,1\],D0=0D\_\{0\}=0; binary effortet∈\{eL,eH\}e\_\{t\}\\in\\\{e\_\{L\},e\_\{H\}\\\},0<eL<eH≤10<e\_\{L\}<e\_\{H\}\\leq 1; transitionDt\+1=min⁡\(1,Dt\+κ​et\)D\_\{t\+1\}=\\min\(1,D\_\{t\}\+\\kappa e\_\{t\}\); rewardr​\(Dt,et\)=etβ​\(1−Dt\)2r\(D\_\{t\},e\_\{t\}\)=e\_\{t\}^\{\\beta\}\(1\-D\_\{t\}\)^\{2\}forβ∈\(0,1\]\\beta\\in\(0,1\]; returnG​\(τ\)=∑t=0H−1r​\(Dt,et\)G\(\\tau\)=\\sum\_\{t=0\}^\{H\-1\}r\(D\_\{t\},e\_\{t\}\)\. This satisfies Assumption[1](https://arxiv.org/html/2605.26657#Thmassumption1)and is the skeleton on which the step\-0 mechanism operates\.

###### Lemma 1\(Lipschitz bound onVπV^\{\\pi\}\)\.

Lete¯β=12​\(eLβ\+eHβ\)\\bar\{e\}\_\{\\beta\}=\\tfrac\{1\}\{2\}\(e\_\{L\}^\{\\beta\}\+e\_\{H\}^\{\\beta\}\)be the expected per\-step effort reward under the uniform policy atD\>0D\>0\. Under the deterministic transitionDt\+1=Dt\+κ​etD\_\{t\+1\}=D\_\{t\}\+\\kappa e\_\{t\}\(prior to clipping at11\), the continuation value satisfies

\|Vπ​\(κ​eH\)−Vπ​\(κ​eL\)\|≤2​\(H−1\)​e¯β​κ​\(eH−eL\)\.\\bigl\|V^\{\\pi\}\(\\kappa e\_\{H\}\)\-V^\{\\pi\}\(\\kappa e\_\{L\}\)\\bigr\|\\;\\leq\\;2\(H\-1\)\\,\\bar\{e\}\_\{\\beta\}\\,\\kappa\(e\_\{H\}\-e\_\{L\}\)\.*Proof\.*The rewardr​\(D,e\)=eβ​\(1−D\)2r\(D,e\)=e^\{\\beta\}\(1\-D\)^\{2\}satisfies\|∂r/∂D\|=2​eβ​\(1−D\)≤2​eβ\|\\partial r/\\partial D\|=2e^\{\\beta\}\(1\-D\)\\leq 2e^\{\\beta\}on\[0,1\]\[0,1\]\. Under the uniform policy fort≥1t\\geq 1, damage evolves deterministically, so∂Dt/∂D1=1\\partial D\_\{t\}/\\partial D\_\{1\}=1for allt≥1t\\geq 1\. The continuation valueVπ​\(κ​e\)V^\{\\pi\}\(\\kappa e\)covers stepst=1,…,H−1t=1,\\ldots,H\-1\(i\.e\.,H−1H\-1steps\)\. By the chain rule:

\|d​Vπd​D1\|≤∑t=1H−1𝔼​\[2​etβ\]≤2​\(H−1\)​e¯β\.\\left\|\\frac\{dV^\{\\pi\}\}\{dD\_\{1\}\}\\right\|\\leq\\sum\_\{t=1\}^\{H\-1\}\\mathbb\{E\}\\bigl\[2e\_\{t\}^\{\\beta\}\\bigr\]\\leq 2\(H\-1\)\\,\\bar\{e\}\_\{\\beta\}\.Applying this to the gapVπ​\(κ​eH\)−Vπ​\(κ​eL\)V^\{\\pi\}\(\\kappa e\_\{H\}\)\-V^\{\\pi\}\(\\kappa e\_\{L\}\)via the mean\-value theorem inD1D\_\{1\}gives the stated bound\.□\\square

#### Policy and first\-update gradient\.

Parameterise theD=0D=0action distribution byθ∈ℝ\\theta\\in\\mathbb\{R\}:πθ​\(eH∣D=0\)=σ​\(θ\)\\pi\_\{\\theta\}\(e\_\{H\}\\mid D=0\)=\\sigma\(\\theta\), initialised atθ0=0\\theta\_\{0\}=0\(uniform\)\. Action selection atD\>0D\>0is held at uniformPr⁡\(eH\)=1/2\\Pr\(e\_\{H\}\)=1/2; we analyse only the first update onθ\\theta\. The REINFORCE gradient is∇θJ​\(θ\)=𝔼π​\[∇θlog⁡πθ​\(a0∣D0\)​G​\(τ\)\]\\nabla\_\{\\theta\}J\(\\theta\)=\\mathbb\{E\}\_\{\\pi\}\\bigl\[\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\_\{0\}\\mid D\_\{0\}\)\\,G\(\\tau\)\\bigr\]\. Atθ0=0\\theta\_\{0\}=0,σ​\(θ0\)=1/2\\sigma\(\\theta\_\{0\}\)=1/2and∇θlog⁡πθ0​\(eH∣0\)=\+1/2\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\_\{0\}\}\(e\_\{H\}\\mid 0\)=\+1/2,∇θlog⁡πθ0​\(eL∣0\)=−1/2\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\_\{0\}\}\(e\_\{L\}\\mid 0\)=\-1/2\. Splitting the expectation by the first action and identifying the inner expectations asQπ​\(0,eH\)Q^\{\\pi\}\(0,e\_\{H\}\)andQπ​\(0,eL\)Q^\{\\pi\}\(0,e\_\{L\}\)yields

∇θJ​\(θ\)\|θ0=14​\[Qπ​\(0,eH\)−Qπ​\(0,eL\)\]\.\\nabla\_\{\\theta\}J\(\\theta\)\\big\|\_\{\\theta\_\{0\}\}=\\tfrac\{1\}\{4\}\\bigl\[Q^\{\\pi\}\(0,e\_\{H\}\)\-Q^\{\\pi\}\(0,e\_\{L\}\)\\bigr\]\.\(1\)The sign of the*expected*first update equals the sign of the state\-action\-value gap atD=0D=0\. Using the Bellman decompositionQπ​\(0,e\)=r​\(0,e\)\+Vπ​\(κ​e\)Q^\{\\pi\}\(0,e\)=r\(0,e\)\+V^\{\\pi\}\(\\kappa e\)and Lemma[1](https://arxiv.org/html/2605.26657#Thmlemma1):

Qπ​\(0,eH\)−Qπ​\(0,eL\)≥\(eHβ−eLβ\)−2​\(H−1\)​κ​e¯β​\(eH−eL\)\.Q^\{\\pi\}\(0,e\_\{H\}\)\-Q^\{\\pi\}\(0,e\_\{L\}\)\\geq\(e\_\{H\}^\{\\beta\}\-e\_\{L\}^\{\\beta\}\)\-2\(H\-1\)\\,\\kappa\\,\\bar\{e\}\_\{\\beta\}\\,\(e\_\{H\}\-e\_\{L\}\)\.\(2\)Equivalently, writingeHβ−eLβ=Δβe\_\{H\}^\{\\beta\}\-e\_\{L\}^\{\\beta\}=\\Delta\_\{\\beta\}for brevity:

Qπ​\(0,eH\)−Qπ​\(0,eL\)≥Δβ−2​\(H−1\)​κ​e¯β​\(eH−eL\)\.Q^\{\\pi\}\(0,e\_\{H\}\)\-Q^\{\\pi\}\(0,e\_\{L\}\)\\geq\\Delta\_\{\\beta\}\-2\(H\-1\)\\,\\kappa\\,\\bar\{e\}\_\{\\beta\}\\,\(e\_\{H\}\-e\_\{L\}\)\.\(3\)
###### Proposition 1\(Step\-0 commitment\)\.

If

eHβ−eLβ\>2​\(H−1\)​κ​e¯β​\(eH−eL\),e\_\{H\}^\{\\beta\}\-e\_\{L\}^\{\\beta\}\\;\>\\;2\(H\-1\)\\,\\kappa\\,\\bar\{e\}\_\{\\beta\}\\,\(e\_\{H\}\-e\_\{L\}\),then the*expected*first policy\-gradient update is positive,𝔼​\[∇θJ​\(θ\)\|θ0\]\>0\\mathbb\{E\}\[\\nabla\_\{\\theta\}J\(\\theta\)\|\_\{\\theta\_\{0\}\}\]\>0: the expected update shiftsθ\\thetatowardeHe\_\{H\}atD=0D=0\. Because the condition depends only on the MDP parameters\(β,κ,H,eL,eH\)\(\\beta,\\kappa,H,e\_\{L\},e\_\{H\}\)and not on random seed, the*expected*gradient favourseHe\_\{H\}for every seed\. Individual stochastic gradient estimates may differ in sign \(consistent with the two escaping seeds in §[6\.2](https://arxiv.org/html/2605.26657#S6.SS2)\)\.

#### Critical horizon\.

Define

H∗=1\+eHβ−eLβ2​κ​e¯β​\(eH−eL\)\.H^\{\*\}=1\+\\frac\{e\_\{H\}^\{\\beta\}\-e\_\{L\}^\{\\beta\}\}\{2\\,\\kappa\\,\\bar\{e\}\_\{\\beta\}\\,\(e\_\{H\}\-e\_\{L\}\)\}\.Condition \([3](https://arxiv.org/html/2605.26657#A2.E3)\) holds for allH≤H∗H\\leq H^\{\*\}\.

#### Scope and self\-reinforcement\.

The proposition establishes the direction of the expected first update only\. It does not prove convergence to theeH=1e\_\{H\}=1attractor\. Self\-reinforcement bridges the direction of the first update to empirical saturation: \(i\) after the first expected update𝔼​\[θ1\]\>0\\mathbb\{E\}\[\\theta\_\{1\}\]\>0, rollouts fromD0=0D\_\{0\}=0increasingly sampleeHe\_\{H\}, concentrating subsequent gradient signal oneHe\_\{H\}\-prefixed trajectories; \(ii\) early\-rollout damage is small, so reward suppression is dominated by theeHβ−eLβe\_\{H\}^\{\\beta\}\-e\_\{L\}^\{\\beta\}reward gap for the first∼1/κ\\sim 1/\\kappasteps; \(iii\) subsequent updates receive a stronger signal of the same sign, drivingθ\\thetamonotonically untilπ​\(eH∣D0=0\)→1\\pi\(e\_\{H\}\\mid D\_\{0\}=0\)\\to 1\. This is consistent with empirical saturation within the first 1% of training for all 8 reactive\-cluster seeds \(§[6\.2](https://arxiv.org/html/2605.26657#S6.SS2)\)\. The proposition does not cover the multi\-activity, continuous\-effort testbeds\. It is a motivating formal result, not a direct theorem about those environments\.

#### BricklayerH∗H^\{\*\}calculation\.

In the bricklayer, under greedy full effort \(eH=1e\_\{H\}=1\) on the dominant activity, the per\-step damage rate isκ≈damage\_scale×load≈0\.083×0\.9≈0\.075\\kappa\\approx\\text\{damage\\\_scale\}\\times\\text\{load\}\\approx 0\.083\\times 0\.9\\approx 0\.075\. Usingβ=0\.6\\beta=0\.6,eL≈0\.05e\_\{L\}\\approx 0\.05,eH=1\.0e\_\{H\}=1\.0:

eHβ−eLβ=1\.0−0\.050\.6≈1\.0−0\.178=0\.822,e¯β=12​\(0\.178\+1\.0\)=0\.589,eH−eL=0\.95\.e\_\{H\}^\{\\beta\}\-e\_\{L\}^\{\\beta\}=1\.0\-0\.05^\{0\.6\}\\approx 1\.0\-0\.178=0\.822,\\quad\\bar\{e\}\_\{\\beta\}=\\tfrac\{1\}\{2\}\(0\.178\+1\.0\)=0\.589,\\quad e\_\{H\}\-e\_\{L\}=0\.95\.H∗=1\+0\.8222×0\.075×0\.589×0\.95=1\+0\.8220\.0840≈10\.8\.H^\{\*\}=1\+\\frac\{0\.822\}\{2\\times 0\.075\\times 0\.589\\times 0\.95\}=1\+\\frac\{0\.822\}\{0\.0840\}\\approx 10\.8\.The analytical guarantee therefore coversH≤10H\\leq 10\. All four bricklayer training horizons \(H∈\{13,20,30,49\}H\\in\\\{13,20,30,49\\\}\) lie aboveH∗H^\{\*\}, yet empirical basin entry occurs at all four \(8/10 seeds each,Mfinal=0\.523±0\.001M\_\{\\text\{final\}\}=0\.523\\pm 0\.001\)\. This is consistent with the condition being sufficient but not necessary: the proposition predicts commitment whenH≤H∗H\\leq H^\{\*\}, while the empirical results show that the basin remains dominant well beyond the analytical guarantee, as in the NBA environment\.

*NBA\.*The NBA environment has two distinct damage channels: the primary cartilage channel \(κ=δ=0\.055\\kappa=\\delta=0\.055\) and the secondary meniscal channel \(κ=μeff=0\.15\\kappa=\\mu\_\{\\text\{eff\}\}=0\.15\)\. Usingβ=0\.6\\beta=0\.6,eL=0\.05e\_\{L\}=0\.05,eH=1\.0e\_\{H\}=1\.0\(consistent with the assumption0<eL<eH0<e\_\{L\}<e\_\{H\}and the NBA minimum\-effort floor\):

eHβ−eLβ=0\.822,e¯β=0\.589,eH−eL=0\.95\.e\_\{H\}^\{\\beta\}\-e\_\{L\}^\{\\beta\}=0\.822,\\quad\\bar\{e\}\_\{\\beta\}=0\.589,\\quad e\_\{H\}\-e\_\{L\}=0\.95\.Hprimary∗=1\+0\.8222×0\.055×0\.589×0\.95≈1\+0\.8220\.0615≈14\.4,H^\{\*\}\_\{\\text\{primary\}\}=1\+\\frac\{0\.822\}\{2\\times 0\.055\\times 0\.589\\times 0\.95\}\\approx 1\+\\frac\{0\.822\}\{0\.0615\}\\approx 14\.4,Hsecondary∗=1\+0\.8222×0\.15×0\.589×0\.95≈1\+0\.8220\.168≈5\.9\.H^\{\*\}\_\{\\text\{secondary\}\}=1\+\\frac\{0\.822\}\{2\\times 0\.15\\times 0\.589\\times 0\.95\}\\approx 1\+\\frac\{0\.822\}\{0\.168\}\\approx 5\.9\.The overall guarantee floor isH∗≈6H^\{\*\}\\approx 6\(secondary channel binding\) and ceilingH∗≈14H^\{\*\}\\approx 14\(primary channel\), giving a range ofH∗∈\[6,14\]H^\{\*\}\\in\[6,14\]depending on which channel is the binding constraint\. The horizonH=20H=20lies above both bounds\. Empirical basin entry at 4/10 seeds is consistent with the condition being sufficient but not necessary\. The horizon\-sweep outlier atH=15H=15lies just aboveHprimary∗≈14H^\{\*\}\_\{\\text\{primary\}\}\\approx 14: it is the first integer horizon outside the formal commitment region under the primary channel, making theH=15H=15exception the sharpest available confirmation of theH∗H^\{\*\}boundary on the discrete horizon grid\.

## Appendix CBiological Parameter Provenance

The bricklayer environment models a 49\-year career with seven activities parameterised by energy cost, hazard coefficient, and performance coefficient \(Table[4](https://arxiv.org/html/2605.26657#A3.T4)\)\. Per\-step cumulative damage evolves as

Dt\+1=max⁡\(0,min⁡\(1,Dt\+damage\_scale⋅loadt⋅b​\(BMIt\)⋅m​\(Mt\)1\.3−r​\(aget\)\)\),D\_\{t\+1\}=\\max\\\!\\Bigl\(0,\\;\\min\\\!\\bigl\(1,\\;D\_\{t\}\+\\text\{damage\\\_scale\}\\cdot\\text\{load\}\_\{t\}\\cdot b\(\\mathrm\{BMI\}\_\{t\}\)\\cdot m\(M\_\{t\}\)^\{1\.3\}\-r\(\\text\{age\}\_\{t\}\)\\bigr\)\\Bigr\),\(4\)whereloadt=0\.40⋅stresst\+0\.35⋅straint\+0\.25⋅sheart\\text\{load\}\_\{t\}=0\.40\\cdot\\text\{stress\}\_\{t\}\+0\.35\\cdot\\text\{strain\}\_\{t\}\+0\.25\\cdot\\text\{shear\}\_\{t\}is the weighted load\. For each activityii, the load components are derived from the single hazard scalarhih\_\{i\}asstressi=0\.45​hi\\text\{stress\}\_\{i\}=0\.45\\,h\_\{i\},straini=0\.35​hi\\text\{strain\}\_\{i\}=0\.35\\,h\_\{i\},sheari=0\.20​hi\\text\{shear\}\_\{i\}=0\.20\\,h\_\{i\}\(proportions calibrated to knee\-joint biomechanicsJensen[2008](https://arxiv.org/html/2605.26657#bib.bib20)\), soloadt=\(0\.40×0\.45\+0\.35×0\.35\+0\.25×0\.20\)​∑isi,t​ei,t​hi=0\.355​∑isi,t​ei,t​hi\\text\{load\}\_\{t\}=\\bigl\(0\.40\\times 0\.45\+0\.35\\times 0\.35\+0\.25\\times 0\.20\\bigr\)\\sum\_\{i\}s\_\{i,t\}\\,e\_\{i,t\}\\,h\_\{i\}=0\.355\\,\\sum\_\{i\}s\_\{i,t\}\\,e\_\{i,t\}\\,h\_\{i\}\.damage\_scale=0\.083\\text\{damage\\\_scale\}=0\.083calibrated to a no\-regulation end\-of\-careerD≈0\.75D\\approx 0\.75\[Coggonet al\.,[2000](https://arxiv.org/html/2605.26657#bib.bib19), Jensen,[2008](https://arxiv.org/html/2605.26657#bib.bib20)\],b​\(BMI\)=1\+0\.07​max⁡\(0,BMI−22\)b\(\\mathrm\{BMI\}\)=1\+0\.07\\,\\max\(0,\\mathrm\{BMI\}\-22\)is a BMI multiplier, andr​\(age\)=0\.015⋅max⁡\(0,1−\(age−30\)/50\)r\(\\text\{age\}\)=0\.015\\cdot\\max\(0,1\-\(\\text\{age\}\-30\)/50\)is a mild age\-dependent recovery term \(clamped to maintainD≥0D\\geq 0\)\.

Meniscal degradation evolves as

Mt\+1=Mt−base\_rate⋅sheart⋅a​\(Mt\)⋅f​\(aget\),M\_\{t\+1\}=M\_\{t\}\-\\text\{base\\\_rate\}\\cdot\\text\{shear\}\_\{t\}\\cdot a\(M\_\{t\}\)\\cdot f\(\\text\{age\}\_\{t\}\),withbase\_rate=0\.075\\text\{base\\\_rate\}=0\.075derived from a target end\-of\-careerM≈0\.65M\\approx 0\.65under 70% block\_laying share at 70% effort\[Rytteret al\.,[2009](https://arxiv.org/html/2605.26657#bib.bib21)\]\. The self\-amplification factor is

a​\(M\)=\{1\+3​\(0\.6−M\)if​M<0\.6,1otherwise,a\(M\)=\\begin\{cases\}1\+3\(0\.6\-M\)&\\text\{if \}M<0\.6,\\\\ 1&\\text\{otherwise,\}\\end\{cases\}and the age\-onset factor is

f​\(age\)=\{1\+0\.5​\(age−45\)/20if age≥45,1otherwise\.f\(\\text\{age\}\)=\\begin\{cases\}1\+0\.5\\,\(\\text\{age\}\-45\)/20&\\text\{if age\}\\geq 45,\\\\ 1&\\text\{otherwise\.\}\\end\{cases\}Baratz amplificationm​\(M\)=1/\(0\.45\+0\.55​M\)m\(M\)=1/\(0\.45\+0\.55M\)matches the 55% contact\-area reduction inBaratzet al\.\[[1986](https://arxiv.org/html/2605.26657#bib.bib22)\]\.

The proxy signal isSt=1\+αs​DtγsS\_\{t\}=1\+\\alpha\_\{s\}D\_\{t\}^\{\\gamma\_\{s\}\}withαs=2\.5\\alpha\_\{s\}=2\.5,γs=1\.8\\gamma\_\{s\}=1\.8\(sensitisation parameters fit to the pressure\-pain threshold data ofArendt\-Nielsenet al\.[2010](https://arxiv.org/html/2605.26657#bib.bib23)\)\. The reward function is

rt=hocc​\(Dt,St\)​∑isi,t​ei,tβ​perfi/perfmax,r\_\{t\}=h\_\{\\text\{occ\}\}\(D\_\{t\},S\_\{t\}\)\\;\\sum\_\{i\}s\_\{i,t\}\\,e\_\{i,t\}^\{\\,\\beta\}\\,\\mathrm\{perf\}\_\{i\}/\\mathrm\{perf\}\_\{\\max\},wherehocc​\(D,S\)=max⁡\(0,1−𝟏D\>Dclin⋅ν​\(D−Dclin\)⋅S\)h\_\{\\text\{occ\}\}\(D,S\)=\\max\(0,1\-\\mathbf\{1\}\_\{D\>D\_\{\\text\{clin\}\}\}\\cdot\\nu\(D\-D\_\{\\text\{clin\}\}\)\\cdot S\)is the HMS capacity\-damage feedback withDclin=0\.30D\_\{\\text\{clin\}\}=0\.30andν=1\.5\\nu=1\.5, andβ=0\.6\\beta=0\.6\.

Table 4:Bricklayer activity parameters \(energy/hazard scale 0–100; performance scale 0–110, where values above 100 encode above\-average task output\)\.
## Appendix DHyperparameters and Implementation

#### PPO\.

Discountγ=1\.0\\gamma=1\.0\(undiscounted episodic return\); rbar exponential moving\-average baseliner¯←\(1−η\)​r¯\+η​rt\\bar\{r\}\\leftarrow\(1\-\\eta\)\\bar\{r\}\+\\eta r\_\{t\},η=0\.01\\eta=0\.01\[Mahadevan,[1996](https://arxiv.org/html/2605.26657#bib.bib3)\]; training steps10610^\{6\}\(primary\),H⋅2×104H\\cdot 2\\times 10^\{4\}\(horizon sweep\); clipϵ=0\.2\\epsilon=0\.2; entropy coefficientcent=0\.05c\_\{\\text\{ent\}\}=0\.05; value coefficientcv=0\.5c\_\{v\}=0\.5; learning rate3×10−43\\times 10^\{\-4\}\(Adam\); minibatch 64; epochs 4; rollout 2048\.

#### Dyna variants\[Sutton,[1990](https://arxiv.org/html/2605.26657#bib.bib33)\]\.

The Dyna agent interleaves real\-environment rollouts with planning steps using the ground\-truth simulator as the world model\. Thefixed\-sharecondition freezes activity shares at the age\-stratified DP\-optimal allocation and learns only per\-activity effort\. Theunrestrictedcondition learns both shares and efforts\. Both use theno\_exitflag \(role\_exit, capacity\_exit, and age\_limit checks disabled during training\); enforcement is always active at evaluation\.

#### Policy network\.

Shared trunk: two FC layers \(128 units, ReLU\); Dirichlet share head; Normal effort head \(concatenates trunk output and sampled shares; clamped to\[0,1\]\[0,1\]\)\.

#### Dynamic programming\.

Backward induction on\(D,M,age\)\(D,M,\\text\{age\}\)grid, resolution\(0\.01,0\.01,1\)\(0\.01,0\.01,1\), effort grid\{0\.20,0\.25,…,0\.80\}\\\{0\.20,0\.25,\\ldots,0\.80\\\}at the DP\-optimal share allocation\. The age\-stratified DP\-optimal shares used in thefixed\_sharecondition are read directly from the argmax of this backward\-induction pass\. Compute time: under 2 hr on a single CPU\.

#### Initialization intervention\.

Effort head final\-layer bias set to−1\.386\-1\.386soσ​\(−1\.386\)≈0\.20\\sigma\(\-1\.386\)\\approx 0\.20; all other hyperparameters and seeds\{0,…,9\}\\\{0,\\ldots,9\\\}identical to the fixed\-share sweep\. Pre\-registered binding thresholds frozen 2026\-04\-20 atprompts/e1\_prereg\.md\.

#### Evaluation\.

100 independent rollouts per seed; 95% CIs from empirical distribution across seeds\.

Table 5:Initialization\-bias intervention \(10 seeds\)\. Reactive\-basin membership requires initial\-state effort\>0\.80\>0\.80, mid\-horizon effort\>0\.40\>0\.40, late\-horizon effort<0\.30<0\.30, and monotone decline \(tolerance0\.020\.02\)\. Pre\-registered classifier frozen 2026\-04\-20\. The four escaping seeds \(s∈\{0,4,7,9\}s\\in\\\{0,4,7,9\\\}\) split into three distinct attractors; only seeds 4 and 7 reach the DP\-competitive region\.

## Appendix EFull\-action\-space DP: CMA\-ES partial relaxation

We use CMA\-ES \(population 50\) to optimise a 98\-dimensional vector parameterising per\-year dominant\-activity shares∈\[0\.15,0\.80\]s\\in\[0\.15,0\.80\]and efforte∈\[0\.05,1\.00\]e\\in\[0\.05,1\.00\]for each of the 49 career years \(non\-dominant shares uniform; non\-dominant efforts fixed at0\.400\.40\)\. Over 300 generations \(15,000 evaluations\), CMA\-ES converges to a solution matching fixed\-share DP to three significant figures in reward,MfinalM\_\{\\text\{final\}\}, andDfinalD\_\{\\text\{final\}\}: within the partial relaxation tested, the fixed\-share restriction is not binding\. A true full\-action DP \(14\-dimensional per year\) remains to be computed; if it yieldsMfinal\>0\.794M\_\{\\text\{final\}\}\>0\.794, the0\.2710\.271gap widens; it is already a lower bound on the shortfall\.

## Appendix FRole\-violation sensitivity

A joint sweep overW∈\{3,5,7\}W\\in\\\{3,5,7\\\}andα∈\{0\.10,0\.15,0\.20\}\\alpha\\in\\\{0\.10,0\.15,0\.20\\\}\(5 seeds per cell, 90 runs\) finds the completion pattern fully robust: PPO\-real achieves completion rateexactly 0across all 45 runs; fixed\-share Dyna achieves completion rateexactly 1across all 45 runs at every\(W,α\)\(W,\\alpha\)cell\. The completion failure of PPO\-real is not an artefact of the specific threshold\.

## Appendix GNBA Replication: Parameter Provenance

Six activities with post\_play as dominant activity \(unique reward maximiser:perf=110\\mathrm\{perf\}=110; unique hazard maximiser:hazard=90\\mathrm\{hazard\}=90\)\. Normalised\-hazard load model:load=∑isi​ei​hi/hmax\\mathrm\{load\}=\\sum\_\{i\}s\_\{i\}e\_\{i\}h\_\{i\}/h\_\{\\max\},hmax=90h\_\{\\max\}=90\. Damage:Dt\+1=max⁡\(0,min⁡\(1,Dt\+δ​loadt​m​\(Mt\)1\.3−ρ​\(aget\)\)\)D\_\{t\+1\}=\\max\\bigl\(0,\\min\\bigl\(1,\\;D\_\{t\}\+\\delta\\,\\mathrm\{load\}\_\{t\}\\,m\(M\_\{t\}\)^\{1\.3\}\-\\rho\(\\mathrm\{age\}\_\{t\}\)\\bigr\)\\bigr\),δ=0\.055\\delta=0\.055\. Meniscal:μeff=0\.15\\mu\_\{\\text\{eff\}\}=0\.15, amplification and age\-onset factors use the same functional forms as Appendix[C](https://arxiv.org/html/2605.26657#A3)with parameters re\-calibrated to the NBA hazard array\. Role\-viability: trailing 3\-season post\_play share<α=0\.12<\\alpha=0\.12triggers release\. DP grid resolutionΔ​D=0\.02\\Delta D=0\.02,Δ​M=0\.053\\Delta M=0\.053; solve time∼12\\sim 12s\. Capacity feedback, HMS pathway, and Baratz amplification use the same functional forms as the bricklayer \(scaled by the NBA hazard array\)\. Pre\-registration:docs/prereg\_nba\.md\.

Table 6:NBA power\-forward activities\.
## NeurIPS Paper Checklist

1. 1\.Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: \[Yes\] Justification: The abstract’s four claims \(decomposition \(§[3\.2](https://arxiv.org/html/2605.26657#S3.SS2)\), completion gap \(§[5](https://arxiv.org/html/2605.26657#S5)\), optimality gap \(§[6](https://arxiv.org/html/2605.26657#S6)\), and NBA replication \(§[7](https://arxiv.org/html/2605.26657#S7)\)\) are each supported by experiments in the cited sections\. The “harmful horizon access” claim is explicitly scoped to PPO with a linear soft penalty in both the abstract and the contributions list\.
2. 2\.Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: \[Yes\] Justification: An explicit Limitations paragraph in §[8](https://arxiv.org/html/2605.26657#S8)covers: same\-engine replication scope \(parameter\-regime generality only\), Proposition–testbed scope gap \(binary\-action MDP vs\. continuous multi\-activity testbeds\), algorithm/penalty specificity of P2 \(PPO with linear soft penalty only\), pre\-registered nuanced result \(initialization intervention\), ground\-truth model assumption \(exact transition function in Dyna\), partial observability of the trailing\-window role\-viability constraint, and shared\-trunk coupling between the share and effort heads\.
3. 3\.Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete \(and correct\) proof? Answer: \[Yes\] Justification: Proposition[1](https://arxiv.org/html/2605.26657#Thmproposition1)is stated under Assumption[1](https://arxiv.org/html/2605.26657#Thmassumption1)with the additional assumption0<eL<eH≤10<e\_\{L\}<e\_\{H\}\\leq 1, and proved in Appendix[B](https://arxiv.org/html/2605.26657#A2)\. The supporting Lemma[1](https://arxiv.org/html/2605.26657#Thmlemma1)\(Lipschitz bound onVπV^\{\\pi\}\) is stated and proved in full\. Scope \(binary\-action minimal MDP, direction of expected first update only, not convergence\) is demarcated in both the main text and the appendix\.
4. 4\.Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper \(regardless of whether the code and data are provided or not\)? Answer: \[Yes\] Justification: Hyperparameters are in Appendix[D](https://arxiv.org/html/2605.26657#A4); biological parameters including explicit formulas for the self\-amplification factora​\(M\)a\(M\), age\-onset factorf​\(age\)f\(\\text\{age\}\), proxy signalStS\_\{t\}, reward functionrtr\_\{t\}, and load decomposition are in Appendix[C](https://arxiv.org/html/2605.26657#A3); NBA parameters are in Appendix[G](https://arxiv.org/html/2605.26657#A7)\. Configuration files are released in the supplementary repository\.
5. 5\.Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: \[Yes\] Justification: Code, configuration files, and per\-seedsummary\.jsonresult files are released alongside the paper in the supplementary repository, including theEnergyCareerEnvsimulator, training scripts, DP solver, and analysis pipelines\.
6. 6\.Experimental setting/details Question: Does the paper specify all the training and test details \(e\.g\., data splits, hyperparameters, how they were chosen, type of optimizer\) necessary to understand the results? Answer: \[Yes\] Justification: Full training details are in Appendix[D](https://arxiv.org/html/2605.26657#A4): optimizer \(Adam,lr=3×10−4\\text\{lr\}=3\\times 10^\{\-4\}\), PPO clip \(ϵ=0\.2\\epsilon=0\.2\), entropy and value coefficients, rollout length, minibatch size, and training steps\. The role\-viability rule andno\_exitflag semantics are defined in §[5](https://arxiv.org/html/2605.26657#S5)\. The fixed\-share PPO\-real ablation protocol is reported in §[5](https://arxiv.org/html/2605.26657#S5)\.
7. 7\.Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: \[Yes\] Justification: All main results include appropriate statistics: standard deviations on exit age; 95% bootstrap CIs onΔ​Mfinal\\Delta M\_\{\\text\{final\}\}; Welch’stt\-test with explicit degrees of freedom and Mann\-WhitneyUUfor exit\-age comparisons; 95% Clopper\-Pearson CIs on basin\-dominance rates; Fisher exact test for the cross\-environment dominance\-rate comparison\. The source of variability \(random seed\) is stated throughout\.
8. 8\.Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the computer resources \(type of compute workers, memory, time of execution\) needed to reproduce the experiments? Answer: \[Yes\] Justification: Appendix[D](https://arxiv.org/html/2605.26657#A4)reports: each PPO/Dyna run∼\\sim13 min on a single CPU node; DP solve under 2 hr on a single CPU; role\-sensitivity sweep∼\\sim20 HPC node\-hours; NBA sweeps∼\\sim15 node\-hours total\.
9. 9\.Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics? Answer: \[Yes\] Justification: No human subjects, crowdsourcing, or proprietary data are involved\. The environments are calibrated exclusively to published epidemiological aggregates\. No individual\-prediction API is released\.
10. 10\.Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: \[Yes\] Justification: §[8](https://arxiv.org/html/2605.26657#S8)discusses beneficial uses \(ergonomic interventions extending productive working life, public health policy\) and risks \(age\-discriminatory hiring or insurance pricing\), with three explicit mitigations: population\-level synthetic agents only, aggregate epidemiological calibration, and no individual\-prediction API in released code\.
11. 11\.Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse? Answer: \[N/A\] Justification: The paper releases a simulation framework and analysis code only, not a pre\-trained model or scraped dataset\. The simulator produces synthetic population\-level trajectories and poses no direct misuse risk\.
12. 12\.Licenses for existing assets Question: Are the creators or original owners of assets \(e\.g\., code, data, models\), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: \[Yes\] Justification: All biological and sports\-medicine literature is cited at point of use and in the parameter appendices\. All code dependencies \(PyTorch, NumPy, etc\.\) are open\-source\. No proprietary datasets or models are used\.
13. 13\.New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: \[Yes\] Justification: The supplementary repository contains theEnergyCareerEnvsimulator, training and DP solver scripts, analysis pipelines, and configuration files for both testbeds, with a README describing installation and reproduction steps\.
14. 14\.Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation \(if any\)? Answer: \[N/A\] Justification: The paper involves no crowdsourcing and no research with human subjects\. All environments are synthetic simulators calibrated to published epidemiological data\.
15. 15\.Institutional review board \(IRB\) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board \(IRB\) approvals \(or an equivalent approval/review based on the requirements of your country or institution\) were obtained? Answer: \[N/A\] Justification: No human subjects are involved; IRB approval is not required\.
16. 16\.Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non\-standard component of the core methods in this research? Answer: \[N/A\] Justification: LLMs were used only for writing and editing assistance and are not a component of the core methodology, scientific contributions, or experimental results\.