OR-Transformer: Scaling Real-Time Decision-Making to 1,000 Items

# OR-Transformer: Scaling Real-Time Decision-Making to 1,000 Items
Source: [https://arxiv.org/html/2609.01933](https://arxiv.org/html/2609.01933)
Shuze Daniel LiuDavid Simchi\-LeviAffiliation:Massachusetts Institute of TechnologyAffiliation:Purdue UniversityClaire ChenAffiliation:California Institute of TechnologyChutong GaoAffiliation:Massachusetts Institute of TechnologyAffiliation:Purdue UniversityShangtong ZhangAffiliation:University of Virginia

###### Abstract

Modern supply chain operations can require coordinating replenishment across thousands of heterogeneous items under correlated stochastic demand, heterogeneous lead times, and shared fixed ordering costs, yielding observation spaces exceeding10410^\{4\}dimensions\. At this scale, rolling\-horizon stochastic mixed\-integer linear programs \(MILPs\) become prohibitively slow, while standard reinforcement learning \(RL\) methods face increasingly challenging credit assignment in high\-dimensional action spaces\. We introduceOR\-Transformer, a deep reinforcement learning framework for joint replenishment under stochastic demand, with an item\-permutation\-equivariant Transformer architecture and pathwise\-gradient training through the inventory dynamics\. Across problem sizes up to 1,024 inventory items, OR\-Transformer increasingly outperforms learning\-based and rolling\-horizon MILP baselines as scale grows\. It also reduces online decision\-making time by over 4 million times relative to MILP solvers, enabling real\-time, large\-scale deep RL in supply chain operations\.

## 1Introduction

Inventory management is a fundamental decision problem in retail and supply chain operations, concerning when to place orders and how much of each item to order over time\. The economic consequences of these decisions can be substantial\. For example, IHL Group estimates that out\-of\-stocks and overstocks together cost retailers approximately $1\.7 trillion annually, corresponding to about 6\.2% of global retail sales\[[13](https://arxiv.org/html/2609.01933#bib.bib26)\]\.

Figure 1\.1:Scaling performance from 1 to 1,024 items\.OR\-Transformer increasingly outperforms learning\-based and rolling\-horizon MILP baselines as problem size grows; lower cost is better\.At large retailers, replenishment decisions are often coordinated across multiple products\. For example,\[[17](https://arxiv.org/html/2609.01933#bib.bib24)\]reports that large supermarket chains such as Wal\-Mart and Yonghui use joint replenishment strategies to reduce shipping costs\. In such settings, a shared fixed ordering cost links the decisions across products: each joint order incurs this cost, and the decision maker must choose which products to include and in what quantities\. This coupling gives rise to the*joint replenishment problem*\(JRP\), a classical problem in operations research \(OR\)\[[9](https://arxiv.org/html/2609.01933#bib.bib25),[14](https://arxiv.org/html/2609.01933#bib.bib23)\]\.

Scaling stochastic joint replenishment is difficult for both optimization and learning\. With more than 1,000 items\[[36](https://arxiv.org/html/2609.01933#bib.bib34)\], observation and action spaces in our setting exceed10410^\{4\}and10310^\{3\}dimensions, respectively, making rolling\-horizon stochastic MILPs prohibitively slow\[[34](https://arxiv.org/html/2609.01933#bib.bib29),[2](https://arxiv.org/html/2609.01933#bib.bib35)\]and RL credit assignment increasingly difficult\. Heterogeneous lead times and correlated demand introduce additional dependencies across time and items\[[27](https://arxiv.org/html/2609.01933#bib.bib32),[8](https://arxiv.org/html/2609.01933#bib.bib30)\]\. Moreover, the shared fixed ordering cost creates a discontinuity between zero and any positive order, challenging standard continuous\-action RL methods\[[30](https://arxiv.org/html/2609.01933#bib.bib21),[11](https://arxiv.org/html/2609.01933#bib.bib22),[1](https://arxiv.org/html/2609.01933#bib.bib20)\]\. Prior work has studied learning\-based joint replenishment, differentiable inventory control, and Transformer\-based inventory policies, but existing approaches either omit the shared fixed\-cost coupling considered here or study it at substantially smaller action dimensions\[[31](https://arxiv.org/html/2609.01933#bib.bib31),[27](https://arxiv.org/html/2609.01933#bib.bib32),[1](https://arxiv.org/html/2609.01933#bib.bib20),[16](https://arxiv.org/html/2609.01933#bib.bib33)\]\.

We introduceOR\-Transformer, a deep RL framework for stochastic joint replenishment\. It uses an item\-permutation\-equivariant Transformer to allow arbitrary reordering of inventory items, while self\-attention captures dependencies across item states\. The training method uses differentiable inventory dynamics to propagate pathwise gradients directly to continuous order quantities, enabling direct credit assignment in high\-dimensional action spaces\.

We evaluate OR\-Transformer against learning\-based baselines spanning multiple model architectures and training algorithms, as well as classical rolling\-horizon stochastic MILP baselines\[[34](https://arxiv.org/html/2609.01933#bib.bib29),[2](https://arxiv.org/html/2609.01933#bib.bib35)\]\. Our experiments demonstrate that OR\-Transformer increasingly outperforms the baselines as problem size grows; see Figure[1\.1](https://arxiv.org/html/2609.01933#S1.F1)\. At 1,024 inventory items, OR\-Transformer achieves approximately 0\.35M discounted cost, reducing cost by 74\.7%–90\.8% relative to the learning\-based baselines and by about 96% relative to rolling\-horizon MILP controllers despite allowing up to 10 minutes of optimization per decision\. In runtime comparisons, OR\-Transformer reduces online decision\-making time by more than 4 million times relative to MILP solvers, offering a practical path toward real\-time inventory control at large scale\.

## 2Problem Formulation

We consider a stochastic joint replenishment problem withnnitems\. Each itemiihas holding costhih\_\{i\}, backlog costbib\_\{i\}, lead timeLiL\_\{i\}, and maximum order quantityq¯i\\bar\{q\}\_\{i\}\. Placing any joint order incurs a shared fixed costKK, and unmet demand is backlogged\. LetDi,tD\_\{i,t\}denote the stochastic demand for itemiiat timett\. We model demands across items as correlated through an observed common stochastic factorFt∈ℝF\_\{t\}\\in\\mathbb\{R\}\. The full demand specification and additional formulation details are provided in Appendix[B](https://arxiv.org/html/2609.01933#A2)\.

Let𝐈t\\mathbf\{I\}\_\{t\}denote the vector of net inventories,𝐏t\\mathbf\{P\}\_\{t\}the outstanding orders scheduled to arrive in future time steps, andΘ\\Thetathe time\-invariant parameters defining the problem instance\. The state isst=\(𝐈t,𝐏t,Ft,Θ\)s\_\{t\}=\(\\mathbf\{I\}\_\{t\},\\mathbf\{P\}\_\{t\},F\_\{t\};\\Theta\)\. At each time stept∈ℕt\\in\\mathbb\{N\}, letYt∈\{0,1\}Y\_\{t\}\\in\\\{0,1\\\}indicate whether a joint order is placed and let𝐐t=\(Q1,t,…,Qn,t\)⊤\\mathbf\{Q\}\_\{t\}=\(Q\_\{1,t\},\\ldots,Q\_\{n,t\}\)^\{\\top\}denote the order quantities, with0≤Qi,t≤q¯i0\\leq Q\_\{i,t\}\\leq\\bar\{q\}\_\{i\}\. The action isat=\(Yt,𝐐t\)a\_\{t\}=\(Y\_\{t\},\\mathbf\{Q\}\_\{t\}\)\. Orders for itemiiarrive after lead timeLiL\_\{i\}, with outstanding orders tracked in𝐏t\\mathbf\{P\}\_\{t\}\.

Let\[x\]\+≐max⁡\{x,0\}\[x\]^\{\+\}\\doteq\\max\\\{x,0\\\}\. The one\-step cost isCt=K​Yt\+∑i=1n\(hi​\[Ii,t−Di,t\]\+\+bi​\[Di,t−Ii,t\]\+\)C\_\{t\}=KY\_\{t\}\+\\sum\_\{i=1\}^\{n\}\\bigl\(h\_\{i\}\[I\_\{i,t\}\-D\_\{i,t\}\]^\{\+\}\+b\_\{i\}\[D\_\{i,t\}\-I\_\{i,t\}\]^\{\+\}\\bigr\)\.

Letπ\\pidenote the replenishment policy andγ∈\(0,1\)\\gamma\\in\(0,1\)the discount factor\. The objective is to minimize the expected infinite\-horizon discounted costminπ⁡𝔼π​\[∑t=0∞γt​Ct\]\.\\min\_\{\\pi\}\\;\\mathbb\{E\}\_\{\\pi\}\\\!\\left\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}C\_\{t\}\\right\]\.

## 3Method

Figure 3\.1:OR\-Transformer architecture and training\.Independent Transformer networks produce the order\-opening decision, item quantities, and value estimate\. Score\-based gradients train the discrete opening decision, while pathwise gradients propagate through the inventory dynamics to train continuous quantities\.OR\-Transformer consists of an item\-permutation\-equivariant Transformer policy and a training procedure that propagates gradients through the inventory dynamics\. Figure[3\.1](https://arxiv.org/html/2609.01933#S3.F1)summarizes the architecture and gradient flow\. Full details are provided in Appendix[C](https://arxiv.org/html/2609.01933#A3)\.

##### Item\-permutation\-equivariant Transformer\.

The state is encoded into one global token and one token for each inventory item\. Three independent Transformer networks parameterize the order\-opening decision, the order quantities, and a critic that estimates expected discounted future cost\. No positional or item\-index embeddings are used, so arbitrarily reordering the inventory items leaves the global outputs unchanged and reorders the quantity outputs accordingly\. Self\-attention allows each item representation to incorporate information from other item states, capturing cross\-item dependencies\.

Letptp\_\{t\}denote the probability of opening an order andQi,tQ\_\{i,t\}the quantity for itemii\. The policy outputs

Yt∼Bernoulli⁡\(pt\),Qi,t=q¯i​sigmoid⁡\(𝐰Q⊤​𝐡i,tQ\+cQ\),Y\_\{t\}\\sim\\operatorname\{Bernoulli\}\(p\_\{t\}\),\\qquad Q\_\{i,t\}=\\bar\{q\}\_\{i\}\\,\\operatorname\{sigmoid\}\\\!\\left\(\\mathbf\{w\}\_\{Q\}^\{\\top\}\\mathbf\{h\}\_\{i,t\}^\{Q\}\+c\_\{Q\}\\right\),\(1\)where𝐡i,tQ\\mathbf\{h\}\_\{i,t\}^\{Q\}is the final Transformer representation of itemii,𝐰Q\\mathbf\{w\}\_\{Q\}is the learned weight vector of the shared quantity\-output head, andcQc\_\{Q\}is its learned scalar bias; and we recall thatq¯i\\bar\{q\}\_\{i\}is the maximum order quantity for itemii\.

Writingp⁡\(s\)p\(s\),𝐐⁡\(s\)\\mathbf\{Q\}\(s\), andV⁡\(s\)V\(s\)for the order\-opening probability, quantity vector, and value estimate, respectively, for any item\-permutation matrix𝐏\\mathbf\{P\},

p⁡\(𝐏​s\)=p⁡\(s\),V⁡\(𝐏​s\)=V⁡\(s\),𝐐⁡\(𝐏​s\)=𝐏𝐐⁡\(s\)\.p\(\\mathbf\{P\}s\)=p\(s\),\\qquad V\(\\mathbf\{P\}s\)=V\(s\),\\qquad\\mathbf\{Q\}\(\\mathbf\{P\}s\)=\\mathbf\{P\}\\mathbf\{Q\}\(s\)\.\(2\)Thus, the order\-opening probability and value estimate are permutation invariant, while the quantity vector is permutation equivariant\.

##### Pathwise training\.

The order\-opening decisionYtY\_\{t\}is discrete, whereas the quantities𝐐t\\mathbf\{Q\}\_\{t\}are continuous\. We trainYtY\_\{t\}with a score\-based policy gradient\. Conditional on sampled order openings and demands, the inventory dynamics are differentiable with respect to𝐐t\\mathbf\{Q\}\_\{t\}; we therefore backpropagate pathwise gradients through the resulting inventory trajectory and future costs\. Consequently, an order quantity at timettreceives gradients from later costs through its effects on future inventory and outstanding orders\. This yields more direct credit assignment for high\-dimensional quantity decisions than relying only on sampled returns\. A critic estimates expected future cost and provides a value estimate used during training\.

## 4Experiments

##### Experimental setting\.

We evaluate joint replenishment problems withn∈\{1,4,16,64,1,024\}n\\in\\\{1,4,16,64,1\{,\}024\\\}items under correlated stochastic demand, heterogeneous lead times, heterogeneous item costs, and a shared setup cost\. We compare OR\-Transformer with three learning baselines\. Transformer\-PPO uses the Transformer backbone with PPO policy gradients for the quantity decisions; HPO uses a two\-hidden\-layer multilayer perceptron \(MLP\) with pathwise gradients\[[1](https://arxiv.org/html/2609.01933#bib.bib20)\]; and PPO\[[30](https://arxiv.org/html/2609.01933#bib.bib21)\]uses a two\-hidden\-layer MLP with PPO policy gradients\. We also compare against rolling\-horizon stochastic MILPs solved by Gurobi\[[10](https://arxiv.org/html/2609.01933#bib.bib37)\]and HiGHS\[[12](https://arxiv.org/html/2609.01933#bib.bib36)\]\. All learned methods use the same number of optimization updates and the same number of samples per update\. Performance is evaluated on a held\-out set of 128 episodes using discounted inventory cost, where lower is better\. Full experimental specifications are provided in Appendix[D](https://arxiv.org/html/2609.01933#A4)\.

Figure 4\.1:OR\-Transformer versus Gurobi at 1,024 items\.With Gurobi allowed up to six hours of optimization per decision, OR\-Transformer achieves 19\.1% lower inventory cost and over4×1064\\times 10^\{6\}times faster online decision\-making on the same held\-out episodes\. Error bars show one standard error; lower is better\.
##### Scaling performance\.

Figure[1\.1](https://arxiv.org/html/2609.01933#S1.F1)shows that the baselines degrade at different scales as problem size grows\. At 4 items, PPO is the first method to diverge\. At 16 items, HPO also diverges, indicating that high\-dimensional joint replenishment remains challenging for an MLP policy even with pathwise gradients\. At 64 items, Transformer\-PPO diverges as well, indicating that PPO policy gradients become increasingly difficult for the high\-dimensional quantity decisions even with the Transformer architecture\. OR\-Transformer remains stable and scales to 1,024 items, where its cost falls below the rolling\-horizon MILP baselines\. At 1,024 items, over a 50\-decision evaluation horizon, OR\-Transformer achieves approximately 0\.35M discounted cost, compared with 1\.39M for the best learning baseline, a reduction of approximately 75%\. Complete cost comparisons across all problem sizes, including rolling\-horizon MILP results, are reported in Appendix[D\.3](https://arxiv.org/html/2609.01933#A4.SS3)\.

##### Online decision quality and computation\.

We further compare OR\-Transformer with rolling\-horizon Gurobi at 1,024 items over an eight\-decision evaluation horizon, while allowing Gurobi up tosix hoursof optimization for each replenishment decision\. As shown in Figure[4\.1](https://arxiv.org/html/2609.01933#S4.F1), OR\-Transformer achieves126\.61±4\.23126\.61\\pm 4\.23K discounted cost, compared with156\.58±4\.93156\.58\\pm 4\.93K for Gurobi, corresponding to a19\.1% cost reduction\. The OR\-Transformer decisions require 0\.0432 seconds in total, compared with 48 hours 49 minutes for Gurobi, making OR\-Transformer more thanfour million times fasterin online decision\-making\. Full comparisons are reported in Appendix[D\.2](https://arxiv.org/html/2609.01933#A4.SS2)\.

## 5Conclusion

We introducedOR\-Transformer, a deep RL framework for large\-scale stochastic joint replenishment with an item\-permutation\-equivariant Transformer architecture and pathwise\-gradient training through the inventory dynamics\. Across problem sizes up to 1,024 items, OR\-Transformer increasingly outperforms learning\-based and rolling\-horizon MILP baselines as scale grows, while reducing online decision\-making time by more than 4 million times relative to MILP solvers\. These results suggest a path toward efficient, real\-time decision\-making for large\-scale stochastic inventory systems\.

## Acknowledgments

David Simchi\-Levi was supported in part by the Purdue University Center for Data Science\. Shangtong Zhang was supported in part by the U\.S\. National Science Foundation under Awards III\-2128019, SLES\-2331904, and CAREER\-2442098; the Commonwealth Cyber Initiative’s Central Virginia Node under Award VV\-1Q26\-001; and a Cisco Faculty Research Award\.

## References

- \[1\]M\. Alvo, D\. Russo, and Y\. Kanoria\(2026\)Policy optimization in hybrid discrete\-continuous action spaces via mixed gradients\.arXiv preprint arXiv:2605\.14297\.Cited by:[§1](https://arxiv.org/html/2609.01933#S1.p3.1),[§4](https://arxiv.org/html/2609.01933#S4.SS0.SSS0.Px1.p1.1)\.
- \[2\]R\. Barati and Q\. V\. Hu\(2026\)Gym\-invmgmt: an open benchmarking framework for inventory management methods\.arXiv preprint arXiv:2605\.11355\.Cited by:[§1](https://arxiv.org/html/2609.01933#S1.p3.1),[§1](https://arxiv.org/html/2609.01933#S1.p5.1)\.
- \[3\]C\. Chen, S\. D\. Liu, L\. Luo, R\. Chandra, N\. Jiang, and S\. Zhang\(2026\)Robust data\-collection policy learning for low\-variance online policy evaluation\.arXiv preprint arXiv:2608\.24146\.Cited by:[§D\.1\.3](https://arxiv.org/html/2609.01933#A4.SS1.SSS3.p1.1)\.
- \[4\]C\. Chen, S\. Liu, and S\. Zhang\(2025\)Efficient policy evaluation with safety constraint for reinforcement learning\.InProceedings of the International Conference on Learning Representations,Cited by:[§D\.1\.3](https://arxiv.org/html/2609.01933#A4.SS1.SSS3.p1.1)\.
- \[5\]C\. Chen, Y\. Zhang, X\. Liu, Z\. Xie, S\. D\. Liu, and N\. Jiang\(2026\)Offline two\-player zero\-sum markov games with KL regularization\.InForty\-third International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=cQD2wxFhFG)Cited by:[§C\.3](https://arxiv.org/html/2609.01933#A3.SS3.p7.3)\.
- \[6\]C\. Chen and Y\. Zhang\(2026\)Fast rates inα\\alpha\-potential games via regularized mirror descent\.ArXiv Preprint arXiv:2605\.00268\.Cited by:[§C\.3](https://arxiv.org/html/2609.01933#A3.SS3.p7.3)\.
- \[7\]C\. Chen and Y\. Zhang\(2026\)Pessimism\-free offline learning in general\-sum games via KL regularization\.ArXiv Preprint arXiv:2605\.00264\.Cited by:[§C\.3](https://arxiv.org/html/2609.01933#A3.SS3.p7.3)\.
- \[8\]Y\. Chen, G\. Iyengar, and C\. Wang\(2023\)Robust inventory management: a cycle\-based approach\.Manufacturing and Service Operations Management \(M&SOM\)\.Cited by:[§1](https://arxiv.org/html/2609.01933#S1.p3.1)\.
- \[9\]S\. K\. Goyal\(1974\)Determination of optimum packaging frequency of items jointly replenished\.Management Science\.Cited by:[Appendix B](https://arxiv.org/html/2609.01933#A2.p2.1),[§1](https://arxiv.org/html/2609.01933#S1.p2.1)\.
- \[10\]Gurobi Optimization, LLC\(2026\)Gurobi Optimizer Reference Manual\.Note:[https://www\.gurobi\.com](https://www.gurobi.com/), self\-published by the vendor, no publication year intrinsic to the document; retrieved 2026\-08\-30Cited by:[§4](https://arxiv.org/html/2609.01933#S4.SS0.SSS0.Px1.p1.1)\.
- \[11\]T\. Haarnoja, A\. Zhou, P\. Abbeel, and S\. Levine\(2018\)Soft actor\-critic: off\-policy maximum entropy deep reinforcement learning with a stochastic actor\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§1](https://arxiv.org/html/2609.01933#S1.p3.1)\.
- \[12\]Q\. Huangfu and J\. A\. J\. Hall\(2018\)Parallelizing the dual revised simplex method\.Mathematical Programming Computation\.Cited by:[§4](https://arxiv.org/html/2609.01933#S4.SS0.SSS0.Px1.p1.1)\.
- \[13\]IHL Group\(2026\)The 2026 Inventory Distortion Study\.Note:[https://www\.ihlservices\.com/product/inventory\-distortion\-study\-2026/](https://www.ihlservices.com/product/inventory-distortion-study-2026/)Cited by:[§1](https://arxiv.org/html/2609.01933#S1.p1.1)\.
- \[14\]M\. Khouja and S\. Goyal\(2008\)A review of the joint replenishment problem literature: 1989–2005\.European Journal of Operational Research \(EJOR\)\.Cited by:[Appendix B](https://arxiv.org/html/2609.01933#A2.p2.1),[§1](https://arxiv.org/html/2609.01933#S1.p2.1)\.
- \[15\]J\. Lee, Y\. Lee, J\. Kim, A\. R\. Kosiorek, S\. Choi, and Y\. W\. Teh\(2019\)Set transformer: A framework for attention\-based permutation\-invariant neural networks\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§C\.2](https://arxiv.org/html/2609.01933#A3.SS2.p1.1)\.
- \[16\]M\. Liu, Y\. Bai, M\. Qi, and Z\. \(\. Shen\(2026\)Inventory management with transformer: automated decision making for order timing and quantity\.Service Science\.Cited by:[§1](https://arxiv.org/html/2609.01933#S1.p3.1)\.
- \[17\]S\. Liu, O\. Liu, and X\. Jiang\(2023\)An efficient algorithm for the joint replenishment problem with quantity discounts, minimum order quantity and transport capacity constraints\.Mathematics\.Cited by:[§1](https://arxiv.org/html/2609.01933#S1.p2.1)\.
- \[18\]S\. Liu, C\. Chen, and S\. Zhang\(2025\)Doubly optimal policy evaluation for reinforcement learning\.InProceedings of the International Conference on Learning Representations,Cited by:[§D\.1\.3](https://arxiv.org/html/2609.01933#A4.SS1.SSS3.p1.1)\.
- \[19\]S\. Liu, Y\. Chen, and S\. Zhang\(2025\)Efficient multi\-policy evaluation for reinforcement learning\.InProceedings of the AAAI Conference on Artificial Intelligence,Cited by:[§D\.1\.3](https://arxiv.org/html/2609.01933#A4.SS1.SSS3.p1.1)\.
- \[20\]S\. D\. Liu, C\. Chen, J\. Wang, and D\. Simchi\-Levi\(2026\)Pessimistic minimax learning for public\-private information games under unilateral coverage\.Note:ManuscriptCited by:[§C\.3](https://arxiv.org/html/2609.01933#A3.SS3.p7.3)\.
- \[21\]S\. D\. Liu, C\. Chen, J\. S\. Xiao, X\. Chen, and D\. Simchi\-Levi\(2026\)Strategic bargaining in multi\-buyer markets: reinforcement learning from verifiable rewards for LLM negotiations\.arXiv preprint arXiv:2607\.05863\.Cited by:[Appendix B](https://arxiv.org/html/2609.01933#A2.SS0.SSS0.Px5.p1.4)\.
- \[22\]S\. D\. Liu, C\. Chen, J\. S\. Xiao, L\. Lei, Y\. Zhang, Y\. Yue, and D\. Simchi\-Levi\(2026\)Instructing LLMs to negotiate using reinforcement learning with verifiable rewards\.arXiv preprint arXiv:2604\.09855\.Cited by:[Appendix B](https://arxiv.org/html/2609.01933#A2.SS0.SSS0.Px5.p1.4)\.
- \[23\]S\. D\. Liu, S\. Chen, and S\. Zhang\(2025\)The ode method for stochastic approximation and reinforcement learning with markovian noise\.Journal of Machine Learning Research26\(24\),pp\. 1–76\.Cited by:[§C\.3](https://arxiv.org/html/2609.01933#A3.SS3.p8.1)\.
- \[24\]S\. D\. Liu\(2025\)Efficient and robust policy evaluation for reinforcement learning\.Ph\.D\. Thesis,University of Virginia\.Cited by:[§D\.1\.3](https://arxiv.org/html/2609.01933#A4.SS1.SSS3.p1.1)\.
- \[25\]S\. Liu, W\. Shen, and H\. Xu\(2021\)Optimal pricing of information\.InProceedings of the 22nd ACM Conference on Economics and Computation,pp\. 693\.Cited by:[Appendix B](https://arxiv.org/html/2609.01933#A2.SS0.SSS0.Px1.p3.1)\.
- \[26\]S\. Liu and S\. Zhang\(2024\)Efficient policy evaluation with offline data informed behavior policy design\.InProceedings of the International Conference on Machine Learning,Cited by:[§D\.1\.3](https://arxiv.org/html/2609.01933#A4.SS1.SSS3.p1.1)\.
- \[27\]D\. Madeka, K\. Torkkola, C\. Eisenach, A\. Luo, D\. P\. Foster, and S\. M\. Kakade\(2022\)Deep inventory management\.arXiv preprint arXiv:2210\.03137\.Cited by:[§1](https://arxiv.org/html/2609.01933#S1.p3.1)\.
- \[28\]V\. Mahadevan, C\. Chen, S\. D\. Liu, and S\. Zhang\(2026\)Convergence of two\-timescale Markovian stochastic approximations with applications in reinforcement learning\.InProceedings of the 43rd International Conference on Machine Learning,Cited by:[§C\.3](https://arxiv.org/html/2609.01933#A3.SS3.p8.1)\.
- \[29\]J\. Schulman, N\. Heess, T\. Weber, and P\. Abbeel\(2015\)Gradient estimation using stochastic computation graphs\.InConference on Neural Information Processing Systems \(NeurIPS\),Cited by:[§C\.3](https://arxiv.org/html/2609.01933#A3.SS3.p1.1)\.
- \[30\]J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov\(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§C\.3](https://arxiv.org/html/2609.01933#A3.SS3.p5.2),[§1](https://arxiv.org/html/2609.01933#S1.p3.1),[§4](https://arxiv.org/html/2609.01933#S4.SS0.SSS0.Px1.p1.1)\.
- \[31\]N\. Vanvuchelen, J\. Gijsbrechts, and R\. Boute\(2020\)Use of proximal policy optimization for the joint replenishment problem\.Computers in Industry\.Cited by:[§1](https://arxiv.org/html/2609.01933#S1.p3.1)\.
- \[32\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, L\. Kaiser, and I\. Polosukhin\(2017\)Attention is all you need\.InConference on Neural Information Processing Systems \(NeurIPS\),Cited by:[§C\.2](https://arxiv.org/html/2609.01933#A3.SS2.p1.1)\.
- \[33\]J\. Wang, J\. Srinivasa, C\. Chen, S\. D\. Liu, A\. Payani, and S\. Zhang\(2026\)Predicting plasticity in deep continual learning: a theoretical perspective\.arXiv preprint arXiv:2605\.09044\.Cited by:[§D\.3](https://arxiv.org/html/2609.01933#A4.SS3.p2.1)\.
- \[34\]M\. Xiang, R\. Rossi, and S\. A\. Tarim\(2019\)An\(R,S\)\(R,S\)based heuristic model for the stochastic joint replenishment problem\.arXiv preprint arXiv:1902\.11025\.Cited by:[§1](https://arxiv.org/html/2609.01933#S1.p3.1),[§1](https://arxiv.org/html/2609.01933#S1.p5.1)\.
- \[35\]Z\. Xie, X\. Liu, C\. Chen, S\. D\. Liu, R\. Chandra, and S\. Zhang\(2026\)Beyond linear attention: Softmax transformers implement in\-context reinforcement learning\.arXiv preprint arXiv:2605\.07333\.Cited by:[§C\.2](https://arxiv.org/html/2609.01933#A3.SS2.p1.1)\.
- \[36\]C\. Zhang, X\. Wang, W\. Jiang, X\. Yang, S\. Wang, L\. Song, and J\. Bian\(2024\)Whittle index with multiple actions and state constraint for inventory management\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2609.01933#S1.p3.1)\.
- \[37\]Y\. Zhang, C\. Chen, and N\. Jiang\(2026\)Beyond pessimism: offline learning in kl\-regularized games\.arXiv preprint arXiv:2604\.06738\.Cited by:[§C\.3](https://arxiv.org/html/2609.01933#A3.SS3.p7.3)\.

## Appendix ANotation

CategorySymbolDefinitionSets and indicesℐ=\{1,…,n\}\\mathcal\{I\}=\\\{1,\\ldots,n\\\}Item set containingnnitems\.nnNumber of items\.i,ji,jItem index and dummy item index used in symmetric sums\.ttTime step\.mmTemporal lag used in generalized advantage estimation\.ℓ\\ellArrival offset associated with an in\-transit quantity\.LmaxL\_\{\\max\}Maximum admissible lead time\.B,eB,eNumber of parallel rollouts and rollout index\.TrollT\_\{\\mathrm\{roll\}\}Number of time steps in one training rollout\.Instance parametersKKCommon setup cost incurred when a joint order is opened\.γ\\gammaInfinite\-horizon discount factor\.hi,bih\_\{i\},b\_\{i\}Per\-unit holding cost and per\-unit backlog cost of itemii\.λi\\lambda\_\{i\}Baseline demand rate of itemii\.βi\\beta\_\{i\}Loading of itemiion the common market factor\.ρ\\rhoAutoregressive coefficient of the market factor\.LiL\_\{i\}Deterministic lead time of itemii\.q¯i\\bar\{q\}\_\{i\}Order\-quantity cap of itemii\.Θ\\ThetaCollection of fixed parameters defining one problem instance\.Demand and exogenous variablesFtF\_\{t\}Observed scalar market factor at time steptt\.εt\\varepsilon\_\{t\}Standard\-normal innovation of the market factor\.λi,t\\lambda\_\{i,t\}Conditional demand rate of itemiiat time steptt\.Di,tD\_\{i,t\}Realized demand of itemiiat time steptt\.StateIi,tI\_\{i,t\}Net inventory of itemiiimmediately before demand at time steptt; negative values represent backlog\.Pi,ℓ,tP\_\{i,\\ell,t\}In\-transit quantity of itemiithat becomes available at time stept\+ℓt\+\\ell\.𝐈t\\mathbf\{I\}\_\{t\}Net\-inventory vector at time steptt\.𝐏t\\mathbf\{P\}\_\{t\}Array collecting all in\-transit quantities at time steptt\.sts\_\{t\}Complete policy state at time steptt\.s0s\_\{0\}Initial state\.Action and dynamicsata\_\{t\}Hybrid replenishment action at time steptt\.YtY\_\{t\}Binary order\-opening decision\.Qi,t,𝐐tQ\_\{i,t\},\\mathbf\{Q\}\_\{t\}Proposed quantity of itemiiand proposed\-quantity vector\.Oi,tO\_\{i,t\}Executed quantity,Oi,t=Yt​Qi,tO\_\{i,t\}=Y\_\{t\}Q\_\{i,t\}\.Zi,tZ\_\{i,t\}Post\-demand net inventory,Zi,t=Ii,t−Di,tZ\_\{i,t\}=I\_\{i,t\}\-D\_\{i,t\}\.𝟏​\{⋅\}\\mathbf\{1\}\\\{\\cdot\\\}Indicator function\.\[x\]\+\[x\]^\{\+\}Positive part of scalarxx, equal tomax⁡\{x,0\}\\max\\\{x,0\\\}\.Cost and control objectiveCtC\_\{t\}One\-step cost\.π\\piStationary hybrid policy\.𝔼π\\mathbb\{E\}\_\{\\pi\}Expectation under policy randomization and the exogenous processes\.JΘ​\(π,s0\)J\_\{\\Theta\}\(\\pi;s\_\{0\}\)Expected infinite\-horizon discounted cost from initial states0s\_\{0\}\.Feature representationμi\\mu\_\{i\}Lead\-time\-plus\-one\-step demand scale,μi=λi​\(Li\+1\)\\mu\_\{i\}=\\lambda\_\{i\}\(L\_\{i\}\+1\)\.σi\\sigma\_\{i\}Item reference scale,σi=μi\\sigma\_\{i\}=\\sqrt\{\\mu\_\{i\}\}\.cscalec\_\{\\mathrm\{scale\}\}Symmetric cost scale used in token normalization\.σref\\sigma\_\{\\mathrm\{ref\}\}Symmetric demand scale used in token normalization\.𝐱i,t\\mathbf\{x\}\_\{i,t\}Eleven\-dimensional item token\.𝐠t\\mathbf\{g\}\_\{t\}Four\-dimensional global token\.Transformer architecturer∈\{Y,Q,V\}r\\in\\\{Y,Q,V\\\}Network label for the order\-opening network, quantity network, or critic\.ddToken width\.MMNumber of Transformer blocks\.HattnH\_\{\\mathrm\{attn\}\}Number of attention heads\.EGr,EIrE\_\{G\}^\{r\},E\_\{I\}^\{r\}Global\-token embedding and shared item\-token embedding of networkrr\.𝒯r\\mathcal\{T\}^\{r\}Transformer encoder of networkrr\.𝐡0,tr\\mathbf\{h\}\_\{0,t\}^\{r\}Final global representation produced by networkrr\.𝐇tr\\mathbf\{H\}\_\{t\}^\{r\}Matrix of final item representations produced by networkrr\.𝐡i,tr\\mathbf\{h\}\_\{i,t\}^\{r\}Final representation of itemiiproduced by networkrr\.𝐰Y,cY\\mathbf\{w\}\_\{Y\},c\_\{Y\}Output weight and bias of the order\-opening head\.𝐰Q,cQ\\mathbf\{w\}\_\{Q\},c\_\{Q\}Shared output weight and bias of the quantity head\.𝐰V,cV\\mathbf\{w\}\_\{V\},c\_\{V\}Output weight and bias of the critic head\.ϕ,κ,ψ\\phi,\\kappa,\\psiParameter collections of the order\-opening network, quantity network, and critic\.s,y,𝐪s,y,\\mathbf\{q\}Generic state, generic order\-opening value, and generic proposed\-quantity vector\.pϕ​\(s\),ptp\_\{\\phi\}\(s\),p\_\{t\}Order\-opening probability function and its value at time steptt\.Vψ​\(s\)V\_\{\\psi\}\(s\)Normalized value\-function estimate at statess\.πϕY​\(y∣s\)\\pi\_\{\\phi\}^\{Y\}\(y\\mid s\)Bernoulli order\-opening distribution\.fκQ​\(s\)f\_\{\\kappa\}^\{Q\}\(s\)Deterministic proposed\-quantity vector\.𝐯,δ𝐯\\mathbf\{v\},\\delta\_\{\\mathbf\{v\}\}Generic vector and point mass at that vector\.πϕ,κ\\pi\_\{\\phi,\\kappa\}Complete hybrid policy\.τ,𝐏τ\\tau,\\mathbf\{P\}\_\{\\tau\}Item permutation and its permutation matrix\.Trainingcrefc\_\{\\mathrm\{ref\}\}Fixed instance\-level cost normalizer\.C~t\\widetilde\{C\}\_\{t\}Normalized one\-step cost,Ct/crefC\_\{t\}/c\_\{\\mathrm\{ref\}\}\.λGAE\\lambda\_\{\\mathrm\{GAE\}\}Trace parameter for generalized advantage estimation\.δt\\delta\_\{t\}One\-step temporal\-difference residual in normalized\-cost units\.A^tC\\widehat\{A\}\_\{t\}^\{C\}Estimated cost advantage\.G^t\\widehat\{G\}\_\{t\}Bootstrapped target for the normalized value function\.sg⁡\(⋅\)\\operatorname\{sg\}\(\\cdot\)Stop\-gradient operator\.ψ¯\\bar\{\\psi\}Frozen critic parameters used in the update of the quantity network\.ℒY\\mathcal\{L\}\_\{Y\}score\-based loss of the order\-opening network\.ℒPW\\mathcal\{L\}\_\{\\mathrm\{PW\}\}Pathwise objective of the quantity network\.ℒV\\mathcal\{L\}\_\{V\}Critic regression loss\.ℋ\\mathcal\{H\}Shannon entropy of the Bernoulli order\-opening distribution\.αPW,αV,αH\\alpha\_\{\\mathrm\{PW\}\},\\alpha\_\{V\},\\alpha\_\{H\}Weights of the pathwise, critic, and entropy terms\.ℒ\\mathcal\{L\}Complete training objective\.
## Appendix BDetailed Problem Formulation

This appendix provides the detailed formulation of the stochastic JRP summarized in Section[2](https://arxiv.org/html/2609.01933#S2), including the demand process, state and action definitions, lead\-time dynamics, and discounted\-cost objective\.

We study a stochastic joint replenishment problem \(JRP\), in which a common setup cost couples the replenishment decisions of multiple items\[[9](https://arxiv.org/html/2609.01933#bib.bib25),[14](https://arxiv.org/html/2609.01933#bib.bib23)\]\. Letn∈ℕn\\in\\mathbb\{N\}be the number of items, and defineℐ=\{1,…,n\}\\mathcal\{I\}=\\\{1,\\ldots,n\\\}as the item set\. Leti∈ℐi\\in\\mathcal\{I\}denote an item, and lett∈ℕ0t\\in\\mathbb\{N\}\_\{0\}be a time step\. For itemii, denote its per\-unit holding cost, per\-unit backlog cost, baseline demand rate, loading on the common market factor, and order\-quantity cap byhi\>0h\_\{i\}\>0,bi\>0b\_\{i\}\>0,λi\>0\\lambda\_\{i\}\>0,βi≥0\\beta\_\{i\}\\geq 0, andq¯i\>0\\bar\{q\}\_\{i\}\>0, respectively\. DefineLmax∈ℕL\_\{\\max\}\\in\\mathbb\{N\}as the maximum admissible lead time andLi∈\{1,…,Lmax\}L\_\{i\}\\in\\\{1,\\ldots,L\_\{\\max\}\\\}as the deterministic lead time of itemii\. We defineK≥0K\\geq 0as the fixed joint setup cost incurred whenever an order is placed, regardless of the number of items assigned positive quantities\. We consider an infinite\-horizon planning model with a discount factorγ∈\(0,1\)\\gamma\\in\(0,1\)\. The system operates under backlogging, meaning any unmet demand is backlogged and fulfilled in subsequent periods rather than lost\.

##### Correlated demand\.

LetFt∈ℝF\_\{t\}\\in\\mathbb\{R\}denote the observed scalar market factor at time steptt, and letρ∈\[0,1\)\\rho\\in\[0,1\)be its autoregressive coefficient\. Denote the standard\-normal noise byεt∈ℝ\\varepsilon\_\{t\}\\in\\mathbb\{R\}and the initial market factor byF0F\_\{0\}\. The factor process satisfies

Ft\+1=ρ​Ft\+1−ρ2​εt\+1,F0,ε1,ε2,…​∼i\.i\.d\.​𝒩​\(0,1\)\.F\_\{t\+1\}=\\rho F\_\{t\}\+\\sqrt\{1\-\\rho^\{2\}\}\\,\\varepsilon\_\{t\+1\},\\qquad F\_\{0\},\\varepsilon\_\{1\},\\varepsilon\_\{2\},\\ldots\\overset\{\\mathrm\{i\.i\.d\.\}\}\{\\sim\}\\mathcal\{N\}\(0,1\)\.\(3\)
Letλi\\lambda\_\{i\}be the base demand rate of itemii\. At timett, conditional onFtF\_\{t\}, the realized demandDi,tD\_\{i,t\}for itemiisatisfy

Di,t\|Ft∼Poisson⁡\(λi,t\),\\displaystyle D\_\{i,t\}\\mid F\_\{t\}\\sim\\operatorname\{Poisson\}\(\\lambda\_\{i,t\}\),\(4\)where​λi,t:=λi​exp⁡\(βi​Ft−βi22\)\.\\displaystyle\\text\{where\}~\\lambda\_\{i,t\}:=\\lambda\_\{i\}\\exp\\\!\\left\(\\beta\_\{i\}F\_\{t\}\-\\frac\{\\beta\_\{i\}^\{2\}\}\{2\}\\right\)\.\(5\)We have𝔼⁡\[λi,t\]=λi\\mathbb\{E\}\[\\lambda\_\{i,t\}\]=\\lambda\_\{i\}\. The factor process and demand realizations are exogenous, i\.e\., their distributions do not depend on the action\. Here, the market factor is observed without cost; extending the model to endogenous acquisition or pricing of demand information would connect inventory control with information\-design problems\[[25](https://arxiv.org/html/2609.01933#bib.bib14)\]\.

##### State\.

LetIi,t∈ℝI\_\{i,t\}\\in\\mathbb\{R\}denote the net inventory of itemiiimmediately before demand at time steptt; a negative value represents backlog\. Letℓ∈\{1,…,Lmax−1\}\\ell\\in\\\{1,\\ldots,L\_\{\\max\}\-1\\\}be an arrival offset, and letPi,ℓ,t≥0P\_\{i,\\ell,t\}\\geq 0denote the in\-transit quantity of itemiithat becomes available at time stept\+ℓt\+\\ell\. Under this convention, the in\-transit quantity with arrival offset one becomes available only after demand and cost at the current time step have been processed\. Define𝐈t=\(I1,t,…,In,t\)⊤∈ℝn\\mathbf\{I\}\_\{t\}=\(I\_\{1,t\},\\ldots,I\_\{n,t\}\)^\{\\top\}\\in\\mathbb\{R\}^\{n\}as the net\-inventory vector and𝐏t=\(Pi,ℓ,t\)i∈ℐ,1≤ℓ<Lmax∈ℝ\+n×\(Lmax−1\)\\mathbf\{P\}\_\{t\}=\(P\_\{i,\\ell,t\}\)\_\{i\\in\\mathcal\{I\},\\,1\\leq\\ell<L\_\{\\max\}\}\\in\\mathbb\{R\}\_\{\+\}^\{n\\times\(L\_\{\\max\}\-1\)\}as the array collecting all in\-transit quantities\. We also define

Θ=\(K,γ,ρ,\{hi,bi,λi,βi,Li,q¯i\}i=1n\)\\Theta=\\left\(K,\\gamma,\\rho,\\\{h\_\{i\},b\_\{i\},\\lambda\_\{i\},\\beta\_\{i\},L\_\{i\},\\bar\{q\}\_\{i\}\\\}\_\{i=1\}^\{n\}\\right\)\(6\)as the collection of fixed parameters defining one problem instance, and

st=\(𝐈t,𝐏t,Ft,Θ\)s\_\{t\}=\(\\mathbf\{I\}\_\{t\},\\mathbf\{P\}\_\{t\},F\_\{t\};\\Theta\)\(7\)as the state observed at time steptt\. The parameters inΘ\\Thetaremain fixed along a trajectory but are included in the policy input so that one policy can act across heterogeneous instances\.

##### Action and within\-step timing\.

LetYt∈\{0,1\}Y\_\{t\}\\in\\\{0,1\\\}be the binary order\-opening decision at time steptt, letQi,t∈\[0,q¯i\]Q\_\{i,t\}\\in\[0,\\bar\{q\}\_\{i\}\]denote theproposedquantity of itemii, and define𝐐t=\(Q1,t,…,Qn,t\)⊤\\mathbf\{Q\}\_\{t\}=\(Q\_\{1,t\},\\ldots,Q\_\{n,t\}\)^\{\\top\}as the proposed\-quantity vector\. Denote the hybrid replenishment actionata\_\{t\}selected at the beginning of time stepttby

at=\(Yt,𝐐t\),Yt∈\{0,1\},0≤Qi,t≤q¯i\.a\_\{t\}=\(Y\_\{t\},\\mathbf\{Q\}\_\{t\}\),\\qquad Y\_\{t\}\\in\\\{0,1\\\},\\qquad 0\\leq Q\_\{i,t\}\\leq\\bar\{q\}\_\{i\}\.\(8\)We also defineOi,t=Yt​Qi,tO\_\{i,t\}=Y\_\{t\}Q\_\{i,t\}as theexecutedquantity of itemii\. After the action is selected, demand is realized\. LetZi,tZ\_\{i,t\}denote the post\-demand net inventory, defined by

Zi,t=Ii,t−Di,t\.Z\_\{i,t\}=I\_\{i,t\}\-D\_\{i,t\}\.\(9\)Holding, backlog, and setup costs are then assessed before any due in\-transit quantity becomes available\.

##### Inventory dynamics\.

RecallPi,ℓ,tP\_\{i,\\ell,t\}as the in\-transit quantity of itemiiat timettwhich is scheduled to become available at timet\+ℓt\+\\ell, and recallOi,tO\_\{i,t\}as the executed quantity\. Let𝟏​\{⋅\}\\mathbf\{1\}\\\{\\cdot\\\}be the indicator function\. We usePi,Lmax,t≡0P\_\{i,L\_\{\\max\},t\}\\equiv 0as a boundary convention\. After the current cost is assessed, the in\-transit quantity with arrival offset one becomes available, each remaining in\-transit quantity moves to the next smaller arrival offset, and the new order is assigned the arrival offset implied byLiL\_\{i\}\. The transition satisfies

Ii,t\+1\\displaystyle I\_\{i,t\+1\}=Zi,t\+Pi,1,t\+𝟏\{Li=1\}Oi,t,\\displaystyle=Z\_\{i,t\}\+P\_\{i,1,t\}\+\\mathbf\{1\}\\\{L\_\{i\}=1\\\}O\_\{i,t\},\(10\)Pi,ℓ,t\+1\\displaystyle P\_\{i,\\ell,t\+1\}=Pi,ℓ\+1,t\+𝟏\{ℓ=Li−1\}Oi,t,ℓ=1,…,Lmax−1\.\\displaystyle=P\_\{i,\\ell\+1,t\}\+\\mathbf\{1\}\\\{\\ell=L\_\{i\}\-1\\\}O\_\{i,t\},\\qquad\\ell=1,\\ldots,L\_\{\\max\}\-1\.\(11\)Consequently, an order placed at time stepttfirst becomes available for demand at time stept\+Lit\+L\_\{i\}\.

##### Cost and control objective\.

For any scalarx∈ℝx\\in\\mathbb\{R\}, let\[x\]\+=max⁡\{x,0\}\[x\]^\{\+\}=\\max\\\{x,0\\\}denote its positive part\. RecallZi,tZ\_\{i,t\}as the post\-demand net inventory\. DefineCtC\_\{t\}as the one\-step cost,

Ct=K​Yt\+∑i=1n\(hi​\[Zi,t\]\+\+bi​\[−Zi,t\]\+\)\.C\_\{t\}=KY\_\{t\}\+\\sum\_\{i=1\}^\{n\}\\left\(h\_\{i\}\[Z\_\{i,t\}\]^\{\+\}\+b\_\{i\}\[\-Z\_\{i,t\}\]^\{\+\}\\right\)\.\(12\)Letπ\\pibe a stationary hybrid policy that maps a state to a distribution over the binary order\-opening decision and a feasible proposed\-quantity vector, and lets0s\_\{0\}denote the initial state\. Denote expectation over policy randomization, market\-factor innovations, and demand realizations underπ\\piby𝔼π\\mathbb\{E\}\_\{\\pi\}\. We define

JΘ\(π;s0\):=𝔼π\[∑t=0∞γtCt\|s0,Θ\]J\_\{\\Theta\}\(\\pi;s\_\{0\}\):=\\mathbb\{E\}\_\{\\pi\}\\\!\\left\[\\left\.\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}C\_\{t\}\\right\|s\_\{0\},\\Theta\\right\]\(13\)as the expected infinite\-horizon discounted cost\. The control problem is

minπJΘ\(π;s0\)subject to \([3](https://arxiv.org/html/2609.01933#A2.E3)\)–\([11](https://arxiv.org/html/2609.01933#A2.E11)\)\.\\min\_\{\\pi\}\\;J\_\{\\Theta\}\(\\pi;s\_\{0\}\)\\quad\\text\{subject to \\eqref\{eq:factor\_process\}\-\-\\eqref\{eq:in\_transit\_transition\}\.\}\(14\)The resulting model combines a discontinuous global order\-opening decision, a high\-dimensional continuous quantity decision, correlated stochastic demand, and delayed item\-specific action effects\. More broadly, structured reinforcement\-learning problems also arise beyond inventory control, including automated negotiation and multi\-buyer bargaining\[[22](https://arxiv.org/html/2609.01933#bib.bib17),[21](https://arxiv.org/html/2609.01933#bib.bib15)\]\.

## Appendix CDetailed OR\-Transformer Method

This appendix provides additional details for OR\-Transformer, summarized in Section[3](https://arxiv.org/html/2609.01933#S3), including the state representation, the item\-permutation\-equivariant Transformer architecture, and the training objectives\. The architecture uses separate networks for the discrete order\-opening decision, the continuous order quantities, and the critic, allowing the two action components to be trained with different gradient estimators\.

### C\.1Symmetry\-preserving state representation

Recallℐ\\mathcal\{I\}as the item set andnnas the number of items\. Also recallIi,tI\_\{i,t\}as net inventory,Pi,ℓ,tP\_\{i,\\ell,t\}as an in\-transit quantity,λi\\lambda\_\{i\}as the baseline demand rate,βi\\beta\_\{i\}as the market\-factor loading,LiL\_\{i\}as the deterministic lead time,hih\_\{i\}andbib\_\{i\}as the holding and backlog costs, andq¯i\\bar\{q\}\_\{i\}as the order\-quantity cap of itemii\. Usej∈ℐj\\in\\mathcal\{I\}as a dummy item in symmetric averages\. Defineμi\\mu\_\{i\}as a lead\-time\-plus\-one\-step demand scale,σi\\sigma\_\{i\}as its square\-root reference scale,cscalec\_\{\\mathrm\{scale\}\}as the symmetric cost scale, andσref\\sigma\_\{\\mathrm\{ref\}\}as the symmetric demand scale:

μi=λi​\(Li\+1\),σi=μi,cscale=1n​∑j=1n\(hj\+bj\),σref=1n​∑j=1nσj\.\\mu\_\{i\}=\\lambda\_\{i\}\(L\_\{i\}\+1\),\\qquad\\sigma\_\{i\}=\\sqrt\{\\mu\_\{i\}\},\\qquad c\_\{\\mathrm\{scale\}\}=\\frac\{1\}\{n\}\\sum\_\{j=1\}^\{n\}\(h\_\{j\}\+b\_\{j\}\),\\qquad\\sigma\_\{\\mathrm\{ref\}\}=\\frac\{1\}\{n\}\\sum\_\{j=1\}^\{n\}\\sigma\_\{j\}\.\(15\)RecallLmaxL\_\{\\max\}as the maximum admissible lead time\. The reported architecture fixesLmax=4L\_\{\\max\}=4\. Let𝐱i,t∈ℝ11\\mathbf\{x\}\_\{i,t\}\\in\\mathbb\{R\}^\{11\}denote the item token of itemiiat time steptt:

𝐱i,t=\[Ii,t−μiσi,Pi,1,tσi,Pi,2,tσi,Pi,3,tσi,hicscale,bicscale,log⁡\(1\+λi\),λiσref,q¯iσi,βi,Li\]⊤\.\\mathbf\{x\}\_\{i,t\}=\\left\[\\frac\{I\_\{i,t\}\-\\mu\_\{i\}\}\{\\sigma\_\{i\}\},\\;\\frac\{P\_\{i,1,t\}\}\{\\sigma\_\{i\}\},\\;\\frac\{P\_\{i,2,t\}\}\{\\sigma\_\{i\}\},\\;\\frac\{P\_\{i,3,t\}\}\{\\sigma\_\{i\}\},\\;\\frac\{h\_\{i\}\}\{c\_\{\\mathrm\{scale\}\}\},\\;\\frac\{b\_\{i\}\}\{c\_\{\\mathrm\{scale\}\}\},\\;\\log\(1\+\\lambda\_\{i\}\),\\;\\frac\{\\sqrt\{\\lambda\_\{i\}\}\}\{\\sigma\_\{\\mathrm\{ref\}\}\},\\;\\frac\{\\bar\{q\}\_\{i\}\}\{\\sigma\_\{i\}\},\\;\\beta\_\{i\},\\;L\_\{i\}\\right\]^\{\\\!\\top\}\.\(16\)RecallKKas the common setup cost,γ\\gammaas the discount factor, andFtF\_\{t\}as the observed market factor\. We also define𝐠t∈ℝ4\\mathbf\{g\}\_\{t\}\\in\\mathbb\{R\}^\{4\}as the global token at time steptt:

𝐠t=\[Kcscale​σref,γ,1100​\(1−γ\),Ft\]⊤\.\\mathbf\{g\}\_\{t\}=\\left\[\\frac\{K\}\{c\_\{\\mathrm\{scale\}\}\\sigma\_\{\\mathrm\{ref\}\}\},\\;\\gamma,\\;\\frac\{1\}\{100\(1\-\\gamma\)\},\\;F\_\{t\}\\right\]^\{\\\!\\top\}\.\(17\)The numerical constant100100is a fixed feature\-scaling constant\. Every pooled normalizer in \([15](https://arxiv.org/html/2609.01933#A3.E15)\) is a symmetric function of the item set\. Consequently, permuting item indices permutes the item tokens and leaves the global token unchanged\.

### C\.2Item\-permutation\-compatible Transformer policy

We adapt full self\-attention\[[32](https://arxiv.org/html/2609.01933#bib.bib27),[35](https://arxiv.org/html/2609.01933#bib.bib13)\]to an unordered item set, following the permutation\-symmetric design principle of set attention models\[[15](https://arxiv.org/html/2609.01933#bib.bib28)\]\. Letr∈\{Y,Q,V\}r\\in\\\{Y,Q,V\\\}label the order\-opening network, quantity network, or critic, respectively, and letd∈ℕd\\in\\mathbb\{N\}be the token width\. For eachrr, denote its global\-token embedding, shared item\-token embedding, and Transformer encoder byEGr:ℝ4→ℝdE\_\{G\}^\{r\}:\\mathbb\{R\}^\{4\}\\to\\mathbb\{R\}^\{d\},EIr:ℝ11→ℝdE\_\{I\}^\{r\}:\\mathbb\{R\}^\{11\}\\to\\mathbb\{R\}^\{d\}, and𝒯r\\mathcal\{T\}^\{r\}, respectively\. Let𝐡0,tr∈ℝd\\mathbf\{h\}\_\{0,t\}^\{r\}\\in\\mathbb\{R\}^\{d\}be the final global representation and𝐇tr∈ℝn×d\\mathbf\{H\}\_\{t\}^\{r\}\\in\\mathbb\{R\}^\{n\\times d\}be the matrix of final item representations\. We write𝐡i,tr∈ℝd\\mathbf\{h\}\_\{i,t\}^\{r\}\\in\\mathbb\{R\}^\{d\}for rowiiof𝐇tr\\mathbf\{H\}\_\{t\}^\{r\}\. The encoder computation is

\[\(𝐡0,tr\)⊤𝐇tr\]=𝒯r​\(\[EGr​\(𝐠t\)⊤EIr​\(𝐱1,t\)⊤EIr​\(𝐱n,t\)⊤\]\),r∈\{Y,Q,V\}\.\\begin\{bmatrix\}\(\\mathbf\{h\}\_\{0,t\}^\{r\}\)^\{\\top\}\\\\ \\mathbf\{H\}\_\{t\}^\{r\}\\end\{bmatrix\}=\\mathcal\{T\}^\{r\}\\\!\\left\(\\begin\{bmatrix\}E\_\{G\}^\{r\}\(\\mathbf\{g\}\_\{t\}\)^\{\\top\}\\\\ E\_\{I\}^\{r\}\(\\mathbf\{x\}\_\{1,t\}\)^\{\\top\}\\\\ \\vdots\\\\ E\_\{I\}^\{r\}\(\\mathbf\{x\}\_\{n,t\}\)^\{\\top\}\\end\{bmatrix\}\\right\),\\qquad r\\in\\\{Y,Q,V\\\}\.\(18\)The three encoders have disjoint parameters\. They use full noncausal attention, no positional embeddings, no item\-index embeddings, and a mask only for padded item slots\. LetM=4M=4be the number of pre\-normalized Transformer blocks, and letHattn=8H\_\{\\mathrm\{attn\}\}=8be the number of attention heads in the reported large\-scale configuration\. The same configuration usesd=128d=128, a feed\-forward width of4​d4d, and zero dropout\.

Denote the complete parameter collections of the order\-opening network, quantity network, and critic byϕ\\phi,κ\\kappa, andψ\\psi, respectively\. Recallsts\_\{t\}as the state at time stepttandYtY\_\{t\}as the binary order\-opening decision\. Letssbe a generic state\. Let𝐰Y∈ℝd\\mathbf\{w\}\_\{Y\}\\in\\mathbb\{R\}^\{d\}andcY∈ℝc\_\{Y\}\\in\\mathbb\{R\}be the output weight and bias of the order\-opening head\. Definepϕ​\(s\)∈\(0,1\)p\_\{\\phi\}\(s\)\\in\(0,1\)as the order\-opening probability at statess, and writept=pϕ​\(st\)p\_\{t\}=p\_\{\\phi\}\(s\_\{t\}\)for its value at time steptt:

pt=sigmoid⁡\(𝐰Y⊤​𝐡0,tY\+cY\),Yt∼Bernoulli⁡\(pt\)\.p\_\{t\}=\\operatorname\{sigmoid\}\\\!\\left\(\\mathbf\{w\}\_\{Y\}^\{\\top\}\\mathbf\{h\}\_\{0,t\}^\{Y\}\+c\_\{Y\}\\right\),\\qquad Y\_\{t\}\\sim\\operatorname\{Bernoulli\}\(p\_\{t\}\)\.\(19\)RecallQi,tQ\_\{i,t\}as the proposed quantity andq¯i\\bar\{q\}\_\{i\}as its cap\. Let𝐰Q∈ℝd\\mathbf\{w\}\_\{Q\}\\in\\mathbb\{R\}^\{d\}andcQ∈ℝc\_\{Q\}\\in\\mathbb\{R\}denote the shared output weight and bias of the quantity head\. The deterministic quantity network outputs

Qi,t=q¯i​sigmoid⁡\(𝐰Q⊤​𝐡i,tQ\+cQ\)\.Q\_\{i,t\}=\\bar\{q\}\_\{i\}\\,\\operatorname\{sigmoid\}\\\!\\left\(\\mathbf\{w\}\_\{Q\}^\{\\top\}\\mathbf\{h\}\_\{i,t\}^\{Q\}\+c\_\{Q\}\\right\)\.\(20\)Let𝐰V∈ℝd\\mathbf\{w\}\_\{V\}\\in\\mathbb\{R\}^\{d\}andcV∈ℝc\_\{V\}\\in\\mathbb\{R\}be the output weight and bias of the critic head, and letVψ​\(st\)∈ℝV\_\{\\psi\}\(s\_\{t\}\)\\in\\mathbb\{R\}denote the critic output:

Vψ​\(st\)=𝐰V⊤​𝐡0,tV\+cV\.V\_\{\\psi\}\(s\_\{t\}\)=\\mathbf\{w\}\_\{V\}^\{\\top\}\\mathbf\{h\}\_\{0,t\}^\{V\}\+c\_\{V\}\.\(21\)
Lety∈\{0,1\}y\\in\\\{0,1\\\}be a generic order\-opening value, and let𝐪∈∏i=1n\[0,q¯i\]\\mathbf\{q\}\\in\\prod\_\{i=1\}^\{n\}\[0,\\bar\{q\}\_\{i\}\]denote a generic proposed\-quantity vector\. Denote the Bernoulli distribution produced by the order\-opening network byπϕY​\(y∣s\)\\pi\_\{\\phi\}^\{Y\}\(y\\mid s\)and the deterministic quantity vector produced by the quantity network byfκQ​\(s\)f\_\{\\kappa\}^\{Q\}\(s\)\. For a generic vector𝐯\\mathbf\{v\}, letδ𝐯\\delta\_\{\\mathbf\{v\}\}denote a point mass at𝐯\\mathbf\{v\}\. We also defineπϕ,κ\\pi\_\{\\phi,\\kappa\}as the complete hybrid policy, which factorizes as

πϕ,κ​\(y,𝐪∣s\)=πϕY​\(y∣s\)​δfκQ​\(s\)​\(𝐪\)\.\\pi\_\{\\phi,\\kappa\}\(y,\\mathbf\{q\}\\mid s\)=\\pi\_\{\\phi\}^\{Y\}\(y\\mid s\)\\,\\delta\_\{f\_\{\\kappa\}^\{Q\}\(s\)\}\(\\mathbf\{q\}\)\.\(22\)
Letτ\\taube an arbitrary permutation of the item indices, and denote its permutation matrix by𝐏τ∈\{0,1\}n×n\\mathbf\{P\}\_\{\\tau\}\\in\\\{0,1\\\}^\{n\\times n\}\. We use𝐏τ​s\\mathbf\{P\}\_\{\\tau\}sfor the state obtained by applying𝐏τ\\mathbf\{P\}\_\{\\tau\}to every item\-indexed component ofsswhile leaving global components unchanged\. The architecture satisfies

pϕ​\(𝐏τ​s\)=pϕ​\(s\),Vψ​\(𝐏τ​s\)=Vψ​\(s\),fκQ​\(𝐏τ​s\)=𝐏τ​fκQ​\(s\)\.p\_\{\\phi\}\(\\mathbf\{P\}\_\{\\tau\}s\)=p\_\{\\phi\}\(s\),\\qquad V\_\{\\psi\}\(\\mathbf\{P\}\_\{\\tau\}s\)=V\_\{\\psi\}\(s\),\\qquad f\_\{\\kappa\}^\{Q\}\(\\mathbf\{P\}\_\{\\tau\}s\)=\\mathbf\{P\}\_\{\\tau\}f\_\{\\kappa\}^\{Q\}\(s\)\.\(23\)Thus, the order\-opening probability andVψV\_\{\\psi\}are permutation invariant\. The proposed\-quantity vector is permutation equivariant\.

### C\.3Hybrid pathwise training

RecallYtY\_\{t\}as the discrete order\-opening decision and𝐐t\\mathbf\{Q\}\_\{t\}as the continuous proposed\-quantity vector\. The sampledYtY\_\{t\}prevents ordinary end\-to\-end differentiation\. Conditional on sampled order\-opening decisions and exogenous factor and demand realizations, however, the simulator is piecewise differentiable in the proposed quantities\. We therefore use a score\-based gradient for the order\-opening network and a pathwise gradient for the quantity network, treating each rollout as a stochastic computation graph\[[29](https://arxiv.org/html/2609.01933#bib.bib19)\]\.

RecallCtC\_\{t\}as the one\-step cost,KKas the setup cost,hih\_\{i\}andbib\_\{i\}as the holding and backlog costs, andσi\\sigma\_\{i\}as the item reference scale\. Definecref\>0c\_\{\\mathrm\{ref\}\}\>0as the fixed instance\-level cost normalizer andC~t\\widetilde\{C\}\_\{t\}as the normalized one\-step cost:

cref=K\+∑i=1n\(hi\+bi\)​σi,C~t=Ctcref\.c\_\{\\mathrm\{ref\}\}=K\+\\sum\_\{i=1\}^\{n\}\(h\_\{i\}\+b\_\{i\}\)\\sigma\_\{i\},\\qquad\\widetilde\{C\}\_\{t\}=\\frac\{C\_\{t\}\}\{c\_\{\\mathrm\{ref\}\}\}\.\(24\)The critic is trained to estimate discounted future cost in these normalized\-cost units\.

Recallγ\\gammaas the discount factor\. LetTroll∈ℕT\_\{\\mathrm\{roll\}\}\\in\\mathbb\{N\}be the number of time steps in one training rollout, and letm∈ℕ0m\\in\\mathbb\{N\}\_\{0\}be a temporal lag used in generalized advantage estimation\. LetλGAE∈\[0,1\]\\lambda\_\{\\mathrm\{GAE\}\}\\in\[0,1\]denote its trace parameter\. Defineδt\\delta\_\{t\}as the one\-step temporal\-difference residual in normalized\-cost units,A^tC\\widehat\{A\}\_\{t\}^\{C\}as the estimated cost advantage, andG^t\\widehat\{G\}\_\{t\}as the bootstrapped value target:

δt\\displaystyle\\delta\_\{t\}=C~t\+γ​Vψ​\(st\+1\)−Vψ​\(st\),\\displaystyle=\\widetilde\{C\}\_\{t\}\+\\gamma V\_\{\\psi\}\(s\_\{t\+1\}\)\-V\_\{\\psi\}\(s\_\{t\}\),\(25\)A^tC\\displaystyle\\widehat\{A\}\_\{t\}^\{C\}=∑m=0Troll−1−t\(γ​λGAE\)m​δt\+m,\\displaystyle=\\sum\_\{m=0\}^\{T\_\{\\mathrm\{roll\}\}\-1\-t\}\(\\gamma\\lambda\_\{\\mathrm\{GAE\}\}\)^\{m\}\\delta\_\{t\+m\},\(26\)G^t\\displaystyle\\widehat\{G\}\_\{t\}=A^tC\+Vψ​\(st\)\.\\displaystyle=\\widehat\{A\}\_\{t\}^\{C\}\+V\_\{\\psi\}\(s\_\{t\}\)\.\(27\)A positiveA^tC\\widehat\{A\}\_\{t\}^\{C\}indicates that the estimated cost\-to\-go exceeds the critic’s baseline atsts\_\{t\}\.

LetB∈ℕB\\in\\mathbb\{N\}be the number of parallel rollouts in one update, and lete∈\{1,…,B\}e\\in\\\{1,\\ldots,B\\\}index a rollout\. Before updating the order\-opening network, we standardize the cost advantages over all rollout–time pairs:

A^e,tC,std=A^e,tC−meane,t⁡\(A^e,tC\)stde,t⁡\(A^e,tC\)\+10−8\.\\widehat\{A\}\_\{e,t\}^\{C,\\mathrm\{std\}\}=\\frac\{\\widehat\{A\}\_\{e,t\}^\{C\}\-\\operatorname\{mean\}\_\{e,t\}\\\!\\left\(\\widehat\{A\}\_\{e,t\}^\{C\}\\right\)\}\{\\operatorname\{std\}\_\{e,t\}\\\!\\left\(\\widehat\{A\}\_\{e,t\}^\{C\}\\right\)\+10^\{\-8\}\}\.\(28\)Here, the mean and standard deviation are computed over allB​TrollBT\_\{\\mathrm\{roll\}\}rollout–time pairs\. The critic targetG^e,t\\widehat\{G\}\_\{e,t\}remains defined from the unstandardized advantage in \([27](https://arxiv.org/html/2609.01933#A3.E27)\)\.

For any differentiable quantityuu, letsg⁡\(u\)\\operatorname\{sg\}\(u\)denote the stop\-gradient operator, which preservesuuin the forward computation and assigns it zero derivative in the backward computation\. The order\-opening network is trained with the score\-based loss

ℒY​\(ϕ\)=1B​Troll​∑e=1B∑t=0Troll−1log⁡πϕY​\(Ye,t∣sg⁡\(se,t\)\)​sg⁡\(A^e,tC,std\)\.\\mathcal\{L\}\_\{Y\}\(\\phi\)=\\frac\{1\}\{BT\_\{\\mathrm\{roll\}\}\}\\sum\_\{e=1\}^\{B\}\\sum\_\{t=0\}^\{T\_\{\\mathrm\{roll\}\}\-1\}\\log\\pi\_\{\\phi\}^\{Y\}\\\!\\left\(Y\_\{e,t\}\\mid\\operatorname\{sg\}\(s\_\{e,t\}\)\\right\)\\operatorname\{sg\}\\\!\\left\(\\widehat\{A\}\_\{e,t\}^\{C,\\mathrm\{std\}\}\\right\)\.\(29\)This loss uses uniform weighting across time steps\. Because each rollout is used for a single on\-policy update, its gradient coincides with the initial actor gradient of the PPO surrogate\[[30](https://arxiv.org/html/2609.01933#bib.bib21)\]\.

For the quantity network, sampled order\-opening decisions, demand realizations, and factor innovations are fixed during differentiation, while gradients propagate through every proposed quantity, inventory transition, and future cost in the rollout\. Letψ¯=sg⁡\(ψ\)\\bar\{\\psi\}=\\operatorname\{sg\}\(\\psi\)be a frozen copy of the critic parameters used in the update of the quantity network\. Denote the pathwise objective of the quantity network byℒPW​\(κ\)\\mathcal\{L\}\_\{\\mathrm\{PW\}\}\(\\kappa\):

ℒPW​\(κ\)=1B​∑e=1B\[∑t=0Troll−1γt​C~e,t\+γTroll​Vψ¯​\(se,Troll\)\]\.\\mathcal\{L\}\_\{\\mathrm\{PW\}\}\(\\kappa\)=\\frac\{1\}\{B\}\\sum\_\{e=1\}^\{B\}\\left\[\\sum\_\{t=0\}^\{T\_\{\\mathrm\{roll\}\}\-1\}\\gamma^\{t\}\\widetilde\{C\}\_\{e,t\}\+\\gamma^\{T\_\{\\mathrm\{roll\}\}\}V\_\{\\bar\{\\psi\}\}\(s\_\{e,T\_\{\\mathrm\{roll\}\}\}\)\\right\]\.\(30\)The frozen terminal value approximates the normalized value beyond the truncated rollout while preserving gradients through the terminal state intoκ\\kappa\.

LetℒV​\(ψ\)\\mathcal\{L\}\_\{V\}\(\\psi\)denote the critic regression loss:

ℒV​\(ψ\)=12​B​Troll​∑e=1B∑t=0Troll−1\(Vψ​\(sg⁡\(se,t\)\)−sg⁡\(G^e,t\)\)2\.\\mathcal\{L\}\_\{V\}\(\\psi\)=\\frac\{1\}\{2BT\_\{\\mathrm\{roll\}\}\}\\sum\_\{e=1\}^\{B\}\\sum\_\{t=0\}^\{T\_\{\\mathrm\{roll\}\}\-1\}\\left\(V\_\{\\psi\}\(\\operatorname\{sg\}\(s\_\{e,t\}\)\)\-\\operatorname\{sg\}\(\\widehat\{G\}\_\{e,t\}\)\\right\)^\{2\}\.\(31\)Recallssas a generic state\. Letℋ\(πϕY\(⋅∣s\)\)\\mathcal\{H\}\(\\pi\_\{\\phi\}^\{Y\}\(\\cdot\\mid s\)\)denote the Shannon entropy of the Bernoulli order\-opening distribution at statess\. LetαPW≥0\\alpha\_\{\\mathrm\{PW\}\}\\geq 0,αV≥0\\alpha\_\{V\}\\geq 0, andαH≥0\\alpha\_\{H\}\\geq 0be the weights of the pathwise, critic, and entropy terms, respectively\. We defineℒ\\mathcal\{L\}as the complete training objective:

ℒ=ℒY\+αPWℒPW\+αVℒV−αHB​Troll∑e=1B∑t=0Troll−1ℋ\(πϕY\(⋅∣sg\(se,t\)\)\)\.\\mathcal\{L\}=\\mathcal\{L\}\_\{Y\}\+\\alpha\_\{\\mathrm\{PW\}\}\\mathcal\{L\}\_\{\\mathrm\{PW\}\}\+\\alpha\_\{V\}\\mathcal\{L\}\_\{V\}\-\\frac\{\\alpha\_\{H\}\}\{BT\_\{\\mathrm\{roll\}\}\}\\sum\_\{e=1\}^\{B\}\\sum\_\{t=0\}^\{T\_\{\\mathrm\{roll\}\}\-1\}\\mathcal\{H\}\\\!\\left\(\\pi\_\{\\phi\}^\{Y\}\(\\cdot\\mid\\operatorname\{sg\}\(s\_\{e,t\}\)\)\\right\)\.\(32\)The entropy term regularizes the order\-opening policy\. More broadly, regularization and pessimism have been used to control policy optimization in offline two\-player, general\-sum, and potential games\[[37](https://arxiv.org/html/2609.01933#bib.bib8),[5](https://arxiv.org/html/2609.01933#bib.bib9),[7](https://arxiv.org/html/2609.01933#bib.bib10),[6](https://arxiv.org/html/2609.01933#bib.bib11),[20](https://arxiv.org/html/2609.01933#bib.bib6)\]\.

After each update, the simulator trajectories continue, but the net\-inventory vector and the array containing the in\-transit quantities are detached from the previous computation graph\. This implements truncated backpropagation through time for a continuing, non\-episodic control process\[[23](https://arxiv.org/html/2609.01933#bib.bib18),[28](https://arxiv.org/html/2609.01933#bib.bib16)\]\.

## Appendix DExperimental Details

### D\.1Method, training, and baselines configuration

#### D\.1\.1Method and training configuration

OR\-Transformer uses three independent Transformer encoders: one for the order\-opening network, one for the quantity network, and one for the critic\. The discrete order\-opening decision is trained with a score\-based estimator, whereas gradients for the continuous quantity vector are propagated pathwise through the differentiable simulator\. Table[D\.1](https://arxiv.org/html/2609.01933#A4.T1)collects the settings needed to reproduce the reported model and training procedure\.

Table D\.1:OR\-Transformer architecture and training settings\.
#### D\.1\.2OR\-Transformer training algorithm

RecallBBas the number of parallel rollouts in one update andTrollT\_\{\\mathrm\{roll\}\}as the number of time steps in one training rollout \(Section[C\.3](https://arxiv.org/html/2609.01933#A3.SS3)\), and defineNupdateN\_\{\\mathrm\{update\}\}as the number of training updates\. LetStep⁡\(s,Y,𝐎\)\\operatorname\{Step\}\(s,Y,\\mathbf\{O\}\)denote one simulator transition, including exogenous sampling, cost evaluation by \([12](https://arxiv.org/html/2609.01933#A2.E12)\), normalization by \([24](https://arxiv.org/html/2609.01933#A3.E24)\), and the dynamics \([10](https://arxiv.org/html/2609.01933#A2.E10)\)–\([11](https://arxiv.org/html/2609.01933#A2.E11)\)\. During the pathwise update, sampled order\-opening decisions and exogenous realizations are treated as constants\.

Algorithm 1OR\-Transformer hybrid pathwise training1:

BB,

TrollT\_\{\\mathrm\{roll\}\},

NupdateN\_\{\\mathrm\{update\}\},

γ\\gamma,

λGAE\\lambda\_\{\\mathrm\{GAE\}\},

αPW\\alpha\_\{\\mathrm\{PW\}\},

αV\\alpha\_\{V\},

αH\\alpha\_\{H\}; initial parameters

\(ϕ,κ,ψ\)\(\\phi,\\kappa,\\psi\)
2:Initialize

\{se,0\}e=1B\\\{s\_\{e,0\}\\\}\_\{e=1\}^\{B\}
3:for

j=1,…,Nupdatej=1,\\ldots,N\_\{\\mathrm\{update\}\}do

4:for

t=0,…,Troll−1t=0,\\ldots,T\_\{\\mathrm\{roll\}\}\-1do

5:for all

e=1,…,Be=1,\\ldots,Bin paralleldo

6:

𝐐e,t←fκQ​\(se,t\)\\mathbf\{Q\}\_\{e,t\}\\leftarrow f\_\{\\kappa\}^\{Q\}\(s\_\{e,t\}\),

Ye,t∼πϕY\(⋅∣se,t\)Y\_\{e,t\}\\sim\\pi\_\{\\phi\}^\{Y\}\(\\cdot\\mid s\_\{e,t\}\)
7:

𝐎e,t←Ye,t​𝐐e,t\\mathbf\{O\}\_\{e,t\}\\leftarrow Y\_\{e,t\}\\mathbf\{Q\}\_\{e,t\}
8:

\(C~e,t,se,t\+1\)←Step⁡\(se,t,Ye,t,𝐎e,t\)\(\\widetilde\{C\}\_\{e,t\},s\_\{e,t\+1\}\)\\leftarrow\\operatorname\{Step\}\(s\_\{e,t\},Y\_\{e,t\},\\mathbf\{O\}\_\{e,t\}\)
9:endfor

10:endfor

11:Compute

\{A^e,tC,G^e,t\}e,t\\\{\\widehat\{A\}\_\{e,t\}^\{C\},\\widehat\{G\}\_\{e,t\}\\\}\_\{e,t\}by \([25](https://arxiv.org/html/2609.01933#A3.E25)\)–\([27](https://arxiv.org/html/2609.01933#A3.E27)\)

12:

ℒY←\\mathcal\{L\}\_\{Y\}\\leftarrowone\-update PPO surrogate of \([29](https://arxiv.org/html/2609.01933#A3.E29)\)

13:

ψ¯←sg⁡\(ψ\)\\bar\{\\psi\}\\leftarrow\\operatorname\{sg\}\(\\psi\); compute

ℒPW\\mathcal\{L\}\_\{\\mathrm\{PW\}\}and

ℒV\\mathcal\{L\}\_\{V\}by \([30](https://arxiv.org/html/2609.01933#A3.E30)\) and \([31](https://arxiv.org/html/2609.01933#A3.E31)\)

14:Form

ℒ\\mathcal\{L\}by \([32](https://arxiv.org/html/2609.01933#A3.E32)\)

15:Adam step on

\(ϕ,κ,ψ\)\(\\phi,\\kappa,\\psi\)with per\-network clipping

16:

se,0←Detach𝐈,𝐏⁡\(se,Troll\)s\_\{e,0\}\\leftarrow\\operatorname\{Detach\}\_\{\\mathbf\{I\},\\mathbf\{P\}\}\(s\_\{e,T\_\{\\mathrm\{roll\}\}\}\),

e=1,…,Be=1,\\ldots,B
17:endfor

#### D\.1\.3Baselines details

Table[D\.2](https://arxiv.org/html/2609.01933#A4.T2)lists the environment setup, the baseline architectures, and the training and evaluation protocol for Figure[1\.1](https://arxiv.org/html/2609.01933#S1.F1)\. All methods are evaluated on the same fixed held\-out set of 128 episodes, following the standard policy\-evaluation setting of estimating performance\[[18](https://arxiv.org/html/2609.01933#bib.bib3),[26](https://arxiv.org/html/2609.01933#bib.bib4),[19](https://arxiv.org/html/2609.01933#bib.bib5),[4](https://arxiv.org/html/2609.01933#bib.bib2),[24](https://arxiv.org/html/2609.01933#bib.bib12),[3](https://arxiv.org/html/2609.01933#bib.bib1)\]\.

Table D\.2:Environment setup and the baseline architectures for Figure[1\.1](https://arxiv.org/html/2609.01933#S1.F1)\.

### D\.2MILP solver setup

At each time step, Gurobi and HiGHS implement the same rolling\-horizon policy by solving the same mixed\-integer linear program \(MILP\)\. Recallℐ=\{1,…,n\}\\mathcal\{I\}=\\\{1,\\ldots,n\\\}as the item set,sts\_\{t\}as the observed state,Ii,tI\_\{i,t\}as the net inventory of itemiiimmediately before demand at time steptt, andPi,ℓ,tP\_\{i,\\ell,t\}as the in\-transit quantity that becomes available at time stept\+ℓt\+\\ell\. Also recallFtF\_\{t\}as the observed market factor;YtY\_\{t\}as the joint order\-opening decision;Oi,tO\_\{i,t\}as the executed quantity;KKas the common setup cost;hih\_\{i\}andbib\_\{i\}as the holding and backlog costs;LiL\_\{i\}andLmaxL\_\{\\max\}as the item lead time and maximum lead time;q¯i\\bar\{q\}\_\{i\}as the order cap; andγ\\gammaas the discount factor\.

##### Scenario construction\.

At time steptt, condition on the observed statests\_\{t\}\. DefineH∈ℕH\\in\\mathbb\{N\}as the MILP planning horizon and𝒰=\{0,…,H−1\}\\mathcal\{U\}=\\\{0,\\ldots,H\-1\\\}as the set of look\-ahead steps\. The local indexu∈𝒰u\\in\\mathcal\{U\}is reset to zero at each MILP solve, withu=0u=0representing the current decision\. LetΩ=\{1,…,S\}\\Omega=\\\{1,\\ldots,S\\\}denote the set ofSSsampled demand scenarios, and setpω=1/Sp\_\{\\omega\}=1/Sas the weight of scenarioω∈Ω\\omega\\in\\Omega\.

For each scenarioω\\omega, defineF0ω=FtF\_\{0\}^\{\\omega\}=F\_\{t\}as the observed market factor at the root\. Recallρ\\rhoas the factor autocorrelation,λi\\lambda\_\{i\}as the baseline demand rate of itemii, andβi\\beta\_\{i\}as its factor loading\. Letεu\+1ω∼𝒩⁡\(0,1\)\\varepsilon\_\{u\+1\}^\{\\omega\}\\sim\\mathcal\{N\}\(0,1\)be the factor innovation at look\-ahead stepu\+1u\+1, sampled independently across scenarios and look\-ahead steps\. The sampled factor path then follows

Fu\+1ω=ρFuω\+1−ρ2εu\+1ω,u=0,…,H−2\.F\_\{u\+1\}^\{\\omega\}=\\rho F\_\{u\}^\{\\omega\}\+\\sqrt\{1\-\\rho^\{2\}\}\\,\\varepsilon\_\{u\+1\}^\{\\omega\},\\qquad u=0,\\ldots,H\-2\.\(33\)Denote the sampled demand of itemiiat look\-ahead stepuuin scenarioω\\omegabyDi,uωD\_\{i,u\}^\{\\omega\}\. Conditional on each sampled factor path, item demands are independent and are generated from the same demand model as the environment:

Di,uω∣Fuω∼Poisson\(λiexp\(βiFuω−βi22\)\),i∈ℐ,u∈𝒰,ω∈Ω\.D\_\{i,u\}^\{\\omega\}\\mid F\_\{u\}^\{\\omega\}\\sim\\operatorname\{Poisson\}\\\!\\left\(\\lambda\_\{i\}\\exp\\\!\\left\(\\beta\_\{i\}F\_\{u\}^\{\\omega\}\-\\frac\{\\beta\_\{i\}^\{2\}\}\{2\}\\right\)\\right\),\\qquad i\\in\\mathcal\{I\},\\ u\\in\\mathcal\{U\},\\ \\omega\\in\\Omega\.\(34\)Once sampled,FuωF\_\{u\}^\{\\omega\}andDi,uωD\_\{i,u\}^\{\\omega\}are data, rather than decision variables, in the deterministic\-equivalent MILP\.

##### Decision variables\.

DefineIi,uω∈ℝI\_\{i,u\}^\{\\omega\}\\in\\mathbb\{R\}as the net inventory immediately before demand at look\-ahead stepuuin scenarioω\\omega\. LetYuω∈\{0,1\}Y\_\{u\}^\{\\omega\}\\in\\\{0,1\\\}denote the scenario\-ω\\omegaorder\-opening decision, and letOi,uω≥0O\_\{i,u\}^\{\\omega\}\\geq 0be the corresponding executed quantity\. We also defineRi,u\+,ω≥0R\_\{i,u\}^\{\+,\\omega\}\\geq 0andRi,u−,ω≥0R\_\{i,u\}^\{\-,\\omega\}\\geq 0as auxiliary variables for, respectively, on\-hand inventory and backlog after demand\. DefineY0∈\{0,1\}Y\_\{0\}\\in\\\{0,1\\\}andOi,0≥0O\_\{i,0\}\\geq 0as the shared root opening decision and shared root executed quantity\. Use the aliasesY0ω=Y0Y\_\{0\}^\{\\omega\}=Y\_\{0\}andOi,0ω=Oi,0O\_\{i,0\}^\{\\omega\}=O\_\{i,0\}for everyω∈Ω\\omega\\in\\Omega\. Future variablesYuωY\_\{u\}^\{\\omega\}andOi,uωO\_\{i,u\}^\{\\omega\}withu≥1u\\geq 1are scenario specific\.

##### Rolling horizon MILP\.

For notational convenience, extend the observed in\-transit state by settingPi,ℓ,t=0P\_\{i,\\ell,t\}=0wheneverℓ∉\{1,…,Lmax−1\}\\ell\\notin\\\{1,\\ldots,L\_\{\\max\}\-1\\\}, and setOi,vω=0O\_\{i,v\}^\{\\omega\}=0for every negative local indexv<0v<0\. DefineLmin=mini∈ℐ⁡LiL\_\{\\min\}=\\min\_\{i\\in\\mathcal\{I\}\}L\_\{i\}as the shortest item lead time\. The rolling\-horizon optimization problem is

min\\displaystyle\\min\\quad∑ω∈Ωpω​∑u=0H−1γu​\[K​Yuω\+∑i∈ℐ\(hi​Ri,u\+,ω\+bi​Ri,u−,ω\)\]\\displaystyle\\sum\_\{\\omega\\in\\Omega\}p\_\{\\omega\}\\sum\_\{u=0\}^\{H\-1\}\\gamma^\{u\}\\left\[KY\_\{u\}^\{\\omega\}\+\\sum\_\{i\\in\\mathcal\{I\}\}\\left\(h\_\{i\}R\_\{i,u\}^\{\+,\\omega\}\+b\_\{i\}R\_\{i,u\}^\{\-,\\omega\}\\right\)\\right\]\(35\)s\.t\.Ii,0ω=Ii,t,\\displaystyle I\_\{i,0\}^\{\\omega\}=I\_\{i,t\},i∈ℐ,ω∈Ω,\\displaystyle i\\in\\mathcal\{I\},\\ \\omega\\in\\Omega,\(36\)Ri,u\+,ω≥Ii,uω−Di,uω,\\displaystyle R\_\{i,u\}^\{\+,\\omega\}\\geq I\_\{i,u\}^\{\\omega\}\-D\_\{i,u\}^\{\\omega\},i∈ℐ,u∈𝒰,ω∈Ω,\\displaystyle i\\in\\mathcal\{I\},\\ u\\in\\mathcal\{U\},\\ \\omega\\in\\Omega,\(37\)Ri,u−,ω≥Di,uω−Ii,uω,\\displaystyle R\_\{i,u\}^\{\-,\\omega\}\\geq D\_\{i,u\}^\{\\omega\}\-I\_\{i,u\}^\{\\omega\},i∈ℐ,u∈𝒰,ω∈Ω,\\displaystyle i\\in\\mathcal\{I\},\\ u\\in\\mathcal\{U\},\\ \\omega\\in\\Omega,\(38\)Ii,u\+1ω=Ii,uω−Di,uω\+Pi,u\+1,t\+Oi,u\+1−Liω,\\displaystyle I\_\{i,u\+1\}^\{\\omega\}=I\_\{i,u\}^\{\\omega\}\-D\_\{i,u\}^\{\\omega\}\+P\_\{i,u\+1,t\}\+O\_\{i,u\+1\-L\_\{i\}\}^\{\\omega\},i∈ℐ,ω∈Ω,u=0,…,H−2,\\displaystyle\\begin\{aligned\} i&\\in\\mathcal\{I\},\\ \\omega\\in\\Omega,\\\\\[\-2\.0pt\] u&=0,\\ldots,H\-2,\\end\{aligned\}\(39\)0≤Oi,uω≤q¯i​Yuω,\\displaystyle 0\\leq O\_\{i,u\}^\{\\omega\}\\leq\\bar\{q\}\_\{i\}Y\_\{u\}^\{\\omega\},i∈ℐ,u∈𝒰,ω∈Ω,\\displaystyle i\\in\\mathcal\{I\},\\ u\\in\\mathcal\{U\},\\ \\omega\\in\\Omega,\(40\)Y0ω=Y0,\\displaystyle Y\_\{0\}^\{\\omega\}=Y\_\{0\},ω∈Ω,\\displaystyle\\omega\\in\\Omega,\(41\)Oi,0ω=Oi,0,\\displaystyle O\_\{i,0\}^\{\\omega\}=O\_\{i,0\},i∈ℐ,ω∈Ω,\\displaystyle i\\in\\mathcal\{I\},\\ \\omega\\in\\Omega,\(42\)Oi,uω=0,\\displaystyle O\_\{i,u\}^\{\\omega\}=0,i∈ℐ,ω∈Ω,u=H−Li,…,H−1,\\displaystyle\\begin\{aligned\} i&\\in\\mathcal\{I\},\\ \\omega\\in\\Omega,\\\\\[\-2\.0pt\] u&=H\-L\_\{i\},\\ldots,H\-1,\\end\{aligned\}\(43\)Yuω=0,\\displaystyle Y\_\{u\}^\{\\omega\}=0,ω∈Ω,u=H−Lmin,…,H−1,\\displaystyle\\omega\\in\\Omega,\\quad u=H\-L\_\{\\min\},\\ldots,H\-1,\(44\)Ii,uω∈ℝ,Ri,u\+,ω,Ri,u−,ω≥0,Yuω∈\{0,1\},\\displaystyle I\_\{i,u\}^\{\\omega\}\\in\\mathbb\{R\},\\quad R\_\{i,u\}^\{\+,\\omega\},R\_\{i,u\}^\{\-,\\omega\}\\geq 0,\\quad Y\_\{u\}^\{\\omega\}\\in\\\{0,1\\\},i∈ℐ,u∈𝒰,ω∈Ω\.\\displaystyle i\\in\\mathcal\{I\},\\ u\\in\\mathcal\{U\},\\ \\omega\\in\\Omega\.\(45\)
Constraints \([37](https://arxiv.org/html/2609.01933#A4.E37)\)–\([38](https://arxiv.org/html/2609.01933#A4.E38)\) represent the positive and negative parts of post\-demand net inventory exactly: becausehi\>0h\_\{i\}\>0andbi\>0b\_\{i\}\>0, both inequalities are tight at an optimum\. Equation \([39](https://arxiv.org/html/2609.01933#A4.E39)\) is the environment dynamics\. The termPi,u\+1,tP\_\{i,u\+1,t\}delivers an order in transit, whileOi,u\+1−LiωO\_\{i,u\+1\-L\_\{i\}\}^\{\\omega\}makes an order placed at look\-ahead stepu\+1−Liu\+1\-L\_\{i\}available after its item\-specific lead time\. Constraint \([40](https://arxiv.org/html/2609.01933#A4.E40)\) forces every executed quantity to zero when the common order is closed and retains the item cap when it is open\. Constraints \([41](https://arxiv.org/html/2609.01933#A4.E41)\)–\([42](https://arxiv.org/html/2609.01933#A4.E42)\) share the root action\. The implemented controller is feasible because it executes only the common root action\.

##### Notation used only in the MILP setup\.

For the operating procedure below, defineHdecH\_\{\\mathrm\{dec\}\}as the number of real time steps in an evaluation trajectory andτmax\\tau\_\{\\max\}as the solver time limit for one decision\. We also defineεMIP\\varepsilon\_\{\\mathrm\{MIP\}\}as the requested relative MILP gap\. Table[D\.3](https://arxiv.org/html/2609.01933#A4.T3)summarizes all local notation introduced in this subsection\.

Table D\.3:Notation introduced only for the rolling\-horizon MILP\.
##### Rolling\-horizon Algorithm\.

RecallHdecH\_\{\\mathrm\{dec\}\}as the evaluation horizon,τmax\\tau\_\{\\max\}as the per\-decision time limit, andεMIP\\varepsilon\_\{\\mathrm\{MIP\}\}as the requested relative gap\. At every real time step, the model is rebuilt from the realized state; scenario\-specific future decisions from the preceding solve are discarded\. Algorithm[2](https://arxiv.org/html/2609.01933#alg2)formalizes this operation\.

Algorithm 2Rolling\-horizon MILP controller1:Initial state

s0s\_\{0\}; scenario count

SS; planning horizon

HH; decision horizon

HdecH\_\{\\mathrm\{dec\}\}; solver time limit

τmax\\tau\_\{\\max\}; relative gap

εMIP\\varepsilon\_\{\\mathrm\{MIP\}\}
2:for

t=0,…,Hdec−1t=0,\\ldots,H\_\{\\mathrm\{dec\}\}\-1do

3:Observe

Ii,tI\_\{i,t\},

Pi,ℓ,tP\_\{i,\\ell,t\}, and

FtF\_\{t\}for all relevant items and arrival offsets\.

4:Sample

SSconditional factor\-and\-demand paths using \([33](https://arxiv.org/html/2609.01933#A4.E33)\)–\([34](https://arxiv.org/html/2609.01933#A4.E34)\)\.

5:Build \([35](https://arxiv.org/html/2609.01933#A4.E35)\)–\([45](https://arxiv.org/html/2609.01933#A4.E45)\) from the observed state and sampled paths\.

6:Supply the all\-zero order plan as a feasible initial solution\.

7:Solve the MILP\.

8:ifthe solver returns a feasible incumbentthen

9:Set

Yt=Y0⋆Y\_\{t\}=Y\_\{0\}^\{\\star\}and

Oi,t=Oi,0⋆O\_\{i,t\}=O\_\{i,0\}^\{\\star\}for every item

ii, using the incumbent root action\.

10:else

11:Define

Yt=0Y\_\{t\}=0and

Oi,t=0O\_\{i,t\}=0for every item

ii\.

12:endif

13:Execute only

\(Yt,\(Oi,t\)i∈ℐ\)\(Y\_\{t\},\(O\_\{i,t\}\)\_\{i\\in\\mathcal\{I\}\}\); discard all actions with

u≥1u\\geq 1\.

14:Observe realized demand, advance the inventory system by one real time step, and retain the resulting in\-transit quantities\.

15:endfor

Table D\.4:Rolling\-horizon MILP solver setup\.

### D\.3Full scaling results

Table[D\.5](https://arxiv.org/html/2609.01933#A4.T5)reports the complete cost comparison across problem sizes over a 50\-decision evaluation horizon\. The rolling\-horizon MILP baselines use the 10\-minute\-per\-decision solver budget specified in Table[D\.4](https://arxiv.org/html/2609.01933#A4.T4)\. HPO achieves the lowest cost at one and four items, while OR\-Transformer achieves the lowest cost from 16 items onward\. At 64 items, OR\-Transformer reduces cost by approximately 20% relative to Gurobi\. At 1,024 items, it reduces cost by approximately 75% relative to the best learned baseline and by 96% relative to Gurobi\.

Several learned baselines diverge as problem size increases \(Figure[1\.1](https://arxiv.org/html/2609.01933#S1.F1)\), highlighting the challenge of maintaining stable learning at scale\. Representation plasticity has been studied as one factor associated with learning stability in deep networks over extended training\[[33](https://arxiv.org/html/2609.01933#bib.bib7)\]\.

Table D\.5:Mean discounted cost over 50 decisions, with one standard error\. Lower is better; the best entry in each row is bold and the next best is underlined\. K and M denote thousands and millions\. The solver baselines use 100 demand scenarios, a 50\-period planning horizon, and a 10\-minute limit per replenishment decision\.Forn≤64n\\leq 64, Gurobi achieves lower discounted cost than HiGHS whenever the two solver results differ\. At 1,024 items, however, the 100\-scenario, 50\-period MILP contains approximately 20 million columns, and all 6,400 evaluated decisions for both solvers reach the 10\-minute time limit\. Both solvers return zero replenishment quantities at every decision, producing the same inventory trajectory and therefore the same reported cost\. Thus, the identical 1,024\-item solver results reflect the common no\-order trajectory under the time limit rather than convergence to a common optimized solution\.