SAT: Sequential Agent Tuning for Coordinator Free Plug and Play Multi-LLM Training with Monotonic Improvement Guarantees

# SAT: Sequential Agent Tuning for Coordinator‑Free Plug‑and‑Play Multi‑LLM Training with Monotonic Improvement Guarantees
Source: [https://arxiv.org/html/2605.05216](https://arxiv.org/html/2605.05216)
\\setcopyright

ifaamas\\acmConference\[AAMAS ’26\]Proc\. of the 25th International Conference on Autonomous Agents and Multiagent Systems \(AAMAS 2026\)May 25 – 29, 2026 Paphos, CyprusAuthor list: Yi Xie, Yangyang Xu, Yi Fan\(the work is not related to their position at Amazon Inc\), Bo Liu\\copyrightyear2026\\acmYear2026\\acmDOI\\acmPrice\\acmISBN\\affiliation\\institutionDepartment of Electrical & Computer Engineering, University of Arizona\\cityTucson, Arizona\\countryUSA\\affiliation\\institutionDepartment of Mathematical Sciences, Rensselaer Polytechnic Institute\\cityTroy, New York\\countryUSA\\affiliation\\institutionAmazon Web Services\\cityNew York\\countryUSA\\affiliation\\institutionDepartment of Electrical & Computer Engineering, University of Arizona\\cityTucson, Arizona\\countryUSA

###### Abstract\.

Large language models \(LLMs\) with a large number of parameters achieve strong performance but are often prohibitively expensive to deploy\. Recent work explores using teams of smaller, more efficient LLMs that collectively match or even outperform a single large model\. However, jointly updating multiple agents introduces compounding distribution shifts, making coordination and stability during training difficult\. We address this by introducing Sequential Agent Tuning \(SAT\), a coordinator\-free training paradigm\. SAT represents the team as a factorized policy and employs block\-coordinate updates over agents, enabling scalable, decentralized training without a central controller\. Specifically, we develop a sequence\-aware, on\-policy advantage estimator that conditions on the evolving team policy, coupled with per\-agent KL trust regions that isolate occupancy drift\. Theoretically, this framework provides two critical guarantees\. First, it ensures monotonic improvement, stabilizing the training process\. Second, it establishes provable plug\-and\-play invariance: any agent can be upgraded to a stronger model without retraining the rest of the team, with a formal guarantee that the performance bound improves\. Empirically, a team of three 4B agents \(12B total\) trained with SAT surpasses the much larger Qwen3\-32B on AIME24/25 benchmarks by 3\.9% on average\. We validate our plug\-and\-play theory by swapping in two 8B agents, which boosts the composite score by 10\.4%\. We provide code and appendix of proof at https://github\.com/Yydc/SAT\-AAMAS

###### Key words and phrases:

Multi\-Agent System, LLM, Reinforcement Learning

## 1\.Introduction

Large language models \(LLMs\) have become effective problem solvers across a wide range of domains\(openai2023gpt4;wei2022chain;wang2023self;yao2023treeofthoughts;shinn2023reflexion\)\. Despite their impressive capabilities, state\-of\-the\-art LLMs demand significant memory, computational, and energy resources, making them unavailable for resource\-limited scenarios\(patterson2021carbon;alizadeh2024flash\)\. This disconnect between potential and deployability motivates a key question:*Can teams of small, efficient models collectively achieve or surpass the performance of one well\-tuned large model?*

Recent research has investigated multi\-agent LLM systems by employing various coordination strategies, such as role assignment \(planner, solver\), hierarchical workflow, and ensemble refinement protocols\(wu2023autogen;li2023camel;yao2023treeofthoughts;wang2024moa;liu2023improving;liu2025grasp\)\. While these methods show promising empirical results, they are constrained by two main limitations\. First, rely on predefined role assignment, which enforces interaction dynamics and may constrain the potential of LLMs systems\(yi2025from\)\. Second, existing methods generally lack robust theoretical foundations that guarantee multi\-agent LLM systems achieve a strong ability, leaving open questions around why multi\-LLMs\(wang2024moa;shinn2023reflexion\)\.

We address these gaps through a theoretical analysis of sequential agent training with factorized product policies and per\-agent trust regions\. We establish three properties under a sequence\-aware, on\-policy advantage estimator that conditions on intermediate policies\. Sequence\-agnostic improvement states that stage\-wise bounds hold regardless of the agent update sequence chosen before each update within the stage\. Plug\-and\-play invariance ensures that agents can be upgraded via a stage\-0 KL projection without retraining others while preserving certificates\. Certificate tightening indicates that upgrades can increase surrogate values at fixed radii or achieve comparable gains with smaller radii, thereby reducing the cumulative penalty that scales with the sum of the square roots of the radii\. The main challenge, covariate shift arising from sequential updates, is addressed by evaluating the advantages under the current intermediate occupancy and constraining the per\-agent per\-state KL divergence; we also demonstrate that naive rollout reuse without such conditioning fails to guarantee improvement, whereas our approach yields monotonic bounds\.

Empirical evaluation across general reasoning, active reasoning and planning \(AutoLogi PlanBench \) demonstrates that SAT\-trained small\-model teams match or exceed large baselines under matched evaluation protocols\. These gains align with our order‑agnostic stage bounds and certificate tightening under per‑agent KL trust regions\. We summarize our main contributions as follows:

- •We present a theoretical framework for multi\-agent LLM finetuning with monotonic improvement bounds, sequence\-agnostic guarantees, and plug\-and\-play invariance \(Sec\.[4](https://arxiv.org/html/2605.05216#S4)\)\.
- •We instantiated the theoretical framework through Sequential Agent Tuning \(SAT\), a coordinator\-free training paradigm utilizing sequence\-aware block optimization \(Sec\.[5](https://arxiv.org/html/2605.05216#S5)\)\.
- •We demonstrate that SAT\-trained teams outperform strong baselines on 7 benchmarks, with further improvements achieved through plug\-and\-play paradigm upgrades \(Sec \.[6](https://arxiv.org/html/2605.05216#S6)\)\.

## 2\.Related Work

##### Multi\-agent LLM Systems and Orchestration\.

A line of work explores orchestrating multiple LLMs via roles, tool use, and multi\-turn protocols, often with an explicit judge or controller that assigns subtasks or aggregates opinions\(yao2022react\)\. Ensembling\-style procedures improve reliability by aggregating diverse completions before selection\(wang2023self\)\. Structured search over thoughts extends this idea with deliberate branching and pruning at the sequence level\(yao2023treeofthoughts\)\. Planning\-inspired search further guides generation with MCTS\-style rollouts that critique and refine candidates\(feng2023alphazero;xie2025haer\)\. In contrast, our method is coordinator\-free and employs stage\-wise sequential updates with per\-agent trust regions and an analysis\-driven surrogate, yielding provable joint\-stage improvement\.

##### Monotonic Policy Improvement and Trust Regions\.

Conservative Policy Iteration establishes monotonic improvement under carefully controlled policy updates\(kakade2002cpi\)\. Trust Region Policy Optimization enforces KL trust regions to stabilize updates with theoretical guarantees\(schulman2015trpo\)\. Proximal Policy Optimization adopts a clipped surrogate that approximates trust\-region behavior in practice\(schulman2017proximal\)\. And some early works applying BCD to RL includes\(zhang2021mean;xie2018block\)\. Our development departs by updating agents sequentially while conditioning analysis on the intermediate product policy, and by providing a joint\-stage lower bound where per\-agent KL penalties accumulate as∑i=1nδi\\sum\_\{i=1\}^\{n\}\\sqrt\{\\delta\_\{i\}\}, justifying small per\-agent trust regions\.

##### Off\-Policy Advantage Estimation under Distribution Shift\.

Generalized Advantage Estimation \(GAE\) trades bias for variance reduction by utilizing multi\-step returns and a value baseline\(schulman2016gae;liu2023improving\)\. Retrace introduces safe multi\-step targets via truncated importance weights for off\-policy data\(munos2016safe;xie2025acorn\)\. V\-trace extends this idea to scalable actor–critic training with robust corrections\(espeholt2018impala\)\. In LLM pipelines, iterative procedures such as self\-consistency can shift the data distribution across rounds\(wang2023self\)\. Sequence\-level search similarly modifies the sampling distribution as the tree expands and prunes candidates\(yao2023treeofthoughts\)\. Our estimator conditions on the updated intermediate policy and uses clipped multi\-step ratios to manage bias, leveraging sequence information gathered within a stage\.

##### Dynamic Routing and Plug\-and\-Play Compute\.

Cascaded inference routes easier inputs to cheaper models and escalates only when necessary, reducing latency without sacrificing accuracy\(leviathan2023fast\)\. Previous multi\-agent frameworks may swap tools or models, but they rarely provide guarantees that persist after replacement\. Our plug\-and\-play analysis shows that monotonic improvement certificates remain invariant under agent replacement, provided that the surrogate and trust\-region constraints are respected\.

## 3\.Preliminaries

##### Environment and policies\.

Letℳ=\(𝒮,\{𝒜i\}i=1n,P,r,γ\)\\mathcal\{M\}=\(\\mathcal\{S\},\\\{\\mathcal\{A\}\_\{i\}\\\}\_\{i=1\}^\{n\},P,r,\\gamma\)be a discounted MDP withγ∈\(0,1\)\\gamma\\in\(0,1\)and bounded rewards\|r\|≤Rmax\|r\|\\leq R\_\{\\max\}\. The joint action space is the product𝒜=𝒜1×⋯×𝒜n\\mathcal\{A\}=\\mathcal\{A\}\_\{1\}\\times\\cdots\\times\\mathcal\{A\}\_\{n\}\. A team ofnnexecution agents forms a factorized policy over*joint actions*

π​\(a∣s\)=∏i=1nπ\(i\)​\(ai∣s\),a=\(a1,…,an\)∈𝒜\.\\pi\(a\\mid s\)=\\prod\_\{i=1\}^\{n\}\\pi^\{\(i\)\}\(a\_\{i\}\\mid s\),\\qquad a=\(a\_\{1\},\\ldots,a\_\{n\}\)\\in\\mathcal\{A\}\.Thusπ\(⋅∣s\)\\pi\(\\cdot\\mid s\)is a valid distribution on𝒜\\mathcal\{A\}, notnnagents “taking the same action\.” Letdπd^\{\\pi\}denote the discounted state visitation measure:

J​\(π\)=𝔼μ,s0∼μ,π​\[∑t≥0γt​rt\]=11−γ​𝔼s∼dπ,a∼π\(⋅∣s\)​\[r​\(s,a\)\]\.J\(\\pi\)=\\mathbb\{E\}\_\{\\mu,s\_\{0\}\\sim\\mu,\\,\\pi\}\\Big\[\\sum\_\{t\\geq 0\}\\gamma^\{t\}r\_\{t\}\\Big\]=\\frac\{1\}\{1\-\\gamma\}\\;\\mathbb\{E\}\_\{s\\sim d^\{\\pi\},\\,a\\sim\\pi\(\\cdot\\mid s\)\}\[\\,r\(s,a\)\\,\]\.In practice, some tasks activate only a subset of agents at a given state \(for example, token heads, tool routers, or role\-switching schedulers\)\. To faithfully cover both simultaneous and interleaved execution while preserving the product\-policy analysis, we adopt a masked activation view: at each statess, letℐ​\(s\)⊆\{1,…,n\}\\mathcal\{I\}\(s\)\\subseteq\\\{1,\\ldots,n\\\}denote the set of active agents and interpret the joint policy over active heads asπ​\(a∣s\)=∏i∈ℐ​\(s\)π\(i\)​\(ai∣s\)\\pi\(a\\mid s\)=\\prod\_\{i\\in\\mathcal\{I\}\(s\)\}\\pi^\{\(i\)\}\(a\_\{i\}\\mid s\), while inactive heads take a fixed no\-op\. All per\-state divergences and trust\-region constraints below are evaluated over the active heads at that state; when all agents act simultaneously,ℐ​\(s\)=\{1,…,n\}\\mathcal\{I\}\(s\)=\\\{1,\\ldots,n\\\}and the definitions reduce to the standard full\-product case\. This formalization of masking is consistent with our later sequence\-level objective and does not alter the statements that only depend on adjacent intermediate policies\. If an implementation never uses masked activation, settingℐ​\(s\)\\mathcal\{I\}\(s\)to the full set recovers the original expressions\.

We writeQπ,VπQ^\{\\pi\},V^\{\\pi\}for action/value functions andAπ​\(s,a\)=Qπ​\(s,a\)−Vπ​\(s\)A^\{\\pi\}\(s,a\)=Q^\{\\pi\}\(s,a\)\-V^\{\\pi\}\(s\)\. We update agents sequentially in orderσ​\(1\),…,σ​\(n\)\\sigma\(1\),\\ldots,\\sigma\(n\):

π^0=πcur,π^i=\(πtarσ​\(1\),…,πtarσ​\(i\),πcurσ​\(i\+1\),…,πcurσ​\(n\)\),π¯=π^n\.\\hat\{\\pi\}^\{0\}=\\pi\_\{\\mathrm\{cur\}\},\\;\\hat\{\\pi\}^\{i\}=\(\\pi^\{\\sigma\(1\)\}\_\{\\mathrm\{tar\}\},\\ldots,\\pi^\{\\sigma\(i\)\}\_\{\\mathrm\{tar\}\},\\pi^\{\\sigma\(i\+1\)\}\_\{\\mathrm\{cur\}\},\\ldots,\\pi^\{\\sigma\(n\)\}\_\{\\mathrm\{cur\}\}\),\\;\\bar\{\\pi\}=\\hat\{\\pi\}^\{n\}\.

##### Per\-state divergences and trust regions\.

For policiesπ1,π2\\pi\_\{1\},\\pi\_\{2\}define the per\-state maximal divergences

DKLmax​\(π1∥π2\)\\displaystyle D\_\{\\mathrm\{KL\}\}^\{\\max\}\(\\pi\_\{1\}\\\|\\pi\_\{2\}\)=supsDKL\(π1\(⋅∣s\)∥π2\(⋅∣s\)\),\\displaystyle=\\sup\_\{s\}D\_\{\\mathrm\{KL\}\}\\\!\\big\(\\pi\_\{1\}\(\\cdot\\mid s\)\\\|\\pi\_\{2\}\(\\cdot\\mid s\)\\big\),DTVmax​\(π1∥π2\)\\displaystyle D\_\{\\mathrm\{TV\}\}^\{\\max\}\(\\pi\_\{1\}\\\|\\pi\_\{2\}\)=sups12∥π1\(⋅∣s\)−π2\(⋅∣s\)∥1\.\\displaystyle=\\sup\_\{s\}\\tfrac\{1\}\{2\}\\\|\\pi\_\{1\}\(\\cdot\\mid s\)\-\\pi\_\{2\}\(\\cdot\\mid s\)\\\|\_\{1\}\.and use Pinsker’s inequality to link them:DTVmax≤12​DKLmaxD\_\{\\mathrm\{TV\}\}^\{\\max\}\\leq\\sqrt\{\\tfrac\{1\}\{2\}\\,D\_\{\\mathrm\{KL\}\}^\{\\max\}\}\. At stepii, agentσ​\(i\)\\sigma\(i\)obeys the per\-state trust region

DKLmax​\(πtarσ​\(i\)∥πcurσ​\(i\)\)≤δi\(uniform case:​δi≡δ\)\.D\_\{\\mathrm\{KL\}\}^\{\\max\}\\\!\\big\(\\pi^\{\\sigma\(i\)\}\_\{\\mathrm\{tar\}\}\\\|\\pi^\{\\sigma\(i\)\}\_\{\\mathrm\{cur\}\}\\big\)\\leq\\delta\_\{i\}\\quad\(\\text\{uniform case: \}\\delta\_\{i\}\\equiv\\delta\)\.We also defineAmax≔sups,a\|Aπ​\(s,a\)\|≤2​Rmax1−γA\_\{\\max\}\\coloneqq\\sup\_\{s,a\}\|A^\{\\pi\}\(s,a\)\|\\leq\\tfrac\{2R\_\{\\max\}\}\{1\-\\gamma\}for later bounds\.

##### Sequence\-aware surrogate objective\.

When updating agentσ​\(i\)\\sigma\(i\)we evaluate on\-policy under the current intermediate occupancy:

Liseq​\(πtarσ​\(i\)\)=11−γ​𝔼s∼dπ^i−1,a∼π^i\(⋅∣s\)​\[A^oni−1​\(s,a\)\],L\_\{i\}^\{\\textsc\{seq\}\}\(\\pi^\{\\sigma\(i\)\}\_\{\\mathrm\{tar\}\}\)=\\frac\{1\}\{1\-\\gamma\}\\;\\mathbb\{E\}\_\{s\\sim d^\{\\hat\{\\pi\}^\{\\,i\-1\}\},\\,a\\sim\\hat\{\\pi\}^\{\\,i\}\(\\cdot\\mid s\)\}\\big\[\\widehat\{A\}^\{\\,i\-1\}\_\{\\textsc\{on\}\}\(s,a\)\\big\],whereA^oni−1\\widehat\{A\}^\{\\,i\-1\}\_\{\\textsc\{on\}\}is a multi\-step on\-policy estimator computed from trajectories ofπ^i−1\\hat\{\\pi\}^\{\\,i\-1\}\(for example, GAE with traceλ\\lambda\)\. This evaluates full episodes under the intermediate policy and is consistent with masked activation, as interpreted by considering the joint action as the product of the active heads at each visited state\.

##### Two standard tools\.

The performance difference lemma states thatJ​\(π′\)−J​\(π\)=11−γ​𝔼s∼dπ′,a∼π′​\[Aπ​\(s,a\)\]J\(\\pi^\{\\prime\}\)\-J\(\\pi\)=\\tfrac\{1\}\{1\-\\gamma\}\\,\\mathbb\{E\}\_\{s\\sim d^\{\\pi^\{\\prime\}\},\\,a\\sim\\pi^\{\\prime\}\}\[A^\{\\pi\}\(s,a\)\]\. An occupancy\-shift bound for bounded test functionsffis given by\|𝔼dπ′​\[f\]−𝔼dπ​\[f\]\|≤2​γ1−γ​DTVmax​\(π′∥π\)​‖f‖∞\\big\|\\mathbb\{E\}\_\{d^\{\\pi^\{\\prime\}\}\}\[f\]\-\\mathbb\{E\}\_\{d^\{\\pi\}\}\[f\]\\big\|\\leq\\tfrac\{2\\gamma\}\{1\-\\gamma\}\\,D\_\{\\mathrm\{TV\}\}^\{\\max\}\(\\pi^\{\\prime\}\\\|\\pi\)\\,\\\|f\\\|\_\{\\infty\}\.

## 4\.Theoretical Framework

We study a coordinator\-free, plug\-and\-play training paradigm in which a team ofnnexecution agents is updated sequentially via block\-coordinate ascent\. The main technical challenge is distribution shift: when one agent updates, the effective evaluation distribution for subsequent agents changes\. We address this issue with a sequence\-aware, on\-policy advantage estimator that conditions on the current intermediate team policy, along with per\-agent\. These per\-state KL trust regions limit occupancy drift\. These ingredients yield single\-step and joint\-stage monotonic improvement bounds, sequence\-agnostic guarantees compatible with learned schedulers and plug\-ins, and information\-theoretic envelopes that quantify what is achievable under a fixed sampling budget\. Full proofs are presented in the Appendix, with a table that maps the results\.

### 4\.1\.Main Theoretical Guarantee

Statement \(one stage, sequence\-agnostic, plug\-and\-play\)\.For a full training stage ofnnsequential agent updates with per\-agent, per\-state KL radii\{δi\}i=1n\\\{\\delta\_\{i\}\\\}\_\{i=1\}^\{n\}and on\-policy budgets\{Ni\}i=1n\\\{N\_\{i\}\\\}\_\{i=1\}^\{n\}, we provide a high\-probability certificate of monotonic improvement\. In particular, with probability at least1−δconf1\-\\delta\_\{\\mathrm\{conf\}\},

J​\(π¯\)−J​\(πcur\)≥\\displaystyle J\(\\bar\{\\pi\}\)\-J\(\\pi\_\{\\mathrm\{cur\}\}\)\\ \\geq∑i=1n\(κireg​δi−aireg​δi\)⏟Information\-Geometric Gain−2​γ\(1−γ\)2​Amax​∑i=1n12​δi⏟Occupancy\-Shift Penalty\\displaystyle\\ \\underbrace\{\\sum\_\{i=1\}^\{n\}\\Big\(\\kappa\_\{i\}^\{\\mathrm\{reg\}\}\\sqrt\{\\delta\_\{i\}\}\-a\_\{i\}^\{\\mathrm\{reg\}\}\\delta\_\{i\}\\Big\)\}\_\{\\text\{Information\-Geometric Gain\}\}\-\\underbrace\{\\frac\{2\\gamma\}\{\(1\-\\gamma\)^\{2\}\}A\_\{\\max\}\\sum\_\{i=1\}^\{n\}\\sqrt\{\\tfrac\{1\}\{2\}\\,\\delta\_\{i\}\}\}\_\{\\text\{Occupancy\-Shift Penalty\}\}−11−γ​∑i=1nζi⏟Estimator\-Bias Penalty−∑i=1nAmax1−γ​log⁡\(2​n/δconf\)2​Ni⏟Finite\-Sample Error\.\\displaystyle\-\\underbrace\{\\frac\{1\}\{1\-\\gamma\}\\sum\_\{i=1\}^\{n\}\\zeta\_\{i\}\}\_\{\\text\{Estimator\-Bias Penalty\}\}\-\\underbrace\{\\sum\_\{i=1\}^\{n\}\\frac\{A\_\{\\max\}\}\{1\-\\gamma\}\\sqrt\{\\frac\{\\log\(2n/\\delta\_\{\\mathrm\{conf\}\}\)\}\{2N\_\{i\}\}\}\}\_\{\\text\{Finite\-Sample Error\}\}\.
where the gain coefficientsκireg=2​gi⊤​\(Fireg\)−1​gi\\kappa\_\{i\}^\{\\mathrm\{reg\}\}=\\sqrt\{2\\,g\_\{i\}^\{\\\!\\top\}\(F\_\{i\}^\{\\mathrm\{reg\}\}\)^\{\-1\}g\_\{i\}\}and curvature termsaireg=Liloc/λmin​\(Fireg\)a\_\{i\}^\{\\mathrm\{reg\}\}=L\_\{i\}^\{\\mathrm\{loc\}\}/\\lambda\_\{\\min\}\(F\_\{i\}^\{\\mathrm\{reg\}\}\)are defined in Theorem[4\.7](https://arxiv.org/html/2605.05216#S4.Thmtheorem7)\. The three penalty terms have precise interpretations: the*occupancy\-shift penalty*2​γ\(1−γ\)2​Amax​∑i=1n12​δi\\frac\{2\\gamma\}\{\(1\-\\gamma\)^\{2\}\}A\_\{\\max\}\\sum\_\{i=1\}^\{n\}\\sqrt\{\\tfrac\{1\}\{2\}\\,\\delta\_\{i\}\}captures the cumulative cost of distribution shift as agents update sequentially \(Theorem[4\.4](https://arxiv.org/html/2605.05216#S4.Thmtheorem4)\); the*estimator\-bias penalty*11−γ​∑i=1nζi\\frac\{1\}\{1\-\\gamma\}\\sum\_\{i=1\}^\{n\}\\zeta\_\{i\}arises from our sequence\-aware advantage estimator \(Theorem[4\.2](https://arxiv.org/html/2605.05216#S4.Thmtheorem2)\); and the*finite\-sample error*∑i=1nAmax1−γ​log⁡\(2​n/δconf\)2​Ni\\sum\_\{i=1\}^\{n\}\\frac\{A\_\{\\max\}\}\{1\-\\gamma\}\\sqrt\{\\frac\{\\log\(2n/\\delta\_\{\\mathrm\{conf\}\}\)\}\{2N\_\{i\}\}\}accounts for statistical uncertainty from usingNiN\_\{i\}samples per step \(Theorem[4\.9](https://arxiv.org/html/2605.05216#S4.Thmtheorem9)\)\.

Crucially, this policy improvement guarantee is realized via a provably convergent optimization procedure\. The sequential projected block updates on the stage surrogateG​\(θ\)=∑iLiseqG\(\\theta\)=\\sum\_\{i\}L\_\{i\}^\{\\textsc\{seq\}\}satisfy the standardO​\(1/K\)O\(1/K\)convergence rate for the projected\-gradient mapping overKKblock steps \(Theorem[4\.8](https://arxiv.org/html/2605.05216#S4.Thmtheorem8)\)\. All statements are*sequence\-agnostic*inσ\\sigmaand remain valid under*plug\-and\-play*replacements, provided the new agent is initialized within the same per\-state KL trust region \(via Stage\-0 alignment in the Appendix\)\.

##### Remark \(High\-level interpretation\)\.

The inequality above certifies that the*information\-geometric gain*—scaled by natural\-gradient structure throughδi\\sqrt\{\\delta\_\{i\}\}terms—minus three*controllable costs*\(distribution shift, estimator bias, and sampling noise\) remains nonnegative in aggregate\. At the same time, the optimizer reliably finds such updates at a rateO​\(1/K\)O\(1/K\)\. Proof ingredients are modular: occupancy shift via per\-state TV/KL bounds \(§[4\.2](https://arxiv.org/html/2605.05216#S4.SS2)\), single\-step⇒\\Rightarrowjoint\-stage telescoping \(Theorems[4\.2](https://arxiv.org/html/2605.05216#S4.Thmtheorem2)–[4\.4](https://arxiv.org/html/2605.05216#S4.Thmtheorem4)\), information\-geometric lower bounds forκireg,aireg\\kappa\_\{i\}^\{\\mathrm\{reg\}\},a\_\{i\}^\{\\mathrm\{reg\}\}\(Theorem[4\.7](https://arxiv.org/html/2605.05216#S4.Thmtheorem7)\), finite\-sample concentration \(Theorem[4\.9](https://arxiv.org/html/2605.05216#S4.Thmtheorem9)\), and sequential projected\-gradient convergence \(Theorem[4\.8](https://arxiv.org/html/2605.05216#S4.Thmtheorem8)\)\. For tighter certificates, one may replaceδi\\delta\_\{i\}by any expected\-KL radiusδ¯i≤δi\\bar\{\\delta\}\_\{i\}\\leq\\delta\_\{i\}in the gain terms \(Theorem[4\.7](https://arxiv.org/html/2605.05216#S4.Thmtheorem7)\)\.

### 4\.2\.Occupancy Shift: definition and per\-state control

##### Definition \(discounted occupancy shift\)\.

For two policiesπ′\\pi^\{\\prime\}andπ\\pi, the discounted occupancy shift measures how much their state visitation distributions differ:

Δocc​\(π′,π;f\)≜\|𝔼dπ′​\[f\]−𝔼dπ​\[f\]\|,Δocc​\(π′,π\)≜sup‖f‖∞≤1Δocc​\(π′,π;f\)\.\\Delta\_\{\\mathrm\{occ\}\}\(\\pi^\{\\prime\},\\pi;f\)\\ \\triangleq\\ \\big\|\\mathbb\{E\}\_\{d^\{\\pi^\{\\prime\}\}\}\[f\]\-\\mathbb\{E\}\_\{d^\{\\pi\}\}\[f\]\\big\|,\\qquad\\Delta\_\{\\mathrm\{occ\}\}\(\\pi^\{\\prime\},\\pi\)\\ \\triangleq\\ \\sup\_\{\\\|f\\\|\_\{\\infty\}\\leq 1\}\\ \\Delta\_\{\\mathrm\{occ\}\}\(\\pi^\{\\prime\},\\pi;f\)\.
###### Lemma 4\.1\(Occupancy\-shift bound\)\.

With per\-state divergencesDTVmax\(π′∥π\)≜ess​supsDTV\(π′\(⋅\|s\),π\(⋅\|s\)\)D\_\{\\mathrm\{TV\}\}^\{\\max\}\(\\pi^\{\\prime\}\\\|\\pi\)\\\!\\triangleq\\\!\\operatorname\*\{ess\\,sup\}\_\{s\}\\,D\_\{\\mathrm\{TV\}\}\\\!\\big\(\\pi^\{\\prime\}\(\\cdot\|s\),\\pi\(\\cdot\|s\)\\big\)andDKLmax\(π′∥π\)≜ess​supsDKL\(π′\(⋅\|s\)∥π\(⋅\|s\)\)D\_\{\\mathrm\{KL\}\}^\{\\max\}\(\\pi^\{\\prime\}\\\|\\pi\)\\\!\\triangleq\\\!\\operatorname\*\{ess\\,sup\}\_\{s\}\\,D\_\{\\mathrm\{KL\}\}\\\!\\big\(\\pi^\{\\prime\}\(\\cdot\|s\)\\\|\\pi\(\\cdot\|s\)\\big\), one has the worst\-case bound

Δocc​\(π′,π\)≤2​γ1−γ​DTVmax​\(π′∥π\)≤2​γ1−γ​12​DKLmax​\(π′∥π\)\.\\Delta\_\{\\mathrm\{occ\}\}\(\\pi^\{\\prime\},\\pi\)\\ \\leq\\ \\frac\{2\\gamma\}\{1\-\\gamma\}\\,D\_\{\\mathrm\{TV\}\}^\{\\max\}\(\\pi^\{\\prime\}\\\|\\pi\)\\ \\leq\\ \\frac\{2\\gamma\}\{1\-\\gamma\}\\,\\sqrt\{\\tfrac\{1\}\{2\}\\,D\_\{\\mathrm\{KL\}\}^\{\\max\}\(\\pi^\{\\prime\}\\\|\\pi\)\}\.

When plugged into the performance\-difference identity \(which carries an extra factor1/\(1−γ\)1/\(1\-\\gamma\)\), this yields a*single\-step penalty*2​γ\(1−γ\)2​Amax​12​DKLmax\\tfrac\{2\\gamma\}\{\(1\-\\gamma\)^\{2\}\}\\,A\_\{\\max\}\\sqrt\{\\tfrac\{1\}\{2\}\\,D\_\{\\mathrm\{KL\}\}^\{\\max\}\}in Theorem[4\.2](https://arxiv.org/html/2605.05216#S4.Thmtheorem2)\.

##### Per\-state control \(our remedy\)\.

At sequential stepii, we*cap*the per\-state KL between the updated agent and its current version:

DKL\(πtarσ​\(i\)\(⋅\|s\)∥πcurσ​\(i\)\(⋅\|s\)\)≤δi\(s\)∀s\.D\_\{\\mathrm\{KL\}\}\\\!\\big\(\\pi^\{\\sigma\(i\)\}\_\{\\mathrm\{tar\}\}\(\\cdot\|s\)\\,\\big\\\|\\,\\pi^\{\\sigma\(i\)\}\_\{\\mathrm\{cur\}\}\(\\cdot\|s\)\\big\)\\ \\leq\\ \\delta\_\{i\}\(s\)\\quad\\forall s\.This impliesDKLmax​\(π^i∥π^i−1\)≤δimax:=ess​sups⁡δi​\(s\)D\_\{\\mathrm\{KL\}\}^\{\\max\}\\\!\\big\(\\hat\{\\pi\}^\{\\,i\}\\\|\\hat\{\\pi\}^\{\\,i\-1\}\\big\)\\leq\\delta\_\{i\}^\{\\max\}:=\\operatorname\*\{ess\\,sup\}\_\{s\}\\delta\_\{i\}\(s\)and therefore

Δocc​\(π^i,π^i−1\)≤2​γ1−γ​12​δimax\.\\Delta\_\{\\mathrm\{occ\}\}\\\!\\big\(\\hat\{\\pi\}^\{\\,i\},\\hat\{\\pi\}^\{\\,i\-1\}\\big\)\\ \\leq\\ \\frac\{2\\gamma\}\{1\-\\gamma\}\\,\\sqrt\{\\tfrac\{1\}\{2\}\\,\\delta\_\{i\}^\{\\max\}\}\.Consequently, the only distribution\-shift cost entering our single\-step and joint\-stage guarantees is the explicitδi\\sqrt\{\\delta\_\{i\}\}penalty \(Theorems[4\.2](https://arxiv.org/html/2605.05216#S4.Thmtheorem2)and[4\.4](https://arxiv.org/html/2605.05216#S4.Thmtheorem4)\), which we control directly by choosing the per\-state radii\{δi​\(s\)\}\\\{\\delta\_\{i\}\(s\)\\\}\(and enforcing them via trust\-region updates / Stage\-0 KL projection\)\.

### 4\.3\.Monotonic Improvement Guarantees

###### Theorem 4\.2\(Single\-step monotonic improvement\)\.

For stepi∈\{1,…,n\}i\\in\\\{1,\\ldots,n\\\},

J​\(π^i\)−J​\(π^i−1\)≥Liseq​\(πtarσ​\(i\)\)−2​γ\(1−γ\)2​Amax​12​DKLmax​\(π^i∥π^i−1\)−ζi1−γ\.J\(\\hat\{\\pi\}^\{\\,i\}\)\-J\(\\hat\{\\pi\}^\{\\,i\-1\}\)\\ \\geq\\ L\_\{i\}^\{\\textsc\{seq\}\}\(\\pi^\{\\sigma\(i\)\}\_\{\\mathrm\{tar\}\}\)\-\\frac\{2\\gamma\}\{\(1\-\\gamma\)^\{2\}\}\\,A\_\{\\max\}\\sqrt\{\\tfrac\{1\}\{2\}\\,D\_\{\\mathrm\{KL\}\}^\{\\max\}\\\!\\big\(\\hat\{\\pi\}^\{\\,i\}\\\|\\hat\{\\pi\}^\{\\,i\-1\}\\big\)\}\-\\frac\{\\zeta\_\{i\}\}\{1\-\\gamma\}\.SinceAmax≤2​Rmax1−γA\_\{\\max\}\\leq\\tfrac\{2R\_\{\\max\}\}\{1\-\\gamma\}, the penalty equals4​γ​Rmax\(1−γ\)3​12​DKLmax​\(π^i∥π^i−1\)\\tfrac\{4\\gamma R\_\{\\max\}\}\{\(1\-\\gamma\)^\{3\}\}\\sqrt\{\\tfrac\{1\}\{2\}\\,D\_\{\\mathrm\{KL\}\}^\{\\max\}\\\!\\big\(\\hat\{\\pi\}^\{\\,i\}\\\|\\hat\{\\pi\}^\{\\,i\-1\}\\big\)\}\.

###### Proof sketch\.

The performance difference lemma gives

J​\(π^i\)−J​\(π^i−1\)=11−γ​𝔼s∼dπ^i,a∼π^i\(⋅\|s\)​\[Aπ^i−1​\(s,a\)\]\.J\(\\hat\{\\pi\}^\{\\,i\}\)\-J\(\\hat\{\\pi\}^\{\\,i\-1\}\)=\\frac\{1\}\{1\-\\gamma\}\\mathbb\{E\}\_\{s\\sim d^\{\\hat\{\\pi\}^\{\\,i\}\},a\\sim\\hat\{\\pi\}^\{\\,i\}\(\\cdot\|s\)\}\[A^\{\\hat\{\\pi\}^\{\\,i\-1\}\}\(s,a\)\]\.Apply the Lemma[4\.1](https://arxiv.org/html/2605.05216#S4.Thmtheorem1)above to replacedπ^id^\{\\hat\{\\pi\}^\{\\,i\}\}bydπ^i−1d^\{\\hat\{\\pi\}^\{\\,i\-1\}\}\(TV/KL penalty\)\. Then replaceAπ^i−1A^\{\\hat\{\\pi\}^\{\\,i\-1\}\}by the on\-policy estimator and account forζi\\zeta\_\{i\}\. Full details are in the Appendix \(Single\-step\)\. ∎

###### Lemma 4\.3\(KL accumulation for product policies\)\.

Letp​\(𝐚\|s\)=∏j=1npj​\(a\(j\)\|s\)p\(\\mathbf\{a\}\|s\)=\\prod\_\{j=1\}^\{n\}p\_\{j\}\(a^\{\(j\)\}\|s\)andq​\(𝐚\|s\)=∏j=1nqj​\(a\(j\)\|s\)q\(\\mathbf\{a\}\|s\)=\\prod\_\{j=1\}^\{n\}q\_\{j\}\(a^\{\(j\)\}\|s\)\. Then

DKL\(p\(⋅\|s\)∥q\(⋅\|s\)\)=∑j=1nDKL\(pj\(⋅\|s\)∥qj\(⋅\|s\)\)\.D\_\{\\mathrm\{KL\}\}\\\!\\big\(p\(\\cdot\|s\)\\\|q\(\\cdot\|s\)\\big\)=\\sum\_\{j=1\}^\{n\}D\_\{\\mathrm\{KL\}\}\\\!\\big\(p\_\{j\}\(\\cdot\|s\)\\\|q\_\{j\}\(\\cdot\|s\)\\big\)\.At stepii,π^i\\hat\{\\pi\}^\{\\,i\}andπ^i−1\\hat\{\\pi\}^\{\\,i\-1\}differ only in factorσ​\(i\)\\sigma\(i\), hence for allss,

DKL\(π^i\(⋅\|s\)∥π^i−1\(⋅\|s\)\)=DKL\(πtarσ​\(i\)\(⋅\|s\)∥πcurσ​\(i\)\(⋅\|s\)\)≤δi\.D\_\{\\mathrm\{KL\}\}\\\!\\big\(\\hat\{\\pi\}^\{\\,i\}\(\\cdot\|s\)\\\|\\hat\{\\pi\}^\{\\,i\-1\}\(\\cdot\|s\)\\big\)=D\_\{\\mathrm\{KL\}\}\\\!\\big\(\\pi^\{\\sigma\(i\)\}\_\{\\mathrm\{tar\}\}\(\\cdot\|s\)\\\|\\pi^\{\\sigma\(i\)\}\_\{\\mathrm\{cur\}\}\(\\cdot\|s\)\\big\)\\leq\\delta\_\{i\}\.

###### Theorem 4\.4\(Joint\-stage monotonic improvement\)\.

After a stage with radii\{δi\}i=1n\\\{\\delta\_\{i\}\\\}\_\{i=1\}^\{n\},

J​\(π¯\)−J​\(πcur\)≥∑iLiseq−2​γ\(1−γ\)2​∑iAmax\(i\)​12​δi−11−γ​∑iζi\.J\(\\bar\{\\pi\}\)\-J\(\\pi\_\{\\mathrm\{cur\}\}\)\\ \\geq\\ \\sum\_\{i\}L\_\{i\}^\{\\textsc\{seq\}\}\-\\frac\{2\\gamma\}\{\(1\-\\gamma\)^\{2\}\}\\sum\_\{i\}A\_\{\\max\}^\{\(i\)\}\\sqrt\{\\tfrac\{1\}\{2\}\\,\\delta\_\{i\}\}\-\\frac\{1\}\{1\-\\gamma\}\\sum\_\{i\}\\zeta\_\{i\}\.In the uniform caseδi≡δ\\delta\_\{i\}\\equiv\\delta, the penalty scales asO​\(n​δ\)O\(n\\sqrt\{\\delta\}\), whereAmax\(i\)≜sups,a\|Aπ^i−1​\(s,a\)\|A\_\{\\max\}^\{\(i\)\}\\triangleq\\sup\_\{s,a\}\\big\|A^\{\\hat\{\\pi\}^\{\\,i\-1\}\}\(s,a\)\\big\|\.

###### Proof sketch\.

Telescoping givesJ​\(π¯\)−J​\(πcur\)=∑i=1n\[J​\(π^i\)−J​\(π^i−1\)\]J\(\\bar\{\\pi\}\)\-J\(\\pi\_\{\\mathrm\{cur\}\}\)=\\sum\_\{i=1\}^\{n\}\[J\(\\hat\{\\pi\}^\{\\,i\}\)\-J\(\\hat\{\\pi\}^\{\\,i\-1\}\)\]\. Apply Theorem[4\.2](https://arxiv.org/html/2605.05216#S4.Thmtheorem2)to each term\. KL penalties add directly because each step constrains only one agent \(Lemma[4\.3](https://arxiv.org/html/2605.05216#S4.Thmtheorem3)\); there are no cross\-terms\. Full details are in the Appendix \(Joint\-stage\)\. ∎

##### Structural properties\.

Sequence\-agnosticism: Theorem[4\.4](https://arxiv.org/html/2605.05216#S4.Thmtheorem4)holds for any update orderσ\\sigma, including data\-dependent choices; the numeric value of the lower bound can still depend on the realized order\. Plug\-and\-play invariance: Replacing agents with stronger models preserves guarantees if they optimize the same surrogate under the same constraints; this is realized via a Stage‑0 alignment that starts within the trust region \(Appendix Stage‑0\)\. Certificate tightening: Upgrades either increasesupLiseq\\sup L\_\{i\}^\{\\textsc\{seq\}\}at fixedδi\\delta\_\{i\}or achieve the same surrogate with smaller radiiδi′<δi\\delta^\{\\prime\}\_\{i\}<\\delta\_\{i\}, reducing∑iδi\\sum\_\{i\}\\sqrt\{\\delta\_\{i\}\}\. A high\-probability relaxation can replaceδi\\delta\_\{i\}withδi\+ϵ​\(Ni,η\)\\delta\_\{i\}\+\\epsilon\(N\_\{i\},\\eta\), whereϵ​\(Ni,η\)=log⁡\(2/η\)/\(2​Ni\)\\epsilon\(N\_\{i\},\\eta\)=\\sqrt\{\\log\(2/\\eta\)/\(2N\_\{i\}\)\}\(DKW; see Appendix Stage‑0\)\.

### 4\.4\.Information\-Theoretic Envelopes

We now establish fundamental limits on achievable improvements under KL constraints and finite sampling budgets\. LetNiN\_\{i\}denote the number of on\-policy episodes at stepiiunderdπ^i−1d^\{\\hat\{\\pi\}^\{\\,i\-1\}\}\.

##### Information\-geometric preliminaries\.

AssumeLiseqL\_\{i\}^\{\\textsc\{seq\}\}is twice differentiable wrt the parametersθi\\theta\_\{i\}of agentσ​\(i\)\\sigma\(i\)nearπcurσ​\(i\)\\pi^\{\\sigma\(i\)\}\_\{\\mathrm\{cur\}\}\. Letgi=∇θiLiseqg\_\{i\}=\\nabla\_\{\\theta\_\{i\}\}L\_\{i\}^\{\\textsc\{seq\}\}and

Fi=𝔼s∼dπ^i−1​\[𝔼a∼πθi\(⋅\|s\)​\[∇θilog⁡πθi​\(a\|s\)​∇θilog⁡πθi​\(a\|s\)⊤\]\]\.F\_\{i\}=\\mathbb\{E\}\_\{s\\sim d^\{\\hat\{\\pi\}^\{\\,i\-1\}\}\}\\left\[\\mathbb\{E\}\_\{a\\sim\\pi\_\{\\theta\_\{i\}\}\(\\cdot\|s\)\}\\left\[\\nabla\_\{\\theta\_\{i\}\}\\log\\pi\_\{\\theta\_\{i\}\}\(a\|s\)\\nabla\_\{\\theta\_\{i\}\}\\log\\pi\_\{\\theta\_\{i\}\}\(a\|s\)^\{\\top\}\\right\]\\right\]\.Local smoothness:Liseq​\(θi\+Δ\)≥Liseq​\(θi\)\+gi⊤​Δ−Liloc2​‖Δ‖2L\_\{i\}^\{\\textsc\{seq\}\}\(\\theta\_\{i\}\+\\Delta\)\\geq L\_\{i\}^\{\\textsc\{seq\}\}\(\\theta\_\{i\}\)\+g\_\{i\}^\{\\top\}\\Delta\-\\frac\{L\_\{i\}^\{\\mathrm\{loc\}\}\}\{2\}\\\|\\Delta\\\|^\{2\}\. Fisher–KL bridge:𝔼s​\[DKL​\(πθi\+Δ∥πθi\)\]≈12​Δ⊤​Fi​Δ\\mathbb\{E\}\_\{s\}\[D\_\{\\mathrm\{KL\}\}\(\\pi\_\{\\theta\_\{i\}\+\\Delta\}\\\|\\pi\_\{\\theta\_\{i\}\}\)\]\\approx\\frac\{1\}\{2\}\\Delta^\{\\top\}F\_\{i\}\\Deltafor small‖Δ‖\\\|\\Delta\\\|\.

###### Theorem 4\.5\(Oracle single\-step upper bound\)\.

UnderDKLmax​\(π^i∥π^i−1\)≤δiD\_\{\\mathrm\{KL\}\}^\{\\max\}\(\\hat\{\\pi\}^\{\\,i\}\\\|\\hat\{\\pi\}^\{\\,i\-1\}\)\\leq\\delta\_\{i\}and\|A\|≤Amax\|A\|\\leq A\_\{\\max\},

J​\(π^i\)−J​\(π^i−1\)≤Amax1−γ​2​δi\.J\(\\hat\{\\pi\}^\{\\,i\}\)\-J\(\\hat\{\\pi\}^\{\\,i\-1\}\)\\ \\leq\\ \\frac\{A\_\{\\max\}\}\{1\-\\gamma\}\\,\\sqrt\{2\\,\\delta\_\{i\}\}\.

###### Theorem 4\.6\(Finite\-budget single\-step envelope\)\.

WithNiN\_\{i\}on\-policy episodes at stepii, with probability at least1−δ1\-\\delta,

J​\(π^i\)−J​\(π^i−1\)≤Amax1−γ​2​δi\+Amax1−γ​log⁡\(2/δ\)2​Ni\.J\(\\hat\{\\pi\}^\{\\,i\}\)\-J\(\\hat\{\\pi\}^\{\\,i\-1\}\)\\ \\leq\\ \\frac\{A\_\{\\max\}\}\{1\-\\gamma\}\\sqrt\{2\\,\\delta\_\{i\}\}\\ \+\\ \\frac\{A\_\{\\max\}\}\{1\-\\gamma\}\\sqrt\{\\frac\{\\log\(2/\\delta\)\}\{2N\_\{i\}\}\}\.

###### Theorem 4\.7\(Budget\-aware stage lower bound\)\.

Under the local smoothness and Fisher–KL bridge assumptions, withFireg=Fi\+ϵ​IF\_\{i\}^\{\\mathrm\{reg\}\}=F\_\{i\}\+\\epsilon Iand noting that𝔼s​DKL≤DKLmax\\mathbb\{E\}\_\{s\}D\_\{\\mathrm\{KL\}\}\\leq D\_\{\\mathrm\{KL\}\}^\{\\max\}\(so any effective radiusδ¯i≤δi\\bar\{\\delta\}\_\{i\}\\leq\\delta\_\{i\}may be used\), with probability at least1−δconf1\-\\delta\_\{\\mathrm\{conf\}\},

J​\(π¯\)−J​\(πcur\)≥\\displaystyle J\(\\bar\{\\pi\}\)\-J\(\\pi\_\{\\mathrm\{cur\}\}\)\\ \\geq∑i=1n\(κireg​δ¯i−aireg​δ¯i\)−2​γ\(1−γ\)2​Amax​∑i=1n12​δi\\displaystyle\\sum\_\{i=1\}^\{n\}\\big\(\\kappa\_\{i\}^\{\\mathrm\{reg\}\}\\sqrt\{\\bar\{\\delta\}\_\{i\}\}\-a\_\{i\}^\{\\mathrm\{reg\}\}\\bar\{\\delta\}\_\{i\}\\big\)\-\\ \\frac\{2\\gamma\}\{\(1\-\\gamma\)^\{2\}\}A\_\{\\max\}\\sum\_\{i=1\}^\{n\}\\sqrt\{\\tfrac\{1\}\{2\}\\,\\delta\_\{i\}\}−11−γ​∑i=1nζi−∑i=1nAmax1−γ​log⁡\(2​n/δconf\)2​Ni,\\displaystyle\\ \-\\ \\frac\{1\}\{1\-\\gamma\}\\sum\_\{i=1\}^\{n\}\\zeta\_\{i\}\\ \\ \-\\ \\sum\_\{i=1\}^\{n\}\\frac\{A\_\{\\max\}\}\{1\-\\gamma\}\\sqrt\{\\frac\{\\log\(2n/\\delta\_\{\\mathrm\{conf\}\}\)\}\{2N\_\{i\}\}\},whereκireg=2​gi⊤​\(Fireg\)−1​gi\\kappa\_\{i\}^\{\\mathrm\{reg\}\}=\\sqrt\{2\\,g\_\{i\}^\{\\top\}\(F\_\{i\}^\{\\mathrm\{reg\}\}\)^\{\-1\}g\_\{i\}\}andaireg=Liloc/λmin​\(Fireg\)a\_\{i\}^\{\\mathrm\{reg\}\}=L\_\{i\}^\{\\mathrm\{loc\}\}/\\lambda\_\{\\min\}\(F\_\{i\}^\{\\mathrm\{reg\}\}\)\.

### 4\.5\.Convergence and Finite\-Sample Analysis

##### Stage objective and block updates\.

Letθ=\(θ1,…,θn\)\\theta=\(\\theta\_\{1\},\\ldots,\\theta\_\{n\}\)parameterize the team policyπθ=∏j=1nπθj\(j\)\\pi\_\{\\theta\}=\\prod\_\{j=1\}^\{n\}\\pi^\{\(j\)\}\_\{\\theta\_\{j\}\}\. Given the sequence\-aware surrogates\{Liseq\}i=1n\\\{L\_\{i\}^\{\\textsc\{seq\}\}\\\}\_\{i=1\}^\{n\}under the intermediate occupancies\{dπ^i−1\}\\\{d^\{\\hat\{\\pi\}^\{\\,i\-1\}\}\\\}, define

G​\(θ\)≜∑i=1nLiseq​\(θ\)=∑j=1nGj​\(θj\),Gj​\(θj\):=Lσ−1​\(j\)seq​\(θj\)\.G\(\\theta\)\\ \\triangleq\\ \\sum\_\{i=1\}^\{n\}L\_\{i\}^\{\\textsc\{seq\}\}\(\\theta\)\\ =\\ \\sum\_\{j=1\}^\{n\}G\_\{j\}\(\\theta\_\{j\}\),\\qquad G\_\{j\}\(\\theta\_\{j\}\):=L^\{\\textsc\{seq\}\}\_\{\\sigma^\{\-1\}\(j\)\}\(\\theta\_\{j\}\)\.By construction, eachLiseqL\_\{i\}^\{\\textsc\{seq\}\}depends only on blockθσ​\(i\)\\theta\_\{\\sigma\(i\)\}under fixeddπ^i−1d^\{\\hat\{\\pi\}^\{\\,i\-1\}\}, hence mixed second derivatives vanish\.

At the*start of the stage*, fix per\-agent trust regions\.

Θj:=\{θj:DKL\(πθj\(j\)\(⋅∣s\)∥πcur\(j\)\(⋅∣s\)\)≤δj\(s\)∀s\},Θ:=∏j=1nΘj,\\Theta\_\{j\}\\ :=\\ \\Big\\\{\\theta\_\{j\}:\\ D\_\{\\mathrm\{KL\}\}\\\!\\Big\(\\pi^\{\(j\)\}\_\{\\theta\_\{j\}\}\(\\cdot\\mid s\)\\,\\big\\\|\\,\\pi^\{\(j\)\}\_\{\\mathrm\{cur\}\}\(\\cdot\\mid s\)\\Big\)\\ \\leq\\ \\delta\_\{j\}\(s\)\\ \\ \\forall s\\Big\\\},\\qquad\\Theta:=\\prod\_\{j=1\}^\{n\}\\Theta\_\{j\},which remain*fixed within the stage*\. Perform one sweep of*sequential*block updates followingσ\\sigma:

θji=ProjΘj​\(θji−1\+ηj​∇θjG​\(θi−1\)\),θ−ji=θ−ji−1,j=σ​\(i\),\\theta\_\{j\}^\{\\,i\}\\ =\\ \\mathrm\{Proj\}\_\{\\Theta\_\{j\}\}\\\!\\Big\(\\theta\_\{j\}^\{\\,i\-1\}\+\\eta\_\{j\}\\,\\nabla\_\{\\theta\_\{j\}\}G\(\\theta^\{\\,i\-1\}\)\\Big\),\\qquad\\theta\_\{\-j\}^\{\\,i\}=\\theta\_\{\-j\}^\{\\,i\-1\},\\quad j=\\sigma\(i\),And define the block projected\-gradient mapping\.gηj\(j\)​\(θi−1\):=1ηj​\(θji−θji−1\)\.g\_\{\\eta\_\{j\}\}^\{\(j\)\}\(\\theta^\{\\,i\-1\}\)\\ :=\\ \\tfrac\{1\}\{\\eta\_\{j\}\}\\big\(\\theta\_\{j\}^\{\\,i\}\-\\theta\_\{j\}^\{\\,i\-1\}\\big\)\.

###### Theorem 4\.8\(Sequential block\-coordinate gradient descent \(BCGD\) on a fixed trust region: monotonicity and rate\)\.

Assume\|A^oni−1\|≤Amax\|\\widehat\{A\}^\{\\,i\-1\}\_\{\\textsc\{on\}\}\|\\leq A\_\{\\max\}and bounded log\-policy derivatives∥∇θlogπθ\(⋅∣s\)∥≤B1\\\|\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(\\cdot\\mid s\)\\\|\\leq B\_\{1\},∥∇θ2logπθ\(⋅∣s\)∥op≤B2\\\|\\nabla\_\{\\theta\}^\{2\}\\log\\pi\_\{\\theta\}\(\\cdot\\mid s\)\\\|\_\{\\mathrm\{op\}\}\\leq B\_\{2\}onΘ\\Theta\. Then eachGjG\_\{j\}isLblkL\_\{\\mathrm\{blk\}\}\-smooth withLblk=Amax1−γ​\(B2\+B12\)L\_\{\\mathrm\{blk\}\}=\\frac\{A\_\{\\max\}\}\{1\-\\gamma\}\(B\_\{2\}\+B\_\{1\}^\{2\}\)\. Ifηj≤1/Lblk\\eta\_\{j\}\\leq 1/L\_\{\\mathrm\{blk\}\}, every block step is ascent and

G​\(θi\)−G​\(θi−1\)≥\(ηj−Lblk​ηj22\)​‖gηj\(j\)​\(θi−1\)‖2≥ηj2​‖gηj\(j\)​\(θi−1\)‖2\.G\(\\theta^\{\\,i\}\)\-G\(\\theta^\{\\,i\-1\}\)\\ \\geq\\ \\Big\(\\eta\_\{j\}\-\\tfrac\{L\_\{\\mathrm\{blk\}\}\\eta\_\{j\}^\{2\}\}\{2\}\\Big\)\\,\\big\\\|g\_\{\\eta\_\{j\}\}^\{\(j\)\}\(\\theta^\{\\,i\-1\}\)\\big\\\|^\{2\}\\ \\geq\\ \\tfrac\{\\eta\_\{j\}\}\{2\}\\,\\big\\\|g\_\{\\eta\_\{j\}\}^\{\(j\)\}\(\\theta^\{\\,i\-1\}\)\\big\\\|^\{2\}\.AfterKKblock updates \(e\.g\., one sweep hasK=nK=n\),

1K​∑t=1K‖gηjt\(jt\)​\(θt−1\)‖2≤2​\(G⋆−G​\(θ0\)\)\(mint⁡ηjt\)​K,G⋆=supθ∈ΘG​\(θ\)\.\\frac\{1\}\{K\}\\sum\_\{t=1\}^\{K\}\\big\\\|g\_\{\\eta\_\{j\_\{t\}\}\}^\{\(j\_\{t\}\)\}\(\\theta^\{\\,t\-1\}\)\\big\\\|^\{2\}\\ \\leq\\ \\frac\{2\\,\(G^\{\\star\}\-G\(\\theta^\{\\,0\}\)\)\}\{\(\\min\_\{t\}\\eta\_\{j\_\{t\}\}\)\\,K\},\\qquad G^\{\\star\}=\\sup\_\{\\theta\\in\\Theta\}G\(\\theta\)\.*Proof sketch\.*Block smoothness from bounded Hessians, one\-step descent lemma on the active block, and Euclidean projection optimality; telescope across blocks\. Full proof: Appendix \(PGD\)\.

###### Theorem 4\.9\(Finite\-sample concentration forLiseqL\_\{i\}^\{\\textsc\{seq\}\}\)\.

LetL^iseq\\widehat\{L\}\_\{i\}^\{\\textsc\{seq\}\}be the empirical surrogate fromNiN\_\{i\}on\-policy episodes at stepiiand setB:=Amax/\(1−γ\)B:=A\_\{\\max\}/\(1\-\\gamma\)\. For anyδ∈\(0,1\)\\delta\\in\(0,1\), with i\.i\.d\. episodes,

\|L^iseq−Liseq\|≤B​log⁡\(2/δ\)2​Niwith probability at least​1−δ\.\\big\|\\widehat\{L\}\_\{i\}^\{\\textsc\{seq\}\}\-L\_\{i\}^\{\\textsc\{seq\}\}\\big\|\\ \\leq\\ B\\sqrt\{\\frac\{\\log\(2/\\delta\)\}\{2N\_\{i\}\}\}\\quad\\text\{with probability at least \}1\-\\delta\.Underβ\\beta\-mixing episodes with∑t≥1β​\(t\)<∞\\sum\_\{t\\geq 1\}\\beta\(t\)<\\infty, the same bound holds withNiN\_\{i\}replaced byNi,eff=Ni/\(1\+2​∑t≥1β​\(t\)\)N\_\{i,\\mathrm\{eff\}\}=N\_\{i\}/\(1\+2\\sum\_\{t\\geq 1\}\\beta\(t\)\)\. Variance\-aware and robust MoM variants are given in the Appendix: Finite‑sample\.

## 5\.Algorithm

We instantiate our framework with SAT, which updates agents sequentially using sequence\-level optimization\. Unlike token\-level baselines \(e\.g\., PPO\), SAT uses a group\-relative, sequence\-level objective that evaluates complete trajectories and normalizes advantages within prompt\-level groups such as GRPO and DAPO\(shao2024deepseekmath;yu2025dapo\)\. This is consistent with the masked\-activation product\-policy view \(Sec\.[3](https://arxiv.org/html/2605.05216#S3)\): only active heads contribute to joint actions and divergences at each state, and standard factorization is recovered in the fully simultaneous case\. Each part below reflects an assumption or value in Sec \.[4](https://arxiv.org/html/2605.05216#S4): the sequence\-level surrogate targetsLiseqL\_\{i\}^\{\\textsc\{seq\}\}, clipped advantages ensure boundary conditions forAmaxA\_\{\\max\}, and the trust\-region controller enforces per\-agent per\-stateDKLmaxD\_\{\\mathrm\{KL\}\}^\{\\max\}\.

##### Sequence\-level optimization\.

Token\-level RLHF methods compute advantages per action and update policies using per\-step likelihood ratios\. In multi\-agent settings with sequential updates, such token\-wise advantages may induce off\-policy errors as agents update one by one; credit assignment across agents is also ambiguous\. SAT operates at the sequence level, evaluating complete trajectories and normalizing advantages within groups of completions for the same prompt\. This matches our theory, where the surrogateLiseqL\_\{i\}^\{\\textsc\{seq\}\}evaluates full episodes under the intermediate policyπ^i−1\\hat\{\\pi\}^\{\\,i\-1\}, reducing distribution mismatch and fitting stage\-wise bounds in Sec \.[4](https://arxiv.org/html/2605.05216#S4)\.

##### Sequential updates with intermediate policies\.

At each stage, agents are updated in orderσ​\(1\),…,σ​\(n\)\\sigma\(1\),\\ldots,\\sigma\(n\)\. When updating agentσ​\(i\)\\sigma\(i\), we form the intermediate policy

π^i−1=\(πtarσ​\(1\),…,πtarσ​\(i−1\),πcurσ​\(i\),…,πcurσ​\(n\)\)\.\\hat\{\\pi\}^\{\\,i\-1\}=\\big\(\\pi^\{\\sigma\(1\)\}\_\{\\mathrm\{tar\}\},\\ldots,\\pi^\{\\sigma\(i\-1\)\}\_\{\\mathrm\{tar\}\},\\;\\pi^\{\\sigma\(i\)\}\_\{\\mathrm\{cur\}\},\\ldots,\\pi^\{\\sigma\(n\)\}\_\{\\mathrm\{cur\}\}\\big\)\.\(1\)The sequence\-aware advantage estimator conditions onπ^i−1\\hat\{\\pi\}^\{\\,i\-1\}\. At the beginning of each stage, we collect fresh on\-policy rollouts underπ^0=πcur\\hat\{\\pi\}^\{\\,0\}=\\pi\_\{\\mathrm\{cur\}\}, and within the stage, we reuse these trajectories by recomputing per\-timestep importance ratios

ρt=π^i−1​\(at∣st\)πcur​\(at∣st\)with truncated weightsct=min⁡\{1,ρt\}\\rho\_\{t\}=\\frac\{\\hat\{\\pi\}^\{\\,i\-1\}\(a\_\{t\}\\mid s\_\{t\}\)\}\{\\pi\_\{\\mathrm\{cur\}\}\(a\_\{t\}\\mid s\_\{t\}\)\}\\quad\\text\{with truncated weights\}\\quad c\_\{t\}=\\min\\\{1,\\rho\_\{t\}\\\}in the multi\-step estimator; advantages use GAE withλ=0\.95\\lambda=0\.95\. This realizes the “sequence\-aware” evaluation required by Sec \.[4](https://arxiv.org/html/2605.05216#S4), and the truncation\-induced bias is explicitly captured by the estimator termζi\\zeta\_\{i\}that appears in the improvement bounds\.

##### Group\-relative normalization\.

For each prompt, we sampleGgrpG\_\{\\text\{grp\}\}trajectories underπ^i−1\\hat\{\\pi\}^\{\\,i\-1\}and compute the group\-normalized advantageA~g\\tilde\{A\}\_\{g\}forg∈\{1,…,Ggrp\}g\\in\\\{1,\\ldots,G\_\{\\text\{grp\}\}\\\}:

A~g=A^gi−1−μσ\+ϵ,μ=1Ggrp​∑j=1GgrpA^ji−1,σ2=1Ggrp​∑j=1Ggrp\(A^ji−1−μ\)2\.\\tilde\{A\}\_\{g\}=\\frac\{\\widehat\{A\}^\{\\,i\-1\}\_\{g\}\-\\mu\}\{\\sigma\+\\epsilon\},\\quad\\mu=\\frac\{1\}\{G\_\{\\text\{grp\}\}\}\\sum\_\{j=1\}^\{G\_\{\\text\{grp\}\}\}\\widehat\{A\}^\{\\,i\-1\}\_\{j\},\\quad\\sigma^\{2\}=\\frac\{1\}\{G\_\{\\text\{grp\}\}\}\\sum\_\{j=1\}^\{G\_\{\\text\{grp\}\}\}\(\\widehat\{A\}^\{\\,i\-1\}\_\{j\}\-\\mu\)^\{2\}\.\(2\)Here,A^gi−1\\widehat\{A\}^\{\\,i\-1\}\_\{g\}is the aggregated per\-timestep advantage\. We useGgrp∈\{4,8\}G\_\{\\text\{grp\}\}\\in\\\{4,8\\\}to balance variance and cost\. Finally, we apply symmetric clipping\|A~g\|≤Aclip\|\\tilde\{A\}\_\{g\}\|\\leq A\_\{\\text\{clip\}\}\(settingAmax:=AclipA\_\{\\max\}:=A\_\{\\text\{clip\}\}\) to satisfy the bounded\-advantage assumption in Sec\.[4](https://arxiv.org/html/2605.05216#S4)\.

##### Trust\-region update for agentσ​\(i\)\\sigma\(i\)\.

Let𝒯i​\(τ\)\\mathcal\{T\}\_\{i\}\(\\tau\)denote the timesteps where agentσ​\(i\)\\sigma\(i\)acts along trajectoryτ\\tau\.

ui​\(τ\)\\displaystyle u\_\{i\}\(\\tau\)=∑t∈𝒯i​\(τ\)\(log⁡πσ​\(i\)​\(ai,t∣st\)−log⁡πcurσ​\(i\)​\(ai,t∣st\)\),\\displaystyle=\\sum\_\{t\\in\\mathcal\{T\}\_\{i\}\(\\tau\)\}\\Big\(\\log\\pi^\{\\sigma\(i\)\}\(a\_\{i,t\}\\\!\\mid s\_\{t\}\)\-\\log\\pi^\{\\sigma\(i\)\}\_\{\\mathrm\{cur\}\}\(a\_\{i,t\}\\\!\\mid s\_\{t\}\)\\Big\),\(3\)ri​\(τ\)\\displaystyle r\_\{i\}\(\\tau\)=exp⁡\(ui​\(τ\)\)\.\\displaystyle=\\exp\\big\(u\_\{i\}\(\\tau\)\\big\)\.\(4\)We optimize a clipped sequence\-level objective with a per\-agent KL penalty and a high\-quantile monitor forDKLmaxD\_\{\\mathrm\{KL\}\}^\{\\max\}constraint:

ℒi=𝔼​\[min⁡\{ri​\(τ\)​A~g,exp⁡\(clip​\(ui​\(τ\),log⁡\(1−ε\),log⁡\(1\+ε\)\)\)​A~g\}\]−β⋅𝔼s\[DKL\(πσ​\(i\)\(⋅∣s\)∥πcurσ​\(i\)\(⋅∣s\)\)\],\\begin\{split\}\\mathcal\{L\}\_\{i\}=&\\;\\mathbb\{E\}\\\!\\left\[\\min\\\!\\left\\\{r\_\{i\}\(\\tau\)\\,\\tilde\{A\}\_\{g\},\\;\\exp\\\!\\big\(\\mathrm\{clip\}\(u\_\{i\}\(\\tau\),\\log\(1\-\\varepsilon\),\\log\(1\+\\varepsilon\)\)\\big\)\\,\\tilde\{A\}\_\{g\}\\right\\\}\\right\]\\\\ &\\;\-\\;\\beta\\cdot\\mathbb\{E\}\_\{s\}\\\!\\left\[D\_\{\\mathrm\{KL\}\}\\\!\\big\(\\pi^\{\\sigma\(i\)\}\(\\cdot\\mid s\)\\,\\\|\\,\\pi^\{\\sigma\(i\)\}\_\{\\mathrm\{cur\}\}\(\\cdot\\mid s\)\\big\)\\right\],\\end\{split\}\(5\)withε=0\.2\\varepsilon=0\.2by default\. The penalty coefficientβ\\betais adapted online, and updates are backtracked whenever the empirical\(1−α\)\(1\-\\alpha\)\-quantile of per\-state KL exceeds the target radiusδi\\delta\_\{i\}\. This enforces the per\-agent trust region in the sense required by Sec\.[4](https://arxiv.org/html/2605.05216#S4): the quantile controller yields a high\-probability relaxation ofDKLmaxD\_\{\\mathrm\{KL\}\}^\{\\max\}that maps the theoretical radius to an effectiveδi\+ϵ​\(N,η\)\\delta\_\{i\}\+\\epsilon\(N,\\eta\), closing the gap between implementation and the bounds in Theorems[4\.2](https://arxiv.org/html/2605.05216#S4.Thmtheorem2)–[4\.4](https://arxiv.org/html/2605.05216#S4.Thmtheorem4)\.

Algorithm 1Sequential Agent Tuning \(SAT\)1:Input:Team

\{π\(j\)\}j=1n\\\{\\pi^\{\(j\)\}\\\}\_\{j=1\}^\{n\}, prompts

𝒟\\mathcal\{D\}, KL radii

\{δj\}\\\{\\delta\_\{j\}\\\}, group size

GgrpG\_\{\\text\{grp\}\}, clip

ε\\varepsilon, penalty

β\\beta
2:Initialize:

πcur←\(π\(1\),…,π\(n\)\)\\pi\_\{\\text\{cur\}\}\\leftarrow\(\\pi^\{\(1\)\},\\ldots,\\pi^\{\(n\)\}\)
3:forstage

k=1,2,…k=1,2,\\ldotsdo

4:

ℬ←ROLLOUT​\(πcur,𝒟\)\\mathcal\{B\}\\leftarrow\\text\{ROLLOUT\}\(\\pi\_\{\\text\{cur\}\},\\mathcal\{D\}\);

σ←ORDERAGENTS​\(ℬ\)\\sigma\\leftarrow\\text\{ORDERAGENTS\}\(\\mathcal\{B\}\)⊳\\trianglerightOn\-policy data; update order

5:for

i=1i=1to

nndo

6:

π^i−1←\(πtarσ​\(1\),…,πtarσ​\(i−1\),πcurσ​\(i\),…,πcurσ​\(n\)\)\\hat\{\\pi\}^\{\\,i\-1\}\\leftarrow\(\\pi^\{\\sigma\(1\)\}\_\{\\text\{tar\}\},\\ldots,\\pi^\{\\sigma\(i\-1\)\}\_\{\\text\{tar\}\},\\pi^\{\\sigma\(i\)\}\_\{\\text\{cur\}\},\\ldots,\\pi^\{\\sigma\(n\)\}\_\{\\text\{cur\}\}\)
7:

\{A~g\}←SEQADVANTAGES​\(ℬ,π^i−1,Ggrp\)\\\{\\tilde\{A\}\_\{g\}\\\}\\leftarrow\\text\{SEQADVANTAGES\}\(\\mathcal\{B\},\\hat\{\\pi\}^\{\\,i\-1\},G\_\{\\text\{grp\}\}\)⊳\\trianglerightGAE \+ group\-norm \+ clip toAmaxA\_\{\\max\}

8:

ℒi←𝔼​\[min⁡\{ri​A~g,exp⁡\(clip​\(ui,log⁡\(1±ε\)\)\)​A~g\}\]−β​𝔼s​\[DKL​\(πσ​\(i\)∥πcurσ​\(i\)\)\]\\mathcal\{L\}\_\{i\}\\leftarrow\\mathbb\{E\}\[\\min\\\{r\_\{i\}\\tilde\{A\}\_\{g\},\\exp\(\\text\{clip\}\(u\_\{i\},\\log\(1\\pm\\varepsilon\)\)\)\\tilde\{A\}\_\{g\}\\\}\]\-\\beta\\mathbb\{E\}\_\{s\}\[D\_\{\\text\{KL\}\}\(\\pi^\{\\sigma\(i\)\}\\\|\\pi^\{\\sigma\(i\)\}\_\{\\text\{cur\}\}\)\]
9:

πtarσ​\(i\)←OPTIMIZE​\(ℒi\)\\pi^\{\\sigma\(i\)\}\_\{\\text\{tar\}\}\\leftarrow\\text\{OPTIMIZE\}\(\\mathcal\{L\}\_\{i\}\)⊳\\trianglerightTrust\-region update

10:if

Quantile1−α​\[DKL​\(πtarσ​\(i\)∥πcurσ​\(i\)\)\]\>δi\\text\{Quantile\}\_\{1\-\\alpha\}\[D\_\{\\text\{KL\}\}\(\\pi^\{\\sigma\(i\)\}\_\{\\text\{tar\}\}\\\|\\pi^\{\\sigma\(i\)\}\_\{\\text\{cur\}\}\)\]\>\\delta\_\{i\}thenbacktrackand increase

β\\beta⊳\\trianglerightEnforceDKLmax≤δiD\_\{\\text\{KL\}\}^\{\\max\}\\leq\\delta\_\{i\}

11:endif

12:

πcurσ​\(i\)←πtarσ​\(i\)\\pi^\{\\sigma\(i\)\}\_\{\\text\{cur\}\}\\leftarrow\\pi^\{\\sigma\(i\)\}\_\{\\text\{tar\}\}
13:endfor

14:

πcur←\(πcur\(1\),…,πcur\(n\)\)\\pi\_\{\\text\{cur\}\}\\leftarrow\(\\pi^\{\(1\)\}\_\{\\text\{cur\}\},\\ldots,\\pi^\{\(n\)\}\_\{\\text\{cur\}\}\)⊳\\trianglerightComplete stage:π¯\\bar\{\\pi\}in Thm\.[4\.4](https://arxiv.org/html/2605.05216#S4.Thmtheorem4)

15:endfor

16:return

πcur\\pi\_\{\\text\{cur\}\}

##### Plug\-and\-play agent upgrades\.

When replacing agentjjwith a stronger pretrained modelπpre\(j\)\\pi\_\{\\text\{pre\}\}^\{\(j\)\}, we perform a Stage\-0 alignment step that projectsπpre\(j\)\\pi\_\{\\text\{pre\}\}^\{\(j\)\}onto the trust region around theπcur\(j\)\\pi\_\{\\mathrm\{cur\}\}^\{\(j\)\}:

πnew\(j\)=\\displaystyle\\pi\_\{\\text\{new\}\}^\{\(j\)\}=\{\}argminπ𝔼s∼dπcur\[DKL\(π\(⋅∣s\)∥πpre\(j\)\(⋅∣s\)\)\]\\displaystyle\\arg\\min\_\{\\pi\}\\;\\mathbb\{E\}\_\{s\\sim d^\{\\pi\_\{\\mathrm\{cur\}\}\}\}\\\!\\left\[D\_\{\\mathrm\{KL\}\}\\big\(\\pi\(\\cdot\\mid s\)\\,\\\|\\,\\pi\_\{\\text\{pre\}\}^\{\(j\)\}\(\\cdot\\mid s\)\\big\)\\right\]\(6\)subject toDKL\(π\(⋅∣s\)∥πcur\(j\)\(⋅∣s\)\)≤δ0\(s\)∀s\.\\displaystyle\\text\{subject to\}\\quad D\_\{\\mathrm\{KL\}\}\\\!\\big\(\\pi\(\\cdot\\mid s\)\\,\\\|\\,\\pi\_\{\\mathrm\{cur\}\}^\{\(j\)\}\(\\cdot\\mid s\)\\big\)\\leq\\delta\_\{0\}\(s\)\\;\\;\\forall s\.The Lagrangian yields a closed\-form geometric mixture for statess:

πnew\(j\)​\(a∣s\)=πpre\(j\)​\(a∣s\)1/\(1\+λ​\(s\)\)​\(πcur\(j\)​\(a∣s\)\)λ​\(s\)/\(1\+λ​\(s\)\)∑a′πpre\(j\)​\(a′∣s\)1/\(1\+λ​\(s\)\)​\(πcur\(j\)​\(a′∣s\)\)λ​\(s\)/\(1\+λ​\(s\)\),\\pi\_\{\\text\{new\}\}^\{\(j\)\}\(a\\mid s\)=\\frac\{\\pi\_\{\\text\{pre\}\}^\{\(j\)\}\(a\\mid s\)^\{1/\(1\+\\lambda\(s\)\)\}\\,\\big\(\\pi\_\{\\mathrm\{cur\}\}^\{\(j\)\}\(a\\mid s\)\\big\)^\{\\lambda\(s\)/\(1\+\\lambda\(s\)\)\}\}\{\\sum\_\{a^\{\\prime\}\}\\pi\_\{\\text\{pre\}\}^\{\(j\)\}\(a^\{\\prime\}\\mid s\)^\{1/\(1\+\\lambda\(s\)\)\}\\,\\big\(\\pi\_\{\\mathrm\{cur\}\}^\{\(j\)\}\(a^\{\\prime\}\\mid s\)\\big\)^\{\\lambda\(s\)/\(1\+\\lambda\(s\)\)\}\},\(7\)where the state\-dependent Lagrange multiplierλ​\(s\)≥0\\lambda\(s\)\\geq 0is chosen by binary search to satisfy the per\-state KL constraint with equality when the pretrained policy would otherwise violate it\. This initialization ensuresπnew\(j\)\\pi\_\{\\text\{new\}\}^\{\(j\)\}starts within the trust region, so Theorems[4\.2](https://arxiv.org/html/2605.05216#S4.Thmtheorem2)–[4\.4](https://arxiv.org/html/2605.05216#S4.Thmtheorem4)remain valid\. Moreover, ifπpre\(j\)\\pi\_\{\\text\{pre\}\}^\{\(j\)\}is superior, the stage surrogateLiseqL\_\{i\}^\{\\textsc\{seq\}\}improves at fixedδi\\delta\_\{i\}, or the same surrogate value can be achieved with tighter radii, reducing the occupancy\-shift\.

## 6\.Experiments

We evaluate SAT across three domains: mathematical reasoning, active reasoning, and planning\. Our experiments address three questions in a single unified protocol: whether SAT\-trained small\-model teams can match or exceed much larger monolithic models, whether empirical improvements align with the theory in Sec\.[4](https://arxiv.org/html/2605.05216#S4), and how plug\-and\-play upgrades affect performance under fixed trust\-region budgets\. Unless explicitly noted, we follow the masked\-activation product\-policy view for action composition\.

### 6\.1\.Experimental Setup

#### 6\.1\.1\.Models and Datasets

We evaluate SAT by finetuning three small LLMs with parameters ranging from1\.51\.5B to88B: including LLaMA\-3\.3 \(3B\-instruct\)\(dubey2024llama\), Qwen2\.5 \(1\.5B\-instruct, 3B\-instruct, 7B\-instruct\)\(qwen2025qwen25technicalreport\), and Qwen3 \(4B\-instruct, 8B\-instruct\)\(yang2025qwen3technicalreport\)\. To demonstrate that SAT\-trained small models can achieve competitive performance, we compare them against larger baseline models ranging from3030B to7070B parameters, such as LLaMA\-3\.3 \(70B\-instruct\), Qwen2\.5 \(32B\-instruct, 72B\-instruct\), and Qwen3 \(30B\-A3B\-Instruct\-2507, 32B\), as well as publicly available thinking/non\-thinking systems where applicable\. For mathematical reasoning, we use AIME 2024/2025\(aime2024\), ZebraLogic\(lin2025zebralogic\), and MATH\-500\(hendrycks2021math\)\. For active reasoning, we utilize ARBench\(zhou2025passive\)to evaluate the capacity of LLMs to formulate suitable questions to acquire additional information\. For planning, we use AutoLogi\(zhu2025autologiautomatedgenerationlogic\)and PlanBench\(valmeekam2024planbench\), including BlocksWorld and logistics domains\. If the exact released parameter counts of some checkpoints differ slightly from the labels above, this variance does not alter the evaluation protocol; the citations remain as the authoritative references\.

#### 6\.1\.2\.Training Data\.

We prepare our training code with the VeRL framework\(sheng2024hybridflow\)with temperature set to0\.80\.8, top\_p set to1\.01\.0, and maximum output length set to32,76832\{,\}768tokens for inference\. Due to the high variance of the outputs from reasoning models, we report avg@K \(pass@1 performance averaged overKKoutputs\) and pass@K for each benchmark\. For benchmarks with few samples \(AIME24/25 and ZebraLogic\), we set a largerK=64K=64\. We useK=25K=25for ARBench,K=8K=8for planning benchmarks, andK=4K=4for MATH\-500\. To ensure accurate evaluation, we adopt the verification functions from DeepScaleR for mathematics problems\. However, whether all external baselines used the same verifiers is uncertain; therefore, we retain their reported numbers and citations as is\. For math reasoning ability, we use datasets from DeepScaleR\(luo2025deepcoder\)and DAPO\(yu2025dapo\); for active reasoning and planning, we use the training sets from ARBench and PlanBench directly\.

### 6\.2\.Main Results

Table[1](https://arxiv.org/html/2605.05216#S6.T1)presents performance comparisons across general reasoning, active reasoning, and planning\. A team of three Qwen3\-4B models \(12B total parameters\) achieves70\.3%70\.3\\%on AIME24,63\.4%63\.4\\%on AIME25, and86\.1%86\.1\\%on MATH\-500, while the three\-agent LLaMA 3\.1\-8B SAT configuration \(24B total\) reaches86\.3%86\.3\\%\(AIME24\),76\.7%76\.7\\%\(AIME25\),99\.1%99\.1\\%\(MATH\-500\),93\.1%93\.1\\%\(ZebraLogic\), and42\.7%42\.7\\%\(PlanBench\-BW NL\)\. These results indicate that SAT\-trained teams of small models can be competitive with, and in some cases surpass, much larger baselines in the listed settings, while using substantially fewer parameters\. The precise relative ranking varies by benchmark, and we maintain every cited baseline as reported\. The empirical trends are consistent with the theoretical guidance that per\-agent KL radii and sequence\-aware estimation control drift, yielding near\-monotonic stage\-wise improvements\.

Table 1\.Performance comparison across general reasoning, active reasoning, and planning\. Publicly verifiable entries are kept as reported\. Baseline results for Qwen3 and GPT\-4 are from\(yang2025qwen3technicalreport\)\. We mark the best result inboldand the second\-best withunderline\.
### 6\.3\.Heterogeneous Models Analysis

Table[2](https://arxiv.org/html/2605.05216#S6.T2)explores parameter\-size and model\-family heterogeneity under SAT without predefined roles\. Performance improves from22\.522\.5\(2×\\times0\.6B\+1\.7B configuration\) to52\.152\.1\(2×\\times4B\+8B configuration\) as total capacity increases, with clear diminishing returns: moving from sub\-2B to 4B models yields substantial improvements, while adding 8B models provides more modest gains\. This pattern aligns with our theoretical analysis, which shows super\-linear prefix penalties for large cumulative updates\. Model selection significantly impacts team performance—the heterogeneous configuration \(2×\\timesQwen3\-4B \+ LLaMA\-8B\) achieves an average score of50\.450\.4relative to the homogeneous 3×\\timesQwen3\-4B baseline \(50\.750\.7\)\. Adding a stronger agent \(Gemma\-3\-12B\-IT\) improves performance to53\.453\.4, although at an increased computational cost\. Mixing thinking and non\-thinking agents often degrades performance, consistent with the need for agents trained under compatible sequence\-level surrogates\.

Table 2\.Heterogeneous team configurations under SAT \(3 agents, no predefined roles\)\. Datasets: AIME24, AIME25, ARBench\-DC \(process@25\), PlanBench\-BW \(NL\)\.Params \(B\)is the sum of the three agents’ parameters \(no weight sharing\)\.Heterogeneity tags: size = different parameter scales \(capacity\), model = different model families / pretraining methods and corpora, thinking = different thinking styles \(e\.g\., deliberate/CoT\-distilled vs non\-thinking\)\. We mark the best column for the average result inboldand the second\-best withunderline\.ConfigurationParams \(B\)AIME24AIME25ARBench\-DCPlanBench\-BW \(NL\)AverageParameter\-Size Heterogeneity2×\\timesQwen3\-0\.6B Base \+ Qwen3\-1\.7B Base3\.022\.117\.326\.124\.522\.5Qwen3\-0\.6B Base\+ Qwen3\-1\.7B Base\+ Qwen3\-4B Base6\.349\.738\.531\.532\.938\.22×\\timesQwen3\-1\.7B Base\+ Qwen3\-4B Base7\.455\.944\.333\.134\.942\.1Qwen3\-1\.7B Base\+ Qwen3\-4B Base\+ Qwen3\-8B Base13\.766\.956\.335\.536\.748\.92×\\timesQwen3\-4B Base\+ Qwen3\-8B Base16\.072\.161\.136\.738\.552\.1Model Heterogeneity2×\\timesQwen3\-4B Base\+ Qwen2\.5\-3B Base11\.068\.157\.335\.136\.349\.2LLaMA 3\.1\-8B Base\+ LLaMA 3\.3\-4B Base\+ Qwen2\.5\-3B Base15\.064\.754\.334\.735\.947\.42×\\timesQwen3\-4B Base\+ LLaMA 3\.1\-8B Base16\.069\.558\.735\.737\.750\.4Qwen3\-4B Base\+ LLaMA 3\.1\-8B Base\+ Gemma\-3\-12B\-IT24\.074\.364\.136\.938\.353\.4Thinking Modes \+ Non\-thinking MixDeepSeek\-R1\-Distill\-\(Qwen\-1\.5B \+ Llama3\.1\-8B Base\)↓\\downarrow\+ Qwen3\-4B \(non\-thinking\)13\.558\.347\.133\.335\.143\.52×\\timesQwen3\-0\.6B \(thinking\) \+ Qwen3\-4B \(non\-thinking\)5\.241\.732\.530\.131\.934\.12×\\timesQwen3\-4B \(thinking\) \+ Qwen3\-1\.7B \(non\-thinking\)9\.762\.952\.334\.336\.146\.42×\\timesQwen3\-4B \(thinking\) \+ Qwen3\-8B \(non\-thinking\)16\.064\.353\.734\.736\.747\.3
### 6\.4\.Stage\-wise Improvement Analysis

Table 3\.Plug\-and\-play under SAT\. The baseline is Qwen3 trained by SAT\. “Composite” is the mean of AIME24, AIME25, ARBench\-DC under the same evaluation protocol as Table[1](https://arxiv.org/html/2605.05216#S6.T1)\. We compare static heterogeneity \(train from scratch\) with PnP upgrades on the same baseline; KL is per\-agent per\-state; Viol\.% is the first\-stage monotonicity violation rate; Cost is relative to baseline parameters\. The best performance is inbold\.Figure[1](https://arxiv.org/html/2605.05216#S6.F1)illustrates training dynamics and trust\-region behavior across the full optimization horizon\. Panels \(a\) and \(b\) show SAT training curves on AIME24 and ARBench, respectively, compared against DAPO and GRPO baselines using Qwen2\.5\-32B\. Horizontal dashed lines denote the performance of the 3\-round3×3\\timesGPT\-4o\-mini debate and GPT\-o1\-preview as reference points\. On AIME24, SAT exhibits rapid early gains before stabilizing around 53% accuracy, while ARBench demonstrates more gradual but steady improvement to 41% success rate\. Panel \(c\) provides empirical validation of our trust\-region analysis during AIME24 training\. The violation rate, measuring the fraction of states where the per\-state KL constraint is exceeded, scales approximately asδ\\sqrt\{\\delta\}\(black dashed line\), consistent with the occupancy\-shift penalty in Theorem[4\.4](https://arxiv.org/html/2605.05216#S4.Thmtheorem4)\. Across multiple agents and training stages, violations concentrate in early steps and diminish as agents align, matching Theorem[4\.9](https://arxiv.org/html/2605.05216#S4.Thmtheorem9)\.

![Refer to caption](https://arxiv.org/html/2605.05216v1/x1.png)Figure 1\.Stage\-wise performance and trust\-region validation with SAT\.\(a\) AIME24 2024 accuracy over training steps; \(b\) ARBench overall success rate; \(c\) trust\-region violation rate vs\. KL radiusδ\\deltaon AIME24 training\. Horizontal dashed lines in \(a\) and \(b\) denote 3\-round3×3\\timesGPT\-4o\-mini debate and GPT\-o1\-preview baselines\. SAT is compared against Qwen2\.5\-32B trained with DAPO and GRPO\. SAT demonstrates rapid early gains on AIME24 and steady improvements on ARBench\. Panel \( \(c\) shows that violation rate scales asδ\\sqrt\{\\delta\}\(dashed theory line\), with the operating region \(yellow\) achieving low violations while enabling effective exploration, consistent with Theorems[4\.4](https://arxiv.org/html/2605.05216#S4.Thmtheorem4)and[4\.9](https://arxiv.org/html/2605.05216#S4.Thmtheorem9)\.
### 6\.5\.Plug\-and\-Play Upgrades

Table[3](https://arxiv.org/html/2605.05216#S6.T3)evaluates plug\-and\-play capabilities under SAT without role assignments, comparing PnP upgrades against static heterogeneous training from scratch using the composite metric\. Single\-agent replacement with LLaMA 3\.1\-8B slightly decreases the composite from55\.255\.2to54\.654\.6underδ=0\.010\\delta=0\.010, while replacing two agents with 8B models increases it to65\.665\.6\. Full PnP to three 8B models reaches69\.169\.1composite with the lowest violation rate \(2\.5%2\.5\\%\) at2\.00×2\.00\\timescost\. Downgrade experiments quantify capacity performance trade offs: replacing with Qwen3\-0\.6B reduces composite to50\.650\.6at0\.72×0\.72\\timescost\.

## 7\.Conclusion

We have introduced Sequential Agent Tuning \(SAT\), a coordinator\-free framework that empowers teams of small LLMs to surpass the performance of larger models\. Our method integrates sequential agent updates with a sequence\-aware advantage estimator and per\-agent KL trust regions, ensuring stable and monotonic improvements\. The central element of our theoretical contribution is the principle of “plug\-and\-play” invariance, which allows individual agents to be upgraded without incurring the cost of retraining the entire team\. We further validated our plug\-and\-play theory by demonstrating that modular agent upgrades yield significant performance gains, in line with analytical expectations\. These results position SAT as a practical and scalable path for deploying high\-performance AI systems, especially in resource\-constrained environments\. We hope this work encourages a shift from simply scaling monolithic models to strategically advancing teams of smaller agents for LLM community\.

\{acks\}

Y\. Fan’s work is not related to the position at Amazon Inc\. Y\. Xu’s work is funded by the National Science Foundation \(NSF\) under grant NSF IIS1910794 and DMS\-2406896;

## References