Very interesting new work from Microsoft.

(bookmark it)

This work is related to this emerging theme of leveraging harnesses for model post-training.

Modern agents run inside a harness that owns tools, context, and control flow. When you train them, the harness owns the environment loop and the trainer only sees LLM request and response pairs.

How it works.

Agent Lightning v1.0 connects any harness to RL through an endpoint proxy in about 3,500 lines, then works through what breaks in that setup, retokenization, sample merging, advantage calculation, loss normalization, and backend scheduling.

Using 6K training examples and modest compute, it moves Qwen3.5-9B on SWE-bench Verified from 41.8% to 56.4%.

Paper: https://arxiv.org/abs/2608.17528

Track more trending AI papers in our academy: https://academy.dair.ai

1The overall framework of Agent Lightning v1.0.

Source: https://arxiv.org/html/2608.17528 August, 2026

Agent Lightning v1.0: Towards Harnessed Agentic RL

Zhiyuan He1,∗,‡Siwei Zhang2,∗Zhiwen Zhou3,∗Yuqing Yang1,‡Yu Kang1 Yuge Zhang1Luna K. Qiu1Tin Yan Tsui4Jiahang Xu1Chong Luo1

1Microsoft2Fudan University3Zhejiang University4University of Edinburgh

Figure 1:The overall framework of Agent Lightning v1.0.

Abstract

Abstract.Modern agents do not operate as standalone LLMs. They run insideagent harnessesthat manage tools, context, and control flow, which makes the harness a critical component. Our original Agent Lightning work introduced a disaggregated architecture that connects arbitrary agents to reinforcement learning (RL) training through an LLM endpoint proxy. Recent frameworks such as verl Uni-Agent, AReaL 2.0, slime v0.3.0, and Polar have followed this proxy-based approach. Such a proxy-based training approach enables RL training with the harness. In this work, we use the termharnessed agentic RLto describe this paradigm, in which the deploy-time harness is directly involved in model post-training, thereby narrowing the gap between training and actual use.

We find that harnessed agentic RL differs fundamentally from traditional agentic RL and introduces a new set of challenges. In traditional agentic RL, the training engine owns the environment interaction loop. In harnessed agentic RL, the harness owns this loop, while the training engine observes only a sequence of LLM request-response pairs. How to model and assemble these calls into training samples remains an open question. Through a careful study, we identify several challenges of harnessed agentic RL, including retokenization, sample merging, advantage calculation, loss normalization, and training backend scheduling. We find that, if not properly addressed, these challenges can lead to ineffective or unstable training. Existing frameworks generally leave these issues underspecified. In this paper, we provide the first comprehensive elaboration of them.

We further present Agent Lightning v1.0, a lightweight framework for harnessed agentic RL. We treat simplicity as a first principle, implementing the framework inonly approximately 3,500 lines of code.Its compact design supports arbitrary agent harnesses and provides a practical testbed for studying these challenges. We validate Agent Lightning v1.0 on general instruction-following agent, search agent, and coding agent. For coding agent, we find that existing RL frameworks provide limited support, including a lack of data and complete training scripts, as well as a reliance on large-scale computational resources. To address this gap, we provide a complete data-cleaning pipeline and reproducible training scripts based on open-source dataset and models.Using only 6K training examples and modest compute, RL improves Qwen3.5-9B on SWE-bench Verified from 41.8% to 56.4%, an absolute 14.6% gain.We release the complete workflow and scripts to facilitate reproducible harnessed agentic RL in Agent Lightning v1.0.

Introduction

Modern agents do not operate as standalone LLMs. They run insideagent harnessesthat manage tools, execution environments, context, and control flow. The harness therefore determines how an agent observes its environment, acts over long horizons, and recovers from failures, making it a central part of the agent’s capabilities. Prominent examples include coding-agent harnesses such as mini-SWE-agent[25], OpenHands[19], OpenCode[18], Claude Code[2], and Codex[16], as well as general-purpose harnesses such as OpenClaw[17]and Hermes[15].

Early reinforcement learning (RL) frameworks, including verl[24], AReaL[6], and slime[34], generally require users to implement the agent loop directly inside the training framework. Integrating existing agent harnesses is therefore difficult because they often have complex implementations and their own dependencies, making them hard to integrate directly into RL frameworks. Our original Agent Lightning work[14]introduced a disaggregated architecture for training and agent execution. It connects arbitrary agents to RL training through an LLM endpoint, with almost no changes to the agent. More recently, such proxy-based approach has become more common in frameworks such as verl Uni-Agent[5], AReaL 2.0[28], slime v0.3.0[34], and Polar[27], which naturally enables enables RL training with agent harnesses.

We use the termharnessed agentic RLfor RL training conducted through the same agent harness used at deployment. The harness, rather than the trainer, owns context construction, tool execution, and the agent–environment interaction loop, while the training system observes and optimizes the resulting model calls across a service boundary. This formulation preserves the harness’s deployment-time context policy, tool protocols, and execution semantics without requiring its agent loop to be reimplemented inside the RL framework.

Both traditional agentic RL and harnessed agentic RL can be modeled as partially observable Markov decision processes, but they differ in their latent state and in the observations presented to the policy model. In traditional agentic RL, the latent state is primarily the environment state. The policy model interacts almost directly with the environment through a transparent layer. The model produces action tokens, the environment returns an observation, and the tokenized observation extends the existing history aspt=(pt−1,at−1,ot)p_{t}=\bigl(p_{t-1},a_{t-1},o_{t}\bigr). Here,ptp_{t}is the token history presented to the model at steptt,at−1a_{t-1}is the action generated at the previous step, andoto_{t}is the latest environment observation. Consequently, the policy observes one continuously extended token history, and a rollout naturally forms one linear token trajectory.

In harnessed agentic RL, the policy model no longer interacts directly with the environment. The latent state contains both the harness state and the environment state. The harness owns context construction, control flow, tool execution, and agent orchestration, and independently constructs the request prompt for each model call. The policy observes only the exact prompt delivered through an LLM API and generates a response conditioned on that prompt. A rollout is therefore exposed at the model boundary as a sequence of request–response pairs,

(p1,a1),(p2,a2),…,(p_{1},a_{1}),(p_{2},a_{2}),\ldots,wherepip_{i}is the prompt sent in theii-th LLM call andaia_{i}is the corresponding model response. The intervening harness and environment state transitions remain latent. Figure2summarizes this difference. It also gives rise to the implementation challenges below, for which existing frameworks make different choices that can affect algorithmic correctness and training stability.

Agentic RLHarnessed Agentic RLStateEnvironmentHarness + environmentModel inputContinuous token historyPer-call promptsAgentsSingle ReAct agentMulti-agent, subagents, and handoffs

Figure 2:Comparison of traditional agentic RL and harnessed agentic RL. Both admit a POMDP formulation, but harnessed agentic RL adds harness state to the latent execution state and exposes the policy model to separately constructed model-call prompts. This change also shifts control and orchestration into the harness and makes the number of training samples more dynamic.The first challenge is retokenization and sample merging.Agent harnesses usually communicate with model APIs through text messages, while RL training operates on tokens. Most frameworks merge two consecutive calls whenpi+1p_{i+1}contains(pi,ai)(p_{i},a_{i})as a complete prefix at the token level. However, after retokenization, even when the text is unchanged, the token IDs ofaia_{i}inpi+1p_{i+1}can differ from those originally sampled by the model. This breaks token-level continuity and prevents the two calls from being safely merged.

Second, advantage calculation.In traditional agentic RL, each rollout is a Markov process that maps to a unique training sample. In harnessed agentic RL, one rollout may instead produce a dynamic number of training samples. This can result not only from the retokenization issue described above, but also from harness operations such as spawning subagents and summarizing context. These dynamic samples challenge how rewards and advantages should be assigned to training samples.

The third challenge is loss normalization.In harnessed agentic RL, a rollout may map to multiple training samples, making the number of samples in each training batch dynamic. Loss normalization therefore becomes nontrivial. For example, some existing frameworks still normalize losses at the sample level, giving greater optimization weight to rollouts that produce more samples, which may make the training unstable.

The fourth challenge is training backend scheduling under dynamic sample counts.The number of samples produced by a rollout batch is known only after harness execution and sample construction, while the number of training GPUs and their parallel configuration remain fixed. The backend must partition this variable sample set into training steps and mini-batches while balancing the workload across fixed GPU workers.

In this work, we provide the first systematic characterization of these challenges, and further present Agent Lightning v1.0, a complete refactoring of the original Agent Lightning. It is a lightweight framework for RL training with arbitrary agent harnesses. Our design principle is to keep the system as simple as possible, implemented in approximately 3,500 lines of code. Its training pipeline also embeds our own design choices for the challenges described above, providing a practical testbed for studying them.

We use Agent Lightning v1.0 to train general instruction-following agent, search agent, and coding agent. In particular, existing agent frameworks provide limited support for coding agent, including a lack of data and complete training scripts, possibly because of the complexity of data cleaning, the difficulty of environment setup, and the substantial computing resources required. To address this gap, we build on the open-source SWE-smith dataset and Qwen3.5-9B[22]to provide a complete data-cleaning pipeline and reproducible training scripts. Our final RL run uses only 6K training samples and modest computing resources. Using RL alone, our trained model improves on SWE-bench Verified from 41.8% to 56.4%, an absolute 14.6% gain. We release the complete workflow and scripts to the community to facilitate reproducibility.

Challenges

Harnessed agentic RL changes how a rollout is observed and modeled by the training engine. In traditional agentic RL, the training engine owns the environment interaction loop and maintains the complete token history. Here,ptp_{t}denotes the prompt tokens at steptt,ata_{t}denotes the action corresponding to the response tokens, andoto_{t}is the tokenized environment observation. The next prompt is constructed as

pt=(pt−1,at−1,ot).p_{t}=(p_{t-1},a_{t-1},o_{t}).(1)The overall rollout follows the sequence(p1,a1,o1,a2,o2,a3,…)(p_{1},a_{1},o_{1},a_{2},o_{2},a_{3},\ldots). This forms a well-defined Markov process and maps naturally to one linear training sample.

In harnessed agentic RL, the harness owns the environment interaction loop and the message state. The training engine can only observe calls made through the LLM endpoint. For a rolloutρ\rho, it records a sequence

𝒞⁡(ρ)=((p1,a1),(p2,a2),…,(pTρ,aTρ)),\mathcal{C}(\rho)=\bigl((p_{1},a_{1}),(p_{2},a_{2}),\ldots,(p_{T_{\rho}},a_{T_{\rho}})\bigr),(2)wherepip_{i}is the prompt tokens andaia_{i}is the exact response tokens sampled by the model. The environment interactions and harness state transitions between these calls are not directly visible. Consequently, assembling the observed call sequence into training samples becomes a modeling problem. This difference introduces several implementation challenges. Existing frameworks make different choices when addressing them, which can affect algorithmic correctness and training stability.

More formally, both traditional agentic RL and harnessed agentic RL admit a partially observable Markov decision process formulation. Their distinction lies in the latent state and the observation presented to the policy model. In harnessed agentic RL, let

st=(stharness,stenv)s_{t}=\bigl(s_{t}^{\mathrm{harness}},s_{t}^{\mathrm{env}}\bigr)(3)denote the latent execution state maintained jointly by the harness and the environment. The model does not observests_{t}directly. Instead, the harness constructs a message-level context and renders it into the exact token-level prompt used for generation:

Ctmsg\displaystyle C_{t}^{\mathrm{msg}}=ContextH⁡(stharness),\displaystyle=\operatorname{Context}_{H}\bigl(s_{t}^{\mathrm{harness}}\bigr),(4)pttok\displaystyle p_{t}^{\mathrm{tok}}=Tok⁡(Template⁡(Ctmsg)).\displaystyle=\operatorname{Tok}\bigl(\operatorname{Template}(C_{t}^{\mathrm{msg}})\bigr).(5)Each policy decision is therefore recorded as a call-level transition

zt=(pttok,attok),attok∼πθ(⋅∣pttok).z_{t}=\bigl(p_{t}^{\mathrm{tok}},a_{t}^{\mathrm{tok}}\bigr),\qquad a_{t}^{\mathrm{tok}}\sim\pi_{\theta}\bigl(\cdot\mid p_{t}^{\mathrm{tok}}\bigr).(6)A rollout yields a variable-length collection of such transitions, and no exact token-prefix relation between consecutive prompts is assumed. Any sequence construction performed for training must preserve the prompt under which each recorded action was actually sampled. We next describe several new challenges specific to harnessed agentic RL.

Retokenization and Sample Merging

Why Token-Prefix Continuity Breaks.

Calliias sampledpitokp_{i}^{\mathrm{tok}}havingaitoka_{i}^{\mathrm{tok}}Naive retokenization ofpi+1tokp_{i+1}^{\mathrm{tok}}pitokp_{i}^{\mathrm{tok}}havingnew turnpi+1tokp_{i+1}^{\mathrm{tok}}≠aitok\neq a_{i}^{\mathrm{tok}}: cannot mergeFigure 3:A retokenization example. The wordhavingsampled as the two tokenshandavingin callii(top) can be retokenized into different token boundaries,havanding, when the updated message history is retokenized for calli+1i{+}1(bottom). Although the underlying text is identical, the token boundaries differ. This breaks the token-level prefix condition in Equation9even though the text-level prefix condition in Equation8still holds.Agent harness typically communicates with model APIs through text messages. From the harness perspective, a rollout usually consists of turn-level calls

𝒞text​(ρ)=((p1text,a1text),(p2text,a2text),…),\mathcal{C}^{\mathrm{text}}(\rho)=\bigl((p_{1}^{\mathrm{text}},a_{1}^{\mathrm{text}}),(p_{2}^{\mathrm{text}},a_{2}^{\mathrm{text}}),\ldots\bigr),(7)where the harness sendspitextp_{i}^{\mathrm{text}}and receivesaitexta_{i}^{\mathrm{text}}. Typically, for a multi-turn ReAct-style[32]agent, the previous call forms a complete text-level prefix of the next prompt, e.g.

(pitext,aitext)⪯pi+1text.(p_{i}^{\mathrm{text}},a_{i}^{\mathrm{text}})\preceq p_{i+1}^{\mathrm{text}}.(8)Besides, when the harness spawns subagents or summarizes its context, the training engine may receive subsequent calls with different prompt histories that do not satisfy this relation.

RL training operates on exact token IDs and rollout log probabilities. The RL system represents each call using the token sequencespitokp_{i}^{\mathrm{tok}}andaitoka_{i}^{\mathrm{tok}}, whereaitok∼πθ(⋅∣pitok)a_{i}^{\mathrm{tok}}\sim\pi_{\theta}(\cdot\mid p_{i}^{\mathrm{tok}}). The response returned to the harness isaitext=Decode⁡(aitok)a_{i}^{\mathrm{text}}=\operatorname{Decode}(a_{i}^{\mathrm{tok}}). Training therefore observes the token-level sequence𝒞tok​(ρ)=((p1tok,a1tok),(p2tok,a2tok),…)\mathcal{C}^{\mathrm{tok}}(\rho)=((p_{1}^{\mathrm{tok}},a_{1}^{\mathrm{tok}}),(p_{2}^{\mathrm{tok}},a_{2}^{\mathrm{tok}}),\ldots).

Token-prefix continuity between consecutive calls requires

(pitok,aitok)⪯pi+1tok,\bigl(p_{i}^{\mathrm{tok}},a_{i}^{\mathrm{tok}}\bigr)\preceq p_{i+1}^{\mathrm{tok}},(9)where⪯\preceqdenotes an exact token-level prefix.

In practice, Equation8holding does not guarantee that Equation9holds. The next request is obtained by applying a chat template and tokenizer to the updated message history. After retokenization, the token IDs corresponding toaitexta_{i}^{\mathrm{text}}insidepi+1tokp_{i+1}^{\mathrm{tok}}can differ from the originally sampled IDs inaitoka_{i}^{\mathrm{tok}}.

This mismatch can arise through at least three mechanisms that we observe in our study:

1. (1)Chat-template non-compositionality.Rendering a complete message history is not necessarily equivalent to concatenating the renderings of its parts: Template⁡(A∥B)≠Template⁡(A)∥Template⁡(B).\operatorname{Template}(A\mathbin{\|}B)\neq\operatorname{Template}(A)\mathbin{\|}\operatorname{Template}(B).(10)A template may insert delimiters or connecting newlines at message boundaries, or omit markers that appeared in the original generation. In practice, for example, we find that Qwen’s chat template can remove an earlier<think>marker, breaking token-prefix continuity.
2. (2)Decode–retokenize drift.Token decoding is not injective, so converting sampled tokens to text and tokenizing that text again need not recover the original token IDs: Tok⁡(Decode⁡(aitok))≠aitok.\operatorname{Tok}\bigl(\operatorname{Decode}(a_{i}^{\mathrm{tok}})\bigr)\neq a_{i}^{\mathrm{tok}}.(11)Figure3gives a concrete example: the wordhavingis sampled as the two tokenshandaving, but retokenizing the same text inside a later prompt may instead producehavanding. Although the decoded text is unchanged, the token boundaries differ and break token-level continuity between the two calls.
3. (3)Inference-time output transformation.Tool-call and structured-output handlers may parse, normalize, repair, and reserialize a sampled response before returning it to the harness. Such processing can change whitespace, delimiters, JSON structure, or invalid syntax, so the response incorporated into a later prompt may differ even at the text level from the response represented by the sampled token IDs.

Mitigating Broken Token-Prefix Continuity.

The most direct strategy is to train every model call independently. Each generated token is treated as part of its call’s action, and the loss is computed on(pitok,aitok)(p_{i}^{\mathrm{tok}},a_{i}^{\mathrm{tok}})without assuming any relation to adjacent calls. This guarantees token-level correctness, but is computationally inefficient because long prompt prefixes shared across calls are repeatedly computed. Different frameworks handle it with different strategies.

AReaL[28]and verl Uni-Agent[5]maintain a request buffer in the LLM proxy that stores the historical text and tokens of each call. When a new request arrives and its text matches the buffered history exactly, the framework replaces the segment of the new prompt tokens that corresponds to the previous response text with the buffered response tokens, which guarantees that the token-prefix condition always holds. slime[34]and Polar[27]do not perform this replacement.

This replacement does more than recover computational reuse: when the buffered token IDs differ from those in the actual next request, it changes the prompt under which the next response is evaluated. Suppose the actual next prompt contains a reconstructed versiona^itok\widehat{a}_{i}^{\mathrm{tok}}of the previous response:

pi+1tok=pitok∥a^itok∥Δi+1.p_{i+1}^{\mathrm{tok}}=p_{i}^{\mathrm{tok}}\mathbin{\|}\widehat{a}_{i}^{\mathrm{tok}}\mathbin{\|}\Delta_{i+1}.(12)Replacing this segment with the originally sampledaitoka_{i}^{\mathrm{tok}}produces a stitched prompt

p~i+1tok=pitok∥aitok∥Δi+1,p~i+1tok≠pi+1tok.\widetilde{p}_{i+1}^{\mathrm{tok}}=p_{i}^{\mathrm{tok}}\mathbin{\|}a_{i}^{\mathrm{tok}}\mathbin{\|}\Delta_{i+1},\qquad\widetilde{p}_{i+1}^{\mathrm{tok}}\neq p_{i+1}^{\mathrm{tok}}.(13)The responseai+1toka_{i+1}^{\mathrm{tok}}was sampled from the policy conditioned onpi+1tokp_{i+1}^{\mathrm{tok}}, not onp~i+1tok\widetilde{p}_{i+1}^{\mathrm{tok}}. Training it under the stitched prompt therefore introduces an off-policy discrepancy. Exact token-prefix overlap should be used for merging only when it preserves the prompt actually consumed during rollout.

A second approach is prefix-shared or tree-structured training. Exact common token prefixes are represented once, while a branch-aware causal attention mask ensures that tokens attend only to their ancestors and earlier tokens on the same branch. This can reproduce the result of training independent causal sequences while reusing prefix computation. However, tree packing, custom attention masks or kernels, partitioning, and distributed gradient handling require substantial training backend support.

A third practical approach is best-effort sequence merging. Two consecutive calls are merged only when their observed token IDs satisfy Equation9. When the condition holds, only the unmatched suffix and next action are appended. When it fails, the current sequence is closed and a new one begins. This preserves the prompts consumed during rollout and works with standard dense causal kernels, while retokenization drift merely lowers the merge ratio. Agent Lightning v1.0 adopts this strategy as a middle ground between independent-call recomputation and backend-intensive tree training.

These approaches have different trade-offs. Buffered token replacement can increase the merge ratio, but becomes off-policy stitching when it changes the prompt actually consumed during rollout. Independent-call training and best-effort merging guarantee token-level correctness but may leave redundant prefix computation, whereas tree-structured training can recover more reuse at the cost of a substantially more complex backend.

Advantage Calculation

After LLM calls are merged, one rolloutρ\rhocan produce a different number of training samplesNρN_{\rho}, known only after execution and sample construction. Retokenization is one source of this dynamic sample count. The harness can also spawn subagents, creating branches that do not share one linear history, or summarize its context, replacing the previous token prefix with a new one. These operations can split one task-level rollout into multiple training samples. This is not an edge case in practice: in our coding-agent training runs (Figure10), only 36% of rollouts on average remain as a single training sample, and each rollout yields 2.4 training samples on average.

In practice, reward is still outcome-based and is assigned to every sample within the rollout that produced it. This raises a natural question: when computing advantage, should the group statistics be computed at the rollout level or the sample level? We find that existing frameworks make different choices. verl Uni-Agent[5]and Polar[27]compute advantage at the rollout level, while slime[34]and AReaL[28]compute it at the sample level.

Figure4gives a concrete example. Suppose Rollout 1 and Rollout 2 are generated from the same prompt in one GRPO[23]group, with Rollout 1 receiving reward 1 and Rollout 2 receiving reward 0. In traditional RL (left), each rollout maps to exactly one training sample, and computing the baseline asr¯=(1+0)/2=1/2\bar{r}=(1+0)/2=1/2is uncontroversial. In harnessed agentic RL (right), Rollout 1 has three samples (Sample 1, Sample 2, and Sample 3) while Rollout 2 remains a single sample (Sample 4): rollout-level advantage calculation still gives baseliner¯rollout=(1+0)/2=1/2\bar{r}_{\mathrm{rollout}}=(1+0)/2=1/2, whereas sample-level advantage calculation instead givesr¯sample=(1+1+1+0)/4=3/4\bar{r}_{\mathrm{sample}}=(1+1+1+0)/4=3/4.

Figure 4:Traditional agentic RL, where each rollout is one training sample (left), versus harnessed agentic RL, where a rollout can expand into a dynamic number of samples that inherit its reward (right).We believe rollout-level advantage is the more principled choice, for the following reasons. Retokenization is an incidental phenomenon, and advantage assignment should not change simply because retokenization happened to split a rollout into more samples. Likewise, subagent spawning and context summarization are internal operations of the harness and should not be allowed to change the baseline of the entire group. Future work may still be needed to design better credit assignment across the samples within a rollout.

Loss Normalization

Dynamic sample counts also make loss normalization a nontrivial choice. Consider a training batch ofRRrollouts, where rolloutρ\rhoproducesNρN_{\rho}samples, and samplejjof rolloutρ\rhohasLρ,jL_{\rho,j}response tokens with per-token lossℓρ,j,t\ell_{\rho,j,t}fort=1,…,Lρ,jt=1,\ldots,L_{\rho,j}. Most frameworks directly reuse a sample-level normalization inherited from traditional agentic RL. The first is the token-mean loss used by DAPO[33], which sums the loss over every token in the batch and normalizes by the total number of response tokens:

ℒtoken​-​mean=∑ρ=1R∑j=1Nρ∑t=1Lρ,jℓρ,j,t∑ρ=1R∑j=1NρLρ,j.\mathcal{L}_{\mathrm{token\text{-}mean}}=\frac{\sum_{\rho=1}^{R}\sum_{j=1}^{N_{\rho}}\sum_{t=1}^{L_{\rho,j}}\ell_{\rho,j,t}}{\sum_{\rho=1}^{R}\sum_{j=1}^{N_{\rho}}L_{\rho,j}}.(14)The second is the seq-mean-token-mean loss used by GRPO[23], which first averages the loss within each sample and then averages these sample means uniformly over all samples in the batch:

ℒseq​-​mean=1∑ρ=1RNρ​∑ρ=1R∑j=1Nρ1Lρ,j​∑t=1Lρ,jℓρ,j,t.\mathcal{L}_{\mathrm{seq\text{-}mean}}=\frac{1}{\sum_{\rho=1}^{R}N_{\rho}}\sum_{\rho=1}^{R}\sum_{j=1}^{N_{\rho}}\frac{1}{L_{\rho,j}}\sum_{t=1}^{L_{\rho,j}}\ell_{\rho,j,t}.(15)slime[34]implements a rollout-level token-mean loss, which first pools all response tokens of a rollout together and then averages uniformly over rollouts:

ℒrollout​-​mean=1R​∑ρ=1R∑j=1Nρ∑t=1Lρ,jℓρ,j,t∑j=1NρLρ,j.\mathcal{L}_{\mathrm{rollout\text{-}mean}}=\frac{1}{R}\sum_{\rho=1}^{R}\frac{\sum_{j=1}^{N_{\rho}}\sum_{t=1}^{L_{\rho,j}}\ell_{\rho,j,t}}{\sum_{j=1}^{N_{\rho}}L_{\rho,j}}.(16) We provide a more concrete example in Figure5. Suppose a batch contains three rollouts: Rollout A produces two samplesA1A_{1}andA2A_{2}with response lengths 50 and 100; Rollout B produces three samplesB1B_{1},B2B_{2}, andB3B_{3}, each with response length 30; Rollout C produces a single sampleC1C_{1}with response length 40. LetA1,A2,B1,B2,B3,C1A_{1},A_{2},B_{1},B_{2},B_{3},C_{1}also denote the sum of per-token losses within each sample, i.e.A1=∑tℓA,1,tA_{1}=\sum_{t}\ell_{A,1,t}, and so on. Then:

• •ℒtoken​-​mean=(A1+A2+B1+B2+B3+C1)/(50+100+30+30+30+40)\mathcal{L}_{\mathrm{token\text{-}mean}}=(A_{1}+A_{2}+B_{1}+B_{2}+B_{3}+C_{1})/(50+100+30+30+30+40).
• •ℒseq​-​mean=16​(A1/50+A2/100+B1/30+B2/30+B3/30+C1/40)\mathcal{L}_{\mathrm{seq\text{-}mean}}=\tfrac{1}{6}(A_{1}/50+A_{2}/100+B_{1}/30+B_{2}/30+B_{3}/30+C_{1}/40).
• •ℒrollout​-​mean=13​((A1+A2)/(50+100)+(B1+B2+B3)/(30+30+30)+C1/40)\mathcal{L}_{\mathrm{rollout\text{-}mean}}=\tfrac{1}{3}\bigl((A_{1}+A_{2})/(50+100)+(B_{1}+B_{2}+B_{3})/(30+30+30)+C_{1}/40\bigr).

Figure 5:An example batch with three rollouts of different sample counts and response lengths.For loss normalization, we hold the same view as for advantage calculation: sample count should not be allowed to affect gradient normalization, because it is often driven by incidental factors such as retokenization. Under this view, the seq-mean-token-mean loss in Equation15is problematic because it varies with how many samples a single rollout happens to produce, and in general gives disproportionately more weight to rollouts with more samples. We therefore believe the token-mean loss in Equation14and the rollout-level token-mean loss in Equation16are more principled in theory. In practice, however, we find that the token-mean loss is sensitive to long sequences: when many long negative samples appear in a batch, it can cause instability later in training. We therefore prefer the rollout-level token-mean loss in Equation16.

Training Backend Complexity

Dynamic sample counts also complicate the interface with the training backend. The number and lengths of samples in a rollout batch are known only after harness execution and sample construction. In contrast, the number of training GPUs and the data-, tensor-, and pipeline-parallel configuration are typically fixed throughout training. The backend must therefore map a variable workload onto a fixed set of workers at every iteration.

After sample construction, the backend may flatten the resulting sequences into a physical tensor batch, but this transformation must preserve their statistical provenance. In particular, every sequence should retain its rollout identifier and prompt-group identifier:

ℬtrain=⋃ρ∈ℬrollout{(Sρ,j,ρ,gρ)|1≤j≤Nρ},\mathcal{B}_{\mathrm{train}}=\bigcup_{\rho\in\mathcal{B}_{\mathrm{rollout}}}\left\{\bigl(S_{\rho,j},\rho,g_{\rho}\bigr)\;\middle|\;1\leq j\leq N_{\rho}\right\},(17)whereSρ,jS_{\rho,j}is thejj-th sequence constructed from rolloutρ\rhoandgρg_{\rho}identifies the group of rollouts sampled from the same prompt. Flattening changes only the physical representation; it must not change rollout membership or cause a rollout to receive additional statistical weight merely because it produced more sequences.

Rollout boundaries also constrain batch scheduling. Row-based tensor batches, data-parallel partitions, and micro-batch schedules cannot be planned from the prompt or rollout count alone becauseNρN_{\rho}is known only after execution. Moreover, sequences from one rollout should remain in the same optimizer update. Splitting them across updates would evaluate different parts of one rollout under different policy versions, introducing within-rollout policy skew. A backend must therefore balance token workload across fixed workers while preserving these rollout-level statistical and update boundaries.

System Design

When the trainer and agent harness are disaggregated, no single process owns the complete rollout lifecycle. The trainer owns model inference and optimization, while the harness owns context construction, control flow, tool use, and environment interaction. Agent execution may run remotely, persist beyond any single API request or worker process, and fail independently of the training process. Operationalizing this architecture therefore requires a lightweight control plane that coordinates durable rollout state, external execution, partial failures, and resource usage without pulling harness logic back into the trainer.

Agent Lightning v1.0 builds this control plane around a declarative rollout abstraction and a reconciliation loop. The trainer declares rollouts through the API Gateway, which serves as the source of truth for lifecycle state and append-only events. The Rollout Controller continuously reconciles this state with agent executions running as Kubernetes Jobs or local processes. This separation makes Kubernetes an interchangeable execution backend rather than part of the rollout abstraction itself.

The same control plane provides explicit reliability and observability semantics across the service boundary. Control-plane operations are idempotent, generation attempts are recorded and resolved explicitly, and a rollout identifier links model requests, rewards, custom events, and execution logs into one diagnostic record. Finally, the API Gateway coordinates inference admission during collocated asynchronous RL, allowing rollout and weight update to time-share one GPU pool without exposing phase switches to the external harness.

As shown in Figure1, Agent Lightning v1.0 bridges the training cluster (a GPU cluster running model inference and training) and the agent execution cluster through three components. The API Gateway is an API service that stores rollouts, models, and events, and forwards LLM calls from agent harnesses to the model endpoints the trainer has registered. The Rollout Controller manages agent execution on top of a Kubernetes cluster (or a local process pool), polling rollouts from the API Gateway and launching the corresponding agent tasks. The Customized Trainer, built on top of VERL[24], registers rollouts with the API Gateway, waits for the Rollout Controller to drive them to completion, and then retrieves their recorded events to assemble training samples. Through this chain, the trainer only creates rollouts and collects trajectories, any harness can connect by switching its LLM endpoint to the proxy, and training and execution resources can be provisioned independently and even run in different locations.

Agent Lightning v1.0 is designed to be as simple as possible: the whole system is implemented in approximately 3,500 lines of code, with each component having a clear responsibility. Agent Lightning v1.0 also incorporates our own design choices for the challenges discussed in Section2, such as rollout-level reward and advantage calculation and rollout-level loss normalization.

We describe each component in detail in AppendixA, and highlight several features in the following sections.

Collocated Async RL

In agentic RL, the synchronous RL setup is slow: all rollouts in a batch must finish before the training step can update the model, so the training step must wait for the slowest rollout in the batch, leaving many GPUs idle. Asynchronous RL, proposed by AReaL[6], solves the GPU idling problem by splitting rollout and update onto two separate pools of machines, each occupying different GPUs, so the rollout GPUs can keep working while the update is running. However, we find that this requires more GPUs overall, which demands more resources than smaller teams may be able to afford. It also requires separately managing a rollout queue and an update queue, which is more complex because the two queues can progress at different rates in practice.

Figure 6:Sync RL, async RL, and our collocated async RL. Collocated async RL shares the same GPUs between rollout and update while still avoiding the need to wait for the slowest rollout.We instead proposecollocated async, illustrated in Figure6. In collocated async RL, rollout and weight update share the same pool of GPUs. Once enough rollout data has been collected, the update step begins. The API Gateway simultaneously stops accepting new requests and waits for the current ones to complete. If a new request arrives afterward, the Gateway pauses it until the system re-enters the rollout phase. As a result, the switch is invisible to the agent harness.

Collocated async lets the same GPUs be time-shared between rollout and training, so we use fewer GPUs than async RL, while also avoiding the wait for the slowest rollout. In our experiments, collocated async RL achieves roughly a 2x end-to-end speedup over synchronous RL while also using fewer GPUs.

Network Issues

In harnessed agentic RL, agents are separate from the training engine, which causes network issues: calls between the two sides travel over the network and are not always reliable. Two kinds of calls are affected: calls from the Trainer or the Rollout Controller to the API Gateway, and calls from the agent harness to the LLM inference endpoint through the proxy. Both can experience network interruptions, and the calling side commonly retries after such a failure. We address this with two measures.

Idempotent API Gateway endpoints.

We design every rollout API Gateway endpoint to be idempotent, so that repeating the same call any number of times has the same effect as calling it once. This lets a caller retry freely after a network failure without worrying that the retry itself will corrupt state.

Deduplication of repeated LLM API calls.

A retried LLM API call cannot be made idempotent in the same way, since each retry is a new generation request and may return a different response. Instead, when the Customized Trainer assembles training samples, it deduplicatesmodel_requestevents that share the same prompt: if a rollout recorded multiple calls with an identical prompt, only the last (most recent) call is kept, and the earlier ones, which correspond to retried or superseded calls, are discarded.

Kubernetes Integration

During the rollout phase of RL training, many agents must run concurrently, which requires substantial compute resources. Existing harnessed agentic RL frameworks commonly turn to commercial sandbox services to meet this demand. For example, verl Uni-Agent[5]launches agent execution on managed sandbox offerings such as Modal Sandbox and Volcano veFaas, while slime[34]relies on E2B. These services provide convenient, ready-to-use sandboxing, but they can be expensive at the scale of RL training.

Agent Lightning v1.0 instead runs agents directly on a Kubernetes cluster through its Rollout Controller. Each agent execution is scheduled as a standard Kubernetes Job, so users can rely entirely on self-hosted or on-premise compute rather than a commercial sandbox provider. This avoids the recurring cost of commercial sandbox services and keeps the entire training stack open-source.

Monitoring

During training, we find that agents themselves can run into problems such as reward hacking, bad agent behavior, and network connectivity issues. Manually inspecting rollouts to catch these problems is inconvenient, so the Customized Trainer includes a monitoring system that records training and validation rollouts, together with pod-level logs from Kubernetes, described further in AppendixA. This system lets us use AI agents to automatically identify such issues, and we have indeed found several reward-hacking examples this way, as shown in Section4.3.2.

Experiments

We evaluate Agent Lightning v1.0 in three practical agent training settings: search, general instruction following, and coding. We follow the experimental setup of Search-R1[11]to train search agents and the setup of LLM-in-Sandbox[4]to train general instruction-following agents. For coding agents, we build our training data from SWE-smith[30].

We find that existing frameworks rarely provide a complete, reproducible coding-agent training example, possibly because of the complexity of data cleaning, the difficulty of environment setup, and the substantial computing resources required. We therefore focus on the coding agent and describe its training process in detail, aiming to provide a complete and reproducible example that requires only modest resources.

Search Agent

We follow the experimental setting of Search-R1[11]to train a search agent that interleaves reasoning with search-engine queries and uses retrieved passages to answer knowledge-intensive questions. We use Llama-3.2-3B-Instruct[7]as the policy model and optimize it with GRPO[23]. We train on the training split of HotpotQA[31]. For evaluation, we sample 50 examples from each of HotpotQA, 2WikiMultiHopQA[8], MuSiQue[26], Bamboogle[21], TriviaQA[12], and Natural Questions[13]. We set the training batch size to 512, sample 4 rollouts per prompt, and evaluate the model every 10 training steps. We use exact match (EM) as the reward metric.

Figure7summarizes the training dynamics. The mean training reward increases steadily. On the validation set, the reward improves from 25.1% to 41.7%, an absolute 16.6% gain.

Figure 7:Search-agent training dynamics. From left to right: mean training reward and mean validation reward.

General Instruction-Following Agent

We follow the experimental setting of LLM-in-Sandbox[4]to train a general instruction-following agent. The agent solves diverse non-coding tasks by using a computer sandbox to access external resources, manage files, and execute code. We use the agent harness provided by the original authors. We use Qwen3-4B-Instruct-2507[29]as the policy model and optimize it with RLOO[1]. We use the dataset released by Instruction Pre-Training[3]and split it into 80% for training and 20% for evaluation. We set the training batch size to 8, sample 8 rollouts per prompt, and evaluate the model every 20 training steps.

Figure8shows that the batch-level training reward is noisy, whereas the validation reward exhibits a clear upward trend. It improves from 51.9% to 70.2%, an absolute 18.3% improvement.

Figure 8:General instruction-following agent training dynamics. From left to right: mean training reward and mean validation reward.

Coding Agent

We train a coding agent based on Qwen3.5-9B[22]using tasks derived from the SWE-smith dataset[30]. We use mini-SWE-agent[25]as the agent harness for interacting with repository environments, executing commands, and producing code changes. To obtain reliable training signals, we apply a detailed data-filtering pipeline, introduce safeguards against reward hacking, and implement the additional measures required for robust coding-agent training. The following sections describe these implementation details.

Dataset Preprocessing and Filtering

SWE-smith is a large-scale dataset of executable software-engineering tasks constructed by introducing bugs into real Python repositories. It contains 59,136 tasks from 128 repositories, with problem statements, code patches, and tests for verifying solutions[30]. Its Docker images occupy only 295 GB[30], substantially less than the 4 TB required by R2E-Gym[9]and the 6 TB required by SWE-Gym[20].

For each task, SWE-smith first switches the repository to the corresponding problem branch and asks the coding agent to modify the codebase. It then runs a task-specific test suite to determine whether the submitted changes resolve the problem.

We identify the following issues in the released data:

• •Among the 59,136 records, 18,033 have an empty problem statement.
• •For 1,265 records, the corresponding problem branch is missing from the provided Docker image.
• •Some tasks require large test suites. For example,python-jsonschemarequires executing more than 7,000 tests, consuming substantial CPU and memory.

We therefore remove tasks with an empty problem statement, a missing problem branch, or more than 200 tests. The remaining tasks still have a highly skewed difficulty distribution and provide limited training signal, so we apply an additional model-based difficulty filter. We run Qwen3.5-9B four times on every candidate: tasks solved in all four rollouts are removed, while tasks with both successful and failed rollouts are retained, yielding approximately 5,000 examples. To avoid making the resulting set overly easy, we additionally sample 1,000 tasks that fail in all four rollouts. The final split contains approximately 6,000 training examples and 400 test examples.

Preventing Reward Hacking

During training, we observe several reward-hacking behaviors in which the agent bypasses the intended problem-solving process and obtains the reference source code directly:

1. 1.Using Git history to locate the gold commit.
2. 2.Usingwgetorcurlto retrieve upstream source code from GitHub.
3. 3.Usingpipto download a package’s source code.
4. 4.Using Python networking libraries, such asurllib, to download source code.

We introduce two safeguards. First, we disable Git commands and hide the.gitdirectory from the agent, preventing it from inspecting commit history. Second, we enforce a Kubernetes network policy that blocks general outbound network access and permits connections only to explicitly whitelisted services. Together, these measures require the agent to solve each task using only the provided problem statement and local information.

Training Dynamics

Figure 9:Coding-agent training dynamics for Sample-level Advantage, Rollout-level Advantage, and Rollout-level Advantage + Rollout-level Norm. Left: validation reward. Right: policy entropy.Figure 10:Rollout-merging behavior for the Rollout-level Advantage + Rollout-level Norm run. Left: fraction of rollouts that yield exactly one training sample. Right: average number of training samples produced per rollout. Dashed lines mark the mean over training.As discussed in Section2, because a rollout’s sample count is often driven by incidental factors such as retokenization, both the advantage calculation (Section2.2) and the loss normalization (Section2.3) should be computed at the rollout level rather than the sample level. We validate this design choice on our coding-agent training run by comparing three settings, all using the same underlying GRPO objective:

• •Sample-level Advantage, which combines sample-level advantage with the token-mean loss in Equation14;
• •Rollout-level Advantage, which switches only the advantage calculation to the rollout level while keeping the token-mean loss; and
• •Rollout-level Advantage + Rollout-level Norm, which additionally replaces the token-mean loss with the rollout-level token-mean loss in Equation16.

Figure9compares the three settings. The last variant produces the highest observed validation reward, reaching 38.2% at step 128, compared with 35.0% for the baseline and 33.1% when only the rollout-advantage fix is applied. Its policy entropy also grows more slowly and stays more stable over training than the variant with only the rollout-advantage fix. These results suggest that loss normalization controls the entropy increase introduced by the corrected rollout advantages while improving validation reward. We also evaluate the Rollout-level Advantage + Rollout-level Norm checkpoint on SWE-bench Verified[10], where it improves from 41.8% to 56.4% at step 208. Because coding-agent trajectories vary widely in length, the rollout-level advantage variants merge trajectories into training rows wherever possible to reduce padding waste, which produces a dynamic number of training samples per rollout: on average, only 36% of rollouts remain as a single, fully merged row, and each rollout yields 2.41 training samples on average (Figure10).

Related Work

Traditional RL frameworks such as verl[24], AReaL[6], and slime[34]originally required the agent loop to be implemented directly inside the training framework, following the classic ReAct-style[32]Markov formulation in which the training engine owns the environment interaction loop. This makes it difficult to reuse independently maintained agent harnesses, such as mini-SWE-agent[25], OpenHands[19], OpenCode[18], Claude Code[2], Codex[16], OpenClaw[17], and Hermes[15], since each would need to be reimplemented inside the training stack. Our original Agent Lightning[14]work introduced a disaggregated architecture that instead connects arbitrary agent harnesses to RL training through an LLM endpoint, and this proxy-based approach has since been adopted by verl Uni-Agent[5], AReaL 2.0[28], slime v0.3.0[34], and Polar[27]. As discussed in Section2, these frameworks make different, sometimes conflicting, design choices when handling retokenization, advantage calculation, and loss normalization under a dynamic number of training samples per rollout, and they commonly rely on commercial sandbox services, such as Modal Sandbox, Volcano veFaas, and E2B, to execute agents at scale. Agent Lightning v1.0 instead runs entirely on a self-hosted Kubernetes cluster and implements the whole system in approximately 3,500 lines of code, providing a compact and transparent testbed for studying these design choices, which we validate on the search-agent, instruction-following, and coding-agent settings of Search-R1[11], LLM-in-Sandbox[4], and SWE-smith[30], respectively.

Conclusion

We characterizeharnessed agentic RL, a paradigm in which the deploy-time agent harness, not the training engine, owns the environment interaction loop, and identify the resulting challenges in retokenization, advantage calculation, loss normalization, and training backend scheduling. We present Agent Lightning v1.0, an approximately 3,500-line framework that supports arbitrary agent harnesses and embeds our rollout-level design choices for these challenges. We validate it on search, instruction-following, and coding agents, and release a complete data pipeline and reward-hacking safeguards that let RL alone improve Qwen3.5-9B on SWE-bench Verified from 41.8% to 56.4%, a gain of 14.6 percentage points, using only about 6K training examples. We release the full codebase and scripts to facilitate reproducible harnessed agentic RL research.

References

• [1](2024)Back to basics: revisiting REINFORCE style optimization for learning from human feedback in LLMs.arXiv preprint arXiv:2402.14740.External Links:Document,LinkCited by:§4.2.
• [2]Anthropic(2026)Claude code overview.Note:https://code.claude.com/docs/en/overviewAccessed: 2026-07-22Cited by:§1,§5.
• [3]D. Cheng, Y. Gu, S. Huang, J. Bi, M. Huang, and F. Wei(2024)Instruction pre-training: language models are supervised multitask learners.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp. 2529–2550.External Links:Document,LinkCited by:§4.2.
• [4]D. Cheng, S. Huang, Y. Gu, H. Song, G. Chen, L. Dong, W. X. Zhao, J. Wen, and F. Wei(2026)Computer environments elicit general agentic intelligence in LLMs.arXiv preprint arXiv:2601.16206.External Links:Document,LinkCited by:§4.2,§4,§5.
• [5]Y. Ding, B. Wen, X. Cao, Z. Zhai, G. Sheng, X. Wu, J. Li, M. Zhang, and Uni-Agent Contributors(2026)Uni-Agent: build, run, and train agents at scale.Note:https://github.com/verl-project/uni-agentAccessed: 2026-07-29Cited by:§1,§2.1,§2.2,§3.3,§5.
• [6]W. Fu, J. Gao, X. Shen, C. Zhu, Z. Mei, C. He, S. Xu, G. Wei, J. Mei, J. Wang, T. Yang, B. Yuan, and Y. Wu(2025)AReaL: a large-scale asynchronous reinforcement learning system for language reasoning.arXiv preprint arXiv:2505.24298.External Links:Document,LinkCited by:§1,§3.1,§5.
• [7]A. Grattafioriet al.(2024)The Llama 3 herd of models.arXiv preprint arXiv:2407.21783.External Links:Document,LinkCited by:§4.1.
• [8]X. Ho, A. D. Nguyen, S. Sugawara, and A. Aizawa(2020)Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps.InProceedings of the 28th International Conference on Computational Linguistics,pp. 6609–6625.External Links:Document,LinkCited by:§4.1.
• [9]N. Jain, J. Singh, M. Shetty, L. Zheng, K. Sen, and I. Stoica(2025)R2E-Gym: procedural environments and hybrid verifiers for scaling open-weights SWE agents.arXiv preprint arXiv:2504.07164.External Links:Document,LinkCited by:§4.3.1.
• [10]C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan(2024)SWE-bench: can language models resolve real-world github issues?.InInternational Conference on Learning Representations,External Links:LinkCited by:§4.3.3.
• [11]B. Jin, H. Zeng, Z. Yue, J. Yoon, S. Arik, D. Wang, H. Zamani, and J. Han(2025)Search-R1: training LLMs to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516.External Links:Document,LinkCited by:§4.1,§4,§5.
• [12]M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer(2017)TriviaQA: a large scale distantly supervised challenge dataset for reading comprehension.InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),pp. 1601–1611.External Links:Document,LinkCited by:§4.1.
• [13]T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M. Chang, A. M. Dai, J. Uszkoreit, Q. Le, and S. Petrov(2019)Natural questions: a benchmark for question answering research.Transactions of the Association for Computational Linguistics7,pp. 452–466.External Links:Document,LinkCited by:§4.1.
• [14]X. Luo, Y. Zhang, Z. He, Z. Wang, S. Zhao, D. Li, L. K. Qiu, and Y. Yang(2025)Agent lightning: train ANY AI agents with reinforcement learning.arXiv preprint arXiv:2508.03680.External Links:Document,LinkCited by:§1,§5.
• [15]Nous Research(2026)Hermes agent.Note:https://github.com/NousResearch/hermes-agentAccessed: 2026-07-22Cited by:§1,§5.
• [16]OpenAI(2026)Codex cli.Note:https://github.com/openai/codexAccessed: 2026-07-22Cited by:§1,§5.
• [17]OpenClaw Foundation(2026)OpenClaw: personal ai assistant.Note:https://github.com/openclaw/openclawAccessed: 2026-07-22Cited by:§1,§5.
• [18]OpenCode(2026)OpenCode: the open source ai coding agent.Note:https://github.com/anomalyco/opencodeAccessed: 2026-07-22Cited by:§1,§5.
• [19]OpenHands(2026)OpenHands.Note:https://github.com/OpenHands/OpenHandsAccessed: 2026-07-22Cited by:§1,§5.
• [20]J. Pan, X. Wang, G. Neubig, N. Jaitly, H. Ji, A. Suhr, and Y. Zhang(2024)Training software engineering agents and verifiers with SWE-Gym.arXiv preprint arXiv:2412.21139.External Links:Document,LinkCited by:§4.3.1.
• [21]O. Press, M. Zhang, S. Min, L. Schmidt, N. A. Smith, and M. Lewis(2023)Measuring and narrowing the compositionality gap in language models.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp. 5687–5711.External Links:Document,LinkCited by:§4.1.
• [22]Qwen Team(2026)Qwen3.5: towards native multimodal agents.External Links:LinkCited by:§1,§4.3.
• [23]Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo(2024)DeepSeekMath: pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300.External Links:Document,LinkCited by:§A.1,§2.2,§2.3,§4.1.
• [24]G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu(2025)HybridFlow: a flexible and efficient RLHF framework.InProceedings of the Twentieth European Conference on Computer Systems,pp. 1279–1297.External Links:Document,LinkCited by:§A.3,§1,§3,§5.
• [25]SWE-agent Team(2026)Mini-swe-agent: the minimal ai software engineering agent.Note:https://github.com/SWE-agent/mini-swe-agentAccessed: 2026-07-22Cited by:§1,§4.3,§5.
• [26]H. Trivedi, N. Balasubramanian, T. Khot, and A. Sabharwal(2022)MuSiQue: multihop questions via single-hop question composition.Transactions of the Association for Computational Linguistics10,pp. 539–554.External Links:Document,LinkCited by:§4.1.
• [27]B. Xu, H. Zhang, S. Zhang, S. Han, M. Liu, J. Hu, S. Diao, Z. Jin, Y. Zou, M. Demoret, J. Kautz, and Y. Dong(2026)Polar: agentic RL on any harness at scale.arXiv preprint arXiv:2605.24220.External Links:Document,LinkCited by:§1,§2.1,§2.2,§5.
• [28]R. Yan, W. Fu, J. Li, S. Xu, Z. Mei, J. Gao, J. Zhang, W. Zhang, H. Dai, X. Shen, C. He, Z. Pu, J. Mei, Z. Lin, H. Wang, Z. Ding, J. Zhang, H. Wang, R. Xu, H. Dong, Y. Jiang, Y. Wu, T. Yang, and B. Yuan(2026)Next-generation agentic reinforcement learning systems enable self-evolving agents.arXiv preprint arXiv:2607.01120.External Links:LinkCited by:§1,§2.1,§2.2,§5.
• [29]A. Yanget al.(2025)Qwen3 technical report.arXiv preprint arXiv:2505.09388.External Links:Document,LinkCited by:§4.2.
• [30]J. Yang, K. Lieret, C. E. Jimenez, A. Wettig, K. Khandpur, Y. Zhang, B. Hui, O. Press, L. Schmidt, and D. Yang(2025)SWE-smith: scaling data for software engineering agents.InAdvances in Neural Information Processing Systems,D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen (Eds.),Vol.38.External Links:LinkCited by:§4.3.1,§4.3,§4,§5.
• [31]Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning(2018)HotpotQA: a dataset for diverse, explainable multi-hop question answering.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,pp. 2369–2380.External Links:Document,LinkCited by:§4.1.
• [32]S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao(2023)ReAct: synergizing reasoning and acting in language models.InInternational Conference on Learning Representations (ICLR),Cited by:§2.1,§5.
• [33]Q. Yuet al.(2025)DAPO: an open-source LLM reinforcement learning system at scale.arXiv preprint arXiv:2503.14476.External Links:Document,LinkCited by:§2.3.
• [34]Z. Zhu, C. Xie, X. Lv, and slime Contributors(2025)Slime: an LLM post-training framework for RL scaling.Note:https://github.com/THUDM/slimeAccessed: 2026-07-29Cited by:§1,§2.1,§2.2,§2.3,§3.3,§5.

Appendix ADetailed System Design

This appendix provides a detailed description of Agent Lightning v1.0’s three components introduced in Section3: the API Gateway, the Rollout Controller, and the Customized Trainer.

API Gateway

The API Gateway is the central component of Agent Lightning v1.0, kept deliberately lightweight: a single stateful service that stores rollouts, models, and events, and exposes them through a minimal API. Figure11summarizes these objects and their relationships.

Figure 11:The objects stored by the API Gateway.##### Rollout.

Arolloutis one agent execution, identified by a unique rollout ID. It stores an input (derived from a training example), a status, and user-defined metadata. The status follows the state machine in Figure11:queuing,running, and the terminal statessucceededandfailed. Rollouts are not one-to-one with training examples: GRPO[23], for instance, generates multiple independent rollouts, each with its own ID and trajectory, from the same example.

Model.

Amodelidentifies an LLM inference endpoint by its name and address. The trainer registers models with the API Gateway, which then routes agent harness requests to the corresponding inference server.

Event.

Aneventattaches arbitrary data to a rollout. By default, Agent Lightning v1.0 records amodel_requestevent (prompt token IDs, response token IDs, and response log probabilities) for every LLM interaction, and arewardevent that the agent typically reports once at the end of the rollout with a scalar reward. Users can also define custom event types.

Table1lists the API Gateway’s endpoints, which fall into two APIs: the rollout API and the proxy API.

Table 1:API Gateway endpoints.

Rollout API.

The trainer creates rollouts through this API. The Rollout Controller polls queued rollouts, launches agents, and updates their status as execution progresses. Rewards and other user-defined events are uploaded the same way.

Proxy API.

This API forwards LLM calls from agent harnesses to the model endpoints the trainer has registered. An agent harness only needs to point its OpenAI-compatible client at the proxy. Since the proxy path embeds the rollout ID, every call can be attributed to its rollout automatically. Each call’s prompt token IDs, response token IDs, and log probabilities are recorded as amodel_requestevent, which the trainer later exports for training.

This simple API fully decouples RL training from agent execution: the trainer only creates rollouts and collects trajectories, any harness can connect by switching its LLM endpoint to the proxy, and training and execution resources can be provisioned independently and even run in different locations.

Rollout Controller

The Rollout Controller manages agent execution on top of Kubernetes. As shown in Figure12, it periodically fetches active rollouts from the API Gateway, launches the corresponding agent tasks, monitors them, and reports status back through the Gateway API. Its primary backend is the K8s Reconciler, which targets an open-source Kubernetes cluster. For debugging purposes, it also provides a Local Reconciler, which targets a local process pool.

Figure 12:The Rollout Controller reconciles rollout status in the API Gateway with agent executions running as Kubernetes Jobs or local processes.##### K8s Reconciler.

For eachqueuingrollout without an existing Kubernetes Job, it creates one from a user-provided template. It watches the Kubernetes API for Job updates so terminal states propagate with low latency, and periodically lists all managed Jobs to recover any watch events that were missed – the standard Kubernetes controller pattern.

Local Reconciler.

It launches agents in a local process pool instead. Since it owns the process handles directly, periodic polling alone is enough, and no separate watch mechanism is needed.

State Consistency.

The API Gateway’s rollout status is the ground truth. The execution state observed from Kubernetes may lag behind it because of network failures or delayed updates. The K8s Reconciler simply retries synchronization on its next cycle, so the two sides converge once communication resumes. This design guarantees only best-effort eventual consistency.

Customized Trainer

Built on top of VERL[24], the Customized Trainer keeps a lightweight design while connecting the training backend to the API Gateway: at each step it registers rollouts for the current batch, waits for them to reach a terminal state, then retrieves theirmodel_requestandrewardevents and assembles them into training samples. It consists of the following two components.

Dedicated Sample Adapter.

The adapter reflects our design choices for the challenges described in Section2.

• •Sample merging.We keep the API Gateway as simple as possible: it does not maintain a server-side request buffer, which keeps training consistent with deployment. The adapter merges two consecutive model requests into one training sample only when the later prompt is an exact token-level prefix match of the earlier request and response.
• •Advantage calculation.The adapter computes baselines and advantages at the rollout level, which we believe is the more principled choice.
• •Loss normalization.The adapter implements the rollout-level token-mean loss discussed in Section2, normalizing so every rollout carries equal weight regardless of its sample count.

Trajectory Monitoring.

Because agentic training can produce reward hacking, runaway trajectories, or silent failures, the trainer exposes every training and validation rollout’s input, status, model requests, rewards, token/turn statistics, and custom events, with execution logs kept in Kubernetes, so we can inspect them manually or with AI agents to diagnose unusual behavior.