Direct Diversity Optimization for Diverse Successful Trajectories in Preference Post-Training

# Direct Diversity Optimization for Diverse Successful Trajectories in Preference Post-Training
Source: [https://arxiv.org/html/2609.10052](https://arxiv.org/html/2609.10052)
###### Abstract

LLM agents for sequential decision tasks are often post\-trained with trajectory\-level outcome labels, but such labels provide little supervision for preserving multiple successful branches from the same decision state\. We study this problem as successful strategy coverage: how broadly a model realizes distinct successful strategies under a fixed rollout budget\. We present Direct Diversity Optimization \(DDO\), an offline post\-training method that combines Divergence\-Tree Collection \(DTC\) with the Reference\-Relative Target\-Odds Objective \(RTO\)\.111Code is available at[https://github\.com/koguma00/direct\_diverse\_optimization](https://github.com/koguma00/direct_diverse_optimization)\.DTC constructs state\-aligned branch sets rooted at shared decision states, and RTO trains the model to match reference\-relative targets over successful alternatives\. DDO achieves the strongest task success and successful strategy coverage among the compared post\-training methods across BabyAI, BabaIsAI, and WebShop\. It also achieves the highest recovery rate after local action replacement and higher task success and coverage than successful\-only imitation and decoding\-time diversification controls\.

## 1Introduction

Recent work on Large Language Models \(LLMs\) commonly adapts pretrained models to downstream tasks through post\-training\([Ouyang et al\., 2022](https://arxiv.org/html/2609.10052#bib.bib22);[Touvron et al\., 2023](https://arxiv.org/html/2609.10052#bib.bib23);[Grattafiori et al\., 2024](https://arxiv.org/html/2609.10052#bib.bib1);[Yang et al\., 2025](https://arxiv.org/html/2609.10052#bib.bib2)\)\. The same approach is now applied to LLM agents in sequential decision tasks such as web navigation, embodied control, and tool use\([Zeng et al\., 2024](https://arxiv.org/html/2609.10052#bib.bib6);[Song et al\., 2024](https://arxiv.org/html/2609.10052#bib.bib5);[Wei et al\., 2025](https://arxiv.org/html/2609.10052#bib.bib14)\)\. In these tasks, dense per\-step supervision is rarely available, and post\-training typically relies on trajectory\-level success\-or\-failure feedback\. Under such coarse supervision, standard post\-training tends to narrow the model’s output distribution\([Kirk et al\., 2024](https://arxiv.org/html/2609.10052#bib.bib8);[Slocum et al\., 2025](https://arxiv.org/html/2609.10052#bib.bib10);[Lanchantin et al\., 2025](https://arxiv.org/html/2609.10052#bib.bib12)\)\. In sequential decision tasks, this narrowing can cause the policy to collapse onto a single successful trajectory even when several viable alternatives remain\.

![Refer to caption](https://arxiv.org/html/2609.10052v1/intro_emnlp_ko.png)Figure 1:A motivating online\-shopping example: Model A, a standard post\-trained model, completes the purchase through a single trajectory, while Model B retains multiple successful trajectories that can be used as fallbacks when conditions change\.[Figure1](https://arxiv.org/html/2609.10052#S1.F1)illustrates this collapse in an online\-shopping task where an LLM agent must buy a laptop charger under a price constraint\. A standard post\-trained model \(Model A\) commits to a single brand\-page trajectory, whereas a model trained to retain multiple successful trajectories \(Model B\) can succeed through distinct search, filtering, and comparison strategies\. Both models succeed under normal conditions\. When the brand\-page trajectory is blocked, Model A has no fallback, whereas Model B retains an alternative path to success\.

The contrast between Models A and B highlights a limitation of standard preference post\-training: it lacks state\-aligned supervision over alternative branches\. Trajectory\-level success labels identify a successful trajectory, but do not show the outcomes of alternative actions at the same decision state\. Consequently, DPO\-style training\([Rafailov et al\., 2023](https://arxiv.org/html/2609.10052#bib.bib4)\)with success–failure trajectory preferences distinguishes successful from failed trajectories but provides no explicit target for allocating probability among multiple successful branches from the same state\. The learned policy is therefore prone to placing most of its probability mass on one successful branch, leaving little for the remaining alternatives\.

To address this limitation, we present Direct Diversity Optimization \(DDO\), an offline post\-training method that couples a data\-construction procedure with a reference\-relative optimization objective\. Given a successful source trajectory, Divergence\-Tree Collection \(DTC\) restores an intermediate decision state and constructs an outcome\-labeled branch set containing the source and alternative branches\. Reference\-Relative Target\-Odds \(RTO\) replaces the single\-winner preference target with a reference\-relative target distribution over successful branches and fits the model to match its induced pairwise odds\. Together, DTC and RTO convert trajectory\-level outcomes into preference supervision over multiple successful alternatives from the same decision state\.

We evaluate DDO on BabyAI, BabaIsAI, and WebShop, which cover navigation, object interaction, multi\-stage instruction following, rule manipulation, and web interaction\. Across the three benchmarks, DDO achieves the strongest joint performance on task success and successful strategy coverage among the compared post\-training methods\. These gains also translate into more robust recovery after local action replacement\. DDO also achieves higher task success and coverage than successful\-only imitation and decoding\-time diversification controls\. Component analyses show complementary roles for DTC and RTO, whose combination yields the largest joint gains\.

## 2Related Work

### 2\.1Diversity\- and tie\-aware post\-training\.

Preference\- or reward\-based post\-training is now a standard mechanism for adapting LLMs with feedback\([Stiennon et al\., 2020](https://arxiv.org/html/2609.10052#bib.bib24);[Ouyang et al\., 2022](https://arxiv.org/html/2609.10052#bib.bib22);[Bai et al\., 2022](https://arxiv.org/html/2609.10052#bib.bib25);[Rafailov et al\., 2023](https://arxiv.org/html/2609.10052#bib.bib4)\)\. However, it has also been shown to narrow the output distribution of trained models:[Kirk et al\. \(2024\)](https://arxiv.org/html/2609.10052#bib.bib8)and[Padmakumar and He \(2024\)](https://arxiv.org/html/2609.10052#bib.bib9)report that RLHF and DPO reduce output diversity relative to the base model, and[Slocum et al\. \(2025\)](https://arxiv.org/html/2609.10052#bib.bib10)show that KL\-regularized preference learning can amplify dominant preferences\. To mitigate this collapse, recent approaches preserve multiple acceptable responses in the preference signal\. DivPO encourages diverse preferred responses through diversity\-aware preference updates\([Lanchantin et al\., 2025](https://arxiv.org/html/2609.10052#bib.bib12)\), while tie\-aware DPO variants allow similarly preferred responses to be modeled as ties rather than forcing an arbitrary winner–loser direction\([Chen et al\., 2025](https://arxiv.org/html/2609.10052#bib.bib11)\)\. These approaches reduce the pressure to collapse acceptable outputs into a single mode, but they operate at the prompt\-response or pairwise\-comparison level\. Related goals have long been studied in reinforcement learning: maximum\-entropy RL encourages stochastic high\-return behavior, while skill\-discovery and quality\-diversity methods learn distinct skills or policy repertoires\([Haarnoja et al\., 2018](https://arxiv.org/html/2609.10052#bib.bib26);[Eysenbach et al\., 2019](https://arxiv.org/html/2609.10052#bib.bib27);[Pugh et al\., 2016](https://arxiv.org/html/2609.10052#bib.bib28);[Pierrot et al\., 2022](https://arxiv.org/html/2609.10052#bib.bib29)\)\. DDO targets offline preference post\-training for sequential LLM agents, using state\-aligned outcome\-labeled branches and reference\-relative targets to retain multiple observed successful branches\.

### 2\.2Post\-training LLMs for sequential tasks\.

Preference\-based post\-training has been extended to sequential decision tasks using branching rollouts or step\-level feedback\. Chain of Preference Optimization\([Zhang et al\., 2024](https://arxiv.org/html/2609.10052#bib.bib7)\)constructs preference data from alternative reasoning trajectories, while ETO\([Song et al\., 2024](https://arxiv.org/html/2609.10052#bib.bib5)\)contrasts successful and failed exploration trajectories at the trajectory level\. Other methods provide per\-step signal: IPR\([Xiong et al\., 2024](https://arxiv.org/html/2609.10052#bib.bib13)\)builds step\-level contrastive pairs along expert trajectories, and AgentPRM\([Xi et al\., 2026](https://arxiv.org/html/2609.10052#bib.bib15)\)trains a process reward model for step\-wise progress\. Across these methods, the supervision signal primarily distinguishes better from worse trajectories or steps\. When multiple branches from the same state are successful, they are not explicitly modeled as a set of valid alternatives; they are typically paired against failures, folded into independent comparisons, or scored by a progress signal\.

DDO uses the labeled branch set at a decision state as the supervision unit\. It preserves multiple successful branches as valid positives while distinguishing them from failed branches, without forcing an arbitrary ranking among the successful ones\. This couples diversity preservation directly with task success\.

![Refer to caption](https://arxiv.org/html/2609.10052v1/method_emnlp_ko.png)Figure 2:DDO pipeline\. DTC constructs state\-aligned branch sets, and RTO defines a reference\-relative target distribution whose induced pairwise targets are optimized with a soft logistic objective\.

## 3Method

We introduce Direct Diversity Optimization \(DDO\), an offline post\-training method for retaining multiple successful strategies in LLM agents trained from trajectory\-level outcome labels\. DDO couples Divergence\-Tree Collection \(DTC;[Section3\.2](https://arxiv.org/html/2609.10052#S3.SS2)\), which constructs state\-aligned branch sets with rollout outcome labels, with the Reference\-Relative Target\-Odds Objective \(RTO;[Section3\.3](https://arxiv.org/html/2609.10052#S3.SS3)\), which fits the model to reference\-relative targets over successful branches\. This coupling transforms trajectory\-level outcome supervision into same\-state branch comparisons and a target distribution over successful alternatives\.[Figure2](https://arxiv.org/html/2609.10052#S2.F2)summarizes the pipeline\.

### 3\.1Problem Setting and Notations

We address post\-training of LLM agents on sequential decision tasks with textual observations and executable actions\. At each steptt, the agent observes a textual statextx\_\{t\}that includes the current observation and the recent interaction history, and produces an outputyty\_\{t\}; the environment executes the action parsed fromyty\_\{t\}to produce the next state\. A trajectoryτ=\(x0,y0,…,yT−1,xT\)\\tau=\(x\_\{0\},y\_\{0\},\\ldots,y\_\{T\-1\},x\_\{T\}\)is judged only at termination, receiving a binary resultR⁡\(τ\)∈\{0,1\}R\(\\tau\)\\in\\\{0,1\\\}provided by the environment\. This trajectory\-level signal is the sole supervision; no per\-step rewards or learned reward model are assumed\. We writeπθ​\(y∣x\)\\pi\_\{\\theta\}\(y\\mid x\)for the trainable model andπref​\(y∣x\)\\pi\_\{\\mathrm\{ref\}\}\(y\\mid x\)for the frozen reference model\.

### 3\.2Divergence\-Tree Collection

DTC takes a successful trajectory and produces, at selected decision steps, branch sets rooted at a shared state with rollout outcome labels\. For a source trajectoryτ\\tau, the candidate divergence steps are the interior decision points1,…,\|τ\|−11,\\ldots,\|\\tau\|\-1\. Under a fixed collection budget, DTC selects divergence steps to cover different depths along the source, so that branch roots are spread along the trajectory rather than concentrated near its initial prefix\. At each selected divergence step, DTC restores the corresponding decision state as the shared statexx\([Algorithm1](https://arxiv.org/html/2609.10052#alg1)\)\. It keeps the source output as one branch and queries an expert model for alternative outputs whose parsed actions differ from the source action\. Each retained output is executed to termination by the expert model and receives a binary success label\. If the state cannot be restored or no distinct valid alternative is obtained, DTC skips that state\.

Algorithm 1Divergence\-Tree Collection \(DTC\)\. Implementation details ofSelect,Restore,Alt, andRejectare provided in Appendix[B\.2](https://arxiv.org/html/2609.10052#A2.SS2)\.1:Input:successful sources

𝒯\+\\mathcal\{T\}^\{\+\}, expert model

EE, budgets

Kd,KaK\_\{d\},K\_\{a\}
2:Output:DTC records

𝒟DTC\\mathcal\{D\}\_\{\\mathrm\{DTC\}\}
3:

𝒟DTC←∅\\mathcal\{D\}\_\{\\mathrm\{DTC\}\}\\leftarrow\\emptyset
4:for

τ∈𝒯\+\\tau\\in\\mathcal\{T\}^\{\+\}do

5:

D←Select​\(\{1,…,\|τ\|−1\},Kd\)D\\leftarrow\\textsc\{Select\}\(\\\{1,\\ldots,\|\\tau\|\-1\\\},K\_\{d\}\)
6:for

d∈Dd\\in Ddo

7:

x←Restore​\(τ,d\)x\\leftarrow\\textsc\{Restore\}\(\\tau,d\)
8:if

x=⊥x=\\botthen

9:continue

10:

B←\{jsrc\}B\\leftarrow\\\{j\_\{\\mathrm\{src\}\}\\\},

yjsrc←ydτy\_\{j\_\{\\mathrm\{src\}\}\}\\leftarrow y\_\{d\}^\{\\tau\},

rjsrc←1r\_\{j\_\{\\mathrm\{src\}\}\}\\leftarrow 1
11:

𝒴←Alt​\(E,x,ydτ,Ka\)\\mathcal\{Y\}\\leftarrow\\textsc\{Alt\}\(E,x,y\_\{d\}^\{\\tau\},K\_\{a\}\)
12:for

y∈𝒴y\\in\\mathcal\{Y\}do

13:if

Reject​\(y,B\)\\textsc\{Reject\}\(y,B\)then

14:continue

15:

r←R​\(Rollout​\(E,x,y\)\)r\\leftarrow R\(\\textsc\{Rollout\}\(E,x,y\)\)
16:add new

jjwith

\(yj,rj\)←\(y,r\)\(y\_\{j\},r\_\{j\}\)\\leftarrow\(y,r\)to

BB
17:if

B≠\{jsrc\}B\\neq\\\{j\_\{\\mathrm\{src\}\}\\\}then

18:add

Record​\(x,B\)\\textsc\{Record\}\(x,B\)to

𝒟DTC\\mathcal\{D\}\_\{\\mathrm\{DTC\}\}
19:return

𝒟DTC\\mathcal\{D\}\_\{\\mathrm\{DTC\}\}

Each collected branch set is represented as one DTC record:

C=\(x,B,\{\(yj,rj\)\}j∈B\)\.C=\(x,B,\\\{\(y\_\{j\},r\_\{j\}\)\\\}\_\{j\\in B\}\)\.\(1\)Here,xxis the shared decision state,BBis a finite set of branch indices,yjy\_\{j\}is the LLM output of branchjj, andrj∈\{0,1\}r\_\{j\}\\in\\\{0,1\\\}is the binary result obtained after executing that branch\. For each record we partitionBBinto successful branchesS\+=\{j:rj=1\}S^\{\+\}=\\\{j:r\_\{j\}=1\\\}and non\-successful branchesS0=\{j:rj=0\}S^\{0\}=\\\{j:r\_\{j\}=0\\\}\. RTO usesS\+S^\{\+\}for target\-odds estimation andS0S^\{0\}for boundary comparisons between successful and non\-successful observed branches\.

### 3\.3Reference\-Relative Target\-Odds Objective

Pairwise preference objectives that contrast a single winner against a single loser with hard binary labels risk collapsingπθ\\pi\_\{\\theta\}onto a single successful strategy\. The Reference\-Relative Target\-Odds \(RTO\) objective addresses this by aligning the log\-oddsπθ\\pi\_\{\\theta\}assigns to pairs of successful branches with the log\-odds prescribed by a fixed distributionqqoverS\+S^\{\+\}\. For each recordC=\(x,B,\{\(yj,rj\)\}j∈B\)C=\(x,B,\\\{\(y\_\{j\},r\_\{j\}\)\\\}\_\{j\\in B\}\), RTO constructsqqfromπref\\pi\_\{\\mathrm\{ref\}\}and matches the two log\-odds in reference\-relative form, aligningπθ\\pi\_\{\\theta\}’s shift overπref\\pi\_\{\\mathrm\{ref\}\}withqq’s shift overqrefq\_\{\\mathrm\{ref\}\}\. Two pair types are handled asymmetrically: success–success pairs useqqto set the desired log\-odds withinS\+S^\{\+\}, and success–failure pairs acrossS\+S^\{\+\}andS0S^\{0\}use task success labels to order successful branches above failed ones\.

#### Branch distribution\.

OnS\+S^\{\+\}, define the renormalized reference distributionqrefq\_\{\\mathrm\{ref\}\}as

qref​\(j\)=πref​\(yj∣x\)∑h∈S\+πref​\(yh∣x\)\.q\_\{\\mathrm\{ref\}\}\(j\)=\\frac\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{j\}\\mid x\)\}\{\\sum\_\{h\\in S^\{\+\}\}\\pi\_\{\\mathrm\{ref\}\}\(y\_\{h\}\\mid x\)\}\.\(2\)The distributionqqis defined as

q⁡\(j\)=qref​\(j\)α∑h∈S\+qref​\(h\)α,q\(j\)=\\frac\{q\_\{\\mathrm\{ref\}\}\(j\)^\{\\alpha\}\}\{\\sum\_\{h\\in S^\{\+\}\}q\_\{\\mathrm\{ref\}\}\(h\)^\{\\alpha\}\},\(3\)whereα∈\[0,1\]\\alpha\\in\[0,1\]controls how stronglyqqpreservesπref\\pi\_\{\\mathrm\{ref\}\}’s relative weighting among successful branches: smallerα\\alphaflattens these differences, largerα\\alphakeeps them\. Atα=0\\alpha=0,qqis uniform onS\+S^\{\+\}, providing maximal within\-success diversity pressure; atα=1\\alpha=1,q=qrefq=q\_\{\\mathrm\{ref\}\}and the success–success targets reduce to preservingπref\\pi\_\{\\mathrm\{ref\}\}’s relative odds among successful branches\. Bothqqandqrefq\_\{\\mathrm\{ref\}\}are computed once fromπref\\pi\_\{\\mathrm\{ref\}\}and held fixed during training\. We useα=0\.5\\alpha=0\.5unless otherwise stated\.

#### Distribution margin\.

For an ordered pairu,v∈S\+u,v\\in S^\{\+\}of successful branches, the target–reference log\-odds margin is given by

m⋆=log⁡q⁡\(u\)q⁡\(v\)−log⁡qref​\(u\)qref​\(v\)\.m^\{\\star\}=\\log\\frac\{q\(u\)\}\{q\(v\)\}\-\\log\\frac\{q\_\{\\mathrm\{ref\}\}\(u\)\}\{q\_\{\\mathrm\{ref\}\}\(v\)\}\.\(4\)The marginm⋆m^\{\\star\}is therefore the reference\-relative log\-odds shift needed forπθ\\pi\_\{\\theta\}to matchqqat\(u,v\)\(u,v\)\. The corresponding target isp⋆=σ⁡\(β​m⋆\)p^\{\\star\}=\\sigma\(\\beta\\,m^\{\\star\}\)with logistic scaleβ\>0\\beta\>0; for a success–failure pair withu∈S\+u\\in S^\{\+\}andv∈S0v\\in S^\{0\},qqis undefined onvvand we setp⋆=1p^\{\\star\}=1directly, so task success labels enter the objective only through these pairs\.

#### Model margin\.

Forπθ\\pi\_\{\\theta\}, the analogous margin is defined in the same form so that it can be directly compared tom⋆m^\{\\star\}:

mθ=log⁡πθ​\(yu∣x\)πθ​\(yv∣x\)−log⁡πref​\(yu∣x\)πref​\(yv∣x\)\.m\_\{\\theta\}=\\log\\frac\{\\pi\_\{\\theta\}\(y\_\{u\}\\mid x\)\}\{\\pi\_\{\\theta\}\(y\_\{v\}\\mid x\)\}\-\\log\\frac\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{u\}\\mid x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{v\}\\mid x\)\}\.\(5\)Aligningmθm\_\{\\theta\}withm⋆m^\{\\star\}is equivalent to matchingπθ\\pi\_\{\\theta\}’s log\-odds at\(u,v\)\(u,v\)toqq’s, which is what the loss enforces\.

#### Objective function\.

We trainπθ\\pi\_\{\\theta\}by minimizing

ℒDDO\(θ\)=𝔼\[DKL\(P⋆∥Pθ\)\],\\mathcal\{L\}\_\{\\mathrm\{DDO\}\}\(\\theta\)=\\mathbb\{E\}\\bigl\[D\_\{\\mathrm\{KL\}\}\\bigl\(P^\{\\star\}\\,\\\|\\,P\_\{\\theta\}\\bigr\)\\bigr\],\(6\)whereP⋆=Bern⁡\(p⋆\)P^\{\\star\}=\\mathrm\{Bern\}\(p^\{\\star\}\)andPθ=Bern⁡\(pθ\)P\_\{\\theta\}=\\mathrm\{Bern\}\(p\_\{\\theta\}\)are Bernoulli distributions with parametersp⋆p^\{\\star\}andpθ=σ⁡\(β​mθ\)p\_\{\\theta\}=\\sigma\(\\beta\\,m\_\{\\theta\}\)\. The expectation is estimated over materialized pairs: each unordered success–success pair is included once, success–failure pairs are oriented toward the successful branch, and state\-normalized weights prevent larger branch sets from dominating\. Up to aθ\\theta\-independent constant, this is equivalent to soft binary cross\-entropy with targetp⋆p^\{\\star\}and predictionpθp\_\{\\theta\}\. The success–success terms match the target oddsq⁡\(u\)/q⁡\(v\)q\(u\)/q\(v\)between successful branches, while success–failure terms setp⋆=1p^\{\\star\}=1and reduce to the DPO\-style loss−log⁡σ⁡\(β​mθ\)\-\\log\\sigma\(\\beta m\_\{\\theta\}\), withu∈S\+u\\in S^\{\+\}andv∈S0v\\in S^\{0\}\.

## 4Experimental Setup

### 4\.1Benchmarks

We evaluate DDO on three benchmarks that span complementary forms of sequential text\-agent behavior\. BabyAI\([Chevalier\-Boisvert et al\., 2019](https://arxiv.org/html/2609.10052#bib.bib16)\)probes controlled navigation, object interaction, and multi\-stage instruction following\. BabaIsAI\([Cloos et al\., 2024](https://arxiv.org/html/2609.10052#bib.bib17)\)targets rule manipulation, where the agent must change or exploit environment dynamics\. For BabyAI and BabaIsAI, we use the BALROG text\-action implementations\([Paglieri et al\., 2025](https://arxiv.org/html/2609.10052#bib.bib18)\), which provide standardized textual observations and executable action interfaces\. WebShop\([Yao et al\., 2022](https://arxiv.org/html/2609.10052#bib.bib19)\)examines web\-style shopping behavior involving search, product inspection, option selection, and purchase decisions\. Details of each benchmark are provided in Appendix[C\.1](https://arxiv.org/html/2609.10052#A3.SS1)\.

### 4\.2Baselines and Comparison Protocol

We compare DDO against DPO and diversity\-aware post\-training baselines\. DPO is the standard Direct Preference Optimization baseline\([Rafailov et al\., 2023](https://arxiv.org/html/2609.10052#bib.bib4)\)\. DivFreq and DivProb adapt DivPO to state\-aligned branch data by treating each shared state as a prompt, its executable branch outputs as candidate responses, and terminal rollout outcomes as quality labels\([Lanchantin et al\., 2025](https://arxiv.org/html/2609.10052#bib.bib12)\)\. They retain DivPO’s frequency\- and probability\-based pair\-selection criteria, respectively\. TieDPO\-RK and TieDPO\-Dav are tie\-aware DPO baselines based on the Rao\-Kupper and Davidson variants of[Chen et al\. \(2025\)](https://arxiv.org/html/2609.10052#bib.bib11)\.

We useQwen3\-1\.7Bas the target model andQwen3\.5\-122B\-A10B\-FP8as the expert model for DTC collection\([Yang et al\., 2025](https://arxiv.org/html/2609.10052#bib.bib2);[Qwen Team, 2026](https://arxiv.org/html/2609.10052#bib.bib3)\)\. Base denotes the unadapted target model before task SFT or preference optimization\. Reference denotes the frozen task\-SFT model, fine\-tuned from the base on the successful trajectories that also serve as DTC sources\. It is used both as the initialization for all post\-training methods and as the reference model for reference\-relative objectives\.

For fair comparison, all post\-training methods within a benchmark use the same DTC\-collected branch sets, so the comparison isolates objective\-side differences while holding the data side fixed\. The DTC ablation in[Section5\.1](https://arxiv.org/html/2609.10052#S5.SS1)separately varies the data side by replacing this resource with comparisons formed without DTC, allowing us to test the contribution of the collection procedure itself\. Training and evaluation otherwise follow the same task budgets and decoding protocols within each benchmark\. Full experimental details are provided in Appendix[C](https://arxiv.org/html/2609.10052#A3)\.

### 4\.3Metrics

We evaluate each model along two axes: task success and successful strategy coverage under a fixed rollout budget\. A rollout is*valid*if it terminates without system failures or missing model outputs\. Success rate is the fraction of valid rollouts that solve the task\. BabyAI and BabaIsAI count a rollout as successful when the benchmark progression reaches1\.01\.0\. WebShop returns a graded purchase score rather than a binary success label; we treat scores of at least0\.90\.9as successful, using the same fixed threshold across all methods\.

To measure successful strategy coverage, we group successful rollouts into trajectory classes using a per\-benchmark equivalence relation that abstracts surface variation in the action sequence \(see Appendix[C\.1](https://arxiv.org/html/2609.10052#A3.SS1)for the per\-benchmark rules\)\. Two successful rollouts belong to the same class when they are equivalent under this relation\. Within each benchmark, we apply the same equivalence relation to the rollouts from all evaluated methods\. We use effective strategy diversity \(ESD\) and entropy effective strategy diversity \(H\-ESD\) to summarize the resulting classes\. Both metrics are success\-restricted and budget\-normalized: they are computed from successful trajectory classes but normalized by the total number of valid rollouts\.

For each evaluation itemii, letKiK\_\{i\}be the number of valid rollouts andUiU\_\{i\}the number of unique successful trajectory classes\. LetHiH\_\{i\}be the entropy of the empirical distribution over these classes:

ESD⁡\(i\)=UiKi,H​\-​ESD​\(i\)=2HiKi\.\\mathrm\{ESD\}\(i\)=\\frac\{U\_\{i\}\}\{K\_\{i\}\},\\qquad\\mathrm\{H\\text\{\-\}ESD\}\(i\)=\\frac\{2^\{H\_\{i\}\}\}\{K\_\{i\}\}\.ESD counts how many distinct successful trajectory classes are observed under the rollout budget, while H\-ESD is smaller when successful rollouts concentrate on a few repeated trajectory classes\. Normalizing byKiK\_\{i\}, rather than by the number of successful rollouts, makes the metric reflect finite\-budget successful strategy coverage: a model receives a high score only when it both solves the task and produces distinct successful trajectory classes\. Reported scores are uniform means of the item\-level ESD and H\-ESD over the evaluation items in each task\. Both metrics are zero when no successful rollout is observed\.

## 5Results

We first evaluate DTC as a data\-construction procedure by measuring collection\-time successful strategy coverage and downstream performance with and without DTC\. We then fix DTC and compare post\-training objectives across the three benchmarks, isolating RTO under the same state\-aligned branch supervision\.

### 5\.1DTC Improves Branch Supervision

Figure 3:Effect of DTC on BabyAI and BabaIsAI\. Each curve reports the task\-averaged number of unique successful trajectories at a matched per\-task request budget, under benchmark\-specific trajectory normalization\.To isolate the data side of DDO, we run DPO and DDO with and without DTC\. DTC converts trajectory\-level outcome labels into same\-state branch supervision\. For the collection\-efficiency comparison, Without DTC independently samples full trajectories with the same expert under the matched request budget\. For downstream training, settings without DTC form same\-task comparisons from separately sampled rollouts matched to the training comparison exposure\.

[Figure3](https://arxiv.org/html/2609.10052#S5.F3)shows that DTC discovers more unique successful trajectories under the same request budget on both BabyAI and BabaIsAI, thereby increasing collection\-time successful strategy coverage before post\-training begins\.

Figure 4:Distribution of normalized pair\-formation positions along source trajectories with and without DTC\. Positions are normalized by trajectory length\.DTC pairs are distributed across the trajectory, with56\.4%56\.4\\%formed at or beyond the midpoint\. Pairs without DTC are concentrated near the initial prefix:79\.3%79\.3\\%occur within the first20%20\\%of the source trajectory\. This structural difference matters for sequential strategies, because early, middle, and late decisions can play different roles in reaching success\. DTC therefore provides branch\-level supervision across a broader range of decision depths\.

Table 1:Effect of DTC on DPO and DDO\.Table 2:Main results across BabyAI, BabaIsAI, and WebShop\. BabyAI and BabaIsAI report task averages, WebShop reports the shopping evaluation summary, and Avg\. is the unweighted mean across benchmarks\.These collection differences translate into post\-training results\.[Table1](https://arxiv.org/html/2609.10052#S5.T1)compares DPO and DDO with and without DTC\. For DPO, adding DTC raises task success by88percentage points \(pp\) on BabyAI and1616pp on BabaIsAI, although its effect on coverage is mixed on BabyAI\. For DDO, adding DTC raises task success by55pp on BabyAI and1616pp on BabaIsAI\. Adding DTC also improves DDO’s coverage metrics: on BabyAI, H\-ESD and ESD each increase by0\.060\.06; on BabaIsAI, they increase by0\.100\.10and0\.120\.12, respectively\. Averaged across the two benchmarks, adding DTC to DDO improves task success by10\.510\.5pp, H\-ESD by0\.080\.08, and ESD by0\.090\.09\.

DTC broadens the collected set of successful trajectories and improves downstream post\-training\. Under matched request budgets, DTC discovers more unique successful trajectories and provides branch supervision at decision states spanning early, middle, and late portions of the source trajectories\. The factorization in[Table1](https://arxiv.org/html/2609.10052#S5.T1)shows complementary contributions from DTC and RTO: DTC expands branch supervision and predominantly raises success, whereas RTO improves both success and coverage under either collection condition\. Combining DTC and RTO yields the strongest joint success and coverage result on both BabyAI and BabaIsAI\. We next fix the DTC resource and compare post\-training objectives under the same state\-aligned branch supervision\.

### 5\.2RTO Improves the Success–Coverage Frontier

With DTC fixed, we compare post\-training objectives on the same state\-aligned branch sets\.[Table2](https://arxiv.org/html/2609.10052#S5.T2)reports benchmark averages; full tables appear in Appendix[A\.6](https://arxiv.org/html/2609.10052#A1.SS6)\. The three benchmarks probe different aspects of successful strategy coverage—diverse successful action trajectories for the same instruction \(BabyAI\), diverse exploitable rule configurations \(BabaIsAI\), and diverse shopping behaviors ending in valid purchases \(WebShop\)\.

#### Joint success and coverage gains\.

DDO achieves the highest benchmark\-average success \(0\.760\.76\), H\-ESD \(0\.350\.35\), and ESD \(0\.400\.40\)\. Relative to the DPO row, these values correspond to gains of88pp,0\.090\.09, and0\.090\.09, respectively\. The largest relative gain appears on WebShop, where H\-ESD nearly doubles from0\.110\.11to0\.210\.21and ESD nearly doubles from0\.120\.12to0\.220\.22, alongside a1010pp success gain from0\.260\.26to0\.360\.36\. Because WebShop trajectory classes incorporate purchase realization, the gain reflects broader successful strategy coverage under the composite class definition\.

#### Coverage relative to task\-SFT initialization\.

DDO raises task success by1010pp and both coverage metrics by0\.040\.04over the shared Reference initialization\. The remaining post\-training methods stay below Reference on both coverage metrics; DPO, for example, raises average success from0\.660\.66to0\.680\.68while H\-ESD and ESD each fall by0\.050\.05\.

#### Other post\-training variants\.

Across the compared methods, DDO leads every benchmark\-level aggregate and all three overall averages\. DivFreq and DivProb, which filter pairs by frequency\- or probability\-based diversity, change success and coverage by at most±0\.02\\pm 0\.02on average\. TieDPO\-RK and TieDPO\-Dav reach lower average success than DPO \(−3\-3and−8\-8pp\), with the largest drops on BabaIsAI \(−0\.14\-0\.14and−0\.17\-0\.17success\) and, for TieDPO\-Dav, WebShop \(−0\.10\-0\.10\); their coverage matches DPO on average\.

## 6Analysis

### 6\.1Strategic Recovery

We use strategic recovery to test whether broader successful strategy coverage provides alternative routes after a local disruption\. For each successful source trajectory, we first sample one interior decision point\. We then replace the source action at that point with a different valid task action and roll out the same model from the edited prefix\. Because every source trajectory solves the task before the edit, the score measures recovery from the local action replacement rather than ordinary task success\. Appendix[C](https://arxiv.org/html/2609.10052#A3)details the probe construction\.

DDO achieves the highest recovery rate,75\.2%75\.2\\%, compared with70\.1%70\.1\\%for DivFreq and69\.7%69\.7\\%for DPO\. After the initial trajectory is disrupted, DDO more often finds another viable trajectory to success, showing that its broader successful strategy coverage is accompanied by higher recovery\.

Figure 5:Strategic recovery on BabyAI after local action replacement in successful source trajectories\.
### 6\.2Imitation Control

To separate successful\-branch exposure from same\-state outcome comparisons and reference\-relative targets, we train successful\-only SFT models on the same successful DTC branches used by DDO\. We compare DDO with two imitation settings matched by optimizer steps and nominal epochs, respectively\. At matched optimizer steps, successful\-only imitation reaches H\-ESD0\.390\.39and ESD0\.440\.44, compared with DDO’s0\.420\.42and0\.500\.50, while DDO retains a3232pp advantage in task success\. At7\.30×7\.30\\timesexposure, imitation reaches H\-ESD0\.410\.41and ESD0\.470\.47, while DDO retains a2424pp advantage in task success\. Successful\-only SFT broadens coverage, while DDO retains a2424–3232pp advantage in task success under both matching conditions\.

### 6\.3Decoding Control

We also test whether inference\-time diversification can recover successful strategy coverage without post\-training changes\. We compare DDO against DPO under a range of sampling temperatures\.

Table 3:Successful\-only imitation control averaged over BabyAI and BabaIsAI\.Table 4:DPO temperature sweep averaged across BabyAI, BabaIsAI, and WebShop; DDO atT=0\.6T=0\.6is shown for comparison\.Simply increasing DPO’s sampling temperature raises strategy coverage, but the gain comes with lower task success\. AtT=1\.5T\{=\}1\.5, DPO reaches an ESD of0\.390\.39, close to DDO’s0\.400\.40, while DDO retains a99pp advantage in task success\. Increasing temperature therefore recovers coverage by sacrificing success, whereas DDO improves the joint success and coverage result through post\-training\.

## 7Conclusion

We present Direct Diversity Optimization \(DDO\), an offline post\-training method for preserving multiple successful strategies in LLM agents trained from trajectory\-level outcome labels\. DDO consists of two components: Divergence\-Tree Collection \(DTC\) and the Reference\-Relative Target\-Odds Objective \(RTO\)\. DTC builds state\-aligned branch sets with per\-branch outcome labels, and RTO trains the model toward reference\-relative targets over successful branches\.

Across all benchmarks, DDO achieves the strongest joint performance in task success and successful strategy coverage among the compared post\-training methods\. DDO also achieves the highest recovery rate in the local action replacement evaluation\. Compared with successful\-only imitation and decoding\-time diversification, DDO achieves broader coverage together with higher task success by training on same\-state outcome comparisons with reference\-relative targets\. These results establish DDO as a training\-time method that improves task success while broadening successful strategy coverage in sequential decision tasks\.

## Limitations

For controlled benchmarking and method comparison, our experiments are confined to environments that support state reconstruction and the execution of alternative branches from a shared decision state under a common success predicate\. This setting makes the outcomes of same\-state alternative branches directly observable, but it excludes continuous control, partially observable or stochastic dynamics, multimodal observations, and real\-world agents whose actions induce non\-reversible external side effects\. Extending DTC to settings without exact state reconstruction would require approximate state\-aligned branch sets\. Such sets could be constructed from rollouts generated by learned simulators or world models, or from logged trajectories that contain comparable decision contexts and alternative outcomes\.

## Ethical Considerations

#### License of Existing Assets\.

The models used in this work, Qwen3\-1\.7B and Qwen3\.5\-122B\-A10B\-FP8, are released under the Apache 2\.0 license\. The benchmark environments are used under their respective licenses: BabyAI under the BSD 3\-Clause License, BabaIsAI under the MIT License, and WebShop under the MIT License with copyright attributed to Princeton Natural Language Processing\. All assets and environments were used for academic, non\-commercial evaluation purposes and in accordance with the applicable license terms\.

## Acknowledgments

This work was supported by the Artificial Intelligence Industrial Convergence Cluster Development Project, funded by the Ministry of Science and ICT \(MSIT\), Korea, and Gwangju Metropolitan City\.

## References

- Baiet al\.\(2022\)Y\. Bai, A\. Jones, K\. Ndousse, A\. Askell, A\. Chen, N\. DasSarma, D\. Drain, S\. Fort, D\. Ganguli, T\. Henighan, N\. Joseph, S\. Kadavath, J\. Kernion, T\. Conerly, S\. El\-Showk, N\. Elhage, Z\. Hatfield\-Dodds, D\. Hernandez, T\. Hume, S\. Johnston, S\. Kravec, L\. Lovitt, N\. Nanda, C\. Olsson, D\. Amodei, T\. Brown, J\. Clark, S\. McCandlish, C\. Olah, B\. Mann, and J\. KaplanTraining a helpful and harmless assistant with reinforcement learning from human feedback\.External Links:2204\.05862,[Document](https://dx.doi.org/10.48550/arXiv.2204.05862),[Link](https://arxiv.org/abs/2204.05862)Cited by:[§2\.1](https://arxiv.org/html/2609.10052#S2.SS1.p1.1)\.
- Chenet al\.\(2025\)J\. Chen, G\. Yang, W\. Lin, J\. Mei, C\. Lyu, and B\. ByrneOn extending direct preference optimization to accommodate ties\.InAdvances in Neural Information Processing Systems,Vol\.38, Main Conference,pp\. 53932–53977\.External Links:[Document](https://dx.doi.org/10.52202/085713-1801),[Link](https://proceedings.neurips.cc/paper_files/paper/2025/hash/4dbf3488a8da908a8177d525ede1a43a-Abstract-Conference.html)Cited by:[§2\.1](https://arxiv.org/html/2609.10052#S2.SS1.p1.1),[§4\.2](https://arxiv.org/html/2609.10052#S4.SS2.p1.1)\.
- Chevalier\-Boisvertet al\.\(2019\)M\. Chevalier\-Boisvert, D\. Bahdanau, S\. Lahlou, L\. Willems, C\. Saharia, T\. H\. Nguyen, and Y\. BengioBabyAI: a platform to study the sample efficiency of grounded language learning\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=rJeXCo0cYX)Cited by:[§4\.1](https://arxiv.org/html/2609.10052#S4.SS1.p1.1)\.
- Clooset al\.\(2024\)N\. Cloos, M\. Jens, M\. Naim, Y\. Kuo, I\. Cases, A\. Barbu, and C\. J\. CuevaBaba is AI: break the rules to beat the benchmark\.InICML 2024 Workshop on LLMs and Cognition,External Links:[Link](https://github.com/nacloos/baba-is-ai)Cited by:[§4\.1](https://arxiv.org/html/2609.10052#S4.SS1.p1.1)\.
- Eysenbachet al\.\(2019\)B\. Eysenbach, A\. Gupta, J\. Ibarz, and S\. LevineDiversity is all you need: learning skills without a reward function\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=SJx63jRqFm)Cited by:[§2\.1](https://arxiv.org/html/2609.10052#S2.SS1.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan, A\. Yang, A\. Fan, A\. Goyal, A\. Hartshorn, A\. Yang, A\. Mitra, A\. Sravankumar, A\. Korenev, A\. Hinsvark, A\. Rao, A\. Zhang, A\. Rodriguez, A\. Gregerson, A\. Spataru, B\. Roziere, B\. Biron, B\. Tang, B\. Chern, C\. Caucheteux, C\. Nayak, C\. Bi, C\. Marra, C\. McConnell, C\. Keller, C\. Touret, C\. Wu, C\. Wong, C\. C\. Ferrer, C\. Nikolaidis, D\. Allonsius, D\. Song, D\. Pintz, D\. Livshits, D\. Wyatt, D\. Esiobu, D\. Choudhary, D\. Mahajan, D\. Garcia\-Olano, D\. Perino, D\. Hupkes, E\. Lakomkin, E\. AlBadawy, E\. Lobanova, E\. Dinan, E\. M\. Smith, F\. Radenovic, F\. Guzmán, F\. Zhang, G\. Synnaeve, G\. Lee, G\. L\. Anderson, G\. Thattai, G\. Nail, G\. Mialon, G\. Pang, G\. Cucurell, H\. Nguyen, H\. Korevaar, H\. Xu, H\. Touvron, I\. Zarov, I\. A\. Ibarra, I\. Kloumann, I\. Misra, I\. Evtimov, J\. Zhang, J\. Copet, J\. Lee, J\. Geffert, J\. Vranes, J\. Park, J\. Mahadeokar, J\. Shah, J\. van der Linde, J\. Billock, J\. Hong, J\. Lee, J\. Fu, J\. Chi, J\. Huang, J\. Liu, J\. Wang, J\. Yu, J\. Bitton, J\. Spisak, J\. Park, J\. Rocca, J\. Johnstun, J\. Saxe, J\. Jia, K\. V\. Alwala, K\. Prasad, K\. Upasani, K\. Plawiak, K\. Li, K\. Heafield, K\. Stone, K\. El\-Arini, K\. Iyer, K\. Malik, K\. Chiu, K\. Bhalla, K\. Lakhotia, L\. Rantala\-Yeary, L\. van der Maaten, L\. Chen, L\. Tan, L\. Jenkins, L\. Martin, L\. Madaan, L\. Malo, L\. Blecher, L\. Landzaat, L\. de Oliveira, M\. Muzzi, M\. Pasupuleti, M\. Singh, M\. Paluri, M\. Kardas, M\. Tsimpoukelli, M\. Oldham, M\. Rita, M\. Pavlova, M\. Kambadur, M\. Lewis, M\. Si, M\. K\. Singh, M\. Hassan, N\. Goyal, N\. Torabi, N\. Bashlykov, N\. Bogoychev, N\. Chatterji, N\. Zhang, O\. Duchenne, O\. Çelebi, P\. Alrassy, P\. Zhang, P\. Li, P\. Vasic, P\. Weng, P\. Bhargava, P\. Dubal, P\. Krishnan, P\. S\. Koura, P\. Xu, Q\. He, Q\. Dong, R\. Srinivasan, R\. Ganapathy, R\. Calderer, R\. S\. Cabral, R\. Stojnic, R\. Raileanu, R\. Maheswari, R\. Girdhar, R\. Patel, R\. Sauvestre, R\. Polidoro, R\. Sumbaly, R\. Taylor, R\. Silva, R\. Hou, R\. Wang, S\. Hosseini, S\. Chennabasappa, S\. Singh, S\. Bell, S\. S\. Kim, S\. Edunov, S\. Nie, S\. Narang, S\. Raparthy, S\. Shen, S\. Wan, S\. Bhosale, S\. Zhang, S\. Vandenhende, S\. Batra, S\. Whitman, S\. Sootla, S\. Collot, S\. Gururangan, S\. Borodinsky, T\. Herman, T\. Fowler, T\. Sheasha, T\. Georgiou, T\. Scialom, T\. Speckbacher, T\. Mihaylov, T\. Xiao, U\. Karn, V\. Goswami, V\. Gupta, V\. Ramanathan, V\. Kerkez, V\. Gonguet, V\. Do, V\. Vogeti, V\. Albiero, V\. Petrovic, W\. Chu, W\. Xiong, W\. Fu, W\. Meers, X\. Martinet, X\. Wang, X\. Wang, X\. E\. Tan, X\. Xia, X\. Xie, X\. Jia, X\. Wang, Y\. Goldschlag, Y\. Gaur, Y\. Babaei, Y\. Wen, Y\. Song, Y\. Zhang, Y\. Li, Y\. Mao, Z\. D\. Coudert, Z\. Yan, Z\. Chen, Z\. Papakipos, A\. Singh, A\. Srivastava, A\. Jain, A\. Kelsey, A\. Shajnfeld, A\. Gangidi, A\. Victoria, A\. Goldstand, A\. Menon, A\. Sharma, A\. Boesenberg, A\. Baevski, A\. Feinstein, A\. Kallet, A\. Sangani, A\. Teo, A\. Yunus, A\. Lupu, A\. Alvarado, A\. Caples, A\. Gu, A\. Ho, A\. Poulton, A\. Ryan, A\. Ramchandani, A\. Dong, A\. Franco, A\. Goyal, A\. Saraf, A\. Chowdhury, A\. Gabriel, A\. Bharambe, A\. Eisenman, A\. Yazdan, B\. James, B\. Maurer, B\. Leonhardi, B\. Huang, B\. Loyd, B\. D\. Paola, B\. Paranjape, B\. Liu, B\. Wu, B\. Ni, B\. Hancock, B\. Wasti, B\. Spence, B\. Stojkovic, B\. Gamido, B\. Montalvo, C\. Parker, C\. Burton, C\. Mejia, C\. Liu, C\. Wang, C\. Kim, C\. Zhou, C\. Hu, C\. Chu, C\. Cai, C\. Tindal, C\. Feichtenhofer, C\. Gao, D\. Civin, D\. Beaty, D\. Kreymer, D\. Li, D\. Adkins, D\. Xu, D\. Testuggine, D\. David, D\. Parikh, D\. Liskovich, D\. Foss, D\. Wang, D\. Le, D\. Holland, E\. Dowling, E\. Jamil, E\. Montgomery, E\. Presani, E\. Hahn, E\. Wood, E\. Le, E\. Brinkman, E\. Arcaute, E\. Dunbar, E\. Smothers, F\. Sun, F\. Kreuk, F\. Tian, F\. Kokkinos, F\. Ozgenel, F\. Caggioni, F\. Kanayet, F\. Seide, G\. M\. Florez, G\. Schwarz, G\. Badeer, G\. Swee, G\. Halpern, G\. Herman, G\. Sizov, G\. Zhang, G\. Lakshminarayanan, H\. Inan, H\. Shojanazeri, H\. Zou, H\. Wang, H\. Zha, H\. Habeeb, H\. Rudolph, H\. Suk, H\. Aspegren, H\. Goldman, H\. Zhan, I\. Damlaj, I\. Molybog, I\. Tufanov, I\. Leontiadis, I\. Veliche, I\. Gat, J\. Weissman, J\. Geboski, J\. Kohli, J\. Lam, J\. Asher, J\. Gaya, J\. Marcus, J\. Tang, J\. Chan, J\. Zhen, J\. Reizenstein, J\. Teboul, J\. Zhong, J\. Jin, J\. Yang, J\. Cummings, J\. Carvill, J\. Shepard, J\. McPhie, J\. Torres, J\. Ginsburg, J\. Wang, K\. Wu, K\. H\. U, K\. Saxena, K\. Khandelwal, K\. Zand, K\. Matosich, K\. Veeraraghavan, K\. Michelena, K\. Li, K\. Jagadeesh, K\. Huang, K\. Chawla, K\. Huang, L\. Chen, L\. Garg, L\. A, L\. Silva, L\. Bell, L\. Zhang, L\. Guo, L\. Yu, L\. Moshkovich, L\. Wehrstedt, M\. Khabsa, M\. Avalani, M\. Bhatt, M\. Mankus, M\. Hasson, M\. Lennie, M\. Reso, M\. Groshev, M\. Naumov, M\. Lathi, M\. Keneally, M\. Liu, M\. L\. Seltzer, M\. Valko, M\. Restrepo, M\. Patel, M\. Vyatskov, M\. Samvelyan, M\. Clark, M\. Macey, M\. Wang, M\. J\. Hermoso, M\. Metanat, M\. Rastegari, M\. Bansal, N\. Santhanam, N\. Parks, N\. White, N\. Bawa, N\. Singhal, N\. Egebo, N\. Usunier, N\. Mehta, N\. P\. Laptev, N\. Dong, N\. Cheng, O\. Chernoguz, O\. Hart, O\. Salpekar, O\. Kalinli, P\. Kent, P\. Parekh, P\. Saab, P\. Balaji, P\. Rittner, P\. Bontrager, P\. Roux, P\. Dollar, P\. Zvyagina, P\. Ratanchandani, P\. Yuvraj, Q\. Liang, R\. Alao, R\. Rodriguez, R\. Ayub, R\. Murthy, R\. Nayani, R\. Mitra, R\. Parthasarathy, R\. Li, R\. Hogan, R\. Battey, R\. Wang, R\. Howes, R\. Rinott, S\. Mehta, S\. Siby, S\. J\. Bondu, S\. Datta, S\. Chugh, S\. Hunt, S\. Dhillon, S\. Sidorov, S\. Pan, S\. Mahajan, S\. Verma, S\. Yamamoto, S\. Ramaswamy, S\. Lindsay, S\. Lindsay, S\. Feng, S\. Lin, S\. C\. Zha, S\. Patil, S\. Shankar, S\. Zhang, S\. Zhang, S\. Wang, S\. Agarwal, S\. Sajuyigbe, S\. Chintala, S\. Max, S\. Chen, S\. Kehoe, S\. Satterfield, S\. Govindaprasad, S\. Gupta, S\. Deng, S\. Cho, S\. Virk, S\. Subramanian, S\. Choudhury, S\. Goldman, T\. Remez, T\. Glaser, T\. Best, T\. Koehler, T\. Robinson, T\. Li, T\. Zhang, T\. Matthews, T\. Chou, T\. Shaked, V\. Vontimitta, V\. Ajayi, V\. Montanez, V\. Mohan, V\. S\. Kumar, V\. Mangla, V\. Ionescu, V\. Poenaru, V\. T\. Mihailescu, V\. Ivanov, W\. Li, W\. Wang, W\. Jiang, W\. Bouaziz, W\. Constable, X\. Tang, X\. Wu, X\. Wang, X\. Wu, X\. Gao, Y\. Kleinman, Y\. Chen, Y\. Hu, Y\. Jia, Y\. Qi, Y\. Li, Y\. Zhang, Y\. Zhang, Y\. Adi, Y\. Nam, Y\. Wang, Y\. Zhao, Y\. Hao, Y\. Qian, Y\. Li, Y\. He, Z\. Rait, Z\. DeVito, Z\. Rosnbrick, Z\. Wen, Z\. Yang, Z\. Zhao, and Z\. MaThe Llama 3 herd of models\.External Links:2407\.21783,[Document](https://dx.doi.org/10.48550/arXiv.2407.21783),[Link](https://arxiv.org/abs/2407.21783)Cited by:[§1](https://arxiv.org/html/2609.10052#S1.p1.1)\.
- Haarnojaet al\.\(2018\)T\. Haarnoja, A\. Zhou, P\. Abbeel, and S\. LevineSoft actor\-critic: off\-policy maximum entropy deep reinforcement learning with a stochastic actor\.InProceedings of the 35th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.80,pp\. 1861–1870\.External Links:[Link](https://proceedings.mlr.press/v80/haarnoja18b.html)Cited by:[§2\.1](https://arxiv.org/html/2609.10052#S2.SS1.p1.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[§C\.2](https://arxiv.org/html/2609.10052#A3.SS2.p2.1)\.
- Kirket al\.\(2024\)R\. Kirk, I\. Mediratta, C\. Nalmpantis, J\. Luketina, E\. Hambro, E\. Grefenstette, and R\. RaileanuUnderstanding the effects of RLHF on LLM generalisation and diversity\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 20620–20653\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/hash/5a68d05006d5b05dd9463dd9c0219db0-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2609.10052#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.10052#S2.SS1.p1.1)\.
- Lanchantinet al\.\(2025\)J\. Lanchantin, A\. Chen, S\. Dhuliawala, P\. Yu, J\. Weston, S\. Sukhbaatar, and I\. KulikovDiverse preference optimization\.External Links:2501\.18101,[Document](https://dx.doi.org/10.48550/arXiv.2501.18101),[Link](https://arxiv.org/abs/2501.18101)Cited by:[§1](https://arxiv.org/html/2609.10052#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.10052#S2.SS1.p1.1),[§4\.2](https://arxiv.org/html/2609.10052#S4.SS2.p1.1)\.
- Loshchilov and Hutter \(2019\)I\. Loshchilov and F\. HutterDecoupled weight decay regularization\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Bkg6RiCqY7)Cited by:[§C\.2](https://arxiv.org/html/2609.10052#A3.SS2.p2.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. F\. Christiano, J\. Leike, and R\. LoweTraining language models to follow instructions with human feedback\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 27730–27744\.External Links:[Document](https://dx.doi.org/10.52202/068431-2011),[Link](https://proceedings.neurips.cc/paper_files/paper/2022/hash/b1efde53be364a73914f58805a001731-Abstract.html)Cited by:[§1](https://arxiv.org/html/2609.10052#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.10052#S2.SS1.p1.1)\.
- Padmakumar and He \(2024\)V\. Padmakumar and H\. HeDoes writing with language models reduce content diversity?\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 642–669\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/hash/02dec8877fb7c6aa9a79f81661baca7c-Abstract-Conference.html)Cited by:[§2\.1](https://arxiv.org/html/2609.10052#S2.SS1.p1.1)\.
- Paglieriet al\.\(2025\)D\. Paglieri, B\. Cupiał, S\. Coward, U\. Piterbarg, M\. Wołczyk, A\. Khan, E\. Pignatelli, Ł\. Kuciński, L\. Pinto, R\. Fergus, J\. Foerster, J\. Parker\-Holder, and T\. RocktäschelBALROG: benchmarking agentic LLM and VLM reasoning on games\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 96666–96702\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/hash/f0b1515be276f6ba82b4f2b25e50bef0-Abstract-Conference.html)Cited by:[§4\.1](https://arxiv.org/html/2609.10052#S4.SS1.p1.1)\.
- Pierrotet al\.\(2022\)T\. Pierrot, V\. Macé, F\. Chalumeau, A\. Flajolet, G\. Cideron, K\. Beguir, A\. Cully, O\. Sigaud, and N\. Perrin\-GilbertDiversity policy gradient for sample efficient quality\-diversity optimization\.InProceedings of the Genetic and Evolutionary Computation Conference,pp\. 1075–1083\.External Links:[Document](https://dx.doi.org/10.1145/3512290.3528845),[Link](https://doi.org/10.1145/3512290.3528845)Cited by:[§2\.1](https://arxiv.org/html/2609.10052#S2.SS1.p1.1)\.
- Pughet al\.\(2016\)J\. K\. Pugh, L\. B\. Soros, and K\. O\. StanleyQuality diversity: a new frontier for evolutionary computation\.Frontiers in Robotics and AI3,pp\. 40\.External Links:[Document](https://dx.doi.org/10.3389/frobt.2016.00040),[Link](https://doi.org/10.3389/frobt.2016.00040)Cited by:[§2\.1](https://arxiv.org/html/2609.10052#S2.SS1.p1.1)\.
- Qwen Team \(2026\)Qwen TeamQwen3\.5: towards native multimodal agents\.External Links:[Link](https://qwen.ai/blog?id=qwen3.5)Cited by:[§C\.2](https://arxiv.org/html/2609.10052#A3.SS2.p1.1),[§4\.2](https://arxiv.org/html/2609.10052#S4.SS2.p2.1)\.
- Rafailovet al\.\(2023\)R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. FinnDirect preference optimization: your language model is secretly a reward model\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 53728–53741\.External Links:[Document](https://dx.doi.org/10.52202/075280-2338),[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html)Cited by:[§C\.2](https://arxiv.org/html/2609.10052#A3.SS2.p1.1),[§1](https://arxiv.org/html/2609.10052#S1.p3.1),[§2\.1](https://arxiv.org/html/2609.10052#S2.SS1.p1.1),[§4\.2](https://arxiv.org/html/2609.10052#S4.SS2.p1.1)\.
- Slocumet al\.\(2025\)S\. Slocum, A\. Parker\-Sartori, and D\. Hadfield\-MenellDiverse preference learning for capabilities and alignment\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 24760–24790\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/hash/3df1eca840e82b11bbc33f68c773c38e-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2609.10052#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.10052#S2.SS1.p1.1)\.
- Songet al\.\(2024\)Y\. Song, D\. Yin, X\. Yue, J\. Huang, S\. Li, and B\. Y\. LinTrial and error: exploration\-based trajectory optimization of LLM agents\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 7584–7600\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.409),[Link](https://aclanthology.org/2024.acl-long.409/)Cited by:[§1](https://arxiv.org/html/2609.10052#S1.p1.1),[§2\.2](https://arxiv.org/html/2609.10052#S2.SS2.p1.1)\.
- Stiennonet al\.\(2020\)N\. Stiennon, L\. Ouyang, J\. Wu, D\. Ziegler, R\. Lowe, C\. Voss, A\. Radford, D\. Amodei, and P\. F\. ChristianoLearning to summarize with human feedback\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 3008–3021\.External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/1f89885d556929e98d3ef9b86448f951-Abstract.html)Cited by:[§2\.1](https://arxiv.org/html/2609.10052#S2.SS1.p1.1)\.
- Touvronet al\.\(2023\)H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale, D\. Bikel, L\. Blecher, C\. C\. Ferrer, M\. Chen, G\. Cucurull, D\. Esiobu, J\. Fernandes, J\. Fu, W\. Fu, B\. Fuller, C\. Gao, V\. Goswami, N\. Goyal, A\. Hartshorn, S\. Hosseini, R\. Hou, H\. Inan, M\. Kardas, V\. Kerkez, M\. Khabsa, I\. Kloumann, A\. Korenev, P\. S\. Koura, M\. Lachaux, T\. Lavril, J\. Lee, D\. Liskovich, Y\. Lu, Y\. Mao, X\. Martinet, T\. Mihaylov, P\. Mishra, I\. Molybog, Y\. Nie, A\. Poulton, J\. Reizenstein, R\. Rungta, K\. Saladi, A\. Schelten, R\. Silva, E\. M\. Smith, R\. Subramanian, X\. E\. Tan, B\. Tang, R\. Taylor, A\. Williams, J\. X\. Kuan, P\. Xu, Z\. Yan, I\. Zarov, Y\. Zhang, A\. Fan, M\. Kambadur, S\. Narang, A\. Rodriguez, R\. Stojnic, S\. Edunov, and T\. ScialomLlama 2: open foundation and fine\-tuned chat models\.External Links:2307\.09288,[Document](https://dx.doi.org/10.48550/arXiv.2307.09288),[Link](https://arxiv.org/abs/2307.09288)Cited by:[§1](https://arxiv.org/html/2609.10052#S1.p1.1)\.
- Weiet al\.\(2025\)Z\. Wei, W\. Yao, Y\. Liu, W\. Zhang, Q\. Lu, L\. Qiu, C\. Yu, P\. Xu, C\. Zhang, B\. Yin, H\. Yun, and L\. LiWebAgent\-r1: training web agents via end\-to\-end multi\-turn reinforcement learning\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,Suzhou, China,pp\. 7909–7928\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.401),[Link](https://aclanthology.org/2025.emnlp-main.401/)Cited by:[§1](https://arxiv.org/html/2609.10052#S1.p1.1)\.
- Xiet al\.\(2026\)Z\. Xi, C\. Liao, G\. Li, Z\. Zhang, W\. Chen, B\. Wang, S\. Jin, Y\. Zhou, J\. Guan, W\. Wu, T\. Ji, T\. Gui, Q\. Zhang, and X\. HuangAgentPRM: process reward models for LLM agents via step\-wise promise and progress\.InProceedings of the ACM Web Conference 2026,pp\. 4184–4195\.External Links:[Document](https://dx.doi.org/10.1145/3774904.3792551),[Link](https://doi.org/10.1145/3774904.3792551)Cited by:[§2\.2](https://arxiv.org/html/2609.10052#S2.SS2.p1.1)\.
- Xionget al\.\(2024\)W\. Xiong, Y\. Song, X\. Zhao, W\. Wu, X\. Wang, K\. Wang, C\. Li, W\. Peng, and S\. LiWatch every step\! LLM agent learning via iterative step\-level process refinement\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Miami, Florida, USA,pp\. 1556–1572\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.93),[Link](https://aclanthology.org/2024.emnlp-main.93/)Cited by:[§2\.2](https://arxiv.org/html/2609.10052#S2.SS2.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. QiuQwen3 technical report\.External Links:2505\.09388,[Document](https://dx.doi.org/10.48550/arXiv.2505.09388),[Link](https://arxiv.org/abs/2505.09388)Cited by:[§C\.2](https://arxiv.org/html/2609.10052#A3.SS2.p1.1),[§1](https://arxiv.org/html/2609.10052#S1.p1.1),[§4\.2](https://arxiv.org/html/2609.10052#S4.SS2.p2.1)\.
- Yaoet al\.\(2022\)S\. Yao, H\. Chen, J\. Yang, and K\. NarasimhanWebShop: towards scalable real\-world web interaction with grounded language agents\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 20744–20757\.External Links:[Document](https://dx.doi.org/10.52202/068431-1508),[Link](https://proceedings.neurips.cc/paper_files/paper/2022/hash/82ad13ec01f9fe44c01cb91814fd7b8c-Abstract-Conference.html)Cited by:[§4\.1](https://arxiv.org/html/2609.10052#S4.SS1.p1.1)\.
- Zenget al\.\(2024\)A\. Zeng, M\. Liu, R\. Lu, B\. Wang, X\. Liu, Y\. Dong, and J\. TangAgentTuning: enabling generalized agent abilities for LLMs\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 3053–3077\.External Links:[Link](https://aclanthology.org/2024.findings-acl.181/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.181)Cited by:[§1](https://arxiv.org/html/2609.10052#S1.p1.1)\.
- Zhanget al\.\(2024\)X\. Zhang, C\. Du, T\. Pang, Q\. Liu, W\. Gao, and M\. LinChain of preference optimization: improving chain\-of\-thought reasoning in LLMs\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 333–356\.External Links:[Document](https://dx.doi.org/10.52202/079017-0011),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/00d80722b756de0166523a87805dd00f-Abstract-Conference.html)Cited by:[§2\.2](https://arxiv.org/html/2609.10052#S2.SS2.p1.1)\.

## Appendix AAdditional Results

### A\.1Model\-Scale Sensitivity

We vary the expert and target models separately on BabyAI while keeping all other collection, training, and evaluation settings fixed \([Table5](https://arxiv.org/html/2609.10052#A1.T5)\)\.

Table 5:Model\-scale sensitivity on BabyAI\. Each block varies one model while holding the other fixed\.Reducing the expert model size from 122B to 35B changes task success from0\.920\.92to0\.870\.87, H\-ESD from0\.450\.45to0\.410\.41, and ESD from0\.520\.52to0\.480\.48\. Even with the 35B expert model, DDO retains more than90%90\\%of the corresponding 122B value for each metric:94\.6%94\.6\\%for task success,91\.1%91\.1\\%for H\-ESD, and92\.3%92\.3\\%for ESD\. The number of completed alternative branches also remains similar \(3,3293\{,\}329versus3,4063\{,\}406\)\.

DDO improves over its corresponding Reference at both target model sizes\. For the 1\.7B target model, DDO improves task success, H\-ESD, and ESD by0\.100\.10,0\.040\.04, and0\.030\.03, respectively\. For the 4B target model, the corresponding gains are0\.060\.06,0\.030\.03, and0\.030\.03\.

### A\.2Coverage Growth

Beyond final\-budget scores,[Figure6](https://arxiv.org/html/2609.10052#A1.F6)traces the cumulative number of unique successful trajectories found within the firstNNrollouts on BabyAI, BabaIsAI, and WebShop\. DDO stays above the other plotted methods across the sampled budget range, so its coverage gain is visible throughout sampling and at the final evaluation budget\.

Figure 6:Coverage growth over BabyAI, BabaIsAI, and WebShop\. Each curve indicates the cumulative number of unique successful trajectories found within the firstNNrollouts\.
### A\.3Shared Cross\-Task Post\-Training

The main benchmark uses SFT and preference adapters for each task to provide controlled, matched comparisons\. As an additional setting, we test whether DDO remains effective when both the initializer and post\-training model are shared across tasks\. On BabyAI, a single BabyAI\-4 SFT adapter initializes all four tasks, and preference methods are trained on pooled BabyAI\-4 preference data\. The table reports BabyAI task averages under the same evaluation budgets and decoding settings as the main results\.

Table 6:Shared cross\-task SFT and post\-training on BabyAI using one BabyAI\-4 SFT initialization and pooled preference data\.[Table6](https://arxiv.org/html/2609.10052#A1.T6)shows that DDO consistently improves both task success and successful strategy coverage in the shared\-model setting\. DDO achieves0\.920\.92success,0\.480\.48H\-ESD, and0\.520\.52ESD, extending its joint success and coverage gains from task\-specific adapters to pooled cross\-task post\-training\.

### A\.4Branch\-Set Target Distribution

In Eq\. \([3](https://arxiv.org/html/2609.10052#S3.E3)\), we interpret the RTO target as a log\-odds interpolation between two endpoints: a uniform distribution onS\+S^\{\+\}\(α=0\\alpha=0, encoding within\-success coverage\) and the reference distribution restricted toS\+S^\{\+\}\(α=1\\alpha=1, preserving the relative ordering inherited fromπref\\pi\_\{\\mathrm\{ref\}\}\)\. The intermediate settingα=0\.5\\alpha=0\.5retains both signals\.[Table7](https://arxiv.org/html/2609.10052#A1.T7)compares these three regimes on BabyAI\.

Table 7:Effect of the target\-distribution coefficientα\\alphaon BabyAI\.The intermediate target \(α=0\.5\\alpha=0\.5\) achieves the highest success \(0\.920\.92\), H\-ESD \(0\.450\.45\), and ESD \(0\.520\.52\)\. At the endpoints,α=1\\alpha=1preserves the reference\-relative weighting over successful branches, whileα=0\\alpha=0uses a uniform target\. The intermediate setting combines both target components and yields the strongest joint success and coverage result\.

### A\.5Margin Sharpness Sensitivity

The inverse\-temperature parameterβ\\betacontrols the sharpness of the pairwise margin loss, determining how strongly deviations from the target pair odds are penalized\.

Figure 7:Effect of the margin sharpnessβ\\betaon DDO, averaged over BabyAI and BabaIsAI\. Axes report success rate and H\-ESD\.[Figure7](https://arxiv.org/html/2609.10052#A1.F7)reports the DDOβ\\betasweep over\{0\.1,0\.5,1\.0\}\\\{0\.1,0\.5,1\.0\\\}\. Across this range, success changes only mildly, while H\-ESD remains nearly unchanged after rounding\. DDO therefore varies little with margin sharpness across the tested values\.

### A\.6Full Main Benchmark Tables

We provide the full benchmark results in[Tables8](https://arxiv.org/html/2609.10052#A1.T8),[9](https://arxiv.org/html/2609.10052#A1.T9)and[10](https://arxiv.org/html/2609.10052#A1.T10)\. Bold marks the strongest preference\-trained result in each column, including ties\.

#### BabyAI\.

DDO leads on average success rate, H\-ESD, and ESD\. The task columns show the same pattern: success and H\-ESD increase for Goto, Pick, Open, and Comp, and ESD increases on three tasks and reaches0\.570\.57on Comp\. The Open and Comp columns make this pattern visible in tasks involving object interaction and multi\-step composition; DDO improves both success and coverage there, so broader successful strategy coverage accompanies higher task success\.

#### BabaIsAI\.

DDO is again the strongest method on average success rate, H\-ESD, and ESD, with gains across Basic, Room, Stop, and Flex\. The Stop column is particularly informative: DDO raises success to1\.001\.00while also improving both coverage metrics, indicating that the gains extend to rule manipulation tasks where successful behavior depends on changing the rule structure\. DivProb and the tie\-aware baselines are strongest or tied in a few individual coverage columns\. DDO leads the benchmark averages while improving success and coverage together\.

#### WebShop\.

DDO is the strongest method across all three reported metrics\. Because WebShop counts unique successful trajectories using both trajectory structure and purchase realization, the gains reflect broader successful strategy coverage under the composite trajectory\-class definition\.

Table 8:Main results on BabyAI\. Avg\. is the unweighted mean across the four tasks\.Table 9:Main results on BabaIsAI\. Avg\. is the unweighted mean across the four tasks\.Table 10:Main results on WebShop\.

## Appendix BAdditional Method Details

### B\.1Action Parsing and Branch Output Probabilities

At each environment step, the LLM emits a model outputyty\_\{t\}, and a benchmark\-specific parser extracts the executable action for the environment\. Some prompts structureyty\_\{t\}as reasoning text followed by an action field; DDO assigns sequence probability to the complete model output and uses the parsed action for execution\. For a DTC recordC=\(x,B,\{\(yj,rj\)\}j∈B\)C=\(x,B,\\\{\(y\_\{j\},r\_\{j\}\)\\\}\_\{j\\in B\}\),πθ​\(yj∣x\)\\pi\_\{\\theta\}\(y\_\{j\}\\mid x\)is the sequence probability of the model output under standard left\-to\-right token factorization\. The rollout determines the branch outcome labelrjr\_\{j\}\.

### B\.2DTC Collection Details

The helper routines in[Algorithm1](https://arxiv.org/html/2609.10052#alg1)are instantiated as follows\. Across benchmarks, we setKdK\_\{d\}to at most55andKaK\_\{a\}to at most33; shorter trajectories and branch rejection can yield fewer retained points or alternatives\.Selectallocates the divergence budgetKdK\_\{d\}over interior decision points1,…,T−11,\\ldots,T\-1so that selected steps cover different trajectory depths\.Restorereturns the decision state associated with the selected source prefix, or⊥\\botwhen the state cannot be constructed under the benchmark environment\.Altqueries the expert modelEEfor at mostKaK\_\{a\}alternative outputs from the shared statexx, required to differ from the source output under the benchmark’s executable\-action normalization\.Rejectremoves outputs that cannot be executed or that do not provide a distinct executable branch decision, and retained outputs are rolled out byEEto obtain outcome labels\.

## Appendix CExperimental Details

### C\.1Benchmark Details

BabyAI provides controlled navigation and object interaction tasks with executable text actions\. We use four tasks:goto,pickup,open, andpick\-up sequence go\-to\. They are reported in[Table8](https://arxiv.org/html/2609.10052#A1.T8)as Goto, Pick, Open, and Comp\. For trajectory\-class normalization, we collapse modulo\-four same\-direction turn repetitions and remove alternating turn blocks that cancel out, both of which leave the agent’s pose unchanged\.

BabaIsAI provides tasks based on rule manipulation with executable text actions\. We use four tasks:goto,two\-room goto,two\-room break\-stop goto, andtwo\-room optional break\-stop goto\. They are reported in[Table9](https://arxiv.org/html/2609.10052#A1.T9)as Basic, Room, Stop, and Flex\. For trajectory\-class normalization, we drop non\-terminal steps with no observation change and two\-step inverse moves that return the agent to a prior state, both of which leave the environment state unchanged\.

WebShop provides shopping tasks in which an agent searches, inspects products, selects options, and purchases an item for a user instruction\. For trajectory\-class normalization, the class combines the purchased item, selected options, and normalized trajectory structure, making coverage invariant to surface variation in the action trace\. Figure[8](https://arxiv.org/html/2609.10052#A3.F8)shows how this rule merges query variants along the same route while distinguishing different search and navigation routes to the same purchase\.

Same class \(query variation\)\.Two successful rollouts begin withsearch\[blue coated steel end table\]andsearch\[blue\-coated steel end table\], follow the same normalized trajectory structure, and purchase itemB08MF23ZPLwith theblueoption\.Different classes \(route variation\)\.Two successful DDO rollouts purchase itemB09P572DP9with theredblackoption\. Their normalized routes are search→\\rightarrowproduct inspection→\\rightarrowrefined search→\\rightarrowpurchase and search→\\rightarrowpagination→\\rightarrowrefined search→\\rightarrowpurchase\.Figure 8:WebShop trajectory\-class examples\.Example input prompts and LLM outputs for all three benchmarks are shown in Appendix[D](https://arxiv.org/html/2609.10052#A4)\.

### C\.2Models and Optimization

For the experiments, expert trajectories are collected withQwen3\.5\-122B\-A10B\-FP8under the shared thought\-action format, and all optimized models useQwen3\-1\.7B\([Yang et al\., 2025](https://arxiv.org/html/2609.10052#bib.bib2);[Qwen Team, 2026](https://arxiv.org/html/2609.10052#bib.bib3)\)\. The FP8 model provides the expert trajectory source\. SFT and preference post\-training for the target model use bf16 mixed precision\. Base denotes the unadapted target model before SFT or preference optimization for each task\. For each benchmark task, the Reference model is a shared SFT adapter trained from the base model for 10 epochs\. DPO[Rafailov et al\. \(2023\)](https://arxiv.org/html/2609.10052#bib.bib4), DivFreq, DivProb, TieDPO\-RK, TieDPO\-Dav, and DDO all start from this same Reference adapter, which is also used as the frozen reference model\.

LoRA\([Hu et al\., 2022](https://arxiv.org/html/2609.10052#bib.bib20)\)is applied to the Q/K/V/O attention projections and MLP projections, with rank3232, alpha3232, and dropout0\.050\.05\. All preference runs use AdamW\([Loshchilov and Hutter, 2019](https://arxiv.org/html/2609.10052#bib.bib21)\)with betas\(0\.9,0\.999\)\(0\.9,0\.999\), epsilon10−810^\{\-8\}, weight decay0\.00\.0, gradient clipping at norm1\.01\.0, and a linear learning\-rate schedule\. Unless otherwise stated, preference\-trained methods use learning rate5×10−65\\times 10^\{\-6\}, warmup ratio0\.030\.03, andβ=0\.1\\beta=0\.1\. DDO usesα=0\.5\\alpha=0\.5\. To constructqrefq\_\{\\mathrm\{ref\}\}, the reference model scores the executable action field of each branch output\.

### C\.3DTC Collection Cost

Table 11:Collection cost per unique successful trajectory on BabyAI and BabaIsAI\.On BabyAI and BabaIsAI, DTC uses 44% of the expert requests, 38% of the expert tokens, and 75% of the environment steps used by Without DTC per unique successful trajectory\. RTO requires no additional expert calls or environment interactions\.

### C\.4Evaluation and Ablation Protocols

A fixed seed protocol is used throughout\. Training collection and evaluation use disjoint task seeds or WebShop evaluation sessions\. BabyAI and BabaIsAI use the epoch\-5 preference policies, while WebShop uses the epoch\-15 preference policies\. Main BabyAI and BabaIsAI results use sampled decoding with a temperature of0\.60\.6, top\-ppof0\.950\.95, and a max token limit of81928192\. WebShop coverage evaluation uses the same model and decoding/environment configuration as the corresponding success\-rate evaluation\.

#### Main results\.

For BabyAI and BabaIsAI, success rate is computed from 50 rollouts per evaluation item\. H\-ESD and ESD use 20 rollouts for each of three seeds used for coverage evaluation and are averaged\. For WebShop, success rate is reported over 50 evaluation sessions\. WebShop coverage uses sessions500500,501501, and502502, with 20 rollouts per session and varied LLM sampling seeds\. Failed purchases, invalid actions, retry exhaustion, and max\-step failures count toward the fixed rollout denominator\. Rollouts interrupted by system failures or missing model outputs are excluded and replaced with reruns\.

#### Component comparison\.

The component comparison in[Table1](https://arxiv.org/html/2609.10052#S5.T1)uses the BabyAI and BabaIsAI evaluation protocol described above\. Settings without DTC train on same\-state comparisons derived from separately sampled rollouts: successful and failed trajectories are sampled for the same task seed, converted into a preference comparison at their first shared decision state divergence, and matched to the corresponding budget for training comparisons\. Settings with DTC use the state\-aligned branch sets\. All four settings use the corresponding DPO and DDO hyperparameters in this appendix and report benchmark averages\.

#### Strategic recovery\.

The strategic recovery probe in[Figure5](https://arxiv.org/html/2609.10052#S6.F5)is run on BabyAI\. For each method\-task pair, we use successful evaluation rollouts, sample one interior decision point per rollout, replace the source action with a different valid task action, and roll out the same model from the edited prefix\. To keep the probe balanced, the probe uses up to 30 successful rollouts per method\-task pair\. The reported value is success over valid edited rollouts\.

#### Imitation control\.

The imitation control in[Table3](https://arxiv.org/html/2609.10052#S6.T3)is run on BabyAI and BabaIsAI\. It starts from the same SFT initialization for each task and continues SFT on all successful DTC branches\. The matched condition uses the same optimizer step count as DDO\. The full\-epoch condition runs successful\-only imitation for the same nominal epoch count as DDO\. Exposure is the number of processed training examples normalized by the DDO comparison exposure\. Metrics use the two\-benchmark aggregate\.

#### Decoding controls\.

The decoding diversification ablation in[Table4](https://arxiv.org/html/2609.10052#S6.T4)reports an equal\-weight benchmark macro average over BabyAI, BabaIsAI, and WebShop\. The temperature sweep evaluates DPO atT∈\{0\.0,0\.1,0\.6,1\.0,1\.5\}T\\in\\\{0\.0,0\.1,0\.6,1\.0,1\.5\\\}with one rollout per item\. The DDO setting uses sampled decoding atT=0\.6T=0\.6\. All preference methods use epoch\-5 policies for BabyAI and BabaIsAI and epoch\-15 policies for WebShop\.

#### Coverage growth\.

Coverage growth curves use the BabyAI, BabaIsAI, and WebShop rollouts used for coverage evaluation\. For eachNN, task averages are computed within each benchmark and then summarized with the three\-benchmark aggregate\.

#### Branch\-Set Target Distribution\.

The target distribution comparison in[Table7](https://arxiv.org/html/2609.10052#A1.T7)usesβ=0\.1\\beta=0\.1\. Under the parameterization in Eq\. \([3](https://arxiv.org/html/2609.10052#S3.E3)\),α=0\.0\\alpha=0\.0targets a uniform distribution over the observed successful branches,α=0\.5\\alpha=0\.5contracts the reference log\-odds among successful branches halfway toward uniformity, andα=1\.0\\alpha=1\.0preserves the reference distribution restricted to successful branches\.

#### Margin sharpness sensitivity\.

Theβ\\betasensitivity figure fixesα=0\.5\\alpha=0\.5and variesβ\\beta, using the two\-benchmark aggregate\.

## Appendix DExample LLM Calls

This appendix shows representative input prompts and LLM outputs for each benchmark used in our evaluation\. The three benchmarks share a common thought\-action response format: the model produces a reasoning trace followed by an executable action, which a benchmark\-specific parser extracts and forwards to the environment \(Appendix[B\.1](https://arxiv.org/html/2609.10052#A2.SS1)\)\. Figures[9](https://arxiv.org/html/2609.10052#A4.F9)–[12](https://arxiv.org/html/2609.10052#A4.F12)illustrate this format on a single decision step in each benchmark\.

#### BabyAI\.

The input prompt in Figure[9](https://arxiv.org/html/2609.10052#A4.F9)contains the task instruction, a textual rendering of the gridworld observation around the agent, the recent interaction history, and the admissible action set\. The LLM output reasons over the visible objects and the current goal before emitting a single low\-level navigation or manipulation action\.

#### BabaIsAI\.

The input prompt in Figure[10](https://arxiv.org/html/2609.10052#A4.F10)additionally exposes the active rule configuration of the puzzle, since success depends on identifying and, when necessary, manipulating the rules\. The model reasons over the grid state and active rules, then outputs either a goal\-directed action or an action that changes the puzzle’s rule configuration\.

#### WebShop\.

The input prompt in Figure[12](https://arxiv.org/html/2609.10052#A4.F12)provides the user’s purchase instruction, the contents of the current page, and the available interaction options such as search, product clicks, option selection, and purchase\. The LLM output reasons about product attributes relative to the instruction before emitting a single page interaction\.

BabyAIInput prompt\.[⬇](data:text/plain;base64,W3VzZXIgbWVzc2FnZSAxXQpZb3UgYXJlIGFuIGFnZW50IHBsYXlpbmcgYSBzaW1wbGUgbmF2aWdhdGlvbiBnYW1lLiBZb3VyIGdvYWwgaXMgdG8gZ28gdG8gdGhlIGdyZWVuIGJveC4gVGhlIGZvbGxvd2luZyBhcmUgdGhlIHBvc3NpYmxlIGFjdGlvbnMgeW91IGNhbiB0YWtlIGluIHRoZSBnYW1lLCBmb2xsb3dlZCBieSBhIHNob3J0IGRlc2NyaXB0aW9uIG9mIGVhY2ggYWN0aW9uOgoKdHVybiBsZWZ0OiB0dXJuIHRvIHRoZSBsZWZ0LAp0dXJuIHJpZ2h0OiB0dXJuIHRvIHRoZSByaWdodCwKZ28gZm9yd2FyZDogdGFrZSBvbmUgc3RlcCBmb3J3YXJkLApwaWNrIHVwOiBwaWNrIHVwIHRoZSBvYmplY3QgZGlyZWN0bHkgaW4gZnJvbnQgb2YgeW91ICgxIHN0ZXAgZm9yd2FyZCksCmRyb3A6IHBsYWNlIHRoZSBvYmplY3QgeW91IGFyZSBjYXJyeWluZyBvbiBhbiBlbXB0eSB0aWxlIGRpcmVjdGx5IGluIGZyb250IG9mIHlvdSAoMSBzdGVwIGZvcndhcmQpLAp0b2dnbGU6IGludGVyYWN0IHdpdGggdGhlIG9iamVjdCBkaXJlY3RseSBpbiBmcm9udCBvZiB5b3U7IGVmZmVjdHMgZGVwZW5kIG9uIG9iamVjdCBhbmQgc3RhdGUgKDEgc3RlcCBmb3J3YXJkKS4KCkluIGEgbW9tZW50IEkgd2lsbCBwcmVzZW50IHlvdSBhbiBvYnNlcnZhdGlvbi4KClRpcHM6Ci0gVXNlICdwaWNrIHVwJyB0byBjb2xsZWN0IGNhcnJ5YWJsZSBpdGVtcyBkaXJlY3RseSBpbiBmcm9udCBvZiB5b3UgKDEgc3RlcCBmb3J3YXJkKS4KLSBVc2UgJ3RvZ2dsZScgdG8gaW50ZXJhY3Qgd2l0aCB0aGUgb2JqZWN0IGRpcmVjdGx5IGluIGZyb250IG9mIHlvdS4KLSBVc2UgJ2Ryb3AnIHRvIHBsYWNlIHRoZSBjYXJyaWVkIG9iamVjdCBvbiBhbiBlbXB0eSB0aWxlIGRpcmVjdGx5IGluIGZyb250IG9mIHlvdSAoMSBzdGVwIGZvcndhcmQpLgotIFNvbWUgYWN0aW9ucyBtYXkgaGF2ZSBubyBlZmZlY3QgaWYgdGhlaXIgcHJlY29uZGl0aW9ucyBhcmUgbm90IG1ldCAoZm9yIGV4YW1wbGUsIGJsb2NrZWQgZnJvbnQgdGlsZSBvciBubyBjYXJyaWVkIG9iamVjdCkuCi0gQXZvaWQgcmVwZWF0aW5nIHRoZSBzYW1lIGFjdGlvbiBvdmVyIGFuZCBvdmVyIGlmIHRoZSBvYnNlcnZhdGlvbiBkb2Vzbid0IGNoYW5nZS4KClBMQVkhCgpbdXNlciBtZXNzYWdlIDJdCkN1cnJlbnQgT2JzZXJ2YXRpb246CmEgd2FsbCA2IHN0ZXBzIGZvcndhcmQKYSB3YWxsIDMgc3RlcHMgbGVmdAphIGdyZWVuIGJveCAxIHN0ZXAgcmlnaHQgYW5kIDUgc3RlcHMgZm9yd2FyZAoKWW91ciByZXNwb25zZSBzaG91bGQgdXNlIHRoZSBmb2xsb3dpbmcgZm9ybWF0OgoKVGhvdWdodDogPHlvdXIgdGhvdWdodHM+CkFjdGlvbjogPHlvdXIgbmV4dCBhY3Rpb24+Cg==)\[usermessage1\]Youareanagentplayingasimplenavigationgame\.Yourgoalistogotothegreenbox\.Thefollowingarethepossibleactionsyoucantakeinthegame,followedbyashortdescriptionofeachaction:turnleft:turntotheleft,turnright:turntotheright,goforward:takeonestepforward,pickup:pickuptheobjectdirectlyinfrontofyou\(1stepforward\),drop:placetheobjectyouarecarryingonanemptytiledirectlyinfrontofyou\(1stepforward\),toggle:interactwiththeobjectdirectlyinfrontofyou;effectsdependonobjectandstate\(1stepforward\)\.InamomentIwillpresentyouanobservation\.Tips:\-Use’pickup’tocollectcarryableitemsdirectlyinfrontofyou\(1stepforward\)\.\-Use’toggle’tointeractwiththeobjectdirectlyinfrontofyou\.\-Use’drop’toplacethecarriedobjectonanemptytiledirectlyinfrontofyou\(1stepforward\)\.\-Someactionsmayhavenoeffectiftheirpreconditionsarenotmet\(forexample,blockedfronttileornocarriedobject\)\.\-Avoidrepeatingthesameactionoverandoveriftheobservationdoesn’tchange\.PLAY\!\[usermessage2\]CurrentObservation:awall6stepsforwardawall3stepsleftagreenbox1steprightand5stepsforwardYourresponseshouldusethefollowingformat:Thought:<yourthoughts\>Action:<yournextaction\>LLM output\.[⬇](data:text/plain;base64,VGhvdWdodDogSSBuZWVkIHRvIHJlYWNoIHRoZSBncmVlbiBib3guIFRoZSBncmVlbiBib3ggaXMgMSBzdGVwIHJpZ2h0IGFuZCA1IHN0ZXBzIGZvcndhcmQuIEJ1dCB0aGVyZSBpcyBhIHdhbGwgNiBzdGVwcyBmb3J3YXJkIGFuZCBhIHdhbGwgMyBzdGVwcyBsZWZ0LiBJIHNob3VsZCBmaXJzdCBtb3ZlIGZvcndhcmQgdG8gYXZvaWQgdGhlIHdhbGxzLiBMZXQncyB0cnkgbW92aW5nIGZvcndhcmQuCgpBY3Rpb246IGdvIGZvcndhcmQK)Thought:Ineedtoreachthegreenbox\.Thegreenboxis1steprightand5stepsforward\.Butthereisawall6stepsforwardandawall3stepsleft\.Ishouldfirstmoveforwardtoavoidthewalls\.Let’strymovingforward\.Action:goforwardFigure 9:Example BabyAI input prompt and LLM output\.BabaIsAIInput prompt: task instructions\.[⬇](data:text/plain;base64,W3VzZXIgbWVzc2FnZSAxXQpCYWJhSXNBSSBpcyBhIHB1enpsZSBnYW1lIHdoZXJlIHlvdSBjYW4gbWFuaXB1bGF0ZSB0aGUgcnVsZXMgb2YgZWFjaCBsZXZlbC4gVGhlIGZvbGxvd2luZyBhcmUgdGhlIHBvc3NpYmxlIGFjdGlvbnMgeW91IGNhbiB0YWtlIGluIHRoZSBnYW1lLCBmb2xsb3dlZCBieSBhIHNob3J0IGRlc2NyaXB0aW9uIG9mIGVhY2ggYWN0aW9uOgoKaWRsZTogd2FpdCBmb3Igb25lIHN0ZXAsCnVwOiB0YWtlIG9uZSBzdGVwIHVwLApyaWdodDogdGFrZSBvbmUgc3RlcCB0byB0aGUgcmlnaHQsCmRvd246IHRha2Ugb25lIHN0ZXAgZG93biwKbGVmdDogdGFrZSBvbmUgc3RlcCB0byB0aGUgbGVmdC4KClRpcHM6Ci0gRXhhbWluZSB0aGUgbGV2ZWwgY2FyZWZ1bGx5LCBub3RpbmcgYWxsIG9iamVjdHMgYW5kIHRleHQgYmxvY2tzIHByZXNlbnQuCi0gSWRlbnRpZnkgdGhlIGN1cnJlbnQgcnVsZXMsIHdoaWNoIGFyZSBmb3JtZWQgYnkgdGV4dCBibG9ja3MgaW4gdGhlIGZvcm1hdCAiW1N1YmplY3RdIElTIFtQcm9wZXJ0eV0iIChlLmcuICJCQUJBIElTIFlPVSIpLgotIENvbnNpZGVyIGhvdyB5b3UgY2FuIGNoYW5nZSBvciBjcmVhdGUgbmV3IHJ1bGVzIGJ5IG1vdmluZyB0ZXh0IGJsb2NrcyBhcm91bmQuCi0gUmVtZW1iZXIgdGhhdCB5b3UgY2FuIG9ubHkgbW92ZSBvYmplY3RzIG9yIHRleHQgdGhhdCBhcmUgbm90IGRlZmluZWQgYXMgIlNUT1AiIG9yIHNpbWlsYXIgaW1tb3ZhYmxlIHByb3BlcnRpZXMuCi0gWW91ciBnb2FsIGlzIHVzdWFsbHkgdG8gcmVhY2ggYW4gb2JqZWN0IGRlZmluZWQgYXMgIldJTiIsIGJ1dCB0aGlzIGNhbiBiZSBjaGFuZ2VkLgotIFRoaW5rIGNyZWF0aXZlbHkgYWJvdXQgaG93IGNoYW5naW5nIHJ1bGVzIGNhbiBhbHRlciB0aGUgcHJvcGVydGllcyBhbmQgYmVoYXZpb3JzIG9mIG9iamVjdHMgaW4gdW5leHBlY3RlZCB3YXlzLgotIElmIHN0dWNrLCB0cnkgYnJlYWtpbmcgYXBhcnQgZXhpc3RpbmcgcnVsZXMgb3IgZm9ybWluZyBjb21wbGV0ZWx5IG5ldyBvbmVzLgotIFNvbWV0aW1lcyB0aGUgc29sdXRpb24gaW52b2x2ZXMgbWFraW5nIHlvdXJzZWxmIGEgZGlmZmVyZW50IG9iamVjdCBvciBjaGFuZ2luZyB3aGF0IGNvdW50cyBhcyB0aGUgd2luIGNvbmRpdGlvbi4KClBMQVkhCgpbdXNlciBtZXNzYWdlIDJdCkN1cnJlbnQgT2JzZXJ2YXRpb246CkFjdGl2ZSBydWxlczoKYmFsbCBpcyB3aW4Kd2FsbCBpcyBzdG9wCmJhYmEgaXMgeW91CgpPYmplY3RzIG9uIHRoZSBtYXA6CndhbGwgNCBzdGVwcyB0byB0aGUgcmlnaHQgYW5kIDQgc3RlcCB1cApydWxlIGBiYWxsYCA3IHN0ZXBzIHRvIHRoZSByaWdodCBhbmQgNCBzdGVwIHVwCnJ1bGUgYGlzYCA4IHN0ZXBzIHRvIHRoZSByaWdodCBhbmQgNCBzdGVwIHVwCnJ1bGUgYHdpbmAgOSBzdGVwcyB0byB0aGUgcmlnaHQgYW5kIDQgc3RlcCB1cApydWxlIGB3YWxsYCAxIHN0ZXAgdG8gdGhlIGxlZnQgYW5kIDMgc3RlcCB1cApydWxlIGBpc2AgMyBzdGVwIHVwCnJ1bGUgYHN0b3BgIDEgc3RlcCB0byB0aGUgcmlnaHQgYW5kIDMgc3RlcCB1cAp3YWxsIDQgc3RlcHMgdG8gdGhlIHJpZ2h0IGFuZCAzIHN0ZXAgdXAKYmFsbCA1IHN0ZXBzIHRvIHRoZSByaWdodCBhbmQgMyBzdGVwIHVwCndhbGwgNCBzdGVwcyB0byB0aGUgcmlnaHQgYW5kIDIgc3RlcCB1cAp3YWxsIDQgc3RlcHMgdG8gdGhlIHJpZ2h0IGFuZCAxIHN0ZXAgdXAKd2FsbCA0IHN0ZXBzIHRvIHRoZSByaWdodAp3YWxsIDQgc3RlcHMgdG8gdGhlIHJpZ2h0IGFuZCAxIHN0ZXAgZG93bgpydWxlIGBiYWJhYCAxIHN0ZXAgdG8gdGhlIGxlZnQgYW5kIDIgc3RlcHMgZG93bgpydWxlIGBpc2AgMiBzdGVwcyBkb3duCnJ1bGUgYHlvdWAgMSBzdGVwIHRvIHRoZSByaWdodCBhbmQgMiBzdGVwcyBkb3duCndhbGwgNCBzdGVwcyB0byB0aGUgcmlnaHQgYW5kIDIgc3RlcHMgZG93bgoKWW91ciByZXNwb25zZSBzaG91bGQgdXNlIHRoZSBmb2xsb3dpbmcgZm9ybWF0OgoKVGhvdWdodDogPG9uZSBzaG9ydCB0aG91Z2h0PgpBY3Rpb246IDxleGFjdGx5IG9uZSBvZjogaWRsZSwgdXAsIHJpZ2h0LCBkb3duLCBsZWZ0PgoKUnVsZXM6Ci0gVGhlIEFjdGlvbiBsaW5lIG11c3QgYmUgZXhhY3RseSBvbmUgb2Y6IGlkbGUsIHVwLCByaWdodCwgZG93biwgbGVmdC4KLSBEbyBub3QgYWRkIGFueSBvdGhlciB3b3JkcywgcHVuY3R1YXRpb24sIG9yIGV4cGxhbmF0aW9uIG9uIHRoZSBBY3Rpb24gbGluZS4KLSBPdXRwdXQgZXhhY3RseSBvbmUgVGhvdWdodCBsaW5lIGFuZCBleGFjdGx5IG9uZSBBY3Rpb24gbGluZS4KLSBEbyBub3Qgb3V0cHV0IGFueXRoaW5nIGFmdGVyIHRoZSBBY3Rpb24gbGluZS4K)\[usermessage1\]BabaIsAIisapuzzlegamewhereyoucanmanipulatetherulesofeachlevel\.Thefollowingarethepossibleactionsyoucantakeinthegame,followedbyashortdescriptionofeachaction:idle:waitforonestep,up:takeonestepup,right:takeonesteptotheright,down:takeonestepdown,left:takeonesteptotheleft\.Tips:\-Examinethelevelcarefully,notingallobjectsandtextblockspresent\.\-Identifythecurrentrules,whichareformedbytextblocksintheformat"\[Subject\]IS\[Property\]"\(e\.g\."BABAISYOU"\)\.\-Considerhowyoucanchangeorcreatenewrulesbymovingtextblocksaround\.\-Rememberthatyoucanonlymoveobjectsortextthatarenotdefinedas"STOP"orsimilarimmovableproperties\.\-Yourgoalisusuallytoreachanobjectdefinedas"WIN",butthiscanbechanged\.\-Thinkcreativelyabouthowchangingrulescanalterthepropertiesandbehaviorsofobjectsinunexpectedways\.\-Ifstuck,trybreakingapartexistingrulesorformingcompletelynewones\.\-Sometimesthesolutioninvolvesmakingyourselfadifferentobjectorchangingwhatcountsasthewincondition\.PLAY\!Figure 10:Example BabaIsAI input prompt \(part 1\)\.BabaIsAI \(continued\)Input prompt: current state and output format\.[⬇](data:text/plain;base64,W3VzZXIgbWVzc2FnZSAxXQpCYWJhSXNBSSBpcyBhIHB1enpsZSBnYW1lIHdoZXJlIHlvdSBjYW4gbWFuaXB1bGF0ZSB0aGUgcnVsZXMgb2YgZWFjaCBsZXZlbC4gVGhlIGZvbGxvd2luZyBhcmUgdGhlIHBvc3NpYmxlIGFjdGlvbnMgeW91IGNhbiB0YWtlIGluIHRoZSBnYW1lLCBmb2xsb3dlZCBieSBhIHNob3J0IGRlc2NyaXB0aW9uIG9mIGVhY2ggYWN0aW9uOgoKaWRsZTogd2FpdCBmb3Igb25lIHN0ZXAsCnVwOiB0YWtlIG9uZSBzdGVwIHVwLApyaWdodDogdGFrZSBvbmUgc3RlcCB0byB0aGUgcmlnaHQsCmRvd246IHRha2Ugb25lIHN0ZXAgZG93biwKbGVmdDogdGFrZSBvbmUgc3RlcCB0byB0aGUgbGVmdC4KClRpcHM6Ci0gRXhhbWluZSB0aGUgbGV2ZWwgY2FyZWZ1bGx5LCBub3RpbmcgYWxsIG9iamVjdHMgYW5kIHRleHQgYmxvY2tzIHByZXNlbnQuCi0gSWRlbnRpZnkgdGhlIGN1cnJlbnQgcnVsZXMsIHdoaWNoIGFyZSBmb3JtZWQgYnkgdGV4dCBibG9ja3MgaW4gdGhlIGZvcm1hdCAiW1N1YmplY3RdIElTIFtQcm9wZXJ0eV0iIChlLmcuICJCQUJBIElTIFlPVSIpLgotIENvbnNpZGVyIGhvdyB5b3UgY2FuIGNoYW5nZSBvciBjcmVhdGUgbmV3IHJ1bGVzIGJ5IG1vdmluZyB0ZXh0IGJsb2NrcyBhcm91bmQuCi0gUmVtZW1iZXIgdGhhdCB5b3UgY2FuIG9ubHkgbW92ZSBvYmplY3RzIG9yIHRleHQgdGhhdCBhcmUgbm90IGRlZmluZWQgYXMgIlNUT1AiIG9yIHNpbWlsYXIgaW1tb3ZhYmxlIHByb3BlcnRpZXMuCi0gWW91ciBnb2FsIGlzIHVzdWFsbHkgdG8gcmVhY2ggYW4gb2JqZWN0IGRlZmluZWQgYXMgIldJTiIsIGJ1dCB0aGlzIGNhbiBiZSBjaGFuZ2VkLgotIFRoaW5rIGNyZWF0aXZlbHkgYWJvdXQgaG93IGNoYW5naW5nIHJ1bGVzIGNhbiBhbHRlciB0aGUgcHJvcGVydGllcyBhbmQgYmVoYXZpb3JzIG9mIG9iamVjdHMgaW4gdW5leHBlY3RlZCB3YXlzLgotIElmIHN0dWNrLCB0cnkgYnJlYWtpbmcgYXBhcnQgZXhpc3RpbmcgcnVsZXMgb3IgZm9ybWluZyBjb21wbGV0ZWx5IG5ldyBvbmVzLgotIFNvbWV0aW1lcyB0aGUgc29sdXRpb24gaW52b2x2ZXMgbWFraW5nIHlvdXJzZWxmIGEgZGlmZmVyZW50IG9iamVjdCBvciBjaGFuZ2luZyB3aGF0IGNvdW50cyBhcyB0aGUgd2luIGNvbmRpdGlvbi4KClBMQVkhCgpbdXNlciBtZXNzYWdlIDJdCkN1cnJlbnQgT2JzZXJ2YXRpb246CkFjdGl2ZSBydWxlczoKYmFsbCBpcyB3aW4Kd2FsbCBpcyBzdG9wCmJhYmEgaXMgeW91CgpPYmplY3RzIG9uIHRoZSBtYXA6CndhbGwgNCBzdGVwcyB0byB0aGUgcmlnaHQgYW5kIDQgc3RlcCB1cApydWxlIGBiYWxsYCA3IHN0ZXBzIHRvIHRoZSByaWdodCBhbmQgNCBzdGVwIHVwCnJ1bGUgYGlzYCA4IHN0ZXBzIHRvIHRoZSByaWdodCBhbmQgNCBzdGVwIHVwCnJ1bGUgYHdpbmAgOSBzdGVwcyB0byB0aGUgcmlnaHQgYW5kIDQgc3RlcCB1cApydWxlIGB3YWxsYCAxIHN0ZXAgdG8gdGhlIGxlZnQgYW5kIDMgc3RlcCB1cApydWxlIGBpc2AgMyBzdGVwIHVwCnJ1bGUgYHN0b3BgIDEgc3RlcCB0byB0aGUgcmlnaHQgYW5kIDMgc3RlcCB1cAp3YWxsIDQgc3RlcHMgdG8gdGhlIHJpZ2h0IGFuZCAzIHN0ZXAgdXAKYmFsbCA1IHN0ZXBzIHRvIHRoZSByaWdodCBhbmQgMyBzdGVwIHVwCndhbGwgNCBzdGVwcyB0byB0aGUgcmlnaHQgYW5kIDIgc3RlcCB1cAp3YWxsIDQgc3RlcHMgdG8gdGhlIHJpZ2h0IGFuZCAxIHN0ZXAgdXAKd2FsbCA0IHN0ZXBzIHRvIHRoZSByaWdodAp3YWxsIDQgc3RlcHMgdG8gdGhlIHJpZ2h0IGFuZCAxIHN0ZXAgZG93bgpydWxlIGBiYWJhYCAxIHN0ZXAgdG8gdGhlIGxlZnQgYW5kIDIgc3RlcHMgZG93bgpydWxlIGBpc2AgMiBzdGVwcyBkb3duCnJ1bGUgYHlvdWAgMSBzdGVwIHRvIHRoZSByaWdodCBhbmQgMiBzdGVwcyBkb3duCndhbGwgNCBzdGVwcyB0byB0aGUgcmlnaHQgYW5kIDIgc3RlcHMgZG93bgoKWW91ciByZXNwb25zZSBzaG91bGQgdXNlIHRoZSBmb2xsb3dpbmcgZm9ybWF0OgoKVGhvdWdodDogPG9uZSBzaG9ydCB0aG91Z2h0PgpBY3Rpb246IDxleGFjdGx5IG9uZSBvZjogaWRsZSwgdXAsIHJpZ2h0LCBkb3duLCBsZWZ0PgoKUnVsZXM6Ci0gVGhlIEFjdGlvbiBsaW5lIG11c3QgYmUgZXhhY3RseSBvbmUgb2Y6IGlkbGUsIHVwLCByaWdodCwgZG93biwgbGVmdC4KLSBEbyBub3QgYWRkIGFueSBvdGhlciB3b3JkcywgcHVuY3R1YXRpb24sIG9yIGV4cGxhbmF0aW9uIG9uIHRoZSBBY3Rpb24gbGluZS4KLSBPdXRwdXQgZXhhY3RseSBvbmUgVGhvdWdodCBsaW5lIGFuZCBleGFjdGx5IG9uZSBBY3Rpb24gbGluZS4KLSBEbyBub3Qgb3V0cHV0IGFueXRoaW5nIGFmdGVyIHRoZSBBY3Rpb24gbGluZS4K)\[usermessage2\]CurrentObservation:Activerules:balliswinwallisstopbabaisyouObjectsonthemap:wall4stepstotherightand4stepuprule‘ball‘7stepstotherightand4stepuprule‘is‘8stepstotherightand4stepuprule‘win‘9stepstotherightand4stepuprule‘wall‘1steptotheleftand3stepuprule‘is‘3stepuprule‘stop‘1steptotherightand3stepupwall4stepstotherightand3stepupball5stepstotherightand3stepupwall4stepstotherightand2stepupwall4stepstotherightand1stepupwall4stepstotherightwall4stepstotherightand1stepdownrule‘baba‘1steptotheleftand2stepsdownrule‘is‘2stepsdownrule‘you‘1steptotherightand2stepsdownwall4stepstotherightand2stepsdownYourresponseshouldusethefollowingformat:Thought:<oneshortthought\>Action:<exactlyoneof:idle,up,right,down,left\>Rules:\-TheActionlinemustbeexactlyoneof:idle,up,right,down,left\.\-Donotaddanyotherwords,punctuation,orexplanationontheActionline\.\-OutputexactlyoneThoughtlineandexactlyoneActionline\.\-DonotoutputanythingaftertheActionline\.LLM output\.[⬇](data:text/plain;base64,VGhvdWdodDogSSBuZWVkIHRvIGJyZWFrIHRoZSAiV0FMTCBJUyBTVE9QIiBydWxlIHRvIHBhc3MgdGhyb3VnaCB0aGUgd2FsbCBiYXJyaWVyIGJsb2NraW5nIHRoZSBwYXRoIHRvIHRoZSBiYWxsLgpBY3Rpb246IGxlZnQK)Thought:Ineedtobreakthe"WALLISSTOP"ruletopassthroughthewallbarrierblockingthepathtotheball\.Action:leftFigure 11:Example BabaIsAI input prompt and LLM output \(part 2\)\.WebShopInput prompt\.[⬇](data:text/plain;base64,W3VzZXIgbWVzc2FnZSAxXQpZb3UgYXJlIGEgc2VxdWVudGlhbCBXZWJTaG9wIHRleHQtZW52aXJvbm1lbnQgYWdlbnQuCllvdXIgbWlzc2lvbiBpcyB0byBzYXRpc2Z5IHRoZSBzaG9wcGluZyBpbnN0cnVjdGlvbiBieSBuYXZpZ2F0aW5nIHBhZ2VzLApzZWxlY3RpbmcgcmVxdWlyZWQgb3B0aW9ucywgYW5kIGJ1eWluZyB0aGUgYmVzdCBtYXRjaGluZyBwcm9kdWN0LgoKVmFsaWQgYWN0aW9uIGZvcm1zOgotIHNlYXJjaFtrZXl3b3Jkc106IHNlYXJjaCBmcm9tIHRoZSBzZWFyY2ggcGFnZSBvbmx5LgotIGNsaWNrW3ZhbHVlXTogY2xpY2sgYSB2aXNpYmxlIGJ1dHRvbiwgcHJvZHVjdCBpZCwgbmF2aWdhdGlvbiBjb250cm9sLCBvcHRpb24sCiAgb3IgY2xpY2tbYnV5IG5vd10uCgpDaG9vc2UgZXhhY3RseSBvbmUgdmFsaWQgYWN0aW9uIGF0IGVhY2ggZGVjaXNpb24gc3RlcC4KClt1c2VyIG1lc3NhZ2UgMl0KQ3VycmVudCBPYnNlcnZhdGlvbjoKSW5zdHJ1Y3Rpb246IFtTRVBdIGkgYW0gbG9va2luZyBmb3Igb3JhbCBoeWdpZW5lIGRlbnRhbCB0b29scyBvZiBkZXNpZ246IHNldCBvZiA0LCBhbmQgcHJpY2UgbG93ZXIgdGhhbiA0MC4wMCBkb2xsYXJzIFtTRVBdIEJhY2sgdG8gU2VhcmNoIFtTRVBdIDwgUHJldiBbU0VQXSBkZXNpZ24gW1NFUF0gZGVudGFsIGN1cmV0dGUgW1NFUF0gZGVudGFsIHByb2JlIFtTRVBdIG1vdXRoIG1pcnJvciBbU0VQXSBzZXQgb2YgNCAoZGVudGFsIGh5Z2llbmUga2l0KSBbU0VQXSB0YXJ0ZXIgc2NyYXBlciBbU0VQXSBPcmFsIENhcmUgRGVudGFsIFRvb2xzLCBQcm9mZXNzaW9uYWwgRGVudGFsIFBpY2tzIGZvciBIb21lIHVzZSwgRGVudGFsIFRhcnRlciBTY3JhcGVyIGZvciBUb290aCBTdGFpbnMgUmVtb3ZlciwgUGxhcXVlIFJlbW92ZXIgZm9yIFRlZXRoLCBEZW50aXN0IFBpY2sgU3RhaW5sZXNzIFN0ZWVsIChEZW50YWwgUHJvYmUpIFtTRVBdIFByaWNlOiAkNC45OSBbU0VQXSBSYXRpbmc6IE4uQS4gW1NFUF0gRGVzY3JpcHRpb24gW1NFUF0gRmVhdHVyZXMgW1NFUF0gUmV2aWV3cyBbU0VQXSBCdXkgTm93CgpBdmFpbGFibGUgYWN0aW9ucyBmb3IgdGhpcyBzdGF0ZToKLSBjbGlja1tiYWNrIHRvIHNlYXJjaF0KLSBjbGlja1s8IHByZXZdCi0gY2xpY2tbZGVzY3JpcHRpb25dCi0gY2xpY2tbZmVhdHVyZXNdCi0gY2xpY2tbcmV2aWV3c10KLSBjbGlja1tidXkgbm93XQotIGNsaWNrW2RlbnRhbCBjdXJldHRlXQotIGNsaWNrW2RlbnRhbCBwcm9iZV0KLSBjbGlja1ttb3V0aCBtaXJyb3JdCi0gY2xpY2tbc2V0IG9mIDQgKGRlbnRhbCBoeWdpZW5lIGtpdCldCi0gY2xpY2tbdGFydGVyIHNjcmFwZXJdCgpPdXRwdXQgZXhhY3RseSB0d28gbGluZXMgYW5kIG5vdGhpbmcgZWxzZToKClRob3VnaHQ6IDx5b3VyIHRob3VnaHRzPgpBY3Rpb246IDxvbmUgdmFsaWQgYWN0aW9uPgo=)\[usermessage1\]YouareasequentialWebShoptext\-environmentagent\.Yourmissionistosatisfytheshoppinginstructionbynavigatingpages,selectingrequiredoptions,andbuyingthebestmatchingproduct\.Validactionforms:\-search\[keywords\]:searchfromthesearchpageonly\.\-click\[value\]:clickavisiblebutton,productid,navigationcontrol,option,orclick\[buynow\]\.Chooseexactlyonevalidactionateachdecisionstep\.\[usermessage2\]CurrentObservation:Instruction:\[SEP\]iamlookingfororalhygienedentaltoolsofdesign:setof4,andpricelowerthan40\.00dollars\[SEP\]BacktoSearch\[SEP\]<Prev\[SEP\]design\[SEP\]dentalcurette\[SEP\]dentalprobe\[SEP\]mouthmirror\[SEP\]setof4\(dentalhygienekit\)\[SEP\]tarterscraper\[SEP\]OralCareDentalTools,ProfessionalDentalPicksforHomeuse,DentalTarterScraperforToothStainsRemover,PlaqueRemoverforTeeth,DentistPickStainlessSteel\(DentalProbe\)\[SEP\]Price:$4\.99\[SEP\]Rating:N\.A\.\[SEP\]Description\[SEP\]Features\[SEP\]Reviews\[SEP\]BuyNowAvailableactionsforthisstate:\-click\[backtosearch\]\-click\[<prev\]\-click\[description\]\-click\[features\]\-click\[reviews\]\-click\[buynow\]\-click\[dentalcurette\]\-click\[dentalprobe\]\-click\[mouthmirror\]\-click\[setof4\(dentalhygienekit\)\]\-click\[tarterscraper\]Outputexactlytwolinesandnothingelse:Thought:<yourthoughts\>Action:<onevalidaction\>LLM output\.[⬇](data:text/plain;base64,VGhvdWdodDogVGhlIGN1cnJlbnQgaXRlbSBpcyBhIGRlbnRhbCBwcm9iZSwgbm90IGEgc2V0IG9mIDQuIEkgbmVlZCB0byBzZWxlY3QgdGhlIHNldCBvcHRpb24uCkFjdGlvbjogY2xpY2tbc2V0IG9mIDQgKGRlbnRhbCBoeWdpZW5lIGtpdCldCg==)Thought:Thecurrentitemisadentalprobe,notasetof4\.Ineedtoselectthesetoption\.Action:click\[setof4\(dentalhygienekit\)\]Figure 12:Example WebShop input prompt and LLM output\.