Crayotter: Learning Long-Horizon Video Editing Agents via Group-Relative Preference Backpropagation

# Crayotter: Learning Long-Horizon Video Editing Agents via Group-Relative Preference Backpropagation
Source: [https://arxiv.org/html/2608.02694](https://arxiv.org/html/2608.02694)
Lecheng Yan1Jianze Lin2Yichong Zhang3Ben Pan4Wenxi Li5Chenyang Lyu6Liting Zhou7Cathal Gurrin71University of Science and Technology of China2Beijing Normal University3Jilin University4Tianjin University5East China Normal University6Alibaba Group7Dublin City University

###### Abstract

Long\-horizon video editing agents receive final\-product feedback only after many interdependent decisions\. Yet editing quality is subjective, admits multiple valid solutions, and is not meaningfully calibrated across heterogeneous requests, making a global scalar objective both ambiguous and temporally uninformative\. Our key observation is that fixing the request, materials, and production constraints converts this subjective objective into an ordinal comparison among directly comparable alternatives\. We introduceGroup\-Relative Preference Backpropagation \(GRPB\), which transforms same\-task rankings into zero\-sum advantages and redistributes them as bounded credit over semantic editing segments\. A lagged allocator and guarded transmission prevent current judgments or unreliable estimates from directly shaping the same rollout group\. We manually construct a project\-disjoint, horizon\-stratified suite of realistic editing tasks for training and controlled evaluation\. Across matched baselines, credit interventions, external benchmarking, and blinded human evaluation, GRPB improves both editing behavior and rendered products\. The resulting 9B Crayotter model surpasses several proprietary systems on AgenticVBench, supporting task\-local preference reduction as a practical approach to learning from subjective, delayed outcomes\. Code and all supporting materials are publicly available at[https://github\.com/idwts/Crayotter](https://github.com/idwts/Crayotter)\.

## 1Introduction

An autonomous video editor must inspect source footage, select temporal regions, construct a timeline, render, and revise\. Recent systems expose this multistage structure through clip filtering and composition\(Yanget al\.,[2024](https://arxiv.org/html/2608.02694#bib.bib31)\), editor–critic interaction\(Sandoval\-Castanedaet al\.,[2025](https://arxiv.org/html/2608.02694#bib.bib32)\), and planning over hours\-long footage\(Zhaoet al\.,[2026](https://arxiv.org/html/2608.02694#bib.bib33)\)\. They do not, however, resolve how a judgment of the rendered video should train the earlier decisions that produced it\.

Execution traces provide verifiable feedback about tool completion, artifact validity, duration, and stage coverage, but not semantic selection, narrative continuity, pacing, or stylistic fidelity\. These qualities become observable only after rendering, many decisions after their causes\. They are also request dependent: different tasks admit different valid edits and weight quality dimensions differently\. An absolute score across tasks therefore entangles product quality, task difficulty, and judge\-scale variation\.

The key reduction is to condition comparison on the task\. Instead of regressing a globally calibrated quality value, we sample alternative trajectories for the same request, materials, target duration, and constraints, and ask only which final edit is preferred\. The underlying judgment may remain subjective, but the controlled context makes the candidates directly comparable and yields a concrete ordinal relation\. This converts a subjective, multi\-solution generation objective into a well\-posed within\-task preference\-learning problem without requiring cross\-task score calibration\. It also changes the appropriate unit of data construction: training examples should preserve projects and their alternative trajectories as comparison groups, rather than pool independently scored outputs across unrelated tasks\. The remaining challenge is temporal: the preference belongs to a final video, whereas policy optimization requires credit at the editing decisions that produced it\.

![Refer to caption](https://arxiv.org/html/2608.02694v1/x1.png)Figure 1:Why editing preference needs segment\-level credit\. Same\-task alternative edits make final\-product preference meaningful, but terminal rank alone does not identify where policy credit should be placed\. GRPB uses a learned, lagged allocator to localize this task\-relative signal\.Return redistribution addresses delayed outcomes\(Arjona\-Medinaet al\.,[2019](https://arxiv.org/html/2608.02694#bib.bib26)\), agentic methods estimate progress toward verifiable goals\(Wanget al\.,[2025](https://arxiv.org/html/2608.02694#bib.bib39); Xiet al\.,[2025](https://arxiv.org/html/2608.02694#bib.bib41)\), and preference\-based RL replaces scalar specification with behavioral comparisons\(Christianoet al\.,[2017](https://arxiv.org/html/2608.02694#bib.bib25)\)\. None directly resolves the combination needed here: task\-local subjective outcomes whose credit must be assigned to structured editing stages\.

We introduceGroup\-Relative Preference Backpropagation \(GRPB\)\. For each request, GRPB orders rollouts produced from the same materials and constraints, converts the order into a zero\-sum rank advantage, and allocates it over contiguous editing segments with a sparse Bradley–Terry model\. Policy credit uses the allocator before it learns from the current group\. Reliability gating, per\-segment caps, and exact return conservation bound early estimates, while raw judge magnitudes never enter the policy objective\.

We instantiate GRPB in the Crayotter editing environment\(Yanet al\.,[2026](https://arxiv.org/html/2608.02694#bib.bib54)\)\. To support training and controlled evaluation, we manually collect source materials from realistic editing projects and construct a horizon\-stratified task suite with project\-disjoint splits\. We evaluate the resulting 9B policy through matched training comparisons, an external post\-production benchmark, controlled segment\-feature interventions, and blinded human preferences\.

Our contributions are three\-fold\.

1. 1\.We formulate long\-horizon video editing as*task\-local artifact preference credit assignment*, reducing uncalibrated cross\-task quality scores to directly comparable within\-task preferences while separating verifiable process feedback from subjective final\-product assessment\.
2. 2\.We propose GRPB, which combines zero\-sum within\-task rank advantages with a lagged Bradley–Terry segment allocator, reliability gating, capped allocation, and exact return conservation\.
3. 3\.We manually construct a project\-disjoint suite of realistic editing tasks across three horizon levels\. Matched 9B experiments demonstrate gains in post\-production performance, credit localization, and blinded human preference\.

## 2Related Work

### 2\.1Relative Outcomes and Process Credit

Delayed credit assignment is a longstanding reinforcement\-learning problem\. Reward shaping introduces intermediate feedback under policy\-invariance conditions\(Nget al\.,[1999](https://arxiv.org/html/2608.02694#bib.bib48)\), whereas RUDDER redistributes delayed returns toward influential events\(Arjona\-Medinaet al\.,[2019](https://arxiv.org/html/2608.02694#bib.bib26)\)\. PPO and GAE stabilize optimization once rewards have been defined\(Schulmanet al\.,[2017](https://arxiv.org/html/2608.02694#bib.bib37);[2016](https://arxiv.org/html/2608.02694#bib.bib38)\), but do not determine how a judgment of a completed artifact should be assigned to the stages that produced it\.

Recent agent\-learning methods decompose interaction traces or infer stepwise progress from terminal supervision\(Luoet al\.,[2025](https://arxiv.org/html/2608.02694#bib.bib27); Wanget al\.,[2025](https://arxiv.org/html/2608.02694#bib.bib39); Tanet al\.,[2026](https://arxiv.org/html/2608.02694#bib.bib40); Xiet al\.,[2025](https://arxiv.org/html/2608.02694#bib.bib41)\)\. Related outcome\-to\-process methods derive process rewards without step annotations\(Yuanet al\.,[2024](https://arxiv.org/html/2608.02694#bib.bib42); Cuiet al\.,[2025](https://arxiv.org/html/2608.02694#bib.bib43); Xieet al\.,[2025](https://arxiv.org/html/2608.02694#bib.bib44)\)\. Most assume an externally verifiable terminal condition\. Group\-relative optimization instead avoids a global reward scale by comparing responses to the same prompt\(Shaoet al\.,[2024](https://arxiv.org/html/2608.02694#bib.bib23)\), but ordinarily broadcasts one trajectory\-level advantage to every action\. GRPB combines these ideas differently: it first constructs a zero\-sum ordinal advantage within a task group, then learns how to conserve and redistribute that advantage over semantically distinct segments\.

### 2\.2Preference Learning for Subjective Artifacts

Preference\-based RL replaces a difficult scalar objective with comparisons of behaviors or ranked trajectories\(Christianoet al\.,[2017](https://arxiv.org/html/2608.02694#bib.bib25); Brownet al\.,[2019](https://arxiv.org/html/2608.02694#bib.bib45); Verma and Metcalf,[2024](https://arxiv.org/html/2608.02694#bib.bib46)\)\. This is especially relevant when valid outputs are diverse and quality is not comparable across tasks\. GRPB restricts each comparison to rollouts sharing the same request, materials, and constraints, thereby reducing subjective final\-product assessment to a task\-local ordering problem\. Unlike methods that treat an outcome label as dense step correctness, it infers relative segment attribution\. Because learned proxies can decouple from held\-out quality and model judges exhibit systematic biases\(Gaoet al\.,[2023](https://arxiv.org/html/2608.02694#bib.bib51); Zhenget al\.,[2023](https://arxiv.org/html/2608.02694#bib.bib49); Liet al\.,[2025](https://arxiv.org/html/2608.02694#bib.bib30)\), GRPB discards raw judge magnitudes, uses a pre\-update allocator for policy credit, and bounds the amount assigned to any segment\.

Video editing provides a concrete instance of this setting: existing systems formulate trimming, nonlinear editing, and long\-footage composition as multistage agent workflows\(Yanget al\.,[2024](https://arxiv.org/html/2608.02694#bib.bib31); Sandoval\-Castanedaet al\.,[2025](https://arxiv.org/html/2608.02694#bib.bib32); Zhaoet al\.,[2026](https://arxiv.org/html/2608.02694#bib.bib33); Liet al\.,[2026](https://arxiv.org/html/2608.02694#bib.bib35); Zhanget al\.,[2026](https://arxiv.org/html/2608.02694#bib.bib36)\), while prior RL work optimizes sequential editing decisions\(Huet al\.,[2024](https://arxiv.org/html/2608.02694#bib.bib24)\)\. Artifact evaluators can estimate dimensions of rendered\-video quality but remain imperfect proxies\(Heet al\.,[2024](https://arxiv.org/html/2608.02694#bib.bib28); Liuet al\.,[2024](https://arxiv.org/html/2608.02694#bib.bib21)\)\. GRPB addresses the complementary learning problem of connecting task\-local judgments of rendered artifacts to reliable intermediate policy credit\.

![Refer to caption](https://arxiv.org/html/2608.02694v1/x2.png)Figure 2:GRPB training architecture\. Executable process feedback and task\-relative final\-product preference meet at segment endpoints\. Policy credit uses the pre\-update allocator; same\-group comparisons train only the allocator used by subsequent batches\.

## 3Method: Group\-Relative Preference Backpropagation

We instantiate GRPB in the Crayotter editing environment\(Yanet al\.,[2026](https://arxiv.org/html/2608.02694#bib.bib54)\), while formulating the method over an artifact\-observable interface rather than a system\-specific workflow\.

### 3\.1Crayotter Editing Environment

The Crayotter editing environment presents the agent with a request, source\-material pool, multimodal analyses, editing blueprint, and workspace artifacts\. Structured analysis, timeline, inspection, repair, and export calls update the workspace while recording execution and artifact diagnostics\. GRPB operates on four observables—segmented traces, rendered outcomes, process diagnostics, and same\-task rollout groups—provided by this environment\.

### 3\.2Problem Formulation and Process Return

We model editing as a finite\-horizon augmented MDP

ℳ=\(𝒮,𝒜,P,ρ0,r,γ,T\)\.\\mathcal\{M\}=\(\\mathcal\{S\},\\mathcal\{A\},P,\\rho\_\{0\},r,\\gamma,T\)\.The statests\_\{t\}summarizes the request and constraints, artifact tree, blueprint coverage, workspace, tool history, rendered previews, and prior video for revision tasks;ata\_\{t\}is a structured editing\-tool call\.

For a fixed taskqq, we sample a rollout group𝒢q=\{τ\(k\)\}k=1K\\mathcal\{G\}\_\{q\}=\\\{\\tau^\{\(k\)\}\\\}\_\{k=1\}^\{K\}whose members share the same request, fixture, materials, target duration, and constraints\. Only a valid export is preference\-eligible\. A product\-only multimodal judge observes the request and ordered frames of the final video and returnsJk∈\[0,100\]J\_\{k\}\\in\[0,100\]; it does not observe the hidden tool trace or blueprint\.JkJ\_\{k\}is used only for within\-group order, never as a calibrated reward magnitude\.

The process channel

rtproc=ρproc​\(st,at,st\+1\),r\_\{t\}^\{\\mathrm\{proc\}\}=\\rho\_\{\\mathrm\{proc\}\}\(s\_\{t\},a\_\{t\},s\_\{t\+1\}\),scores tool and artifact validity, export completion, duration, and stage progress\. It supports executable rollouts, while final\-product preference covers qualities observable only after rendering\. Consecutive events with the same editing\-stage label form a segment; process residuals and preference credit are placed at segment endpoints\.

#### Artifact\-level interface\.

Process diagnostics verify executable progress but cannot judge narrative rhythm, continuity, or stylistic fidelity; final\-product preference captures these qualities but does not identify their causes\. Persistent workspace artifacts bridge the two: contiguous events that transform the same editing stage define a segment to which delayed preference can be returned\. GRPB therefore requires observable intermediate artifacts, meaningful segmentation, and same\-task alternative trajectories rather than the Crayotter agent’s internal organization\.

### 3\.3Group\-Relative Rank Advantage

For an eligible group of at leastKminK\_\{\\min\}exports, GRPB converts judge scores to pairwise win/loss advantage with tie toleranceϵJ\\epsilon\_\{J\}:

Akrank=1K−1​∑j≠k\[𝟏​\(Jk−Jj\>ϵJ\)−𝟏​\(Jj−Jk\>ϵJ\)\]\.A\_\{k\}^\{\\mathrm\{rank\}\}=\\frac\{1\}\{K\-1\}\\sum\_\{j\\neq k\}\\left\[\\mathbf\{1\}\(J\_\{k\}\-J\_\{j\}\>\\epsilon\_\{J\}\)\-\\mathbf\{1\}\(J\_\{j\}\-J\_\{k\}\>\\epsilon\_\{J\}\)\\right\]\.This statistic depends only on within\-task order, lies in\[−1,1\]\[\-1,1\], and satisfies∑kAkrank=0\\sum\_\{k\}A\_\{k\}^\{\\mathrm\{rank\}\}=0exactly\. Thus score offsets, scale changes that do not alter ties, and cross\-task calibration do not affect the policy signal\. When rollouts share an explicitly recorded counterfactual prefix, only segments after the branch point are attributable; otherwise the full trajectory is used\.

### 3\.4Lagged Pairwise Segment Allocator

For segmentmmof rolloutkk, a sparse vectorfk,mf\_\{k,m\}encodes stage identity, execution and artifact statistics, trajectory position, repair evidence, request alignment, and optional semantic deltas\. We define its predictive contributionuk,m=w⊤​fk,mu\_\{k,m\}=w^\{\\top\}f\_\{k,m\}and aggregate rollout utilityz^k=Mk−1​∑muk,m=w⊤​f¯k\\hat\{z\}\_\{k\}=M\_\{k\}^\{\-1\}\\sum\_\{m\}u\_\{k,m\}=w^\{\\top\}\\bar\{f\}\_\{k\}, wheref¯k=Mk−1​∑mfk,m\\bar\{f\}\_\{k\}=M\_\{k\}^\{\-1\}\\sum\_\{m\}f\_\{k,m\}\. A Bradley–Terry model\(Bradley and Terry,[1952](https://arxiv.org/html/2608.02694#bib.bib53)\)predicts

pw​\(k≻j\)=σ​\(z^k−z^j\)\.p\_\{w\}\(k\\succ j\)=\\sigma\(\\hat\{z\}\_\{k\}\-\\hat\{z\}\_\{j\}\)\.For every non\-tied ordered pair in the current group, the allocator minimizes

ℒBT​\(w\)=−∑Jk−Jj\>ϵJlog⁡pw​\(k≻j\)\+λw2​∥w∥22,\\mathcal\{L\}\_\{\\mathrm\{BT\}\}\(w\)=\-\\sum\_\{J\_\{k\}\-J\_\{j\}\>\\epsilon\_\{J\}\}\\log p\_\{w\}\(k\\succ j\)\+\\frac\{\\lambda\_\{w\}\}\{2\}\\lVert w\\rVert\_\{2\}^\{2\},using online AdaGrad with clipped weights\. This is weak supervision at the rollout level: it assumes that predicted final\-product utility can be represented by the mean of segment feature contributions\. Under this additive model, the regularized objective selects a uniquewwanduk,mu\_\{k,m\}is exactly the term contributed by segmentmmtoz^k\\hat\{z\}\_\{k\}, so the score used for allocation is algebraically consistent with the score used for ranking\. We interpretuk,mu\_\{k,m\}as predictive policy credit rather than an independently observed causal effect; Section[4\.4](https://arxiv.org/html/2608.02694#S4.SS4)evaluates whether this learned decomposition localizes held\-out single\-segment feature interventions\.

The update order is essential\. Letw−w^\{\-\}be the state loaded before observing the current group\. GRPB first computes all policy credit withw−w^\{\-\}, then evaluates its pre\-update ranking accuracy, and only afterward updates and savesww\. Consequently, current labels cannot be memorized and immediately returned as credit to the same batch\. Validation freezes both allocator updates and policy\-side preference credit\.

### 3\.5Reliability\-Gated Preference Budget

Early allocator estimates are gated by a reliability coefficient\. WithNcalN\_\{\\mathrm\{cal\}\}accumulated pre\-update pair observations, warm\-up targetNwarmN\_\{\\mathrm\{warm\}\}, exponentially averaged accuracyaa, and target accuracya∗a\_\{\*\}, we use

g=min⁡\(1,Ncal8​Nwarm\)​clip⁡\(a−0\.5a∗−0\.5,0,1\)\.g=\\min\\\!\\left\(1,\\sqrt\{\\frac\{N\_\{\\mathrm\{cal\}\}\}\{8N\_\{\\mathrm\{warm\}\}\}\}\\right\)\\operatorname\{clip\}\\\!\\left\(\\frac\{a\-0\.5\}\{a\_\{\*\}\-0\.5\},0,1\\right\)\.Credit is disabled before warm\-up or whenggis below a threshold\. Otherwise, the shared group budget is

B=g​min⁡\(Bmax,Mmin​C\),bk=Akrank​B,B=g\\min\(B\_\{\\max\},M\_\{\\min\}C\),\\qquad b\_\{k\}=A\_\{k\}^\{\\mathrm\{rank\}\}B,whereMminM\_\{\\min\}is the minimum attributable segment count andCCis the per\-segment absolute cap\. Scaling by the least segment\-rich rollout ensures that everybkb\_\{k\}is feasible without giving longer traces a larger budget\.

![Refer to caption](https://arxiv.org/html/2608.02694v1/x3.png)Figure 3:Lagged and bounded segment allocation\. The sparse pre\-update Bradley–Terry model produces feature\-additive segment scores\. Signed capped allocation conserves each rollout budget and yields exact zero\-sum preference pressure across the task\-local group\.
### 3\.6Signed Capped Segment Allocation

Letsk=sgn⁡\(bk\)s\_\{k\}=\\operatorname\{sgn\}\(b\_\{k\}\)\. For each segment, define

qk,m=exp⁡\(sk​uk,m/τalloc\),uk,m=\(w−\)⊤​fk,m,q\_\{k,m\}=\\exp\\\!\\left\(s\_\{k\}u\_\{k,m\}/\\tau\_\{\\mathrm\{alloc\}\}\\right\),\\qquad u\_\{k,m\}=\(w^\{\-\}\)^\{\\top\}f\_\{k,m\},using the allocator frozen before the current rollout group\. Forbk≠0b\_\{k\}\\neq 0and nonconstant scores, GRPB applies

ck,m=sk​min⁡\{C,λk​qk,m\},∑mmin⁡\{C,λk​qk,m\}=\|bk\|,c\_\{k,m\}=s\_\{k\}\\min\\\{C,\\lambda\_\{k\}q\_\{k,m\}\\\},\\qquad\\sum\_\{m\}\\min\\\{C,\\lambda\_\{k\}q\_\{k,m\}\\\}=\|b\_\{k\}\|,whereλk\\lambda\_\{k\}is the smallest nonnegative solution\. The implementation finds it by water\-filling: fix every segment whose proportional share reachesCC, remove it from the active set, and redistribute the remaining mass according to the unchangedqk,mq\_\{k,m\}\. Thus positive budgets favor high scores, whereas negative budgets assign larger\-magnitude penalties to low scores\. Tied segments receive equal shares; if all scores in any rollout are equal within10−810^\{\-8\}, group credit is withheld rather than uniformly broadcast\.

#### Conservation\.

Feasibility follows from\|bk\|≤B≤Mmin​C≤Mk​C\|b\_\{k\}\|\\leq B\\leq M\_\{\\min\}C\\leq M\_\{k\}C\. The function∑mmin⁡\{C,λ​qk,m\}\\sum\_\{m\}\\min\\\{C,\\lambda q\_\{k,m\}\\\}is continuous and nondecreasing from0toMk​CM\_\{k\}C, so a solution exists and obeys\|ck,m\|≤C\|c\_\{k,m\}\|\\leq Cand∑mck,m=bk\\sum\_\{m\}c\_\{k,m\}=b\_\{k\}\. Group\-level zero\-sum then follows directly:

∑k∑mck,m=∑kbk=B​∑kAkrank=0\.\\sum\_\{k\}\\sum\_\{m\}c\_\{k,m\}=\\sum\_\{k\}b\_\{k\}=B\\sum\_\{k\}A\_\{k\}^\{\\mathrm\{rank\}\}=0\.

#### Reward placement\.

Credits are stored to six decimals, with any rounding residual assigned to an unsaturated segment so that conservation remains exact\. Allocation is over semantic editing segments rather than token counts:ck,mc\_\{k,m\}is placed at the final trainable assistant position associated with that segment, preventing long verbal spans from receiving more preference mass merely because they contain more tokens\. The rule\-based return recorded for the trajectory is unchanged, while the policy\-side return gainsbkb\_\{k\}\. Raw judge scores remain available for audit but never enter the reward tensor\.

### 3\.7Critic\-Based Policy Optimization

If segmentmmof rolloutkkends at steptt, its credit is added to the process reward:

rt\(k\)=rtproc,\(k\)\+∑m:e​\(k,m\)=tck,m\.r\_\{t\}^\{\(k\)\}=r\_\{t\}^\{\\mathrm\{proc\},\(k\)\}\+\\sum\_\{m:\\,e\(k,m\)=t\}c\_\{k,m\}\.Given a value functionVψV\_\{\\psi\}, we compute

δt=rt\+γ​Vψ​\(st\+1\)−Vψ​\(st\),A^tGAE=∑ℓ=0T−1−t\(γ​λGAE\)ℓ​δt\+ℓ\.\\delta\_\{t\}=r\_\{t\}\+\\gamma V\_\{\\psi\}\(s\_\{t\+1\}\)\-V\_\{\\psi\}\(s\_\{t\}\),\\quad\\hat\{A\}\_\{t\}^\{\\mathrm\{GAE\}\}=\\sum\_\{\\ell=0\}^\{T\-1\-t\}\(\\gamma\\lambda\_\{\\mathrm\{GAE\}\}\)^\{\\ell\}\\delta\_\{t\+\\ell\}\.The policy is updated with the clipped PPO surrogate\(Schulmanet al\.,[2017](https://arxiv.org/html/2608.02694#bib.bib37)\)

ℒCLIP​\(θ\)=𝔼t​\[min⁡\(ρt​\(θ\)​A^tGAE,clip⁡\(ρt​\(θ\),1−ϵ,1\+ϵ\)​A^tGAE\)\],\\mathcal\{L\}\_\{\\mathrm\{CLIP\}\}\(\\theta\)=\\mathbb\{E\}\_\{t\}\\\!\\left\[\\min\\\!\\left\(\\rho\_\{t\}\(\\theta\)\\hat\{A\}\_\{t\}^\{\\mathrm\{GAE\}\},\\operatorname\{clip\}\\\!\\left\(\\rho\_\{t\}\(\\theta\),1\-\\epsilon,1\+\\epsilon\\right\)\\hat\{A\}\_\{t\}^\{\\mathrm\{GAE\}\}\\right\)\\right\],whereρt​\(θ\)=πθ​\(at∣st\)/πθold​\(at∣st\)\\rho\_\{t\}\(\\theta\)=\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)/\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(a\_\{t\}\\mid s\_\{t\}\)\. The distinction from GRPO is not the clipped likelihood ratio, which both methods use\. GRPB retains the learned value baseline and applies GAE\(Schulmanet al\.,[2016](https://arxiv.org/html/2608.02694#bib.bib38)\)to obtain step\-varying advantages from the localized segment rewards; the rollout group is used only to construct preference budgets\. GRPO instead replaces the critic\-based advantage with a trajectory\-level group\-relative scalar that is ordinarily shared by all actions in the response\. Complete implementation and optimization configurations are provided in the Supplementary Material\.

## 4Experiments

### 4\.1Experimental Setup

#### Training and evaluation data\.

We manually collect source materials for 23 realistic post\-production projects and construct each project with a user brief, target duration, multimodal analyses, and an executable workspace; revision tasks additionally include an earlier render and natural\-language feedback\. The resulting 70 tasks comprise 12 normal, 12 medium\-horizon, and 46 long\-horizon cases, progressing from single\-pass production to multi\-constraint composition and iterative revision through increased stage coverage, dependency, and revision depth\. We split projects before task construction: 18 yield 54 training tasks \(9/9/36 by horizon\) and five yield 16 held\-out tasks \(3/3/10\), with no request, footage, render, or revision chain crossing partitions\. Alternative rollouts within a task share the complete production context and form the comparison groups required by GRPB\. AgenticVBench\(Caoet al\.,[2026](https://arxiv.org/html/2608.02694#bib.bib55)\)is a separate post\-training benchmark\.

#### Matched protocol\.

All variants share the same 9B initialization, tool interface, rollout conditions, process supervision, product judge, and optimization budget\. Process feedback rewards executable progress, whereas the compared methods differ only in how same\-task final\-product preference is converted into policy credit\. Evaluation combines matched baselines and ablations, the disjoint external benchmark, controlled credit\-localization interventions, and blinded human preference\.

![Refer to caption](https://arxiv.org/html/2608.02694v1/x4.png)Figure 4:GRPB training dynamics\. Thin curves are per\-update statistics; thick curves are trailing seven\-update EWMAs\. Panels show \(a\) value loss, \(b\) valid export rate, \(c\) frozen\-allocator accuracy before the update, and \(d\) reliability versus applied preference credit\.\(a\) AgenticVBench results \(%\)

\(b\) Training means over Steps 11–100 \(%\)

Table 1:Matched 9B comparison\. AgenticVBench values are final\-checkpoint scores; training statistics are averaged over Steps 11–100\. Credit coverage is the fraction of episodes receiving nonzero preference credit\.

### 4\.2Training Dynamics

Figure[4](https://arxiv.org/html/2608.02694#S4.F4)separates critic fitting, executable output, allocator calibration, and credit transmission\. Value loss declines while valid export remains high across varying task groups\. Pre\-update allocator accuracy stays above chance over most updates, excluding same\-batch fitting as its explanation\. Reliability and applied credit rise as calibration evidence accumulates; their gap reflects the additional validity, ranking, and segment\-contrast conditions required before preference reaches the policy\.

### 4\.3Comparative Evaluation and Ablations

Base denotes the original 9B weights\. Process\-PPO uses process reward only; Terminal Rank PPO places rank credit at the trajectory endpoint; and uniform allocation divides it over segments\. The no\-lag and no\-safeguard variants respectively remove pre\-update allocation and guarded transmission\. All trained variants otherwise use matched conditions\.

GRPB obtains the highest matched AVG \(15\.7\) and Repurpose score \(23\.0\), exceeding Process\-PPO by 2\.0 and 2\.8 points\. Terminal and uniform credit both underperform, showing that task\-local preference must be localized\. Removing lagging or safeguards also falls below Process\-PPO, whereas full GRPB credits 49\.4% of episodes while retaining 81\.3% valid export\. The gain therefore requires differentiated and guarded credit\.

#### External benchmark\.

AgenticVBench\(Caoet al\.,[2026](https://arxiv.org/html/2608.02694#bib.bib55)\)evaluates repurposing, sequencing, repair, and assembly on 100 disjoint tasks\. Table[2](https://arxiv.org/html/2608.02694#S4.T2)follows its official rubrics, weighting, and judges\. Because model and harness both vary, it measures external performance rather than isolating GRPB\.

\(a\) Closed\-source models

\(b\) Open\-source models

Table 2:AgenticVBench results \(%\) for representative closed\- and open\-source models\. Crayotter\-9B is our 9B model trained with GRPB\. All scores follow the official Pillar 0–4 rubrics, judge models \(Claude Opus 4\.7 and Gemini 3\.1 Pro\), and 100\-task weighting\. Repurpose, the most editing\-aligned pillar, is shaded\.Crayotter\-9B ranks third among the listed systems\(Team,[2026a](https://arxiv.org/html/2608.02694#bib.bib56);[2025](https://arxiv.org/html/2608.02694#bib.bib57); Qwen Team,[2026](https://arxiv.org/html/2608.02694#bib.bib58); Team,[2026b](https://arxiv.org/html/2608.02694#bib.bib60); Baiet al\.,[2025](https://arxiv.org/html/2608.02694#bib.bib59)\)and exceeds several proprietary alternatives\. It matches Gemini 3\.1 Pro on Repurpose but remains weaker on sequencing\. Together, Table[1](https://arxiv.org/html/2608.02694#S4.T1)attributes the gain to localized, guarded credit, while Table[2](https://arxiv.org/html/2608.02694#S4.T2)shows transfer beyond the training environment without claiming uniform improvement across post\-production skills\.

### 4\.4Credit\-Assignment Analysis

We construct 168 evaluation\-only controlled pairs from 64 trajectories in the five held\-out projects\. Each pair degrades only one segment’s execution, artifact, and semantic features while fixing the request and all other segments; the probes never update the allocator\. They cover material selection, temporal assembly, narration/subtitle alignment, and final checks\. Top\-1 localization and target mass measure where credit is placed, while pairwise accuracy and log loss test whether the frozen predictor orders the favorable record above its degraded counterpart\. The diagnostic bypasses reliability gating to evaluate the learned score but retains the common cap and conservation projection\.

GRPB raises Top\-1 localization from 11\.9% to 20\.8% and reaches 85\.7% pairwise accuracy without cap violations\. Uniform allocation cannot distinguish the intervened segment, while terminal broadcast places reward at an unrelated endpoint and violates the cap by construction\. GRPB improves both localization and ordering while preserving the exact budget\. The smaller changes in target mass and log loss indicate that identifying a useful credit direction is easier than calibrating its full distribution, motivating reliability gating and bounded allocation on the policy path\.

### 4\.5Blinded Human Preference Evaluation

Three human judges independently compare anonymized, randomly ordered outputs from identical held\-out requests, sources, and decoding budgets, considering instruction satisfaction, content selection, coherence, fluency, and audiovisual quality\. Win and loss rates are averaged per judge to weight evaluators equally, and net win rate is reported from the GRPB perspective\.

Table 3:Credit localization on 168 held\-out single\-segment feature interventions\. Target mass is the fraction of absolute credit assigned to the changed segment\. Conservation error is zero for all methods\.Table 4:Blinded same\-task preference from the GRPB perspective, averaged across three human judges\.GRPB achieves positive net win rates against every opponent: 34\.2 points over Base, 8\.0 over Process\-PPO, 3\.2 over Terminal Rank PPO, and 11\.8 over uniform allocation\. The gain over the original weights confirms improved rendered products, while the margins over Terminal Rank PPO and uniform allocation support task\-local ordering and differentiated segment credit\.

## 5Conclusion

We introduced GRPB, a preference\-to\-process reinforcement learning method that converts same\-task comparisons into zero\-sum rank advantages, localizes them through a lagged and bounded segment allocator, and optimizes the resulting credit with critic\-based PPO and GAE\. Matched experiments, controlled credit interventions, external benchmarking, and blinded human evaluation show that localized preference credit improves agent behavior and rendered\-video quality; Crayotter\-9B also ranks above several proprietary systems on AgenticVBench\. More broadly, task\-local comparison, conservative credit allocation, and artifact\-level evaluation provide a practical foundation for other subjective, multi\-solution, multistage production tasks\.

## References

- J\. A\. Arjona\-Medina, M\. Gillhofer, M\. Widrich, T\. Unterthiner, J\. Brandstetter, and S\. Hochreiter \(2019\)RUDDER: return decomposition for delayed rewards\.InAdvances in Neural Information Processing Systems,Vol\.32\.Cited by:[§1](https://arxiv.org/html/2608.02694#S1.p4.1),[§2\.1](https://arxiv.org/html/2608.02694#S2.SS1.p1.1)\.
- Qwen3\-vl technical report\.arXiv preprint arXiv:2511\.21631\.Cited by:[§4\.3](https://arxiv.org/html/2608.02694#S4.SS3.SSS0.Px1.p2.1)\.
- R\. A\. Bradley and M\. E\. Terry \(1952\)Rank analysis of incomplete block designs: i\. the method of paired comparisons\.Biometrika39\(3/4\),pp\. 324–345\.Cited by:[§3\.4](https://arxiv.org/html/2608.02694#S3.SS4.p1.6)\.
- D\. S\. Brown, W\. Goo, P\. Nagarajan, and S\. Niekum \(2019\)Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations\.InInternational Conference on Machine Learning,pp\. 783–792\.Cited by:[§2\.2](https://arxiv.org/html/2608.02694#S2.SS2.p1.1)\.
- Z\. Cao, Y\. Zheng, R\. Song, and X\. Hu \(2026\)AgenticVBench: can ai agents complete real\-world post\-production tasks?\.arXiv preprint arXiv:2605\.27705\.Cited by:[§4\.1](https://arxiv.org/html/2608.02694#S4.SS1.SSS0.Px1.p1.1),[§4\.3](https://arxiv.org/html/2608.02694#S4.SS3.SSS0.Px1.p1.1)\.
- P\. F\. Christiano, J\. Leike, T\. B\. Brown, M\. Martic, S\. Legg, and D\. Amodei \(2017\)Deep reinforcement learning from human preferences\.InAdvances in Neural Information Processing Systems,Vol\.30\.Cited by:[§1](https://arxiv.org/html/2608.02694#S1.p4.1),[§2\.2](https://arxiv.org/html/2608.02694#S2.SS2.p1.1)\.
- G\. Cui, L\. Yuan, Z\. Wang, H\. Wang, W\. Li, B\. He, Y\. Fan, T\. Yu, Q\. Xu, W\. Chen,et al\.\(2025\)Process reinforcement through implicit rewards\.arXiv preprint arXiv:2502\.01456\.Cited by:[§2\.1](https://arxiv.org/html/2608.02694#S2.SS1.p2.1)\.
- L\. Gao, J\. Schulman, and J\. Hilton \(2023\)Scaling laws for reward model overoptimization\.InInternational Conference on Machine Learning,pp\. 10835–10866\.Cited by:[§2\.2](https://arxiv.org/html/2608.02694#S2.SS2.p1.1)\.
- X\. He, D\. Jiang, G\. Zhang, M\. Ku, A\. Soni, S\. Siu, H\. Chen, A\. Chandra, Z\. Jiang, A\. Arulraj, K\. Wang, Q\. D\. Do, Y\. Zhang, B\. Luo, S\. Geng, Z\. Wang, and W\. Chen \(2024\)VideoScore: building automatic metrics to simulate fine\-grained human feedback for video generation\.External Links:2406\.15252,[Document](https://dx.doi.org/10.48550/arXiv.2406.15252)Cited by:[§2\.2](https://arxiv.org/html/2608.02694#S2.SS2.p2.1)\.
- P\. Hu, N\. Xiao, F\. Li, Y\. Chen, and R\. Huang \(2024\)A reinforcement learning\-based automatic video editing method using pre\-trained vision\-language model\.External Links:2411\.04942,[Document](https://dx.doi.org/10.48550/arXiv.2411.04942)Cited by:[§2\.2](https://arxiv.org/html/2608.02694#S2.SS2.p2.1)\.
- D\. Li, B\. Jiang, L\. Huang, A\. Beigi, C\. Zhao, Z\. Tan, A\. Bhattacharjee, Y\. Jiang, C\. Chen, T\. Wu, K\. Shu, L\. Cheng, and H\. Liu \(2025\)From generation to judgment: opportunities and challenges of llm\-as\-a\-judge\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.138)Cited by:[§2\.2](https://arxiv.org/html/2608.02694#S2.SS2.p1.1)\.
- K\. Li, M\. Li, J\. Chen, J\. Chen, Z\. Zheng, S\. Wang, and X\. Chen \(2026\)DIRECT: video mashup creation via hierarchical multi\-agent planning and intent\-guided editing\.arXiv preprint arXiv:2604\.04875\.Cited by:[§2\.2](https://arxiv.org/html/2608.02694#S2.SS2.p2.1)\.
- Y\. Liu, X\. Cun, X\. Liu, X\. Wang, Y\. Zhang, H\. Chen, Y\. Liu, T\. Zeng, R\. Chan, and Y\. Shan \(2024\)Evalcrafter: benchmarking and evaluating large video generation models\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 22139–22149\.Cited by:[§2\.2](https://arxiv.org/html/2608.02694#S2.SS2.p2.1)\.
- X\. Luo, Y\. Zhang, Z\. He, Z\. Wang, S\. Zhao, D\. Li, L\. K\. Qiu, and Y\. Yang \(2025\)Agent lightning: train any AI agents with reinforcement learning\.External Links:2508\.03680,[Document](https://dx.doi.org/10.48550/arXiv.2508.03680)Cited by:[§2\.1](https://arxiv.org/html/2608.02694#S2.SS1.p2.1)\.
- A\. Y\. Ng, D\. Harada, and S\. Russell \(1999\)Policy invariance under reward transformations: theory and application to reward shaping\.InInternational Conference on Machine Learning,pp\. 278–287\.Cited by:[§2\.1](https://arxiv.org/html/2608.02694#S2.SS1.p1.1)\.
- Qwen Team \(2026\)Qwen3\.5: towards native multimodal agents\.External Links:[Link](https://qwen.ai/blog?id=qwen3.5)Cited by:[§4\.3](https://arxiv.org/html/2608.02694#S4.SS3.SSS0.Px1.p2.1)\.
- M\. Sandoval\-Castaneda, B\. Russell, J\. Sivic, G\. Shakhnarovich, and F\. Caba Heilbron \(2025\)EditDuet: a multi\-agent system for video non\-linear editing\.External Links:2509\.10761,[Link](https://arxiv.org/abs/2509.10761)Cited by:[§1](https://arxiv.org/html/2608.02694#S1.p1.1),[§2\.2](https://arxiv.org/html/2608.02694#S2.SS2.p2.1)\.
- J\. Schulman, P\. Moritz, S\. Levine, M\. Jordan, and P\. Abbeel \(2016\)High\-dimensional continuous control using generalized advantage estimation\.InInternational Conference on Learning Representations,Cited by:[§2\.1](https://arxiv.org/html/2608.02694#S2.SS1.p1.1),[§3\.7](https://arxiv.org/html/2608.02694#S3.SS7.p1.5)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§2\.1](https://arxiv.org/html/2608.02694#S2.SS1.p1.1),[§3\.7](https://arxiv.org/html/2608.02694#S3.SS7.p1.6)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\.K\. Li, Y\. Wu, and D\. Guo \(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.External Links:2402\.03300,[Document](https://dx.doi.org/10.48550/arXiv.2402.03300)Cited by:[§2\.1](https://arxiv.org/html/2608.02694#S2.SS1.p2.1)\.
- H\. Tan, X\. Yang, H\. Chen, J\. Shao, Y\. Wen, Y\. Shen, W\. Luo, X\. Du, L\. Guo, and Y\. Li \(2026\)Hindsight credit assignment for long\-horizon llm agents\.arXiv preprint arXiv:2603\.08754\.Cited by:[§2\.1](https://arxiv.org/html/2608.02694#S2.SS1.p2.1)\.
- G\. Team \(2026a\)Gemma 4 technical report\.External Links:2607\.02770,[Link](https://arxiv.org/abs/2607.02770)Cited by:[§4\.3](https://arxiv.org/html/2608.02694#S4.SS3.SSS0.Px1.p2.1)\.
- K\. Team \(2026b\)Kimi k3: open frontier intelligence\.External Links:2607\.24653,[Link](https://arxiv.org/abs/2607.24653)Cited by:[§4\.3](https://arxiv.org/html/2608.02694#S4.SS3.SSS0.Px1.p2.1)\.
- Q\. Team \(2025\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§4\.3](https://arxiv.org/html/2608.02694#S4.SS3.SSS0.Px1.p2.1)\.
- M\. Verma and K\. Metcalf \(2024\)Hindsight priors for reward learning from human preferences\.arXiv preprint arXiv:2404\.08828\.Cited by:[§2\.2](https://arxiv.org/html/2608.02694#S2.SS2.p1.1)\.
- H\. Wang, C\. T\. Leong, J\. Wang, J\. Wang, and W\. Li \(2025\)SPA\-rl: reinforcing llm agents via stepwise progress attribution\.arXiv preprint arXiv:2505\.20732\.Cited by:[§1](https://arxiv.org/html/2608.02694#S1.p4.1),[§2\.1](https://arxiv.org/html/2608.02694#S2.SS1.p2.1)\.
- Z\. Xi, C\. Liao, G\. Li, Y\. Yang, W\. Chen, Z\. Zhang, B\. Wang, S\. Jin, Y\. Zhou, J\. Guan,et al\.\(2025\)AgentPRM: process reward models for llm agents via step\-wise promise and progress\.arXiv preprint arXiv:2511\.08325\.Cited by:[§1](https://arxiv.org/html/2608.02694#S1.p4.1),[§2\.1](https://arxiv.org/html/2608.02694#S2.SS1.p2.1)\.
- B\. Xie, B\. Xu, Y\. Yuan, S\. Zhu, and H\. Shen \(2025\)From outcomes to processes: guiding prm learning from orm for inference\-time alignment\.arXiv preprint arXiv:2506\.12446\.Cited by:[§2\.1](https://arxiv.org/html/2608.02694#S2.SS1.p2.1)\.
- L\. Yan, Y\. Zhang, X\. Xu, J\. Lin, B\. Pan, X\. Zheng, J\. Qian, A\. Wu, J\. Geng, R\. Li, F\. Cai, J\. Niu, R\. Li, W\. Li, and C\. Lyu \(2026\)Crayotter: traceable multi\-agent workflows for long\-form video editing\.arXiv \(Cornell University\)\.Cited by:[§1](https://arxiv.org/html/2608.02694#S1.p6.1),[§3](https://arxiv.org/html/2608.02694#S3.p1.1)\.
- L\. Yang, Z\. Chen, X\. Li, P\. Jia, L\. Long, and J\. Yang \(2024\)Agent\-based video trimming\.External Links:2412\.09513,[Link](https://arxiv.org/abs/2412.09513)Cited by:[§1](https://arxiv.org/html/2608.02694#S1.p1.1),[§2\.2](https://arxiv.org/html/2608.02694#S2.SS2.p2.1)\.
- L\. Yuan, W\. Li, H\. Chen, G\. Cui, N\. Ding, K\. Zhang, B\. Zhou, Z\. Liu, and H\. Peng \(2024\)Free process rewards without process labels\.arXiv preprint arXiv:2412\.01981\.Cited by:[§2\.1](https://arxiv.org/html/2608.02694#S2.SS1.p2.1)\.
- P\. Zhang, C\. Zhou, Z\. Zhang, H\. Liu, C\. Zhang, J\. Liu, X\. Zhou, X\. Chen, S\. Weng, S\. Li,et al\.\(2026\)A benchmark and multi\-agent system for instruction\-driven cinematic video compilation\.arXiv preprint arXiv:2604\.10456\.Cited by:[§2\.2](https://arxiv.org/html/2608.02694#S2.SS2.p2.1)\.
- S\. Zhao, Y\. Hu, Y\. Shan, Y\. Wei, and X\. Cun \(2026\)CutClaw: agentic hours\-long video editing via music synchronization\.External Links:2603\.29664,[Link](https://arxiv.org/abs/2603.29664)Cited by:[§1](https://arxiv.org/html/2608.02694#S1.p1.1),[§2\.2](https://arxiv.org/html/2608.02694#S2.SS2.p2.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing,et al\.\(2023\)Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.InAdvances in Neural Information Processing Systems,Cited by:[§2\.2](https://arxiv.org/html/2608.02694#S2.SS2.p1.1)\.