Rank-Then-Act: Reward-Free Control from Frame-Order Progress
Summary
Rank-Then-Act (RTA) is a framework for learning control policies from expert video demonstrations without environment rewards, using a Vision-Language Model as a progress-based ordinal scorer with correlation-based rewards. It achieves stable cross-task transfer and outperforms prior methods on discrete and continuous control benchmarks.
View Cached Full Text
Cached at: 07/09/26, 07:53 AM
Paper page - Rank-Then-Act: Reward-Free Control from Frame-Order Progress
Source: https://huggingface.co/papers/2607.01897
Abstract
Rank-Then-Act framework learns control policies from video demonstrations using a vision-language model as an ordinal scorer with correlation-based rewards, enabling stable cross-task transfer without environment rewards.
We introduce Rank-Then-Act (RTA), a framework for learning control policies from expert video demonstrations without environment rewards. RTA trains aVision-Language Model(VLM) offline as a progress-basedordinal scorer, using aGroup Relative Policy Optimization(GRPO) objective over shuffled frame sequences, which forces the model to recover temporal ordering from visual semantics rather than trivial time cues. Importantly, instead of using the scorer directly as a scalar reward model, we propose a correlation-based reward function forreinforcement learning: at each interaction window, we compute theSpearman rank correlationbetween predicted progress rankings and true temporal indices, yielding a bounded, scale-invariant learning signal. This design decouples reward learning from absolute calibration and enables stable transfer across tasks and environments. We evaluate RTA on discrete control benchmarks (PyBoy: Catrap, Kirby) and continuous control tasks (PointMaze, MetaWorld). RTA consistently matches or outperforms priorvideo-based reward learningmethods and rank-based baselines, while demonstrating strongcross-task reuseof a single pretrained progress scorer. Our results suggest that correlation-structured supervision over video-derived ordinal signals is sufficient forpolicy learning, offering a scalable alternative to explicit reward design.
View arXiv pageView PDFProject pageGitHub6Add to collection
Get this paper in your agent:
hf papers read 2607\.01897
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2607.01897 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2607.01897 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2607.01897 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
@SharonYixuanLi: Scaling outcome-based RL won't solve long-horizon agentic tasks. Credit assignment is the bottleneck, and turn-level re…
TRACE introduces a turn-level reward assignment method using frozen reference model log-probabilities and temporal-difference learning to address credit assignment in long-horizon agentic tasks, achieving significant improvements in search benchmarks without critic or process labels.
RTPO: Reverse-Turn Policy Optimization for Stabilizing Agentic RL Training
The paper proposes Reverse-Turn Policy Optimization (RTPO) to stabilize multi-turn agentic reinforcement learning training by organizing rollouts as reverse trees and performing turn-level updates in reverse order, showing improvements over baselines.
RewardVerse: Rubric-Guided Policy Optimization for Video Reward Modeling
RewardVerse presents a rubric-based framework for video reward modeling to mitigate scalar drift and provide stable evaluation criteria, enhancing reinforcement learning in video generation.
RL^2-VLA: Adaptive RL Latent Compositional Steering with Test-Time Scaling for Vision-Language-Action Models
This paper introduces RL^2, an adaptive inference-time steering framework for Vision-Language-Action models that uses offline RL on latent representations to compose action flows, activating steering only when failure is predicted. It achieves up to +17.3% success rate improvements on SIMPLER and PolaRiS benchmarks and demonstrates real-world transfer.
StructRL: Online Structured Reinforcement Learning for Long-Horizon Vision-Language-Action Tasks
StructRL introduces an online reinforcement learning framework that uses structured intermediate rewards from verifiable subtasks to enhance long-horizon vision-language-action tasks, demonstrating superior performance on benchmarks like RoboCasa365 and LIBERO-Long.