Rank-Then-Act: Reward-Free Control from Frame-Order Progress
Summary
Rank-Then-Act (RTA) is a framework for learning control policies from expert video demonstrations without environment rewards, using a Vision-Language Model as a progress-based ordinal scorer with correlation-based rewards. It achieves stable cross-task transfer and outperforms prior methods on discrete and continuous control benchmarks.
View Cached Full Text
Cached at: 07/09/26, 07:53 AM
Paper page - Rank-Then-Act: Reward-Free Control from Frame-Order Progress
Source: https://huggingface.co/papers/2607.01897
Abstract
Rank-Then-Act framework learns control policies from video demonstrations using a vision-language model as an ordinal scorer with correlation-based rewards, enabling stable cross-task transfer without environment rewards.
We introduce Rank-Then-Act (RTA), a framework for learning control policies from expert video demonstrations without environment rewards. RTA trains aVision-Language Model(VLM) offline as a progress-basedordinal scorer, using aGroup Relative Policy Optimization(GRPO) objective over shuffled frame sequences, which forces the model to recover temporal ordering from visual semantics rather than trivial time cues. Importantly, instead of using the scorer directly as a scalar reward model, we propose a correlation-based reward function forreinforcement learning: at each interaction window, we compute theSpearman rank correlationbetween predicted progress rankings and true temporal indices, yielding a bounded, scale-invariant learning signal. This design decouples reward learning from absolute calibration and enables stable transfer across tasks and environments. We evaluate RTA on discrete control benchmarks (PyBoy: Catrap, Kirby) and continuous control tasks (PointMaze, MetaWorld). RTA consistently matches or outperforms priorvideo-based reward learningmethods and rank-based baselines, while demonstrating strongcross-task reuseof a single pretrained progress scorer. Our results suggest that correlation-structured supervision over video-derived ordinal signals is sufficient forpolicy learning, offering a scalable alternative to explicit reward design.
View arXiv pageView PDFProject pageGitHub6Add to collection
Get this paper in your agent:
hf papers read 2607\.01897
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2607.01897 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2607.01897 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2607.01897 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies
Researchers propose APT, a two-stage training method that pretrains action experts on vision-action pairs before integrating language conditioning, significantly improving out-of-distribution instruction generalization for Vision-Language-Action policies.
Hierarchical Advantage Weighting for Online RL Fine-Tuning of VLAs from Sparse Episode Outcomes
This paper proposes Hierarchical Advantage-Weighted Behavior Cloning (HABC) for fine-tuning Vision-Language-Action (VLA) policies using online reinforcement learning with sparse binary episode outcomes. HABC separates viability and efficiency objectives via adaptive critic heads and intervention-aware credit assignment, significantly improving success rates on contact-rich bimanual manipulation tasks.
Sample-efficient Transfer Reinforcement Learning via Adaptive Reward Shaping and Policy-Ratio Reweighting Strategy
This paper proposes a safe transfer reinforcement learning framework for autonomous highway lane-changing, using adaptive teacher intervention and reward shaping to improve sample efficiency and safety. Experiments show over 52% improvement in safety and 5% improvement in efficiency over baselines.
TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning
TRACE is a unified rollout budget allocation framework that enhances reward contrast in multi-turn agentic reinforcement learning by dynamically distributing resources across tree-structured rollouts based on prefix-level informativeness. It improves efficiency and accuracy on agentic benchmarks like Multi-Hop QA.
ProcessThinker: Enhancing Multi-modal Large Language Models Reasoning via Rollout-based Process Reward
ProcessThinker introduces a practical post-training pipeline that provides step-level process rewards without training an explicit process reward model. It uses rollout-based rewards to give dense credit assignment for multi-step reasoning in multimodal LLMs, consistently improving performance on video benchmarks.