Video-RSI: Recursive Self-Improvement of Video Understanding Agents via Harness Evolution

Hugging Face Daily Papers Papers

Summary

Video-RSI 提出一种通过 harness 进化实现视频理解智能体递归自我改进的框架,无需更新模型权重,通过重新审视训练视频、收集证据并诊断失败来迭代改进提示、工具和记忆,从而在更少帧处理量下提升准确率。

Video understanding agents acquire evidence through an executable harness that controls what they observe and how they use those observations. However, execution traces contain only the evidence acquired by the current harness, leaving competing explanations for failure unresolved and limiting the basis for self-improvement. We introduce Video-RSI, a framework for recursive self-improvement in which a video understanding agent uses its own language model to revise its harness. Through active video investigation, the model revisits the original training videos to test competing failure explanations with additional observations, grounding proposed changes in evidence beyond the existing trace. Cost-aware harness evolution turns these diagnoses into reusable revisions and determines which revisions to retain by considering both answer accuracy and visual cost. Across our evaluation settings on video understanding benchmarks, the evolved agent improves accuracy while processing fewer frames and achieves competitive accuracy-efficiency trade-offs against existing video understanding agents. These results demonstrate the potential for video understanding agents to improve their own evidence acquisition and use through harness evolution. Code is available at https://github.com/bingjunluo/Video-RSI .
Original Article
View Cached Full Text

Cached at: 10/12/26, 06:55 AM

Paper page - Video-RSI: Recursive Self-Improvement of Video Understanding Agents via Harness Evolution

Source: https://huggingface.co/papers/2609.37950 We introduceVideo-RSI, a framework for recursive self-improvement of video understanding agents through harness evolution. Instead of updating model weights, Video-RSI enables the agent to revisit the original training videos used for evolution, gather additional visual, transcript, and OCR evidence, and test competing explanations for its failures. Evidence-supported diagnoses are then translated into reusable changes to the agent’s prompts, tools, memory, and execution logic.

Video-RSI uses a cost-aware selection process to retain revisions based on both answer accuracy and visual processing cost. Across the evaluated video understanding benchmarks, the evolved harness improves accuracy while processing fewer frames, yielding competitive accuracy–efficiency trade-offs.

We release the inference and evaluation code, the initial and final harnesses, and the implementation of the harness-evolution workflow. We welcome feedback and discussion from the community.

Similar Articles

Self-Evolving Agent Harness

Hugging Face Daily Papers

提出一个自我进化代理框架(Harness RSI),在保持模型权重固定的情况下,通过反馈驱动的修订与进化优化来持续改进代理的软件脚手架(提示、记忆、工作流和可执行组件),并借助持续 PAC 学习给出泛化与保留的理论保证。

RRSI: Regularized Recursive Self-Improvement of Agent Harnesses

Hugging Face Daily Papers

This paper introduces Regularized Recursive Self-Improvement (RRSI) for AI agent harnesses, which applies regularization to prevent overfitting during recursive evolution, demonstrating performance gains on multiple benchmarks.

Reels-RSI: Program-Level Self-Improvement for Code-Rendered Explainer Videos

arXiv cs.AI

本文提出 Reels-RSI,一个开源的可编程短视频(explainer reel)导演系统,将讲解视频视为程序而非像素,通过代码层面的日志、可执行规则与回顾反思实现 agent 的程序级自我改进,并在 111 个 reel 的实验中验证了规则审计效果及偏好信号的失败模式。

EasyVideoR1: Easier RL for Video Understanding

Hugging Face Daily Papers

EasyVideoR1 is an efficient reinforcement learning framework for training large vision-language models on video understanding tasks, featuring offline preprocessing with tensor caching for 1.47x throughput improvement, a task-aware reward system covering 11 problem types, and evaluation across 22 video benchmarks. It also supports joint image-video training and a mixed offline-online data training paradigm.