Ego2Act: Evaluating Goal-Directed Manipulation in Egocentric Video Generation

Hugging Face Daily Papers Papers
video-generation benchmark egocentric-vision embodied-ai world-models evaluation

Summary

The paper introduces Ego2Act, a goal-directed benchmark of 2,640 egocentric videos across 110 real-world tasks for evaluating whether video generation models can simulate multi-step physical manipulation, along with Ego2Act Judge, a reference-free evaluation pipeline that aligns better with human judgments of task completion and physics plausibility.

Video generation models are increasingly being explored as world simulators for embodied planning and learning. To do so effectively, these models must not only generate visually appealing frames, but also predict how environments dynamically evolve when executing goal-directed actions. While evaluating these capabilities is crucial, existing benchmarks focus mainly on single short actions or step-by-step instructions. This leaves multi-step physical reasoning underexplored, especially in egocentric video generation that requires planning to simulate proper execution to accomplish high-level goals by carrying out multiple real-world manipulations. We introduce Ego2Act, a goal-directed benchmark featuring 2,640 videos from 110 real-world tasks across day-to-day settings, varying object clutter and multi-step complexity. Given an initial scene image and a high-level goal, Ego2Act evaluates whether video generation models can produce realistic egocentric videos of a hand manipulating objects to carry out the task. To support scalable evaluation, we also introduce Ego2ActJudge, a reference-free evaluation pipeline that achieves better task completion and physics plausibility evaluation alignment with human consensus compared to relevant baselines. Our findings reveal that models' generated simulations often skip or partially execute steps, leaving later steps missing dependent states, which leads to unfulfilled goal. Furthermore, models consistently fail at fine-grained physical dynamics, particularly during complex object manipulation and persistent world modeling. We hope Ego2Act provides a rigorous testbed for advancing video models toward physically plausible, goal-directed simulation.
Original Article
View Cached Full Text

Cached at: 10/03/26, 12:32 AM

Paper page - Ego2Act: Evaluating Goal-Directed Manipulation in Egocentric Video Generation

Source: https://huggingface.co/papers/2610.01092 Authors:

,

,

,

,

,

,

,

,

,

,

,

Abstract

Videogenerationmodelsareincreasinglybeingexploredasworldsimulatorsforembodiedplanningandlearning.Todosoeffectively,thesemodelsmustnotonlygeneratevisuallyappealingframes,butalsopredicthowenvironmentsdynamicallyevolvewhenexecutinggoal-directedactions.Whileevaluatingthesecapabilitiesiscrucial,existingbenchmarksfocusmainlyonsingleshortactionsorstep-by-stepinstructions.Thisleavesmulti-stepphysicalreasoningunderexplored,especiallyinegocentricvideogenerationthatrequiresplanningtosimulateproperexecutiontoaccomplishhigh-levelgoalsbycarryingoutmultiplereal-worldmanipulations.WeintroduceEgo2Act,agoal-directedbenchmarkfeaturing2,640videosfrom110real-worldtasksacrossday-to-daysettings,varyingobjectclutterandmulti-stepcomplexity.Givenaninitialsceneimageandahigh-levelgoal,Ego2Actevaluateswhethervideogenerationmodelscanproducerealisticegocentricvideosofahandmanipulatingobjectstocarryoutthetask.Tosupportscalableevaluation,wealsointroduceEgo2ActJudge,areference-freeevaluationpipelinethatachievesbettertaskcompletionandphysicsplausibilityevaluationalignmentwithhumanconsensuscomparedtorelevantbaselines.Ourfindingsrevealthatmodels’generatedsimulationsoftenskiporpartiallyexecutesteps,leavinglaterstepsmissingdependentstates,whichleadstounfulfilledgoal.Furthermore,modelsconsistentlyfailatfine-grainedphysicaldynamics,particularlyduringcomplexobjectmanipulationandpersistentworldmodeling.WehopeEgo2Actprovidesarigoroustestbedforadvancingvideomodelstowardphysicallyplausible,goal-directedsimulation.

View arXiv pageView PDFProject pageGitHub0Add to collection

Get this paper in your agent:

hf papers read 2610\.01092

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.01092 in a model README.md to link it from this page.

Datasets citing this paper2

#### ego2act/ego2act-bench Viewer• Updatedabout 11 hours ago • 110 • 231 #### ego2act/ego2act-vidgen Viewer• Updatedabout 11 hours ago • 18.3k • 77

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.01092 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Ego2Robot: Scalable Robot Data Synthesis from Egocentric Human Data

Hugging Face Daily Papers

Ego2Robot is a scalable pipeline that converts egocentric human manipulation videos into robot training data via action retargeting and visual synthesis, producing 18,561 hours of data across 15 robot morphologies. Experiments show that joint pretraining on this synthesized data improves out-of-distribution generalization for vision-language-action models, including on real-robot deployment.

EgoPhys: Learning Generalizable Physics Models of Deformable Objects from Egocentric Video

Hugging Face Daily Papers

EgoPhys introduces a framework to construct deformable physical digital twins from egocentric RGB video using generalizable priors and a compact codebook, enabling zero-shot generalization to unseen objects without per-spring optimization. The system is demonstrated on a real robot, showing that egocentric human play video can serve as internal world representation for deformable-object planning.

EgoTools: Towards Tool-Centric Reasoning in Real-World Egocentric Videos

Hugging Face Daily Papers

EgoTools introduces a 100-hour egocentric tool-use corpus and a 1,000-question diagnostic benchmark, showing current multimodal models (e.g., Gemini-3.1-Pro at 51.7% on Perception & Grounding) still struggle with tool-centric embodied reasoning, while fine-tuning Qwen3-VL-8B on the data lifts accuracy from 50.0% to 60.9%.