EgoCaptioner: Towards Comprehensive and Fine-Grained Egocentric Video Captioning
Summary
The paper presents EgoCaptioner, a coarse-to-fine egocentric video captioning framework that combines video-level overviews with timestamped, perception-adaptive step narration, trained with GRPO using factual and temporal rewards, achieving leading results on five egocentric benchmarks and surpassing proprietary baselines like Gemini 3.1 Pro on PhysCapBench.
View Cached Full Text
Cached at: 11/20/26, 07:37 AM
Paper page - EgoCaptioner: Towards Comprehensive and Fine-Grained Egocentric Video Captioning
Source: https://huggingface.co/papers/2611.26380 Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
Egocentricvideocaptioningrequiresfine-graineddetailandaccuratetiming,yetcurrentmultimodallargelanguagemodels(MLLMs)oftenmissbriefactions,confusetheirorder,ormisinterpretthewearer’smotion.Existingcaptionformatsalsoofferlimitedsupportforcombiningdetailednarrationwithprecisetiming.WepresentEgoCaptioner,whichpairsavideo-leveloverviewofthesetting,participants,andthewearer’soverallactivitywithtimedstepswhosenarrationadaptstovaryingperceptualdemandsandmarkskeymomentsatspecifictimestamps.Wedevelopacoarse-to-fineannotationpipelinethatestablisheseventstructurebeforeexpandingstepdescriptions,yielding50,000trainingexamplesfromeightegocentricvideosources.Complementaryrewardsassessfactualaccuracyandcoverage,measurestep-leveltemporalalignment,andpenalizefragmentedorredundantnarration,guidingcaptionrefinementthroughGroupRelativePolicyOptimization(GRPO).Toimprovethereliabilityoffactualsupervision,wesupplementcaptionreferenceswithpose-derivedmotionfactswhensensordataareavailable.Acrossfiveegocentricbenchmarks,EgoCaptionerachievesleadingperformanceamongevaluatedopen-sourcemodelsincaptionquality,caption-conditionedquestionanswering,andtemporalalignment.ItalsosurpassesGemini3.1ProonPhysCapBenchandallevaluatedproprietarybaselinesonMinerva-Egounderthecaption-to-QAprotocol.Ablationssupportthecomplementaryrolesoffactualandtemporalrewardsandshowthatcoarse-to-fineannotationprovidesmoreeffectivesupervisionthansingle-passannotation.
View arXiv pageView PDFProject pageGitHub1Add to collection
Get this paper in your agent:
hf papers read 2611\.26380
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2611.26380 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2611.26380 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2611.26380 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
CapMem: A Benchmark for Caption-Based Episodic Memory in Egocentric Video
CapMem is a human-annotated benchmark for episodic memory in egocentric video using captions, showing that caption-based QA outperforms direct video QA on long videos.
RefCaptioner: Multi-Reference Image-Grounded Video Captioning
Introduces multi-reference image-grounded video captioning and proposes RefCaptioner, a two-stage post-training framework with mixed-data SFT and hierarchical coverage-discounted GRPO. The paper also presents MRVBench, a benchmark for evaluating caption factuality and multi-reference grounding.
EgoTools: Towards Tool-Centric Reasoning in Real-World Egocentric Videos
EgoTools introduces a 100-hour egocentric tool-use corpus and a 1,000-question diagnostic benchmark, showing current multimodal models (e.g., Gemini-3.1-Pro at 51.7% on Perception & Grounding) still struggle with tool-centric embodied reasoning, while fine-tuning Qwen3-VL-8B on the data lifts accuracy from 50.0% to 60.9%.
Ego2Act: Evaluating Goal-Directed Manipulation in Egocentric Video Generation
The paper introduces Ego2Act, a goal-directed benchmark of 2,640 egocentric videos across 110 real-world tasks for evaluating whether video generation models can simulate multi-step physical manipulation, along with Ego2Act Judge, a reference-free evaluation pipeline that aligns better with human judgments of task completion and physics plausibility.
Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos
Vinci2 presents a proactive assistance system for continuous egocentric videos, introducing a new benchmark EgoServe and a memory-augmented agent EgoMemo that determines when to intervene based on temporal context.