EgoCaptioner: Towards Comprehensive and Fine-Grained Egocentric Video Captioning

Hugging Face Daily Papers Papers

Summary

The paper presents EgoCaptioner, a coarse-to-fine egocentric video captioning framework that combines video-level overviews with timestamped, perception-adaptive step narration, trained with GRPO using factual and temporal rewards, achieving leading results on five egocentric benchmarks and surpassing proprietary baselines like Gemini 3.1 Pro on PhysCapBench.

Egocentric video captioning requires fine-grained detail and accurate timing, yet current multimodal large language models (MLLMs) often miss brief actions, confuse their order, or misinterpret the wearer's motion. Existing caption formats also offer limited support for combining detailed narration with precise timing. We present EgoCaptioner, which pairs a video-level overview of the setting, participants, and the wearer's overall activity with timed steps whose narration adapts to varying perceptual demands and marks key moments at specific timestamps. We develop a coarse-to-fine annotation pipeline that establishes event structure before expanding step descriptions, yielding 50,000 training examples from eight egocentric video sources. Complementary rewards assess factual accuracy and coverage, measure step-level temporal alignment, and penalize fragmented or redundant narration, guiding caption refinement through Group Relative Policy Optimization (GRPO). To improve the reliability of factual supervision, we supplement caption references with pose-derived motion facts when sensor data are available. Across five egocentric benchmarks, EgoCaptioner achieves leading performance among evaluated open-source models in caption quality, caption-conditioned question answering, and temporal alignment. It also surpasses Gemini 3.1 Pro on PhysCapBench and all evaluated proprietary baselines on Minerva-Ego under the caption-to-QA protocol. Ablations support the complementary roles of factual and temporal rewards and show that coarse-to-fine annotation provides more effective supervision than single-pass annotation.
Original Article
View Cached Full Text

Cached at: 11/20/26, 07:37 AM

Paper page - EgoCaptioner: Towards Comprehensive and Fine-Grained Egocentric Video Captioning

Source: https://huggingface.co/papers/2611.26380 Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

Egocentricvideocaptioningrequiresfine-graineddetailandaccuratetiming,yetcurrentmultimodallargelanguagemodels(MLLMs)oftenmissbriefactions,confusetheirorder,ormisinterpretthewearer’smotion.Existingcaptionformatsalsoofferlimitedsupportforcombiningdetailednarrationwithprecisetiming.WepresentEgoCaptioner,whichpairsavideo-leveloverviewofthesetting,participants,andthewearer’soverallactivitywithtimedstepswhosenarrationadaptstovaryingperceptualdemandsandmarkskeymomentsatspecifictimestamps.Wedevelopacoarse-to-fineannotationpipelinethatestablisheseventstructurebeforeexpandingstepdescriptions,yielding50,000trainingexamplesfromeightegocentricvideosources.Complementaryrewardsassessfactualaccuracyandcoverage,measurestep-leveltemporalalignment,andpenalizefragmentedorredundantnarration,guidingcaptionrefinementthroughGroupRelativePolicyOptimization(GRPO).Toimprovethereliabilityoffactualsupervision,wesupplementcaptionreferenceswithpose-derivedmotionfactswhensensordataareavailable.Acrossfiveegocentricbenchmarks,EgoCaptionerachievesleadingperformanceamongevaluatedopen-sourcemodelsincaptionquality,caption-conditionedquestionanswering,andtemporalalignment.ItalsosurpassesGemini3.1ProonPhysCapBenchandallevaluatedproprietarybaselinesonMinerva-Egounderthecaption-to-QAprotocol.Ablationssupportthecomplementaryrolesoffactualandtemporalrewardsandshowthatcoarse-to-fineannotationprovidesmoreeffectivesupervisionthansingle-passannotation.

View arXiv pageView PDFProject pageGitHub1Add to collection

Get this paper in your agent:

hf papers read 2611\.26380

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2611.26380 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2611.26380 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2611.26380 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

RefCaptioner: Multi-Reference Image-Grounded Video Captioning

Hugging Face Daily Papers

Introduces multi-reference image-grounded video captioning and proposes RefCaptioner, a two-stage post-training framework with mixed-data SFT and hierarchical coverage-discounted GRPO. The paper also presents MRVBench, a benchmark for evaluating caption factuality and multi-reference grounding.

EgoTools: Towards Tool-Centric Reasoning in Real-World Egocentric Videos

Hugging Face Daily Papers

EgoTools introduces a 100-hour egocentric tool-use corpus and a 1,000-question diagnostic benchmark, showing current multimodal models (e.g., Gemini-3.1-Pro at 51.7% on Perception & Grounding) still struggle with tool-centric embodied reasoning, while fine-tuning Qwen3-VL-8B on the data lifts accuracy from 50.0% to 60.9%.

Ego2Act: Evaluating Goal-Directed Manipulation in Egocentric Video Generation

Hugging Face Daily Papers

The paper introduces Ego2Act, a goal-directed benchmark of 2,640 egocentric videos across 110 real-world tasks for evaluating whether video generation models can simulate multi-step physical manipulation, along with Ego2Act Judge, a reference-free evaluation pipeline that aligns better with human judgments of task completion and physics plausibility.