Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos

Hugging Face Daily Papers Papers
egocentric-video proactive-assistance memory-augmented-agent benchmark computer-vision ai-assistant

Summary

Vinci2 presents a proactive assistance system for continuous egocentric videos, introducing a new benchmark EgoServe and a memory-augmented agent EgoMemo that determines when to intervene based on temporal context.

When should an intelligent assistant speak up without being asked? Continuous egocentric video offers rich, evolving context that enables a new form of assistance: one that is proactive rather than merely reactive. Yet existing approaches either wait passively for user queries or treat every detected event as requiring a response, without considering the user's history, current activity, or whether assistance would actually be welcome. We reframe proactive assistance as a context-dependent decision problem: the agent must not only perceive what is happening, but reason over accumulated temporal context to determine when and whether to intervene. To this end, we present Vinci2, a proactive egocentric assistance system that advances the on-device assistant Vinci from reactive response toward proactivity. On the evaluation side, we present EgoServe, the first large-scale benchmark for proactive assistance in continuous egocentric video. EgoServe comprises over 3,000 service instances organized along 4 temporal memory horizons, ranging from immediate safety alerts to long-term habit coaching, across 10 service categories. On the modeling side, we propose EgoMemo, a training-free, memory-augmented agent that maintains three complementary memory representations: multi-scale temporal summaries, a semantic knowledge graph, and visual embedding archives. At each timestep, EgoMemo performs retrieval-augmented reasoning to determine whether assistance is warranted and, if so, produces contextually grounded responses. Experiments demonstrate that EgoMemo establishes strong baselines on EgoServe while remaining competitive on existing egocentric benchmarks. Our benchmark and code are publicly available at https://sitonggong.github.io/EgoServe-page/{Vinci2}.
Original Article
View Cached Full Text

Cached at: 07/16/26, 05:41 AM

Paper page - Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos

Source: https://huggingface.co/papers/2607.11523

Abstract

Whenshouldanintelligentassistantspeakupwithoutbeingasked?Continuousegocentricvideooffersrich,evolvingcontextthatenablesanewformofassistance:onethatisproactiveratherthanmerelyreactive.Yetexistingapproacheseitherwaitpassivelyforuserqueriesortreateverydetectedeventasrequiringaresponse,withoutconsideringtheuser’shistory,currentactivity,orwhetherassistancewouldactuallybewelcome.Wereframeproactiveassistanceasacontext-dependentdecisionproblem:theagentmustnotonlyperceivewhatishappening,butreasonoveraccumulatedtemporalcontexttodeterminewhenandwhethertointervene.Tothisend,wepresentVinci2,aproactiveegocentricassistancesystemthatadvancestheon-deviceassistantVincifromreactiveresponsetowardproactivity.Ontheevaluationside,wepresentEgoServe,thefirstlarge-scalebenchmarkforproactiveassistanceincontinuousegocentricvideo.EgoServecomprisesover3,000serviceinstancesorganizedalong4temporalmemoryhorizons,rangingfromimmediatesafetyalertstolong-termhabitcoaching,across10servicecategories.Onthemodelingside,weproposeEgoMemo,atraining-free,memory-augmentedagentthatmaintainsthreecomplementarymemoryrepresentations:multi-scaletemporalsummaries,asemanticknowledgegraph,andvisualembeddingarchives.Ateachtimestep,EgoMemoperformsretrieval-augmentedreasoningtodeterminewhetherassistanceiswarrantedand,ifso,producescontextuallygroundedresponses.ExperimentsdemonstratethatEgoMemoestablishesstrongbaselinesonEgoServewhileremainingcompetitiveonexistingegocentricbenchmarks.Ourbenchmarkandcodearepubliclyavailableathttps://sitonggong.github.io/EgoServe-page/{Vinci2}.

View arXiv pageView PDFProject pageGitHub2Add to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.11523 in a model README.md to link it from this page.

Datasets citing this paper1

#### SitongGong/EgoServe Preview• Updated1 day ago • 154

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.11523 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

MemLife: Curating and Reasoning over Long-Term Egocentric Video Memories

Hugging Face Daily Papers

MemLife introduces a multimodal memory system for long-term egocentric videos that builds entity-grounded text episodes and retrieves them with a time-indexed agentic reader, improving over training-free baselines by 4.6-12.0% on long-horizon benchmarks. A reinforcement learning framework called MemOpt further optimizes the memory writer for faithfulness and retrievability, yielding consistent 2.7-5.0% gains.

ProPerAgent: Toward Memory-Driven Proactive Assistance

arXiv cs.CL

ProPerAgent: introduces ProPerMem, a benchmark for memory-driven proactive assistance pairing egocentric video with year-long user memories, and ProPerAgent, a graph-organized memory harness with RL post-training that lifts strict accuracy from under 10% to 42.4%, showing content and timing are separately necessary for proactive AI assistants.