Rethinking Long-Video Efficiency: A Joint Allocation Perspective on Frames, Pixels, and Front-End Latency

Hugging Face Daily Papers Papers
long-video-understanding vision-language-models efficiency frame-sampling token-budget decoding-latency

Summary

The paper proposes LoHi, a training-free framework that jointly allocates frames, resolution, and front-end decoding latency for long-video understanding with VLMs, combining dense low-resolution video streams with sparse high-resolution frames, improving accuracy by up to 10.6 points over baselines and reducing decoding latency up to 7x on hour-long videos.

Efficient long-video understanding with vision-language models (VLMs) is often framed as selecting informative frames or visual tokens at a fixed native resolution. We show that per-frame resolution can instead be traded for denser temporal coverage, while front-end decoding latency depends on the size of the candidate pool rather than the final token budget. An empirical study across multiple VLMs and long-video benchmarks yields three findings: dense low-resolution sampling outperforms sparse native-resolution sampling at matched token budgets; resolution-sensitive tasks benefit from selected high-resolution frames; and front-end decoding dominates wall time for hour-long videos. Motivated by these findings, we introduce LoHi, a training-free, single-pass framework that combines a dense low-resolution video stream with sparse high-resolution image frames through the VLM's native video and image pathways. LoHi-Anchor selects high-resolution frames using codec I-frame metadata, while LoHi-SemDiv uses query relevance and visual diversity over CLIP features. Across three long-video benchmarks, LoHi improves average accuracy by 10.6 percentage points over the native-resolution baseline at a matched token budget and by 5.2 percentage points over the strongest prior efficiency method. It also reduces front-end decoding latency by up to 7x on hour-long videos. Project page: https://sixundong.com/projects/lohi
Original Article
View Cached Full Text

Cached at: 10/06/26, 04:53 AM

Paper page - Rethinking Long-Video Efficiency: A Joint Allocation Perspective on Frames, Pixels, and Front-End Latency

Source: https://huggingface.co/papers/2610.04318

Abstract

Efficientlong-videounderstandingwithvision-languagemodels(VLMs)isoftenframedasselectinginformativeframesorvisualtokensatafixednativeresolution.Weshowthatper-frameresolutioncaninsteadbetradedfordensertemporalcoverage,whilefront-enddecodinglatencydependsonthesizeofthecandidatepoolratherthanthefinaltokenbudget.AnempiricalstudyacrossmultipleVLMsandlong-videobenchmarksyieldsthreefindings:denselow-resolutionsamplingoutperformssparsenative-resolutionsamplingatmatchedtokenbudgets;resolution-sensitivetasksbenefitfromselectedhigh-resolutionframes;andfront-enddecodingdominateswalltimeforhour-longvideos.Motivatedbythesefindings,weintroduceLoHi,atraining-free,single-passframeworkthatcombinesadenselow-resolutionvideostreamwithsparsehigh-resolutionimageframesthroughtheVLM’snativevideoandimagepathways.LoHi-Anchorselectshigh-resolutionframesusingcodecI-framemetadata,whileLoHi-SemDivusesqueryrelevanceandvisualdiversityoverCLIPfeatures.Acrossthreelong-videobenchmarks,LoHiimprovesaverageaccuracyby10.6percentagepointsoverthenative-resolutionbaselineatamatchedtokenbudgetandby5.2percentagepointsoverthestrongestpriorefficiencymethod.Italsoreducesfront-enddecodinglatencybyupto7xonhour-longvideos.Projectpage:https://sixundong.com/projects/lohi

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2610\.04318

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.04318 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.04318 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.04318 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

LiteFrame Scales Video LLM Efficiency (6 minute read)

TLDR AI

LiteFrame introduces a highly efficient video encoder for Video LLMs that uses Compressed Token Distillation to enable up to 8x more frames and 35% latency reduction while maintaining accuracy, setting a new Pareto frontier for long-form video understanding.

Video2LoRA: Parametric Video Internalization for Vision-Language Models

Hugging Face Daily Papers

This paper introduces Video2LoRA, a method that predicts Low-Rank Adaptation (LoRA) weights directly from video representations, enabling efficient video processing in frozen vision-language models. It reduces visual token load by up to 1500x and query TTFT by 6-80x while maintaining performance on video summarization and captioning benchmarks.