VideoMM: Adaptive Macro-Micro Inference for Efficient Video MLLMs

# Adaptive Macro-Micro Inference for Efficient Video MLLMs
Source: [https://arxiv.org/html/2609.16722](https://arxiv.org/html/2609.16722)
## VideoMM: Adaptive Macro\-Micro Inference for Efficient Video MLLMsThanks:∗\\astEqual Contribution: \{haoyuguo, yfung\}@mail\.ustc\.edu\.cnThanks:†\\daggerCorresponding author: xkxie@ustc\.edu\.cn

Haoyu Guo1,3,∗\\ast, Yuan Feng2,3,∗\\ast, Junlin Lv2,3, Mingjun Xiao2,3, S Kevin Zhou1,3, Xike Xie1,3,†\\daggerAffiliation:1School of Biomedical Engineering, University of Science and Technology of China 2School of Computer Science, USTC 3Data Darkness Lab, MIRACLE Center, Suzhou Institute for Advanced Research,China

###### Abstract\.

Scaling Multimodal Large Language Models \(MLLMs\) to long\-form video understanding is bottlenecked by the explosion of visual tokens, which saturates context windows and incurs prohibitive costs\. Current solutions predominantly rely on auxiliary models for token reduction but face a fundamental dilemma: lightweight encoder\-driven approaches often overlook critical semantic information, whereas heavyweight MLLM\-driven reduction negates the efficiency gains\. In this work, we identify a more fundamental inefficiency underlying this dilemma: while fine\-grained visual details are essential for detailed understanding, they are largely redundant for the preliminary task of selecting semantically relevant regions\. Motivated by this, we introduceVideoMM, which marks a paradigm shift from model\-centric downsizing to adaptive perceptual granularity\. Specifically, our framework decouples selection from reasoning by executing semantic filtering on a cost\-effectiveMacro Proxy\(derived from downscaled frames\), and projecting the selected regions onto high\-fidelityMicro Tokensfor detailed understanding only when necessary\. Extensive evaluations show that VideoMM significantly outperforms existing solutions\. It achieves a 6\.13×\\timesspeedup and a 7\.4% accuracy gain over full\-context baselines on LongVideoBench, and further accelerates inference by 2\.73×\\timesover current leading methods, establishing a highly scalable paradigm for long\-video understanding\. Our code is available at: https://github\.com/adfh917k/VideoMM\.

## 1\.Introduction

Multimodal Large Language Models \(MLLMs\)\([Team, 2025](https://arxiv.org/html/2609.16722#bib.bib27);[Bai et al\., 2025](https://arxiv.org/html/2609.16722#bib.bib3);[Team et al\., 2025](https://arxiv.org/html/2609.16722#bib.bib28);[Zohar et al\., 2024](https://arxiv.org/html/2609.16722#bib.bib45);[Tang et al\., 2025](https://arxiv.org/html/2609.16722#bib.bib25)\)have recently achieved remarkable success in interpreting static images and short video clips\. However, extending these capabilities to long\-form video understanding remains a critical challenge\. While long\-video analysis enables high\-value applications—such as video QA, temporal event localization, and long\-horizon action recognition\([Liu et al\., 2025a](https://arxiv.org/html/2609.16722#bib.bib20);[AlShami et al\., 2024](https://arxiv.org/html/2609.16722#bib.bib2);[Liu et al\., 2021](https://arxiv.org/html/2609.16722#bib.bib22)\)—it introduces severe scalability bottlenecks\. The primary constraint is the massive influx of visual tokens: for instance, a single 10\-minute 720p video sampled at 1 fps generates over 300k tokens in the widely adopted Qwen\-3\-VL model\. This volume rapidly exhausts the limited context windows of current models, leading to prohibitive computational costs\.

\(a\)Qwen2\.5\-VL\-7B\(b\)GLM\-4\.1V\-9B\(c\)Qwen3\-VL\-8B
Figure 1\.VideoMM sets a new accuracy–efficiency frontier on LongVideoBench, achieving a6\.13×6\.13\\timesinference speedup with a7\.4%7\.4\\%accuracy gain over the vanilla baseline, and a2\.73×2\.73\\timesspeedup over FlexSelect without accuracy loss\.To mitigate this, existing research has focused on reducing the number of visual tokens via auxiliary models prior to full inference, yet these approaches face afundamental dilemma\. Lightweight encoder\-driven methods\([Bolya et al\., 2023](https://arxiv.org/html/2609.16722#bib.bib4);[Liu et al\., 2025b](https://arxiv.org/html/2609.16722#bib.bib23);[Yang et al\., 2024](https://arxiv.org/html/2609.16722#bib.bib39)\)are efficient but often discard semantically critical information due to limited semantic awareness, while MLLM\-driven selection\([zhang et al\., 2025](https://arxiv.org/html/2609.16722#bib.bib43)\)improves accuracy at the cost of substantial computational overhead\. As a result, existing model\-centric downsizing methods struggle to reconcile the trade\-off between accuracy and efficiency in token selection\. We argue that this bottleneck arises froma more fundamental redundancy: in long\-video understanding, different stages of inference require fundamentally different levels of perceptual fidelity\.

While high\-fidelity visual details are essential for precise semantic understanding, they are largely unnecessary for the preliminary task of identifying which regions are relevant\. Consequently, performing semantic filtering directly on original high\-resolution visual tokens is inherently wasteful\.

Drawing inspiration from the coarse\-to\-fine nature of human perception\-where one first grasps the global context \(macro\) before focusing on specific details \(micro\)—we proposeVideoMM, a macro\-micro paradigm that decouples semantic localization from fine\-grained reasoning\. VideoMM initially identifies relevant regions using a lightweightMacro Proxyderived from downscaled frames\. Subsequently, it employs a consensus mechanism to govern adaptive computation: high\-resolutionMicro Tokensare selectively activated only when the necessity for fine\-grained inference is confirmed\. By strictly adhering to this coarse\-to\-fine progression, VideoMM ensures that heavy computation is reserved solely for regions requiring granular scrutiny, thereby minimizing redundant computation without compromising semantic fidelity\. Extensive evaluations across three benchmarks validate the effectiveness of our approach\. Notably, as shown in Figure[1](https://arxiv.org/html/2609.16722#S1.F1), VideoMM achieves an average speedup of 6\.13× and an accuracy gain of 7\.0% over vanilla baselines and accelerates state\-of\-the\-art methods by over 2\.73× with comparable accuracy on the challenging LongVideoBench\.

To summarize, our contributions are as follows:

- •We identify afundamental dilemmain existing token reduction methods rooted in model downsizing, which struggle to balance accuracy with efficiency\. To resolve this, we propose a paradigm shift from model scale toperceptual granularity, exploiting the coarse\-to\-fine nature of visual information to reconcile the conflict between selection precision and overhead\.
- •We propose VideoMM, a macro\-micro paradigm mimicking human coarse\-to\-fine perception:Grouped Selection with Macro\-view Proxydecouples token selection from dense processing via lightweight proxies, whileAdaptive Macro\-Micro Inferencedynamically recruits specific details only when necessary for ambiguous cases\.
- •Extensive evaluations validate the effectiveness of VideoMM, achieving a 6\.1× speedup and a 7\.0% accuracy gain on LongVideoBench\. Through further analysis of macro\-micro variants, we demonstrate the intrinsic efficacy of this perceptual granularity\-oriented paradigm, establishing a foundational and promising direction that moves beyond the existing model\-centric perspective\.

## 2\.Related Works

Given the escalating visual token counts in video MLLMs, token reduction is essential for efficient inference\. Current strategies typically employ an additional downscaling model to prune unimportant tokens prior to full inference\. Depending on the downscaling model used, these strategies fall into two main categories:\(1\) lightweight Encoder\-Driven methods\([Bolya et al\., 2023](https://arxiv.org/html/2609.16722#bib.bib4);[Yang et al\., 2024](https://arxiv.org/html/2609.16722#bib.bib39);[Liu et al\., 2025b](https://arxiv.org/html/2609.16722#bib.bib23);[Tao et al\., 2025](https://arxiv.org/html/2609.16722#bib.bib26)\)selects tokens using only the ViT encoder\. For example, VisionZip\([Yang et al\., 2024](https://arxiv.org/html/2609.16722#bib.bib39)\)first selects a subset of dominant visual tokens using ViT attention and then merges the remaining less important tokens through averaging based on embedding similarity, thereby achieving effective token reduction\. In contrast, VidCom\([Liu et al\., 2025b](https://arxiv.org/html/2609.16722#bib.bib23)\)uses the diversity of ViT embeddings to represent frame\-wise importance, subsequently applying different token compression ratios to different frames\. However, these methods are prone to erroneous pruning, as lightweight ViTs lack the capacity to fully capture complex video semantics\.\(2\) Recent heavyweight MLLM\-driven methods\([Chen et al\., 2024b](https://arxiv.org/html/2609.16722#bib.bib8);[Zhang et al\., 2025](https://arxiv.org/html/2609.16722#bib.bib42);[zhang et al\., 2025](https://arxiv.org/html/2609.16722#bib.bib43)\)further leverage a small\-scale MLLM to more accurately detect and preserve essential visual tokens\. For instance, the current SOTA method, FlexSelect\([zhang et al\., 2025](https://arxiv.org/html/2609.16722#bib.bib43)\), employs a reduced\-size MLLM to identify significant tokens via cross\-modal attention, subsequently feeding only these selected tokens and the text query into the target MLLM for inference\. While this approach improves accuracy, it remains computationally expensive; the overhead introduced by the auxiliary MLLM effectively negates the latency gains achieved through token reduction\.

Overall, the prevailing emphasis on the downscaling model’s size for fast token reduction results in an accuracy–efficiency dilemma\. In contrast, VideoMM introduces a paradigm shift by shifting the focus from model scale to perceptual granularity\. By constructing a macro\-level proxy for fast token reduction, VideoMM substantially reduces the computational overhead of this process—even when using a MLLM to guide compression—without sacrificing semantic depth\. Empirically, VideoMM achieves faster inference than lightweight encoder\-driven methods while preserving the accuracy advantages of MLLM\-driven strategies\.111Broader related works are discussed in Appendix[A](https://arxiv.org/html/2609.16722#A1)

![Refer to caption](https://arxiv.org/html/2609.16722v1/architecture.png)Figure 2\.VideoMM Framework\. \(a\) Stage 1: Grouped Selection with Macro\-view Proxy efficiently identifies informative regions\. \(b\) Stage 2: Adaptive Macro\-Micro Inference assesses semantic sufficiency via consensus, integrating Micro\-view video only when necessary\.
## 3\.Methods

We consider standard MLLMs generally consisting of two primary components: a vision transformer encoder \(ViT\), and a large language modelMM\. Given an input video𝐕∈ℝT×H×W×3\\mathbf\{V\}\\in\\mathbb\{R\}^\{T\\times H\\times W\\times 3\}, a ViT encoder directly processes the frames into a sequence of visual tokens:

\(1\)𝐗V=ViT​\(𝐕\)∈ℝN×D,\\mathbf\{X\}\_\{V\}=\\text\{ViT\}\(\\mathbf\{V\}\)\\in\\mathbb\{R\}^\{N\\times D\},whereN=T×Hp×WpN=T\\times H\_\{p\}\\times W\_\{p\}is the total number of visual tokens,TTis the number of sampled video frames,Hp,WpH\_\{p\},W\_\{p\}denotes the patched height and patched width respectively, andDDis the hidden dimension of the ViT\. Finally, the visual tokens𝐗V\\mathbf\{X\}\_\{V\}are concatenated with the text query tokens𝐗Q\\mathbf\{X\}\_\{Q\}\. The LLMMMgenerates the response𝐘=\{yi\}i=1L\\mathbf\{Y\}=\\\{y\_\{i\}\\\}\_\{i=1\}^\{L\}autoregressively by modeling the conditional probability:

p⁡\(𝐘∣𝐗V,𝐗Q\)=∏i=1Lp⁡\(yi∣𝐗V,𝐗Q,y<i\),p\(\\mathbf\{Y\}\\mid\\mathbf\{X\}\_\{V\},\\mathbf\{X\}\_\{Q\}\)=\\prod\_\{i=1\}^\{L\}p\(y\_\{i\}\\mid\\mathbf\{X\}\_\{V\},\\mathbf\{X\}\_\{Q\},y\_\{<i\}\),wherey<iy\_\{<i\}represents the tokens generated prior to stepii\.

### 3\.1\.Overall Architecture of VideoMM

We present VideoMM, a framework that operationalizes the coarse\-to\-fine nature of human perception to resolve the efficiency\-accuracy dilemma in long\-form video understanding\. The core insight driving our architecture is that while long videos contain massive tokens, semantic density is sparse: a global macro\-view is often sufficient for general reasoning, whereas micro details are only necessary for specific, ambiguous segments\. As illustrated in Figure[2](https://arxiv.org/html/2609.16722#S2.F2), VideoMM implements this philosophy through pipelined collaboration between two distinct stages\. \(1\)Grouped Selection with Macro Proxy: This stage acts as a lightweight semantic scout\. By operating on a spatially downscaled proxy, it swiftly isolates semantically relevant temporal regions under MLLM guidance, thereby circumventing the prohibitive computational cost of processing raw high\-resolution frames\. \(2\)Adaptive Macro\-Micro Inference: This stage mimics the human cognitive process of attentional zooming, where fine\-grained details are recruited only when global perception proves ambiguous\. Instead of indiscriminately processing high\-resolution tokens, the model assesses the sufficiency of macro\-view via a consensus mechanism\. If the global context yields a confident answer, inference terminates early; the model activates high\-resolution micro tokens only when necessary to resolve uncertainty\. This hierarchical design ensures that computational resources are allocated precisely where needed, achieving substantial inference acceleration without compromising the model’s ability to resolve subtle visual details\.

### 3\.2\.Stage I: Grouped Selection with Macro Proxy

To balance the precision of MLLM\-guided token selection with computational efficiency, we operate on a spatially downscaled representation of the input video\. This strategy is grounded in the insight that downscaling significantly reduces data volume while preserving the spatial consistency of critical features\. By identifying salient regions on this lightweight proxy, we substantially minimize the computational overhead of the selection process\. Please refer to Algorithm[1](https://arxiv.org/html/2609.16722#alg1)for the Stage I pseudocode\.

Algorithm 1Stage I: Grouped Selection with Macro Proxy1:Video

𝐕∈ℝT×H×W×3\\mathbf\{V\}\\in\\mathbb\{R\}^\{T\\times H\\times W\\times 3\}, Query

QQ, Scale

kk, Groups

GG, Token budget

BB
2:Macro\-level token set

𝒯′\\mathcal\{T\}^\{\\prime\}
3:1\. Spatial Downscaling to Macro Proxy

4:

𝐕′←Downscale​\(𝐕,k\)\\mathbf\{V\}^\{\\prime\}\\leftarrow\\text\{Downscale\}\(\\mathbf\{V\},k\)⊳\\triangleright𝐕′∈ℝT×Hk×Wk×3\\mathbf\{V\}^\{\\prime\}\\in\\mathbb\{R\}^\{T\\times\\frac\{H\}\{k\}\\times\\frac\{W\}\{k\}\\times 3\}

5:

𝐗𝐕′←ViT⁡\(𝐕′\)\\mathbf\{X\}\_\{\\mathbf\{V\}^\{\\prime\}\}\\leftarrow\\mathrm\{ViT\}\(\\mathbf\{V\}^\{\\prime\}\)⊳\\trianglerightToken count reduced byk2k^\{2\}

6:

𝒯′←∅\\mathcal\{T\}^\{\\prime\}\\leftarrow\\emptyset
7:2\. Grouped Selection via MLLM Guidance

8:Interleave frames

f1:Tf\_\{1:T\}into

GGgroups

\{ℱj\}\\\{\\mathcal\{F\}\_\{j\}\\\}
9:foreach group

ℱj\\mathcal\{F\}\_\{j\}do

10:Extract cross\-modal

Aq,i\(l,h\)A^\{\(l,h\)\}\_\{q,i\}for visual token

ii
11:

si=maxl∈ℒmid⁡maxh​maxq​Aq,i\(l,h\)s\_\{i\}=\\max\_\{l\\in\\mathcal\{L\}\_\{\\text\{mid\}\}\}\\max\_\{h\}\\max\_\{q\}A^\{\(l,h\)\}\_\{q,i\}
12:

sf←∑i∈fsis\_\{f\}\\leftarrow\\sum\_\{i\\in f\}s\_\{i\}for each frame

f∈ℱjf\\in\\mathcal\{F\}\_\{j\}
13:// Retain top\-50% frames

14:

ℱj′←TopRatio⁡\(ℱj,α,key=sf\)\\mathcal\{F\}\_\{j\}^\{\\prime\}\\leftarrow\\operatorname\{TopRatio\}\(\\mathcal\{F\}\_\{j\},\\alpha,\\text\{key\}=s\_\{f\}\)
15:// Select top tokens from retained frames

16:

𝒯j′←TopK\(\{𝐗V′∈ℱj′\},B/G,key=si\)\\mathcal\{T\}\_\{j\}^\{\\prime\}\\leftarrow\\operatorname\{TopK\}\(\\\{\\mathbf\{X\}\_\{V^\{\\prime\}\}\\in\\mathcal\{F\}\_\{j\}^\{\\prime\}\\\},B/G,\\text\{key\}=s\_\{i\}\)
17:

𝒯′←𝒯′∪𝒯j′\\mathcal\{T\}^\{\\prime\}\\leftarrow\\mathcal\{T\}^\{\\prime\}\\cup\\mathcal\{T\}\_\{j\}^\{\\prime\}
18:endfor

19:return

𝒯′\\mathcal\{T\}^\{\\prime\}

Spatial Downscaling to Macro Proxy\.Given an input video𝐕∈ℝT×H×W×3\\mathbf\{V\}\\in\\mathbb\{R\}^\{T\\times H\\times W\\times 3\}, we generate a*macro*proxy𝐕′∈ℝT×Hk×Wk×3\\mathbf\{V\}^\{\\prime\}\\in\\mathbb\{R\}^\{T\\times\\frac\{H\}\{k\}\\times\\frac\{W\}\{k\}\\times 3\}by uniformly downscaling spatial dimensions by a factor222We set the downscaling factor to 2 in the main experiments, while exploring a more aggressive factor of 3 in Section[4\.3](https://arxiv.org/html/2609.16722#S4.SS3)\.ofkk\. This transformation reduces the token count byk2k^\{2\}:

𝐗𝐕′\\displaystyle\\mathbf\{X\}\_\{\\mathbf\{V\}^\{\\prime\}\}=ViT⁡\(𝐕′\)∈ℝN/k2×D\\displaystyle=\\mathrm\{ViT\}\(\\mathbf\{V\}^\{\\prime\}\)\\in\\mathbb\{R\}^\{N/k^\{2\}\\times D\}By replacing𝐕\\mathbf\{V\}with𝐕′\\mathbf\{V\}^\{\\prime\}for selection, we drastically reduce the number of visual tokens by a factor ofk2k^\{2\}which facilitates subsequent MLLM\-guided selection while preserving the global layout of salient features\.

![Refer to caption](https://arxiv.org/html/2609.16722v1/backup1.png)Figure 3\.Illustration of Stage II: Adaptive Macro\-Micro Inference\.Top:In unambiguous cases, the voting mechanism across macro groups reaches a consensus, enabling the model to generate an immediate response\.Bottom:When macro\-views diverge \(indicating ambiguity\), VideoMM maps the selected macro regions to micro\-level tokens for high\-resolution refinement\.Grouped Selection via MLLM Guidance\.To identify informative tokens within context constraints, we process the macro\-view proxy𝐕′\\mathbf\{V\}^\{\\prime\}using an interleaved grouping strategy\. We partition theTTframes \(denoted asf1,f2,…,fTf\_\{1\},f\_\{2\},\\dots,f\_\{T\}\) intoGGdistinct groups:

\(2\)ℱj=\{fi∣i∈\[1,T\],i≡j\(modG\)\},\\mathcal\{F\}\_\{j\}=\\\{f\_\{i\}\\mid i\\in\[1,T\],\\ i\\equiv j\\pmod\{G\}\\\},wherej∈\{0,…,G−1\}j\\in\\\{0,\\dots,G\-1\\\}\. This decomposition ensures that each group functions as a sparse, global representation, enabling the MLLM to form an independent yet comprehensive understanding of the video content within each group\.

Within each group, we quantify the semantic relevance of visual tokens based on the cross\-modal attention weights from the text query𝐗Q∈ℝNQ×D\\mathbf\{X\}\_\{Q\}\\in\\mathbb\{R\}^\{N\_\{Q\}\\times D\}\. Specifically, for a headhhin layerll, let𝐖Q\(l,h\),𝐖K\(l,h\)∈ℝD×dh\\mathbf\{W\}\_\{Q\}^\{\(l,h\)\},\\mathbf\{W\}\_\{K\}^\{\(l,h\)\}\\in\\mathbb\{R\}^\{D\\times d\_\{h\}\}be the projection matrices, wheredhd\_\{h\}is the head dimension\. The attention weights from text to visual tokens are computed as:

\(3\)𝐀\(l,h\)\\displaystyle\\mathbf\{A\}^\{\(l,h\)\}∈ℝNQ×N/k2\\displaystyle\\in\\mathbb\{R\}^\{N\_\{Q\}\\times N/k^\{2\}\}=Softmax⁡\(\(𝐗Q​𝐖Q\(l,h\)\)​\(𝐗V′​𝐖K\(l,h\)\)⊤dh\)\\displaystyle=\\operatorname\{Softmax\}\\left\(\\frac\{\(\\mathbf\{X\}\_\{Q\}\\mathbf\{W\}\_\{Q\}^\{\(l,h\)\}\)\(\\mathbf\{X\}\_\{V^\{\\prime\}\}\\mathbf\{W\}\_\{K\}^\{\(l,h\)\}\)^\{\\top\}\}\{\\sqrt\{d\_\{h\}\}\}\\right\)where𝐀q,i\(l,h\)\\mathbf\{A\}^\{\(l,h\)\}\_\{q,i\}represents the attention weight from theqq\-th text token to theii\-th visual token\. We focus on intermediate layersℒmid\\mathcal\{L\}\_\{\\text\{mid\}\}, as recent studies on long\-context LLM inference\([Wang et al\., 2025](https://arxiv.org/html/2609.16722#bib.bib29);[Feng et al\., 2024](https://arxiv.org/html/2609.16722#bib.bib11);[Xiao et al\., 2024a](https://arxiv.org/html/2609.16722#bib.bib37);[Wu et al\., 2024b](https://arxiv.org/html/2609.16722#bib.bib35)\)suggest that semantic retrieval is predominantly localized in the middle layers\.333While profiling specific retrieval layers even heads\([zhang et al\., 2025](https://arxiv.org/html/2609.16722#bib.bib43);[Fu et al\., 2024](https://arxiv.org/html/2609.16722#bib.bib14)\)will yield additional gains, we opted for a fixed selection of three intermediate layers to ensure both simplification and broad applicability\.Inspired by the design of the recent worst\-case importance indicator\([Feng et al\., 2025a](https://arxiv.org/html/2609.16722#bib.bib10)\), we employ maximum aggregation to distill these signals into a robust importance scoresis\_\{i\}

\(4\)si=maxl∈ℒmid⁡maxh​maxq​Aq,i\(l,h\)s\_\{i\}=\\max\_\{l\\in\\mathcal\{L\}\_\{\\text\{mid\}\}\}\\max\_\{h\}\\max\_\{q\}A^\{\(l,h\)\}\_\{q,i\}
With the relevance scoressis\_\{i\}established, we implement a hierarchical selection process to filter redundancy at both the frame and token levels\. First, at the frame level, we aggregate the importance of each frameffby summing its constituent token scores, denoted asSf=∑i∈fsiS\_\{f\}=\\sum\_\{i\\in f\}s\_\{i\}\. For each groupℱj\\mathcal\{F\}\_\{j\}, we retain the topα\\alphafraction of frames \(whereα=0\.5\\alpha=0\.5in our experiments\) with the highestSfS\_\{f\}values, forming a refined subsetℱj′\\mathcal\{F\}\_\{j\}^\{\\prime\}\.444Thisα\\alpha\-fraction frame selection primarily aims to reduce the computational overhead of re\-encoding micro tokens using the ViT encoder in the subsequent stage\.Subsequently, at the token level, we select theBBtokens uniformly from each frame groups to construct the final macro\-level token set𝒯′\\mathcal\{T\}^\{\\prime\}:

\(5\)𝒯′=⋃j=0G−1TopK\(\{𝐗V′∈ℱj′\},B/G,key=si\)\\small\\mathcal\{T\}^\{\\prime\}=\\bigcup\_\{j=0\}^\{G\-1\}\\operatorname\{TopK\}\(\\\{\\mathbf\{X\}\_\{V^\{\\prime\}\}\\in\\mathcal\{F\}\_\{j\}^\{\\prime\}\\\},B/G,\\text\{key\}=s\_\{i\}\)
Algorithm 2Stage II: Adaptive Macro\-Micro InferenceMacro token set

𝒯′\\mathcal\{T\}^\{\\prime\}\(from Stage I\), Query

QQ, Downscale factor

kk, Frame Group

ℱj\\mathcal\{F\}\_\{j\}, Model

ℳ\\mathcal\{M\}
Final Answer

RR
1\. Macro\-Consensus Verification

Yresponse←\{ℳ⁡\(𝒯′,Q\)\}Y\_\{\\text\{response\}\}\\leftarrow\\\{\\mathcal\{M\}\(\\mathcal\{T\}^\{\\prime\},Q\)\\\}

foreach group

ℱj\\mathcal\{F\}\_\{j\}do

Extract a subset

𝒯j′⊆𝒯′\\mathcal\{T\}^\{\\prime\}\_\{j\}\\subseteq\\mathcal\{T\}^\{\\prime\}belonging to group

ℱj\\mathcal\{F\}\_\{j\}
a​n​sj←ℳ⁡\(𝒯j,Q\)ans\_\{j\}\\leftarrow\\mathcal\{M\}\(\\mathcal\{T\}\_\{j\},Q\)⊳\\trianglerightGenerate answer per group

Add

a​n​sjans\_\{j\}to

YresponseY\_\{\\text\{response\}\}
endfor

ifAll responses in

YresponseY\_\{\\text\{response\}\}are consistentthen

returnthe consensus result from

YresponseY\_\{\\text\{response\}\}
endif

2\. Micro\-Detail Recruitment

Obtain raw frames

𝐕^\\mathbf\{\\hat\{V\}\}associated with

𝒯′\\mathcal\{T\}^\{\\prime\}
Encode micro features:

𝐗V^←ViT​\(𝐕^\)\\mathbf\{X\}\_\{\\hat\{V\}\}\\leftarrow\\text\{ViT\}\(\\mathbf\{\\hat\{V\}\}\)
// Macro\-to\-Micro token mapping via Eq\.[7](https://arxiv.org/html/2609.16722#S3.E7)

𝒯←Gather​\(𝐗V^,𝒯′\)\\mathcal\{T\}\\leftarrow\\text\{Gather\}\(\\mathbf\{X\}\_\{\\hat\{V\}\},\\mathcal\{T\}^\{\\prime\}\)

return

ℳ⁡\(𝒯,Q\)\\mathcal\{M\}\(\\mathcal\{T\},Q\)

### 3\.3\.Stage II: Adaptive Macro\-Micro Inference

While Stage I identifieswhereto look, Stage II determineshow deepto look\. As illustrated in Figure[3](https://arxiv.org/html/2609.16722#S3.F3), this stage is grounded in the principle of computational efficiency: most video queries can be answered via global semantic cues \(Macro\), while only a fraction require high\-resolution scrutiny \(Micro\)\. To exploit this, VideoMM employs an adaptive granularity inference path, prioritizing low\-cost macro inference and escalating to high\-cost micro reasoning only when ambiguity is detected\. Please refer to Algorithm[2](https://arxiv.org/html/2609.16722#alg2)for the Stage II pseudocode\.

Macro\-Consensus Verification\.We employ aConsensus\-Based Votingmechanism\([Wang et al\., 2023](https://arxiv.org/html/2609.16722#bib.bib31)\)to evaluate the sufficiency of the macro\-view\. Each groupℱj\\mathcal\{F\}\_\{j\}, alongside the jointly selected macro token set𝒯′\\mathcal\{T\}^\{\\prime\}\(treated as an additional independent group\), processed independently, acts as a weak estimator generating a preliminary response\. We posit that prediction invariance across disjoint temporal subsets serves as a robust proxy for confidence\. If all independent groups converge on a consensus, macro\-level semantics are deemed sufficient555Other voting strategies are also applicable; notably, majority voting boosts throughput with little accuracy drop \(Section[4\.3](https://arxiv.org/html/2609.16722#S4.SS3)\)\.\. In such cases, the model terminates inference early using the consensus result, bypassing the overhead of micro\-level processing\.666Given that prefilling is completed in Stage I and each group contains only a few macro Tokens, the overhead of this consensus voting is negligible relative to its substantial benefits\.

Micro\-Detail Recruitment\.When macro\-views diverge—indicating ambiguity and the need for finer detail—, VideoMM activates micro\-level refinement\. This process necessitates encoding the relevant raw video frames into micro tokens:

\(6\)𝐗V^=ViT​\(𝐕^\)∈ℝα​N×D\\mathbf\{X\}\_\{\\hat\{V\}\}=\\text\{ViT\}\(\\mathbf\{\\hat\{V\}\}\)\\in\\mathbb\{R\}^\{\\alpha N\\times D\}
where𝐕^\\mathbf\{\\hat\{V\}\}denotes the raw video input constructed from the frames selected in the first stage, consisting of only theα\\alpha\-fraction of the original frames\. This effectively controls the computational overhead of ViT re\-encoding for these micro tokens\. We then map the critical macro tokens𝒯′\\mathcal\{T\}^\{\\prime\}selected in Stage I back to their corresponding micro\-level representations𝒯\\mathcal\{T\}\. Specifically, for any macro token in𝒯′\\mathcal\{T\}^\{\\prime\}identified by\(t,h′,w′\)\(t,h^\{\\prime\},w^\{\\prime\}\)in frame and spatial coordinates, it corresponds to a micro\-token set from the newly encoded𝐗V^\\mathbf\{X\}\_\{\\hat\{V\}\}within the same spatial region:

\(7\)𝒯\(t,h,w\)=\{𝐗V^​\[t,h,w\]∣\{h′​k≤h<\(h′\+1\)​k,w′​k≤w<\(w′\+1\)​k\}\.\\mathcal\{T\}\_\{\(t,h,w\)\}=\\left\\\{\\mathbf\{X\}\_\{\\hat\{V\}\}\[t,h,w\]\\mid\\begin\{cases\}h^\{\\prime\}k\\leq h<\(h^\{\\prime\}\+1\)k,\\\\ w^\{\\prime\}k\\leq w<\(w^\{\\prime\}\+1\)k\\end\{cases\}\\right\\\}\.
The complete micro token set𝒯\\mathcal\{T\}is then the union of these subsets:𝒯=⋃𝒯\(t,h,w\)\\mathcal\{T\}=\\bigcup\\mathcal\{T\}\_\{\(t,h,w\)\}, which is then fed into the models for the definitive response\. This paradigm balances efficiency and accuracy by using the macro stage as a filter for noise and the micro stage as a specialized solver for detail\.

Table 1\.Performance comparison on video understanding benchmarks\. Throughput denotes inference speed in samples per minute\. The best two results arebolded\. VideoMM demonstrates leading performance in both accuracy and throughput, whereas other methods typically prioritize one over the other\.MethodLongVideoBenchVideoMMELVBenchAverageAccuracyThroughputAccuracyThroughputAccuracyThroughputAccuracyThroughputQwen2\.5\-VL\-7BVanilla57\.670\.7761\.560\.8539\.570\.6552\.930\.76VisionZip58\.23 \(↑1\.0%\)3\.63 \(4\.7x\)62\.14 \(↑0\.9%\)3\.84 \(4\.5x\)41\.38 \(↑4\.6%\)3\.10 \(4\.8x\)53\.92 \(↑1\.9%\)3\.52 \(4\.6x\)VidCom54\.60 \(↓5\.3%\)4\.27 \(5\.5x\)61\.56 \(\-0\.0%\)4\.42 \(5\.2x\)40\.80 \(↑3\.1%\)4\.32 \(6\.6x\)52\.32 \(↓1\.2%\)4\.34 \(5\.7x\)Flexselect64\.10 \(↑11\.1%\)1\.83 \(2\.4x\)68\.85 \(↑11\.8%\)1\.84 \(2\.2x\)51\.45 \(↑30\.0%\)1\.50 \(2\.3x\)61\.47 \(↑16\.1%\)1\.72 \(2\.3x\)VideoMM64\.70 \(↑12\.2%\)5\.16 \(6\.7x\)68\.41 \(↑11\.1%\)5\.47 \(6\.4x\)50\.94 \(↑28\.7%\)4\.20 \(6\.5x\)61\.35 \(↑15\.9%\)4\.94 \(6\.5x\)GLM\-4\.1V\-9BVanilla58\.980\.5166\.670\.5847\.900\.4257\.850\.50VisionZip55\.35 \(↓6\.2%\)4\.10 \(8\.0x\)59\.55 \(↓10\.7%\)4\.30 \(7\.4x\)36\.41 \(↓24\.0%\)3\.60 \(8\.6x\)50\.44 \(↓12\.8%\)4\.00 \(8\.0x\)VidCom51\.68 \(↓12\.4%\)3\.50 \(6\.9x\)56\.48 \(↓15\.3%\)3\.70 \(6\.4x\)37\.70 \(↓21\.3%\)2\.96 \(7\.0x\)48\.62 \(↓16\.0%\)3\.39 \(6\.8x\)Flexselect60\.66 \(↑2\.8%\)1\.47 \(2\.9x\)65\.15 \(↓2\.3%\)1\.57 \(2\.7x\)50\.16 \(↑4\.7%\)1\.28 \(3\.0x\)58\.66 \(↑1\.4%\)1\.44 \(2\.9x\)VideoMM64\.32 \(↑9\.1%\)3\.77 \(7\.4x\)67\.37 \(↑1\.0%\)4\.21 \(7\.3x\)49\.52 \(↑3\.4%\)2\.87 \(6\.8x\)60\.40 \(↑4\.4%\)3\.62 \(7\.2x\)Qwen3\-VL\-8BVanilla66\.940\.8571\.780\.9754\.10\.7064\.270\.84VisionZip57\.97 \(↓13\.4%\)2\.47 \(2\.9x\)65\.55 \(↓8\.7%\)2\.43 \(2\.5x\)40\.22 \(↓25\.7%\)1\.92 \(2\.7x\)54\.58 \(↓15\.1%\)2\.27 \(2\.7x\)VidCom59\.16 \(↓11\.6%\)4\.39 \(5\.2x\)64\.59 \(↓10\.0%\)3\.04 \(3\.1x\)44\.03 \(↓18\.6%\)2\.39 \(3\.4x\)55\.93 \(↓13\.0%\)3\.27 \(3\.9x\)Flexselect67\.17 \(↑0\.3%\)1\.30 \(1\.5x\)72\.19 \(↑0\.6%\)1\.32 \(1\.4x\)55\.20 \(↑2\.0%\)0\.97 \(1\.4x\)64\.85 \(↑0\.9%\)1\.20 \(1\.4x\)VideoMM67\.54 \(↑0\.9%\)3\.69 \(4\.3x\)71\.67 \(↓0\.2%\)3\.59 \(3\.7x\)54\.36 \(↑0\.5%\)2\.50 \(3\.6x\)64\.52 \(↑0\.4%\)3\.26 \(3\.9x\)

### 3\.4\.Efficiency Analysis

To quantify the efficiency gains of VideoMM, we analyze its asymptotic complexity relative to standard MLLM inference\. We focus on the self\-attention mechanism, which constitutes the primary computational bottleneck in long\-video understanding\. LetNNdenote the number of visual tokens in raw high\-resolution video\. Given thatNNscales significantly with video duration, we analyze complexity with respect toNN, treating text sequence length as negligible\.

Complexity Derivation\.Standard global attention generally scales quadratically, i\.e\.,𝒪⁡\(N2\)\\mathcal\{O\}\(N^\{2\}\)\. VideoMM optimizes this via three key parameters: the macro downscaling factorkk, the number of temporal groupsGG, and the macro\-selection budgetBB\(introduced in Stage I\) combined with the macro\-consensus probabilityα\\alpha\(governed by Stage II\)\. The expected computational cost of VideoMM,𝔼⁡\[ΩOurs\]\\mathbb\{E\}\[\\Omega\_\{\\text\{Ours\}\}\], is the sum of two components:

1. \(1\)Macro Processing \(Stage I & Stage II\-Consensus\):BothGrouped Selection\(Stage I\) andMacro\-Consensus Verification\(Stage II\) operate on sharedMacro Proxy\. Here, token count is reduced toN/k2N/k^\{2\}and divided intoGGgroups\. Since attention is restricted within these groups for relevance estimation \(sis\_\{i\}\) and consensus voting, the computational overhead is:G⋅\(N/k2G\)2=N2G​k4G\\cdot\\left\(\\frac\{N/k^\{2\}\}\{G\}\\right\)^\{2\}=\\frac\{N^\{2\}\}\{Gk^\{4\}\}\.
2. \(2\)Micro Processing \(Stage II\-Refinement\):This term corresponds to the conditional activation of theMicro\-Detail Recruitmentphase in Stage II\. Letρ=\(B⋅k2\)/N\\rho=\(B\\cdot k^\{2\}\)/Ndenote the ratio of selected high\-resolution tokens\. Crucially, this computation is only incurred when the consensus mechanismfails\. Letβ\\betadenote the probability of successfully reaching a consensus \(i\.e\., early exit\)\. The expected cost is:\(1−β\)​\(ρ​N\)2\(1\-\\beta\)\(\\rho N\)^\{2\}\.

Summing these terms, the total expected complexity is:

\(8\)𝔼⁡\[ΩOurs\]=𝒪⁡\(N2​\[1G​k4⏟Macro Proxy\+\(1−β\)​ρ2⏟Micro Refinement\]\)\.\\mathbb\{E\}\[\\Omega\_\{\\text\{Ours\}\}\]=\\mathcal\{O\}\\left\(N^\{2\}\\left\[\\underbrace\{\\frac\{1\}\{Gk^\{4\}\}\}\_\{\\text\{Macro Proxy\}\}\+\\underbrace\{\(1\-\\beta\)\\rho^\{2\}\}\_\{\\text\{Micro Refinement\}\}\\right\]\\right\)\.
Equation[8](https://arxiv.org/html/2609.16722#S3.E8)demonstrates the transformation of quadratic dependency into an efficient form via two benefits:\(1\) Grouping and Downscaling:The term1G​k4\\frac\{1\}\{Gk^\{4\}\}represents the structural cost reduction, wherekkandGGprovide quartic and linear complexity drops, respectively\.\(2\) Adaptive Inference:The term\(1−β\)​ρ2\(1\-\\beta\)\\rho^\{2\}captures the gain from adaptive efficiency\. Empirically, high early\-exit rates \(β≈70%\\beta\\approx 70\\%\) and extreme sparsity \(ρ<3%\\rho<3\\%\) confine heavy computation strictly to ambiguous instances, preserving detail with minimal overhead\.

## 4\.Experiments

### 4\.1\.Settings

Models\.To demonstrate the universality of our framework, we conduct evaluations on three leading open\-source MLLMs: Qwen2\.5\-VL\-7B\([Bai et al\., 2025](https://arxiv.org/html/2609.16722#bib.bib3)\)and GLM\-4\.1\-VL\-9B\-Thinking\([Team et al\., 2025](https://arxiv.org/html/2609.16722#bib.bib28)\)support 64K context windows, while Qwen3\-VL\-8B\([Team, 2025](https://arxiv.org/html/2609.16722#bib.bib27)\)extends to 256K natively\. For GLM\-4\.1\-VL\-9B\-Thinking, we disable its thinking mode to force direct responses and keep evaluation costs manageable\. These models were selected as state\-of\-the\-art for long\-context video understanding; furthermore, their dynamic\-resolution ViTs naturally support varied input resolutions, inherently aligning with our hierarchical macro\-micro paradigm\.

Baselines\.We benchmark our approach against three representative token reduction methods, categorized by their computational characteristics:\(i\) lightweight encoder\-driven methodsand\(ii\) heavyweight MLLM\-driven methods\. For the lightweight category, we evaluate VisionZip\([Yang et al\., 2024](https://arxiv.org/html/2609.16722#bib.bib39)\), which utilizes ViT attention for selection, and VidCom\([Liu et al\., 2025b](https://arxiv.org/html/2609.16722#bib.bib23)\), which employ token uniqueness for compression\. For the heavyweight category, we utilize FlexSelect\([zhang et al\., 2025](https://arxiv.org/html/2609.16722#bib.bib43)\)as a representative baseline\. To ensure a fair comparison and strictly control computational costs, we adopt a unified token budget across all baseline categories\. Specifically, for both lightweight encoder\-driven methods\([Yang et al\., 2024](https://arxiv.org/html/2609.16722#bib.bib39);[Liu et al\., 2025b](https://arxiv.org/html/2609.16722#bib.bib23)\)and the heavyweight MLLM\-driven FlexSelect\([zhang et al\., 2025](https://arxiv.org/html/2609.16722#bib.bib43)\), we retain a fixed 8,192 visual tokens, following common practice\. For VideoMM \(G=4,k=2G=4,k=2\), we align our Micro token budget with this unified 8,192 limit; consequently, the Macro budget is set to 2,048 \(8,192/48,192/4\)\.

Benchmarks\.Our evaluation employs two widely adopted long\-video benchmarks: LongVideoBench\([Wu et al\., 2024a](https://arxiv.org/html/2609.16722#bib.bib34)\), focusing on fine\-grained retrieval and complex reasoning within long\-context videos, and LVBench\([Wang et al\., 2024](https://arxiv.org/html/2609.16722#bib.bib30)\)for testing the capability over hour\-long video analysis\. Furthermore, we assess multi\-scale temporal understanding using VideoMME\([Fu et al\., 2025](https://arxiv.org/html/2609.16722#bib.bib13)\)\. We sample videos uniformly at 1 FPS, up to a maximum of 512 frames\. All evaluations use the LMMs\-Eval framework\([Zhang et al\., 2024a](https://arxiv.org/html/2609.16722#bib.bib41)\)following standard protocols\.

Table 2\.Comparison on long\-video benchmarks under comparable throughput \(FlexSelect reduced to 256 frames; Acc\.: Accuracy \(%\); Thpt\.: Throughput \(samples/min\)\)Figure 4\.Impact of input frame counts on LongVideoBench using Qwen3\-VL\-8B\. Thesolid linesrepresent accuracy \(left axis\), while thetransparent barsrepresent throughput \(right axis\)\.
### 4\.2\.Main Results

Table[1](https://arxiv.org/html/2609.16722#S3.T1)presents a comparative evaluation of accuracy and throughput for VideoMM against various token compression methods across multiple models and benchmarks\. Overall,Encoder\-driven methodsexcel in throughput, whereasMLLM\-driven methodsachieve higher accuracy\. In contrast, VideoMM demonstrates superiority in both accuracy and efficiency, enabled by its novel macro\-micro perceptual granularity\. Detailed performance on the three benchmarks are provided in the appendix\.

Comparison with Encoder\-Driven Methods\.

By leveraging macro\-level MLLM\-guided token selection, VideoMM significantly outperforms lightweight encoder\-driven methods \(e\.g\., VisionZip, VidCom\) in both accuracy and throughput\. For instance, on Qwen3\-VL\-8B and Qwen2\.5\-VL\-7B, VideoMM attains average scores of 64\.52 and 61\.35 across three benchmarks, substantially outperforming VidCom’s 55\.93 and 52\.32, respectively\.Moreover, even in terms of throughput—where encoder\-driven methods typically excel—VideoMM achieves superior performance due to its macro\-micro paradigm\. On the same models, VideoMM reaches average throughputs of 3\.26 and 4\.94 samples/min, delivering comparable or superior overall efficiency compared to VidCom’s 3\.27 and 4\.34\.

Comparison with MLLM\-Driven methods\.By employing Adaptive Macro\-Micro Inference, VideoMM achieves a substantial boost in throughput while preserving accuracy on par with FlexSelect\. For example, on Qwen3 and GLM, VideoMM achieves average scores of 64\.52 and 60\.40 across three benchmarks, close to FlexSelect’s 64\.85 and 58\.66, respectively\. Crucially, VideoMM demonstrates a significant throughput advantage: on Qwen3, it reaches 3\.26 samples/min—surpassing FlexSelect’s 1\.20—which corresponds to a3\.9× speedupover the vanilla\. This gap is even more pronounced on GLM\-4\.1V\-9B, where VideoMM achieves 3\.62 samples/min compared to FlexSelect’s 1\.44, yielding a7\.2× speedupover vanilla\. In real\-world long\-video understanding, balancing accuracy and throughput often necessitates adjusting input frame counts to meet deployment constraints\. Thus, in Table[2](https://arxiv.org/html/2609.16722#S4.T2), we focus specifically on two long\-video benchmarks \(LongVideoBench and LVBench\), employing frame reduction on FlexSelect \(to 256 frames\) to compare accuracy under iso\-throughput conditions\. Overall, our method consistently achieves higher accuracy at comparable throughput\. For instance, with Qwen2\.5\-VL\-7B, VideoMM outperforms FlexSelect by 4\.7 point in average accuracy \(59\.32 vs\. 54\.62\) while simultaneously increasing throughput\. This advantage extends to GLM\-4\.1V\-9B and Qwen3\-VL\-8B, where our method maintains higher accuracy at comparable inference speeds\.

Figure 5\.Accuracy\-efficiency trade\-offs across VideoMM family variants \(Qwen3\-VL\-8B, LongVideoBench\)\. Legend colors from dark to light correspond to frame counts of 512, 386, 256, and 128, respectively\. The VideoMM family establishes a superior Pareto frontier, validating the intrinsic efficacy of the proposed macro\-micro paradigm\.Table 3\.Early\-exit statisticsβ\\beta\(LongVideoBench, Qwen3\-VL\-8B\)\.
### 4\.3\.Analysis Experiments

Detail discussion of Adaptive Computation\.VideoMM employs an adaptive computation mechanism within a macro\-micro hierarchical framework, dynamically allocating computational resources based on sample complexity\. To investigate the accuracy\-efficiency trade\-off from this mechanism, we instantiate two variants by modulating the adaptive strategy777In Appendix[D](https://arxiv.org/html/2609.16722#A4)and[E](https://arxiv.org/html/2609.16722#A5), we explore additional variants within the Macro\-Micro paradigm to further enhance efficiency\.:\(1\) VideoMM\+\(Efficiency\-Prioritized\):This variant adopts a majority\-voting strategy for Macro\-Consensus Verification, allowing early exit upon majority agreement\. This significantly increases the probability of early termination at the cost\-efficient macro level compared to the standard VideoMM, which requires unanimous consensus\.\(2\) VideoMM\-\(Non\-Adaptive Ablation\):Conversely, this variant enforces all samples to undergo fine\-grained inference at the micro level, completely disabling the adaptive computation mechanism\. Figure[5](https://arxiv.org/html/2609.16722#S4.F5)illustrates the accuracy\-efficiency trade\-off across varying frame inputs, while Table[3](https://arxiv.org/html/2609.16722#S4.T3)details the early\-exit statisticsβ\\beta\. Overall, as we transition from VideoMM\-to VideoMM and finally VideoMM\+, both the early\-exit probabilityβ\\betaand inference speed progressively increase, though potentially at the cost of marginal accuracy drops\. Considering both accuracy and efficiency, both adaptive variants \(VideoMM and VideoMM\+\) yield a superior trade\-off compared to the non\-adaptive VideoMM\-, occupying the optimal upper\-right region of the plot\. Between the two, the choice depends on specific application constraints\. For instance, with 512\-frame inputs, VideoMM prioritizes higher accuracy \(67\.54\) with moderate throughput \(3\.69\), whereas VideoMM\+achieves significantly higher throughput \(4\.90\) with slightly lower accuracy \(66\.87\)\. Overall, the VideoMM family establishes a new optimal frontier on accuracy\-throughput landscape\. This success validates the intrinsic efficacy of macro\-micro perceptual granularity paradigm, offering a novel direction that moves beyond the prevailing model\-centric perspective\.

Influence of Input Frames\.As shown in Figure[4](https://arxiv.org/html/2609.16722#S4.F4), we further evaluate the performance of VideoMM on the challenging LongVideoBench benchmark using varying input frame count to simulate diverse real\-world deployment constraints\. Typically, increasing the number of frames improves long\-video understanding accuracy but incurs a substantial token overhead, thereby significantly reducing throughput\. However, VideoMM effectively mitigates the trade\-off between efficiency and accuracy, maintaining superior accuracy while markedly improving throughput across all input frame settings\. For instance, with 512 frames, VideoMM attains a superior accuracy of 67\.54% and achieves speedups of 4\.3× over vanilla and 2\.8× over FlexSelect\. Notably, the throughput of VideoMM at 512 frames is comparable to that of FlexSelect at only 256 frames\. Consequently, in practical applications, Video\-MM enables the efficient processing of longer video inputs, thereby unlocking superior understanding capabilities from these extended contexts\.

Influence of Downscale factorkk\.In our main experiments, we set the downscale factor tok=2k=2, reducing the first\-stage macro proxy tokens to one\-quarter of the original count to maintain high accuracy\. Here, we investigate a more aggressive downscaling strategy by settingk=3k=3—which reduces tokens to 1/9—to evaluate the impact on the accuracy\-throughput trade\-off\. As shown in Table[4](https://arxiv.org/html/2609.16722#S4.T4), increasing downscale factork=3k=3naturally incurs a slight accuracy cost compared tok=2k=2\(65\.89 vs\. 67\.54\)\. However, this yields a substantial gain in computational speed, nearly doubling the throughput from 3\.69 to 6\.74 samples per second\. Crucially, this aggressivek=3k=3configuration establishes a superior Pareto frontier for highly efficient long\-video understanding\. When benchmarked against lightweight Encoder\-driven methods \(e\.g\., VisionZip and VidCom\), VideoMM withk=3k=3demonstrates comprehensive dominance: it not only retains higher accuracy \(65\.89 vs\.≈\\approx64\.0\) but also achieves drastically higher throughput \(7\.9× vs\.≈\\approx2\.2×\)\. The extreme downscaling strategy remains highly effective primarily because our Stage II consensus mechanism acts as a reliable safety net; when severe perceptual ambiguity is induced by thek=3k=3compression, the resulting macro\-level divergence typically triggers a fallback to micro\-level refinement, mitigating the risk of catastrophic reasoning failures\. This result confirms the robustness of our macro\-proxy mechanism even under aggressive downscaling\. Consequently, the downscale factor serves as a practical control knob, allowing users to seamlessly transition between high\-precision and high\-speed regimes to meet diverse deployment constraints\.

Table 4\.Impact of downscale factorkk\. Evaluated on LongVideoBench using Qwen3\-VL\-8B\. Throughput is measured in samples per min\. Best results are highlighted inbold\.Task\-Aware Behavior of Adaptive Computation\.To further investigate the dynamics of our macro\-micro hierarchical framework, we quantitatively analyze the correlation between the macro\-level early\-exit probability \(β\\beta\) and specific task categories on the LongVideoBench\. Following the task taxonomy introduced in the original benchmark\([Wu et al\., 2024a](https://arxiv.org/html/2609.16722#bib.bib34)\), the adaptive mechanism exhibits distinct routing preferences depending on the semantic requirements of the query\. For perception\-centric tasks \(Level\-1\) characterized by explicit temporal anchors—such as Text\-Referred Event \(T2E\) and Object\-Referred Event \(O2E\)—VideoMM triggers early exits at a consistently high rate \(ranging from76%76\\%to83%83\\%\)\. Circumventing the fine\-grained micro level in these localized tasks significantly reduces inference latency\. This phenomenon suggests that coarse\-grained proxy tokens are highly effective for extracting localized semantics, whereas enforcing full\-resolution inference may inadvertently cause the model to over\-assimilate irrelevant visual noise\. Conversely, for relation\-centric tasks \(Level\-2\) that necessitate global temporal logic—such as Sequence of Scenes \(SSS\) and Object Before/After Object \(O3O\)—the early\-exit rate decreases to approximately60%60\\%\. In these complex scenarios, the macro\-level consensus naturally diverges\. This divergence inherently stems from the fact that the spatial downsampling within the macro proxy often obscures subtle visual state transitions and fine\-grained action boundaries essential for stringent temporal deduction\. Recognizing this semantic deficit, the framework conservatively routes the samples to the micro level for comprehensive tracking, ensuring lossless visual features for long\-range dependencies\. Consequently, the macro\-micro paradigm functions as an implicit task\-aware router: it accelerates and filters noise for localized queries while prudently preserving high\-resolution computational capacity for long\-span reasoning\. From a system optimization perspective, this dynamic routing profoundly regulates the inference memory footprint\.

Table 5\.Quantitative analysis of early\-exit behaviors across representative tasks on LongVideoBench\. The macro\-micro paradigm naturally exhibits higher early\-exit rates for localized perception tasks, while conservatively maintaining deep inference \(lowestβ\\beta\) for complex relation tasks\.

## 5\.Conclusion

In this work, we identified a fundamental dilemma in efficient long\-form video understanding: existing token reduction methods rely on model\-centric downsizing for token selection, failing to reconcile the trade\-off between selection precision and computational overhead\. To address this, we proposedVideoMM, a novel framework that orchestrates a paradigm shift from model scale to adaptive perceptual granularity\. By adhering to a coarse\-to\-fine philosophy, VideoMM decouples semantic filtering from dense processing via a cost\-effectiveMacro Proxy, while theAdaptive Macro\-Micro Inferencemechanism dynamically recruits high\-fidelity tokens only when necessary\.Extensive evaluations on LongVideoBench and other benchmarks demonstrate that VideoMM not only breaks the efficiency bottleneck—achieving a6\.13×6\.13\\timesspeedup—but also improves accuracy by over 7% compared to full\-context baselines\. Furthermore, our in\-depth analysis of multiple variants under the Macro\-Micro design demonstrates that the VideoMM family establishes a superior Pareto frontier compared to prior arts, empirically confirming the inherent effectiveness of thehierarchical perceptual granularity paradigm\. We believe this promising perspective opens a new avenue for efficient video understanding, facilitating the real\-world deployment of multimodal systems in long\-context scenarios\.

## References

- AlShami et al\.\(2024\)Ali K\. AlShami, Ryan Rabinowitz, Khang Lam, Yousra Shleibik, Melkamu Mersha, Terrance Boult, and Jugal Kalita\. 2024\.SMART\-vision: survey of modern action recognition techniques in vision\.*Multimedia Tools and Applications*84, 27 \(Dec\. 2024\), 32705–32776\.[doi:10\.1007/s11042\-024\-20484\-5](https://doi.org/10.1007/s11042-024-20484-5)
- Bai et al\.\(2025\)Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin\. 2025\.Qwen2\.5\-VL Technical Report\.*arXiv preprint arXiv:2502\.13923*\(2025\)\.
- Bolya et al\.\(2023\)Daniel Bolya, Cheng\-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman\. 2023\.Token Merging: Your ViT but Faster\. In*International Conference on Learning Representations*\.
- Cai et al\.\(2024\)Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, and Xiao Wen\. 2024\.Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling\.*arXiv preprint arXiv:2406\.02069*\(2024\)\.
- Chen et al\.\(2025\)Boyu Chen, Zhengrong Yue, Siran Chen, Zikang Wang, Yang Liu, Peng Li, and Yali Wang\. 2025\.LVAgent: Long Video Understanding by Multi\-Round Dynamical Collaboration of MLLM Agents\.arXiv:2503\.10200 \[cs\.CV\][https://arxiv\.org/abs/2503\.10200](https://arxiv.org/abs/2503.10200)
- Chen et al\.\(2023\)Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean\-Baptiste Lespiau, Laurent Sifre, and John Jumper\. 2023\.Accelerating Large Language Model Decoding with Speculative Sampling\.arXiv:2302\.01318 \[cs\.CL\][https://arxiv\.org/abs/2302\.01318](https://arxiv.org/abs/2302.01318)
- Chen et al\.\(2024b\)Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang\. 2024b\.An Image is Worth 1/2 Tokens After Layer 2: Plug\-and\-Play Inference Acceleration for Large Vision\-Language Models\.arXiv:2403\.06764 \[cs\.CV\][https://arxiv\.org/abs/2403\.06764](https://arxiv.org/abs/2403.06764)
- Chen et al\.\(2024a\)Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai\. 2024a\.Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual\-Linguistic Tasks\. In*2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*\. 24185–24198\.[doi:10\.1109/CVPR52733\.2024\.02283](https://doi.org/10.1109/CVPR52733.2024.02283)
- Feng et al\.\(2025a\)Yuan Feng, Haoyu Guo, JunLin Lv, S Kevin Zhou, and Xike Xie\. 2025a\.Taming the Fragility of KV Cache Eviction in LLM Inference\.*arXiv preprint arXiv:2510\.13334*\(2025\)\.
- Feng et al\.\(2024\)Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S Kevin Zhou\. 2024\.Ada\-kv: Optimizing kv cache eviction by adaptive budget allocation for efficient llm inference\.*arXiv preprint arXiv:2407\.11550*\(2024\)\.
- Feng et al\.\(2025b\)Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S Kevin Zhou\. 2025b\.Identify critical kv cache in llm inference from an output perturbation perspective\.*arXiv preprint arXiv:2502\.03805*\(2025\)\.
- Fu et al\.\(2025\)Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al\.2025\.Video\-mme: The first\-ever comprehensive evaluation benchmark of multi\-modal llms in video analysis\. In*CVPR*\.
- Fu et al\.\(2024\)Yu Fu, Zefan Cai, Abedelkadir Asi, Wayne Xiong, Yue Dong, and Wen Xiao\. 2024\.Not All Heads Matter: A Head\-Level KV Cache Compression Method with Integrated Retrieval and Reasoning\.*arXiv preprint arXiv:2410\.19258*\(2024\)\.
- Ji et al\.\(2025\)Yicheng Ji, Jun Zhang, Heming Xia, Jinpeng Chen, Lidan Shou, Gang Chen, and Huan Li\. 2025\.Specvlm: Enhancing speculative decoding of video llms via verifier\-guided token pruning\. In*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*\. 7216–7230\.
- Jiang et al\.\(2024\)Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H\. Abdi, Dongsheng Li, Chin\-Yew Lin, Yuqing Yang, and Lili Qiu\. 2024\.MInference 1\.0: Accelerating Pre\-filling for Long\-Context LLMs via Dynamic Sparse Attention\. In*The Thirty\-eighth Annual Conference on Neural Information Processing Systems*\.[https://openreview\.net/forum?id=fPBACAbqSN](https://openreview.net/forum?id=fPBACAbqSN)
- Leviathan et al\.\(2023\)Yaniv Leviathan, Matan Kalman, and Yossi Matias\. 2023\.Fast Inference from Transformers via Speculative Decoding\.arXiv:2211\.17192 \[cs\.LG\][https://arxiv\.org/abs/2211\.17192](https://arxiv.org/abs/2211.17192)
- Li et al\.\(2024\)Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen\. 2024\.SnapKV: LLM Knows What You are Looking for Before Generation\. In*The Thirty\-eighth Annual Conference on Neural Information Processing Systems*\.[https://openreview\.net/forum?id=poE54GOq2l](https://openreview.net/forum?id=poE54GOq2l)
- Li et al\.\(2025\)Yucheng Li, Huiqiang Jiang, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Amir H\. Abdi, Dongsheng Li, Jianfeng Gao, Yuqing Yang, and Lili Qiu\. 2025\.MMInference: Accelerating Pre\-filling for Long\-Context VLMs via Modality\-Aware Permutation Sparse Attention\.arXiv:2504\.16083 \[cs\.CV\][https://arxiv\.org/abs/2504\.16083](https://arxiv.org/abs/2504.16083)
- Liu et al\.\(2025a\)Huabin Liu, Filip Ilievski, and Cees G\. M\. Snoek\. 2025a\.Commonsense Video Question Answering through Video\-Grounded Entailment Tree Reasoning\. In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*\. 3262–3271\.
- Liu et al\.\(2024\)Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee\. 2024\.LLaVA\-NeXT: Improved reasoning, OCR, and world knowledge\.[https://llava\-vl\.github\.io/blog/2024\-01\-30\-llava\-next/](https://llava-vl.github.io/blog/2024-01-30-llava-next/)
- Liu et al\.\(2021\)Xiaolong Liu, Yao Hu, Song Bai, Fei Ding, Xiang Bai, and Philip H\. S\. Torr\. 2021\.Multi\-Shot Temporal Event Localization: A Benchmark\. In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*\. 12596–12606\.
- Liu et al\.\(2025b\)Xuyang Liu, Yiyu Wang, Junpeng Ma, and Linfeng Zhang\. 2025b\.Video Compression Commander: Plug\-and\-Play Inference Acceleration for Video Large Language Models\. In*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*\. Association for Computational Linguistics, Suzhou, China, 1910–1924\.[doi:10\.18653/v1/2025\.emnlp\-main\.98](https://doi.org/10.18653/v1/2025.emnlp-main.98)
- Tang et al\.\(2024\)Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han\. 2024\.Quest: Query\-Aware Sparsity for Efficient Long\-Context LLM Inference\.
- Tang et al\.\(2025\)Yunlong Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin, Rongyi Zhu, Ali Vosoughi, Chao Huang, Zeliang Zhang, Pinxin Liu, Mingqian Feng, Feng Zheng, Jianguo Zhang, Ping Luo, Jiebo Luo, and Chenliang Xu\. 2025\.Video Understanding with Large Language Models: A Survey\.*IEEE Transactions on Circuits and Systems for Video Technology*\(2025\), 1–1\.[doi:10\.1109/TCSVT\.2025\.3566695](https://doi.org/10.1109/TCSVT.2025.3566695)
- Tao et al\.\(2025\)Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang\. 2025\.DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models\. In*Proceedings of the Computer Vision and Pattern Recognition Conference*\. 18992–19001\.
- Team \(2025\)Qwen Team\. 2025\.Qwen3 Technical Report\.arXiv:2505\.09388 \[cs\.CL\][https://arxiv\.org/abs/2505\.09388](https://arxiv.org/abs/2505.09388)
- Team et al\.\(2025\)V Team, Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, Shuaiqi Duan, Weihan Wang, Yan Wang, Yean Cheng, Zehai He, Zhe Su, Zhen Yang, Ziyang Pan, Aohan Zeng, Baoxu Wang, Bin Chen, Boyan Shi, Changyu Pang, Chenhui Zhang, Da Yin, Fan Yang, Guoqing Chen, Jiazheng Xu, Jiale Zhu, Jiali Chen, Jing Chen, Jinhao Chen, Jinghao Lin, Jinjiang Wang, Junjie Chen, Leqi Lei, Letian Gong, Leyi Pan, Mingdao Liu, Mingde Xu, Mingzhi Zhang, Qinkai Zheng, Sheng Yang, Shi Zhong, Shiyu Huang, Shuyuan Zhao, Siyan Xue, Shangqin Tu, Shengbiao Meng, Tianshu Zhang, Tianwei Luo, Tianxiang Hao, Tianyu Tong, Wenkai Li, Wei Jia, Xiao Liu, Xiaohan Zhang, Xin Lyu, Xinyue Fan, Xuancheng Huang, Yanling Wang, Yadong Xue, Yanfeng Wang, Yanzi Wang, Yifan An, Yifan Du, Yiming Shi, Yiheng Huang, Yilin Niu, Yuan Wang, Yuanchang Yue, Yuchen Li, Yutao Zhang, Yuting Wang, Yu Wang, Yuxuan Zhang, Zhao Xue, Zhenyu Hou, Zhengxiao Du, Zihan Wang, Peng Zhang, Debing Liu, Bin Xu, Juanzi Li, Minlie Huang, Yuxiao Dong, and Jie Tang\. 2025\.GLM\-4\.5V and GLM\-4\.1V\-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning\.arXiv:2507\.01006 \[cs\.CV\][https://arxiv\.org/abs/2507\.01006](https://arxiv.org/abs/2507.01006)
- Wang et al\.\(2025\)Jiahui Wang, Zuyan Liu, Yongming Rao, and Jiwen Lu\. 2025\.SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMs\.*arXiv preprint arXiv:2506\.05344*\(2025\)\.
- Wang et al\.\(2024\)Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Shiyu Huang, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang\. 2024\.LVBench: An Extreme Long Video Understanding Benchmark\.arXiv:2406\.08035 \[cs\.CV\]
- Wang et al\.\(2023\)Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou\. 2023\.Self\-Consistency Improves Chain of Thought Reasoning in Language Models\.arXiv:2203\.11171 \[cs\.CL\][https://arxiv\.org/abs/2203\.11171](https://arxiv.org/abs/2203.11171)
- Wang et al\.\(2026b\)Zikang Wang, Boyu Chen, Zhengrong Yue, Yi Wang, Yu Qiao, Limin Wang, and Yali Wang\. 2026b\.VideoChat\-A1: Thinking with Long Videos by Chain\-of\-Shot Reasoning\.arXiv:2506\.06097 \[cs\.CV\][https://arxiv\.org/abs/2506\.06097](https://arxiv.org/abs/2506.06097)
- Wang et al\.\(2026a\)Zheng Wang, Haoran Chen, Haoxuan Qin, Zhipeng Wei, Tianwen Qian, and Cong Bai\. 2026a\.Think, Then Verify: A Hypothesis\-Verification Multi\-Agent Framework for Long Video Understanding\.arXiv:2603\.04977 \[cs\.CV\][https://arxiv\.org/abs/2603\.04977](https://arxiv.org/abs/2603.04977)
- Wu et al\.\(2024a\)Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li\. 2024a\.LongVideoBench: A Benchmark for Long\-context Interleaved Video\-Language Understanding\.arXiv:2407\.15754 \[cs\.CV\][https://arxiv\.org/abs/2407\.15754](https://arxiv.org/abs/2407.15754)
- Wu et al\.\(2024b\)Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu\. 2024b\.Retrieval Head Mechanistically Explains Long\-Context Factuality\.arXiv:2404\.15574 \[cs\.CL\][https://arxiv\.org/abs/2404\.15574](https://arxiv.org/abs/2404.15574)
- Xiao et al\.\(2024b\)Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun\. 2024b\.InfLLM: Training\-Free Long\-Context Extrapolation for LLMs with an Efficient Context Memory\.arXiv:2402\.04617 \[cs\.CL\][https://arxiv\.org/abs/2402\.04617](https://arxiv.org/abs/2402.04617)
- Xiao et al\.\(2024a\)Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han\. 2024a\.DuoAttention: Efficient Long\-Context LLM Inference with Retrieval and Streaming Heads\.arXiv:2410\.10819 \[cs\.CL\][https://arxiv\.org/abs/2410\.10819](https://arxiv.org/abs/2410.10819)
- Yan et al\.\(2026\)Haiyang Yan, Hongyun Zhou, Peng Xu, Xiaoxue Feng, and Mengyi Liu\. 2026\.Symphony: A Cognitively\-Inspired Multi\-Agent System for Long\-Video Understanding\.arXiv:2603\.17307 \[cs\.CV\][https://arxiv\.org/abs/2603\.17307](https://arxiv.org/abs/2603.17307)
- Yang et al\.\(2024\)Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia\. 2024\.VisionZip: Longer is Better but Not Necessary in Vision Language Models\.*arXiv preprint arXiv:2412\.04467*\(2024\)\.
- Zhang et al\.\(2024b\)Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra\. 2024b\.Draft& verify: Lossless large language model acceleration via self\-speculative decoding\. In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*\. 11263–11282\.
- Zhang et al\.\(2024a\)Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu\. 2024a\.LMMs\-Eval: Reality Check on the Evaluation of Large Multimodal Models\.arXiv:2407\.12772 \[cs\.CL\][https://arxiv\.org/abs/2407\.12772](https://arxiv.org/abs/2407.12772)
- Zhang et al\.\(2025\)Yuan Zhang, Chun\-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, et al\.2025\.SparseVLM: Visual Token Sparsification for Efficient Vision\-Language Model Inference\. In*International Conference on Machine Learning*\.
- zhang et al\.\(2025\)Yunzhu zhang, Yu Lu, Tianyi Wang, Fengyun Rao, Yi Yang, and Linchao Zhu\. 2025\.FlexSelect: Flexible Token Selection for Efficient Long Video Understanding\. In*The Thirty\-ninth Annual Conference on Neural Information Processing Systems*\.[https://openreview\.net/forum?id=0D3ja9s17M](https://openreview.net/forum?id=0D3ja9s17M)
- Zhang et al\.\(2023\)Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen\. 2023\.H2O: Heavy\-Hitter Oracle for Efficient Generative Inference of Large Language Models\.[https://arxiv\.org/abs/2306\.14048](https://arxiv.org/abs/2306.14048)
- Zohar et al\.\(2024\)Orr Zohar, Xiaohan Wang, Yann Dubois, Nikhil Mehta, Tong Xiao, Philippe Hansen\-Estruch, Licheng Yu, Xiaofang Wang, Felix Juefei\-Xu, Ning Zhang, Serena Yeung\-Levy, and Xide Xia\. 2024\.Apollo: An Exploration of Video Understanding in Large Multimodal Models\.arXiv:2412\.10360 \[cs\.CV\][https://arxiv\.org/abs/2412\.10360](https://arxiv.org/abs/2412.10360)

## Appendix AAdditional Related Works\.

MLLMs typically encode visual inputs as discrete visual tokens\. Early approaches were restricted to processing videos at fixed resolutions, such as InternVL\([Chen et al\., 2024a](https://arxiv.org/html/2609.16722#bib.bib9)\)and LLaVA\-NeXT\([Liu et al\., 2024](https://arxiv.org/html/2609.16722#bib.bib21)\), which hinders the comprehension of content containing diverse semantic details\. Most recent advancements support native dynamic resolution, enabling direct, patch\-wise partitioning of inputs at arbitrary resolutions\. Indeed, this capability has emerged as the prevailing paradigm in contemporary MLLMs\. Leading open\-source mllms, including Qwen3\-VL\([Team, 2025](https://arxiv.org/html/2609.16722#bib.bib27)\)and GLM\-4V\([Team et al\., 2025](https://arxiv.org/html/2609.16722#bib.bib28)\), have adopted this paradigm, enabling direct patch\-wise tokenization of visual inputs at arbitrary resolutions\. This capability of mllms to handle videos at arbitrary native resolutions serves as the foundation for our VideoMM\. Leveraging this flexibility, VideoMM adaptively modulates video resolution to coordinate video inference on a coarse\-to\-fine basis, enhancing comprehensive video understanding\.

Beyond token reduction, a significant body of work addresses long\-context LLM inference through complementary mechanisms that can be synergistically integrated with our approach\.

KV Cache Evictionaccelerates the decoding process by selectively pruning internally retained KV caches\. Predominantly operating in long text scenarios, these methods optimize compression by assessing cache importance\([Zhang et al\., 2023](https://arxiv.org/html/2609.16722#bib.bib44);[Li et al\., 2024](https://arxiv.org/html/2609.16722#bib.bib18);[Feng et al\., 2025b](https://arxiv.org/html/2609.16722#bib.bib12);[Feng et al\., 2025a](https://arxiv.org/html/2609.16722#bib.bib10)\)or employing dynamic budget allocation\([Cai et al\., 2024](https://arxiv.org/html/2609.16722#bib.bib5);[Feng et al\., 2024](https://arxiv.org/html/2609.16722#bib.bib11);[Fu et al\., 2024](https://arxiv.org/html/2609.16722#bib.bib14)\)\. Recent researches have been extended to MLLMs: notably, SparseMM\([Wang et al\., 2025](https://arxiv.org/html/2609.16722#bib.bib29)\)incorporates inter\-head allocation into multimodal cache management\. However, since these methods compress the cache only after processing all tokens to alleviate memory pressure, they do not reduce the token count itself\. Consequently, they are orthogonal to token reduction strategies\.

Sparse attention methods\([Xiao et al\., 2024b](https://arxiv.org/html/2609.16722#bib.bib36);[Tang et al\., 2024](https://arxiv.org/html/2609.16722#bib.bib24);[Jiang et al\., 2024](https://arxiv.org/html/2609.16722#bib.bib16);[Li et al\., 2025](https://arxiv.org/html/2609.16722#bib.bib19)\)accelerate inference by retaining the full KV cache while selecting only a critical subset for computation\. However, as these approaches do not lower the number of processed tokens, they remain orthogonal to token reduction methods\. Future work could combine lightweight token reduction with sparse attention to further minimize inference overhead\.

Speculative decoding\([Leviathan et al\., 2023](https://arxiv.org/html/2609.16722#bib.bib17);[Chen et al\., 2023](https://arxiv.org/html/2609.16722#bib.bib7);[Zhang et al\., 2024b](https://arxiv.org/html/2609.16722#bib.bib40)\)accelerates inference by utilizing a small model to draft outputs, which are then verified by a larger model, thereby improving decoding efficiency through collaborative inference\. This technique has recently been extended to MLLMs\. For example, SpecVLM\([Ji et al\., 2025](https://arxiv.org/html/2609.16722#bib.bib15)\)enhances decoding efficiency by compressing visual tokens and passing them to a smaller model for drafting, followed by verification from a larger model\. This demonstrates that visual token reduction is orthogonal to the speculative decoding paradigm\. Future work could explore integrating these two approaches more closely\.

Table 6\.Detailed performance comparison on LongVideoBench\. The best two results arebolded\.Table 7\.Detailed performance comparison on LVBench\. The best two results arebolded\.Table 8\.Detailed performance comparison on VideoMME\. The best two results arebolded\.Agent\-based Video Understanding\([Chen et al\., 2025](https://arxiv.org/html/2609.16722#bib.bib6);[Wang et al\., 2026b](https://arxiv.org/html/2609.16722#bib.bib32);[Wang et al\., 2026a](https://arxiv.org/html/2609.16722#bib.bib33);[Yan et al\., 2026](https://arxiv.org/html/2609.16722#bib.bib38)\)tackles long\-video redundancy at a semantic level through interactive reasoning\. LVAgent\([Chen et al\., 2025](https://arxiv.org/html/2609.16722#bib.bib6)\)introduces a multi\-round dynamic collaboration framework among multiple MLLM agents, filtering out suboptimal reasoning through iterative selection, perception, action, and reflection\. VideoChat\-A1\([Wang et al\., 2026b](https://arxiv.org/html/2609.16722#bib.bib32)\)emphasizes the inherent shot\-based structure of videos by proposing a Chain\-of\-Shot reasoning paradigm\. It progressively selects relevant shots, partitions them into fine\-grained subshots via feature clustering, and evaluates reasoning confidence to iteratively refine the temporal context\. VideoHV\-Agent\([Wang et al\., 2026a](https://arxiv.org/html/2609.16722#bib.bib33)\)reformulates long\-video question answering as a hypothesis–verification process: a Thinker converts candidate answers into testable hypotheses, a Judge derives discriminative clues, and a Verifier localizes and examines targeted video evidence before an Answer agent integrates the results\. Symphony\([Yan et al\., 2026](https://arxiv.org/html/2609.16722#bib.bib38)\)employs a cognitively inspired multi\-agent architecture that coordinates task decomposition, evidence grounding, visual perception, and reflection for long\-video reasoning\. Future work could explore synergizing agent\-based semantic exploration with visual token reduction methods for optimal long\-video inference efficiency\.

## Appendix BBenchmark Details\.

Below, we provide a detailed overview of the video understanding benchmarks utilized in the experiments\.

1. \(1\)LongVideoBench\([Wu et al\., 2024a](https://arxiv.org/html/2609.16722#bib.bib34)\)is a comprehensive video question\-answering benchmark comprising 3,763 web\-collected videos \(up to one hour\) with 6,678 human\-annotated multiple\-choice questions across 17 fine\-grained categories\. It introduces a referring reasoning task where models must retrieve and reason over relevant multimodal information from specified video contexts\.
2. \(2\)VideoMME\([Fu et al\., 2025](https://arxiv.org/html/2609.16722#bib.bib13)\)is a full\-spectrum multi\-modal evaluation benchmark for MLLMs comprising 900 videos \(254 hours\) with 2,700 question\-answer pairs\. It spans 6 visual domains, covers short\- to long\-form videos \(11 seconds to 1 hour\), and includes multi\-modal inputs \(frames, subtitles, audio\)\.
3. \(3\)LVBench\([Wang et al\., 2024](https://arxiv.org/html/2609.16722#bib.bib30)\)is a benchmark specifically designed for long video understanding, created to address the gap in evaluating MLLMs for real\-world applications requiring comprehension of videos spanning several hours\. It comprises publicly sourced videos and includes diverse tasks focused on long video comprehension and information extraction\.

## Appendix CModel Details

Our evaluation employs three models, all of which support dynamic resolution:

- •Qwen2\.5\-VL\-7B\-Instruct\([Bai et al\., 2025](https://arxiv.org/html/2609.16722#bib.bib3)\)builds upon the Qwen2\-VL architecture, introducing significant advancements in visual\-language understanding\. Key enhancements include superior comprehension of complex visual elements such as text, charts, and layouts, alongside agentic capabilities for dynamic tool direction\. Architecturally, it extends dynamic resolution to the temporal dimension through dynamic Frame Per Second sampling, complemented by mRoPE updates that facilitate learning of temporal sequences and precise event identification within videos\.
- •GLM4\.1V\-9B\-Thinking\([Team et al\., 2025](https://arxiv.org/html/2609.16722#bib.bib28)\): is a powerful Mllm designed for advanced general\-purpose multimodal reasoning\. At its core is a reasoning\-centric training framework that leverages a robust vision foundation model, initially established through large\-scale pre\-training\. This foundation’s full potential is subsequently unlocked via Reinforcement Learning, significantly enhancing the model’s comprehensive capabilities\. It also supports dynamic\-resolution video inputs\.
- •Qwen3\-VL\-8B\-Instruct\([Team, 2025](https://arxiv.org/html/2609.16722#bib.bib27)\)stands as the most powerful lite mllm in the Qwen series to date, featuring comprehensive upgrades\. It incorporates a novel Interleaved\-MRoPE, which provides full\-frequency allocation across time, width, and height through robust positional embeddings, thereby enhancing long\-horizon video reasoning\. Additionally, it employs Text–Timestamp Alignment, moving beyond T\-RoPE to achieve precise, timestamp\-grounded event localization for stronger video temporal modeling\.

Table 9\.Impact of Lite Models on LongVideoBench based on Qwen3\-VL\-8B\. Acc\.: Accuracy \(%\); Thpt\.: Throughput \(samples/min\)\)
## Appendix DAccelerating Token Compression Using Lightweight MLLMs

While VideoMM primarily focuses on optimizing representation granularity, its framework is orthogonal to—and can be effectively combined with—model\-centric downsizing strategies\. In such a configuration, a lightweight model with fewer parameters conducts the initial token selection, streamlining the process before a larger MLLM performs the final inference\. We try a direct integration\. To demonstrate this flexibility, we instantiate a variant namedVideoMM\-Lite, where we directly employ the off\-the\-shelf Qwen3\-VL\-4B for token selection and the Qwen3\-VL\-8B for the final inference, avoiding additional training overhead888Although the distillation\-based specialization strategy employed by FlexSelect\([zhang et al\., 2025](https://arxiv.org/html/2609.16722#bib.bib43)\)is equally applicable to our framework, we leave such optimization for future work to avoid additional training overhead, maintaining our primary focus on the efficacy of data granularity paradigm\.\. As shown in Table[9](https://arxiv.org/html/2609.16722#A3.T9), compared to the standard VideoMM, VideoMM\-Lite increases throughput from 3\.69 to 4\.19 samples/mins, with only a marginal accuracy dip from 67\.54% to 66\.34%\. This result confirms that integrating a smaller parameter model for selection is a viable pathway to further enhance throughput with minimal impact on performance\.

Table 10\.Impact of multi\-stages of VideoMM on LongVideoBench\.Accuracy represents the percentage of correct predictions, while Throughout denotes the number of samples processed per second\.Table 11\.Comparison of VideoMM and FlexSelect on VideoMME using Qwen\-VL modelsAcc\.: Accuracy \(%\); Thpt\.: Throughput \(samples/min\)
## Appendix EThree\-Stage Micro\-Micro Exploration\.

To further demonstrate the extensibility of our framework, we introduceVideoMM\-multi, a variant that expands the original two\-stage design into a three\-stage cascade\. This is achieved by inserting an additional, coarser\-grained hierarchical level prior to the standard Macro Proxy\. Specifically, we implement anUltra\-Macro Proxywith an aggressive downsampling factor ofk2k^\{2\}\(compared to the standard factorkk\), designed to filter out the most obvious non\-essential regions with extreme efficiency before passing the remaining candidates to the Macro level\.We evaluated this three\-stage design \(k=2k=2\) using Qwen3\-VL\-8B on LongVideoBench\. As shown in Table[10](https://arxiv.org/html/2609.16722#A4.T10), the comparison reveals a distinct trade\-off between granular precision and computational speed\. The standard two\-stage VideoMM maintains superior semantic retention, achieving a higher accuracy of 67\.54 compared to VideoMM\-multi’s 64\.55\. Conversely, by offloading the majority of workload to the new Ultra\-Macro tier, VideoMM\-multi drastically reduces the computational burden, increasing throughput to 5\.91 samples/min—significantly outperforming the standard VideoMM’s 3\.69 samples/min\. These results confirm that dynamically adjusting the depth of the macro\-micro hierarchy serves as a powerful lever for tuning the accuracy\-efficiency trade\-off\.