Reasoning with Memory: A Temporal Granularity-Adaptive Framework for Training-Free Long Video Understanding

# Reasoning with Memory: A Temporal Granularity-Adaptive Framework for Training-Free Long Video Understanding
Source: [https://arxiv.org/html/2607.24794](https://arxiv.org/html/2607.24794)
11institutetext:National University of Singapore, Singapore
11email:menglinghao25@u\.nus\.edu; ymjin@nus\.edu\.sg22institutetext:Institute of High Performance Computing, Agency for Science, Technology and Research \(A\*STAR\), Singapore
22email:hzfu@ieee\.org33institutetext:Nanyang Technological University, Singapore
33email:cs\-qiankun\.li@ntu\.edu\.sg44institutetext:University of Science and Technology of China, Anhui, China55institutetext:Jilin University, Jilin, China
Qiankun Li\*Junyuan MaoPujin LiaoZhicheng HeEnbo ZhangKun WangYang LiuHuazhu Fu\(✉\)Yueming Jin\(✉\)

###### Abstract

While Multimodal Large Language Models \(MLLMs\) demonstrate superior generalization in fundamental video tasks, restricted context windows limit their long video understanding\. To accommodate this constraint, models typically resort to keyframe selection\. However, uniform sampling or static query\-guided selection often overlooks critical temporal context, failing to adapt to the varying query temporal granularities\. In this paper, we proposeReMem, a temporal granularity\-adaptive keyframe selection framework for training\-free LongVideoQA\. ReMem introduces adual\-levelmemory\-augmented adaptation\. At the query level,Memory\-Driven Question Parsingleverages LLM long\-term memory to decode question temporal granularity and extract semantic entities\. At the video level,Synergistic Dual\-Semantic Frame Alignmentexploits intrinsic structural memory to align frames with query semantics, guidingStructure\-Aware Dynamic Frame Routingto cluster events and optimally distribute sampling budgets\. By explicitly preserving temporal information with memory mechanisms, ReMem suppresses redundancy and empowers MLLMs to perform robust multi\-granular video reasoning\. Evaluations across four popular LongVideoQA benchmarks using three MLLMs demonstrate highly efficient, state\-of\-the\-art zero\-shot performance; notably, LLaVA\-Video with ReMem reaches 54\.5% \(\+12\.3%\) on LVBench and 67\.1% \(\+8\.2%\) on LongVideoBench\. Code is available at[ReMem](https://github.com/jinlab-imvr/ReMem)\.

††footnotetext:\*Equal contribution\.
✉Corresponding authors\.## 1Introduction

Recently, Multimodal Large Language Models \(MLLMs\) have achieved remarkable breakthroughs in cross\-modal reasoning by aligning visual encoders with LLMs\[liu2023visual,zhu2023minigpt,bai2023qwen\]\. By integrating temporal modeling, this architecture extends to the video domain, adapting static perception for dynamic understanding\[lin2024video,maaz2024video,li2025videochat,li2024llama,zhang2026bimm\]\. Consequently, MLLMs inherit robust instruction\-following capabilities and demonstrate superior generalization across fundamental video tasks\[jin2024chat,zhang2025collaboratively\]\.

Despite these advancements, standard MLLMs struggle with long video understanding tasks\[fu2025video,zhou2024mlvu,wang2025lvbench\]\. The extensive frame counts exceed restricted context windows, precluding the processing of all frames\[zhang2024long,song2024moviechat,zhang2025q\]\. Consequently, models often resort to sparse uniform sampling, which inadvertently omits crucial evidence and causes erroneous predictions\[wang2023internvid,tang2025adaptive\]\. To address this, recent approaches focus on selecting the most discriminative keyframes within a limited frame budget\[zhang2024simple,tang2025adaptive,li2025frameoracle,yu2023self\]\.

![Refer to caption](https://arxiv.org/html/2607.24794v1/intro.png)Figure 1:Query\-adaptive methods evaluate frames in isolation and suffer from the absence of temporal understanding\. OurReMemovercomes this bottleneck with memory\-augmented temporal granularity\-adaptive sampling, achieving superior accuracy and efficiency for long video reasoning\.However, current methods typically rely on static query\-frame similarity for keyframe selection\[ren2024timechat,tang2025adaptive,zhang2025q\], failing to adapt to the varying temporal granularities of different questions\. As demonstrated in Fig\.[1](https://arxiv.org/html/2607.24794#S1.F1), purely query\-adaptive sampling tends to over\-sample redundant frames from localized high\-scoring segments\. Consequently, while adequate for fine\-grained queries, these approaches overlook essential temporal dependencies, resulting in suboptimal performance on tasks requiring continuous temporal reasoning\. Moreover, existing methods overlook the latent concepts embedded in questions, failing to enhance the textual signal\-to\-noise ratio for effective query\-frame alignment\.

To address these limitations in a training\-free setting, the concept ofmemoryserves as a dual\-faceted mechanism to process and align both the query and the video\. For the query, the inherentlong\-term memoryof LLMs encapsulates rich cognitive priors, providing the intuition to analyze temporal granularity and make semantic associations\[achiam2023gpt\]\. For the visual content, mining the video’s inherentstructural memorycaptures its continuous temporal evolution, effectively bridging the contextual gap between scattered frames\. Building upon this insight, we proposeReMem\(Reasoning withMemory\), a temporal granularity\-adaptive training\-free keyframe selection framework for zero\-shot LongVideoQA augmented by a memory mechanism\.

ReMem formulates a memory\-augmented dual\-level adaptation framework, aligning question semantics with video visual and temporal dynamics\. At thequery level, a Reasoning LLM utilizes long\-term memory to conductMemory\-Driven Question Parsing, evaluating the query temporal granularity and extracting key entities to guide visual retrieval\. At thevideo level,Synergistic Dual\-Semantic Frame Alignmentaligns frames with query semantics, guidingStructure\-Aware Dynamic Frame Routingto cluster events and distribute budgets\. This memory\-augmented strategy suppresses temporal redundancy and preserves causal structures, empowering MLLMs to perform robust multi\-granular video reasoning\.

We evaluate ReMem using three MLLMs across four LongVideoQA benchmarks, where it achieves state\-of\-the\-art zero\-shot performance with high efficiency\. On LVBench\[wang2025lvbench\]and LongVideoBench\[wu2024longvideobench\], LLaVA\-Video with ReMem reaches accuracies of 54\.5%\(\+12\.3%\) and 67\.1%\(\+8\.2%\), respectively\. These substantial gains are consistently observed across all evaluated MLLMs, highlighting ReMem as a generalizable framework that requires no parameter tuning\.

The main contributions of this work are summarized as follows:

- ❶We proposeReMem, a training\-free framework that performs temporal granularity\-adaptive keyframe selection for long video understanding, enabling MLLMs to reason over long videos under strict context constraints\.
- ❷We introduce amemory\-driven dual\-level adaptation mechanismthat leverages LLM long\-term memory to parse query temporal granularity and employs structural video memory to align frames through structure\-aware dynamic routing\.
- ❸Extensive experiments across three MLLM architectures and four popular LongVideoQA benchmarks demonstrate that ReMem achieves new state\-of\-the\-art zero\-shot performance\.

## 2Related Work

### 2\.1Video Large Language Models

Recent Video Large Language Models have witnessed significant advancements in visual\-text alignment and spatial\-temporal modeling\[zhang2024llava,bai2025qwen3,li2024llava,liu2024llavanext,bai2025qwen2,wang2024internvideo2,zhang2025collaboratively,zhang2026bimm\]\. For instance, LLaVA\-OneVision\[li2024llava\]and LLaVA\-NeXT\-QW2\[liu2024llavanext\]introduce AnyRes strategies and dynamic visual pooling to effectively process high\-resolution frames across diverse scenarios\. VideoLLaMA3\[zhang2025videollama\]further strengthens spatial\-temporal representation through an advanced visual\-language architecture, while Chat\-UniVi\-V1\.5\[jin2024chat\]employs dynamic token merging to optimize computational efficiency without compromising performance\.

To address the computational challenges of Long\-Video Understanding, specialized architectures have emerged to decouple sequence length from memory cost\. LongVA\[zhang2024long\]and LongVILA\[chen2024longvila\]significantly extend the context window by incorporating efficient spatial\-temporal pooling and system\-level training optimizations\. Video\-XL\[shu2025video\]compresses hour\-scale videos into manageable KV caches utilizing latent visual summarization tokens\. Similarly, LongVU\[shen2024longvu\]and Video\-CCAM\[fei2024video\]introduce spatiotemporal context compression and causal cross\-attention masking, effectively mitigating the heavy memory burden\. Furthermore, Video\-R1\[feng2025video\]and VideoChat\-R1\[li2025videochat\-r1\]integrate deductive reasoning traces into video comprehension, generating intermediate logical steps and enabling MLLMs to ground with complex temporal evidence and structured visual cues\.

### 2\.2Keyframe Selection

The computational bottleneck of processing dense video tokens motivates the development of Keyframe Selection\[zhao2023search,zhu2025focus,guo2025logic,tang2025adaptive,song2026ktv\], which aims to identify the most informative frames to represent the video content\. These methods can be broadly categorized into training\-based and training\-free approaches\.

Training\-based methods typically learn dedicated modules to extract relevant evidence \(e\.g\., Frame\-Voyager\[yu2024frame\], KeyVideoLLM\[liang2024keyvideollm\], FrameOracle\[li2025frameoracle\]\), or fine\-tune lightweight LLMs as plug\-and\-play filters, such as Selector\[hu2025m\]\. Alternatively, recent works like CoF\[ghazanfari2025chain\]and GenS\[yao2025generative\]integrate selection directly into the reasoning process by training Video\-LLMs to generate temporal traces\.

Training\-free methods offer plug\-and\-play compatibility with off\-the\-shelf models, transitioning from static uniform sampling to query\-adaptive strategies\. For instance, AKS\[tang2025adaptive\]optimizes the balance between relevance and coverage, while Q\-Frame\[zhang2025q\]and FlexSelect\[zhang2025flexselect\]dynamically adjust visual budgets through multi\-resolution and context\-aware probabilistic sampling respectively\. Furthermore, BOLT\[liu2025bolt\]employs dynamic routing to match evidence with intent, and MDP3\[sun2025mdp3\]utilizes determinantal point processes for list\-wise diversity\. However, these approaches predominantly evaluate frames based on isolated static semantics, ignoring the continuous temporal dynamics and long\-range causal dependencies required for complex long video reasoning\.

## 3Method

We propose ReMem, a training\-free framework that adaptively selects the most informative frames for zero\-shot Long Video Question Answering \(LongVideoQA\)\. As shown in Fig\.[2](https://arxiv.org/html/2607.24794#S3.F2), ReMem performs a comprehensive dual\-level memory adaptation for long video understanding: ❶query\-level adaptation, where a reasoning LLM utilizes long\-term memory to evaluate the question’s granularity and extract key entities; ❷video\-level adaptation, where keyframes are selected by memory\-augmented visual evidence distillation\. The overall pipeline consists of three core modules:Memory\-Driven Question Parsing\(⊳\\trianglerightSection[3\.1](https://arxiv.org/html/2607.24794#S3.SS1)\),Synergistic Dual\-Semantic Frame Alignment\(⊳\\trianglerightSection[3\.2](https://arxiv.org/html/2607.24794#S3.SS2)\), andStructure\-Aware Dynamic Frame Routing\(⊳\\trianglerightSection[3\.3](https://arxiv.org/html/2607.24794#S3.SS3)\)\.

![Refer to caption](https://arxiv.org/html/2607.24794v1/method.png)Figure 2:Overview of ReMem\.ReMem performs dual\-level adaptation for training\-free LongVideoQA: ❶query\-level adaptation, where an LLM evaluates the question’s temporal granularity and extracts entities; ❷video\-level adaptation, where memory\-augmented dual\-semantic evidence assists in selecting keyframes\. The pipeline consists of three modules: Memory\-Driven Question Parsing, Synergistic Dual\-Semantic Frame Alignment, and Structure\-Aware Dynamic Frame Routing\.### 3\.1Memory\-Driven Question Parsing

#### Granularity Analysis\.

Previous training\-free LongVideoQA approaches often select informative frames without considering the varying complex temporal granularity in question semantics\. To address this, we formulate granularity analysis as a continuous reasoning processfgranularity:Q→gf\_\{\\text\{granularity\}\}:Q\\to g, where an LLM draws upon itslong\-term memoryto score the queryQQ, yieldingg∈\[0,1\]g\\in\[0,1\]that quantifies its temporal granularity\. Specifically, a lowerggdenotes localized events requiring dense visual evidence from a narrow window, while a higherggreflects overarching queries demanding long\-range contextual integration \(*e\.g*\., “How many times is the sun visible in the video?”→g=0\.90\\to g=0\.90\)\.

#### Entity Extraction\.

Beyond temporal granularity, raw queries often lack explicit visual cues for precise alignment\. To bridge this gap, we formulate entity extraction asfentity:Q,\{Ci\}→\{Ei\}f\_\{\\text\{entity\}\}:Q,\\\{C\_\{i\}\\\}\\to\\\{E\_\{i\}\\\}, where the LLM leverages itslong\-term memoryto extract concrete visual signatures from the questionQQand candidate options\{Ci\}\\\{C\_\{i\}\\\}\(*e\.g*\., “How many Mbappe’s spectacular goals does the video show?”,\{“6”,“5”,“4”,“3”\}→\{\\\{\\text\{\`\`6''\},\\text\{\`\`5''\},\\text\{\`\`4''\},\\text\{\`\`3''\}\\\}\\to\\\{“Mbappe scoring celebration”, “soccer net shaking”, “Mbappe’s jersey number”\}\\\}\)\.

### 3\.2Synergistic Dual\-Semantic Frame Alignment

#### Query Enhancement\.

We extract the CLIP text embeddings𝒗q∈ℝD\\bm\{v\}\_\{q\}\\\!\\in\\\!\\mathbb\{R\}^\{D\}and𝒗e∈ℝN×D\\bm\{v\}\_\{e\}\\\!\\in\\\!\\mathbb\{R\}^\{N\\times D\}for the questionQQand extracted entities\{Ei\}\\\{E\_\{i\}\\\}, respectively\. We dynamically inject𝒗e\\bm\{v\}\_\{e\}into𝒗q\\bm\{v\}\_\{q\}via a cross\-attention residual stream:

𝒗a​n​c=LN​\(𝒗q\+𝑾f​\[𝒗q∥Softmax​\(\(𝑾q​𝒗q\)​\(𝑾k​𝒗e\)⊤D\)​\(𝑾v​𝒗e\)\]\),\\bm\{v\}\_\{anc\}=\\text\{LN\}\\left\(\\bm\{v\}\_\{q\}\+\\bm\{W\}\_\{f\}\\left\[\\bm\{v\}\_\{q\}\\parallel\\text\{Softmax\}\\left\(\\frac\{\(\\bm\{W\}\_\{q\}\\bm\{v\}\_\{q\}\)\(\\bm\{W\}\_\{k\}\\bm\{v\}\_\{e\}\)^\{\\top\}\}\{\\sqrt\{D\}\}\\right\)\(\\bm\{W\}\_\{v\}\\bm\{v\}\_\{e\}\)\\right\]\\right\),\(1\)whereLN​\(⋅\)\\text\{LN\}\(\\cdot\)denotes Layer Normalization,𝑾q,𝑾k\\bm\{W\}\_\{q\},\\bm\{W\}\_\{k\}, and𝑾v∈ℝD×D\\bm\{W\}\_\{v\}\\in\\mathbb\{R\}^\{D\\times D\}are projection matrices,\[⋅∥⋅\]\[\\cdot\\parallel\\cdot\]denotes the concatenation operation along the feature dimension, and𝑾f∈ℝ2​D×D\\bm\{W\}\_\{f\}\\in\\mathbb\{R\}^\{2D\\times D\}is the fusion weight\. This yields𝒗a​n​c\\bm\{v\}\_\{anc\}, a visually\-grounded enhanced query anchor that boasts a significantly improved signal\-to\-noise ratio for subsequent frame retrieval\.

#### Static Visual\-Semantic Alignment\.

Letℐ=\{Ii\}i=1M\\mathcal\{I\}=\\\{I\_\{i\}\\\}\_\{i=1\}^\{M\}denote the sequence of frames uniformly sampled from the raw video\. For each frameIiI\_\{i\}, we extract the CLIP visual embedding𝒗i∈ℝD\\bm\{v\}\_\{i\}\\in\\mathbb\{R\}^\{D\}\. To measure the semantic affinity while controlling scale and higher\-order correlation, ReMem defines the static visual\-semantic similarity as:

𝒮v​\(𝒗i,𝒗a​n​c\)=α​\(𝑾s​𝒗a​n​c\)⊤​\(𝑾s​𝒗i\)‖𝑾s​𝒗a​n​c‖2​‖𝑾s​𝒗i‖2\+β​\(𝒗a​n​c⊤​𝑾c​𝒗iD\)2,\\mathcal\{S\}\_\{\\text\{v\}\}\(\\bm\{v\}\_\{i\},\\bm\{v\}\_\{anc\}\)=\\alpha\\frac\{\(\\bm\{W\}\_\{s\}\\bm\{v\}\_\{anc\}\)^\{\\top\}\(\\bm\{W\}\_\{s\}\\bm\{v\}\_\{i\}\)\}\{\\\|\\bm\{W\}\_\{s\}\\bm\{v\}\_\{anc\}\\\|\_\{2\}\\,\\\|\\bm\{W\}\_\{s\}\\bm\{v\}\_\{i\}\\\|\_\{2\}\}\+\\beta\\left\(\\frac\{\\bm\{v\}\_\{anc\}^\{\\top\}\\bm\{W\}\_\{c\}\\bm\{v\}\_\{i\}\}\{\\sqrt\{D\}\}\\right\)^\{2\},\(2\)where𝑾s,𝑾c∈ℝD×D\\bm\{W\}\_\{s\},\\bm\{W\}\_\{c\}\\in\\mathbb\{R\}^\{D\\times D\}denote the projection weight matrices inherited from the pre\-trained CLIP model,𝑾s\\bm\{W\}\_\{s\}serves as a scale\-controlling projection to alleviate the modality gap, andα,β\\alpha,\\betaare weighting coefficients governing the contribution of each term\.

#### Memory Augmented Temporal\-Semantic Alignment\.

To effectively capture cross\-scene correlations and long\-range dependencies, we construct a temporal\-semantic memory graph𝒢=\(𝒱,ℰ\)\\mathcal\{G\}=\(\\mathcal\{V\},\\mathcal\{E\}\)\. We define the node set𝒱\\mathcal\{V\}to represent core visual anchor states, dynamically setting its sizeK=MK=\\sqrt\{M\}for sub\-linear scaling\. Specifically, we partition the visual features\{𝒗i\}i=1M\\\{\\bm\{v\}\_\{i\}\\\}\_\{i=1\}^\{M\}intoKKclusters\{Ck\}k=1K\\\{C\_\{k\}\\\}\_\{k=1\}^\{K\}with centroids\{𝝁k\}k=1K\\\{\\bm\{\\mu\}\_\{k\}\\\}\_\{k=1\}^\{K\}via K\-Means\. To avoid abstract mean representations, each anchor is instantiated by the actual frame closest to its centroid, i\.e\.,𝑽k=arg⁡min𝒗i∈Ck⁡‖𝒗i−𝝁k‖2\\bm\{V\}\_\{k\}=\\arg\\min\_\{\\bm\{v\}\_\{i\}\\in C\_\{k\}\}\\\|\\bm\{v\}\_\{i\}\-\\bm\{\\mu\}\_\{k\}\\\|\_\{2\}\. These instantiated nodes are chronologically ordered and stacked to form the memory feature matrix𝑽=\[𝑽1,…,𝑽K\]⊤∈ℝK×D\\bm\{V\}=\[\\bm\{V\}\_\{1\},\\dots,\\bm\{V\}\_\{K\}\]^\{\\top\}\\in\\mathbb\{R\}^\{K\\times D\}\. Subsequently, we establish the edge setℰ\\mathcal\{E\}by computing semantic weightses​e​me\_\{sem\}and assigning temporal weightset​e​me\_\{tem\}for local connectivity\. The edge construction is mathematically formulated as:

es​e​mi,j=max⁡\(𝑽i⊤​𝑽j,0\),et​e​mi,j=\{1,\|i−j\|≤10,otherwise\.e\_\{sem\}^\{i,j\}=\\max\(\\bm\{V\}\_\{i\}^\{\\top\}\\bm\{V\}\_\{j\},0\),\\quad e\_\{tem\}^\{i,j\}=\\begin\{cases\}1,&\|i\-j\|\\leq 1\\\\ 0,&\\text\{otherwise\}\\end\{cases\}\.\(3\)To ensure stable energy propagation during graph reasoning, we construct the transition matrices𝑾s​e​m,𝑾t​e​m∈ℝK×K\\bm\{W\}\_\{sem\},\\bm\{W\}\_\{tem\}\\in\\mathbb\{R\}^\{K\\times K\}by explicitly applyingL1L\_\{1\}normalization to the raw edge weightses​e​me\_\{sem\}andet​e​me\_\{tem\}, respectively\. For cross\-scene reasoning, the enhanced semantic anchor𝒗a​n​c\\bm\{v\}\_\{anc\}initializes a seed probability distribution𝒑0∈ℝK\\bm\{p\}^\{0\}\\in\\mathbb\{R\}^\{K\}across the anchor nodes, formulated as𝒑\(0\)=Softmax​\(𝑽​𝒗a​n​c\)\\bm\{p\}^\{\(0\)\}=\\text\{Softmax\}\(\\bm\{V\}\\bm\{v\}\_\{anc\}\)\. This activation energy diffuses through the graph via a random walk mechanism:

𝒑t\+1=\(1−γ\)​\(α​𝑾s​e​m\+\(1−α\)​𝑾t​e​m\)​𝒑t\+γ​𝒑0,\\bm\{p\}^\{t\+1\}=\(1\-\\gamma\)\\Big\(\\alpha\\bm\{W\}\_\{sem\}\+\(1\-\\alpha\)\\bm\{W\}\_\{tem\}\\Big\)\\bm\{p\}^\{t\}\+\\gamma\\bm\{p\}^\{0\},\(4\)whereγ∈\(0,1\)\\gamma\\in\(0,1\)is the restart probability governing the diffusion depth, andα\\alphais a coefficient balancing content\-driven attention and temporal inductive bias\. After 2 steps of diffusion, we obtain𝒑2\\bm\{p\}^\{2\}, which captures neighborhood interactions while preserving discriminative local topologies, thereby encoding the global structural importance of each anchor node\. The temporal\-semantic score is derived by:

𝒮t​\(𝒗i,𝒗a​n​c\)=𝔼k∼𝒑2​\[𝒗i⊤​𝑽k\]=∑k=1K\(\(𝒗i⊤​𝑽k\)​pk2\),\\mathcal\{S\}\_\{\\text\{t\}\}\(\\bm\{v\}\_\{i\},\\bm\{v\}\_\{anc\}\)=\\mathbb\{E\}\_\{k\\sim\\bm\{p\}^\{2\}\}\\big\[\\bm\{v\}\_\{i\}^\{\\top\}\\bm\{V\}\_\{k\}\\big\]=\\sum\_\{k=1\}^\{K\}\\left\(\(\\bm\{v\}\_\{i\}^\{\\top\}\\bm\{V\}\_\{k\}\)p\_\{k\}^\{2\}\\right\),\(5\)where𝔼k∼𝒑2\\mathbb\{E\}\_\{k\\sim\\bm\{p\}^\{2\}\}denotes the expectation over the memory nodes guided by the propagated probabilities, andpk2p\_\{k\}^\{2\}denotes thekk\-th scalar element of𝒑2\\bm\{p\}^\{2\}\.

#### Granularity Guided Dual\-Semantic Fusion\.

To comprehensively evaluate frame relevance, we formulate a joint similarity metric𝒮\\mathcal\{S\}that dynamically arbitrates between instantaneous spatial cues and global topological memory:

α=1−0\.5​g​e−λ​t,𝒮=α​𝒮v\+\(1−α\)​𝒮t,\\alpha=1\-0\.5ge^\{\-\\lambda t\},\\quad\\mathcal\{S\}=\\alpha\\mathcal\{S\}\_\{\\text\{v\}\}\+\(1\-\\alpha\)\\mathcal\{S\}\_\{\\text\{t\}\},\(6\)whereλ\\lambdais a decay constant andttdenotes the video duration\. Crucially, the balancing factorα\\alphais modulated by the query’s temporal granularitygg\. A smallerggindicating localized events yields a higherα\\alpha, prioritizing𝒮v\\mathcal\{S\}\_\{\\text\{v\}\}to capture precise spatial evidence without temporal noise\. Conversely, for a largerggor longer video,α\\alphasmoothly decays, elevating𝒮t\\mathcal\{S\}\_\{\\text\{t\}\}to track evolving actions and long\-range structural dependencies\. Ultimately, based on aggregated𝒮\\mathcal\{S\}, we select the topNframes to construct a synergistic dual\-semanticcandidate pool\.

### 3\.3Structure\-Aware Dynamic Frame Routing

#### Temporally\-Coherent Event Clustering\.

To partition the N selected frames intoLLcoherent events, we jointly optimize the probabilistic assignment matrix𝑼∈\[0,1\]N×L\\bm\{U\}\\in\[0,1\]^\{N\\times L\}, semantic centroids𝒄l\\bm\{c\}\_\{l\}, and temporal centersτl\\tau\_\{l\}by minimizing:

min𝑼,𝒄,𝝉⁡ℒcluster=∑i=1N∑l=1LUi,l​\(1−𝒗i⊤​𝒄l⏟semantic cost\+β​\(ti−τl\)2⏟temporal cost\)−ϵ​ℋ​\(𝑼\),\\min\_\{\\bm\{U\},\\bm\{c\},\\bm\{\\tau\}\}\\mathcal\{L\}\_\{\\text\{cluster\}\}=\\sum\_\{i=1\}^\{N\}\\sum\_\{l=1\}^\{L\}U\_\{i,l\}\\Big\(\\underbrace\{1\-\\bm\{v\}\_\{i\}^\{\\top\}\\bm\{c\}\_\{l\}\}\_\{\{\\color\[rgb\]\{0\.12109375,0\.46484375,0\.70703125\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.12109375,0\.46484375,0\.70703125\}\\text\{semantic cost\}\}\}\+\\beta\\underbrace\{\(t\_\{i\}\-\\tau\_\{l\}\)^\{2\}\}\_\{\{\\color\[rgb\]\{0\.83984375,0\.15234375,0\.15625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.83984375,0\.15234375,0\.15625\}\\text\{temporal cost\}\}\}\\Big\)\-\\epsilon\\mathcal\{H\}\(\\bm\{U\}\),\(7\)whereℋ​\(𝑼\)=−∑i,lUi,l​log⁡Ui,l\\mathcal\{H\}\(\\bm\{U\}\)=\-\\sum\_\{i,l\}U\_\{i,l\}\\log U\_\{i,l\}encourages smooth transitions,ϵ\\epsiloncontrols cluster assignment softness, andβ\\betaregulates temporal compactness\. Frames in the candidate pool are assigned to event clusters\{G1,…,GL\}\\\{G\_\{1\},\\dots,G\_\{L\}\\\}viaarg⁡maxl⁡Ui,l\\arg\\max\_\{l\}U\_\{i,l\}\.

#### Dual\-Semantic Similarity\-Guided Distribution\.

To allocate the total frame budgetBB, we design a dynamic routing mechanism guided by the dual\-semantic score𝒮\\mathcal\{S\}\. We define the information capacityIlI\_\{l\}and frame quotamlm\_\{l\}for each eventGlG\_\{l\}:

Il=∑i∈Gl𝒮i\+η​log⁡\(1\+\|Gl\|\),ml=B​exp⁡\(Il/θ\)∑j=1Lexp⁡\(Ij/θ\),I\_\{l\}=\\sum\_\{i\\in G\_\{l\}\}\\mathcal\{S\}\_\{i\}\+\\eta\\log\(1\+\|G\_\{l\}\|\),\\quad m\_\{l\}=B\\frac\{\\exp\(I\_\{l\}/\\theta\)\}\{\\sum\_\{j=1\}^\{L\}\\exp\(I\_\{j\}/\\theta\)\},\(8\)whereθ\\thetacontrols the distribution sharpness, andη\\etascales the penalty against redundant temporal sampling\.mlm\_\{l\}is rounded to the nearest integer\.

#### Temporal Synchronization and MLLM Integration\.

Based on𝒮\\mathcal\{S\}, we retrieve the topmlm\_\{l\}frames from each fragmentGlG\_\{l\}\. The gatheredBBkeyframes are monotonically sorted into a temporal sequence𝒱select\\mathcal\{V\}\_\{\\text\{select\}\}and interleaved with the system promptPsys\\text\{P\}\_\{\\text\{sys\}\}and queryQ\. Finally, the joint representation𝒳=\{Psys,Q,𝒱select\}\\mathcal\{X\}=\\\{\\text\{P\}\_\{\\text\{sys\}\},\\text\{Q\},\\mathcal\{V\}\_\{\\text\{select\}\}\\\}is fed into the MLLM for cross\-modal reasoning\.

## 4Experiments

Table 1:Performance comparison on four long video understanding benchmarks\. Our approach is evaluated across three MLLMs under fixed input frame settings\. The best results are highlighted inbold\. Methods denoted with∗are training\-based\.### 4\.1Experimental Setup

✤ Evaluation Benchmarks and Metric\.We evaluate our method on four public long video benchmarks: Video\-MME\[fu2025video\], LongVideoBench\[wu2024longvideobench\], MLVU\[zhou2024mlvu\], and LVBench\[wang2025lvbench\], which assess distinct dimensions of prolonged multimodal reasoning\. To focus on visual understanding and weaken MLLM’s language bias, tasks are formulated as subtitle\-free multiple\-choice question answering\. Following standard protocols, we report the answer selection accuracy as the metric\.

✤ Implementation Details\.We instantiate our framework on three foundational Video\-LLMs: LLaVA\-Video\-7B\-Qwen2\[zhang2024llava\]\(frame budgetBBis set to 64\), Qwen2\-VL\-7B\-Instruct\[wang2024qwen2\]and Qwen3\-VL\-8B\-Instruct\[bai2025qwen3\]\(BBis set to 32\)\. We employ GPT\-4o\[achiam2023gpt\]as the reasoning LLM and OpenAI’s CLIP\-L\-14\[radford2021learning\]as the encoder to extract visual and textual representations\. To alleviate computational overhead while maintaining video information completeness, we construct the initial frames sequenceℐ\\mathcal\{I\}by uniformly sampling at 1 fps\. During subsequent selection phases,NNis empirically set to 150, andλ\\lambdais fixed at10−410^\{\-4\}\. All experiments are zero\-shot, training\-free, and performed on 8×\\timesNVIDIA H200 GPUs\.

### 4\.2Main Results

As summarized in Table[4](https://arxiv.org/html/2607.24794#S4), we comprehensively evaluate our proposed framework against state\-of\-the\-art Video\-LLMs and existing frame selection strategies\. The quantitative results unequivocally demonstrate the superiority of ReMem, which we analyze across three critical dimensions\.

Consistent High Performance\.Our framework establishes a new SOTA across multiple challenging benchmarks, demonstrating robust generalizability independent of LLM size or architecture\. When integrated into the77B LLaVA\-Video backbone, ReMem yields a substantial6\.5%↑6\.5\\%\\uparrowgain on MLVU\[zhou2024mlvu\]\. Performance leaps are also striking with the88B Qwen3\-VL backbone, where we observe a compelling10\.6%↑10\.6\\%\\uparrowabsolute improvement on LVBench\[wang2025lvbench\], alongside a significant jump on MLVU\[zhou2024mlvu\]\(63\.6%→77\.6%63\.6\\%\\\!\\rightarrow 77\.6\\%\)\. These consistent improvements validate our temporal granularity\-adaptive framework’s effectiveness\.

Efficacy of the Training\-Free Paradigm\.A hallmark advantage of ReMem is its ability to achieve exceptional performance without the need for expensive, domain\-specific fine\-tuning\. Strikingly, our zero\-shot formulation consistently surpasses recent strongly supervised frame\-selection baselines\. To illustrate, while the training\-intensive method GenS\[yao2025generative\]achieves a respectable66\.9%66\.9\\%on MLVU\[zhou2024mlvu\]using Qwen2\-VL, ReMem vastly outperforms it by reaching72\.8%72\.8\\%, marking a5\.9%↑5\.9\\%\\uparrowadvantage\. Similarly, when compared against the supervised FrameOracle\[li2025frameoracle\]on Qwen3\-VL, ReMem achieves77\.6%77\.6\\%compared to the baseline’s62\.9%62\.9\\%on MLVU\[zhou2024mlvu\], yielding a14\.7%↑14\.7\\%\\uparrowedge\. This confirms that explicitly modeling visual\-semantic\-temporal correlations mathematically is more efficient and accurate than implicitly learning them through brute\-force training\.

Robustness Across Diverse Temporal Granularities\.Our method’s superiority is vividly demonstrated on the Video\-MME\[fu2025video\]benchmark, containing Short, Medium, and Long subsets \(1\.3 to 41 minutes\)\. As video length extends, existing methods suffer severe information degradation\. While AKS\[tang2025adaptive\]enhances LLaVA\-Video on Short videos, it stagnates on the Long subset at54\.1%54\.1\\%\. In contrast, our approach maintains semantic coherence, achieving a remarkable61\.8%61\.8\\%on Long videos \(7\.7%↑7\.7\\%\\uparrowedge over AKS\[tang2025adaptive\]\)\. This robust trend continues with Qwen2\-VL: whereas Q\-Frame\[zhang2025q\]degrades to57\.1%57\.1\\%on extended durations, our framework outperforms it by securing65\.7%65\.7\\%\. These gains prove our dynamic allocation successfully bridges the semantic gap in long\-range reasoning\.

### 4\.3Ablation Studies

Table 2:Ablation study across multiple benchmarks with LLaVA\-Video model\.![Refer to caption](https://arxiv.org/html/2607.24794v1/ablation_qwen.png)\(a\)Ablation study on components across MLVU and LVBench\.
![Refer to caption](https://arxiv.org/html/2607.24794v1/ablation_TSA.png)\(b\)Ablation study on TSA across multiple benchmarks\.

Figure 3:Ablation results of Qwen2\-VL using ReMem\. \(a\) compares components of our proposed framework on Qwen video model\. \(b\) details the impact of removing TSA\.To evaluate the contribution of each component in our proposed ReMem framework, we conduct extensive ablation studies\. We primarily ablate four components: \(1\) whetherEntity Extraction\(EE\) is applied, \(2\) whetherStatic Visual\-Semantic Alignment\(VSA\) is used, \(3\) whetherMemory Augmented Temporal\-Semantic Alignment\(TSA\) is engaged, and \(4\) whetherStructure\-Aware Dynamic Frame Routing\(FR\) is enabled\. To demonstrate the model\-agnostic nature of ReMem, experiments are conducted using both LLaVA\-Video \(Table[2](https://arxiv.org/html/2607.24794#S4.T2)\) and Qwen2\-VL \(Fig\.[3](https://arxiv.org/html/2607.24794#S4.F3)\) across multiple benchmarks\. The results clearly indicate that all components are indispensable and act synergistically\.

Effect of Entity Extraction\.As demonstrated in Table[2](https://arxiv.org/html/2607.24794#S4.T2), incorporating Entity Extraction yields persistent performance gains across all datasets\. Notably, EE integration brings solid improvements on challenging benchmarks, securing a2\.2%↑2\.2\\%\\uparrowon LVBench\[wang2025lvbench\]and a1\.5%↑1\.5\\%\\uparrowon MLVU\[zhou2024mlvu\]\. Extracting entities from the original query provides fine\-grained contextual focus, proving that explicit language priors are highly beneficial for guiding memory graph construction with correct objects and actions\.

Static Visual\-Semantics Alignment\.As co\-validated by Table[2](https://arxiv.org/html/2607.24794#S4.T2)and Fig\.[2\(a\)](https://arxiv.org/html/2607.24794#S4.F2.sf1), equipping the framework with VSA triggers the most dramatic performance surges across different LLM architectures, proving its role as the core grounding mechanism\. For instance, on LVBench\[wang2025lvbench\], integrating VSA recovers a massive 12\.9% accuracy for LLaVA\-Video, alongside a mirrored12\.3%↑12\.3\\%\\uparrowfor Qwen2\-VL\. This consistent, cross\-model boost demonstrates that VSA successfully establishes the visual\-textual bridge, enabling the agent to accurately capture specific visual evidence rather than reasoning blindly\.

Memory Augmented Temporal\-Semantics Alignment\.The critical role of TSA is evident in Table[2](https://arxiv.org/html/2607.24794#S4.T2), demonstrating consistent accuracy gains across all four benchmarks upon its integration\. Further investigated in Fig\.[2\(b\)](https://arxiv.org/html/2607.24794#S4.F2.sf2), integrating TSA into Qwen2\-VL drives striking improvements across the board: an absolute gain of 5\.8% on Video\-MME\[fu2025video\], 5\.4% on MLVU\[zhou2024mlvu\], and a remarkable7\.3%↑7\.3\\%\\uparrowon LVBench\[wang2025lvbench\]\. This confirms that relying solely on isolated, static visual alignments is inadequate for long\-form contexts\. By encoding relative temporal distances and cross\-frame correlations, TSA bridges visually divergent but causally linked events across extended time spans, preserving the intrinsic temporal logic of the video\.

Structure\-Aware Dynamic Frame Routing\.Absolute score rankings inherently cause context redundancy by isolating single dominant events according to Table[2](https://arxiv.org/html/2607.24794#S4.T2)and Fig\.[2\(a\)](https://arxiv.org/html/2607.24794#S4.F2.sf1)\. Replacing this Top\-KKapproach with FR drives a6\.8%↑6\.8\\%\\uparrowaccuracy gain on MLVU\[zhou2024mlvu\]for Qwen2\-VL and consistent improvements for LLaVA\-Video\. This confirms that our temporal granularity\-adaptive routing successfully filters redundancy, retaining visual tokens with optimal semantic relevance and temporal diversity\.

### 4\.4Candidate Pool Capacity Scaling

Table 3:Performance of LLaVA\-Video model with ReMem on Video\-MME and LongVideoBench under varying candidate pool sizes\. We report accuracy and preprocessing latency \(frame selection duration per item\)\.To determine the optimal sizeNNof the candidate pool prior toStructure\-Aware Dynamic Frame Routing, we evaluate the trade\-off between reasoning accuracy and preprocessing latency\. Since the final budget of frames fed into MLLMs remains fixed asBB,NNstrictly controls the sampling capacity\.

As illustrated in Table[3](https://arxiv.org/html/2607.24794#S4.T3), expanding the candidate pool initially drives steady performance gains\. The accuracy climbs asNNincreases from 90 to 150, peaking at 69\.2% and 67\.1% respectively, confirming that a broad candidate pool prevents the premature pruning of vital visual evidence\. However, expanding the pool further triggers a noticeable accuracy drop alongside a linear surge in preprocessing time \(e\.g\.,N=200N=200drops accuracy to 68\.9% on VideoMME\[fu2025video\]\)\. This exposes a characteristic of clustering\-based condensation: an excessively large pool introduces a ‘long tail’ of noisy, low\-relevance frames\. These redundant frames severely distort the clustering feature space, diluting the semantic density of the final condensed budget\. Consequently,N=150N=150emerges as the optimal size, balancing sufficient diversity with required purity\.

![Refer to caption](https://arxiv.org/html/2607.24794v1/lamda.png)Figure 4:Model performance on two benchmarks with varying decay constant\.
![Refer to caption](https://arxiv.org/html/2607.24794v1/granularity.png)Figure 5:Performance comparison of different query temporal granularity analysis\.

### 4\.5Sensitivity Analysis of Decay Constant

To determine the optimal decay constantλ\\lambda, we analyze its impact on reasoning accuracy\. As illustrated in Fig\.[5](https://arxiv.org/html/2607.24794#S4.F5), accuracy on both VideoMME\[fu2025video\]and LongVideoBench\[wu2024longvideobench\]exhibits a clear inverted\-V trend asλ\\lambdaincreases from10−510^\{\-5\}to10−210^\{\-2\}\. Performance peaks atλ=10−4\\lambda=10^\{\-4\}\(achieving 69\.2% and 67\.1% respectively\) and drops significantly asλ\\lambdagrows larger\. This trajectory directly reflects the trade\-off governed byλ\\lambda\. Whenλ\\lambdais excessively small, the decay mechanism is sluggish\. The model over\-relies on static visual evidence for long videos, hindering the timely injection of temporal\-semantic alignment\. Conversely, asλ\\lambdaincreases to a larger value, premature decay forcesα\\alphato 0\.5 too early, neutralizing the temporal granularityggand preventing the model from adequately prioritizing instantaneous spatial cues\. Therefore,λ=10−4\\lambda=10^\{\-4\}provides the optimal balance, ensuring a well\-calibrated transition from localized visual focus to global temporal reasoning\.

### 4\.6Effect of Dynamic Time Granularity

To validate the necessity of the continuous Time Granularity, we compare our dynamic weighting mechanism𝒮=α​𝒮v\+\(1−α\)​𝒮t\\mathcal\{S\}=\\alpha\\mathcal\{S\}\_\{\\text\{v\}\}\+\(1\-\\alpha\)\\mathcal\{S\}\_\{\\text\{t\}\}against rigid binary classification \(short\-termα=0\.9\\alpha=0\.9, long\-termα=0\.5\\alpha=0\.5\)\. As illustrated in Fig\.[5](https://arxiv.org/html/2607.24794#S4.F5), upgrading to our dynamic approach yields consistent and notable performance gains across both LLM architectures\. For instance, on LongVideoBench\[wu2024longvideobench\], our dynamic method boosts Qwen2\-VL accuracy by 3\.3% and LLaVA\-Video by 2\.6%\. This performance gap exposes a fundamental limitation of the binary approach: long videos encapsulate complex events spanning a continuous spectrum of temporal scales\. A simple dichotomy lacks the necessary resolution to capture these subtleties\. Our dynamic weighting gracefully adapts to diverse granularities, establishing a precise alignment between the query’s temporal demands and the video’s topological memory\.

### 4\.7Efficiency Analysis and Trade\-off Discussion

![Refer to caption](https://arxiv.org/html/2607.24794v1/efficiency_vmme.png)

![Refer to caption](https://arxiv.org/html/2607.24794v1/efficiency_lvb.png)

Figure 6:Visualization of LLaVA\-Video reasoning accuracy and latency time with different frame selection methods on Video\-MME \(left\) and LongVideoBench \(right\)\.As illustrated in Fig\.[6](https://arxiv.org/html/2607.24794#S4.F6), our method achieves state\-of\-the\-art accuracy with slightly higher end\-to\-end inference time per sample compared to Uniform Sampling \(used by baseline\) and AKS\[tang2025adaptive\]\. However, this increase in total latency is a deliberate and highly rewarding accuracy\-efficiency trade\-off\.

Favorable Marginal Efficiency\.The increased latency primarily stems from the keyframe selection phase, where our module delves deeper into the video to capture richer spatial\-temporal information\. Importantly, this extra time investment yields exceptionally high returns\. Observing the Pareto frontier in Fig\.[6](https://arxiv.org/html/2607.24794#S4.F6), our method demonstrates superior efficiency: on Video\-MME\[fu2025video\], while AKS\[tang2025adaptive\]improves baseline accuracy by0\.9%0\.9\\%with\+6\.3​s\+6\.3s, our method yields a substantial3\.9%↑3\.9\\%\\uparrowover AKS\[tang2025adaptive\]with\+8\.4​s\+8\.4s, highlighting a much more effective utilization of computational resources\.

Amortized Cost via Plug\-and\-Play Design\.Our keyframe selection module functions as a flexible, model\-agnostic plug\-and\-play interface\. When processing the same benchmark, keyframe indices can be directly reused for inference across different backbones\. This significantly amortizes the selection time, making our pipeline highly efficient for multi\-model testing scenarios\.

### 4\.8Qualitative Analysis

![Refer to caption](https://arxiv.org/html/2607.24794v1/qualitative.png)Figure 7:Case analysis from Video\-MME\. Unlike Uniform Sampling and AKS which miss crucial temporal context, our method accurately routes the relevant frames following the key event \(red dot\) to answer correctly\.To intuitively demonstrate the superiority of our frame selection strategy, we present a qualitative comparison in Fig\.[7](https://arxiv.org/html/2607.24794#S4.F7), visualizing frames sampled by Uniform Sampling, AKS\[tang2025adaptive\], and our ReMem for a complex QA case\. The query is highly challenging as it demands explicit temporal logic reasoning: the model must temporally localize a milestone and exclusively attend to subsequent actions, filtering out prior events\.

As shown, both Uniform Sampling and AKS\[tang2025adaptive\]fail\. Uniform Sampling rigidly distributes frames across the entire video, allocating too much attention to early stages and diluting critical post\-furnace evidence\. Similarly, AKS\[tang2025adaptive\]retrieves visually dynamic but temporally irrelevant frames from the preliminary phase\. Conversely, ReMem bypasses distracting early phases and densely samples the critical frames after the furnace milestone, which explicitly depict the blogger painting the fired ceramics\. This compellingly validates our method’s ability to isolate crucial, query\-aligned visual cues for complex temporal reasoning\.

![Refer to caption](https://arxiv.org/html/2607.24794v1/qualitative_ablation.png)Figure 8:Case analysis from MLVU\. Variants without key components of our proposed ReMem fail\.To complement the quantitative findings in ablation studies, Fig\.[8](https://arxiv.org/html/2607.24794#S4.F8)qualitatively illustrates the necessity of each ReMem module\. As depicted, removing EE does not impact the outcome in this specific instance, as the candidate pool already provides sufficient information\. However, dropping VSA or TSA yields an incomplete temporal span, missing the initial appearance of the older man’s face\. Meanwhile, the variant without FR manages to find the correct visual cues but disrupts temporal coherence, leading to a scrambled event sequence\. Ultimately, the complete ReMem pipeline precisely extracts frames across the timeline, depicting all four anchor events in correct chronological order\. This visualization underscores how our modules work synergistically to guarantee both visual\-semantic relevance and temporal structural integrity\.

## 5Conclusion

In this work, we presentedReMem, a training\-free, temporal granularity\-adaptive keyframe selection framework to enhance zero\-shot LongVideoQA\. By introducing a memory\-augmented dual\-level adaptation, ReMem effectively integrates*query\-level adaptation*through Memory\-Driven Question Parsing and*video\-level adaptation*via Synergistic Dual\-Semantic Frame Alignment and Structure\-Aware Dynamic Frame Routing\. This synergistic combination allows the framework to identify the most discriminative frames while explicitly preserving causal structures and temporal information tailored to the varying query temporal granularities\. Extensive evaluations across four LongVideoQA benchmarks demonstrate that ReMem significantly improves zero\-shot reasoning capabilities across multiple MLLMs, achieving state\-of\-the\-art performance without any parameter tuning\. Our results highlight the potential of unified memory mechanisms as a powerful tool for robust long video understanding, opening avenues for future research in efficient and scalable training\-free spatiotemporal reasoning\.

## Acknowledgements

This work was supported by Ministry of Education Tier 2 grant, Singapore \(T2EP20224\-0028\), and Ministry of Education Tier 1 grant, Singapore \(23\-0651\-P0001\)\.

## References