LEAP: Learned Block-wise Evidence Retrieval for Long Audio-Video Perception

Hugging Face Daily Papers Papers
audio-visual-qa long-context evidence-retrieval omni-modal lora-fine-tuning streaming-inference

Summary

LEAP introduces a block-wise evidence retrieval framework for hour-scale audio-visual question answering that decouples localization from reasoning, keeping context independent of recording duration and improving over Qwen3-Omni-30B-A3B by 4.5-16.8% while transferring to MiniCPM-o4.5.

Hour-scale audio-visual question answering is constrained by a context dilemma: dense whole-recording encoding rapidly exhausts context limits, whereas uniform temporal compression severely dilutes fine-grained acoustic and visual evidence. We introduce LEAP, a framework where the model retrieves its own evidence without placing the whole recording in one context. LEAP divides a recording into fixed-duration blocks, applying a lightweight localization pass to each block to score short candidate windows. The highest-ranked windows are pooled and re-encoded in a single bounded answer pass. Consequently, the answer input and peak context remain independent of the recording duration. By decoupling evidence localization from reasoning, our framework can localize candidate temporal windows over pre-computed transcripts without decoding media frames, while preserving fine-grained visual and non-speech evidence by routing the final answering pass over raw audio-visual streams. LEAP trains both stages: a localization LoRA improves the selected windows, and an answer LoRA improves the answers read from the same windows. The block grid natively supports causal queries, enabling LEAP to support streaming inference without streaming-specific training. Across several AVQA benchmarks, LEAP improves over the Qwen3-Omni-30B-A3B baseline by 4.5-16.8%, and transfers to a second omni-modal backbone, MiniCPM-o 4.5, surpassing its published results by 3.1-13.0%.
Original Article
View Cached Full Text

Cached at: 10/01/26, 04:23 PM

Paper page - LEAP: Learned Block-wise Evidence Retrieval for Long Audio-Video Perception

Source: https://huggingface.co/papers/2609.39938 Published on Sep 30

·

Submitted byhttps://huggingface.co/juyil

juyion Oct 1

Authors:

,

,

,

,

,

,

,

,

,

,

Abstract

Hour-scaleaudio-visualquestionansweringisconstrainedbyacontextdilemma:densewhole-recordingencodingrapidlyexhaustscontextlimits,whereasuniformtemporalcompressionseverelydilutesfine-grainedacousticandvisualevidence.WeintroduceLEAP,aframeworkwherethemodelretrievesitsownevidencewithoutplacingthewholerecordinginonecontext.LEAPdividesarecordingintofixed-durationblocks,applyingalightweightlocalizationpasstoeachblocktoscoreshortcandidatewindows.Thehighest-rankedwindowsarepooledandre-encodedinasingleboundedanswerpass.Consequently,theanswerinputandpeakcontextremainindependentoftherecordingduration.Bydecouplingevidencelocalizationfromreasoning,ourframeworkcanlocalizecandidatetemporalwindowsoverpre-computedtranscriptswithoutdecodingmediaframes,whilepreservingfine-grainedvisualandnon-speechevidencebyroutingthefinalansweringpassoverrawaudio-visualstreams.LEAPtrainsbothstages:alocalizationLoRAimprovestheselectedwindows,andananswerLoRAimprovestheanswersreadfromthesamewindows.Theblockgridnativelysupportscausalqueries,enablingLEAPtosupportstreaminginferencewithoutstreaming-specifictraining.AcrossseveralAVQAbenchmarks,LEAPimprovesovertheQwen3-Omni-30B-A3Bbaselineby4.5-16.8%,andtransferstoasecondomni-modalbackbone,MiniCPM-o4.5,surpassingitspublishedresultsby3.1-13.0%.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2609\.39938

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.39938 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.39938 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.39938 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Evidence-Backed Video Question Answering

Hugging Face Daily Papers

This paper introduces Evidence-Backed Video Question Answering (E-VQA), a new task requiring models to output both semantic answers and precise spatio-temporal evidence like tracked object segmentation masklets. The authors create a human-verified benchmark and a scalable training dataset, showing significant improvements over baselines.

OmniSeek: Native Tool Integration for Multi-turn Audio-Visual Reasoning

Hugging Face Daily Papers

OmniSeek is an agentic framework that turns an Omni-LLM into an active multi-turn reasoning agent with native tool use, dynamically retrieving audio/visual evidence across long contexts. It uses a synthetic OmniTraj-170K trajectory corpus, supervised cold-starting, and two-stage verifiable-reward RL with an Audio-Visual Necessity objective to boost cross-modal reasoning.