OmniPack: Unified Token Compression for Efficient Omni-modal Large Language Models

Hugging Face Daily Papers Papers
token-compression omni-modal llm efficiency training-free multimodal

Summary

OmniPack proposes a training-free token compression framework for omni-modal LLMs, combining structural pre-LLM compression with task-relevant inner-LLM semantic refinement, achieving strong performance-efficiency trade-offs on multiple benchmarks.

Omni-modal large language models (Omni-LLMs) have achieved remarkable performance on audio-visual understanding tasks, but processing long and highly redundant visual and audio token sequences incurs substantial computational overhead, demanding aggressive token compression for efficient deployment. Existing methods often degrade at low token budgets: pre-LLM compression may discard structurally important and globally distributed evidence, whereas inner-LLM compression often underexploits query-conditioned audio-visual collaboration. To address these limitations, we propose OmniPack, a training-free framework that coordinates structural compression before the LLM with task-relevant semantic refinement within the LLM. Before the LLM, OmniPack removes structural redundancy through modality-specific importance, global coverage, and similarity-aware merging. After sufficient multimodal interaction, it further consolidates diverse, task-relevant representations through textual guidance and audio-visual collaboration. Extensive experiments on five benchmarks with three Omni-LLM backbones demonstrate that OmniPack consistently achieves the best performance-efficiency trade-off across diverse retention ratios, outperforming all existing methods. Notably, on Qwen2.5-Omni-7B, OmniPack preserves 98.0% of the original performance while reducing FLOPs to 16.7%, and still retains 92.9% of the original performance with only 6.8% of the original FLOPs.
Original Article
View Cached Full Text

Cached at: 08/05/26, 05:43 AM

Paper page - OmniPack: Unified Token Compression for Efficient Omni-modal Large Language Models

Source: https://huggingface.co/papers/2608.03812 Authors:

,

,

,

,

,

,

,

,

,

,

,

Abstract

Omni-modallargelanguagemodels(Omni-LLMs)haveachievedremarkableperformanceonaudio-visualunderstandingtasks,butprocessinglongandhighlyredundantvisualandaudiotokensequencesincurssubstantialcomputationaloverhead,demandingaggressivetokencompressionforefficientdeployment.Existingmethodsoftendegradeatlowtokenbudgets:pre-LLMcompressionmaydiscardstructurallyimportantandgloballydistributedevidence,whereasinner-LLMcompressionoftenunderexploitsquery-conditionedaudio-visualcollaboration.Toaddresstheselimitations,weproposeOmniPack,atraining-freeframeworkthatcoordinatesstructuralcompressionbeforetheLLMwithtask-relevantsemanticrefinementwithintheLLM.BeforetheLLM,OmniPackremovesstructuralredundancythroughmodality-specificimportance,globalcoverage,andsimilarity-awaremerging.Aftersufficientmultimodalinteraction,itfurtherconsolidatesdiverse,task-relevantrepresentationsthroughtextualguidanceandaudio-visualcollaboration.ExtensiveexperimentsonfivebenchmarkswiththreeOmni-LLMbackbonesdemonstratethatOmniPackconsistentlyachievesthebestperformance-efficiencytrade-offacrossdiverseretentionratios,outperformingallexistingmethods.Notably,onQwen2.5-Omni-7B,OmniPackpreserves98.0%oftheoriginalperformancewhilereducingFLOPsto16.7%,andstillretains92.9%oftheoriginalperformancewithonly6.8%oftheoriginalFLOPs.

View arXiv pageView PDFGitHub1Add to collection

Get this paper in your agent:

hf papers read 2608\.03812

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.03812 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.03812 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.03812 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles