UltraViT: Latency-Optimized On-device Vision Encoder for Large Vision-Language Models

Hugging Face Daily Papers Papers
vision-encoder large-vision-language-models edge-devices latency-optimization pyramidal-architecture generative-pre-training

Summary

UltraViT is a latency-optimized vision encoder for large vision-language models, designed for on-device deployment with a pyramidal architecture and a two-stage generative pre-training strategy, achieving state-of-the-art performance at 1.7x speed.

Large Vision-Language Models (LVLMs) remain bottlenecked by massive computational footprints, precluding their deployment on resource-constrained edge devices. While efforts to compress LVLMs focus heavily on vision token reduction or smaller language models, the vision encoder is largely overlooked, typically deployed as a monolithic, computationally heavy feature extractor. Moreover, there is no previous effort that designs a vision encoder for LVLMs directly optimized for on-device latency. In this paper, we present UltraViT, a vision encoder for LVLMs, explicitly designed and optimized for on-device performance. Specifically, by taking into account real on-device latencies, we systematically design a pyramidal architecture that strategically integrates and adapts heterogeneous spatial mixers at the macro-block level. Furthermore, to pre-train UltraViT, we propose a novel two-stage generative pre-training strategy: cultivating rich spatial features via dense distillation, followed by direct generative supervision from a capacity-mixed frozen LLM. Compared to standard contrastive and SSL, we show that our pre-training is much more effective for achieving high-level semantic grounding for UltraViT needed for the subsequent generative multimodal alignment of LVLM training. Extensive experiments demonstrate that our on-device latency-informed design combined with our tailored training strategy establishes a new state-of-the-art for efficient LVLM encoding, significantly outperforming existing encoder-centric baselines while operating on-device at nearly 1.7xthe speed.
Original Article
View Cached Full Text

Cached at: 07/28/26, 06:26 PM

Paper page - UltraViT: Latency-Optimized On-device Vision Encoder for Large Vision-Language Models

Source: https://huggingface.co/papers/2607.23373

Abstract

LargeVision-LanguageModels(LVLMs)remainbottleneckedbymassivecomputationalfootprints,precludingtheirdeploymentonresource-constrainededgedevices.WhileeffortstocompressLVLMsfocusheavilyonvisiontokenreductionorsmallerlanguagemodels,thevisionencoderislargelyoverlooked,typicallydeployedasamonolithic,computationallyheavyfeatureextractor.Moreover,thereisnopreviouseffortthatdesignsavisionencoderforLVLMsdirectlyoptimizedforon-devicelatency.Inthispaper,wepresentUltraViT,avisionencoderforLVLMs,explicitlydesignedandoptimizedforon-deviceperformance.Specifically,bytakingintoaccountrealon-devicelatencies,wesystematicallydesignapyramidalarchitecturethatstrategicallyintegratesandadaptsheterogeneousspatialmixersatthemacro-blocklevel.Furthermore,topre-trainUltraViT,weproposeanoveltwo-stagegenerativepre-trainingstrategy:cultivatingrichspatialfeaturesviadensedistillation,followedbydirectgenerativesupervisionfromacapacity-mixedfrozenLLM.ComparedtostandardcontrastiveandSSL,weshowthatourpre-trainingismuchmoreeffectiveforachievinghigh-levelsemanticgroundingforUltraViTneededforthesubsequentgenerativemultimodalalignmentofLVLMtraining.Extensiveexperimentsdemonstratethatouron-devicelatency-informeddesigncombinedwithourtailoredtrainingstrategyestablishesanewstate-of-the-artforefficientLVLMencoding,significantlyoutperformingexistingencoder-centricbaselineswhileoperatingon-deviceatnearly1.7xthespeed.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2607\.23373

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.23373 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.23373 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.23373 in a Space README.md to link it from this page.

Collections including this paper1

Similar Articles

LLaVA-UHD v4: What Makes Efficient Visual Encoding in MLLMs?

Hugging Face Daily Papers

This paper introduces LLaVA-UHD v4, which improves visual encoding efficiency in multimodal large language models by using slice-based encoding and intra-ViT early compression. It reduces computational costs by over 55% while maintaining or improving performance on high-resolution image tasks.

VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding

Papers with Code Trending

VideoChat3 is a fully open, efficient, and generalist video-centric multimodal large language model that introduces Inflated 3D Vision Transformer (I3D-ViT) and Adaptive Frame Resolution for streaming video perception, along with scalable video data synthesis pipelines, achieving superior performance with only 4B parameters.

ViT-Up: Faithful Feature Upsampling for Vision Transformers

Hugging Face Daily Papers

ViT-Up introduces a task-agnostic feature upsampler for Vision Transformers that predicts features at arbitrary continuous image coordinates, enabling dense feature maps at any resolution and improving dense prediction and semantic correspondence benchmarks. It outperforms prior state-of-the-art upsamplers, with gains of up to +2.07 mIoU on Cityscapes and +4.17 [email protected] on SPair-71k.