Video2LoRA: Parametric Video Internalization for Vision-Language Models

Hugging Face Daily Papers Papers
video-processing vision-language-models lora efficient-inference video-summarization captioning low-rank-adaptation

Summary

This paper introduces Video2LoRA, a method that predicts Low-Rank Adaptation (LoRA) weights directly from video representations, enabling efficient video processing in frozen vision-language models. It reduces visual token load by up to 1500x and query TTFT by 6-80x while maintaining performance on video summarization and captioning benchmarks.

Processing video in vision-language models is expensive: each frame occupies hundreds of tokens, and inference cost scales with every frame and every repeated query. We introduce Video2LoRA, a method for parametric video internalization. A perceiver hypernetwork reads the intermediate representations produced layer-by-layer as a frozen VLM encodes a video, and generates a Low-Rank Adaptation (LoRA) adapter in a single forward pass. Unlike standard LoRA fine-tuning, which requires iterative gradient updates, Video2LoRA predicts these weights directly from the video. Trained for SmolVLM2 500M and 2.2B on video summarization and captioning, Video2LoRA enables the same frozen VLM to answer queries from the adapter alone, with zero visual tokens in its context at query time. Video2LoRA is statistically non-inferior and equivalent to direct video-in-context inference across all five captioning benchmarks at both model scales, and across seven of eight video question answering benchmark-scale pairings. Although trained only on 12 frames at 384px, it remains stable up to 1,024 frames and 1024px, where direct video-in-context inference often degenerates. Across this sweep, it reduces answer-time visual-token load by up to 1,500x and query TTFT by 6-80x, while preserving video-faithful outputs. We also find that independently generated adapters for non-overlapping video segments can compose in rank space, suggesting a path toward chunked long-video internalization.
Original Article
View Cached Full Text

Cached at: 06/05/26, 06:07 AM

Paper page - Video2LoRA: Parametric Video Internalization for Vision-Language Models

Source: https://huggingface.co/papers/2606.04351

Abstract

Video2LoRA enables efficient video processing in vision-language models by predicting Low-Rank Adaptation weights from video representations, reducing computational costs while maintaining video-faithful outputs.

Processing video in vision-language models is expensive: each frame occupies hundreds of tokens, and inference cost scales with every frame and every repeated query. We introduce Video2LoRA, a method for parametric video internalization. Aperceiver hypernetworkreads theintermediate representationsproduced layer-by-layer as a frozen VLM encodes a video, and generates aLow-Rank Adaptation(LoRA) adapter in a single forward pass. Unlike standardLoRAfine-tuning, which requires iterative gradient updates, Video2LoRApredicts these weights directly from the video. Trained for SmolVLM2 500M and 2.2B onvideo summarizationandcaptioning, Video2LoRAenables the same frozen VLM to answer queries from the adapter alone, with zerovisual tokensin its context at query time. Video2LoRAis statistically non-inferior and equivalent to direct video-in-context inference across all fivecaptioningbenchmarks at both model scales, and across seven of eightvideo question answeringbenchmark-scale pairings. Although trained only on 12 frames at 384px, it remains stable up to 1,024 frames and 1024px, where direct video-in-context inference often degenerates. Across this sweep, it reduces answer-time visual-token load by up to 1,500x andquery TTFTby 6-80x, while preserving video-faithful outputs. We also find that independently generated adapters for non-overlapping video segments can compose inrank space, suggesting a path toward chunked long-video internalization.

View arXiv pageView PDFProject pageGitHub1Add to collection

Get this paper in your agent:

hf papers read 2606\.04351

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper1

#### MananSuri27/Video2LoRA-SmolVLM-ckpts Updatedabout 4 hours ago

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2606.04351 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.04351 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Hybrid-LoRA: Bridging Full Fine-Tuning and Low-Rank Adaptation for Post-Training

arXiv cs.LG

Hybrid-LoRA proposes a framework that selectively applies full fine-tuning to a small subset of modules while using LoRA for the rest, achieving performance near full fine-tuning with significantly lower computational cost. Experiments show improvements of up to 5.65% over existing parameter-efficient baselines.

LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence

Hugging Face Daily Papers

LLaVA-OneVision-2 introduces codec-stream tokenization and windowed attention for efficient video understanding, achieving state-of-the-art performance across multiple multimodal benchmarks including video, spatial, and tracking tasks.