DRIFT: A Residual Flow Adapter for Decoding Continuous Outputs in Vision-Language Models
Summary
DRIFT is a framework that adapts pretrained vision-language models for continuous output decoding by combining coarse prediction with iterative flow matching refinement, improving performance on perception and planning tasks.
View Cached Full Text
Cached at: 06/11/26, 05:35 PM
Paper page - DRIFT: A Residual Flow Adapter for Decoding Continuous Outputs in Vision-Language Models
Source: https://huggingface.co/papers/2606.05758
Abstract
DRIFT is a framework that adapts pretrained vision-language models for continuous decoding tasks by combining coarse prediction with iterative refinement through flow matching, improving performance across perception and planning tasks.
Many modernvision-language models(VLMs) build onautoregressive decodingofdiscrete tokens. While text-based output interfaces enable scalable pretraining and strong zero-shot generalization across diverse tasks, they are poorly suited for problems that require precisecontinuous outputs, such as localizing temporal boundaries of events or generatingrobotic controlactions. To address this challenge, we propose DRIFT, a general framework for adapting pretrained VLMs to continuous decoding tasks. DRIFT combines abase predictor, which provides a coarse estimate of the target output, with agenerative refinement modulebased onflow matchingthat iteratively improves the prediction. Thisresidual formulationtransforms the generative modeling problem from learning aglobal output distributionto modeling alocalized residual distributionaround a strong prior, substantially simplifying optimization. We evaluate DRIFT on both perception and planning tasks, includingvisual groundingandrobotic control. Across multiple tasks and architectures spanning MLLMs, VLAs, and WAMs, DRIFT consistently outperforms a strong set of regression- and generative-based solutions.
View arXiv pageView PDFProject pageGitHub0Add to collection
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.05758 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.05758 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.05758 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Drifting Objectives for Refining Discrete Diffusion Language Models
This paper introduces TokenDrift, a drifting objective that refines discrete diffusion language models by lifting categorical predictions to a continuous semantic space for anti-symmetric drifting, significantly improving generation quality under a fixed number of denoising steps.
LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling
LangFlow presents the first continuous diffusion language model that rivals discrete diffusion approaches, challenging the long-held belief that continuous diffusion is inferior for language modeling. The work introduces key ingredients like optimal Gumbel-based noise scheduling and demonstrates competitive perplexity and transfer learning performance compared to discrete diffusion baselines.
DRIFT: Decoupled Rollouts and Importance-Weighted Fine-Tuning for Efficient Multi-Turn Optimization
This paper proposes DRIFT, a framework that combines offline trajectories with importance-weighted supervised fine-tuning to efficiently achieve multi-turn interactive learning performance comparable to reinforcement learning.
Attention Drift: What Autoregressive Speculative Decoding Models Learn
This paper identifies 'attention drift' in autoregressive speculative decoding models, where drafters' attention shifts from the prompt to their own generated tokens. The authors propose architectural changes, such as post-norm and RMSNorm, which improve acceptance rates and robustness across various benchmarks.
DRIFT: Refining Instruction Data via On-Policy Data Attribution
DRIFT proposes a method that uses on-policy influence functions to refine training data distribution for supervised fine-tuning of large language models, consistently improving performance ceilings over existing baselines.