Boosting Visual Instruction Tuning with Self-Supervised Guidance
Summary
This paper proposes augmenting visual instruction tuning in multimodal language models with self-supervised tasks expressed as natural language instructions, improving vision-centric reasoning without additional architecture or annotations. By reformulating classical self-supervised pretext tasks as image-instruction-response triplets, the method achieves consistent performance improvements across multiple benchmarks by injecting only 3-10% visually grounded instructions into the training data.
View Cached Full Text
Cached at: 04/20/26, 08:27 AM
Paper page - Boosting Visual Instruction Tuning with Self-Supervised Guidance
Source: https://huggingface.co/papers/2604.12966
Abstract
Visual instruction tuning enhanced with naturally phrased self-supervised tasks improves vision-centric reasoning in multimodal language models without additional architecture or annotations.
Multimodal large language models (https://huggingface.co/papers?q=Multimodal%20large%20language%20models) (MLLMs) perform well on many vision-language tasks (https://huggingface.co/papers?q=vision-language%20tasks) but often struggle with vision-centric problems that require fine-grained visual reasoning (https://huggingface.co/papers?q=visual%20reasoning). Recent evidence suggests that this limitation arises not from weak visual representations, but from under-utilization of visual information during instruction tuning (https://huggingface.co/papers?q=instruction%20tuning), where many tasks can be partially solved using language priors alone. We propose a simple and lightweight approach that augments visual instruction tuning (https://huggingface.co/papers?q=instruction%20tuning) with a small number of visually grounded self-supervised tasks expressed as natural language instructions. By reformulating classical self-supervised pretext tasks (https://huggingface.co/papers?q=pretext%20tasks), such as rotation prediction, color matching, and cross-view correspondence, as image-instruction-response triplets (https://huggingface.co/papers?q=image-instruction-response%20triplets), we introduce supervision that cannot be solved without relying on visual evidence. Our approach requires no human annotations, no architectural modifications, and no additional training stages. Across multiple models, training regimes, and benchmarks, injecting only a small fraction (3-10%) of such visually grounded instructions consistently improves performance on vision-centric evaluations. Our findings highlight instruction tuning (https://huggingface.co/papers?q=instruction%20tuning) with visually grounded SSL tasks as a powerful lever for improving visual reasoning (https://huggingface.co/papers?q=visual%20reasoning) in MLLMs through simple adjustments to the training data distribution. Code available at: https://github.com/sirkosophia/V-GIFT
View arXiv page (https://arxiv.org/abs/2604.12966) View PDF (https://arxiv.org/pdf/2604.12966) GitHub13 (https://github.com/sirkosophia/V-GIFT) Add to collection (https://huggingface.co/login?next=%2Fpapers%2F2604.12966)
Get this paper in your agent:
hf papers read 2604.12966
Don’t have the latest CLI? curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2604.12966 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2604.12966 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2604.12966 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to a collection (https://huggingface.co/new-collection) to link it from this page.
Similar Articles
From Direction to Magnitude: How Multimodal Instruction-Tuning Reorganizes the Geometric Encoding of Identity-Specifying Prompts in Transformer Hidden States
This paper investigates how multimodal instruction-tuning reorganizes the geometric encoding of identity-specifying prompts in transformer hidden states, finding a shift from direction-based to magnitude-based encoding after instruction tuning.
Decomposing the Basic Abilities of Large Language Models: Mitigating Cross-Task Interference in Multi-Task Instruct-Tuning
This paper proposes Badit, a method that decomposes large language model parameters into orthogonal high-singular-value LoRA experts to mitigate cross-task interference during multi-task instruction tuning.
Steering Visual Generation in Unified Multimodal Models with Understanding Supervision
This paper introduces UNO, an Understanding-Oriented Post-Training framework that uses comprehension tasks as supervisory signals to enhance image generation and editing in unified multimodal models.
Self-Boosting Vision-Language Models with Noisy Student On-Policy Self-Distillation
Proposes NOPD, a self-distillation method that improves vision-language models without external supervision by leveraging prediction discrepancies between clean and corrupted inputs. Achieves significant gains on visual reasoning tasks, matching or exceeding RL and distillation from external models.
Anchoring What Matters: A Dual-Level Learning Framework for Visually-Grounded Multimodal Reasoning
This paper proposes PIVOT, a dual-level learning framework that enhances visually-grounded reasoning in large vision-language models by using self-calibrated experience replay and vision-guided advantage allocation to optimize reinforcement learning.