See What I Mean: Aligning Vision and Language Representations for Video Fine-grained Object Understanding
Summary
SWIM is a novel training strategy that aligns vision and language representations for fine-grained object understanding using only textual prompts, leveraging mask supervision during training to improve cross-modal attention. It introduces the NL-Refer dataset and achieves superior performance over visual-prompt-based methods.
View Cached Full Text
Cached at: 05/25/26, 06:36 AM
Paper page - See What I Mean: Aligning Vision and Language Representations for Video Fine-grained Object Understanding
Source: https://huggingface.co/papers/2605.18018
Abstract
SWIM is a training approach that aligns vision and language representations for fine-grained object understanding using only textual prompts by addressing cross-modal attention misalignment through mask supervision and a new dataset.
We present SWIM (See What I Mean), a novel training strategy that aligns vision and language representations to enable fine-grained object understanding solely from textual prompts. Unlike existing approaches that require explicit visual prompts, such as masks or points, SWIM leverages mask supervision only during training to guidecross-modal attention, allowing the model to automatically attend to the user-specified object at inference. Our cross-attention analysis of pretrained multimodal large languagemodels (MLLMs) reveals a systematic discrepancy: Attribute words produce sharp, localized activations in the visual modality, whereas object nouns yield diffuse and scattered patterns due to semantic reference bias and distributed high-level representations. To address this misalignment, we construct NL-Refer, an enriched dataset, in which each object mask is paired with a precise natural language referring expression. SWIM extractsmulti-layer cross-attention mapsfrom object nouns and enforcesspatial consistencywith ground-truth masks. Experimental results demonstrate that SWIM substantially improves text-visual alignment and achieves superior performance over visual-prompt-based methods on fine-grained object understanding benchmarks. The code and data are available at https://github.com/HumanMLLM/SWIM{https://github.com/HumanMLLM/SWIM}.
View arXiv pageView PDFProject pageGitHub80Add to collection
Get this paper in your agent:
hf papers read 2605\.18018
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper1
#### BBBBCHAN/SWIM-7B Video-Text-to-Text• 8B• Updated5 days ago • 29 • 2
Datasets citing this paper1
#### BBBBCHAN/NL-Refer Preview• Updated5 days ago • 168 • 2
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.18018 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Which Pretraining Paradigm Better Serves Spatial Intelligence? An Empirical Comparison of Vision-Language and Video Generation Models
This paper presents a systematic frozen-feature probing study comparing vision-language models (VLMs) and video generation models (VGMs) on spatial intelligence tasks. It finds that VLMs excel at semantic tagging and instance grouping, while VGMs provide better dense geometry and camera motion signals, and a naive fusion of both yields strong performance across all axes.
Large Language Model Teaches Visual Students: Cross-Modality Transfer of Fine-Grained Conceptual Knowledge
This paper introduces LaViD, a framework that transfers semantic knowledge from a language-only LLM to a vision student model by generating multiple-choice questions as conceptual signatures, achieving superior fine-grained classification performance and robustness.
PARCEL: Pool-Anchored Resampling with Conditioned Elastic Queries for Efficient Vision-Language Understanding
PARCEL introduces a novel vision-language model architecture that uses pool-anchored resampling and conditioned elastic queries to improve efficiency and performance across different visual-token budgets, outperforming existing matryoshka baselines.
@lukaskuhn77: We introduce LeVLJEPA: the first fully non-contrastive end-to-end vision-language pretraining method competitive with C…
LeVLJEPA is the first fully non-contrastive end-to-end vision-language pretraining method, achieving competitive performance with CLIP and SigLIP without negatives, temperature, or momentum encoder. It learns via cross-modal prediction with stop-gradient targets and per-modality distributional regularization, providing stronger dense semantic features for downstream tasks like VLM backbones and semantic segmentation.
Watch, Remember, Reason: Human-View Video Understanding with MLLMs
A survey presenting a human-view perspective on video understanding with multimodal large language models, organized around watching, remembering, and reasoning abilities, covering challenges, methods, and applications.