VaaWIT: Visual-Aware Adaptation of Large Language Models for Multilingual Web Image Translation

Hugging Face Daily Papers Papers
large-vision-language-models multimodal image-translation visual-aware multilingual fine-tuning attention-mechanism

Summary

VaaWIT is an end-to-end framework enhancing Large Vision-Language Models for multilingual Web image translation via dual-stream attention and visual-aware adapters, outperforming SOTA baselines.

Translating text embedded in Web images is crucial for improving content accessibility and cross-lingual information retrieval, particularly within social media and e-commerce domains. Although Large Vision-Language Models (LVLMs) have advanced multimodal understanding, applying them to Web image translation remains challenging due to the visual representation gap: standard encoders often prioritize high-level semantics over the fine-grained visual details required for recognizing diverse character morphologies. To address this challenge, we propose VaaWIT, an end-to-end framework that adapts Large Language Models for multilingual Web image translation. The framework introduces two key technical contributions: (1) a Dual-Stream Attention Module (DSAM), which facilitates bidirectional interaction between multilingual semantic features and detailed visual representations, thereby synthesizing unified features robust to textual variations; and (2) a Visual-Aware Adapter (VAA), a parameter-efficient fine-tuning strategy that dynamically injects these fused visual cues into the frozen LLM backbone. This design enables the model to align the visual context with linguistic reasoning effectively while minimizing computational costs. Extensive experiments on eight tasks on three public benchmarks demonstrate that VaaWIT significantly outperforms state-of-the-art (SOTA) open-source baselines and achieves competitive performance against proprietary models. These results validate the efficacy of integrating fine-grained visual perception into LLMs for complex Web content analysis.
Original Article
View Cached Full Text

Cached at: 05/26/26, 02:41 AM

Paper page - VaaWIT: Visual-Aware Adaptation of Large Language Models for Multilingual Web Image Translation

Source: https://huggingface.co/papers/2605.24675 Published on May 23

·

Submitted byhttps://huggingface.co/liboaccn

Bo Lion May 25

Abstract

VaaWIT is an end-to-end framework that enhances Large Vision-Language Models for multilingual Web image translation by incorporating fine-grained visual perception through dual-stream attention and visual-aware adapters.

Translating text embedded in Web images is crucial for improving content accessibility and cross-lingual information retrieval, particularly within social media and e-commerce domains. AlthoughLarge Vision-Language Models(LVLMs) have advancedmultimodal understanding, applying them to Web image translation remains challenging due to thevisual representation gap: standard encoders often prioritize high-level semantics over the fine-grained visual details required for recognizing diversecharacter morphologies. To address this challenge, we propose VaaWIT, an end-to-end framework that adapts Large Language Models for multilingual Web image translation. The framework introduces two key technical contributions: (1) aDual-Stream Attention Module(DSAM), which facilitates bidirectional interaction betweenmultilingual semantic featuresand detailed visual representations, thereby synthesizing unified features robust to textual variations; and (2) aVisual-Aware Adapter(VAA), aparameter-efficient fine-tuningstrategy that dynamically injects these fused visual cues into the frozen LLM backbone. This design enables the model to align the visual context with linguistic reasoning effectively while minimizing computational costs. Extensive experiments on eight tasks on three public benchmarks demonstrate that VaaWIT significantly outperforms state-of-the-art (SOTA) open-source baselines and achieves competitive performance against proprietary models. These results validate the efficacy of integrating fine-grained visual perception into LLMs for complex Web content analysis.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2605\.24675

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.24675 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2605.24675 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.24675 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

CaVe-VLM-CoT: An Interpretable Vision-Language Model Framework

arXiv cs.AI

CaVe-VLM-CoT is a modular reflection-based agentic-RAG framework for vision-language models that enforces evidence-grounded reasoning through a five-stage pipeline, achieving 87.1% accuracy on ScienceQA and proposing a suite of 23 metrics for evaluation.

Large Vision-Language Models Get Lost in Attention

arXiv cs.AI

This research paper analyzes the internal mechanics of Large Vision-Language Models (LVLMs) using information theory, revealing that attention mechanisms may be redundant while Feed-Forward Networks drive semantic innovation. The authors demonstrate that replacing learned attention weights with random values can yield comparable performance, suggesting current models 'get lost in attention'.

WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agent

Papers with Code Trending

WebWatcher is a multimodal agent for deep research that uses synthetic trajectories and reinforcement learning to achieve superior performance in complex visual and textual information retrieval tasks. The paper also introduces BrowseComp-VL, a new benchmark for evaluating multimodal agents.