VaaWIT: Visual-Aware Adaptation of Large Language Models for Multilingual Web Image Translation

Hugging Face Daily Papers Papers
large-vision-language-models multimodal image-translation visual-aware multilingual fine-tuning attention-mechanism

Summary

VaaWIT is an end-to-end framework enhancing Large Vision-Language Models for multilingual Web image translation via dual-stream attention and visual-aware adapters, outperforming SOTA baselines.

Translating text embedded in Web images is crucial for improving content accessibility and cross-lingual information retrieval, particularly within social media and e-commerce domains. Although Large Vision-Language Models (LVLMs) have advanced multimodal understanding, applying them to Web image translation remains challenging due to the visual representation gap: standard encoders often prioritize high-level semantics over the fine-grained visual details required for recognizing diverse character morphologies. To address this challenge, we propose VaaWIT, an end-to-end framework that adapts Large Language Models for multilingual Web image translation. The framework introduces two key technical contributions: (1) a Dual-Stream Attention Module (DSAM), which facilitates bidirectional interaction between multilingual semantic features and detailed visual representations, thereby synthesizing unified features robust to textual variations; and (2) a Visual-Aware Adapter (VAA), a parameter-efficient fine-tuning strategy that dynamically injects these fused visual cues into the frozen LLM backbone. This design enables the model to align the visual context with linguistic reasoning effectively while minimizing computational costs. Extensive experiments on eight tasks on three public benchmarks demonstrate that VaaWIT significantly outperforms state-of-the-art (SOTA) open-source baselines and achieves competitive performance against proprietary models. These results validate the efficacy of integrating fine-grained visual perception into LLMs for complex Web content analysis.
Original Article
View Cached Full Text

Cached at: 05/26/26, 02:41 AM

Paper page - VaaWIT: Visual-Aware Adaptation of Large Language Models for Multilingual Web Image Translation

Source: https://huggingface.co/papers/2605.24675 Published on May 23

·

Submitted byhttps://huggingface.co/liboaccn

Bo Lion May 25

Abstract

VaaWIT is an end-to-end framework that enhances Large Vision-Language Models for multilingual Web image translation by incorporating fine-grained visual perception through dual-stream attention and visual-aware adapters.

Translating text embedded in Web images is crucial for improving content accessibility and cross-lingual information retrieval, particularly within social media and e-commerce domains. AlthoughLarge Vision-Language Models(LVLMs) have advancedmultimodal understanding, applying them to Web image translation remains challenging due to thevisual representation gap: standard encoders often prioritize high-level semantics over the fine-grained visual details required for recognizing diversecharacter morphologies. To address this challenge, we propose VaaWIT, an end-to-end framework that adapts Large Language Models for multilingual Web image translation. The framework introduces two key technical contributions: (1) aDual-Stream Attention Module(DSAM), which facilitates bidirectional interaction betweenmultilingual semantic featuresand detailed visual representations, thereby synthesizing unified features robust to textual variations; and (2) aVisual-Aware Adapter(VAA), aparameter-efficient fine-tuningstrategy that dynamically injects these fused visual cues into the frozen LLM backbone. This design enables the model to align the visual context with linguistic reasoning effectively while minimizing computational costs. Extensive experiments on eight tasks on three public benchmarks demonstrate that VaaWIT significantly outperforms state-of-the-art (SOTA) open-source baselines and achieves competitive performance against proprietary models. These results validate the efficacy of integrating fine-grained visual perception into LLMs for complex Web content analysis.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2605\.24675

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.24675 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2605.24675 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.24675 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

VFA: Empowering Multilingual MLLMs via Vision-Free Adaptation

arXiv cs.CL

This paper proposes Vision-Free Adaptation (VFA), a framework that enhances multilingual capabilities in multimodal large language models by merging multilingual and vision-aligned task vectors without visual data, demonstrating improved performance and data efficiency.

CaVe-VLM-CoT: An Interpretable Vision-Language Model Framework

arXiv cs.AI

CaVe-VLM-CoT is a modular reflection-based agentic-RAG framework for vision-language models that enforces evidence-grounded reasoning through a five-stage pipeline, achieving 87.1% accuracy on ScienceQA and proposing a suite of 23 metrics for evaluation.

CLAP: Direct VLM-to-VLA Adaptation via Language-Action Grounding

arXiv cs.AI

CLAP proposes a method to convert pretrained vision-language models (VLMs) into vision-language-action models (VLAs) by prepending natural-language action descriptions to action token sequences, preserving semantic capabilities without architectural changes. It achieves 90.8% on LIBERO and improves robustness.