ReVA: A Region-Aware Visual Assistant for Visually Grounded Question Answering

# A Region-Aware Visual Assistant for Visually Grounded Question Answering
Source: [https://arxiv.org/html/2608.28707](https://arxiv.org/html/2608.28707)
## ReVA: A Region\-Aware Visual Assistant for Visually Grounded Question AnsweringThanks:1The source code and pre\-trained weights for ReVA are publicly available athttps://github\.com/anoop675/reva\.

Anoop Senthil1Affiliation:School of Electronic Engineering and Computer Science Queen Mary University of London London, United Kingdom a\.senthil@se25\.qmul\.ac\.ukAffiliation:

###### Abstract

Multimodal Large Language Models \(MLLMs\) have achieved remarkable progress in Visual Question Answering \(VQA\), yet they continue to struggle with questions requiring precise spatial reasoning and fine\-grained visual understanding\. These limitations often manifest as object, attribute, and spatial hallucinations, where models generate confident but visually unsupported responses due to insufficient region\-level and fine\-grained visual grounding\. To address this challenge, we propose ReVA, a region\-aware VQA model that employs a frozen CLIP ViT\-L/14 Vision Transformer \(ViT\) and a Qwen2\.5\-7B\-Instruct large language model \(LLM\) connected through a dual bridge that aligns both whole\-image and region\-level representations with the LLM’s embedding space\. The image bridge maps final transformer block features into image tokens\. The region bridge maps cropped features from enriched intermediate features across ViT blocks so early texture and later object cues are more evident, intoKKregion tokens for every bounding box\. ReVA uses a detector stack that supplies automatic zero\-shot bounding boxes that are both question\-agnostic and question\-dependent, using RAM\+\+ \(Recognize Anything Model\), spaCy, and Grounding DINO\. The image tokens and region tokens are concatenated as an LLM prompt prefix to jointly encode scene\-level context and fine\-grained regional evidence when answering questions\. Evaluated on VQAv2, MMBench, POPE, and SEED\-Bench, ReVA achieves 82\.85% mean F1 on POPE, compared with 81\.14% for an image\-token baseline without region tokens\. These results demonstrate that explicit region\-aware visual representations reduce object hallucination and improve the factual grounding of MLLMs\.

###### Index Terms:

Visual Question Answering, Multimodal Large Language Models, Region Grounding, Multimodal Context Engineering, Vision–Language Alignment, LoRA Fine\-Tuning

## IIntroduction

This work presentsReVA\(Region\-Aware Visual Assistant\), a region\-aware VQA model built to ground language in both the broader scene and the fine\-grained visual evidence within it \(Fig\.[1](https://arxiv.org/html/2608.28707#S3.F1)\)\. At its core, ReVA pairs a frozen CLIP ViT\-L/14 Vision Transformer \(ViT\)\([Radford et al\. 2021](https://arxiv.org/html/2608.28707#bib.bib31);[Dosovitskiy et al\. 2021](https://arxiv.org/html/2608.28707#bib.bib7)\), operating at 336 px, with a Qwen2\.5\-7B\-Instruct backbone\([Qwen Team 2024](https://arxiv.org/html/2608.28707#bib.bib30)\)adapted via LoRA\([Hu et al\. 2022](https://arxiv.org/html/2608.28707#bib.bib13)\), linked by an image bridge and a region bridge\. The image bridge maps final transformer block features into 576 image tokens\. For local evidence, RAM\+\+ \(Recognize Anything Model\)\([Huang et al\. 2023](https://arxiv.org/html/2608.28707#bib.bib14)\)tags objects in the image, spaCy\([Honnibal et al\. 2020](https://arxiv.org/html/2608.28707#bib.bib12)\)adds nouns from the question so named objects are not missed, and Grounding DINO\([Liu et al\. 2024b](https://arxiv.org/html/2608.28707#bib.bib24)\)localises each tag to a box\. This makes the region detector’s proposals both question\-agnostic and question\-dependent\. The region bridge then crops intermediate ViT\-block features with those boxes and maps each crop to region tokens\. The LLM therefore sees what lies inside each box, and the boxes themselves never appear in the prompt as text\. Training has three stages so each bridge aligns before the LLM combines them: Stage 1 trains only the image bridge on whole\-image captions \(CLIP and Qwen frozen\); Stage 2 trains the region bridge on box\-linked descriptions with frozen backbones; Stage 3 adapts Qwen with LoRA to answer from the concatenated image and region tokens\. In later comparisons, the*image\-token*baseline uses the Stage 1 image bridge plus a separately trained Stage 3 LoRA with no region tokens;*ReVA*is the full model after all three stages, using region tokens from proposed boxes at inference\.

Recent MLLMs—LLaVA\([Liu et al\. 2024a](https://arxiv.org/html/2608.28707#bib.bib22)\), InstructBLIP\([Dai et al\. 2023](https://arxiv.org/html/2608.28707#bib.bib6)\), Qwen\-VL\([Bai et al\. 2023](https://arxiv.org/html/2608.28707#bib.bib2)\)—encode the image, project patches into the LLM, and generate an answer\. That whole\-image path is strong for scene\-level understanding, yet even a dense 576\-token grid at 336 px can leave spatial relations and attribute–instance binding ambiguous without explicit local evidence\. GPT4RoI\([Zhang et al\. 2023](https://arxiv.org/html/2608.28707#bib.bib41)\)supplies learned RoI tokens; Shikra\([Chen et al\. 2023](https://arxiv.org/html/2608.28707#bib.bib4)\)supplies box text\. ReVA follows the RoI\-token approach, expands each box intoKKregion tokens rather than a single pooled vector, and keeps the user question as ordinary natural language\. Local evidence enters only as region tokens, as a prefix of the LLM prompt\.

## IIRelated Work

#### Connecting vision encoders to language models

The dominant recipe for modern MLLMs starts from a pretrained vision encoder and a pretrained LLM, learns a bridge between them, and often adapts the LLM while keeping the vision encoder frozen\. BLIP\-2\([Li et al\. 2023b](https://arxiv.org/html/2608.28707#bib.bib18)\)introduced a lightweight Querying Transformer \(Q\-Former\) that extracts a small set of query embeddings from a frozen image encoder for the LLM to read, and InstructBLIP\([Dai et al\. 2023](https://arxiv.org/html/2608.28707#bib.bib6)\)made this bridge instruction\-aware by also feeding the task instruction into the Q\-Former, so the visual features it extracts depend on the question being asked\. LLaVA\([Liu et al\. 2023](https://arxiv.org/html/2608.28707#bib.bib21)\)showed that an even simpler bridge—a linear projection, upgraded to a two\-layer MLP in LLaVA\-1\.5\([Liu et al\. 2024a](https://arxiv.org/html/2608.28707#bib.bib22)\)—trained on visual\-instruction data is remarkably effective, while Qwen\-VL\([Bai et al\. 2023](https://arxiv.org/html/2608.28707#bib.bib2)\)compressed patch features into a fixed\-length sequence through a position\-aware cross\-attention adapter\. ReVA’s image bridge sits in this lineage: a frozen CLIP encoder\([Radford et al\. 2021](https://arxiv.org/html/2608.28707#bib.bib31)\)feeds a two\-layer MLP \(the Image Feature Projector\) into Qwen2\.5\-7B\-Instruct\([Qwen Team 2024](https://arxiv.org/html/2608.28707#bib.bib30)\)\. On its own, that image bridge presents the image as one holistic token set—strong for scene\-level understanding, but with no explicit handle on individual objects\.

#### Making MLLMs region\-aware

Two broad strategies have emerged to restore that missing locality\. The first encodes location in the text stream: Shikra\([Chen et al\. 2023](https://arxiv.org/html/2608.28707#bib.bib4)\)writes bounding boxes as plain numbers in natural language with no extra vocabulary or detector, Kosmos\-2\([Peng et al\. 2023](https://arxiv.org/html/2608.28707#bib.bib29)\)attaches discrete location tokens to phrases in a Markdown\-style hyperlink format \(trained on its large\-scale GRIT corpus\), and Qwen\-VL\([Bai et al\. 2023](https://arxiv.org/html/2608.28707#bib.bib2)\)wraps normalised box strings in special tokens\. These schemes are elegant, but they ask the LLM to handle spatial position as text or location tokens rather than as cropped visual evidence\. The second strategy instead feeds the LLM visual features taken from the region itself\. GPT4RoI\([Zhang et al\. 2023](https://arxiv.org/html/2608.28707#bib.bib41)\)builds a multi\-level CLIP feature pyramid, incorporates feature coordinates\([Liu et al\. 2018](https://arxiv.org/html/2608.28707#bib.bib23)\)for absolute position, and uses RoI Align\([He et al\. 2017](https://arxiv.org/html/2608.28707#bib.bib11)\)so that each box replaces a<region\>placeholder with region features interleaved in the instruction\. Ferret\([You et al\. 2024](https://arxiv.org/html/2608.28707#bib.bib36);[Zhang et al\. 2024](https://arxiv.org/html/2608.28707#bib.bib40)\)takes a hybrid route: discrete coordinates plus continuous features from a spatial\-aware visual sampler that covers points, boxes, and free\-form shapes\. GLaMM\([Rasheed et al\. 2024](https://arxiv.org/html/2608.28707#bib.bib32)\)pushes grounding down to pixel\-level segmentation masks, and Groma\([Ma et al\. 2024](https://arxiv.org/html/2608.28707#bib.bib27)\)treats regions as first\-class visual tokens—encoding proposed regions into region tokens alongside global tokens so the LLM can ground by referring to those tokens rather than regressing coordinates\. ReVA adopts the region\-feature\-as\-token philosophy for VQA\. Where it differs from GPT4RoI is how a box becomes tokens: rather than fusing the pyramid levels into a single RoI embedding, ReVA mixes them with multi\-head self\-attention and expands each box intoKKtokens through multi\-token attention pooling, giving Qwen a richer local descriptor beside the 576 image tokens\. It also keeps the user question as plain natural language without interleaving boxes, location tokens, or<region\>placeholders into the query text\.

#### Where the regions come from

At inference, GPT4RoI accepts boxes from the user or from an off\-the\-shelf detector; Ferret’s referring path takes user\-specified regions \(points, boxes, or free\-form shapes\) while its grounding path emits boxes in the response; and Groma trains its own region proposer\. ReVA instead builds the region set automatically from off\-the\-shelf open\-vocabulary detectors: RAM\+\+ \(Recognize Anything Model\)\([Huang et al\. 2023](https://arxiv.org/html/2608.28707#bib.bib14)\)proposes open\-set tags for what is present, and Grounding DINO\([Liu et al\. 2024b](https://arxiv.org/html/2608.28707#bib.bib24)\)localises those tags into boxes from a text prompt\. No detector is retrained, and the boxes only decide where to crop features rather than entering the prompt as text\. This design tests whether region\-aware gains are available under standard VQA inputs that use ordinary natural language questions with no additional spatial context interleaved in the question itself\.

## IIISystem Architecture

Fig\.[1](https://arxiv.org/html/2608.28707#S3.F1)shows ReVA’s dual\-bridge pipeline that supplies both image tokens and region tokens into Qwen\. The image is encoded once by CLIP ViT\-L/14, the Vision Transformer in the figure\. The image bridge passes final transformer block features through the Image Feature Projector as 576 image tokens\. The region bridge enriches intermediate transformer block features with CoordConv, per\-level projection, and multi\-head self\-attention, then crops them with proposed box coordinates; they becomeKKregion tokens per box\. Those visual tokens are concatenated with a short text footer \(answer format, question, “Answer:”\); Qwen then generates the reply\. Stages 1–2 train the image bridge and the region bridge with frozen CLIP and Qwen; Stage 3 adds LoRA\([Hu et al\. 2022](https://arxiv.org/html/2608.28707#bib.bib13)\)in Qwen\.

![Refer to caption](https://arxiv.org/html/2608.28707v1/figures/architecture_overview.jpg)Fig\. 1:Overview of ReVA\. The Vision Transformer is CLIP ViT\-L/14\. Image bridge: final transformer block features pass through the Image Feature Projector to image tokens\. Region bridge: intermediate transformer block features are enriched \(CoordConv, per\-level projection, multi\-head self\-attention\), cropped with RoI Align using RAM\+\+ \(Recognize Anything Model\) tags localised by Grounding DINO, then pooled and mapped by the Region Feature Projector to region tokens\.### III\-AImage Alignment \(image bridge\)

The image bridge is a LLaVA\-style two\-layer MLP—the Image Feature Projector—that maps each CLIP patch into Qwen’s embedding space\. Images are resized to336×336336\{\\times\}336and encoded by CLIP ViT\-L/14\([Radford et al\. 2021](https://arxiv.org/html/2608.28707#bib.bib31)\)into a24×2424\{\\times\}24grid of 1024\-d patch tokens \(CLS discarded\)\. Let𝐟patch∈ℝ1024\\mathbf\{f\}^\{\\mathrm\{patch\}\}\\in\\mathbb\{R\}^\{1024\}denote one such CLIP patch token\. The Image Feature Projector maps each patch into Qwen’s 3584\-d space, yielding an image token𝐳image∈ℝ3584\\mathbf\{z\}^\{\\mathrm\{image\}\}\\in\\mathbb\{R\}^\{3584\}:

𝐳image=W2​GELU​\(W1​𝐟patch\+b1\)\+b2,\\mathbf\{z\}^\{\\mathrm\{image\}\}=W\_\{2\}\\,\\mathrm\{GELU\}\(W\_\{1\}\\,\\mathbf\{f\}^\{\\mathrm\{patch\}\}\+b\_\{1\}\)\+b\_\{2\},\(1\)In Stage 1, the 576 image tokens are placed before the caption tokens to form the LLM input\. The training objective is next\-token prediction on the caption only: image\-token positions are masked, so they contribute no loss, and the projector is trained to make the image tokens useful for generating the caption that follows\.

### III\-BRegion Alignment \(region bridge\)

The region bridge maps a bounding box to region tokens that describe its contents, following GPT4RoI\([Zhang et al\. 2023](https://arxiv.org/html/2608.28707#bib.bib41)\)with the modifications below\. It comprises CoordConv, per\-level projection, multi\-head self\-attention, RoI Align, multi\-token attention pooling, and a final Region Feature Projector\. Starting from the same CLIP ViT used for the image bridge, intermediate transformer block features are taken, which represent early texture to higher\-level object cues\. Following CoordConv\([Liu et al\. 2018](https://arxiv.org/html/2608.28707#bib.bib23)\), normalised spatial channels are concatenated onto each24×2424\{\\times\}24feature map so every patch embedding position carries absolute spatial information\. Each depth is then passed through a linear layer and LayerNorm so content and coordinates are mixed and the four levels share a common feature space for multi\-head self\-attention\. Multi\-head self\-attention subsequently mixes them so that shallow texture and deeper semantics exchange information at the same patch embedding position\. RoI Align extracts a fixed\-size window of per\-level cropped features from each boxed region of these enriched per\-level features\. Multi\-token attention pooling then converts each window intoKKtokens rather than a single pooled vector, and the Region Feature Projector maps those tokens into Qwen’s embedding space\. Stage 2 trains the region bridge alone with CLIP and Qwen frozen\. The paragraphs below detail each step\.

#### Intermediate transformer block features

GPT4RoI\([Zhang et al\. 2023](https://arxiv.org/html/2608.28707#bib.bib41)\)also draws region features from multiple CLIP ViT\-L/14 layers, selectingℒ=\{14,17,20,23\}\\mathcal\{L\}=\\\{14,17,20,23\\\}, which are the eleventh\-, eighth\-, fifth\-, and second\-to\-last blocks\. That choice is also consistent with prior evidence that earlier ViT layers preserve texture and spatial layout while the last layer largely drops localisation\([Goldblum et al\. 2022](https://arxiv.org/html/2608.28707#bib.bib9)\), that the penultimate CLIP layer typically outperforms the last for MLLMs\([Chen et al\. 2025](https://arxiv.org/html/2608.28707#bib.bib3)\), and that CLIP’s deepest layers favour image–text alignment over visual discriminability\([Zhou et al\. 2025](https://arxiv.org/html/2608.28707#bib.bib42)\)\. For eachℓ∈ℒ\\ell\\in\\mathcal\{L\}, patch tokens \(CLS discarded\) form map𝐅\(ℓ\)\\mathbf\{F\}^\{\(\\ell\)\}\. The image bridge already uses only the final transformer block, which is the most image–text aligned but least localisation\-preserving representation\. The region bridge is therefore designed to read earlier and penultimate maps instead, so its region tokens retain finer spatial and object\-level detail that the final block tends to lose\.

#### CoordConv

ViT positions are only implicit and may weaken after reshape/crop, so we append normalised spatial coordinate channels\([Liu et al\. 2018](https://arxiv.org/html/2608.28707#bib.bib23)\)to each feature map to encode absolute spatial information\. Let𝐅\(ℓ\)\\mathbf\{F\}^\{\(\\ell\)\}denote the level\-ℓ\\ellvisual map; after appending the two coordinate grids𝐱grid\\mathbf\{x\}\_\{\\mathrm\{grid\}\}and𝐲grid\\mathbf\{y\}\_\{\\mathrm\{grid\}\}, the augmented map is denoted𝐅~\(ℓ\)\\tilde\{\\mathbf\{F\}\}^\{\(\\ell\)\}\. In words, this changes the feature vector at each patch embedding position from 1024 channels \(visual content only\) to 1026 channels \(visual content plus absolutexx/yyposition\)\. CoordConv\([Liu et al\. 2018](https://arxiv.org/html/2608.28707#bib.bib23)\)then mixes these coordinate channels with visual channels via a1×11\{\\times\}1channel\-mixing operation\. In ReVA, this mixing is implemented by the per\-level linear layer in Eq\.[2](https://arxiv.org/html/2608.28707#S3.E2)\.

#### Per\-level projection

After CoordConv, each level is a10261026\-d map that still lives in its own representational space: block 14 has seen far fewer self\-attention blocks than block 23, so raw features are not directly comparable\. We therefore insert a learned projection before multi\-head self\-attention\. A linear layer is chosen because it plays two roles at once\. First, applied independently at each of the576576patch embedding positions it is mathematically equivalent to a1×11\{\\times\}1convolution over the24×2424\{\\times\}24map, so it performs the CoordConv channel mixing after spatial channels are appended, mixing position with visual content\. Second, giving each levelℓ\\ellits own weightsWproj\(ℓ\)W^\{\(\\ell\)\}\_\{\\mathrm\{proj\}\}lets shallower and deeper maps learn different transforms into a common feature space, which is what makes multi\-head self\-attention meaningful\. Operating on the sequence layout rather than asConv2dalso keeps the module compatible with the attention that follows:

𝐡s\(ℓ\)=LN\(ℓ\)​\(Wproj\(ℓ\)​𝐟~s\(ℓ\)\+𝐛proj\(ℓ\)\)∈ℝ1026,s∈\{1,…,576\},ℓ∈ℒ\\begin\{split\}\\mathbf\{h\}^\{\(\\ell\)\}\_\{s\}&=\\mathrm\{LN\}^\{\(\\ell\)\}\\\!\\left\(W^\{\(\\ell\)\}\_\{\\mathrm\{proj\}\}\\,\\tilde\{\\mathbf\{f\}\}^\{\(\\ell\)\}\_\{s\}\+\\mathbf\{b\}^\{\(\\ell\)\}\_\{\\mathrm\{proj\}\}\\right\)\\in\\mathbb\{R\}^\{1026\},\\\\ &\\qquad s\\in\\\{1,\\ldots,576\\\},\\;\\ell\\in\\mathcal\{L\}\\end\{split\}\(2\)LayerNorm is chosen next because the four depths differ not only in semantics but in typical activation scale; without normalisation, multi\-head self\-attention would be biased by magnitude rather than by content\.LN\(ℓ\)\\mathrm\{LN\}^\{\(\\ell\)\}places every level on a consistent scale so that the subsequent multi\-head self\-attention compares compatible vectors\.

#### Multi\-head self\-attention

With all four depths in a shared space, we apply multi\-head self\-attention across levels at every patch embedding position before RoI Align\. Stacking the projected vectors at patch embedding positionssgives a short sequence𝐇s∈ℝ4×1026\\mathbf\{H\}\_\{s\}\\in\\mathbb\{R\}^\{4\\times 1026\}; multi\-head self\-attention over this length\-44sequence mixes the four depths so that shallow texture and deeper semantics exchange information at the same patch embedding position:

𝐇^s=LN⁡\(𝐇s\+MultiHeadAttn⁡\(𝐇s,𝐇s,𝐇s\)\)\\hat\{\\mathbf\{H\}\}\_\{s\}=\\mathrm\{LN\}\\\!\\left\(\\mathbf\{H\}\_\{s\}\+\\mathrm\{MultiHeadAttn\}\(\\mathbf\{H\}\_\{s\},\\mathbf\{H\}\_\{s\},\\mathbf\{H\}\_\{s\}\)\\right\)\(3\)We use six heads and dropout0\.10\.1, with a residual connection and LayerNorm as in a standard transformer block\([Vaswani et al\. 2017](https://arxiv.org/html/2608.28707#bib.bib34)\)\. The576576patch embedding positions are processed independently \(effective batchB×576B\{\\times\}576\), so attention never mixes different patch embedding positions—only the four ViT depths at the same patch embedding position\. The attended sequences are then reshaped back to enriched per\-level features𝐅^\(ℓ\)\\hat\{\\mathbf\{F\}\}^\{\(\\ell\)\}\.

#### RoI Align and multi\-token attention pooling

RoI Align\([He et al\. 2017](https://arxiv.org/html/2608.28707#bib.bib11)\)generally crops each proposed box to a fixedn×nn\{\\times\}nfeature window, preserving the features inside the box\. In ReVA we setn=14n\{=\}14\(scale24/33624/336\) to avoid aggressive feature compression before pooling while keeping compute manageable, and to convert box coordinates from the 336\-pixel image grid to the24×2424\{\\times\}24grid of patch embedding positions before cropping:

𝐑b\(ℓ\)=RoIAlign⁡\(𝐅^\(ℓ\),b,14×14\)∈ℝ1026×14×14,ℓ∈ℒ\\mathbf\{R\}^\{\(\\ell\)\}\_\{b\}=\\mathrm\{RoIAlign\}\\\!\\left\(\\hat\{\\mathbf\{F\}\}^\{\(\\ell\)\},\\,b;\\,14\{\\times\}14\\right\)\\in\\mathbb\{R\}^\{1026\\times 14\\times 14\},\\;\\ell\\in\\mathcal\{L\}\(4\)Wherebbis the bounding box\. During region\-alignment training, GT boxes from the Stage 2 training corpora are used, and inference uses box proposals produced by the region detector\. These per\-level cropped features are then pooled intoKKtokens per level: each level has its ownK=16K\{=\}16learnable queries that soft\-attend over the196196cells\. Here each learnable queryk∈\{1,…,K\}k\\in\\\{1,\\ldots,K\\\}at depthℓ\\ellproduces one pooled vector𝐳b,k\(ℓ\)\\mathbf\{z\}^\{\(\\ell\)\}\_\{b,k\}:

αb,k,i\(ℓ\)=softmaxi​\(\(𝐪k\(ℓ\)\)⊤​WK\(ℓ\)​𝐫b,i\(ℓ\)1026\),ℓ∈ℒ𝐳b,k\(ℓ\)=∑i=1196αb,k,i\(ℓ\)​WV\(ℓ\)​𝐫b,i\(ℓ\)∈ℝ1026,ℓ∈ℒ\\begin\{split\}\\alpha^\{\(\\ell\)\}\_\{b,k,i\}&=\\mathrm\{softmax\}\_\{i\}\\\!\\left\(\\frac\{\(\\mathbf\{q\}^\{\(\\ell\)\}\_\{k\}\)^\{\\top\}W^\{\(\\ell\)\}\_\{K\}\\,\\mathbf\{r\}^\{\(\\ell\)\}\_\{b,i\}\}\{\\sqrt\{1026\}\}\\right\),\\;\\ell\\in\\mathcal\{L\}\\\\ \\mathbf\{z\}^\{\(\\ell\)\}\_\{b,k\}&=\\sum\_\{i=1\}^\{196\}\\alpha^\{\(\\ell\)\}\_\{b,k,i\}\\,W^\{\(\\ell\)\}\_\{V\}\\,\\mathbf\{r\}^\{\(\\ell\)\}\_\{b,i\}\\in\\mathbb\{R\}^\{1026\},\\;\\ell\\in\\mathcal\{L\}\\end\{split\}\(5\)

#### Region Feature Projector

Multi\-token attention pooling yieldsKKvectors at each of the four ViT depths\. For every query indexkk, those four level\-specific vectors are concatenated into one vector𝐮b,k∈ℝ4104\\mathbf\{u\}\_\{b,k\}\\in\\mathbb\{R\}^\{4104\}; the Region Feature Projector—a two\-layer MLP with LayerNorm, playing the same role as the Image Feature Projector—then maps it into Qwen space:

𝐮b,k=Concat⁡\(𝐳b,k\(ℓ\)\)∈ℝ4104,ℓ∈ℒ\\displaystyle\\mathbf\{u\}\_\{b,k\}=\\mathrm\{Concat\}\\\!\\left\(\\mathbf\{z\}^\{\(\\ell\)\}\_\{b,k\}\\right\)\\in\\mathbb\{R\}^\{4104\},\\;\\ell\\in\\mathcal\{L\}\(6\)𝐳b,kregion=LN2​\(W2A​LN1​\(GELU⁡\(W1A​𝐮b,k\+𝐛1A\)\)\+𝐛2A\)\\displaystyle\\mathbf\{z\}^\{\\mathrm\{region\}\}\_\{b,k\}=\\mathrm\{LN\}\_\{2\}\\\!\\left\(W^\{A\}\_\{2\}\\,\\mathrm\{LN\}\_\{1\}\\\!\\left\(\\mathrm\{GELU\}\\\!\\left\(W^\{A\}\_\{1\}\\,\\mathbf\{u\}\_\{b,k\}\+\\mathbf\{b\}^\{A\}\_\{1\}\\right\)\\right\)\+\\mathbf\{b\}^\{A\}\_\{2\}\\right\)\(7\)Stage 2 prepends these to region descriptions \(caption loss on text only\) plus InfoNCE\([Chen et al\. 2020](https://arxiv.org/html/2608.28707#bib.bib5)\)\(τ=0\.07\\tau\{=\}0\.07,λitc=0\.5\\lambda\_\{\\mathrm\{itc\}\}\{=\}0\.5\)\. Each ofNNboxes contributesKKtokens beside the576576image tokens\.

### III\-CRegion Detector

At inference, RoI Align in the region bridge relies on automatic zero\-shot bounding boxes to crop regional features\. RAM\+\+ \(Recognize Anything Model\)\([Huang et al\. 2023](https://arxiv.org/html/2608.28707#bib.bib14)\)produces a tag set of what is present in the image \(at most 20 tags\); this tag set is united with spaCy nouns extracted from the question\. The union ensures that relevant nouns in the question are captured \(via spaCy\), while maintaining context about objects within the image \(via RAM\+\+\)\. Grounding DINO\([Liu et al\. 2024b](https://arxiv.org/html/2608.28707#bib.bib24)\)then localises each tag into at most one box\. Near\-duplicate boxes are removed by NMS at IoU≥0\.95\\geq 0\.95, at most 20 boxes are kept, and if nothing is detected the stack falls back to a single whole\-image box\. The boxes are used as crops for RoI Align only; they are not written into the prompt as text\. A sample token layout of the LLM prompt is shown below\.

\[576 image token embeddings\] \[N×\\timesK region token embeddings\]

Answer the question using a single word or phrase\.

Question: <question\>

Answer:

## IVTraining Regime

Training is split into three stages so each bridge learns one alignment before the LLM combines them \(Table[I](https://arxiv.org/html/2608.28707#S4.T1)\)\. Stages 1–2 train the image bridge and the region bridge only, with CLIP and Qwen frozen; Stage 3 freezes the image bridge and the region bridge and updates only LoRA adapters in Qwen\.

Table I:Overview of the three\-stage ReVA training procedure\.†Stage 1: pHash \(Hamming≤4\\leq 4\) against COCO and Visual Genome; flagged pairs treated as false positives; full 558,128\-image pool retained\. Stage 2: pHash \(Hamming≤4\\leq 4\) against POPE, VQAv2 test2015, MMBench, and SEED\-Bench\. Stage 3: hard\-ID exclusion then pHash against the same eval image union\.‡Stage 3 region crops use up to 20 boxes per image across all sources: VQAv2 and A\-OKVQA use COCO train2014 ground\-truth boxes; GQA, Visual7W, and VCR use their own box annotations\.Table II:Principal hyperparameters for the three\-stage ReVA training pipeline\.### IV\-AStage 1: Image Alignment

Stage 1 aligns the image bridge \(Image Feature Projector\) to Qwen on whole images\. We ran pHash\([Zauner 2010](https://arxiv.org/html/2608.28707#bib.bib38)\)\(Hamming≤4\\leq 4\) on LLaVA\-Pretrain \(LCS\-558K\)\([Liu et al\. 2023](https://arxiv.org/html/2608.28707#bib.bib21)\)against COCO\([Lin et al\. 2014](https://arxiv.org/html/2608.28707#bib.bib20)\)and Visual Genome\([Krishna et al\. 2017](https://arxiv.org/html/2608.28707#bib.bib16)\)\. The flagged pairs were false positives, so we retained the full 558,128\-image pool\.

Only the image bridge \(∼\\sim16\.5M parameters\) is trained; CLIP and Qwen stay frozen, with gradient checkpointing on Qwen \(Table[II](https://arxiv.org/html/2608.28707#S4.T2)\)\.

### IV\-BStage 2: Region Alignment

Stage 2 trains the region bridge to describe box contents using decontaminated region–text pairs: COCO names\([Lin et al\. 2014](https://arxiv.org/html/2608.28707#bib.bib20)\), RefCOCO/\+/g expressions\([Yu et al\. 2016](https://arxiv.org/html/2608.28707#bib.bib37)\), Visual Genome attributes\([Krishna et al\. 2017](https://arxiv.org/html/2608.28707#bib.bib16)\)\(capped at 15 regions/image\), and GRIT\-20M\([Peng et al\. 2023](https://arxiv.org/html/2608.28707#bib.bib29)\)phrases\. For this stage, training uses GT box proposals, while inference later uses Grounding DINO box proposals\([Liu et al\. 2024b](https://arxiv.org/html/2608.28707#bib.bib24)\)\. We use only GRIT metadata shard 0, keep rows with grounded boxes and CLIP ViT\-L/14\([Radford et al\. 2021](https://arxiv.org/html/2608.28707#bib.bib31)\)similarity≥0\.30\\geq 0\.30, and cap at 250,000 rows before download\. pHash \(Hamming≤4\\leq 4\) removes overlap with POPE, VQAv2 test2015, MMBench, and SEED\-Bench\. For each \(image, box, description\) we minimized

L=Lcap\+λitc​LitcL=L\_\{\\mathrm\{cap\}\}\+\\lambda\_\{\\mathrm\{itc\}\}\\,L\_\{\\mathrm\{itc\}\}\(8\)i\.e\. teach fluent box captions \(LcapL\_\{\\mathrm\{cap\}\}\) while also pulling matching region/text pairs together in embedding space and pushing mismatches apart \(LitcL\_\{\\mathrm\{itc\}\}, InfoNCE\([Chen et al\. 2020](https://arxiv.org/html/2608.28707#bib.bib5)\)atτ=0\.07\\tau\{=\}0\.07\)\.KKregion tokens are prepended; only description positions receive loss \(region\-token positions are masked\)\. We setλitc=0\.5\\lambda\_\{\\mathrm\{itc\}\}\{=\}0\.5\. The combined region–text pool is much larger for Visual Genome and COCO than for RefCOCO and GRIT, so a uniform shuffle would train mostly on the two large sources\. We therefore draw each training batch so that the four sources contribute roughly equally, even though that means repeating some RefCOCO and GRIT pairs and skipping some Visual Genome and COCO pairs in a given epoch\.

### IV\-CStage 3: LoRA Instruction Tuning

After the image bridge and the region bridge are trained, Stage 3 teaches Qwen to use image and region tokens for VQA\. We use a Stage 3 pool stored after decontamination, built from Visual7W\([Zhu et al\. 2016](https://arxiv.org/html/2608.28707#bib.bib43)\), GQA\([Hudson and Manning 2019](https://arxiv.org/html/2608.28707#bib.bib15)\), VQAv2\([Goyal et al\. 2017](https://arxiv.org/html/2608.28707#bib.bib10)\), VCR\([Zellers et al\. 2019](https://arxiv.org/html/2608.28707#bib.bib39)\), and A\-OKVQA\([Schwenk et al\. 2022](https://arxiv.org/html/2608.28707#bib.bib33)\), with up to 20 boxes per image: VQAv2 and A\-OKVQA use COCO train2014 ground\-truth boxes\([Lin et al\. 2014](https://arxiv.org/html/2608.28707#bib.bib20)\), while GQA, Visual7W, and VCR use their own box annotations\. Rather than sampling uniformly from the decontaminated pool—which is heavily skewed toward GQA and VQAv2—we target a fixed mix of Visual7W, GQA, VQAv2, VCR, and A\-OKVQA at 35%, 30%, 15%, 10%, and 10%, ensuring adequate exposure to pointing \(Visual7W\), spatial awareness \(GQA\), open\-ended perception \(VQAv2\), commonsense reasoning \(VCR\), and MCQ\-style answering \(A\-OKVQA\)\.

LoRA\([Hu et al\. 2022](https://arxiv.org/html/2608.28707#bib.bib13)\)updates Qwen attention and MLP layers; the image bridge and the region bridge stay frozen\. Training runs for 10,000 optimiser steps with a weighted sampler that applies those source weights \(Table[II](https://arxiv.org/html/2608.28707#S4.T2)\)\.

## VExperimental Setup

### V\-ABenchmarks and Metrics

We report four primary Stage 3 suites, chosen because they stress complementary failure modes of whole\-image MLLMs and are widely used in the literature we compare against\.

VQAv2\([Goyal et al\. 2017](https://arxiv.org/html/2608.28707#bib.bib10)\)\(test\-dev / test\-standard\) measures open\-ended short\-answer accuracy with the official soft\-scoring protocol\. Each question has ten human answers; a prediction that matches at least three annotators receives full credit, while rarer matches receive partial credit\. We follow the official punctuation and digit/article normalisation ofvqaEval\.py\. Gains here indicate that region tokens do not hurt broad question answering\.

MMBench\([Liu et al\. 2024c](https://arxiv.org/html/2608.28707#bib.bib25)\)\(English\) is a multiple\-choice suite covering perception and reasoning skills\. Accuracy under CircularEval \(shuffled options\) reduces lucky guessing and tests whether the model follows the intended choice, not just fluent text\.

POPE\([Li et al\. 2023c](https://arxiv.org/html/2608.28707#bib.bib19)\)probes object hallucination with yes/no questions on COCO in random, popular, and adversarial splits\. We report per\-split and mean F1 \(Table[V](https://arxiv.org/html/2608.28707#S6.T5)\): precision/recall matter because models that over\-answer “yes” can look accurate while hallucinating\. Adversarial POPE is especially relevant to ReVA’s claim that local evidence should suppress false object claims\.

SEED\-Bench\([Li et al\. 2023a](https://arxiv.org/html/2608.28707#bib.bib17)\)image split \(SEED\-Image; evaluation dimensions 1–9\) is a broad multiple\-choice comprehension benchmark\. We report image\-only accuracy, not SEED All, because ReVA is an image VQA system and our evaluator excludes video questions\. Overall SEED\-Image accuracy summarises whether region grounding preserves general multimodal competence\.

Stage 2 and Stage 3 training pools are pHash\-decontaminated against the reported eval image sets; Stage 3 also drops exact question/image ID overlaps\. All reported evaluations use greedy decoding\. Where instance\-level predictions are available, we additionally report two\-sided pairedtt\-tests and McNemar tests on matched questions between the image\-token baseline and the full system \(Tables[VI](https://arxiv.org/html/2608.28707#S6.T6)and[VII](https://arxiv.org/html/2608.28707#S6.T7)\)\.

### V\-BImplementation

Experiments use PyTorch, Hugging Face Transformers, and PEFT LoRA\. Inference proposals: Grounding DINO \(Swin\-T OGC\) and RAM\+\+ \(Swin\-L\) on NVIDIA A100 GPUs \(bf16\)\.

## VIResults and Ablations

### VI\-AImage Backbone Ablation

Stage 1 trains only the image bridge, so the backbone decision is independent of the region bridge and of LoRA\. Table[III](https://arxiv.org/html/2608.28707#S6.T3)compares frozen encoders under that setting; all runs use Qwen2\.5\-7B\-Instruct, and∗marks Stage 1 zero\-shot with frozen CLIP/Qwen\. Raising CLIP from 224 to 336 px improves GQA val by 3\.90 points and VQAv2 val by 3\.75, consistent with a24×2424\{\\times\}24\(576\-token\) grid versus16×1616\{\\times\}16\(256\)\. DINOv2\-ViT\-L/14\([Oquab et al\. 2024](https://arxiv.org/html/2608.28707#bib.bib28)\)trails CLIP\-ViT\-L/14\-336\. We therefore fix CLIP ViT\-L/14 at 336 px for all later stages\.

Table III:Image backbone ablation \(Stage 1, image alignment only\)\. All models use Qwen2\.5\-7B\-Instruct\.∗Stage 1 zero\-shot: frozen CLIP/Qwen; only the image bridge \(Image Feature Projector\) is trained\.
### VI\-BImage\-Token Baseline vs\. Full System

The central comparison is whether region tokens help after LoRA teaches Qwen to use them\. The*image\-token*baseline keeps the Stage 1 image bridge and Stage 3 LoRA\([Hu et al\. 2022](https://arxiv.org/html/2608.28707#bib.bib13)\), but omits region tokens; the*full*system adds the Stage 2 region bridge and, at inference, RAM\+\+\([Huang et al\. 2023](https://arxiv.org/html/2608.28707#bib.bib14)\)/Grounding DINO\([Liu et al\. 2024b](https://arxiv.org/html/2608.28707#bib.bib24)\)boxes\. Table[IV](https://arxiv.org/html/2608.28707#S6.T4)reports that comparison against published MLLMs on VQAv2, MMBench, and SEED\-Image; Table[V](https://arxiv.org/html/2608.28707#S6.T5)isolates object hallucination on POPE\. All accuracy values use the protocols in Section[V\-A](https://arxiv.org/html/2608.28707#S5.SS1): VQAv2 soft accuracy \(test\-dev / test\-standard\), MMBench ENdevunder CircularEval, and SEED\-Image \(dims 1–9\), not SEED All\.Δ\\Deltais the difference between ReVA with image and region tokens and ReVA with image tokens only\.

Table IV:Comparison with state\-of\-the\-art multimodal LLMs on VQAv2, MMBench, and SEED\-Bench\. Values are accuracy \(%\); SEED\-Bench is image\-only \(dims 1–9\), not SEED All\.Δ\\Deltais the difference between ReVA with image and region tokens and ReVA with image tokens only\.∗Benchmark zero\-shot; unmarked VQAv2 rows observe VQAv2 train images\.†LLaVA\-1\.5 re\-evaluation\([Liu et al\. 2024a](https://arxiv.org/html/2608.28707#bib.bib22)\);§DeepSeek\-VL\([Lu et al\. 2024](https://arxiv.org/html/2608.28707#bib.bib26)\);‡DeepSeek\-VL2\([Wu et al\. 2024](https://arxiv.org/html/2608.28707#bib.bib35)\);♯SEED\-Bench \(img\) from LLaVA\-1\.5\([Liu et al\. 2024a](https://arxiv.org/html/2608.28707#bib.bib22)\)Table 4\.Table V:POPE F1 \(%\) on COCO \(random / popular / adversarial; Mean is the unweighted average\)\.∗Benchmark zero\-shot;§from DeepSeek\-VL\([Lu et al\. 2024](https://arxiv.org/html/2608.28707#bib.bib26)\)\.Δ\\Deltais the difference between ReVA with image and region tokens and ReVA with image tokens only\. Baselines from\([Liu et al\. 2024a](https://arxiv.org/html/2608.28707#bib.bib22);[You et al\. 2024](https://arxiv.org/html/2608.28707#bib.bib36);[Zhang et al\. 2024](https://arxiv.org/html/2608.28707#bib.bib40);[Lu et al\. 2024](https://arxiv.org/html/2608.28707#bib.bib26)\)\.On VQAv2, the full system raises soft accuracy from 72\.89 to 73\.28 on test\-dev \(\+0\.39\+0\.39\) and from 73\.14 to 73\.79 on test\-standard \(\+0\.65\+0\.65\)\. Region tokens leave broad multiple\-choice competence essentially unchanged: MMBench drops by1\.11\.1points while SEED\-Image rises by0\.180\.18\. The MMBench dip is a skill\-level trade\-off rather than a uniform regression: localisation and attribute comparison improve, while coarse perception and relation reasoning fall \(Section[VII](https://arxiv.org/html/2608.28707#S7)\)\. The clearer signal is POPE \(Table[V](https://arxiv.org/html/2608.28707#S6.T5)\), where the full system improves mean F1 by\+1\.71\+1\.71over the image\-token baseline \(random\+2\.11\+2\.11, popular\+1\.45\+1\.45, adversarial\+1\.57\+1\.57\), consistent with local evidence helping suppress in\-image object hallucination\. Section[VI\-C](https://arxiv.org/html/2608.28707#S6.SS3)tests that claim with instance\-level pairedtt\-tests and McNemar tests\. Against open\-source peers, ReVA remains below stronger VQAv2 numbers from models that train on VQAv2 images more aggressively, but the image\-token vs\. full contrast is the relevant control for the region\-token claim\. ReVA rows use greedy decoding with option\-letter matching; MMBench is scored with VLMEvalKit\([Duan et al\. 2024](https://arxiv.org/html/2608.28707#bib.bib8)\)CircularEval, and ReVA’s 66\.57 is SEED\-Image from our evaluator \(data\_type=image,question\_type\_id∈\{1,…,9\}\\in\\\{1,\\ldots,9\\\}\)\.

### VI\-CStatistical Significance

To test whether region tokens improve localised grounding and reduce hallucination, we compare the image\-token baseline and the full ReVA system on matched questions using binary correctness \(Tables[VI](https://arxiv.org/html/2608.28707#S6.T6)and[VII](https://arxiv.org/html/2608.28707#S6.T7)\)\. We report a two\-sided pairedtt\-test \(H0:mean⁡\(d\)=0H\_\{0\}\{:\}\\,\\mathrm\{mean\}\(d\)\{=\}0\) and McNemar’s test on discordant pairs \(H0:P⁡\(b01\)=P⁡\(b10\)H\_\{0\}\{:\}\\,P\(b\_\{01\}\)\{=\}P\(b\_\{10\}\)\), both atα=0\.01\\alpha\{=\}0\.01\. Hereb01b\_\{01\}counts questions the base got right and the final system got wrong \(region tokens flipped a correct answer\), whileb10b\_\{10\}counts questions the base got wrong and the final system got right \(region tokens fixed an incorrect answer\)\. POPE is pooled across random/popular/adversarial as accuracy \(not the F1 in Table[V](https://arxiv.org/html/2608.28707#S6.T5)\); MMBench uses per\-question JSONL correctness rather than the CircularEval Overall in Table[IV](https://arxiv.org/html/2608.28707#S6.T4); SEED\-Image uses dims 1–9\. VQAv2 is excluded from these paired tests because labeled per\-question JSONL dumps were unavailable; Table[VI](https://arxiv.org/html/2608.28707#S6.T6)therefore lists EvalAI overall soft scores only\. Fig\.ReVA: A Region\-Aware Visual Assistant for Visually Grounded Question Answeringshows the corresponding null distributions\.

Table VI:Pairedtt\-test \(Base: image tokens; Final: image \+ region\)\.†Significant atα=0\.01\\alpha\{=\}0\.01\.Table VII:McNemar test \(Base: image tokens; Final: image \+ region\) on discordant pairs\.b01b\_\{01\}: base correct, final incorrect \(region tokens flipped a previously correct answer\);b10b\_\{10\}: base incorrect, final correct \(region tokens fixed a previously wrong answer\)\.†Significant atα=0\.01\\alpha\{=\}0\.01\.On POPE accuracy, the full system gains\+1\.18\+1\.18percentage points \(pt=3\.165×10−6p\_\{t\}\{=\}3\.165\{\\times\}10^\{\-6\}; McNemarp=3\.961×10−6p\{=\}3\.961\{\\times\}10^\{\-6\}\), significant atα=0\.01\\alpha\{=\}0\.01under both tests\. MMBench and SEED\-Image show no significant change \(pt=0\.1615p\_\{t\}\{=\}0\.1615and0\.54920\.5492; McNemarp=0\.177p\{=\}0\.177and0\.56460\.5646\)\. Overall, region tokens help most on object\-hallucination probes, where local evidence matters, and do not significantly change broad multiple\-choice scores on MMBench or SEED\-Image\.

## VIIDiscussion

#### Decontamination and data scale

We ran pHash \(Hamming≤4\\leq 4\) on LCS\-558K against COCO and Visual Genome\. The flagged pairs were false positives, so we retained the full 558,128\-image pool \(Table[VIII](https://arxiv.org/html/2608.28707#S7.T8), Kept==Original\)\. Stage 2 training images were decontaminated against POPE, VQAv2, MMBench, and SEED\-Bench using pHash\. Stage 3 fine\-tuning images were decontaminated against POPE, VQAv2, MMBench, and SEED\-Bench using hard\-ID exclusion then pHash\. Although the risk of contamination is lowered for Stages 2–3, residual overlap with evaluation images remains possible; Table[VIII](https://arxiv.org/html/2608.28707#S7.T8)reports how much was removed, not that the remaining pools are fully clean\.

Table VIII:Training\-pool size after decontamination\.
#### MMBench skill trade\-off

The CircularEval Overall drop of1\.11\.1points on MMBench \(Table[IV](https://arxiv.org/html/2608.28707#S6.T4)\) is not uniform across skills\. Relative to the image\-token baseline, the full system gains on object localisation \(\+8\.65\+8\.65\), attribute comparison \(\+6\.82\+6\.82\), identity reasoning \(\+4\.45\+4\.45\), fine\-grained perception, and attribute reasoning—skills that benefit from cropped local evidence\. It reduces performance on coarse perception \(−4\.73\-4\.73\), relation reasoning \(−5\.21\-5\.21\), OCR \(−5\.13\-5\.13\), attribute recognition \(−6\.76\-6\.76\), and on image\-quality judgement \(−24\.53\-24\.53\)\. This fits a capacity trade\-off in which region tokens use prefix space for local object evidence at the expense of holistic scene and relation cues\. The Overall drop is not significant under the pairedtt\-test or McNemar test \(Tables[VI](https://arxiv.org/html/2608.28707#S6.T6)and[VII](https://arxiv.org/html/2608.28707#S6.T7)\), so it is better read as a redistribution of skill than a reliable regression\.

## VIIIFuture Work

#### Train–test box gap

Replacing GT boxes with Grounding DINO proposals at training time—or mixing clean and noisy boxes—would shrink the mismatch that appears at inference\. A light region reranker \(score boxes by question relevance before packing the prefix\) is a practical middle ground: keep at mostNNregions without forcing the LLM to attend to every detector proposal\.

#### Architecture and compute

The region bridge adds up to20×K20\{\\times\}Ktokens on top of the 576 image tokens, plus RAM\+\+/Grounding DINO latency, so future variants should prune low\-confidence or overlapping boxes, try a smallerKK, or distill the region bridge into a lighter encoder\. A CoordConv on/off ablation would isolate how much explicit spatial channels matter once RoI Align already crops local features\. Swapping the frozen vision encoder under the same two\-projector recipe would test whether the gains are backbone\-specific\.

#### Broader grounded tasks

The current version of ReVA is trained on short\-phrase VQA\. Further work can extend it to open\-ended long answers, multi\-turn referring dialogue, grounded captioning, and harder visual reasoning that needs several local clues in sequence\.

#### Multi\-seed replication

Tables[VI](https://arxiv.org/html/2608.28707#S6.T6)–[VII](https://arxiv.org/html/2608.28707#S6.T7)compare one checkpoint pair, so the tests measure question\-level reliability rather than stability across training runs\. Retraining Stages 1–3 with several seeds would show whether the POPE gain, and the non\-significant MMBench and SEED\-Image differences, hold more generally\.

## IXConclusion

This work asks whether ReVA answers visual questions more reliably when the language model receives both image tokens and explicit region tokens\. On POPE, the full system improves mean F1 on every split over the image\-token baseline \(Table[V](https://arxiv.org/html/2608.28707#S6.T5)\)\. On VQAv2, soft accuracy rises on both test\-dev and test\-standard, completing the image\-token versus image \+ region comparison alongside POPE, MMBench, and SEED\-Image \(Table[IV](https://arxiv.org/html/2608.28707#S6.T4)\)\. These results suggest that ReVA answers visual questions more reliably when it receives both the whole image and explicit region tokens from multi\-level RoI\-cropped ViT features, improving object\-level grounding and reducing hallucination while largely preserving general VQA capability relative to a whole\-image encoding alone\.

## Acknowledgment

I thank Prof\. Shalom Lappin for his supervision, guidance, and feedback throughout this project\. I am also grateful to the School of Electronic Engineering and Computer Science at Queen Mary University of London, and to the comp\-teach team that made this work possible\.

## References

- Bai et al\. \(2023\)Bai, J\. et al\. \(2023\), ‘Qwen\-vl: A versatile vision\-language model for understanding, localization, text reading, and beyond’,arXiv preprint arXiv:2308\.12966\.
- Chen et al\. \(2025\)Chen, H\., Lin, J\., Chen, X\., Fan, Y\., Dong, J\., Jin, X\., Su, H\., Fu, J\. and Shen, X\. \(2025\), Multimodal language models see better when they look shallower,in‘EMNLP’\.
- Chen et al\. \(2023\)Chen, K\. et al\. \(2023\), Shikra: Unleashing multimodal LLM’s referential dialogue magic,in‘arXiv preprint arXiv:2306\.15195’\.
- Chen et al\. \(2020\)Chen, T\. et al\. \(2020\), A simple framework for contrastive learning of visual representations,in‘ICML’\.
- Dai et al\. \(2023\)Dai, W\. et al\. \(2023\), InstructBLIP: Towards general\-purpose vision\-language models with instruction tuning,in‘NeurIPS’\.
- Dosovitskiy et al\. \(2021\)Dosovitskiy, A\. et al\. \(2021\), An image is worth 16x16 words: Transformers for image recognition at scale,in‘ICLR’\.
- Duan et al\. \(2024\)Duan, H\. et al\. \(2024\), VLMEvalKit: An open\-source toolkit for evaluating large multi\-modality models,in‘ACM MM’\.arXiv:2407\.11691\.
- Goldblum et al\. \(2022\)Goldblum, M\. et al\. \(2022\), ‘What do vision transformers learn? a visual exploration’,arXiv preprint arXiv:2212\.06727\.
- Goyal et al\. \(2017\)Goyal, Y\. et al\. \(2017\), Making the V in VQA matter: Elevating the role of image understanding in visual question answering,in‘CVPR’\.
- He et al\. \(2017\)He, K\. et al\. \(2017\), Mask R\-CNN,in‘ICCV’\.
- Honnibal et al\. \(2020\)Honnibal, M\., Montani, I\., Landeghem, S\. V\. and Boyd, A\. \(2020\), ‘spaCy: Industrial\-strength natural language processing in Python’\.Zenodo\.
- Hu et al\. \(2022\)Hu, E\. J\. et al\. \(2022\), LoRA: Low\-rank adaptation of large language models,in‘ICLR’\.
- Huang et al\. \(2023\)Huang, X\. et al\. \(2023\), ‘Open\-set image tagging with multi\-grained text supervision’,arXiv preprint arXiv:2310\.15200\.
- Hudson and Manning \(2019\)Hudson, D\. A\. and Manning, C\. D\. \(2019\), GQA: A new dataset for real\-world visual reasoning and compositional question answering,in‘CVPR’\.
- Krishna et al\. \(2017\)Krishna, R\. et al\. \(2017\), Visual genome: Connecting language and vision using crowdsourced dense image annotations,in‘IJCV’\.
- Li et al\. \(2023a\)Li, B\. et al\. \(2023a\), ‘SEED\-Bench: Benchmarking multimodal large language models’,arXiv preprint arXiv:2307\.16125\.
- Li et al\. \(2023b\)Li, J\. et al\. \(2023b\), BLIP\-2: Bootstrapping language\-image pre\-training with frozen image encoders and large language models,in‘ICML’\.
- Li et al\. \(2023c\)Li, Y\. et al\. \(2023c\), Evaluating object hallucination in large vision\-language models,in‘EMNLP’\.
- Lin et al\. \(2014\)Lin, T\.\-Y\. et al\. \(2014\), Microsoft COCO: Common objects in context,in‘ECCV’\.
- Liu et al\. \(2023\)Liu, H\. et al\. \(2023\), Visual instruction tuning,in‘NeurIPS’\.
- Liu et al\. \(2024a\)Liu, H\. et al\. \(2024a\), Improved baselines with visual instruction tuning,in‘CVPR’\.arXiv:2310\.03744\.
- Liu et al\. \(2018\)Liu, R\. et al\. \(2018\), An intriguing failing of convolutional neural networks and the CoordConv solution,in‘NeurIPS’\.
- Liu et al\. \(2024b\)Liu, S\. et al\. \(2024b\), Grounding DINO: Marrying DINO with grounded pre\-training for open\-set object detection,in‘ECCV’\.
- Liu et al\. \(2024c\)Liu, Y\. et al\. \(2024c\), MMBench: Is your multi\-modal model an all\-around player?,in‘ECCV’\.
- Lu et al\. \(2024\)Lu, H\. et al\. \(2024\), ‘Deepseek\-VL: Towards real\-world vision\-language understanding’,arXiv preprint arXiv:2403\.05525\.
- Ma et al\. \(2024\)Ma, C\. et al\. \(2024\), Groma: Localized visual tokenization for grounding multimodal large language models,in‘ECCV’\.
- Oquab et al\. \(2024\)Oquab, M\. et al\. \(2024\), ‘DINOv2: Learning robust visual features without supervision’,Transactions on Machine Learning Research\.
- Peng et al\. \(2023\)Peng, Z\. et al\. \(2023\), ‘Kosmos\-2: Grounding multimodal large language models to the world’,arXiv preprint arXiv:2306\.14824\.
- Qwen Team \(2024\)Qwen Team \(2024\), ‘Qwen2\.5 technical report’,arXiv preprint arXiv:2412\.15115\.
- Radford et al\. \(2021\)Radford, A\. et al\. \(2021\), Learning transferable visual models from natural language supervision,in‘ICML’\.
- Rasheed et al\. \(2024\)Rasheed, H\. et al\. \(2024\), GLaMM: Pixel grounding large multimodal model,in‘CVPR’\.
- Schwenk et al\. \(2022\)Schwenk, D\. et al\. \(2022\), A\-okvqa: A benchmark for visual question answering using world knowledge,in‘ECCV’\.
- Vaswani et al\. \(2017\)Vaswani, A\. et al\. \(2017\), Attention is all you need,in‘NeurIPS’\.
- Wu et al\. \(2024\)Wu, Z\. et al\. \(2024\), ‘Deepseek\-VL2: Mixture\-of\-experts vision\-language models for advanced multimodal understanding’,arXiv preprint arXiv:2412\.10302\.
- You et al\. \(2024\)You, H\. et al\. \(2024\), Ferret: Refer and ground anything anywhere at any granularity,in‘ICLR’\.
- Yu et al\. \(2016\)Yu, L\. et al\. \(2016\), Modeling context in referring expressions,in‘ECCV’\.
- Zauner \(2010\)Zauner, C\. \(2010\), Implementation and benchmarking of perceptual image hash functions, Master’s thesis, Upper Austria University of Applied Sciences, Hagenberg Campus\.
- Zellers et al\. \(2019\)Zellers, R\. et al\. \(2019\), From recognition to cognition: Visual commonsense reasoning,in‘CVPR’\.
- Zhang et al\. \(2024\)Zhang, H\. et al\. \(2024\), ‘Ferret\-v2: An improved baseline for referring and grounding with large language models’,arXiv preprint arXiv:2404\.07973\.
- Zhang et al\. \(2023\)Zhang, S\. et al\. \(2023\), ‘GPT4RoI: Instruction tuning large language model on region\-of\-interest’,arXiv preprint arXiv:2307\.03601\.
- Zhou et al\. \(2025\)Zhou, J\., Jiang, J\. and Zhu, Z\. \(2025\), ‘Improving visual discriminability of CLIP for training\-free open\-vocabulary semantic segmentation’,arXiv preprint arXiv:2510\.23894\.
- Zhu et al\. \(2016\)Zhu, Y\. et al\. \(2016\), Visual7W: Grounded question answering in images,in‘CVPR’\.

![[Uncaptioned image]](https://arxiv.org/html/2608.28707v1/figures/qualitative_region_grounding.jpg)![[Uncaptioned image]](https://arxiv.org/html/2608.28707v1/figures/qualitative_spatial_perturbation.jpg)![[Uncaptioned image]](https://arxiv.org/html/2608.28707v1/figures/examples3_visioQA.jpg)![[Uncaptioned image]](https://arxiv.org/html/2608.28707v1/figures/examples4.jpg)

![[Uncaptioned image]](https://arxiv.org/html/2608.28707v1/figures/significance_pope.png)

![[Uncaptioned image]](https://arxiv.org/html/2608.28707v1/figures/significance_mmbench-dev-en.png)

![[Uncaptioned image]](https://arxiv.org/html/2608.28707v1/figures/significance_seed-image.png)