Position Is All You Need: A Free Lunch Token Compression Strategy for MLLM-based Referring Expression Segmentation

# A Free Lunch Token Compression Strategy for MLLM-based Referring Expression Segmentation
Source: [https://arxiv.org/html/2608.26142](https://arxiv.org/html/2608.26142)
###### Abstract

Referring Expression Segmentation \(RES\) aims to generate pixel\-wise segmentation masks from complex and implicit textual queries\. While recent advances in Multimodal Large Language Models \(MLLMs\) have substantially boosted RES performance, their prohibitive computational overhead remains a critical bottleneck, which, however, is rarely explored\. To fill this gap, we first evaluate typical token compression methods on this task and observe a surprising performance degradation\. In this paper, we aim to understand this phenomenon for a solution\. By extensive experiments, we find that token compression for RES requires preserving the original position embeddings and local neighboring spatial structures, indicating that visual token position information is far more critical than in other tasks\. Building on this insight, we ask:Can we design the token compression methodpurely based on the position information?Therefore, we propose PAYN, a plug\-and\-play, training\-free token compression method that relies solely on position information\. PAYN retains tokens that are adequately distributed in every local neighboring region while strictly preserving original positional indices, thereby maintaining spatial relational consistency\. Experiments on multiple RES benchmarks demonstrate that our method outperforms existing token compression methods, verifying that position is indeed all you need for token compression in the MLLM\-based RES task\. Codes are avaliable at[https://github\.com/YuhanLiu231/PAYN](https://github.com/YuhanLiu231/PAYN)\.

Machine Learning, ICML

![Refer to caption](https://arxiv.org/html/2608.26142v1/x1.png)Figure 1:\(a\) The Referring Expression Segmentation \(RES\) task\. \(b\) While representative token compression methods can preserve over 90% of the original performance on regular vision\-language tasks, they suffer from severe performance degradation on the RES task, retaining barely around 20%, which we aim to understand and handle in this paper\. \(c\) Visualization of the retained tokens and segmentation results of our method and competing method\. Our free\-lunch method, which relies solely on positional information, achieves superior performance\.## 1Introduction

The Referring Expression Segmentation \(RES\) task predicts pixel\-level segmentation masks given complex or implicit textual descriptions\(Huet al\.,[2016](https://arxiv.org/html/2608.26142#bib.bib1); Dinget al\.,[2025](https://arxiv.org/html/2608.26142#bib.bib2)\)\. As shown in Fig\.[1](https://arxiv.org/html/2608.26142#S0.F1)\(a\), unlike traditional category\-based segmentation, which uses labels like “dog”, RES allows free\-form expressions such as “the dog sitting on bench” for object descriptions, offering flexible and user\-friendly segmentation\. To handle this challenging task, current works\(Wanget al\.,[2023](https://arxiv.org/html/2608.26142#bib.bib10); Laiet al\.,[2024](https://arxiv.org/html/2608.26142#bib.bib3); Xiaet al\.,[2024](https://arxiv.org/html/2608.26142#bib.bib4); Piet al\.,[2024](https://arxiv.org/html/2608.26142#bib.bib5); Renet al\.,[2024](https://arxiv.org/html/2608.26142#bib.bib6); Rasheedet al\.,[2024](https://arxiv.org/html/2608.26142#bib.bib9); Lanet al\.,[2025](https://arxiv.org/html/2608.26142#bib.bib7); Weiet al\.,[2025](https://arxiv.org/html/2608.26142#bib.bib8)\)leverage Multimodal Large Language Models \(MLLMs\)\(Baiet al\.,[2023](https://arxiv.org/html/2608.26142#bib.bib12); Liuet al\.,[2024](https://arxiv.org/html/2608.26142#bib.bib38); Luet al\.,[2024](https://arxiv.org/html/2608.26142#bib.bib11); Chenet al\.,[2024c](https://arxiv.org/html/2608.26142#bib.bib13)\)for the understanding of complex text and visual inputs, focusing on designing effective multimodal representation formats for segmentation decoding\. However, due to the quadratic scaling of attention computation with input sequence length, MLLMs suffer from high inference cost\(Chenet al\.,[2024a](https://arxiv.org/html/2608.26142#bib.bib14); Linet al\.,[2025](https://arxiv.org/html/2608.26142#bib.bib15); Yanget al\.,[2025b](https://arxiv.org/html/2608.26142#bib.bib40)\), which limits the practical deployment of RES\. To the best of our knowledge, few works have studied the acceleration for the MLLM\-based RES task\.

To fill up this underexplored area, we begin with token compression methods as a widely\-adopted acceleration strategy\(Lianget al\.,[2022](https://arxiv.org/html/2608.26142#bib.bib16); Bolyaet al\.,[2023](https://arxiv.org/html/2608.26142#bib.bib39)\)for MLLMs, which typically reduces the inference cost by removing or merging substantial redundant visual tokens\(Wanget al\.,[2025](https://arxiv.org/html/2608.26142#bib.bib17); Yanget al\.,[2025a](https://arxiv.org/html/2608.26142#bib.bib31); Chenet al\.,[2026](https://arxiv.org/html/2608.26142#bib.bib32); Zhanget al\.,[2026](https://arxiv.org/html/2608.26142#bib.bib34); Tonget al\.,[2026](https://arxiv.org/html/2608.26142#bib.bib33)\)\. However, by applying prevailing token compression methods to MLLM\-based RES models, we surprisingly find that these methods lead to severe performance degradation, as shown in Fig\.[1](https://arxiv.org/html/2608.26142#S0.F1)\(b\)\. When retaining the same portion of visual tokens, these methods can maintain over 90% of the original performance on regular vision\-language tasks, but can hardly maintain even about 20% of the original performance on the RES task, leading to almost collapsed results after the token compression\.

In this paper, we aim to understand why this problem occurs, and specifically design token compression methods for the MLLM\-based RES task based on this interpretation\. We first take a detailed evaluation of current token compression methods on the RES task\. By \(coarsely\) dividing current methods into attention/similarity\-based ones\(Bolyaet al\.,[2023](https://arxiv.org/html/2608.26142#bib.bib39); Yanget al\.,[2025b](https://arxiv.org/html/2608.26142#bib.bib40); Shanget al\.,[2025](https://arxiv.org/html/2608.26142#bib.bib41)\)and diversity\-based ones\(Wenet al\.,[2025](https://arxiv.org/html/2608.26142#bib.bib42); Alvaret al\.,[2025](https://arxiv.org/html/2608.26142#bib.bib43); Zhanget al\.,[2025a](https://arxiv.org/html/2608.26142#bib.bib44)\), we find that the diversity\-based ones show much better performance, where the core reason is the maintenance of original position embeddings, as widely adopted in diversity\-based methods\. To further understand why position embeddings are much more important in the RES task than in other tasks, through extensive experiments, we find that it is because the RES task requires the precise prediction ofevery visual token’s label, while other tasks typically only require the understanding of thesemantic part of visual tokens, which makes a small perturbation of local neighbor positions \(e\.g\., not maintaining original position embeddings\) influences much more on the RES task than on other tasks\. This leads to our insight for the token compression methods in MLLM\-based RES task:position information of visual tokens is much more important than in other tasks\.

Based on this insight, now that the position information is much more important,can we design the token compression methodpurely based on the position information? Therefore, we propose PAYN, a plug\-and\-play, training\-free token compression method that relies solely on positional information without considering any semantic content, effectively serving as a free\-lunch solution, as illustrated in Fig\.[1](https://arxiv.org/html/2608.26142#S0.F1)\(c\)\. Specifically, our method strictly preserves the original positional indices while aiming to retain tokens that are spatially averagely distributed, thus adequately preserving local neighboring spatial structures\. We provide two instantiations of this approach: checkerboard\-style spatial sampling and farthest point sampling\. Extensive experiments on multiple RES benchmarks demonstrate that our method outperforms existing state\-of\-the\-art token pruning approaches, verifying thatposition is all you need for token compression in the MLLM\-based RES task\.

In summary, our key contributions are as follows:

- •We find that existing token compression methods suffer from severe performance degradation in the MLLM\-based RES task, which is underexplored\.
- •Through experiments and analysis, we observe that positional information of visual tokens plays a substantially more critical role in RES than in other tasks\.
- •Guided by this observation, we propose PAYN, a simple but effective free\-lunch token compression method that relies solely on positional information\.
- •Extensive experiments demonstrate that our method outperforms existing SOTA token pruning approaches, validating that position is all you need for token compression in the MLLM\-based RES task\.

## 2Related Work

### 2\.1MLLM\-Based Referring Expression Segmentation

Referring Expression Segmentation \(RES\) aims to segment the pixels of a target object in an image based on a natural language referring expression\(Laiet al\.,[2024](https://arxiv.org/html/2608.26142#bib.bib3); Zhanget al\.,[2024](https://arxiv.org/html/2608.26142#bib.bib20); Chenet al\.,[2024b](https://arxiv.org/html/2608.26142#bib.bib21); Wuet al\.,[2024](https://arxiv.org/html/2608.26142#bib.bib22); Qianet al\.,[2025](https://arxiv.org/html/2608.26142#bib.bib25); Zhuet al\.,[2025a](https://arxiv.org/html/2608.26142#bib.bib26); Liuet al\.,[2026](https://arxiv.org/html/2608.26142#bib.bib27)\)\. With the rapid development of multimodal large language models \(MLLMs\)\(Liuet al\.,[2024](https://arxiv.org/html/2608.26142#bib.bib38)\)and powerful segmentation models like SAM\(Kirillovet al\.,[2023](https://arxiv.org/html/2608.26142#bib.bib23)\), significant progress has been made in RES\. LISA\(Laiet al\.,[2024](https://arxiv.org/html/2608.26142#bib.bib3)\)pioneers the embedding\-as\-mask paradigm by extending the MLLM vocabulary with a special<<seg\>\>token to guide a segmentation decoder\. Following this paradigm, several subsequent works extend MLLM\-based RES from different perspectives\. GSVA\(Xiaet al\.,[2024](https://arxiv.org/html/2608.26142#bib.bib4)\)employs multiple<<seg\>\>tokens along with a<<rej\>\>token to handle multiple objects and reject null targets\. PixelLM\(Renet al\.,[2024](https://arxiv.org/html/2608.26142#bib.bib6)\)replace SAM with a lightweight pixel decoder and introduce a learnable segmentation codebook for mask generation\. InstructSeg\(Weiet al\.,[2025](https://arxiv.org/html/2608.26142#bib.bib8)\)provides a unified framework for performing language\-instructed segmentation across both images and videos\. In contrast to these embedding\-as\-mask approaches, Text4Seg\(Lanet al\.,[2025](https://arxiv.org/html/2608.26142#bib.bib7)\)proposes a text\-as\-mask paradigm that reformulates image segmentation as a text generation problem by encoding images into sequences of semantic textual descriptors\. Another line of work\(Wanget al\.,[2023](https://arxiv.org/html/2608.26142#bib.bib10); Penget al\.,[2024](https://arxiv.org/html/2608.26142#bib.bib24)\)leverage MLLMs to produce polygon coordinates for mask prediction, which often struggle to achieve satisfactory performance\.

### 2\.2Visual Token Compression

In MLLMs, visual signals exhibit much higher spatial redundancy than text, motivating visual token compression to improve efficiency\. From the perspective of optimization requirements, some methods require finetuning the model\(Caiet al\.,[2025](https://arxiv.org/html/2608.26142#bib.bib30); Yeet al\.,[2025b](https://arxiv.org/html/2608.26142#bib.bib49)\), some determine pruning strategies using a calibration dataset\(Linet al\.,[2025](https://arxiv.org/html/2608.26142#bib.bib15); Yeet al\.,[2025a](https://arxiv.org/html/2608.26142#bib.bib28)\), while others operate in a training\-free manner\(Yanget al\.,[2025b](https://arxiv.org/html/2608.26142#bib.bib40); Zhanget al\.,[2025b](https://arxiv.org/html/2608.26142#bib.bib18)\)\. From the perspective of compression mechanisms, existing methods can be broadly categorized into similarity\-based approaches, e\.g\., ToMe\(Bolyaet al\.,[2023](https://arxiv.org/html/2608.26142#bib.bib39)\), diversity\-based approaches, e\.g\., Dart\(Wenet al\.,[2025](https://arxiv.org/html/2608.26142#bib.bib42)\), DivPrune\(Alvaret al\.,[2025](https://arxiv.org/html/2608.26142#bib.bib43)\), and attention\-based approaches, e\.g\., VisionZip\(Yanget al\.,[2025b](https://arxiv.org/html/2608.26142#bib.bib40)\), PruMerge\(Shanget al\.,[2025](https://arxiv.org/html/2608.26142#bib.bib41)\), Vispruner\(Zhanget al\.,[2025a](https://arxiv.org/html/2608.26142#bib.bib44)\)\. Similarity\-based methods merge highly similar tokens into fewer representative tokens, for example, ToMe implements this using bipartite matching\. Diversity\-based methods also exploit token similarity, but instead aim to retain tokens that are maximally different from each other\. For instance, DivPrune formulates token compression as a Max\-Min diversity problem, selecting a subset of tokens with the largest internal differences\. Attention\-based methods retain high\-attention tokens while pruning low\-attention ones, and can be combined with similarity\- or diversity\-based methods\. While these methods perform well on sparse prediction tasks, visual token compression for dense prediction tasks such as RES is largely underexplored\(Konget al\.,[2025](https://arxiv.org/html/2608.26142#bib.bib36)\)\. In this work, we propose a training\-free, plug\-and\-play token compression method tailored for RES\.

## 3Rethinking Token Compression for Referring Expression Segmentation

### 3\.1Preliminary

MLLM\-based Referring Expression Segmentation \(RES\) aims to generate a pixel\-level segmentation maskℳ\\mathcal\{M\}for a target object in an imageII, conditioned on a natural\-language referring expressionTT\. The input imageIIis encoded by a vision encoder \(e\.g\., SigLIP\(Zhaiet al\.,[2023](https://arxiv.org/html/2608.26142#bib.bib37)\)\) into a sequence of visual tokens𝐗v∈ℝM×d\\mathbf\{X\}\_\{v\}\\in\\mathbb\{R\}^\{M\\times d\}, while the referring expressionTTis encoded into a sequence of language tokens𝐗t∈ℝN×d\\mathbf\{X\}\_\{t\}\\in\\mathbb\{R\}^\{N\\times d\}\. Typically, the number of visual tokensMMis much larger than the number of text tokensNN, leading to substantial computational overhead when processed by the Large Language Model \(LLM\)\. To this end, a compression operatorℱc​o​m​p\\mathcal\{F\}\_\{comp\}is employed to project the raw visual sequence into a compact representation𝐂\\mathbf\{C\}:

𝐂=ℱc​o​m​p​\(𝐗v\),s\.t\.​\|𝐂\|<\|𝐗v\|\\mathbf\{C\}=\\mathcal\{F\}\_\{comp\}\(\\mathbf\{X\}\_\{v\}\),\\quad\\text\{s\.t\. \}\|\\mathbf\{C\}\|<\|\\mathbf\{X\}\_\{v\}\|\(1\)Here𝐂=\{ck\}k=1K\\mathbf\{C\}=\\\{c^\{k\}\\\}\_\{k=1\}^\{K\}denotes a set ofKKcompressed visual tokens\. These tokens are concatenated with the language tokens𝐗t\\mathbf\{X\}\_\{t\}and fed into LLM:

𝐇o​u​t=LLM​\(Concat​\(𝐗t,𝐂\)\)\.\\mathbf\{H\}\_\{out\}=\\text\{LLM\}\(\\text\{Concat\}\(\\mathbf\{X\}\_\{t\},\\mathbf\{C\}\)\)\.\(2\)𝐇o​u​t\\mathbf\{H\}\_\{out\}denotes the high\-level guidance for the segmentation task, which may take the form of<<seg\>\>embeddings, textual representations, or other structured signals\. Finally, the segmentation maskℳ\\mathcal\{M\}is derived through a decoder \(e\.g\. SAM\) conditioned on both the original image featuresfi​m​gf\_\{img\}and the guidance𝐇o​u​t\\mathbf\{H\}\_\{out\}:

ℳ=Decoder​\(fi​m​g,𝐇o​u​t\)\\mathcal\{M\}=\\text\{Decoder\}\(f\_\{img\},\\mathbf\{H\}\_\{out\}\)\(3\)

### 3\.2Evaluation of Existing Token Compression Methods

In prior experiments, we observed that several existing token compression methods suffer from significant performance degradation on the RES task\. To delve deeper into this phenomenon, we conduct a more systematic evaluation of a broader range of token compression strategies\. All experiments are conducted on a unified RES baseline, namely Text4Seg\(Lanet al\.,[2025](https://arxiv.org/html/2608.26142#bib.bib7)\), with LLaVA\-1\.5\-7B\(Liuet al\.,[2024](https://arxiv.org/html/2608.26142#bib.bib38)\)as the backbone, where the number of visual tokens is reduced from 576 to 192\. From the perspective of their underlying mechanisms, these methods can be broadly divided into four categories: \(1\) similarity\-based\(Bolyaet al\.,[2023](https://arxiv.org/html/2608.26142#bib.bib39)\)\(2\) attention & similarity\-based\(Yanget al\.,[2025b](https://arxiv.org/html/2608.26142#bib.bib40); Shanget al\.,[2025](https://arxiv.org/html/2608.26142#bib.bib41)\)\(3\) diversity\-based\(Wenet al\.,[2025](https://arxiv.org/html/2608.26142#bib.bib42); Alvaret al\.,[2025](https://arxiv.org/html/2608.26142#bib.bib43)\)\(4\) attention & diversity\-based\(Zhanget al\.,[2025a](https://arxiv.org/html/2608.26142#bib.bib44)\)\. The results are shown in Table[1](https://arxiv.org/html/2608.26142#S3.T1)\(top\)\.

Table 1:Performance comparison of visual token compression methods on the RES task\. \(Top\) Results of existing methods, where S, A, and D denote similarity\-based, attention\-based, and diversity\-based token selection criteria respectively\. \(Middle\) Ablation on A & S methods by removing S and evaluating the effect of position ids, where preserving position ids yields significantly better performance\. \(Bottom\) Performance drops when removing position ids from D methods further supporting this observation\.In table[1](https://arxiv.org/html/2608.26142#S3.T1)\(top\), we observe that\(attention&\) diversity\-based methods significantly outperform \(attention&\) similarity\-based ones\. Besides, we also observe that \(attention&\) similarity\-based methods often employ token merging or clustering, which reorganizes tokens into a contiguous sequence with re\-indexed positions \(e\.g\., 0, 1, 2\), but \(attention&\) diversity\-based methods typically perform token pruning, which preserves the original position indices of the retained tokens \(Fig\.[2](https://arxiv.org/html/2608.26142#S3.F2)\)\. Therefore, we hypothesize two potential reasons for this disparity\. \(1\) \(Attention&\) diversity\-based methods are inherently more suitable for the RES task\. \(2\) The two types of methods differ in how positional information is handled after token compression\.

To verify which of the two hypotheses holds, we conduct a set of controlled experiments\. We isolate the attention\-based selection component, while keeping the number of retained tokens unchanged, with and without preserving the original position indices\. The results are shown in Table[1](https://arxiv.org/html/2608.26142#S3.T1)\(middle\)\. We observe that preserving position indices yields a clear performance advantage and achieves results comparable to \(attention &\) diversity\-based methods\.

![Refer to caption](https://arxiv.org/html/2608.26142v1/x2.png)Figure 2:Different strategies in position indices of pruned tokens\.Furthermore, we remove the original position indices in \(attention &\) diversity\-based methods and replace them with contiguous ones\. As shown in Table[1](https://arxiv.org/html/2608.26142#S3.T1)\(bottom\), this leads to a sharp performance drop\. Therefore, these results validate the second hypothesis, and we can conclude that:

Insight 1\.For the referring expression segmentation task, preserving the original positional embedding during token compression is essential\.

### 3\.3The Role of Position in Sparse and Dense Prediction Tasks

Building onInsight 1, we also notice that\(Linet al\.,[2025](https://arxiv.org/html/2608.26142#bib.bib15); Yeet al\.,[2025b](https://arxiv.org/html/2608.26142#bib.bib49)\)mentioned that maintaining the original position indices leads to only around 2% difference in performance\. This inspires us to further investigate why preserving the original positional embedding is crucial for RES, while having a much smaller impact on conventional vision\-language tasks\. We hypothesize that the discrepancy arises from the different sensitivity ofdenseandsparseprediction tasks to local spatial consistency\.

![Refer to caption](https://arxiv.org/html/2608.26142v1/x3.png)Figure 3:Different task requirements of sparse and dense prediction tasks lead to distinct sensitivities to relative positional shifts\. The black boxes indicate simulated positional shifts\.As shown in Fig\.[3](https://arxiv.org/html/2608.26142#S3.F3), sparse prediction tasks, such as Visual Question Answering \(VQA\), primarily rely on a few discriminative semantic patches, which can be spatially distant from each other\. As a result, these tasks are less sensitive to precise relative positions, and therefore suffer less performance degradation when positional indices are altered after token compression\. In contrast, dense prediction tasks like RES require consistent modeling of local neighborhoods to produce pixel\-wise predictions\. If the original positional indices are not preserved during token compression, tokens that are far apart in the image may be assigned adjacent indices, leading to severe distortion of local spatial relationships and consequent performance degradation\.

![Refer to caption](https://arxiv.org/html/2608.26142v1/x4.png)Figure 4:Sensitivity of dense and sparse prediction tasks to local positional perturbations\. \(Left\) Two types of perturbations applied to input images, including continuous TPS\-based deformations and discrete local patch shuffle\. \(Right\) Dense prediction tasks suffer substantially larger performance drops than sparse prediction tasks under the same perturbation strength\.To validate the above hypothesis, we design experiments by applying local spatial perturbations to the input images on both tasks\. Specifically, we introduce two types of perturbations that primarily disrupt local spatial structures while largely preserving semantic information, as illustrated in Fig\.[4](https://arxiv.org/html/2608.26142#S3.F4)\(left\)\. The first one employs Thin Plate Spline \(TPS\) interpolation\(Wood,[2003](https://arxiv.org/html/2608.26142#bib.bib45)\)to induce smooth, continuous spatial deformations\. The second divides image patches into local windows and randomly shuffles the patch order within each window, resulting in discrete local spatial reordering\. We evaluate the impact of these perturbations on representative sparse prediction tasks: MME\(Fuet al\.,[2025](https://arxiv.org/html/2608.26142#bib.bib46)\), ScienceQA\(Luet al\.,[2022](https://arxiv.org/html/2608.26142#bib.bib47)\), and TextVQA\(Singhet al\.,[2019](https://arxiv.org/html/2608.26142#bib.bib48)\), as well as dense prediction tasks across multiple RES datasets\.

As shown in Fig\.[4](https://arxiv.org/html/2608.26142#S3.F4)\(right\),under the same local spatial perturbations, dense prediction tasks exhibit a substantially larger relative performance drop compared to sparse prediction tasks\. This observation indicates that dense prediction tasks are indeed more sensitive to local positional variations, thereby validating our hypothesis:

Insight 2\.Referring expression segmentation critically depends on the preservation of local spatial structures\.

Building uponInsights 1 and 2, which highlight the strong dependence of RES on positional information, we naturally raise the following question:Is position all you need for token compression in the MLLM\-based RES task?

## 4Method

![Refer to caption](https://arxiv.org/html/2608.26142v1/x5.png)Figure 5:Our method consists of two components: \(1\) position\-guided token selection, which retains tokens solely based on positional information to ensure adequate retained tokens in every local neighboring region, and is instantiated by two strategies, Checkerboard and FPS; and \(2\) a position index preserving strategy, which aims to preserve original positional information\.Guided by the two insights and the question posed above, we aim to investigate a token compression method for RES thatrelies solely on positional information, as shown in Fig\.[5](https://arxiv.org/html/2608.26142#S4.F5)\. The approach is fully plug\-and\-play, training\-free, and does not even require attention maps or feature information, effectively serving as a free\-lunch solution\.

### 4\.1Position\-Guided Token Selection

According toInsight 2, RES is particularly sensitive to local spatial variations and relies heavily on the preservation of local spatial structures\. To ensure that all local regions are adequately represented after compression, we design to ensure enough maintained tokens ineverylocal neighboring region\. Therefore, we need to ensure broad and uniform spatial coverage of maintained tokens across the image, so we retain tokens that are maximally separated in the spatial positions\. Below, we provide two instantiations of this position\-guided token selection strategyℱc​o​m​p\\mathcal\{F\}\_\{comp\}\.

#### Checkerboard\-Style Spatial Sampling

We adopt a simple and efficient checkerboard\-style spatial sampling strategy on a regular 2D grid of patch\-level visual tokens indexed by spatial coordinates\(x,y\)\(x,y\)\. Given the target number of retained tokensKKand the number of total tokensMM, the retained token setCCis defined as

𝒞=\{\{\(x,y\)\|\(x\+y\)mods=ϕ\},K≤M/2,\{\(x,y\)\|\(x\+y\)mods≠ϕ\},K\>M/2,\\mathcal\{C\}=\\begin\{cases\}\\left\\\{\(x,y\)\\;\\middle\|\\;\(x\+y\)\\bmod s=\\phi\\right\\\},&K\\leq M/2,\\\\\[6\.0pt\] \\left\\\{\(x,y\)\\;\\middle\|\\;\(x\+y\)\\bmod s\\neq\\phi\\right\\\},&K\>M/2,\\end\{cases\}\(4\)whereϕ\\phiis a fixed offset \(set to 0 in our implementation\) that determines the checkerboard sampling pattern\. The stridessis defined as:

s=⌊Mmin⁡\(K,M−K\)⌋\.s=\\left\\lfloor\\dfrac\{M\}\{\\min\(K,\\,M\-K\)\}\\right\\rfloor\.\(5\)The retained tokens preserve their original positional indices\. This strategy results in a spatially uniform, checkerboard\-like distribution of retained tokens, enabling broad coverage of local regions without relying on semantic information\.

#### Farthest Point Sampling \(FPS\)\.

As a more flexible instantiation of position\-guided token selection, especially when the number of retained tokensKKis very small, we perform farthest point sampling \(FPS\) directly in the spatial coordinate space\. Starting from an initial randomly selected token, FPS iteratively selects the token that maximizes its minimum distance to the already selected set\. Formally, at each iterationtt, the next token is chosen as:

ct=argmaxc∈𝐗v∖𝐒t−1minc′∈𝐒t−1∥𝐩c−𝐩c′∥22,c^\{t\}=\\arg\\max\_\{c\\in\\mathbf\{X\}\_\{v\}\\setminus\\mathbf\{S\}\_\{t\-1\}\}\\min\_\{c^\{\\prime\}\\in\\mathbf\{S\}\_\{t\-1\}\}\\lVert\\mathbf\{p\}\_\{c\}\-\\mathbf\{p\}\_\{c^\{\\prime\}\}\\rVert\_\{2\}^\{2\},\(6\)where𝐩c∈ℝ2\\mathbf\{p\}\_\{c\}\\in\\mathbb\{R\}^\{2\}denotes the 2D spatial coordinates of tokencc\. The procedure is repeated until\|𝐒t\|=K\|\\mathbf\{S\}\_\{t\}\|=K, producing a set of well\-dispersed tokens over the spatial domain, while retaining their original positional indices\.

### 4\.2Position Index Preserving

According toInsight 1, preserving the original positional indices of retained image tokens is essential for RES, so we adopt a position index preserving strategy\. Compressed visual tokens are first projected by a multimodal projector and then concatenated with text tokens to form the input sequence for the LLM:

𝐗=\[𝐗s​y​s,𝐗i​m​g,𝐗i​n​s,𝐗o​u​t\]\\mathbf\{X\}=\[\\mathbf\{X\}\_\{sys\},\\mathbf\{X\}\_\{img\},\\mathbf\{X\}\_\{ins\},\\mathbf\{X\}\_\{out\}\]\(7\)where𝐗s​y​s,𝐗i​m​g,𝐗i​n​s,𝐗o​u​t\\mathbf\{X\}\_\{sys\},\\mathbf\{X\}\_\{img\},\\mathbf\{X\}\_\{ins\},\\mathbf\{X\}\_\{out\}denote system prompts, image tokens, user instructions, and output tokens, respectively\. Without position\-preserving reindexing, positional indices are assigned monotonically from0toL−1L\-1, whereLLdenotes the total sequence length\. After compressing, we replace the positions of image tokens by the preserved original indices:

𝐈i​m​gpreserved=\{i1,i2,…,iK\},K<\|𝐗i​m​g\|\\mathbf\{I\}\_\{img\}^\{\\text\{preserved\}\}=\\\{i\_\{1\},i\_\{2\},\\dots,i\_\{K\}\\\},\\quad K<\|\\mathbf\{X\}\_\{img\}\|\(8\)while the positions of subsequent instruction and output tokens are shifted accordingly to maintain the overall sequence order\. Formally, the reindexed positions are:

𝐏r​e​i​n​d​e​x=\[𝐏s​y​s,𝐈i​m​gpreserved,𝐏i​n​sshifted,𝐏o​u​tshifted\]\\mathbf\{P\}\_\{reindex\}=\[\\mathbf\{P\}\_\{sys\},\\mathbf\{I\}\_\{img\}^\{\\text\{preserved\}\},\\mathbf\{P\}\_\{ins\}^\{\\text\{shifted\}\},\\mathbf\{P\}\_\{out\}^\{\\text\{shifted\}\}\]\(9\)Position index preserving guarantees that the relative spatial structure of image tokens is retained in the LLM, enabling more precise dense predictions\. After obtaining the LLM’s output as high\-level guidance, it is fed together with the image into a decoder to produce the segmentation mask\.

## 5Experiments

### 5\.1Experimental Setting

#### Baselines and Models

We adopt two representative RES baselines: Text4Seg\(Lanet al\.,[2025](https://arxiv.org/html/2608.26142#bib.bib7)\)and InstructSeg\(Weiet al\.,[2025](https://arxiv.org/html/2608.26142#bib.bib8)\)\. Text4Seg follows the text\-as\-mask paradigm by encoding images into semantic textual descriptors, with LLaVA\-7B as the MLLM backbone\. In contrast, InstructSeg is a recent state\-of\-the\-art model based on the embedding\-as\-mask paradigm, built upon Mipha\-3B\(Zhuet al\.,[2025b](https://arxiv.org/html/2608.26142#bib.bib50)\)\.

#### Datasets and Metric

Following standard evaluation protocols\(Laiet al\.,[2024](https://arxiv.org/html/2608.26142#bib.bib3)\), we evaluate our method on the RefCOCO, RefCOCO\+\(Kazemzadehet al\.,[2014](https://arxiv.org/html/2608.26142#bib.bib53)\), and RefCOCOg\(Maoet al\.,[2016](https://arxiv.org/html/2608.26142#bib.bib54)\)benchmarks\. These datasets include over 19,000 images for RefCOCO and RefCOCO\+, and 26,711 images for RefCOCOg\. RefCOCO allows expressions that reference both location and appearance attributes\. RefCOCO\+ focuses on appearance\-based descriptions while limiting location cues\. RefCOCOg contains longer and more complex expressions without restrictions on location references\(Dinget al\.,[2025](https://arxiv.org/html/2608.26142#bib.bib2)\)\.

For evaluation, we adopt the cumulative Intersection\-over\-Union \(cIoU\) metric, calculated as the total intersection over the total union of all predicted masks\.

### 5\.2Main Results

We compare our method with other training\-free, calibration\-free token compression methods\(Bolyaet al\.,[2023](https://arxiv.org/html/2608.26142#bib.bib39); Yanget al\.,[2025b](https://arxiv.org/html/2608.26142#bib.bib40); Wenet al\.,[2025](https://arxiv.org/html/2608.26142#bib.bib42); Zhanget al\.,[2025a](https://arxiv.org/html/2608.26142#bib.bib44); Alvaret al\.,[2025](https://arxiv.org/html/2608.26142#bib.bib43); Zhuet al\.,[2026](https://arxiv.org/html/2608.26142#bib.bib55)\)\. Unlike token compression in other tasks, whose baselines are plain MLLM models such as LLaVA, RES baselines vary in model structures and complexity\. We therefore adopt different compression ratios for each RES baseline to balance segmentation performance and inference efficiency\. Table[2](https://arxiv.org/html/2608.26142#S5.T2)reports results on the Text4Seg baseline, shows that our PAYN achieves the best performance across nearly all benchmarks, with its average results substantially outperforming existing methods\. The performance gap is especially noticeable when fewer tokens are retained\. Following\(Zhuet al\.,[2026](https://arxiv.org/html/2608.26142#bib.bib55)\), we further evaluate PAYN on InstructSeg\. As illustrated in Table[3](https://arxiv.org/html/2608.26142#S5.T3), our method continues to outperform alternative approaches\.

Table 2:Performance of PAYN on Text4SegLLaVA\-1\.5\-7B\{\}\_\{\\,\\textup\{LLaVA\-1\.5\-7B\}\}under varying visual token counts\. Avg\. denotes the average cIoU across Referring Expression Segmentation datasets, and Rel\. shows the percentage of performance retained at each token reduction level\.Table 3:Performance of PAYN on InstructSeg under varying visual token counts\.
### 5\.3Ablation Studies

#### Validating thePosition Is All You NeedPerspective

The results in Table[2](https://arxiv.org/html/2608.26142#S5.T2)and Table[3](https://arxiv.org/html/2608.26142#S5.T3)demonstrate that our position\-based approach outperforms attention\-, similarity\-, and diversity\-based token compression approaches\. To explore other factors that might influence token selection, we considered several alternative strategies:

\(1\) Random selection, serving as a baseline; \(2\) Selecting tokens with larger L2 norms, assuming they carry more information; \(3\) Selecting edge or contour tokens to preserve boundary information important for segmentation, using two different edge\-detection operators: Sobel\(Sobelet al\.,[1968](https://arxiv.org/html/2608.26142#bib.bib51)\)and Canny\(Canny,[2009](https://arxiv.org/html/2608.26142#bib.bib52)\); \(4\) Uniform sampling within clusters, where tokens are first clustered and then an equal number of tokens are randomly sampled from each cluster, either on the original image or on patch embeddings; \(5\) Farthest point sampling in the feature space\.

Table 4:Comparison of alternative token selection criteria such as semantic information or edge cues, the original positional indices of the selected tokens are preserved\. The position\-based strategy consistently outperforms all alternatives, answering our earlier question: position is all you need for token compression in the MLLM\-based RES task\.MethodrefCOCOrefCOCO\+refCOCOgAvg\.random64\.757\.560\.160\.8L2 norm52\.947\.650\.450\.3sobel46\.640\.547\.146\.9canny50\.043\.744\.243\.7img clustering64\.056\.659\.360\.0emb clustering63\.956\.758\.559\.8feature FPS65\.859\.461\.462\.3PAYN \(spatial FPS\)69\.261\.864\.265\.2PAYN \(checkerboard\)70\.463\.065\.266\.4

As shown in Table[4](https://arxiv.org/html/2608.26142#S5.T4), our position\-based strategy consistently outperforms all alternative methods\. Methods based on the L2 norm or edge detectors show inferior performance because they tend to select tokens concentrated in high\-activation or boundary regions\. Such selections fail to adequately cover all local neighboring regions, resulting in an imbalanced spatial distribution\. In particular, the comparison between farthest point sampling in the feature space and in the spatial space highlights the dominant role of positional information, which even outweighs feature information for token selection\. These results provide a clear answer to the question raised inChapter 3:Yes, position is all you need for token compression in the MLLM\-based RES task\.

#### Effectiveness of Position Index Preserving

As shown in Table[5](https://arxiv.org/html/2608.26142#S5.T5), we remove the preserved original positional indices in PAYN and replace them with continuous ones, resulting in a severe performance degradation\. This finding is consistent with the results reported in Table[1](https://arxiv.org/html/2608.26142#S3.T1), which show similar trends across other methods\. These results demonstratethe effectiveness of the position index preserving module and further validate Insight 1\.

Table 5:Effectiveness of position index preserving\.
#### Effectiveness of Position\-Guided Token Selection

To validate our position\-guided token selection strategy, which emphasizes retaining sufficient tokens within each local neighboring region, we compare it with the following token selection strategies that are also purely position\-based:

\(1\) Center\-biased sampling, which performs non\-uniform sampling with higher density near the image center\. \(2\) Fixed\-stride row/column sampling, which retains entire rows and columns at a fixed stride\. \(3\) Group\-wise random sampling, which divides tokens into consecutive groups of equal size and randomly selects one token from each group\.

As shown in Table[6](https://arxiv.org/html/2608.26142#S5.T6), our token selection strategy achieves superior performance, which can be attributed to its ability to ensure broad and uniform spatial coverage and to better preserve local spatial structures\. These results demonstratethe effectiveness of the position\-guided token selection and further validate Insight 2\.

Table 6:Effectiveness of position\-guided token selection\.MethodrefCOCOrefCOCO\+refCOCOgAvg\.center\-biased59\.952\.555\.856\.1fixed\-stride row67\.759\.562\.463\.3fixed\-stride column65\.558\.161\.561\.7group\-wise random68\.460\.063\.864\.1PAYN \(spatial FPS\)69\.261\.864\.265\.2PAYN \(checkerboard\)70\.463\.065\.266\.4

#### Efficiency Analysis

As shown in Table[7](https://arxiv.org/html/2608.26142#S5.T7), we evaluate the efficiency of PAYN on both the Text4Seg and InstructSeg baselines, reporting the total runtime, inference speedup, and TFLOPs\. The results demonstrate that PAYN achieves an effective trade\-off between model performance and computational efficiency\. In addition, PAYN performs token selection solely based on positional information, without relying on token features or attention scores\. The selection masks can even be precomputed and reused during inference, introducing no additional computational overhead\.

Table 7:Efficiency analysis of PAYN on Text4Seg and InstructSeg\. Metrics include total runtime \(hour:min:sec\), inference speedup, and TFLOPs on RefCOCO\|\|TestA using a single A6000 GPU\.
#### Extension to Other MLLM Backbone

As shown in Table[8](https://arxiv.org/html/2608.26142#S5.T8), we further evaluate PAYN on the DeepSeekVL\-based\(Luet al\.,[2024](https://arxiv.org/html/2608.26142#bib.bib11)\)Text4Seg baseline\. PAYN consistently outperforms existing token compression methods, demonstrating its effectiveness and generalizability\.

Table 8:Comparison of PAYN and competing token compression methods on the Text4Seg baseline with DeepSeekVL\-7B as the MLLM backbone, where 192 visual tokens are retained\.
#### Pre\-Encoder vs\. Post\-Encoder Token Compression

Since PAYN is completely feature\-agnostic and relies solely on positional information, we further investigate whether positional sampling can be directly applied to input image patches before feature extraction\. As shown in Table[9](https://arxiv.org/html/2608.26142#S5.T9), discarding patches before the vision encoder leads to a complete performance collapse\. We attribute this to the fact that tokens in the vision encoder, especially in early layers, need to interact with neighboring tokens via self\-attention to progressively aggregate local features into global representations; patches discarded prior to the vision encoder cannot participate in this process and their information is therefore permanently lost\. In contrast, our method discards tokens after the vision encoder and before the LLM, where the retained tokens have already integrated contextual information from the discarded ones\.

Table 9:Applying token compression prior to the vision encoder causes severe performance collapse\.
#### Visualization Results

As illustrated in Fig[6](https://arxiv.org/html/2608.26142#S5.F6), we present two cases, where our method effectively preserves spatial structures, thereby achieving strong performance on dense prediction tasks such as RES\.

![Refer to caption](https://arxiv.org/html/2608.26142v1/img/experiment1.png)Figure 6:Visualization Results\.
#### Analysis on Extremely Small Objects

In extreme cases, the target object may occupy only a single image patch\. Although PAYN is feature\-agnostic and therefore cannot explicitly guarantee the retention of this token, object information can still spread to neighboring tokens through interactions in the vision encoder\. Specifically, we select images with extremely small foregrounds and measure how each token’s feature similarity varies with its distance to the foreground token, as shown in Fig\.[8](https://arxiv.org/html/2608.26142#A1.F8)\(a\)\. We observe that, from shallow to deeper layers, feature similarity increases for nearby tokens while decreasing for distant ones\. This indicates that even when a tiny object occupies only a single token, its information can still interact with neighboring patches\. As shown in the left panel of Fig\.[8](https://arxiv.org/html/2608.26142#A1.F8)\(b\), PAYN remains effective for extremely small objects while retaining only 192 tokens \(33\.3%\)\.

![Refer to caption](https://arxiv.org/html/2608.26142v1/x6.png)Figure 7:\(a\) Feature similarity of each token versus its distance to the foreground token in images with extremely small foregrounds\. \(b\) PAYN can preserve local spatial structures and perform well under normal conditions, but may fail in extreme cases where objects are very small and only a few tokens are retained\.

## 6Limitation and Future Direction

While effective in most settings, our method may exhibit limitations in extreme cases, such as when background regions contain little or no informative content while the number of retained tokens is extremely small, as shown in the right panel of Fig\.[8](https://arxiv.org/html/2608.26142#A1.F8)\(b\)\. In such cases, PAYN may fail to preserve local spatial structures and therefore become suboptimal, and prioritizing regions with higher information content could be necessary\. However, these cases are extremely uncommon and are not in conflict with our core perspective\.

## 7Conclusion

In this work, we observe that token compression for the MLLM\-based referring expression segmentation \(RES\) task has been rarely explored, and existing methods developed for other vision\-language tasks often suffer severe performance degradation when applied to RES\. Through experiments and analysis, we find that positional information of visual tokens plays a substantially more critical role in RES than in other tasks\. Motivated by this insight, we introduce PAYN, a plug\-and\-play, training\-free token compression method that relies solely on positional information\. Extensive experiments demonstrate that our method achieves improved inference efficiency while maintaining performance, validating our perspective that position is all you need for token compression in the MLLM\-based RES task\.

## Acknowledgments

This work is supported by the National Natural Science Foundation of China under grants 62206102; the National Key Research and Development Program of China under grant 2024YFC3307900; the National Natural Science Foundation of China under grants 62436003, 62376103 and 62302184; Major Science and Technology Project of Hubei Province under grant 2025BAB011 and 2024BAA008; Hubei Science and Technology Talent Service Project under grant 2024DJC078; and Ant Group through CCF\-Ant Research Fund\. The computation is completed in the HPC Platform of Huazhong University of Science and Technology\.

## Impact Statement

This paper presents work whose goal is to advance the field of Machine Learning\. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here\.

## References

- S\. R\. Alvar, G\. Singh, M\. Akbari, and Y\. Zhang \(2025\)Divprune: diversity\-based visual token pruning for large multimodal models\.InProceedings of the Computer Vision and Pattern Recognition Conference,pp\. 9392–9401\.Cited by:[§1](https://arxiv.org/html/2608.26142#S1.p3.1),[§2\.2](https://arxiv.org/html/2608.26142#S2.SS2.p1.1),[§3\.2](https://arxiv.org/html/2608.26142#S3.SS2.p1.1),[§5\.2](https://arxiv.org/html/2608.26142#S5.SS2.p1.1)\.
- J\. Bai, S\. Bai, Y\. Chu, Z\. Cui, K\. Dang, X\. Deng, Y\. Fan, W\. Ge, Y\. Han, F\. Huang,et al\.\(2023\)Qwen technical report\.arXiv preprint arXiv:2309\.16609\.Cited by:[§1](https://arxiv.org/html/2608.26142#S1.p1.1)\.
- D\. Bolya, C\. Fu, X\. Dai, P\. Zhang, C\. Feichtenhofer, and J\. Hoffman \(2023\)Token merging: your ViT but faster\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.26142#S1.p2.1),[§1](https://arxiv.org/html/2608.26142#S1.p3.1),[§2\.2](https://arxiv.org/html/2608.26142#S2.SS2.p1.1),[§3\.2](https://arxiv.org/html/2608.26142#S3.SS2.p1.1),[§5\.2](https://arxiv.org/html/2608.26142#S5.SS2.p1.1)\.
- M\. Cai, J\. Yang, J\. Gao, and Y\. J\. Lee \(2025\)Matryoshka multimodal models\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 46254–46272\.Cited by:[§2\.2](https://arxiv.org/html/2608.26142#S2.SS2.p1.1)\.
- J\. Canny \(2009\)A computational approach to edge detection\.IEEE Transactions on pattern analysis and machine intelligence\(6\),pp\. 679–698\.Cited by:[§5\.3](https://arxiv.org/html/2608.26142#S5.SS3.SSS0.Px1.p2.1)\.
- L\. Chen, H\. Zhao, T\. Liu, S\. Bai, J\. Lin, C\. Zhou, and B\. Chang \(2024a\)An image is worth 1/2 tokens after layer 2: plug\-and\-play inference acceleration for large vision\-language models\.InEuropean Conference on Computer Vision,pp\. 19–35\.Cited by:[§1](https://arxiv.org/html/2608.26142#S1.p1.1)\.
- X\. Chen, C\. Gao, Y\. Li, X\. Chen,et al\.\(2026\)Balanced token pruning: accelerating vision language models beyond local optimization\.Advances in Neural Information Processing Systems38,pp\. 62740–62764\.Cited by:[§1](https://arxiv.org/html/2608.26142#S1.p2.1)\.
- Y\. Chen, W\. Li, C\. Sun, Y\. F\. Wang, and C\. Chen \(2024b\)Sam4mllm: enhance multi\-modal large language model for referring expression segmentation\.InEuropean Conference on Computer Vision,pp\. 323–340\.Cited by:[§2\.1](https://arxiv.org/html/2608.26142#S2.SS1.p1.6)\.
- Z\. Chen, W\. Wang, H\. Tian, S\. Ye, Z\. Gao, E\. Cui, W\. Tong, K\. Hu, J\. Luo, Z\. Ma,et al\.\(2024c\)How far are we to gpt\-4v? closing the gap to commercial multimodal models with open\-source suites\.Science China Information Sciences67\(12\),pp\. 220101\.Cited by:[§1](https://arxiv.org/html/2608.26142#S1.p1.1)\.
- H\. Ding, S\. Tang, S\. He, C\. Liu, Z\. Wu, and Y\. Jiang \(2025\)Multimodal referring segmentation: a survey\.arXiv preprint arXiv:2508\.00265\.Cited by:[§1](https://arxiv.org/html/2608.26142#S1.p1.1),[§5\.1](https://arxiv.org/html/2608.26142#S5.SS1.SSS0.Px2.p1.1)\.
- C\. Fu, P\. Chen, Y\. Shen, Y\. Qin, M\. Zhang, X\. Lin, J\. Yang, X\. Zheng, K\. Li, X\. Sun,et al\.\(2025\)Mme: a comprehensive evaluation benchmark for multimodal large language models\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track,Cited by:[§3\.3](https://arxiv.org/html/2608.26142#S3.SS3.p3.1)\.
- R\. Hu, M\. Rohrbach, and T\. Darrell \(2016\)Segmentation from natural language expressions\.InEuropean conference on computer vision,pp\. 108–124\.Cited by:[§1](https://arxiv.org/html/2608.26142#S1.p1.1)\.
- S\. Kazemzadeh, V\. Ordonez, M\. Matten, and T\. Berg \(2014\)Referitgame: referring to objects in photographs of natural scenes\.InProceedings of the 2014 conference on empirical methods in natural language processing \(EMNLP\),pp\. 787–798\.Cited by:[Appendix A](https://arxiv.org/html/2608.26142#A1.p1.1),[§5\.1](https://arxiv.org/html/2608.26142#S5.SS1.SSS0.Px2.p1.1)\.
- A\. Kirillov, E\. Mintun, N\. Ravi, H\. Mao, C\. Rolland, L\. Gustafson, T\. Xiao, S\. Whitehead, A\. C\. Berg, W\. Lo,et al\.\(2023\)Segment anything\.InProceedings of the IEEE/CVF international conference on computer vision,pp\. 4015–4026\.Cited by:[Appendix B](https://arxiv.org/html/2608.26142#A2.p2.6),[§2\.1](https://arxiv.org/html/2608.26142#S2.SS1.p1.6)\.
- Z\. Kong, Y\. Li, F\. Zeng, L\. Xin, S\. Messica, X\. Lin, P\. Zhao, M\. Kellis, H\. Tang, and M\. Zitnik \(2025\)Token reduction should go beyond efficiency in generative models–from vision, language to multimodality\.arXiv preprint arXiv:2505\.18227\.Cited by:[§2\.2](https://arxiv.org/html/2608.26142#S2.SS2.p1.1)\.
- X\. Lai, Z\. Tian, Y\. Chen, Y\. Li, Y\. Yuan, S\. Liu, and J\. Jia \(2024\)Lisa: reasoning segmentation via large language model\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 9579–9589\.Cited by:[§1](https://arxiv.org/html/2608.26142#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.26142#S2.SS1.p1.6),[§5\.1](https://arxiv.org/html/2608.26142#S5.SS1.SSS0.Px2.p1.1)\.
- M\. Lan, C\. Chen, Y\. Zhou, J\. Xu, Y\. Ke, X\. Wang, L\. Feng, and W\. Zhang \(2025\)Text4seg: reimagining image segmentation as text generation\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 1634–1661\.Cited by:[Appendix B](https://arxiv.org/html/2608.26142#A2.p1.1),[§1](https://arxiv.org/html/2608.26142#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.26142#S2.SS1.p1.6),[§3\.2](https://arxiv.org/html/2608.26142#S3.SS2.p1.1),[§5\.1](https://arxiv.org/html/2608.26142#S5.SS1.SSS0.Px1.p1.1)\.
- Y\. Liang, C\. Ge, Z\. Tong, Y\. Song, J\. Wang, and P\. Xie \(2022\)Not all patches are what you need: expediting vision transformers via token reorganizations\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.26142#S1.p2.1)\.
- Z\. Lin, M\. Lin, L\. Lin, and R\. Ji \(2025\)Boosting multimodal large language models with visual tokens withdrawal for rapid inference\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 5334–5342\.Cited by:[§1](https://arxiv.org/html/2608.26142#S1.p1.1),[§2\.2](https://arxiv.org/html/2608.26142#S2.SS2.p1.1),[§3\.3](https://arxiv.org/html/2608.26142#S3.SS3.p1.1)\.
- H\. Liu, C\. Li, Y\. Li, and Y\. J\. Lee \(2024\)Improved baselines with visual instruction tuning\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 26296–26306\.Cited by:[§1](https://arxiv.org/html/2608.26142#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.26142#S2.SS1.p1.6),[§3\.2](https://arxiv.org/html/2608.26142#S3.SS2.p1.1)\.
- Y\. Liu, T\. Qu, Z\. Zhong, B\. Peng, S\. Liu, B\. Yu, and J\. Jia \(2026\)VisionReasoner: unified reasoning\-integrated visual perception via reinforcement learning\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§2\.1](https://arxiv.org/html/2608.26142#S2.SS1.p1.6)\.
- H\. Lu, W\. Liu, B\. Zhang, B\. Wang, K\. Dong, B\. Liu, J\. Sun, T\. Ren, Z\. Li, H\. Yang,et al\.\(2024\)Deepseek\-vl: towards real\-world vision\-language understanding\.arXiv preprint arXiv:2403\.05525\.Cited by:[§1](https://arxiv.org/html/2608.26142#S1.p1.1),[§5\.3](https://arxiv.org/html/2608.26142#S5.SS3.SSS0.Px5.p1.1)\.
- P\. Lu, S\. Mishra, T\. Xia, L\. Qiu, K\. Chang, S\. Zhu, O\. Tafjord, P\. Clark, and A\. Kalyan \(2022\)Learn to explain: multimodal reasoning via thought chains for science question answering\.Advances in Neural Information Processing Systems35,pp\. 2507–2521\.Cited by:[§3\.3](https://arxiv.org/html/2608.26142#S3.SS3.p3.1)\.
- J\. Mao, J\. Huang, A\. Toshev, O\. Camburu, A\. L\. Yuille, and K\. Murphy \(2016\)Generation and comprehension of unambiguous object descriptions\.InProceedings of the IEEE conference on computer vision and pattern recognition,pp\. 11–20\.Cited by:[Appendix A](https://arxiv.org/html/2608.26142#A1.p1.1),[§5\.1](https://arxiv.org/html/2608.26142#S5.SS1.SSS0.Px2.p1.1)\.
- Z\. Peng, W\. Wang, L\. Dong, Y\. Hao, S\. Huang, S\. Ma, Q\. Ye, and F\. Wei \(2024\)Grounding multimodal large language models to the world\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 51575–51598\.Cited by:[§2\.1](https://arxiv.org/html/2608.26142#S2.SS1.p1.6)\.
- R\. Pi, L\. Yao, J\. Gao, J\. Zhang, and T\. Zhang \(2024\)Perceptiongpt: effectively fusing visual perception into llm\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 27124–27133\.Cited by:[§1](https://arxiv.org/html/2608.26142#S1.p1.1)\.
- R\. Qian, X\. Yin, and D\. Dou \(2025\)Reasoning to attend: try to understand how seg token works\.InProceedings of the Computer Vision and Pattern Recognition Conference,pp\. 24722–24731\.Cited by:[§2\.1](https://arxiv.org/html/2608.26142#S2.SS1.p1.6)\.
- H\. Rasheed, M\. Maaz, S\. Shaji, A\. Shaker, S\. Khan, H\. Cholakkal, R\. M\. Anwer, E\. Xing, M\. Yang, and F\. S\. Khan \(2024\)Glamm: pixel grounding large multimodal model\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 13009–13018\.Cited by:[§1](https://arxiv.org/html/2608.26142#S1.p1.1)\.
- Z\. Ren, Z\. Huang, Y\. Wei, Y\. Zhao, D\. Fu, J\. Feng, and X\. Jin \(2024\)Pixellm: pixel reasoning with large multimodal model\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 26374–26383\.Cited by:[§1](https://arxiv.org/html/2608.26142#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.26142#S2.SS1.p1.6)\.
- Y\. Shang, M\. Cai, B\. Xu, Y\. J\. Lee, and Y\. Yan \(2025\)Llava\-prumerge: adaptive token reduction for efficient large multimodal models\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 22857–22867\.Cited by:[§1](https://arxiv.org/html/2608.26142#S1.p3.1),[§2\.2](https://arxiv.org/html/2608.26142#S2.SS2.p1.1),[§3\.2](https://arxiv.org/html/2608.26142#S3.SS2.p1.1)\.
- A\. Singh, V\. Natarajan, M\. Shah, Y\. Jiang, X\. Chen, D\. Batra, D\. Parikh, and M\. Rohrbach \(2019\)Towards vqa models that can read\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 8317–8326\.Cited by:[§3\.3](https://arxiv.org/html/2608.26142#S3.SS3.p3.1)\.
- I\. Sobel, G\. Feldman,et al\.\(1968\)A 3x3 isotropic gradient operator for image processing\.a talk at the Stanford Artificial Project in1968,pp\. 271–272\.Cited by:[§5\.3](https://arxiv.org/html/2608.26142#S5.SS3.SSS0.Px1.p2.1)\.
- J\. Tong, W\. Jin, P\. Qin, A\. Li, Y\. Zou, Y\. Li, Y\. Li, and R\. Li \(2026\)Flowcut: rethinking redundancy via information flow for efficient vision\-language models\.Advances in Neural Information Processing Systems38,pp\. 94946–94973\.Cited by:[§1](https://arxiv.org/html/2608.26142#S1.p2.1)\.
- H\. Wang, Z\. Yu, G\. Spadaro, C\. Ju, V\. Quétu, S\. Xiao, and E\. Tartaglione \(2025\)Folder: accelerating multi\-modal large language models with enhanced performance\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 23614–23625\.Cited by:[§1](https://arxiv.org/html/2608.26142#S1.p2.1)\.
- W\. Wang, Z\. Chen, X\. Chen, J\. Wu, X\. Zhu, G\. Zeng, P\. Luo, T\. Lu, J\. Zhou, Y\. Qiao,et al\.\(2023\)Visionllm: large language model is also an open\-ended decoder for vision\-centric tasks\.Advances in Neural Information Processing Systems36,pp\. 61501–61513\.Cited by:[§1](https://arxiv.org/html/2608.26142#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.26142#S2.SS1.p1.6)\.
- C\. Wei, Y\. Zhong, H\. Tan, Y\. Zeng, Y\. Liu, H\. Wang, and Y\. Yang \(2025\)Instructseg: unifying instructed visual segmentation with multi\-modal large language models\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 20193–20203\.Cited by:[Appendix B](https://arxiv.org/html/2608.26142#A2.p1.1),[§1](https://arxiv.org/html/2608.26142#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.26142#S2.SS1.p1.6),[§5\.1](https://arxiv.org/html/2608.26142#S5.SS1.SSS0.Px1.p1.1)\.
- Z\. Wen, Y\. Gao, S\. Wang, J\. Zhang, Q\. Zhang, W\. Li, C\. He, and L\. Zhang \(2025\)Stop looking for “important tokens” in multimodal language models: duplication matters more\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 9972–9991\.Cited by:[§1](https://arxiv.org/html/2608.26142#S1.p3.1),[§2\.2](https://arxiv.org/html/2608.26142#S2.SS2.p1.1),[§3\.2](https://arxiv.org/html/2608.26142#S3.SS2.p1.1),[§5\.2](https://arxiv.org/html/2608.26142#S5.SS2.p1.1)\.
- S\. N\. Wood \(2003\)Thin plate regression splines\.Journal of the Royal Statistical Society Series B: Statistical Methodology65\(1\),pp\. 95–114\.Cited by:[§3\.3](https://arxiv.org/html/2608.26142#S3.SS3.p3.1)\.
- T\. Wu, G\. Biamby, D\. Chan, L\. Dunlap, R\. Gupta, X\. Wang, J\. E\. Gonzalez, and T\. Darrell \(2024\)See say and segment: teaching lmms to overcome false premises\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 13459–13469\.Cited by:[§2\.1](https://arxiv.org/html/2608.26142#S2.SS1.p1.6)\.
- Z\. Xia, D\. Han, Y\. Han, X\. Pan, S\. Song, and G\. Huang \(2024\)Gsva: generalized segmentation via multimodal large language models\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 3858–3869\.Cited by:[§1](https://arxiv.org/html/2608.26142#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.26142#S2.SS1.p1.6)\.
- C\. Yang, Y\. Sui, J\. Xiao, L\. Huang, Y\. Gong, C\. Li, J\. Yan, Y\. Bai, P\. Sadayappan, X\. Hu,et al\.\(2025a\)Topv: compatible token pruning with inference time optimization for fast and low\-memory multimodal vision language model\.InProceedings of the Computer Vision and Pattern Recognition Conference,pp\. 19803–19813\.Cited by:[§1](https://arxiv.org/html/2608.26142#S1.p2.1)\.
- S\. Yang, Y\. Chen, Z\. Tian, C\. Wang, J\. Li, B\. Yu, and J\. Jia \(2025b\)Visionzip: longer is better but not necessary in vision language models\.InProceedings of the Computer Vision and Pattern Recognition Conference,pp\. 19792–19802\.Cited by:[§1](https://arxiv.org/html/2608.26142#S1.p1.1),[§1](https://arxiv.org/html/2608.26142#S1.p3.1),[§2\.2](https://arxiv.org/html/2608.26142#S2.SS2.p1.1),[§3\.2](https://arxiv.org/html/2608.26142#S3.SS2.p1.1),[§5\.2](https://arxiv.org/html/2608.26142#S5.SS2.p1.1)\.
- W\. Ye, Q\. Wu, W\. Lin, and Y\. Zhou \(2025a\)Fit and prune: fast and training\-free visual token pruning for multi\-modal large language models\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 22128–22136\.Cited by:[§2\.2](https://arxiv.org/html/2608.26142#S2.SS2.p1.1)\.
- X\. Ye, Y\. Gan, Y\. Ge, X\. Zhang, and Y\. Tang \(2025b\)ATP\-llava: adaptive token pruning for large vision language models\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 24972–24982\.Cited by:[§2\.2](https://arxiv.org/html/2608.26142#S2.SS2.p1.1),[§3\.3](https://arxiv.org/html/2608.26142#S3.SS3.p1.1)\.
- X\. Zhai, B\. Mustafa, A\. Kolesnikov, and L\. Beyer \(2023\)Sigmoid loss for language image pre\-training\.InProceedings of the IEEE/CVF international conference on computer vision,pp\. 11975–11986\.Cited by:[§3\.1](https://arxiv.org/html/2608.26142#S3.SS1.p1.11)\.
- Q\. Zhang, A\. Cheng, M\. Lu, R\. Zhang, Z\. Zhuo, J\. Cao, S\. Guo, Q\. She, and S\. Zhang \(2025a\)Beyond text\-visual attention: exploiting visual cues for effective token pruning in vlms\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 20857–20867\.Cited by:[§1](https://arxiv.org/html/2608.26142#S1.p3.1),[§2\.2](https://arxiv.org/html/2608.26142#S2.SS2.p1.1),[§3\.2](https://arxiv.org/html/2608.26142#S3.SS2.p1.1),[§5\.2](https://arxiv.org/html/2608.26142#S5.SS2.p1.1)\.
- Q\. Zhang, M\. Liu, L\. Li, M\. Lu, Y\. Zhang, J\. Pan, Q\. She, and S\. Zhang \(2026\)Beyond attention or similarity: maximizing conditional diversity for token pruning in mllms\.Advances in Neural Information Processing Systems38,pp\. 25438–25468\.Cited by:[§1](https://arxiv.org/html/2608.26142#S1.p2.1)\.
- Y\. Zhang, C\. Fan, J\. Ma, W\. Zheng, T\. Huang, K\. Cheng, D\. Gudovskiy, T\. Okuno, Y\. Nakata, K\. Keutzer,et al\.\(2025b\)SparseVLM: visual token sparsification for efficient vision\-language model inference\.InInternational Conference on Machine Learning,Cited by:[§2\.2](https://arxiv.org/html/2608.26142#S2.SS2.p1.1)\.
- Z\. Zhang, Y\. Ma, E\. Zhang, and X\. Bai \(2024\)Psalm: pixelwise segmentation with large multi\-modal model\.InEuropean Conference on Computer Vision,pp\. 74–91\.Cited by:[§2\.1](https://arxiv.org/html/2608.26142#S2.SS1.p1.6)\.
- L\. Zhu, T\. Chen, Q\. Xu, X\. Liu, D\. Ji, H\. Wu, D\. W\. Soh, and J\. Liu \(2025a\)Popen: preference\-based optimization and ensemble for lvlm\-based reasoning segmentation\.InProceedings of the Computer Vision and Pattern Recognition Conference,pp\. 30231–30240\.Cited by:[§2\.1](https://arxiv.org/html/2608.26142#S2.SS1.p1.6)\.
- M\. Zhu, Y\. Zhu, N\. Liu, X\. Liu, Z\. Xu, C\. Shen, and Y\. Peng \(2025b\)A comprehensive overhaul of multimodal assistant with small language models\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 10986–10994\.Cited by:[§5\.1](https://arxiv.org/html/2608.26142#S5.SS1.SSS0.Px1.p1.1)\.
- W\. Zhu, X\. Chen, Z\. Wang, S\. Tang, S\. Ghosh, X\. Dong, R\. Koner, and Y\. Wang \(2026\)EVTP\-ivs: effective visual token pruning for unifying instruction visual segmentation in multi\-modal large language models\.InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision,pp\. 7158–7167\.Cited by:[§5\.2](https://arxiv.org/html/2608.26142#S5.SS2.p1.1)\.

## Appendix ADataset Description

![Refer to caption](https://arxiv.org/html/2608.26142v1/x7.png)Figure 8:Samples from the \(a\) RefCOCO, \(b\) RefCOCO\+, and \(c\) RefCOCOg datasets\.We evaluate our method on the RefCOCO, RefCOCO\+\(Kazemzadehet al\.,[2014](https://arxiv.org/html/2608.26142#bib.bib53)\), and RefCOCOg\(Maoet al\.,[2016](https://arxiv.org/html/2608.26142#bib.bib54)\)benchmarks\. RefCOCOcontains 19,994 images, 142,209 referring expressions, and 50,000 annotated objects\. It supports referring expressions based on both spatial location and visual appearance attributes\. RefCOCO\+includes 19,992 images, 141,564 expressions, and 49,856 annotated objects\. Different from RefCOCO, it restricts the use of location\-related descriptions and mainly focuses on appearance\-based referring expressions\. RefCOCOgconsists of 25,799 images with 95,010 referring expressions and 49,822 annotated objects\. It is characterized by longer and more complex expressions without limitations on location references\.

## Appendix BDetails of Baseline Methods

We employ two representative RES baselines: Text4Seg\(Lanet al\.,[2025](https://arxiv.org/html/2608.26142#bib.bib7)\)and InstructSeg\(Weiet al\.,[2025](https://arxiv.org/html/2608.26142#bib.bib8)\)\.

Text4Seg follows the text\-as\-mask paradigm, where the image is divided into a 16×16 grid of patches, and the MLLM assigns a semantic textual label to each patch in a row\-wise manner\. For example, the segmentation mask for “dog laying down” can be represented as:<<seg\>\>others \*16 \\n dog laying down \*1\|\|others \*15 \\n dog laying down \*4\|\|others \*12 \\n others \*16 …<</seg\>\>\. To further improve the quality of the pixel\-level semantic masks, SAM\(Kirillovet al\.,[2023](https://arxiv.org/html/2608.26142#bib.bib23)\)is employed as a mask refinement module\.

In contrast, InstructSeg is built upon an embedding\-as\-mask paradigm and supports segmentation for both images and videos\. The framework mainly consists of an object\-aware video perceiver \(OVP\), an MLLM, a visual encoder, a vision\-guided multi\-granularity text fusion module \(VMTF\), and a segmentation decoder\. Specifically, the OVP compresses temporal and object\-aware information from video frames into compact tokens, which are jointly processed with text tokens by the MLLM to generate mask embeddings and detailed text embeddings\. Meanwhile, the visual encoder extracts semantic features from the input image or video frames\. The generated embeddings and visual features are further fused by the VMTF module to enable comprehensive vision\-language understanding, whose outputs are subsequently decoded by the segmentation decoder to generate segmentation masks and corresponding confidence scores\.

Due to the differences in model architectures and structural complexity, we adopt different compression ratios for each RES baseline to achieve a better balance between segmentation performance and inference efficiency\. Compared with Text4Seg, InstructSeg contains an additional vision encoder branch that helps alleviate the information loss caused by token compression\. Moreover, InstructSeg follows an embedding\-as\-mask paradigm, where the MLLM predicts position\-related embeddings rather than per\-patch semantic classes as in Text4Seg, making it inherently less sensitive to suboptimal token compression\. Therefore, we apply a higher compression ratio to the InstructSeg baseline\.

## Appendix CHyperparameters of the Compared Methods

We use well\-tuned configurations for all compared methods to ensure a fair comparison\. Under the 192 tokens setting of the Text4Seg baseline, the hyperparameters of the comparison methods are summarized in Table[10](https://arxiv.org/html/2608.26142#A3.T10)\. For InstructSeg, we adopt the same settings\.

Table 10:Hyperparameters of the compared methods\.