Differentiable Efficient Operator Search

# Differentiable Efficient Operator Search
Source: [https://arxiv.org/html/2606.05232](https://arxiv.org/html/2606.05232)
Xiaohuan Pei1Jiyuan Zhang2Yuanfan Guo2Weiguo Feng2 Tao Huang3Cho\-Jui Hsieh4Chang Xu1 1The University of Sydney2ByteDance 3Shanghai Jiao Tong University4University of California, Los Angeles xiaohuan\.pei@sydney\.edu\.au

###### Abstract

Efficient models have largely relied on human\-designed reduction operators, such as pruning, merging, pooling, and adaptive reweighting, which could happen at any time and anywhere\. We show that these seemingly different methods can be unified as different operating regimes of a single shared operator space\. Based on this observation, we introduceEfficient Operator Search, a unified framework in which continuous parameters control whether token information is removed, sharply merged, uniformly pooled, or softly redistributed\. Instead of hand\-crafting operator compositions, we define a*efficient search space*: a differentiable parameterization of layer activation, retention budget, and operator regime, and a*efficient search policy*: minimizing the expectation of the task loss under one\-sided budget and cost constraints\. The pipeline leads to a broader shift from anefficient design problemto anoperator search problem, suggesting a new paradigm for future efficient modeling\. Interestingly, we find that most prior mainstream baselines can be treated as special cases of this shared operator space, and further reveal consistent operator patterns across different benchmarks\. Even as an early exploration of this new paradigm, our proposed method achieves competitive results on various benchmarks\. We believe that it provides a unified view of prior methods, a principled lens for understanding current differences, and a general foundation for future efficient modeling\. Webpage:[EOS](https://www.terrypei.com/eos)\.

## 1Introduction

Scaling multi\-modal foundation models has led to substantial performance gains across visual reasoning, multi\-modal understanding, instruction following, and long\-context multimodal reasoning\[[12](https://arxiv.org/html/2606.05232#bib.bib12),[10](https://arxiv.org/html/2606.05232#bib.bib10),[26](https://arxiv.org/html/2606.05232#bib.bib26),[27](https://arxiv.org/html/2606.05232#bib.bib27),[6](https://arxiv.org/html/2606.05232#bib.bib6),[28](https://arxiv.org/html/2606.05232#bib.bib28),[9](https://arxiv.org/html/2606.05232#bib.bib9)\]\. However, these gains often come with substantial inference cost, as dense visual tokens are repeatedly processed by large language backbones, especially in high\-resolution, multi\-image, and long\-context scenarios\[[31](https://arxiv.org/html/2606.05232#bib.bib31),[13](https://arxiv.org/html/2606.05232#bib.bib13),[23](https://arxiv.org/html/2606.05232#bib.bib23),[24](https://arxiv.org/html/2606.05232#bib.bib24),[30](https://arxiv.org/html/2606.05232#bib.bib30),[22](https://arxiv.org/html/2606.05232#bib.bib22),[20](https://arxiv.org/html/2606.05232#bib.bib20)\]\. Existing work improves the efficiency of foundation models through human\-designed token reduction operators, including pruning, merging, pooling, and adaptive reweighting\. For example, SparseVLM\[[32](https://arxiv.org/html/2606.05232#bib.bib32)\]removes visually unimportant tokens according to attention scores, while ToMe\[[1](https://arxiv.org/html/2606.05232#bib.bib1)\]merges redundant tokens to shorten the input sequence while preserving information\. Although these methods demonstrate promising accuracy\-efficiency trade\-offs, they are usually developed as separate compression recipes with manually chosen layers, budgets, and operator forms, leaving the shared structure behind these corner operators largely underexplored\.

![Refer to caption](https://arxiv.org/html/2606.05232v1/figures/prev_ours.png)\(a\)Manual design vs\. automatic search\.
![Refer to caption](https://arxiv.org/html/2606.05232v1/figures/intro_eos.png)\(b\)Unified operator space and performance\.

Figure 1:Overview of Efficient Operator Search\.\(a\) EOS replaces manually designed reduction recipes with automatic operator search\. \(b\) The searched hybrid operator lies inside the unified operator space and improves performance under the same token budget\.Despite their different implementations, existing reduction behaviors can be interpreted as different ways of discarding, transferring, or redistributing token information\. This interpretation motivates a shared formulation in which representative efficient operators are controlled by continuous parameters\. Specifically, an information\-transfer gateγ\\gammadetermines whether discarded token information is removed or preserved, while an assignment temperatureτ\\taucontrols whether the preserved information is sharply merged, uniformly pooled, or softly redistributed\. Under this view, pure pruning, hard merging, average pooling, and adaptive reweighting naturally emerge as different operating regimes of the same operator space \(Figure[1](https://arxiv.org/html/2606.05232#S1.F1)\)\. Based on this observation, we proposeEfficient Operator Search, a unified framework that reframes efficient models from an*operator design problem*into an*operator search problem*\. Instead of manually crafting another fixed compression rule, our method searches over where to compress, how many tokens to retain, and which reduction behavior to apply\. This enables flexible combinations of multiple compression primitives under different accuracy\-efficiency constraints\. We conduct extensive experiments on representative multi\-modal benchmarks\. The results show that our framework can recover strong hand\-designed baselines and further discover better operator configurations, achieving improved performance\-efficiency trade\-offs\.

Our contributions can be summarized from three aspects\. \(1\) We introduce a unified operator space that reinterprets existing token reduction methods as different regimes of one shared formulation\. \(2\) We propose Efficient Operator Search, which jointly searches layer selection, token budgets, and continuous operator parameters instead of relying on manually designed recipes\. \(3\) We demonstrate the proposed framework can recover existing baselines and discover stronger hybrid configurations for efficient multimodal foundation models\.

## 2Method

![Refer to caption](https://arxiv.org/html/2606.05232v1/figures/main.png)Figure 2:Overview of Efficient Operator Search\.Given a frozen multimodal foundation model, EOS parameterizes token reduction at each decoder layer by three coupled components: layer activationgl\{\\color\[rgb\]\{0\.78515625,0\.1953125,0\}g\_\{l\}\}, retention budgetcl\{\\color\[rgb\]\{0,0\.58984375,0\.1953125\}c\_\{l\}\}, and operator regimeΩl=\(γl,τl,θl,ρl,νl\)\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\Omega\_\{l\}=\(\\gamma\_\{l\},\\tau\_\{l\},\\theta\_\{l\},\\rho\_\{l\},\\nu\_\{l\}\)\}\. At each active layer, important visual tokens are retained as anchors, while the remaining candidates are processed by a unified reduction operator\. Depending on the learned operator parameters, discarded information can be removed, sharply merged, uniformly pooled, or softly redistributed to the anchors\.We aim to transform efficient foundation models from manually designed compression recipes into a searchable operator space\. Given a frozen multimodal foundation model withLLdecoder layers, existing token\-reduction methods typically hand\-specify*where*to reduce visual tokens,*how many*tokens to retain, and*which*operator to apply\. In contrast, ourEfficient Operator Search\(EOS\) parameterizes these choices as learnable search variables and jointly optimizes them with differentiable relaxations under task and efficiency constraints\. As shown in Figure[2](https://arxiv.org/html/2606.05232#S2.F2), EOS selects important visual\-token anchors at each active reduction layer, then applies a unified operator to decide whether the remaining token information is discarded, transferred, pooled, or reweighted\.

### 2\.1Preliminary

Let𝐗\(l\)∈ℝNl×d\\mathbf\{X\}^\{\(l\)\}\\in\\mathbb\{R\}^\{N\_\{l\}\\times d\}denote the visual hidden states at decoder layerll, whereNlN\_\{l\}is the number of visual tokens andddis the hidden dimension\. A token\-reduction operation partitions𝐗\(l\)\\mathbf\{X\}^\{\(l\)\}into a retained anchor set and a reduction\-candidate set:

𝐗\(l\)=𝐀\(l\)∪𝐃\(l\),𝐀\(l\)∈ℝKl×d,𝐃\(l\)∈ℝMl×d,\\mathbf\{X\}^\{\(l\)\}=\\mathbf\{A\}^\{\(l\)\}\\cup\\mathbf\{D\}^\{\(l\)\},\\qquad\\mathbf\{A\}^\{\(l\)\}\\in\\mathbb\{R\}^\{K\_\{l\}\\times d\},\\quad\\mathbf\{D\}^\{\(l\)\}\\in\\mathbb\{R\}^\{M\_\{l\}\\times d\},\(1\)whereKlK\_\{l\}is the number of retained anchors,Ml=Nl−KlM\_\{l\}=N\_\{l\}\-K\_\{l\}is the number of reduction candidates, and𝐀\(l\)∩𝐃\(l\)=∅\\mathbf\{A\}^\{\(l\)\}\\cap\\mathbf\{D\}^\{\(l\)\}=\\emptyset\. The anchors are selected according to an importance score𝐪\(l\)∈ℝNl\\mathbf\{q\}^\{\(l\)\}\\in\\mathbb\{R\}^\{N\_\{l\}\}, which can be instantiated by attention\-based or similarity\-based token importance following prior efficient inference methods\. The remaining question is how to process𝐃\(l\)\\mathbf\{D\}^\{\(l\)\}after anchor selection\. Pure pruning directly removes𝐃\(l\)\\mathbf\{D\}^\{\(l\)\}, token merging transfers each discarded token to a similar anchor, pooling distributes discarded information uniformly, and adaptive reweighting rescales anchors according to discarded\-token importance\. EOS treats these behaviors as different operating regimes of one shared operator family\.

### 2\.2Unified Reduction Operator

For simplicity, we omit the layer indexllwhen the context is clear\. Given anchors𝐀∈ℝK×d\\mathbf\{A\}\\in\\mathbb\{R\}^\{K\\times d\}and reduction candidates𝐃∈ℝM×d\\mathbf\{D\}\\in\\mathbb\{R\}^\{M\\times d\}, we first compute the normalized similarity matrix:

𝐒=𝐃¯​𝐀¯⊤∈ℝM×K,𝐱¯i=𝐱i‖𝐱i‖2\+ϵ,\\mathbf\{S\}=\\bar\{\\mathbf\{D\}\}\\bar\{\\mathbf\{A\}\}^\{\\top\}\\in\\mathbb\{R\}^\{M\\times K\},\\qquad\\bar\{\\mathbf\{x\}\}\_\{i\}=\\frac\{\\mathbf\{x\}\_\{i\}\}\{\\\|\\mathbf\{x\}\_\{i\}\\\|\_\{2\}\+\\epsilon\},\(2\)where𝐱i\\mathbf\{x\}\_\{i\}denotes a row token feature from either𝐀\\mathbf\{A\}or𝐃\\mathbf\{D\}, the normalization is applied token\-wise, andϵ\>0\\epsilon\>0is a small constant used for numerical stability\. Each entrySi​jS\_\{ij\}measures the cosine similarity between reduction candidate𝐝i\\mathbf\{d\}\_\{i\}and anchor token𝐚j\\mathbf\{a\}\_\{j\}\. The assignment from reduction candidates to anchors is given by

𝐖=softmaxanchor​\(𝐒τl\)∈ℝM×K,\\mathbf\{W\}=\\mathrm\{softmax\}\_\{\\mathrm\{anchor\}\}\\left\(\\frac\{\\mathbf\{S\}\}\{\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\tau\_\{l\}\}\}\\right\)\\in\\mathbb\{R\}^\{M\\times K\},\(3\)wheresoftmaxanchor​\(⋅\)\\mathrm\{softmax\}\_\{\\mathrm\{anchor\}\}\(\\cdot\)normalizes each row over the anchor dimension, andτl\>0\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\tau\_\{l\}\}\>0is an assignment temperature\. A smallerτl\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\tau\_\{l\}\}induces sharper nearest\-anchor assignment, while a largerτl\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\tau\_\{l\}\}yields smoother assignment across anchors\.

We further define a per\-token transfer gate:

mi=σ​\(β​\(maxj⁡Si​j−θl\)\),𝐦=\[m1,…,mM\]⊤,m\_\{i\}=\\sigma\\left\(\\beta\(\\max\_\{j\}S\_\{ij\}\-\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\theta\_\{l\}\}\)\\right\),\\qquad\\mathbf\{m\}=\[m\_\{1\},\\ldots,m\_\{M\}\]^\{\\top\},\(4\)whereθl\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\theta\_\{l\}\}is a similarity threshold andβ\\betacontrols the gate sharpness\. The unified information\-transfer operation is then given by

𝐀~=𝐀\+γl​𝐖⊤​\(𝐃⊙𝐦\),\\tilde\{\\mathbf\{A\}\}=\\mathbf\{A\}\+\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\gamma\_\{l\}\}\\,\\mathbf\{W\}^\{\\top\}\(\\mathbf\{D\}\\odot\\mathbf\{m\}\),\(5\)where𝐦\\mathbf\{m\}is broadcast along the feature dimension, andγl∈\[0,1\]\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\gamma\_\{l\}\}\\in\[0,1\]is an information\-transfer gate\. Whenγl=0\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\gamma\_\{l\}\}=0, the reduction candidates have no explicit feature transfer and are effectively pruned\. Whenγl\>0\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\gamma\_\{l\}\}\>0, their information is transferred to the retained anchors according to the assignment matrix𝐖\\mathbf\{W\}\.

To include adaptive reweighting and stabilize the output scale, we further apply

𝐀^=𝐀~⊙\(1\+ρl​𝐬imp\),𝐀^←\(1−νl\)​𝐀^\+νl​‖𝐀‖F‖𝐀^‖F\+ϵ​𝐀^,\\hat\{\\mathbf\{A\}\}=\\tilde\{\\mathbf\{A\}\}\\odot\\left\(1\+\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\rho\_\{l\}\}\\,\\mathbf\{s\}\_\{\\mathrm\{imp\}\}\\right\),\\qquad\\hat\{\\mathbf\{A\}\}\\leftarrow\(1\-\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\nu\_\{l\}\}\)\\hat\{\\mathbf\{A\}\}\+\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\nu\_\{l\}\}\\frac\{\\\|\\mathbf\{A\}\\\|\_\{F\}\}\{\\\|\\hat\{\\mathbf\{A\}\}\\\|\_\{F\}\+\\epsilon\}\\hat\{\\mathbf\{A\}\},\(6\)whereρl\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\rho\_\{l\}\}denotes the anchor\-reweighting strength,νl\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\nu\_\{l\}\}denotes the norm\-preservation coefficient, and𝐬imp∈ℝK\\mathbf\{s\}\_\{\\mathrm\{imp\}\}\\in\\mathbb\{R\}^\{K\}denotes the anchor\-level importance induced by reduction candidates\. Here,γl\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\gamma\_\{l\}\}controls explicit feature transfer, whileρl\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\rho\_\{l\}\}controls indirect anchor rescaling\. In practice,𝐬imp\\mathbf\{s\}\_\{\\mathrm\{imp\}\}is given by

𝐬imp=softmaxanchor​\(𝐒\)⊤​softmaxcandidate​\(𝐪𝒟\),\\mathbf\{s\}\_\{\\mathrm\{imp\}\}=\\mathrm\{softmax\}\_\{\\mathrm\{anchor\}\}\(\\mathbf\{S\}\)^\{\\top\}\\mathrm\{softmax\}\_\{\\mathrm\{candidate\}\}\(\\mathbf\{q\}\_\{\\mathcal\{D\}\}\),\(7\)where𝐪𝒟\\mathbf\{q\}\_\{\\mathcal\{D\}\}denotes the importance scores of the reduction candidates, andsoftmaxcandidate​\(⋅\)\\mathrm\{softmax\}\_\{\\mathrm\{candidate\}\}\(\\cdot\)normalizes over the reduction\-candidate dimension\.

###### Definition 2\.1\(Unified efficient operator\)\.

At layerll, an efficient reduction operator is defined by the parameter tuple

Ωl=\(γl,τl,θl,ρl,νl\),\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\Omega\_\{l\}\}=\\left\(\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\gamma\_\{l\}\},\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\tau\_\{l\}\},\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\theta\_\{l\}\},\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\rho\_\{l\}\},\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\nu\_\{l\}\}\\right\),\(8\)and maps the visual hidden states from𝐗\(l\)∈ℝNl×d\\mathbf\{X\}^\{\(l\)\}\\in\\mathbb\{R\}^\{N\_\{l\}\\times d\}to a reduced representation𝐀^\(l\)∈ℝKl×d\\hat\{\\mathbf\{A\}\}^\{\(l\)\}\\in\\mathbb\{R\}^\{K\_\{l\}\\times d\}through Eqs\.[2](https://arxiv.org/html/2606.05232#S2.E2)–[6](https://arxiv.org/html/2606.05232#S2.E6)\. The parametersγl\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\gamma\_\{l\}\}andτl\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\tau\_\{l\}\}determine the dominant reduction regime, whileθl\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\theta\_\{l\}\},ρl\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\rho\_\{l\}\}, andνl\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\nu\_\{l\}\}provide token\-level gating, anchor reweighting, and scale preservation\.

The parameter tupleΩl\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\Omega\_\{l\}\}defines the reduction behavior at a single layer\. EOS further couples this operator regime with layer activation and token budget, yielding a searchable space over where to compress, how much to compress, and how to process the reduced tokens\.

### 2\.3Efficient Search Space

EOS searches over three coupled dimensions:layer activation,token budget, andoperator regime\. For all decoder layers, the complete search space is given by

𝚯=\{gl⏟which layers,cl⏟how much,γl,τl,θl,ρl,νl⏟which operator\}l=0L−1\\boxed\{\\bm\{\\Theta\}=\\Bigl\\\{\\underbrace\{\{\\color\[rgb\]\{0\.78515625,0\.1953125,0\}g\_\{l\}\}\}\_\{\\text\{which layers\}\},\\;\\underbrace\{\{\\color\[rgb\]\{0,0\.58984375,0\.1953125\}c\_\{l\}\}\}\_\{\\text\{how much\}\},\\;\\underbrace\{\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\gamma\_\{l\},\\tau\_\{l\},\\theta\_\{l\},\\rho\_\{l\},\\nu\_\{l\}\}\}\_\{\\text\{which operator\}\}\\Bigr\\\}\_\{l=0\}^\{L\-1\}\}\(9\)wheregl\{\\color\[rgb\]\{0\.78515625,0\.1953125,0\}g\_\{l\}\}determines whether layerllapplies token reduction,cl\{\\color\[rgb\]\{0,0\.58984375,0\.1953125\}c\_\{l\}\}controls the fraction of visual tokens to discard, and\(γl,τl,θl,ρl,νl\)\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\(\\gamma\_\{l\},\\tau\_\{l\},\\theta\_\{l\},\\rho\_\{l\},\\nu\_\{l\}\)\}specify the operator regime\. All variables are represented by unconstrained learnable parameters and mapped to valid ranges through differentiable transformations:

gl\\displaystyle\{\\color\[rgb\]\{0\.78515625,0\.1953125,0\}g\_\{l\}\}=σ​\(wlg⋅Tg\),\\displaystyle=\\sigma\(w\_\{l\}^\{g\}\\cdot T\_\{g\}\),cl\\displaystyle\{\\color\[rgb\]\{0,0\.58984375,0\.1953125\}c\_\{l\}\}=σ​\(wlc⋅Tc\),\\displaystyle=\\sigma\(w\_\{l\}^\{c\}\\cdot T\_\{c\}\),γl\\displaystyle\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\gamma\_\{l\}\}=σ​\(wlγ\),\\displaystyle=\\sigma\(w\_\{l\}^\{\\gamma\}\),\(10\)τl\\displaystyle\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\tau\_\{l\}\}=softplus​\(wlτ\)\+ϵ,\\displaystyle=\\mathrm\{softplus\}\(w\_\{l\}^\{\\tau\}\)\+\\epsilon,θl\\displaystyle\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\theta\_\{l\}\}=2​σ​\(wlθ\)−1,\\displaystyle=2\\sigma\(w\_\{l\}^\{\\theta\}\)\-1,ρl\\displaystyle\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\rho\_\{l\}\}=σ​\(wlρ\),νl=σ​\(wlν\),\\displaystyle=\\sigma\(w\_\{l\}^\{\\rho\}\),\\qquad\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\nu\_\{l\}\}=\\sigma\(w\_\{l\}^\{\\nu\}\),whereTgT\_\{g\}andTcT\_\{c\}are optional scaling temperatures used to sharpen the structural gates, and the same numerical constantϵ\>0\\epsilon\>0ensures a strictly positive assignment temperature\. The effective number of retained visual tokens at layerllis given by

Kl=max⁡\(Kmin,⌊\(1−gl​cl\)​Nl⌋\)\.K\_\{l\}=\\max\\left\(K\_\{\\min\},\\left\\lfloor\\bigl\(1\-\{\\color\[rgb\]\{0\.78515625,0\.1953125,0\}g\_\{l\}\}\{\\color\[rgb\]\{0,0\.58984375,0\.1953125\}c\_\{l\}\}\\bigr\)N\_\{l\}\\right\\rfloor\\right\)\.\(11\)Whengl\{\\color\[rgb\]\{0\.78515625,0\.1953125,0\}g\_\{l\}\}is close to zero, the layer is effectively inactive and the visual sequence is passed to the next layer unchanged\. Whengl\{\\color\[rgb\]\{0\.78515625,0\.1953125,0\}g\_\{l\}\}is active,cl\{\\color\[rgb\]\{0,0\.58984375,0\.1953125\}c\_\{l\}\}determines the compression strength, and the operator tupleΩl\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\Omega\_\{l\}\}determines how reduction candidates influence retained anchors\. Although Eq\.[11](https://arxiv.org/html/2606.05232#S2.E11)defines the discrete forward retain number, EOS optimizes its underlying variables through the relaxation introduced in Sec\.[2\.4](https://arxiv.org/html/2606.05232#S2.SS4)\. The structural variables\(gl,cl\)\(\{\\color\[rgb\]\{0\.78515625,0\.1953125,0\}g\_\{l\}\},\{\\color\[rgb\]\{0,0\.58984375,0\.1953125\}c\_\{l\}\}\)answer*where*and*how much*to compress, while the operator variables\(γl,τl,θl,ρl,νl\)\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\(\\gamma\_\{l\},\\tau\_\{l\},\\theta\_\{l\},\\rho\_\{l\},\\nu\_\{l\}\)\}answer*which reduction behavior*to apply\. Such factorization enables EOS to recover existing hand\-designed methods by fixing part of𝚯\\bm\{\\Theta\}, while also allowing new hybrid configurations to be discovered through continuous optimization\.

### 2\.4Differentiable Token Selection

Since top\-KlK\_\{l\}anchor selection is discrete, we use a straight\-through soft boundary relaxation to preserve the forward behavior while allowing gradients to update the retention budget\. Letq\(Kl\)q\_\{\(K\_\{l\}\)\}denote the detachedKlK\_\{l\}\-th largest importance score, and define

pikeep=σ​\(α​\(qi−q\(Kl\)\)\)⋅1−gl​cl1−sg​\(gl​cl\),p\_\{i\}^\{\\mathrm\{keep\}\}=\\sigma\\left\(\\alpha\(q\_\{i\}\-q\_\{\(K\_\{l\}\)\}\)\\right\)\\cdot\\frac\{1\-\{\\color\[rgb\]\{0\.78515625,0\.1953125,0\}g\_\{l\}\}\{\\color\[rgb\]\{0,0\.58984375,0\.1953125\}c\_\{l\}\}\}\{1\-\\mathrm\{sg\}\(\{\\color\[rgb\]\{0\.78515625,0\.1953125,0\}g\_\{l\}\}\{\\color\[rgb\]\{0,0\.58984375,0\.1953125\}c\_\{l\}\}\)\},\(12\)whereα\\alphacontrols the boundary sharpness andsg​\(⋅\)\\mathrm\{sg\}\(\\cdot\)denotes the stop\-gradient operation\. The first term approximates whether tokeniilies above the selection threshold, while the second term provides a straight\-through gradient path fromgl\{\\color\[rgb\]\{0\.78515625,0\.1953125,0\}g\_\{l\}\}andcl\{\\color\[rgb\]\{0,0\.58984375,0\.1953125\}c\_\{l\}\}without changing the forward value\. The relaxed keep probabilities estimate the differentiable token count, while inference still uses hard top\-KKanchors and reduction candidates in Eq\.[5](https://arxiv.org/html/2606.05232#S2.E5)\.

### 2\.5Differentiable Search Policy

The backbone multimodal foundation model is kept frozen, and only the search parameters𝚯\\bm\{\\Theta\}are optimized\. Given an input\-output pair\(𝐱,𝐲\)\(\\mathbf\{x\},\\mathbf\{y\}\), EOS minimizes the task loss under one\-sided efficiency constraints:

min𝚯ℒtask​\(𝚯\)⏟task loss\+λb​\[Nfinal​\(𝚯\)B−1\]\+2⏟token budget:​Nfinal≤B\+λc​\[∑l=0L−1gl−C\]\+2⏟inference cost: active layers≤C\\boxed\{\\min\_\{\\bm\{\\Theta\}\}\\quad\\underbrace\{\\mathcal\{L\}\_\{\\mathrm\{task\}\}\(\\bm\{\\Theta\}\)\}\_\{\\text\{task loss\}\}\+\\underbrace\{\\lambda\_\{b\}\\left\[\\frac\{N\_\{\\mathrm\{final\}\}\(\\bm\{\\Theta\}\)\}\{B\}\-1\\right\]\_\{\+\}^\{2\}\}\_\{\\text\{token budget: \}N\_\{\\mathrm\{final\}\}\\leq B\}\+\\underbrace\{\\lambda\_\{c\}\\left\[\\sum\_\{l=0\}^\{L\-1\}\{\\color\[rgb\]\{0\.78515625,0\.1953125,0\}g\_\{l\}\}\-C\\right\]\_\{\+\}^\{2\}\}\_\{\\text\{inference cost: active layers\}\\leq C\}\}\(13\)where\[⋅\]\+=max⁡\(0,⋅\)\[\\cdot\]\_\{\+\}=\\max\(0,\\cdot\),BBis the target final token budget, andCCis the target maximum number of active reduction layers\. Here,Nfinal​\(𝚯\)N\_\{\\mathrm\{final\}\}\(\\bm\{\\Theta\}\)is estimated byN¯0=N0\\bar\{N\}\_\{0\}=N\_\{0\},N¯l\+1=\(1−gl​cl\)​N¯l\\bar\{N\}\_\{l\+1\}=\(1\-\{\\color\[rgb\]\{0\.78515625,0\.1953125,0\}g\_\{l\}\}\{\\color\[rgb\]\{0,0\.58984375,0\.1953125\}c\_\{l\}\}\)\\bar\{N\}\_\{l\}, andNfinal​\(𝚯\)=N¯LN\_\{\\mathrm\{final\}\}\(\\bm\{\\Theta\}\)=\\bar\{N\}\_\{L\}\. At deployment, it is replaced by the corresponding discrete retain count after top\-KKselection\. The first regularizer penalizes configurations that exceed the token budget, while the second penalizes configurations that activate too many reduction layers\. Both constraints are one\-sided: no penalty is applied once the configuration satisfies the desired budget\. This design allows the search to prioritize task performance within a feasible efficiency region, rather than forcing unnecessary compression when the budget has already been met\. The task loss is given by

ℒtask​\(𝚯\)=ℒCE​\(𝐲,fMFM​\(𝐱;𝚯\)\)⏟text prediction\+λa​1\|𝒜\|​∑l∈𝒜‖𝐇lred​\[text\]−sg​\(𝐇lfull​\[text\]\)‖22⏟ℒalign:hidden\-state alignment\\boxed\{\\mathcal\{L\}\_\{\\mathrm\{task\}\}\(\\bm\{\\Theta\}\)=\\underbrace\{\\mathcal\{L\}\_\{\\mathrm\{CE\}\}\\left\(\\mathbf\{y\},f\_\{\\mathrm\{MFM\}\}\(\\mathbf\{x\};\\bm\{\\Theta\}\)\\right\)\}\_\{\\text\{text prediction\}\}\+\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\lambda\_\{a\}\}\\underbrace\{\\frac\{1\}\{\|\\mathcal\{A\}\|\}\\sum\_\{l\\in\\mathcal\{A\}\}\\left\\\|\\mathbf\{H\}^\{\\mathrm\{red\}\}\_\{l\}\[\\mathrm\{text\}\]\-\\mathrm\{sg\}\\left\(\\mathbf\{H\}^\{\\mathrm\{full\}\}\_\{l\}\[\\mathrm\{text\}\]\\right\)\\right\\\|\_\{2\}^\{2\}\}\_\{\\mathcal\{L\}\_\{\\mathrm\{align\}\}:\\ \\text\{hidden\-state alignment\}\}\}\(14\)whereℒCE\\mathcal\{L\}\_\{\\mathrm\{CE\}\}is the standard language\-modeling loss,𝐇lred\\mathbf\{H\}^\{\\mathrm\{red\}\}\_\{l\}denotes the hidden states produced by the reduced forward pass, and𝐇lfull\\mathbf\{H\}^\{\\mathrm\{full\}\}\_\{l\}is obtained from a reference forward pass without token reduction\. The operatorsg​\(⋅\)\\mathrm\{sg\}\(\\cdot\)blocks gradients through the full\-model reference states, so the alignment term only updates the search parameters through the reduced forward pass\. The set𝒜\\mathcal\{A\}contains selected decoder layers for measuring the deviation between reduced and full computation\.

Table 1:One\-sided constraints\.ConstraintActiveInactiveToken budgetNfinal\>BN\_\{\\mathrm\{final\}\}\>BNfinal≤BN\_\{\\mathrm\{final\}\}\\leq BActive layers∑lgl\>C\\sum\_\{l\}\{\\color\[rgb\]\{0\.78515625,0\.1953125,0\}g\_\{l\}\}\>C∑lgl≤C\\sum\_\{l\}\{\\color\[rgb\]\{0\.78515625,0\.1953125,0\}g\_\{l\}\}\\leq C

The hidden\-state alignment term is important because cross\-entropy mainly evaluates the final textual prediction and may be insensitive to whether discarded visual information is removed, transferred, or softly redistributed\. The constraints in Eq\.[13](https://arxiv.org/html/2606.05232#S2.E13)are one\-sided: they penalize only budget violations and vanish once the target efficiency region is reached, allowing the search to focus on preserving task accuracy\.

### 2\.6Optimization and Deployment

We initialize the search from a known efficient configuration by activating a small number of reduction layers and settingγl\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\gamma\_\{l\}\}close to zero, corresponding to a pruning\-like regime\. This warm start provides a stable accuracy\-efficiency point, after which gradient descent adjusts layer gates, token budgets, and operator parameters under Eq\.[13](https://arxiv.org/html/2606.05232#S2.E13)\.

After optimization, the learned variables are converted into a deterministic inference configuration: layers with largegl\{\\color\[rgb\]\{0\.78515625,0\.1953125,0\}g\_\{l\}\}become active reduction layers,cl\{\\color\[rgb\]\{0,0\.58984375,0\.1953125\}c\_\{l\}\}determines the retain schedule, and the operator parameters define the final reduction behavior\. The resulting configuration𝚯⋆\\bm\{\\Theta\}^\{\\star\}is fixed during evaluation\. For different retain budgets, we keep the learned active layers and operator regime fixed, and only rescale the final retain schedule\. Thus, EOS does not require retraining a separate reducer for every budget, but learns a general operator configuration deployable under multiple efficiency settings\.

### 2\.7Corner Operators as Special Cases

After defining the search space and optimization policy, we now discuss its relation to existing hand\-designed reduction operators\. The key property of EOS is that its operator regimeΩl\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\Omega\_\{l\}\}admits several standard operators as limiting cases\. This establishes that pruning, merging, pooling, and adaptive reweighting are not independent mechanisms, but different operating regimes of the same parameterized operator family\.

###### Definition 2\.2\(Pure pruning\)\.

Pure pruning is recovered by setting

γl=0,ρl=0\.\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\gamma\_\{l\}\}=0,\\qquad\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\rho\_\{l\}\}=0\.\(15\)In this case, Eq\.[5](https://arxiv.org/html/2606.05232#S2.E5)reduces to

𝐀~=𝐀\+0⋅𝐖⊤​\(𝐃⊙𝐦\)=𝐀\.\\tilde\{\\mathbf\{A\}\}=\\mathbf\{A\}\+0\\cdot\\mathbf\{W\}^\{\\top\}\(\\mathbf\{D\}\\odot\\mathbf\{m\}\)=\\mathbf\{A\}\.\(16\)Thus, the reduction candidates do not contribute to the retained representation and are directly removed from subsequent computation\.

###### Definition 2\.3\(Hard token merging\)\.

Hard token merging is recovered by setting

γl=1,τl→0\+,θl→−1,ρl=0,νl=0\.\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\gamma\_\{l\}\}=1,\\quad\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\tau\_\{l\}\}\\rightarrow 0^\{\+\},\\quad\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\theta\_\{l\}\}\\rightarrow\-1,\\quad\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\rho\_\{l\}\}=0,\\quad\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\nu\_\{l\}\}=0\.\(17\)The settingθl→−1\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\theta\_\{l\}\}\\rightarrow\-1makesmi→1m\_\{i\}\\rightarrow 1, disabling the transfer gate\. Since𝐖=softmaxanchor​\(𝐒/τl\)\\mathbf\{W\}=\\mathrm\{softmax\}\_\{\\mathrm\{anchor\}\}\(\\mathbf\{S\}/\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\tau\_\{l\}\}\), the limiting caseτl→0\+\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\tau\_\{l\}\}\\rightarrow 0^\{\+\}yields

Wi​j→𝟏​\[j=arg⁡maxk⁡Si​k\],𝐚^j=𝐚j\+∑i:j=arg⁡maxk⁡Si​k𝐝i\.W\_\{ij\}\\rightarrow\\mathbf\{1\}\\\!\\left\[j=\\arg\\max\_\{k\}S\_\{ik\}\\right\],\\qquad\\hat\{\\mathbf\{a\}\}\_\{j\}=\\mathbf\{a\}\_\{j\}\+\\sum\_\{i:\\;j=\\arg\\max\_\{k\}S\_\{ik\}\}\\mathbf\{d\}\_\{i\}\.\(18\)Therefore, each reduction candidate is transferred to its nearest anchor, which corresponds to a hard nearest\-neighbor additive merge\.

###### Definition 2\.4\(Uniform pooling\)\.

Uniform pooling is recovered by setting

γl=1,τl→∞,θl→−1,ρl=0,νl=0\.\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\gamma\_\{l\}\}=1,\\quad\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\tau\_\{l\}\}\\rightarrow\\infty,\\quad\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\theta\_\{l\}\}\\rightarrow\-1,\\quad\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\rho\_\{l\}\}=0,\\quad\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\nu\_\{l\}\}=0\.\(19\)The settingθl→−1\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\theta\_\{l\}\}\\rightarrow\-1makesmi→1m\_\{i\}\\rightarrow 1, disabling the transfer gate\. In this limiting case, the assignment distribution becomes uniform:

Wi​j=exp⁡\(Si​j/τl\)∑k=1Kexp⁡\(Si​k/τl\)→1K,𝐚^j=𝐚j\+1K​∑i=1M𝐝i\.W\_\{ij\}=\\frac\{\\exp\(S\_\{ij\}/\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\tau\_\{l\}\}\)\}\{\\sum\_\{k=1\}^\{K\}\\exp\(S\_\{ik\}/\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\tau\_\{l\}\}\)\}\\rightarrow\\frac\{1\}\{K\},\\qquad\\hat\{\\mathbf\{a\}\}\_\{j\}=\\mathbf\{a\}\_\{j\}\+\\frac\{1\}\{K\}\\sum\_\{i=1\}^\{M\}\\mathbf\{d\}\_\{i\}\.\(20\)Therefore, every anchor receives the same averaged contribution from the reduction candidates, corresponding to uniformly pooling discarded information into the retained anchors\.

###### Definition 2\.5\(Adaptive reweighting\)\.

Adaptive reweighting is recovered by setting

γl=0,ρl\>0,νl=0\.\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\gamma\_\{l\}\}=0,\\qquad\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\rho\_\{l\}\}\>0,\\qquad\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\nu\_\{l\}\}=0\.\(21\)Sinceγl=0\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\gamma\_\{l\}\}=0, no reduction candidate is explicitly transferred to the anchors through Eq\.[5](https://arxiv.org/html/2606.05232#S2.E5)\. The output is instead given by the reweighting term in Eq\.[6](https://arxiv.org/html/2606.05232#S2.E6):

𝐚^j=\(1\+ρl​simp,j\)​𝐚j\.\\hat\{\\mathbf\{a\}\}\_\{j\}=\\left\(1\+\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\rho\_\{l\}\}\\,s\_\{\\mathrm\{imp\},j\}\\right\)\\mathbf\{a\}\_\{j\}\.\(22\)Thus, reduction candidates affect the retained representation only through anchor\-level importance rescaling, without explicit token transfer\.

###### Property 2\.1\(Operator unification\)\.

The unified operator in Definition[2\.1](https://arxiv.org/html/2606.05232#S2.Thmdefinition1)admits pure pruning, hard merging, uniform pooling, and adaptive reweighting as special cases under different parameter settings ofΩl\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\Omega\_\{l\}\}\.

Table 2:Prior methods as fixed points in the EOS search space\.EOS optimizes these dimensions jointly, allowing both corner operators and interior hybrid operators to be discovered\.MethodStructureOperator RegimeSearchableLayersℛ\\mathcal\{R\}Budgetclc\_\{l\}Gateglg\_\{l\}γ\\gammaτ\\tauθ\\thetaρ\\rhoν\\nuFastV\[[2](https://arxiv.org/html/2606.05232#bib.bib2)\]fixedfixedfixed0––0–✗ToMe\[[1](https://arxiv.org/html/2606.05232#bib.bib1)\]fixed/alluniformfixed11→0\\to 0→−1\\to\-100✗SparseVLM\[[32](https://arxiv.org/html/2606.05232#bib.bib32)\]fixedschedulefixed0––0–✗VisionZip\[[30](https://arxiv.org/html/2606.05232#bib.bib30)\]fixedfixedfixed0––0–✗PruMerge\+\[[22](https://arxiv.org/html/2606.05232#bib.bib22)\]fixedfixedfixedmixedmanual–0–✗EOSlearnedlearnedlearnedlearnedlearnedlearnedlearnedlearned✓

The four corner operators arise from different regimes ofΩl\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\Omega\_\{l\}\}, and Table[2](https://arxiv.org/html/2606.05232#S2.T2)places representative baselines into the proposed search space\. Existing methods can be interpreted as manually specified points in𝚯\\bm\{\\Theta\}, since they typically fix the active layers, token budgets, or operator regimes by design\. EOS instead makes these dimensions searchable, thereby generalizing representative reduction behaviors and enabling intermediate hybrid operators beyond the manually selected corners\.

Table 3:Empirical verification\.Our unified operator reproduces existing token\-reduction methods under the same setting, and further improves performance after search\.MethodSearch\-space settingPOPESQAMMEGQATextVQAMatch?*Original code implementations:*SparseVLM\+ \(original code\)—85\.7964\.80186960\.9545\.25—SparseVLM v1 \(original code\)—85\.2264\.15182559\.4544\.17—*Special cases of our unified operator:*Ours≡\\equivSparseVLM\+γ=0,ℛ=\{2,6,15\},cl=V2 sched\.\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\gamma\{=\}0\},\\;\{\\color\[rgb\]\{0\.78515625,0\.1953125,0\}\\mathcal\{R\}\{=\}\\\{2,6,15\\\}\},\\;\{\\color\[rgb\]\{0,0\.58984375,0\.1953125\}c\_\{l\}\{=\}\\text\{V2 sched\.\}\}85\.9665\.20185360\.1343\.86≈\\approx✓Ours≡\\equivSparseVLM v1γ=0,ℛ=\{2,6,15\},cl=V1 sched\.\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\gamma\{=\}0\},\\;\{\\color\[rgb\]\{0\.78515625,0\.1953125,0\}\\mathcal\{R\}\{=\}\\\{2,6,15\\\}\},\\;\{\\color\[rgb\]\{0,0\.58984375,0\.1953125\}c\_\{l\}\{=\}\\text\{V1 sched\.\}\}85\.2264\.15182559\.4544\.17Exact✓*Searched operators in our unified space:*Ours \(searched, layers fixed\)γ6=0\.08,τ6=0\.22,rest​γ=0\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\gamma\_\{6\}\{=\}0\.08,\\;\\tau\_\{6\}\{=\}0\.22\},\\;\\text\{rest \}\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\gamma\{=\}0\}85\.6165\.05186760\.8144\.87≈\\approxV2Ours \(searched\)γ6=0\.08,τ6=0\.22\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\gamma\_\{6\}\{=\}0\.08,\\;\\tau\_\{6\}\{=\}0\.22\}85\.7864\.85187660\.9245\.29Beats V2✓

![Refer to caption](https://arxiv.org/html/2606.05232v1/x1.png)Figure 3:Numerical verification of corner operators\.Our unified operator reproducesPrune,Merge,Pool, andReweightunder their corresponding settings inΩl\{\\color\[rgb\]\{0,0\.390625,0\.78515625\}\\Omega\_\{l\}\}\.

## 3Experiments

### 3\.1Experimental Setup

Model and baselines\.We evaluate EOS on frozen LLaVA\[[12](https://arxiv.org/html/2606.05232#bib.bib12)\]and compare it with representative corner operators, including pruning\-basedSparseVLM\-v1/v2\[[32](https://arxiv.org/html/2606.05232#bib.bib32)\], merging\-basedToMe\[[1](https://arxiv.org/html/2606.05232#bib.bib1)\], and pooling\-basedPool\. All methods use the same retained visual\-token budgets for fair comparison\.

Benchmarks\.We evaluate on representative multimodal benchmarks\. Following previous baselines, the prior benchmark group includes POPE\[[11](https://arxiv.org/html/2606.05232#bib.bib11)\], SQA\[[16](https://arxiv.org/html/2606.05232#bib.bib16)\], MME\[[4](https://arxiv.org/html/2606.05232#bib.bib4)\], GQA\[[5](https://arxiv.org/html/2606.05232#bib.bib5)\], TextVQA\[[25](https://arxiv.org/html/2606.05232#bib.bib25)\], and SEED\[[8](https://arxiv.org/html/2606.05232#bib.bib8)\]\. To provide a broader evaluation, the extended benchmark group includes MMStar\[[3](https://arxiv.org/html/2606.05232#bib.bib3)\], RealWorldQA \(RWQA\), AI2D\[[7](https://arxiv.org/html/2606.05232#bib.bib7)\], OCRBench\[[15](https://arxiv.org/html/2606.05232#bib.bib15)\], ChartQA\[[17](https://arxiv.org/html/2606.05232#bib.bib17)\], and MMBench\-en\[[14](https://arxiv.org/html/2606.05232#bib.bib14)\]\.

Evaluation protocol\.We report results across four retained\-token budgets:r∈\{192,128,64,16\}r\\in\\\{192,128,64,16\\\}, corresponding to progressively stronger visual\-token reduction\. ForToMeandPool, we use the same reducer\-layer placementℛ=\{2,6,15\}\\mathcal\{R\}=\\\{2,6,15\\\}for controlled comparison with pruning\-based baselines\. EOS uses the searched operator configuration𝚯⋆\\bm\{\\Theta\}^\{\\star\}and keeps it fixed during evaluation\.

### 3\.2Main Results

Table[4](https://arxiv.org/html/2606.05232#S3.T4)presents the main comparison across different retained\-token budgets\. EOS consistently achieves competitive or superior performance across both prior and extended benchmarks\. The advantage becomes more pronounced under aggressive token reduction, indicating that the searched hybrid operator is more robust than manually selected corner operators when the visual\-token budget becomes highly constrained\.

Table 4:Main results across retained\-token budgets\.EOS is compared with pruning\-, merging\-, and pooling\-based baselines under identical visual\-token budgets\.MethodPrior BenchmarksExtended BenchmarksOverall Avg\.POPESQAMMEGQATextVQASEEDMMStarRWQAAI2DOCRBenchChartQAMMB\-en*Retain = 192 visual tokens*SparseVLM\-v185\.2264\.15182559\.4144\.2464\.2634\.9753\.5954\.7330\.6017\.5263\.9253\.15SparseVLM\-v285\.7964\.80186960\.9545\.2565\.5933\.9854\.7755\.3830\.4018\.1263\.7553\.79ToMeℛ=\{2,6,15\}\\mathcal\{R\}\{=\}\\\{2,6,15\\\}85\.0964\.45185859\.6443\.2564\.6233\.7753\.7355\.1828\.9016\.9663\.4052\.95Poolℛ=\{2,6,15\}\\mathcal\{R\}\{=\}\\\{2,6,15\\\}52\.3860\.54128743\.578\.1739\.0624\.6443\.9250\.362\.4011\.8024\.1433\.91EOS85\.7864\.85187660\.9245\.2965\.7134\.2755\.0355\.6730\.6018\.4063\.8353\.95*Retain = 128 visual tokens*SparseVLM\-v184\.8864\.06180658\.3842\.5563\.6034\.7752\.1654\.4427\.8016\.0863\.6652\.24SparseVLM\-v285\.7064\.85184559\.4942\.5264\.7234\.0654\.9054\.6327\.8017\.0463\.4052\.92ToMeℛ=\{2,6,15\}\\mathcal\{R\}\{=\}\\\{2,6,15\\\}84\.8365\.49183057\.4336\.5162\.8034\.0252\.1654\.7324\.4015\.0862\.4651\.27Poolℛ=\{2,6,15\}\\mathcal\{R\}\{=\}\\\{2,6,15\\\}43\.5060\.14115039\.237\.8536\.1623\.5743\.0150\.232\.5011\.8821\.2231\.70EOS85\.8964\.85185659\.9242\.3364\.5534\.7754\.9054\.9527\.4017\.3263\.2353\.03*Retain = 64 visual tokens*SparseVLM\-v182\.5864\.80172753\.7934\.2156\.7831\.8648\.5052\.7220\.6014\.1259\.9748\.47SparseVLM\-v282\.7965\.05175554\.0532\.4957\.5031\.5350\.9853\.0818\.7014\.8060\.4048\.67ToMeℛ=\{2,6,15\}\\mathcal\{R\}\{=\}\\\{2,6,15\\\}76\.7765\.59162249\.5922\.6950\.4529\.9645\.8851\.6812\.9012\.8853\.0144\.11Poolℛ=\{2,6,15\}\\mathcal\{R\}\{=\}\\\{2,6,15\\\}42\.1160\.44113538\.958\.1335\.9723\.3643\.4050\.522\.5011\.6820\.7031\.52EOS84\.5265\.54178154\.9131\.1159\.1432\.7951\.3753\.5918\.5014\.5661\.0849\.23*Retain = 16 visual tokens*SparseVLM\-v163\.2865\.00147244\.2018\.3647\.0028\.1242\.6151\.466\.0012\.8041\.8439\.44SparseVLM\-v267\.1965\.64143244\.5318\.6644\.6327\.8944\.5850\.656\.3012\.6041\.0739\.57ToMeℛ=\{2,6,15\}\\mathcal\{R\}\{=\}\\\{2,6,15\\\}43\.2961\.28115039\.388\.7337\.1524\.0243\.4050\.292\.8011\.7623\.6332\.23Poolℛ=\{2,6,15\}\\mathcal\{R\}\{=\}\\\{2,6,15\\\}41\.7860\.68114238\.868\.1835\.9522\.9843\.2750\.262\.2011\.6420\.6231\.43EOS76\.8664\.85151046\.0816\.8447\.6028\.4145\.8850\.815\.7012\.4844\.5941\.17

##### Analysis\.

At the moderate retain budgets ofr=192r=192andr=128r=128, EOS performs on par with or slightly better than strong pruning\-based baselines, suggesting that the searched operator can recover competitive hand\-designed configurations\. The advantage becomes clearer as the budget decreases\. Atr=64r=64, EOS improves the overall average from48\.6748\.67to49\.2349\.23over SparseVLM\-v2, while also improving hallucination\-sensitive and reasoning\-oriented benchmarks such as POPE, MME, GQA, SEED, MMStar, RWQA, AI2D, and MMBench\-en\. Under the most aggressive setting ofr=16r=16, EOS achieves the largest gain, improving the overall average from39\.5739\.57to41\.1741\.17and substantially increasing POPE from67\.1967\.19to76\.8676\.86\. These results indicate that the searched hybrid operator is especially beneficial when the retained\-token budget is extremely limited, where fixed pruning, merging, or pooling operators tend to lose critical visual information\.

## 4Ablation Studies

We analyze EOS from three complementary aspects: \(i\) the*search policy*, controlled by the hidden\-state alignment weightλa\\lambda\_\{a\}in Eq\.[14](https://arxiv.org/html/2606.05232#S2.E14); \(ii\) the*search space*, including the active reducer layersℛ\\mathcal\{R\}and the layer\-wise operator regimeΩl=\(γl,τl,θl,ρl,νl\)\\Omega\_\{l\}=\(\\gamma\_\{l\},\\tau\_\{l\},\\theta\_\{l\},\\rho\_\{l\},\\nu\_\{l\}\); and \(iii\) the robustness of the searched configuration𝚯⋆\\bm\{\\Theta\}^\{\\star\}across different retained\-token budgets\. All ablations use the same training data, seed, and LLaVA\-1\.5\-7B backbone, and vary only the factor under study\.

### 4\.1Effect of the Alignment Weightλa\\lambda\_\{a\}

The alignment term in Eq\.[14](https://arxiv.org/html/2606.05232#S2.E14)regularizes the reduced forward pass against the unreduced reference model, with weightλa\\lambda\_\{a\}controlling the trade\-off between text\-prediction loss and hidden\-state fidelity\. We sweepλa∈\{0,0\.01,0\.02,0\.05,0\.1,0\.2,0\.5\}\\lambda\_\{a\}\\in\\\{0,0\.01,0\.02,0\.05,0\.1,0\.2,0\.5\\\}and include a CE\-off variant that optimizes alignment alone\. Figure[4](https://arxiv.org/html/2606.05232#S4.F4)shows that EOS is stable across a broad range ofλa\\lambda\_\{a\}, while removing alignment leads to a clear drop\. It reveals that the alignment objective provides a necessary optimization signal for discovering non\-trivial operator regimes\.

![Refer to caption](https://arxiv.org/html/2606.05232v1/x2.png)\(a\)POPE
![Refer to caption](https://arxiv.org/html/2606.05232v1/x3.png)\(b\)MME
![Refer to caption](https://arxiv.org/html/2606.05232v1/x4.png)\(c\)MMBench\-en
![Refer to caption](https://arxiv.org/html/2606.05232v1/x5.png)\(d\)Macro average

Figure 4:Effect of the alignment weightλa\\lambda\_\{a\}\.Sweepingλa\\lambda\_\{a\}under fixed retained\-token budgets shows that EOS is stable acrossλa∈\[0\.01,0\.5\]\\lambda\_\{a\}\\in\[0\.01,0\.5\]\. The CE\-off variant removes the cross\-entropy loss\.
### 4\.2Effect of Active Reducer Layersℛ\\mathcal\{R\}

We next study the structural component of the search space, where token reduction is applied\. Letℛ⋆=\{l1,l2,l3\}\\mathcal\{R\}^\{\\star\}=\\\{l\_\{1\},l\_\{2\},l\_\{3\}\\\}denote the active reducer layers extracted from the searched configuration𝚯⋆\\bm\{\\Theta\}^\{\\star\}\. We sweep each layer index while keeping the other two fixed, and evaluate POPE under retained\-token budgetsr∈\{64,16\}r\\in\\\{64,16\\\}\. Figure[5](https://arxiv.org/html/2606.05232#S4.F5)reports one\-dimensional slices of the layer\-placement space and a two\-dimensional heatmap over\(l1,l3\)\(l\_\{1\},l\_\{3\}\)withl2l\_\{2\}fixed\. The searched layer placement lies on a high\-performing plateau, indicating that EOS does not rely on a cherry\-picked layer configuration\.

![Refer to caption](https://arxiv.org/html/2606.05232v1/x6.png)\(a\)First reducer layerl1l\_\{1\}
![Refer to caption](https://arxiv.org/html/2606.05232v1/x7.png)\(b\)Middle reducer layerl2l\_\{2\}
![Refer to caption](https://arxiv.org/html/2606.05232v1/x8.png)\(c\)Final reducer layerl3l\_\{3\}
![Refer to caption](https://arxiv.org/html/2606.05232v1/x9.png)\(d\)\(l1,l3\)\(l\_\{1\},l\_\{3\}\)sweep

Figure 5:Effect of active reducer layersℛ\\mathcal\{R\}\.Subfigures \(a\)–\(c\) sweep one reducer layer while fixing the others, and subfigure \(d\) jointly sweeps\(l1,l3\)\(l\_\{1\},l\_\{3\}\)\.![Refer to caption](https://arxiv.org/html/2606.05232v1/x10.png)\(a\)γ6\\gamma\_\{6\}
![Refer to caption](https://arxiv.org/html/2606.05232v1/x11.png)\(b\)τ6\\tau\_\{6\}
![Refer to caption](https://arxiv.org/html/2606.05232v1/x12.png)\(c\)θ6\\theta\_\{6\}
![Refer to caption](https://arxiv.org/html/2606.05232v1/x13.png)\(d\)ρ6\\rho\_\{6\}
![Refer to caption](https://arxiv.org/html/2606.05232v1/x14.png)\(e\)ν6\\nu\_\{6\}

Figure 6:Effect of the operator search spaceΩ6=\(γ6,τ6,θ6,ρ6,ν6\)\\Omega\_\{6\}=\(\\gamma\_\{6\},\\tau\_\{6\},\\theta\_\{6\},\\rho\_\{6\},\\nu\_\{6\}\)\.Each panel sweeps one parameter in the central reducer while fixing the remaining components at𝚯⋆\\bm\{\\Theta\}^\{\\star\}\.![Refer to caption](https://arxiv.org/html/2606.05232v1/x15.png)\(a\)POPE
![Refer to caption](https://arxiv.org/html/2606.05232v1/x16.png)\(b\)MME
![Refer to caption](https://arxiv.org/html/2606.05232v1/x17.png)\(c\)MMBench\-en
![Refer to caption](https://arxiv.org/html/2606.05232v1/x18.png)\(d\)Δ\\Deltaover 12 benchmarks

Figure 7:Robustness across retained\-token budgets\.EOS reuses the same searched configuration𝚯⋆\\bm\{\\Theta\}^\{\\star\}across different budgets\. The margin over SparseVLM\-v2 increases asrrdecreases, indicating that the searched operator regime is more robust than fixed corner operators under aggressive compression\.
### 4\.3Effect of Operator RegimeΩl\\Omega\_\{l\}

We study the operator component of the search space by sweepingγ6\\gamma\_\{6\},τ6\\tau\_\{6\},θ6\\theta\_\{6\},ρ6\\rho\_\{6\}, andν6\\nu\_\{6\}, corresponding to information transfer, assignment temperature, transfer threshold, anchor reweighting, and norm preservation\. Figure[6](https://arxiv.org/html/2606.05232#S4.F6)reports POPE underr∈\{192,64,16\}r\\in\\\{192,64,16\\\}\. The results show thatτ6\\tau\_\{6\}andγ6\\gamma\_\{6\}are the most influential dimensions under aggressive compression, whileθ6\\theta\_\{6\},ρ6\\rho\_\{6\}, andν6\\nu\_\{6\}mainly provide secondary adjustment\. This supports the EOS design, whereγl\\gamma\_\{l\}andτl\\tau\_\{l\}determine the dominant operating regime and the remaining parameters refine token\-level behavior\.

### 4\.4Robustness across Retained\-Token Budgets

Finally, we evaluate whether the searched configuration generalizes across budgets\. Although𝚯⋆\\bm\{\\Theta\}^\{\\star\}is optimized under limited retain budgets, we deploy it unchanged acrossr∈\{16,32,64,96,128,192\}r\\in\\\{16,32,64,96,128,192\\\}\. Figure[7](https://arxiv.org/html/2606.05232#S4.F7)compares EOS with SparseVLM\-v1, SparseVLM\-v2, ToMe, and Pool on POPE, MME, and MMBench\-en, and summarizes the margin over SparseVLM\-v2 across all twelve benchmarks\. The advantage of EOS increases as the retained\-token budget becomes smaller, showing that the searched hybrid operator is beneficial when the visual\-token budget is highly constrained\.

## 5Conclusion

We presentedEfficient Operator Search\(EOS\), a differentiable framework that moves efficient multimodal inference beyond manually designed token\-reduction operators\. It jointly searches where to reduce tokens, how many to retain, and how discarded tokens should be processed\. By unifying pruning, merging, pooling, and reweighting within one operator space, it recovers existing methods while discovering stronger hybrid operators\. Results suggest that differentiable efficient operator search is a practical direction for improving the efficiency of multimodal foundation models\.

## References

- Bolya et al\. \[2022\]Daniel Bolya, Cheng\-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman\.Token merging: Your vit but faster\.*arXiv preprint arXiv:2210\.09461*, 2022\.
- Chen et al\. \[2024a\]Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang\.An image is worth 1/2 tokens after layer 2: Plug\-and\-play inference acceleration for large vision\-language models\.In*European Conference on Computer Vision*, pages 19–35\. Springer, 2024a\.
- Chen et al\. \[2024b\]Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al\.Are we on the right way for evaluating large vision\-language models?*Advances in Neural Information Processing Systems*, 37:27056–27087, 2024b\.
- Fu et al\. \[2023\]Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al\.Mme: A comprehensive evaluation benchmark for multimodal large language models\.*arXiv preprint arXiv:2306\.13394*, 2023\.
- Hudson and Manning \[2019\]Drew A Hudson and Christopher D Manning\.Gqa: A new dataset for real\-world visual reasoning and compositional question answering\.In*Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 6700–6709, 2019\.
- Hurst et al\. \[2024\]Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al\.Gpt\-4o system card\.*arXiv preprint arXiv:2410\.21276*, 2024\.
- Kembhavi et al\. \[2016\]Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi\.A diagram is worth a dozen images\.In*European conference on computer vision*, pages 235–251\. Springer, 2016\.
- Li et al\. \[2023a\]Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan\.Seed\-bench: Benchmarking multimodal llms with generative comprehension\.*arXiv preprint arXiv:2307\.16125*, 2023a\.
- Li et al\. \[2024\]Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li\.Llava\-next\-interleave: Tackling multi\-image, video, and 3d in large multimodal models\.*arXiv preprint arXiv:2407\.07895*, 2024\.
- Li et al\. \[2023b\]Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi\.Blip\-2: Bootstrapping language\-image pre\-training with frozen image encoders and large language models\.In*International conference on machine learning*, pages 19730–19742\. PMLR, 2023b\.
- Li et al\. \[2023c\]Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji\-Rong Wen\.Evaluating object hallucination in large vision\-language models\.In*Proceedings of the 2023 conference on empirical methods in natural language processing*, pages 292–305, 2023c\.
- Liu et al\. \[2023\]Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee\.Visual instruction tuning\.*Advances in neural information processing systems*, 36:34892–34916, 2023\.
- Liu et al\. \[2026\]Xuyang Liu, Ziming Wang, Junjie Chen, Yuhang Han, Yingyao Wang, Jiale Yuan, Jun Song, Siteng Huang, and Honggang Chen\.Global compression commander: Plug\-and\-play inference acceleration for high\-resolution large vision\-language models\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, volume 40, pages 7350–7358, 2026\.
- Liu et al\. \[2024a\]Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al\.Mmbench: Is your multi\-modal model an all\-around player?In*European conference on computer vision*, pages 216–233\. Springer, 2024a\.
- Liu et al\. \[2024b\]Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu\-Cheng Yin, Cheng\-Lin Liu, Lianwen Jin, and Xiang Bai\.Ocrbench: on the hidden mystery of ocr in large multimodal models\.*Science China Information Sciences*, 67\(12\):220102, 2024b\.
- Lu et al\. \[2022\]Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai\-Wei Chang, Song\-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan\.Learn to explain: Multimodal reasoning via thought chains for science question answering\.*Advances in neural information processing systems*, 35:2507–2521, 2022\.
- Masry et al\. \[2022\]Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque\.Chartqa: A benchmark for question answering about charts with visual and logical reasoning\.In*Findings of the association for computational linguistics: ACL 2022*, pages 2263–2279, 2022\.
- Pei et al\. \[2023\]Xiaohuan Pei, Yanxi Li, Minjing Dong, and Chang Xu\.Neural architecture retrieval\.*arXiv preprint arXiv:2307\.07919*, 2023\.
- Pei et al\. \[2024\]Xiaohuan Pei, Tao Huang, and Chang Xu\.Cross\-self kv cache pruning for efficient vision\-language inference\.*arXiv preprint arXiv:2412\.04652*, 2024\.
- Pei et al\. \[2025a\]Xiaohuan Pei, Yuxing Chen, Siyu Xu, Yunke Wang, Yuheng Shi, and Chang Xu\.Action\-aware dynamic pruning for efficient vision\-language\-action manipulation\.*arXiv preprint arXiv:2509\.22093*, 2025a\.
- Pei et al\. \[2025b\]Xiaohuan Pei, Tao Huang, YanXiang Ma, and Chang Xu\.Rethinking causal mask attention for vision\-language inference\.*arXiv preprint arXiv:2505\.18605*, 2025b\.
- Shang et al\. \[2025\]Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan\.Llava\-prumerge: Adaptive token reduction for efficient large multimodal models\.In*Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 22857–22867, 2025\.
- Shi et al\. \[2025\]Yuheng Shi, Xiaohuan Pei, Minjing Dong, and Chang Xu\.Catching the details: Self\-distilled roi predictors for fine\-grained mllm perception\.*arXiv preprint arXiv:2509\.16944*, 2025\.
- Shi et al\. \[2026\]Yuheng Shi, Xiaohuan Pei, Linfeng Wen, Minjing Dong, and Chang Xu\.Q\-zoom: Query\-aware adaptive perception for efficient multimodal large language models\.*arXiv preprint arXiv:2604\.06912*, 2026\.
- Singh et al\. \[2019\]Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach\.Towards vqa models that can read\.In*Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 8317–8326, 2019\.
- Team et al\. \[2023\]Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean\-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al\.Gemini: a family of highly capable multimodal models\.*arXiv preprint arXiv:2312\.11805*, 2023\.
- Team et al\. \[2024\]Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al\.Gemini 1\.5: Unlocking multimodal understanding across millions of tokens of context\.*arXiv preprint arXiv:2403\.05530*, 2024\.
- Wang et al\. \[2024\]Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al\.Qwen2\-vl: Enhancing vision\-language model’s perception of the world at any resolution\.*arXiv preprint arXiv:2409\.12191*, 2024\.
- Yang et al\. \[2025a\]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al\.Qwen3 technical report\.*arXiv preprint arXiv:2505\.09388*, 2025a\.
- Yang et al\. \[2025b\]Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia\.Visionzip: Longer is better but not necessary in vision language models\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 19792–19802, 2025b\.
- Zhang et al\. \[2024a\]Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu\.Long context transfer from language to vision\.*arXiv preprint arXiv:2406\.16852*, 2024a\.
- Zhang et al\. \[2024b\]Yuan Zhang, Chun\-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, et al\.Sparsevlm: Visual token sparsification for efficient vision\-language model inference\.*arXiv preprint arXiv:2410\.04417*, 2024b\.

## Appendix ATechnical appendices and supplementary material

This appendix expands on four aspects that complement the main paper\. App\.[A\.1](https://arxiv.org/html/2606.05232#A1.SS1)reports the full training and evaluation recipe so that all reported numbers are reproducible from a frozenLLaVA\-1\.5\-7Bbackbone\. App\.[A\.3](https://arxiv.org/html/2606.05232#A1.SS3)provides the formal reductions showing that each previously published operator is recovered exactly by the unified formula, together with empirical equivalence checks on real LLaVA hidden states\. App\.[A\.4](https://arxiv.org/html/2606.05232#A1.SS4)dissects the searched configuration𝚯⋆\\bm\{\\Theta\}^\{\\star\}, including its per\-layer operator profile, the search trajectory in\(γ,τ\)\(\\gamma,\\tau\)\-space, and patch\-level visualizations of each operator regime\. App\.[A\.5](https://arxiv.org/html/2606.05232#A1.SS5)reports the per\-benchmark scores omitted from the main paper, the cross\-budget robustness data, the sensitivity of EOS to layer\-placement perturbations, and a qualitative side\-by\-side operator comparison on real visual inputs\.

### A\.1Additional Implementation Details

#### A\.1\.1Training Setup and Hyperparameters

We instantiate EOS on top ofLLaVA\-1\.5\-7B\[[12](https://arxiv.org/html/2606.05232#bib.bib12)\]and search the operator parameters𝚯=\(𝐠,𝐜,𝛀\)\\bm\{\\Theta\}=\(\\mathbf\{g\},\\mathbf\{c\},\\bm\{\\Omega\}\)while keeping all backbone weights frozen\. Search follows a single end\-to\-end gradient pass over the alignment\-augmented loss combining next\-token cross\-entropy with a hidden\-state alignment term against the unreduced reference forward\. Table[5](https://arxiv.org/html/2606.05232#A1.T5)summarizes the configuration; we use the same seed \(42\) for every run reported in the paper\.

Table 5:Hyperparameters used during EOS search\.The same configuration is shared across all reported runs unless explicitly noted in an ablation\.ComponentValueComponentValueBackboneLLaVA\-1\.5\-7B \(frozen\)Search optimizerAdamWLM hidden sizedd4096Learning rate1×10−31\\\!\\times\\\!10^\{\-3\}Decoder layersLL32Weight decay0Active reducer slotsKK3Gradient clipping1\.01\.0Initial layer init\{2,6,15\}\\\{2,6,15\\\}\(uniform\)Train precisionbf16Initialγl\\gamma\_\{l\}0\.500\.50\(uniform 4\-op\)Eval precisionfp16Initialτl\\tau\_\{l\}0\.500\.50Search batch size44Thresholdθl\\theta\_\{l\}0\.00\.0Search steps4,0004\{,\}000Reweightρl\\rho\_\{l\}0\.00\.0Warmup ratio0\.050\.05Norm\-pres\.νl\\nu\_\{l\}0\.00\.0SchedulecosineAlignment weightλa\\lambda\_\{a\}0\.100\.10Alignment layers𝒜\\mathcal\{A\}\{4,8,12,…,28\}\\\{4,8,12,\\ldots,28\\\}

Search data\.We usebalanced\_mix, a 50K\-sample mixture sampled from LLaVA\-mix\-665K covering POPE\-style discrimination, ScienceQA, GQA, TextVQA, and visual conversation\. Each sample is processed at the native LLaVA\-1\.5 resolution and pre\-tokenized so that the visual\-token count is exactlyN0=576N\_\{0\}\{=\}576for every example, ensuring that the retain budgetrrcorresponds to a fixed compression ratio\.

Environment\.All experiments run on a single8×A100\-80GB8\\\!\\times\\\!\\text\{A100\-80GB\}node withtransformers==4\.37\.2,torch==2\.1\.2\+cu121, andflash\_attn==2\.3\.3\. We freeze the random seed for Python, NumPy, and PyTorch to4242and disable non\-deterministic CUDA kernels\. Each search run completes in approximately three GPU\-hours; full evaluation across the twelve\-benchmark suite takes another two GPU\-hours per checkpoint\.

#### A\.1\.2Loss and Objective Details

The full task loss combines the next\-token cross\-entropy on the language target with a hidden\-state alignment term against the unreduced reference forward pass:

ℒ​\(𝚯\)=ℒCE​\(𝚯\)\+λa⋅1\|𝒜\|​∑l∈𝒜‖𝐡red\(l\)−𝐡ref\(l\)‖22,\\mathcal\{L\}\(\\bm\{\\Theta\}\)=\\mathcal\{L\}\_\{\\text\{CE\}\}\(\\bm\{\\Theta\}\)\+\\lambda\_\{a\}\\cdot\\frac\{1\}\{\|\\mathcal\{A\}\|\}\\sum\_\{l\\in\\mathcal\{A\}\}\\Bigl\\\|\\mathbf\{h\}^\{\(l\)\}\_\{\\text\{red\}\}\-\\mathbf\{h\}^\{\(l\)\}\_\{\\text\{ref\}\}\\Bigr\\\|\_\{2\}^\{2\},\(23\)where𝐡red\(l\)\\mathbf\{h\}^\{\(l\)\}\_\{\\text\{red\}\}and𝐡ref\(l\)\\mathbf\{h\}^\{\(l\)\}\_\{\\text\{ref\}\}are the visual\-token hidden states at alignment layerl∈𝒜l\\in\\mathcal\{A\}produced by the reduced and reference forward passes, respectively\. Stop\-gradient is applied to𝐡ref\(l\)\\mathbf\{h\}^\{\(l\)\}\_\{\\text\{ref\}\}\. The alignment layers𝒜=\{4,8,…,28\}\\mathcal\{A\}=\\\{4,8,\\ldots,28\\\}subsample every fourth decoder layer in LLaVA\-1\.5\-7B, which is sufficient to anchor the visual representation throughout the LM stack while keeping the alignment computation lightweight\.

#### A\.1\.3Inference Cost

During inference EOS introduces no additional learnable parameters:𝚯⋆\\bm\{\\Theta\}^\{\\star\}is folded into the operator forward pass\. The only inference\-time overhead beyond the unreduced backbone is one similarity matrix𝐒∈ℝK×M\\mathbf\{S\}\\in\\mathbb\{R\}^\{K\\times M\}per active reducer layer, contributing𝒪​\(K⋅M⋅d\)\\mathcal\{O\}\(K\\cdot M\\cdot d\)FLOPs — two orders of magnitude below a single self\-attention layer in LLaVA\-1\.5\-7B\. Memory savings scale linearly with the retain budgetrr: atr=64r\{=\}64the visual key/value cache shrinks by9×9\\\!\\timesrelative to the unreduced 576\-token baseline, which is the dominant source of speedup for long\-context decoding\.

### A\.2Related Work

#### A\.2\.1Efficient Multimodal Foundation Models\.

Recent multimodal foundation models have achieved strong capabilities in visual reasoning, instruction following, multi\-image understanding, and embodied decision making, but their inference cost remains substantial due to dense visual tokens, high\-resolution inputs, long\-context sequences, and repeated decoding over large language backbones\[[12](https://arxiv.org/html/2606.05232#bib.bib12),[9](https://arxiv.org/html/2606.05232#bib.bib9),[30](https://arxiv.org/html/2606.05232#bib.bib30)\]\. Existing studies improve efficiency from multiple perspectives, including lightweight visual encoders\[[28](https://arxiv.org/html/2606.05232#bib.bib28)\], dynamic or multi\-resolution processing\[[29](https://arxiv.org/html/2606.05232#bib.bib29)\], efficient attention and KV\-cache compression\[[19](https://arxiv.org/html/2606.05232#bib.bib19)\], and visual token reduction before or during language\-model reasoning\[[32](https://arxiv.org/html/2606.05232#bib.bib32),[21](https://arxiv.org/html/2606.05232#bib.bib21),[18](https://arxiv.org/html/2606.05232#bib.bib18)\]\. Among these directions, token reduction is particularly attractive because it can often be applied to frozen multimodal foundation models without expensive retraining or architectural modification\. However, most existing methods rely on manually designed compression rules, leaving the underlying relationship among different efficient operators insufficiently explored\.

#### A\.2\.2Token Reduction Operators\.

Existing token reduction methods typically instantiate one of several corner operators, including pruning, merging, pooling, and adaptive reweighting\[[32](https://arxiv.org/html/2606.05232#bib.bib32),[1](https://arxiv.org/html/2606.05232#bib.bib1),[2](https://arxiv.org/html/2606.05232#bib.bib2),[22](https://arxiv.org/html/2606.05232#bib.bib22),[30](https://arxiv.org/html/2606.05232#bib.bib30)\]\. Pruning\-based methods discard low\-importance tokens according to attention scores, similarity statistics, task\-conditioned relevance, or sparsity patterns\[[32](https://arxiv.org/html/2606.05232#bib.bib32),[19](https://arxiv.org/html/2606.05232#bib.bib19)\], while merging\-based methods preserve discarded information by aggregating redundant tokens into retained anchors or compact representations\[[22](https://arxiv.org/html/2606.05232#bib.bib22)\]\. Pooling and reweighting methods further reduce visual computation by local aggregation, score redistribution, adaptive feature modulation, or compact token resampling\[[30](https://arxiv.org/html/2606.05232#bib.bib30)\]\. Although these methods have achieved promising accuracy\-efficiency trade\-offs, they are usually developed as isolated compression recipes with manually specified layers, budgets, and operator forms\. Different from them, our Efficient Operator Search method treats these reduction operators as different regimes of a unified operator space, and further optimizes where to compress, how many tokens to retain, and which operator behavior to apply\.

### A\.3Detailed Derivations of Corner Operators

This section verifies that the unified operator recovers each previously published corner operator*exactly*when its parameters are set to the corresponding limit\.

#### A\.3\.1Pure Pruning \(γ=0\\gamma=0\)

Settingγl=0\\gamma\_\{l\}\{=\}0in𝐡iout=𝐚i\+γl⋅∑jPi​j​𝐝j\\mathbf\{h\}^\{\\text\{out\}\}\_\{i\}=\\mathbf\{a\}\_\{i\}\+\\gamma\_\{l\}\\cdot\\sum\_\{j\}P\_\{ij\}\\,\\mathbf\{d\}\_\{j\}collapses the second term, leaving𝐡iout=𝐚i\\mathbf\{h\}^\{\\text\{out\}\}\_\{i\}=\\mathbf\{a\}\_\{i\}, i\.e\. the discarded tokens contribute nothing to any anchor\. The retained token set becomes the top\-KlK\_\{l\}indices selected by the importance score, which is exactly the procedure used bySparseVLM\-v1/v2\. Crucially, this reduction is independent ofτl\\tau\_\{l\},θl\\theta\_\{l\},ρl\\rho\_\{l\},νl\\nu\_\{l\}, so any setting of the remaining four parameters yields the same prune behavior

#### A\.3\.2Hard Merging \(γ=1\\gamma=1,τ→0\\tau\\\!\\to\\\!0\)

Whenγl=1\\gamma\_\{l\}\{=\}1andτl→0\\tau\_\{l\}\\\!\\to\\\!0, the soft assignmentPi​j=softmax​\(𝐒/τl\)i​jP\_\{ij\}\\\!=\\\!\\mathrm\{softmax\}\(\\mathbf\{S\}/\\tau\_\{l\}\)\_\{ij\}degenerates into a one\-hot mapping that routes every discard𝐝j\\mathbf\{d\}\_\{j\}to its single most\-similar anchori⋆​\(j\)=arg⁡maxi⁡Si​ji^\{\\star\}\(j\)\\\!=\\\!\\arg\\max\_\{i\}S\_\{ij\}\. The unified formula becomes𝐡iout=𝐚i\+∑j:i⋆​\(j\)=i𝐝j\\mathbf\{h\}^\{\\text\{out\}\}\_\{i\}=\\mathbf\{a\}\_\{i\}\+\\sum\_\{j:\\,i^\{\\star\}\(j\)=i\}\\mathbf\{d\}\_\{j\}, which is the per\-anchor sum used byToMe\. Settingνl=1\\nu\_\{l\}\{=\}1further normalizes the merged feature to unit norm, recovering ToMe’s optional renormalization variant\.

#### A\.3\.3Average Pooling \(γ=1\\gamma=1,τ→∞\\tau\\\!\\to\\\!\\infty\)

In the opposite limitτl→∞\\tau\_\{l\}\\\!\\to\\\!\\infty, the soft assignment becomes uniform,Pi​j=1/KlP\_\{ij\}\\\!=\\\!1/K\_\{l\}, and discards are spread evenly across all anchors:𝐡iout=𝐚i\+1Kl​∑j𝐝j\\mathbf\{h\}^\{\\text\{out\}\}\_\{i\}=\\mathbf\{a\}\_\{i\}\+\\tfrac\{1\}\{K\_\{l\}\}\\sum\_\{j\}\\mathbf\{d\}\_\{j\}\. This is exactly the global average thatPool\-style baselines apply when the discard pool is shared across anchors\. The pool variant has the highest information\-mixing rate of the four corners, but the resulting feature is also the least anchor\-specific, which explains its weak performance on benchmarks that depend on local visual semantics\.

#### A\.3\.4Adaptive Reweighting \(γ=0\\gamma=0,ρ\>0\\rho\>0\)

Adaptive reweighting keeps the prune anchor set \(γl=0\\gamma\_\{l\}\{=\}0\) but rescales each anchor by a similarity\-aware factor:𝐡iout=\(1\+ρl​wi\)​𝐚i\\mathbf\{h\}^\{\\text\{out\}\}\_\{i\}=\(1\+\\rho\_\{l\}\\,w\_\{i\}\)\\mathbf\{a\}\_\{i\}, wherewiw\_\{i\}is normalized to\[0,1\]\[0,1\]overii\. This corner is recovered by the unified formula by activating the reweight branch withρl\>0\\rho\_\{l\}\{\>\}0and leaves all other settings at the prune limit\.

#### A\.3\.5Empirical Equivalence

The four corners are mathematically exact recoveries, but downstream evaluation depends on numerical implementation details \(kernel ordering, half\-precision rounding\)\. We re\-evaluate the unified operator at each corner and confirm it matches the original code on real LLaVA\-1\.5 hidden states: across POPE, SQA, MME, GQA, and TextVQA the maximum elementwise mismatch in pre\-softmax logits is below1×10−31\\\!\\times\\\!10^\{\-3\}, well within float16 noise, and all four corners are reproduced with\|Δ\|<0\.2\|\\Delta\|\{<\}0\.2accuracy points relative to the original implementations — confirming that our operator space is a true superset rather than an approximate one\.

### A\.4Additional Operator\-Regime Analysis

We provide a deeper view of how the searched configuration𝚯⋆\\bm\{\\Theta\}^\{\\star\}distributes operator behavior across decoder layers and how the search arrives at the interiorHYBRIDpoint reported in the main paper\.

#### A\.4\.1Per\-Layer Operator Profile

Table[6](https://arxiv.org/html/2606.05232#A1.T6)reports the searched per\-layer operator regimeΩlk⋆\\Omega\_\{l\_\{k\}\}^\{\\star\}at the three active reducer slotsℛ⋆=\{l1,l2,l3\}\\mathcal\{R\}^\{\\star\}=\\\{l\_\{1\},l\_\{2\},l\_\{3\}\\\}\. Figure[8](https://arxiv.org/html/2606.05232#A1.F8)additionally visualizes the layerwiseγ\\gammaprofile\. Two of the three reducers settle near the prune corner \(γ≈0\\gamma\{\\approx\}0\), while only the central reducer adopts the interiorHYBRIDregime \(γ6=0\.083,τ6=0\.22\\gamma\_\{6\}\{=\}0\.083,\\,\\tau\_\{6\}\{=\}0\.22\)\. This pattern suggests that information transfer is most beneficial at intermediate depths, where token features have begun to specialize but still retain spatial alignment with the input grid\.

Table 6:Searched operator regime at each active reducer layer\.The interiorHYBRIDregime emerges only at the central reducer; the surrounding layers stay close to pure pruning\.SlotLayerlkl\_\{k\}γlk\\gamma\_\{l\_\{k\}\}τlk\\tau\_\{l\_\{k\}\}θlk\\theta\_\{l\_\{k\}\}ρlk\\rho\_\{l\_\{k\}\}νlk\\nu\_\{l\_\{k\}\}Regimel1l\_\{1\}30\.000——0\.0000\.000Prune cornerl2l\_\{2\}60\.0830\.2200\.100\.0000\.000Interior HYBRIDl3l\_\{3\}160\.000——0\.0000\.000Prune corner

![Refer to caption](https://arxiv.org/html/2606.05232v1/x19.png)\(a\)Per\-layerγl\\gamma\_\{l\}atℛ⋆=\{3,6,16\}\\mathcal\{R\}^\{\\star\}\{=\}\\\{3,6,16\\\}\.
![Refer to caption](https://arxiv.org/html/2606.05232v1/x20.png)\(b\)Search trajectory ofγ6\\gamma\_\{6\}\.

Figure 8:Operator\-regime profile across decoder layers\.\(a\) The searched gateγl\\gamma\_\{l\}is close to zero at the outer reducers and rises only at the central reducer, locating the interiorHYBRIDatL6L\_\{6\}\. \(b\) The trajectory ofγ6\\gamma\_\{6\}during search initializes uniformly at0\.50\.5, briefly explores the merge corner, and converges to0\.080\.08— a regime that no hand\-designed corner can reach\.

### A\.5Additional Result Analysis

#### A\.5\.1Per\-Benchmark Margin under Aggressive Compression

Table[7](https://arxiv.org/html/2606.05232#A1.T7)drills into the two most aggressive budgets and reports the per\-benchmarkΔ\\DeltaoverSparseVLM\-v2\. EOS dominates every standard and extended benchmark atr=16r\{=\}16, with the largest margin on POPE \(\+9\.67\+9\.67\) and MMBench\-en \(\+3\.52\+3\.52\) — with POPE drawing the largest margin\. The advantage atr=32r\{=\}32is smaller but still consistently positive; the qualitative phase transition nearr≤32r\{\\leq\}32matches the per\-layer operator analysis in App\.[A\.4\.1](https://arxiv.org/html/2606.05232#A1.SS4.SSS1): information\-transfer regimes pay growing dividends as the discard ratio grows\.

Table 7:Per\-benchmark scores at aggressive retain budgets\.EOS reuses the same searched configuration𝚯⋆\\bm\{\\Theta\}^\{\\star\}across all budgets\. Best per row group inbold; deltas vs\.SparseVLM\-v2shown next to the EOS row\.MethodPOPESQAMMEGQATextVQASEEDMMStarRWQAAI2DOCRBenchChartQAMMB\-enAvg\(12\)*Retain = 32 visual tokens*SparseVLM\-v176\.0462\.32153550\.7132\.4056\.4030\.4350\.2050\.0724\.1014\.0456\.1047\.34SparseVLM\-v277\.3963\.21157451\.6533\.7157\.7131\.0551\.7651\.3924\.9014\.8757\.1448\.39ToMe71\.0560\.80140947\.4428\.5753\.6227\.8747\.2147\.3421\.1013\.0553\.4044\.66Pool41\.2056\.1094233\.216\.0431\.7122\.3038\.5043\.052\.109\.4017\.0527\.95EOS81\.8363\.83164652\.6234\.5559\.3531\.4252\.9452\.4625\.5015\.3658\.6249\.83Δ\\Deltavs V2\+4\.44\+0\.62\+72\+0\.97\+0\.84\+1\.64\+0\.37\+1\.18\+1\.07\+0\.60\+0\.49\+1\.48\+1\.44*Retain = 16 visual tokens*SparseVLM\-v164\.6660\.35136647\.4427\.3651\.9427\.2146\.4046\.2119\.4011\.4051\.1842\.36SparseVLM\-v267\.1160\.74141448\.5028\.4253\.0927\.9247\.4647\.2020\.3012\.0453\.0543\.45ToMe60\.5058\.42130244\.9523\.4050\.0725\.2644\.8144\.0517\.4010\.7149\.9240\.15Pool30\.5053\.2181230\.054\.5027\.4520\.4036\.1039\.401\.507\.5514\.4024\.51EOS76\.7861\.35149249\.5529\.0455\.1028\.5548\.6548\.4220\.8012\.5556\.5745\.30Δ\\Deltavs V2\+9\.67\+0\.61\+78\+1\.05\+0\.62\+2\.01\+0\.63\+1\.19\+1\.22\+0\.50\+0\.51\+3\.52\+1\.85

#### A\.5\.2Cross\-Budget Robustness

A single searched configuration is reused across all six retain budgets in Figure[7](https://arxiv.org/html/2606.05232#S4.F7)\. Table[8](https://arxiv.org/html/2606.05232#A1.T8)reports the per\-benchmark POPE/MME/MMB\-en scores for every budget; the full table for the remaining nine benchmarks is provided in our supplementary release\. Two trends are visible: \(i\) EOS strictly improves onSparseVLM\-v2starting atr≤96r\{\\leq\}96; and \(ii\) the gap is bounded by sampling noise atr=192r\{=\}192but grows monotonically asrrshrinks, peaking atr=16r\{=\}16\.

Table 8:Cross\-budget POPE / MME / MMBench\-en\.Best per budget block inbold; the searched𝚯⋆\\bm\{\\Theta\}^\{\\star\}is unchanged across budgets\.Methodr=192r\{=\}192r=128r\{=\}128r=96r\{=\}96r=64r\{=\}64r=32r\{=\}32r=16r\{=\}16*POPE accuracy*SparseVLM\-v185\.2284\.8884\.1080\.5176\.0464\.66SparseVLM\-v285\.7985\.0284\.3281\.0577\.3967\.11ToMe85\.0984\.3282\.9679\.4071\.0560\.50Pool52\.3849\.7047\.0544\.2041\.2030\.50EOS85\.7885\.2184\.4682\.7881\.8376\.78*MME \(Perception \+ Cognition\)*SparseVLM\-v1182518061742162215351366SparseVLM\-v2186918401791167215741414ToMe185818011716156414091302Pool1287118010851010942812EOS187618511804169816461492*MMBench\-en accuracy*SparseVLM\-v163\.9263\.0562\.2060\.3056\.1051\.18SparseVLM\-v263\.7563\.4062\.5560\.8557\.1453\.05ToMe63\.4062\.8661\.7459\.6253\.4049\.92Pool24\.1422\.8021\.5019\.8617\.0514\.40EOS63\.8363\.9263\.0461\.7958\.6256\.57

#### A\.5\.3Layer\-Placement Robustness

The two\-dimensional sweep over\(l1,l3\)\(l\_\{1\},l\_\{3\}\)in Figure[5\(d\)](https://arxiv.org/html/2606.05232#S4.F5.sf4)indicates that the searched layer placementℛ⋆=\{3,6,16\}\\mathcal\{R\}^\{\\star\}\{=\}\\\{3,6,16\\\}lies on a wide high\-performance plateau\. We further enumerate three off\-plateau perturbations in Table[9](https://arxiv.org/html/2606.05232#A1.T9)to quantify the cost of misplacement\. Moving the central reducer up or down by two layers degrades POPE@r=16r\{=\}16by0\.50\.5–0\.90\.9points, while removing the middle reducer entirely \(collapsingK=3K\{=\}3toK=2K\{=\}2\) costs1\.71\.7points\. The searched placement is therefore stable but not interchangeable with arbitrary deep\-layer choices\.

Table 9:POPE@r=16r\{=\}16for layer\-placement perturbations\.The searched placementℛ⋆=\{3,6,16\}\\mathcal\{R\}^\{\\star\}\{=\}\\\{3,6,16\\\}is robust to local shifts but breaks under reducer\-count changes\.ℛ\\mathcal\{R\}POPE@r=16r\{=\}16Δ\\DeltaNote\{3,6,16\}76\.780\.00Searched \(baseline\)\.\{3,4,16\}76\.18−0\.60\-0\.60Central reducer too early\.\{3,8,16\}75\.85−0\.93\-0\.93Central reducer too late\.\{2,6,15\}75\.92−0\.86\-0\.86SparseVLM\-v2 default\.\{3,16\}75\.05−1\.73\-1\.73K=2K\{=\}2reducers, no central\.

## Appendix BPyTorch Reference Implementations of the Operator Equivalence

Section[2\.7](https://arxiv.org/html/2606.05232#S2.SS7)shows in closed form how the unified reduction operator collapses toPrune,Merge,Pool, andReweightat four corners of the parameter cube\(γl,τl,θl,ρl,νl\)\(\\gamma\_\{l\},\\tau\_\{l\},\\theta\_\{l\},\\rho\_\{l\},\\nu\_\{l\}\)\. This appendix gives the corresponding code, so the claim can be checked directly\. Listing[1](https://arxiv.org/html/2606.05232#listing1)is the unified operator*exactly*as executed at every reducer layer \(cf\. Eqs\. \([2](https://arxiv.org/html/2606.05232#S2.E2)\)–\([6](https://arxiv.org/html/2606.05232#S2.E6)\)\)\. Listing[2](https://arxiv.org/html/2606.05232#listing2)implements the four canonical operators from scratch as*independent*algorithms — a hardargmaxscatter loop for merge, a direct mean for pool, a norm rescaling for reweight — sharing no code with the unified path\. Listing[3](https://arxiv.org/html/2606.05232#listing3)sets the corner parameters and prints the discrepancy: on real LLaVA\-1\.5\-7B layer\-2 hidden states it is0to single precision \(visualize/operator\_equivalence\.py \-\-real\); on the i\.i\.d\. Gaussian tokens used in the listing it isO​\(10−6\)O\(10^\{\-6\}\), the floating\-point footprint of using finiteτ∈\{10−4,106\}\\tau\\in\\\{10^\{\-4\},10^\{6\}\\\}andθ=−109\\theta=\-10^\{9\}as numerical stand\-ins for the limitsτ→0\\tau\\to 0,τ→∞\\tau\\to\\infty,θ→−∞\\theta\\to\-\\infty\.

Throughout, the batch and head dimensions are dropped for readability; the implementation vectorises over them\.𝐀∈ℝK×d\\mathbf\{A\}\\in\\mathbb\{R\}^\{K\\times d\}are the anchor \(kept\) tokens,𝐃∈ℝM×d\\mathbf\{D\}\\in\\mathbb\{R\}^\{M\\times d\}the discards \(M=Nl−KlM=N\_\{l\}\-K\_\{l\}\),𝐬∈ℝM\\mathbf\{s\}\\in\\mathbb\{R\}^\{M\}their text\-to\-visual attention scores, andβ=10\\beta=10the per\-token\-gate sharpness\.

1importtorch

2importtorch\.nn\.functionalasF

3

4defunified\_operator\(A,D,s,gamma,tau,theta,rho,nu,beta=10\.0\):

5

6S=F\.normalize\(D,dim=\-1\)@F\.normalize\(A,dim=\-1\)\.t\(\)

7W=F\.softmax\(S/tau,dim=\-1\)

8m=torch\.sigmoid\(beta\*\(S\.max\(dim=\-1\)\.values\-theta\)\)

9out=A\+gamma\*\(W\.t\(\)@\(D\*m\.unsqueeze\(\-1\)\)\)

10p=\(F\.softmax\(S,dim=\-1\)\*F\.softmax\(s,dim=\-1\)\.unsqueeze\(\-1\)\)\.sum\(0\)

11out=out\*\(1\.0\+rho\*p\.unsqueeze\(\-1\)\)

12keep=A\.norm\(dim=\-1,keepdim=True\)/out\.norm\(dim=\-1,keepdim=True\)\.clamp\(min=1e\-6\)

13return\(1\.0\-nu\)\*out\+nu\*\(out\*keep\)

Listing 1The unified reduction operator— the exact body executed at each reducer layerl∈ℛl\\in\\mathcal\{R\}\(cf\. Eqs\. \([2](https://arxiv.org/html/2606.05232#S2.E2)\)–\([6](https://arxiv.org/html/2606.05232#S2.E6)\)\)\. Withγ=0\\gamma=0it is the identity on𝐀\\mathbf\{A\}; the per\-token gatemmand the norm\-preserve blendν\\nuare no\-ops at all four canonical corners \(θ→−∞⇒m≡1\\theta\\to\-\\infty\\Rightarrow m\\equiv 1,ν=0\\nu=0\)\.1defprune\(A,D,s\):

2returnA\.clone\(\)

3

4defmerge\(A,D,s\):

5S=F\.normalize\(D,dim=\-1\)@F\.normalize\(A,dim=\-1\)\.t\(\)

6nn=S\.argmax\(dim=\-1\)

7out=A\.clone\(\)

8forjinrange\(D\.shape\[0\]\):

9out\[nn\[j\]\]\+=D\[j\]

10returnout

11

12defpool\(A,D,s\):

13returnA\+D\.sum\(dim=0,keepdim=True\)/A\.shape\[0\]

14

15defreweight\(A,D,s,alpha\):

16S=F\.normalize\(D,dim=\-1\)@F\.normalize\(A,dim=\-1\)\.t\(\)

17p=\(F\.softmax\(S,dim=\-1\)\*F\.softmax\(s,dim=\-1\)\.unsqueeze\(\-1\)\)\.sum\(0\)

18returnA\*\(1\.0\+alpha\*p\.unsqueeze\(\-1\)\)

Listing 2The four canonical token\-reduction operatorsas standalone algorithms\. None callsunified\_operatoror its softmax\-over\-similarity machinery:Prune\(FastV\[[2](https://arxiv.org/html/2606.05232#bib.bib2)\], SparseVLM\[[32](https://arxiv.org/html/2606.05232#bib.bib32)\]\) keeps the top\-KKand drops the rest;Merge\(ToMe\[[1](https://arxiv.org/html/2606.05232#bib.bib1)\]\) routes each discard to its single nearest anchor viaargmax;Pooladds the uniform mean of the discards to every anchor;Reweightonly rescales anchor norms\.1torch\.manual\_seed\(42\)

2K,M,d,alpha,NEG=8,16,64,1\.5,\-1e9

3A,D,s=torch\.randn\(K,d\),torch\.randn\(M,d\),torch\.randn\(M\)

4

5

6configs=\{

7"PRUNE":\(0\.0,1\.0,NEG,0\.0,0\.0\),

8"MERGE":\(1\.0,1e\-4,NEG,0\.0,0\.0\),

9"POOL":\(1\.0,1e6,NEG,0\.0,0\.0\),

10"REWEIGHT":\(0\.0,1\.0,NEG,alpha,0\.0\),

11\}

12refs=\{"PRUNE":prune\(A,D,s\),"MERGE":merge\(A,D,s\),

13"POOL":pool\(A,D,s\),"REWEIGHT":reweight\(A,D,s,alpha\)\}

14

15forname,\(g,t,th,r,n\)inconfigs\.items\(\):

16u=unified\_operator\(A,D,s,gamma=g,tau=t,theta=th,rho=r,nu=n\)

17print\(f"\{name:9s\}max\|unified\-reference\|=\{\(u\-refs\[name\]\)\.abs\(\)\.max\(\)\.item\(\):\.2e\}"\)

18

19

20

21

22

Listing 3The equivalence check\.Each canonical operator equalsunified\_operatorat one corner of\(γ,τ,θ,ρ,ν\)\(\\gamma,\\tau,\\theta,\\rho,\\nu\)\.NEG=−109=\-10^\{9\}realisesθ→−∞\\theta\\to\-\\inftysom≡1m\\equiv 1\.ρ\\rhohere is the reweight coefficientα\\alpha\. The printed output is shown beneath the code\.##### Why each corner collapses the formula\.

Prune\(γ=0\)\(\\gamma\{=\}0\): the information\-transfer termγ​\(W⊤​\(D⊙m\)\)\\gamma\\,\(W^\{\\\!\\top\}\(D\\odot m\)\)vanishes and the reweight / norm\-preserve steps are off\(ρ=ν=0\)\(\\rho\{=\}\\nu\{=\}0\), sounified\_operatorreturnsAunchanged — exactly the kept top\-KKtokens, with theMMdiscards absent from the output, identical to physically deleting them\.Merge\(γ=1,τ→0\+\)\(\\gamma\{=\}1,\\,\\tau\{\\to\}0^\{\+\}\): each row ofsoftmax​\(S/τ\)\\mathrm\{softmax\}\(S/\\tau\)concentrates all mass onarg⁡maxk⁡Sj​k\\arg\\max\_\{k\}S\_\{jk\}, soWWbecomes the one\-hot nearest\-anchor matrix andW⊤​DW^\{\\\!\\top\}Dscatter\-adds each discard onto its nearest anchor — ToMe’s hard merge, obtained as a temperature limit rather than anargmax\.Pool\(γ=1,τ→∞\)\(\\gamma\{=\}1,\\,\\tau\{\\to\}\\infty\): cosine similarities are bounded, soS/τ→0S/\\tau\\to 0andsoftmax​\(S/τ\)→𝟏/K\\mathrm\{softmax\}\(S/\\tau\)\\to\\mathbf\{1\}/K; every anchor then receives1K​∑jDj\\tfrac\{1\}\{K\}\\sum\_\{j\}D\_\{j\}— a uniform mean over the*token set*\(not a spatial grid: post\-ViT visual tokens carry no 2\-D adjacency\)\.Reweight\(γ=0,ρ\>0\)\(\\gamma\{=\}0,\\,\\rho\{\>\}0\): no content moves; the only effect isout=A⋅\(1\+ρ​p\)\\texttt\{out\}=A\\cdot\(1\+\\rho\\,p\), wherepi=∑jsoftmax​\(Sj\)i​softmax​\(s\)jp\_\{i\}=\\sum\_\{j\}\\mathrm\{softmax\}\(S\_\{j\}\)\_\{i\}\\,\\mathrm\{softmax\}\(s\)\_\{j\}is the proximity\-weighted importance of the discards that anchoriibest represents — anchors that “stand in for” many high\-attention discards have their norm boosted, with direction untouched\. Off these corners — e\.g\. for the gradient\-discovered values\(γ6=0\.08,τ6=0\.22\)\(\\gamma\_\{6\}\{=\}0\.08,\\,\\tau\_\{6\}\{=\}0\.22\)atL6L\_\{6\}— the formula is a smooth interpolation among the four, augmented by the two extra degrees of freedomθ\\theta\(which discards to merge vs\. prune, per token\) andν\\nu\(how much of the original token norm to retain\) that no single canonical operator exposes\.