面向多模态智能体的持续工具空间扩展学习

arXiv cs.AI 论文

摘要

该论文提出了 COTEX,一种持续工具空间扩展学习方法,通过施加正交约束的低秩适配器,为已训练的多模态智能体添加新的专用工具,在保留现有工具能力的同时,在九项视觉语言基准测试中取得性能提升。

arXiv:2611.23810v1 Announce Type: new Abstract: Multimodal agents largely rely on generic tools such as cropping, search, and code execution, leaving specialist vision models outside their tool space. However, bringing such a model into an already trained agent is not straightforward. The challenge is to learn each new tool efficiently while retaining the ability to use existing ones. This work proposes COTEX, continual tool-space extension learning, that adds new tools at efficient adapter-level cost. COTEX locates two sets of parameter directions; one triggers a tool call and thereby opens a multi-turn reasoning trajectory, while the other determines which tool is chosen. The adapter of a new tool is restricted to the calling directions that leave earlier tool choices unaffected by orthogonal constraint. We further introduce a protocol that converts specialist models' existing task-specific annotations into verified tool-use trajectories. Experiments on two representative agents and three specialist tools show gains on ten vision-language benchmarks, with at most a 0.3-point drop on existing tools.
查看原文
查看缓存全文

缓存时间: 2026/11/20 09:09

# Continual Tool-Space Extension Learning for Multimodal Agents
Source: [https://arxiv.org/html/2611.23810](https://arxiv.org/html/2611.23810)
Jinbae Seo Jungin Park Hyunsung Jang Namkoo Ha Jiyoung Lee Kwanghoon SohnAffiliation:LIG Defense & Aerospace School of AI and Software, Ewha Womans UniversityAffiliation:Korea Institute of Science and Technology \(KIST\)

###### Abstract

Multimodal agents largely rely on generic tools such as cropping, search, and code execution, leaving specialist vision models outside their tool space\. However, bringing such a model into an already trained agent is not straightforward\. The challenge is to learn each new tool efficiently while retaining the ability to use existing ones\. This work proposes COTEX, continual tool\-space extension learning, that adds new tools at efficient adapter\-level cost\. COTEX locates two sets of parameter directions; One triggers a tool call and thereby opens a multi\-turn reasoning trajectory, while the other determines which tool is chosen\. The adapter of a new tool is restricted to the calling directions that leave earlier tool choices unaffected by orthogonal constraint\. We further introduce a protocol that converts specialist models’ existing task\-specific annotations into verified tool\-use trajectories\. Experiments on two representative agents and three specialist tools show gains on nine vision\-language benchmarks, with at most a 0\.3\-point drop on existing tools\.

![[Uncaptioned image]](https://arxiv.org/html/2611.23810v1/fig1.png)

Figure 1:Intuition of COTEX: \(a\) full re\-training preserves previously acquired tools but incurs high training cost, \(b\) naive parameter\-efficient finetuning \(PEFT\)\[[10](https://arxiv.org/html/2611.23810#bib.bib10)\]is efficient but overwrites previously acquired tools, \(c\) COTEX, that weaves the advantages of both full re\-training and PEFT techniques by orthogonally learning low\-rank adapters for each novel tool\.
00footnotetext:†Corresponding authors## 1Introduction

Multimodal large language models \(MLLMs\) have rapidly evolved from passive visual understanding systems into agentic models capable of interacting with external environments\. By integrating systematic operations into their reasoning process \(*e\.g*\., code execution, search, and image cropping\), recent multimodal agents such as DeepEyes\[[34](https://arxiv.org/html/2611.23810#bib.bib34),[9](https://arxiv.org/html/2611.23810#bib.bib9)\]and Metis\[[30](https://arxiv.org/html/2611.23810#bib.bib30)\]can actively acquire missing evidence beyond parametric inference alone\. Existing works have primarily focused on teaching agents how to invoke and utilize external tools\.

In practice, however, the tool set is rarely fixed\. Many vision tasks are best solved by specialist models, and stronger specialists continue to appear\. As illustrated in[Fig\.1](https://arxiv.org/html/2611.23810#S0.F1), an agent limited to cropping, search, and code often misreads scene text, miscounts objects, and misjudges relative depth\. Incorporating such tools \(*e\.g*\., OCR model, a detector, or a depth estimator\) after the agent has already been trained, which we refer to ascontinual tool\-space extension learning, is therefore a natural requirement for a scalable agent\.

Integrating newly available tools into an already trained agent introduces a new challenge\. Existing options sit at two ends of a trade\-off between cost and retention\. A straightforward approach is agent full training, which re\-runs SFT and RL over the expanded tool space, on newly induced trajectories with a novel tool\. Its cost, however, grows with every tool that is added\. Parameter\-efficient fine\-tuning \(PEFT\)\[[10](https://arxiv.org/html/2611.23810#bib.bib10)\]avoids this cost, yet an adapter trained only on the new tool overwrites the directions the agent relies on for existing tools \(*i\.e*\., catastrophic forgetting problem\)\[[11](https://arxiv.org/html/2611.23810#bib.bib11),[18](https://arxiv.org/html/2611.23810#bib.bib18)\]\. Consequently, a practical agent requires both efficient updates and intact existing tools\.

In this paper, we propose aCOntinualTool\-spaceEXtension learning \(COTEX\)\. COTEX is built on the premise that tool use involves two distinct decisions\. Initiating a tool call reflects the judgment that the evidence at hand is insufficient to answer, which is common to every tool\. Selecting a tool determines what evidence is needed next, which is specific to each tool\. A new tool should therefore reuse the former and leave the latter untouched\. COTEX applies SVD to the weight displacement between the agent and its base MLLM, then ranks the resulting directions by their gradient sensitivity on a small calibration set\. Directions sensitive to call initiation form the shared tool\-call subspace\. Within it, directions sensitive to the selection of existing tools are marked as protected\. Each new tool is learned as a low\-rank adapter that lies inside the shared subspace while having orthogonal directions to the protected directions\. The adapter thus inherits the judgment of when evidence is insufficient, while the choice among existing tools is left intact\.

Learning the adapter still requires tool\-use trajectories, which are rarely available for a newly released model\. We therefore introduce a general protocol that repurposes existing task\-specific supervision into such trajectories\. Each tool corresponds to a well\-defined task for which annotated data already exist, such as bounding boxes for a detector or depth maps for a depth estimator\. These annotations specify what a correct tool output should be, but they do not describe how an agent should invoke the tool and use its output to answer a question\. Our protocol bridges this gap with self\-generated trajectories\. The agent is prompted to invoke the tool on examples from its native dataset, and a task\-specific verifier retains only the trajectories whose final predictions agree with the original annotations\. Through training on these trajectories, the agent learns to decide whether to use the tool and how to invoke it on its own\.

Extensive experiments demonstrate that COTEX effectively grows two agents with three tools, including object detector\[[16](https://arxiv.org/html/2611.23810#bib.bib16)\], OCR\[[5](https://arxiv.org/html/2611.23810#bib.bib5)\], and depth estimator\[[31](https://arxiv.org/html/2611.23810#bib.bib31)\]\. With only about 1K self\-generated demonstrations per tool, COTEX improves performance across benchmarks requiring OCR, object detection, and depth estimation, while largely preserving the original agent’s visual perception and reasoning capabilities\.

In summary, this work provides key contributions:

- •We formulate continual tool\-space extension, where a multimodal agent acquires new specialist tools while retaining existing ones\.
- •We propose COTEX, which identifies the protected tool\-sensitive parameters and learns each new tool orthogonally to it within the shared tool\-call subspace\.
- •We introduce a training protocol that converts native task annotations into verified tool\-use trajectories, and demonstrate across 10 benchmarks that COTEX acquires specialized capabilities through parameter\-efficient updates while preserving general visual perception performance\.

## 2Related Work

![Refer to caption](https://arxiv.org/html/2611.23810v1/fig2.png)Figure 2:The overall training pipeline of COTEX\.### 2\.1Multimodal Agents

In general, multimodal agents\[[7](https://arxiv.org/html/2611.23810#bib.bib7),[34](https://arxiv.org/html/2611.23810#bib.bib34)\]have built on MLLMs by explicitly executing image processing, computation, and information retrieval programs into the reasoning process\. Early models\[[7](https://arxiv.org/html/2611.23810#bib.bib7),[23](https://arxiv.org/html/2611.23810#bib.bib23)\]have generated executable program code in scripts, while GPT4Tools\[[32](https://arxiv.org/html/2611.23810#bib.bib32)\]and LLaVA\-Plus\[[15](https://arxiv.org/html/2611.23810#bib.bib15)\]have learned to invoke predefined tools through instruction tuning within their reasoning chain\. Recently, reinforcement learning \(RL\)\-based optimization\[[21](https://arxiv.org/html/2611.23810#bib.bib21)\]has improved the scalability of tool\-use in agents\[[34](https://arxiv.org/html/2611.23810#bib.bib34),[9](https://arxiv.org/html/2611.23810#bib.bib9),[30](https://arxiv.org/html/2611.23810#bib.bib30)\]\. For instance, DeepEyesV2\[[9](https://arxiv.org/html/2611.23810#bib.bib9)\]integrates visual operations, code execution, and external search into a unified reasoning policy\. Meanwhile, Metis\[[30](https://arxiv.org/html/2611.23810#bib.bib30)\]addresses tool overuse by decoupling task correctness from interaction efficiency and discouraging redundant tool calls among successful trajectories\. These approaches substantially advance the acquisition and utilization of external tools, but mainly focus on improving tool use within a given interaction setting\. Our work instead considers how the available tool space itself can be expanded after post\-training while preserving previously acquired tool\-use behavior\.

### 2\.2Tool\-Space Extension

Tool invocation itself is learned quickly, yet sequential fine\-tuning on a new tool erases previously learned ones almost entirely\[[11](https://arxiv.org/html/2611.23810#bib.bib11)\]\. GeoPilot\[[35](https://arxiv.org/html/2611.23810#bib.bib35)\]repurposes existing task annotations to construct tool\-use supervision, but trains with a predefined set of tools\. Even though COLT\[[18](https://arxiv.org/html/2611.23810#bib.bib18)\]introduces a learnable tool codebook for single\-turn API selection of Video\-LLMs, it still finetunes the full model for each new tool, without controlling interference in weight space\. Instead, we target agentic visual reasoning, in which the forgetting problem is mitigated by restricting new tool updates to directions within the shared tool\-call subspace that are orthogonal to previously learned tool\-selection directions\.

### 2\.3Subspace Isolation and Adaptation

Constraining parameter updates to a subspace adds new capabilities while limiting interference with existing knowledge\[[10](https://arxiv.org/html/2611.23810#bib.bib10),[3](https://arxiv.org/html/2611.23810#bib.bib3)\]\. Recently, Iso\-CTS\[[20](https://arxiv.org/html/2611.23810#bib.bib20)\]identifies common and task\-specific subspaces from weight differences for model merging, while DiFT\[[33](https://arxiv.org/html/2611.23810#bib.bib33)\]identifies capability\-sensitive parameters from activation changes for selective fine\-tuning\. Meanwhile, O\-LoRA\[[27](https://arxiv.org/html/2611.23810#bib.bib27)\]and KeepLoRA\[[19](https://arxiv.org/html/2611.23810#bib.bib19)\]have constrained low\-rank updates to be orthogonal to previously acquired knowledge, while LoDA\[[8](https://arxiv.org/html/2611.23810#bib.bib8)\]has separated shared and task\-specific subspaces to balance knowledge transfer and isolation\. Unlike prior orthogonal adaptation methods\[[27](https://arxiv.org/html/2611.23810#bib.bib27),[19](https://arxiv.org/html/2611.23810#bib.bib19)\], which steer new updates away from the dominant directions of the model, COTEX places them inside the dominant tool\-call directions and excludes only what governs existing tool selection\.

## 3Method

### 3\.1Problem Statement and Overview

Letπθ\\pi\_\{\\theta\}denote a multimodal agent obtained by post\-training a base MLLM with parametersθ⋆\\theta^\{\\star\}on a tool set𝒯old\\mathcal\{T\}\_\{\\mathrm\{old\}\}\(*e\.g*\., cropping, Python execution, and search\)\. New tools\{g1,…,gK\}\\\{g\_\{1\},\\dots,g\_\{K\}\\\}arrive sequentially, where eachgkg\_\{k\}is a frozen specialist model accessed through an executable interface\. Whengkg\_\{k\}arrives, the goal is to enableπθ\\pi\_\{\\theta\}to invoke it while retaining the use of𝒯old\\mathcal\{T\}\_\{\\mathrm\{old\}\}and of the previously added tools\{g1,…,gk−1\}\\\{g\_\{1\},\\dots,g\_\{k\-1\}\\\}\. For efficient training, each tool is acquired through a dedicated low\-rank update\.

As illustrated in[Fig\.2](https://arxiv.org/html/2611.23810#S2.F2), COTEX consists of three components\. First, existing supervision for the task performed bygkg\_\{k\}is converted into a set of verified tool\-use trajectories \([Sec\.3\.2](https://arxiv.org/html/2611.23810#S3.SS2)\)\. Second, we locate the directions that drive call initiation, which might be updateable, and those that determine tool selection, which might be protected \([Sec\.3\.3](https://arxiv.org/html/2611.23810#S3.SS3)\)\. Third, new tools are learned via a low\-rank adapter inside the shared tool\-call subspace while remaining orthogonal to protected tool\-selection directions \([Sec\.3\.4](https://arxiv.org/html/2611.23810#S3.SS4)\)\.

### 3\.2Task\-Specific Models as Expandable Tools

A robust modelgkg\_\{k\}tailored for a specific computer vision task \(*e\.g*\., object detection, OCR, and monocular depth estimation\) could serve as a novel tool to employ an external capability on a multimodal agent\. Integrating a new toolgkg\_\{k\}requires trajectories that show when to call it, how to form its arguments, and how to use its observation\. Prior work synthesizes such trajectories with strong teacher model \(*e\.g*\., GPT\[[9](https://arxiv.org/html/2611.23810#bib.bib9)\]or Gemini\[[25](https://arxiv.org/html/2611.23810#bib.bib25)\]\), which is costly and ties data quality to the teacher\. We observe that the annotated dataset ofgkg\_\{k\}already supplies the two ingredients a teacher would otherwise provide\. Its inputs are precisely the cases in whichgkg\_\{k\}is useful\. Its ground\-truths tell whether a final answer is correct\. With both in hand, trajectories need not be written by a teacher\. The agent attempts each example with access togkg\_\{k\}, and only the attempts whose final answer matches the ground\-truths are kept\.

Concretely, for an image inputxix\_\{i\}, the agent emits a callcic\_\{i\}, receives the observationoio\_\{i\}from executinggkg\_\{k\}, and produces a predictiony^i\\hat\{y\}\_\{i\}\. A task\-specific verifier𝒱k\\mathcal\{V\}\_\{k\}, implemented with rule\-based criteria or an external judge, compares a predictiony^i\\hat\{y\}\_\{i\}withyiy\_\{i\}\. Only trajectoriesτi\\tau\_\{i\}verified as correct are retained in a task dataset,

𝒟k=\{\(xi,τi\)\|𝒱k​\(y^i,yi\)=1\},\\mathcal\{D\}\_\{k\}=\\left\\\{\(x\_\{i\},\\tau\_\{i\}\)\\,\\middle\|\\,\\mathcal\{V\}\_\{k\}\(\\hat\{y\}\_\{i\},y\_\{i\}\)=1\\right\\\},\(1\)where𝒱k\\mathcal\{V\}\_\{k\}returns a binary verification result\. The instruction templates and verifier configurations for each task are provided in Appendix A\.3\. This construction allows existing task\-specific supervision to serve directly as training data for tool integration\.

### 3\.3Low\-Rank Structure of Tool\-Call Sensitivity

Introducing new tools requires adapting an existing agent without disrupting its previously acquired capabilities\. However, unconstrained parameter updates may disable tool\-use, reasoning, and perception capabilities\. A natural remedy is to identify a shared subspace associated with tool\-call behavior and protect the directions within it that are responsible for selecting existing tools\. We therefore investigate whether tool\-call behavior is associated with a compact subspace of the parameter changes introduced during agent training\.

For each adapted linear projectionWℓW^\{\\ell\}in the LLM, excluding vision\-specific modules and the language head, we compute singular value decomposition \(SVD\) for weight displacement,

Δ​Wℓ=Wθℓ−Wθ⋆ℓ=UΔℓ​Σℓ​Vℓ⊤,\\Delta W^\{\\ell\}=W^\{\\ell\}\_\{\\theta\}\-W^\{\\ell\}\_\{\\theta^\{\\star\}\}=U\_\{\\Delta\}^\{\\ell\}\\Sigma^\{\\ell\}\{V^\{\\ell\}\}^\{\\top\},\(2\)whereUΔℓ∈ℝdout×dΔU\_\{\\Delta\}^\{\\ell\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times d\_\{\\Delta\}\}contains the left singular vectors of the post\-training displacement, withdΔ=min⁡\(dout,din\)d\_\{\\Delta\}=\\min\(d\_\{\\mathrm\{out\}\},d\_\{\\mathrm\{in\}\}\)\. These directions reflect every change introduced by post\-training, including general reasoning, so not all of them concern tool use\. The sensitivity of tool\-call behavior to each direction is therefore measured separately\.

For a calibration trajectoryii, letℐi\\mathcal\{I\}\_\{i\}denote the set of target token positions at which the loss is evaluated\. The calibration lossℒi\\mathcal\{L\}\_\{i\}is the mean negative log\-likelihood of the target tokens inℐi\\mathcal\{I\}\_\{i\}underπθ\\pi\_\{\\theta\}, conditioned on the preceding interaction history:

ℒi=−1\|ℐi\|∑t∈ℐilogπθ\(yt∣h<t\),\\mathcal\{L\}\_\{i\}=\-\\frac\{1\}\{\|\\mathcal\{I\}\_\{i\}\|\}\\sum\_\{t\\in\\mathcal\{I\}\_\{i\}\}\\log\\pi\_\{\\theta\}\(y\_\{t\}\\mid h\_\{<t\}\),\(3\)whereyty\_\{t\}is the token at positionttandh<th\_\{<t\}is the preceding interaction history, including tool observations\. A trajectory often contains more than one tool call\. For instance, an agent may crop a region, inspect it, and then call a numeric calculator\. Therefore, the trajectory has two positions where a call is initiated and two positions where a tool is named\. In other words,ℒcall\\mathcal\{L\}\_\{\\mathrm\{call\}\}is evaluated at positions that initiate a tool call \(*e\.g*\.,<code\>,<tool\_call\>\), capturing when the agent decides to invoke a tool\.ℒtool\\mathcal\{L\}\_\{\\mathrm\{tool\}\}is evaluated at positions that specify the selected tool \(*e\.g*\.,crop,search,image\_search\), capturing which tool the agent chooses\.

Gradient\-based tool sensitivity within an orthonormal basisUUis defined as

C⁡\(U,ℒ\)=1N​∑i=1N\(U⊤​∂ℒi∂Wℓ\)​\(U⊤​∂ℒi∂Wℓ\)⊤,C\(U,\\mathcal\{L\}\)=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\left\(U^\{\\top\}\\frac\{\\partial\\mathcal\{L\}\_\{i\}\}\{\\partial W^\{\\ell\}\}\\right\)\\left\(U^\{\\top\}\\frac\{\\partial\\mathcal\{L\}\_\{i\}\}\{\\partial W^\{\\ell\}\}\\right\)^\{\\top\},\(4\)whereNNis the number of calibration trajectories\.[Fig\.3](https://arxiv.org/html/2611.23810#S3.F3)shows the cumulative gradient energy captured by the leading eigenvectors ofC⁡\(UΔℓ,ℒcall\)C\(U\_\{\\Delta\}^\{\\ell\},\\mathcal\{L\}\_\{\\mathrm\{call\}\}\), averaged across layers\. The result supports our hypothesis that tool\-call sensitivity is concentrated in a low\-rank subspace across different agents\[[9](https://arxiv.org/html/2611.23810#bib.bib9),[30](https://arxiv.org/html/2611.23810#bib.bib30)\]\. We next explain how to learn new tools within this shared subspace while remaining orthogonal to directions associated with existing tool selection\.

Figure 3:Low\-rank retention rate of tool\-call gradient energy\. Gradients of the tool\-call loss are projected onto the top\-rrtool\-sensitive directions identified byC⁡\(UΔℓ,ℒcall\)C\(U\_\{\\Delta\}^\{\\ell\},\\mathcal\{L\}\_\{\\mathrm\{call\}\}\)within the left singular\-vector subspace ofΔ​Wℓ\\Delta W^\{\\ell\}, and the retained energy ratioEr/EtotalE\_\{r\}/E\_\{\\mathrm\{total\}\}is averaged over layers\. Atr=64r\{=\}64, under 2% of the full rank, the projection retains 88% of the energy for DeepEyesV2 and 97% for Metis\.
### 3\.4Orthogonal Tool Space Update

To prevent interference between new tool usability and old\-tool application capability, we constrain new tool updates to their remaining orthogonal subspace\[[27](https://arxiv.org/html/2611.23810#bib.bib27),[19](https://arxiv.org/html/2611.23810#bib.bib19)\]\. Specifically, COTEX’s design departs from subspace\-constrained adaptation such as KeepLoRA\[[19](https://arxiv.org/html/2611.23810#bib.bib19)\]in two respects\. KeepLoRA\[[19](https://arxiv.org/html/2611.23810#bib.bib19)\]decomposes the pre\-trained weights and writes new knowledge into the residual subspace, away from their principal components\. COTEX decomposes the post\-training displacement rather than the weights themselves, ranks directions by their gradient sensitivity to tool\-call behavior rather than by singular value magnitude, and places new tools inside the shared tool\-call subspace\. Within this subspace, directions associated with existing tool selection are excluded from new\-tool adaptation\. For notational simplicity, we omit the layer indexℓ\\ellin the following\.

Shared tool\-call basis\.Specifically, given a symmetric matrixCCby[Eq\.4](https://arxiv.org/html/2611.23810#S3.E4), letQ\[j\]​\(C\)Q\_\{\[j\]\}\(C\)denote the eigenvectors associated with itsjjlargest eigenvalues\. We definedcalld\_\{\\mathrm\{call\}\}as the dimensionality of the tool\-call\-sensitive subspace identified byC⁡\(UΔ,ℒcall\)C\(U\_\{\\Delta\},\\mathcal\{L\}\_\{\\mathrm\{call\}\}\)\. We construct an orthonormal basis for the shared tool\-call directions as

Ucall=UΔ​Q\[dcall\]​\(C⁡\(UΔ,ℒcall\)\)\.U\_\{\\mathrm\{call\}\}=U\_\{\\Delta\}Q\_\{\[d\_\{\\mathrm\{call\}\}\]\}\\left\(C\(U\_\{\\Delta\},\\mathcal\{L\}\_\{\\mathrm\{call\}\}\)\\right\)\.\(5\)The columns ofUcallU\_\{\\mathrm\{call\}\}form an orthonormal basis within the dominant displacement subspace\.

Protected tool\-selection basis\.Within this shared subspaceUcallU\_\{\\mathrm\{call\}\}, we further identify directions sensitive to selecting previously learned tools,

Uold=Ucall​Q\[r\]​\(C⁡\(Ucall,ℒtool\)\),U\_\{\\mathrm\{old\}\}=U\_\{\\mathrm\{call\}\}Q\_\{\[r\]\}\\left\(C\(U\_\{\\mathrm\{call\}\},\\mathcal\{L\}\_\{\\mathrm\{tool\}\}\)\\right\),\(6\)This hierarchical construction selects the protected directions within the shared tool\-call subspace, leaving the remaining directions available for new\-tool adaptation\.

Constrained new\-tool adaptation\.For a newly introduced toolgkg\_\{k\}, letU<kU\_\{<k\}be the accumulated orthonormal basis of protected tool\-selection directions, initialized asUoldU\_\{\\mathrm\{old\}\}\. Since these directions lie within the shared tool\-call subspace, the projector onto the available adaptation space is

Pk=Ucall​Ucall⊤−U<k​U<k⊤\.P\_\{k\}=U\_\{\\mathrm\{call\}\}U\_\{\\mathrm\{call\}\}^\{\\top\}\-U\_\{<k\}U\_\{<k\}^\{\\top\}\.\(7\)LetR<kR\_\{<k\}be an orthonormal basis for the range ofPkP\_\{k\}, satisfyingR<k​R<k⊤=PkR\_\{<k\}R\_\{<k\}^\{\\top\}=P\_\{k\}\. We employ a new tool’s low\-rank adapter\[[10](https://arxiv.org/html/2611.23810#bib.bib10)\]as

Δ​Wk=sk​R<k​Bk​Ak,\\Delta W\_\{k\}=s\_\{k\}R\_\{<k\}B\_\{k\}A\_\{k\},\(8\)wheresks\_\{k\}is the scaling factor, and the fixed basisR<kR\_\{<k\}confines every update to the shared tool\-call subspace and excludes the accumulated protected directions\.Δ​Wk\\Delta W\_\{k\}has to be orthogonal to the accumulated protected tool\-selection directions, satisfyingU<k⊤​Δ​Wk=0U\_\{<k\}^\{\\top\}\\Delta W\_\{k\}=0\. OnlyBkB\_\{k\}andAkA\_\{k\}are optimized with the verified tool\-use trajectories𝒟k\\mathcal\{D\}\_\{k\}described in[Sec\.3\.2](https://arxiv.org/html/2611.23810#S3.SS2)\. Note that tool observations are included in the interaction history but excluded from the prediction targets\.

### 3\.5Training for Tool Utilization

SFT\.For each newly introduced toolgkg\_\{k\}, the agent learns on the verified tool\-use trajectories𝒟k\\mathcal\{D\}\_\{k\}by SFT\. We minimize the negative log\-likelihood of the assistant\-generated tokens, including reasoning, tool invocations, and final answers, conditioned on the preceding interaction history\. Tool observations are included in the history but excluded from the prediction targets\. During SFT, we freeze previously learned adapters, and the agent backbone\. The resulting updates are therefore confined to the available tool\-call space and remain orthogonal to the accumulated protected tool\-selection directions\.

RL\.After SFT establishes basic tool\-use capabilities, we further optimize the agent through reinforcement learning to improve when and how to utilize newly introduced tools\. Following Metis\[[30](https://arxiv.org/html/2611.23810#bib.bib30)\], we adopt Hierarchical Decoupled Policy Optimization \(HDPO\), which separately computes advantages for task outcomes and tool\-use efficiency\. The outcome reward combines task accuracy and format compliance with weightswaccw\_\{\\mathrm\{acc\}\}andwfmtw\_\{\\mathrm\{fmt\}\}, while the efficiency reward favors fewer tool interactions among correct rollouts\. The resulting advantages are optimized through

ℒRL=ℒGRPO​\(Advout\)\+weff​ℒGRPO​\(Adveff\),\\mathcal\{L\}\_\{\\mathrm\{RL\}\}=\\mathcal\{L\}\_\{\\mathrm\{GRPO\}\}\(\\textit\{Adv\}\_\{\\mathrm\{out\}\}\)\+w\_\{\\mathrm\{eff\}\}\\mathcal\{L\}\_\{\\mathrm\{GRPO\}\}\(\\textit\{Adv\}\_\{\\mathrm\{eff\}\}\),\(9\)whereAdvout\\textit\{Adv\}\_\{\\mathrm\{out\}\}andAdveff\\textit\{Adv\}\_\{\\mathrm\{eff\}\}denote the independently computed outcome and efficiency advantages, respectively, andweffw\_\{\\mathrm\{eff\}\}controls the contribution of tool\-use efficiency\. During optimization, we update only the task\-specific LoRA adapter while keeping the remaining adapters and backbone frozen\.

Continual protection\.After learningAkA\_\{k\}andBkB\_\{k\}, we evaluate its tool\-selection calibration lossℒtool,k\\mathcal\{L\}\_\{\\mathrm\{tool\},k\}on the adapted agent and extract the corresponding sensitive directions within the available adaptation space:

Uk=R<k​Q\[r\]​\(C⁡\(R<k,ℒtool,k\)\)\.U\_\{k\}=R\_\{<k\}Q\_\{\[r\]\}\\left\(C\(R\_\{<k\},\\mathcal\{L\}\_\{\\mathrm\{tool\},k\}\)\\right\)\.\(10\)We appendUkU\_\{k\}to the accumulated protected basisU<kU\_\{<k\}and use the updated basisU≤kU\_\{\\leq k\}to constrain adaptation for subsequent tools\. For the next tool, we setU<k\+1=U≤kU\_\{<k\+1\}=U\_\{\\leq k\}and recompute the available projector using[Eq\.7](https://arxiv.org/html/2611.23810#S3.E7)\. This procedure progressively reserves tool\-selection\-sensitive directions while allowing subsequent tools to adapt within the remaining shared tool\-call directions\.

## 4Experiments

Table 1:Performance comparisons on visual perception benchmarks, including VStar\[[28](https://arxiv.org/html/2611.23810#bib.bib28)\]and HRBench\[[26](https://arxiv.org/html/2611.23810#bib.bib26)\], and spatial reasoning benchmarks, including TreeBench\[[24](https://arxiv.org/html/2611.23810#bib.bib24)\], MultihopSpatial\[[13](https://arxiv.org/html/2611.23810#bib.bib13)\], and MC\-Bench\[[29](https://arxiv.org/html/2611.23810#bib.bib29)\]\. The best and second\-best scores in each column are shown in bold and underlined, respectively\.Table 2:Performance comparisons on text\-rich VL benchmarks, including SEED2Plus\[[14](https://arxiv.org/html/2611.23810#bib.bib14)\], TextVQA\[[22](https://arxiv.org/html/2611.23810#bib.bib22)\], OCRBench\[[17](https://arxiv.org/html/2611.23810#bib.bib17)\]and depth perception benchmarks, including BLINK\[[6](https://arxiv.org/html/2611.23810#bib.bib6)\], HardBLINK\[[2](https://arxiv.org/html/2611.23810#bib.bib2)\]\. ‘HB\(\#points\)’ denotes the number of comparison points in HardBLINK\.### 4\.1Experimental Setup

Agent and tool space\.We use DeepEyesV2\[[9](https://arxiv.org/html/2611.23810#bib.bib9)\], built on Qwen2\.5\-VL\-7B\[[1](https://arxiv.org/html/2611.23810#bib.bib1)\], as our primary post\-trained multimodal agent and report additional results on Metis\[[30](https://arxiv.org/html/2611.23810#bib.bib30)\]in[Sec\.4\.4](https://arxiv.org/html/2611.23810#S4.SS4)and Appendix\. We sequentially extend the agent with three frozen vision specialists accessed through executable interfaces: SVTRv2\[[5](https://arxiv.org/html/2611.23810#bib.bib5)\]for scene text recognition \(OCR\), GroundingDINO\[[16](https://arxiv.org/html/2611.23810#bib.bib16)\]for open\-set object detection, and DepthAnything\[[31](https://arxiv.org/html/2611.23810#bib.bib31)\]for monocular depth estimation\.

Training data\.Following[Sec\.3\.2](https://arxiv.org/html/2611.23810#S3.SS2), we generate multi\-turn trajectories from annotated datasets for the corresponding tasks and retain only rollouts verified by rule\-based checks\. Each verified set is randomly partitioned into disjoint SFT and RL splits: ICDAR 2015\[[12](https://arxiv.org/html/2611.23810#bib.bib12)\]yields 1,463 SFT and 166 RL examples for OCR, Ref\-Adv\-s\[[4](https://arxiv.org/html/2611.23810#bib.bib4)\]yields 739 SFT and 87 RL examples for object detection, and UniQA\-3D\[[36](https://arxiv.org/html/2611.23810#bib.bib36)\]yields 834 SFT and 77 RL examples for relative depth\. Every retained trajectory contains intermediate reasoning, tool invocation, returned tool observations, and a verified final answer\.

Calibration set\.For the initial shared tool\-call and protected tool\-selection bases \([Sec\.3\.4](https://arxiv.org/html/2611.23810#S3.SS4)\), we compute the decoupled sensitivities in[Eq\.4](https://arxiv.org/html/2611.23810#S3.E4)over publicly released tool\-use trajectories of the frozen initial agent\. During sequential extension, selection\-sensitive directions for each newly learned tool are estimated on its verified SFT trajectories\. All calibration samples are disjoint from benchmarks\.

Benchmarks\.We evaluate four capability groups in the main text: \(1\) high\-resolution visual perception on V\*Bench\[[28](https://arxiv.org/html/2611.23810#bib.bib28)\]and HRBench\[[26](https://arxiv.org/html/2611.23810#bib.bib26)\]to test retention of the native cropping tool, \(2\) text\-rich visual understanding on SEED\-Bench\-2\-Plus\[[14](https://arxiv.org/html/2611.23810#bib.bib14)\], TextVQA\[[22](https://arxiv.org/html/2611.23810#bib.bib22)\], and OCRBench\[[17](https://arxiv.org/html/2611.23810#bib.bib17)\], \(3\) spatial reasoning and visual grounding on TreeBench\[[24](https://arxiv.org/html/2611.23810#bib.bib24)\], MultihopSpatial\[[13](https://arxiv.org/html/2611.23810#bib.bib13)\], and MC\-Bench\[[29](https://arxiv.org/html/2611.23810#bib.bib29)\], and \(4\) relative\-depth perception on BLINK\[[6](https://arxiv.org/html/2611.23810#bib.bib6)\]and multi\-point HardBLINK\[[2](https://arxiv.org/html/2611.23810#bib.bib2)\]\. Retention on eight additional mathematical reasoning, visual search, and factuality VQA benchmarks is reported in the Appendix\. All evaluations follow each benchmark’s standard protocol and metric\.

Baselines\.We compare COTEX against training\-free In\-Context Prompting and three sequential LoRA adaptation methods\. Prompting prepends tool schemas and verified demonstrations to the input prompt while keeping all model weights frozen\. Naive SFT sequentially trains unconstrained per\-tool LoRA adapters on the verified trajectories\. O\-LoRA\[[27](https://arxiv.org/html/2611.23810#bib.bib27)\]penalizes subspace overlap between new and prior adapters, whereas KeepLoRA\[[19](https://arxiv.org/html/2611.23810#bib.bib19)\]projects new updates into the residual subspace orthogonal to pre\-trained weight principal components and the dominant feature directions of previous tasks\. All adaptation baselines share identical training trajectories, LoRA ranks, and optimization budgets\.

Figure 4:Distribution of tool calls on each benchmark before and after training with COTEX\.Implementation details\.We sequentially train tool\-specific LoRA adapters for OCR, object detection, and depth estimation, each with rankr=64r=64and scaling factorα=128\\alpha=128\. During SFT, each adapter is trained for one epoch with a constant learning rate of1×10−61\\times 10^\{\-6\}, a global batch size of 8, and a maximum sequence length of 4,096 tokens\. We then apply HDPO\[[30](https://arxiv.org/html/2611.23810#bib.bib30)\]with group sizeN=16N=16, learning rate1×10−61\\times 10^\{\-6\}, reward weightswacc=0\.9w\_\{\\mathrm\{acc\}\}=0\.9andwfmt=0\.1w\_\{\\mathrm\{fmt\}\}=0\.1, and efficiency weightweff=0\.15w\_\{\\mathrm\{eff\}\}=0\.15\. RL rollouts use temperature 1\.0, top\-pp1\.0, unrestricted top\-kk\(k=−1k=\-1\), up to 10 interaction turns, and a 300\-second tool timeout\.

### 4\.2Main Results

Visual perception and spatial reasoning\.[Tab\.1](https://arxiv.org/html/2611.23810#S4.T1)presents visual perception and spatial reasoning performance on six evaluation settings\. On benchmarks requiring the existing cropping tool \(*i\.e*\., VStar, HRbench\), COTEX maintains or slightly improves performance relative to the original agent\. These results demonstrate that COTEX effectively preserves the visual perception capabilities of the original agent\. Moreover, compared with DeepEyesV2, COTEX consistently improves performance across all three spatial reasoning benchmarks, which require object grounding\. COTEX especially achieves gains of 5\.0 and 12\.2 points on MultihopSpatial and MC\-Bench, respectively\. Interestingly, we empirically found that the substantial gain on MC\-Bench is driven by the detection tool resolving challenging samples that the original agent failed to solve\.

Text\-oriented understanding\.The left part of[Tab\.2](https://arxiv.org/html/2611.23810#S4.T2)reports performance on text\-oriented multimodal benchmarks\. Compared with DeepEyesV2, COTEX improves OCRBench accuracy by 2\.1 points, from 80\.4 to 82\.5, with increased OCR tool usage as shown in[Fig\.4](https://arxiv.org/html/2611.23810#S4.F4), while also improving TextVQA accuracy from 76\.6 to 76\.9 and maintaining comparable performance on SEED\-Bench\-2\-Plus\. These results indicate that COTEX retains previously acquired text\-understanding capabilities while benefiting from the newly introduced OCR tool\.

Relative\-depth perception\.[Tab\.2](https://arxiv.org/html/2611.23810#S4.T2)presents relative\-depth reasoning performance on BLINK and HardBLINK\. Compared with DeepEyesV2, COTEX substantially improves BLINK accuracy from 41\.1 to 87\.1\. More notably, COTEX achieves accuracies of 95\.2, 85\.5, and 88\.7 on the three\-, four\-, and five\-point HardBLINK benchmarks, respectively, outperforming DeepEyesV2 by 64\.5, 64\.5, and 69\.3 points\. These substantial improvements demonstrate the effectiveness of incorporating specialized depth estimation into multimodal reasoning, particularly for challenging multi\-point depth comparisons\.

### 4\.3Tool Call Distribution

[Fig\.4](https://arxiv.org/html/2611.23810#S4.F4)shows how COTEX changes tool selection across benchmarks\. On OCRBench, the agent adopts the OCR tool while reducing the proportion of calls to other Python operations\. The shift is particularly pronounced on BLINK and HardBLINK, where depth estimation accounts for nearly all tool calls after training, replacing the previous reliance on numerical analysis\. Meanwhile, crop remains the most frequently invoked tool on VStar and HRBench 8K, and mark continues to account for a substantial share of calls on MultihopSpatial\. These patterns suggest that COTEX incorporates new specialist tools selectively across benchmarks while continuing to use existing tools\.

![Refer to caption](https://arxiv.org/html/2611.23810v1/qualitative_ocr.png)\(a\)OCR
![Refer to caption](https://arxiv.org/html/2611.23810v1/qualitative_detection.png)\(b\)Object detection
![Refer to caption](https://arxiv.org/html/2611.23810v1/qualitative_depth.png)\(c\)Depth estimation

Figure 5:Qualitative examples of newly introduced tool capabilities\. Each example illustrates the agent’s reasoning, tool invocation, returned observation, and final prediction\. More examples are in Appendix\.
### 4\.4Ablation Studies

Table 3:Comparison of adaptation methods under sequential tool\-space extension\. Each stage cumulatively introduces OCR \(\+O\), detection \(\+G\), and depth \(\+D\)\. All values are accuracies \(%\)\.Adaptation methods and sequential extension\.[Tab\.3](https://arxiv.org/html/2611.23810#S4.T3)compares COTEX with Naive SFT, O\-LoRA\[[27](https://arxiv.org/html/2611.23810#bib.bib27)\], and KeepLoRA\[[19](https://arxiv.org/html/2611.23810#bib.bib19)\]under sequential tool\-space extension\. All methods use identical tool\-use trajectories, LoRA configurations, and optimization budgets\. Across all extension stages, COTEX achieves the highest V\*Bench performance, retaining 80\.6 after integrating all three tools, compared with 75\.9, 78\.0, and 74\.3 for Naive SFT, O\-LoRA, and KeepLoRA, respectively\. For newly acquired capabilities, COTEX achieves the highest OCR accuracy of 46\.4 after the first extension and the highest depth accuracy of 93\.5 after the final extension, surpassing KeepLoRA by 6\.5 points on depth estimation\. Although detection accuracy decreases after the depth extension, these results demonstrate the effectiveness of COTEX in preserving general visual perception while acquiring specialized capabilities through sequential adaptation\.

Table 4:Effect of before/after COTEX with different initial agents\. Performance on other benchmarks is in Appendix\.Different initial agents\.To investigate the generalizability of COTEX, we apply our framework to two post\-trained multimodal agents, DeepEyesV2\[[9](https://arxiv.org/html/2611.23810#bib.bib9)\]and Metis\[[30](https://arxiv.org/html/2611.23810#bib.bib30)\]\. As shown in[Tab\.4](https://arxiv.org/html/2611.23810#S4.T4), COTEX improves V\*Bench accuracy by 1\.6 points for DeepEyesV2 while maintaining comparable HRBench performance\. Similarly, applying COTEX to Metis improves V\*Bench accuracy from 90\.6 to 92\.5 and HRBench accuracy by 1\.6 and 0\.9 points at 4K and 8K resolutions, respectively\. These results demonstrate that COTEX can extend different post\-trained agents while preserving their previously acquired visual perception capabilities\.

Table 5:Effectiveness on SFT and RL from the official DeepEyesV2 weights\. Call denotes the tool invocation rate \(%\)\.SFT & RL\.We further analyze the contributions of SFT tool integration and subsequent RL in[Tab\.5](https://arxiv.org/html/2611.23810#S4.T5)\. Following SFT, RL improves VStar accuracy up to 83\.3, surpassing the initial agent’s performance of 81\.7\. On the task\-specific examples reserved for RL training, optimization further improves OCR and detection accuracies by 1\.8 and 11\.5 points, respectively\. In particular, detection tool invocation increases from 32\.2% to 82\.8%, accompanied by an accuracy improvement from 35\.6 to 47\.1\. In contrast, depth accuracy decreases from 93\.5 to 74\.0 despite an unchanged tool invocation rate of 100\.0%\. These results show that RL improves tool utilization and general visual perception performance\.

### 4\.5Qualitative Results

[Fig\.5](https://arxiv.org/html/2611.23810#S4.F5)presents representative tool\-use trajectories for OCR, object detection, and depth estimation\. Each example shows how the agent invokes a specialized tool to acquire additional visual evidence and incorporates its observations into the final prediction\. These examples complement the quantitative results by demonstrating how newly introduced tools support task\-specific multimodal reasoning\.

## 5Conclusion

We introduce COTEX, a continual tool space extension learning framework for scalable multimodal agents\. COTEX recovers two sets of directions to derive the initiation of a tool call, and to determine which tool is chosen\. Each new tool is learned as a low\-rank adapter inside the former and orthogonal to the latter\. Therefore, the shared invocation ability is reused while existing tool choices are left intact\. Its training data come from existing annotated datasets for each tasks, with no teacher model involved\. In our experiments, the agents gained the new capabilities with almost no loss on the tools they already used\.

## LLM Usage Statement

We used OpenAI ChatGPT to assist with experimental code development, LaTeX formatting, and language editing of the manuscript\. All AI\-assisted code and manuscript content were reviewed and verified by the authors\. We take full responsibility for the final content of this work, including all experimental results, text, and artifacts\.

## References

- \[1\]Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al\.Qwen2\.5\-VL technical report\.*arXiv preprint arXiv:2502\.13923*, 2025\.
- \[2\]Mahtab Bigverdi, Zelun Luo, Cheng\-Yu Hsieh, Ethan Shen, Dongping Chen, Linda G\. Shapiro, and Ranjay Krishna\.Perception tokens enhance visual reasoning in multimodal language models\.In*CVPR*, pages 3836–3845, 2025\.
- \[3\]Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer\.QLoRA: Efficient finetuning of quantized llms\.In*NeurIPS*, 2023\.
- \[4\]Qihua Dong, Kuo Yang, Lin Ju, Handong Zhao, Yitian Zhang, Yizhou Wang, Huimin Zeng, Jianglin Lu, and Yun Fu\.Ref\-Adv: Exploring MLLM visual reasoning in referring expression tasks\.In*ICLR*, 2026\.
- \[5\]Yongkun Du, Zhineng Chen, Hongtao Xie, Caiyan Jia, and Yu\-Gang Jiang\.SVTRv2: CTC beats encoder\-decoder models in scene text recognition\.In*ICCV*, pages 20147–20156, 2025\.
- \[6\]Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A\. Smith, Wei\-Chiu Ma, and Ranjay Krishna\.BLINK: Multimodal large language models can see but not perceive\.In*ECCV*, 2024\.
- \[7\]Tanmay Gupta and Aniruddha Kembhavi\.Visual programming: Compositional visual reasoning without training\.In*CVPR*, pages 14953–14962, 2023\.
- \[8\]Lingfeng He, De Cheng, Huaijie Wang, Xi Yang, Nannan Wang, and Xinbo Gao\.Task\-driven subspace decomposition for knowledge sharing and isolation in lora\-based continual learning\.In*Proceedings of the 43rd International Conference on Machine Learning*, pages 41172–41194\. PMLR, 2026\.
- \[9\]Jack Hong, Chenxiao Zhao, Chenglin Zhu, Weiheng Lu, Guohai Xu, and Xing Yu\.DeepEyesV2: Toward agentic multimodal model\.In*ICLR*, 2026\.
- \[10\]Edward J\. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen\-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen\.LoRA: Low\-rank adaptation of large language models\.In*ICLR*, 2022\.
- \[11\]Jerry Huang, Prasanna Parthasarathi, Mehdi Rezagholizadeh, and Sarath Chandar\.Towards practical tool usage for continually learning LLMs\.*arXiv preprint arXiv:2404\.09339*, 2024\.
- \[12\]Dimosthenis Karatzas, Lluis Gomez\-Bigorda, Anguelos Nicolaou, Suman Ghosh, Andrew Bagdanov, Masakazu Iwamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu, Faisal Shafait, Seiichi Uchida, and Ernest Valveny\.ICDAR 2015 competition on robust reading\.In*International Conference on Document Analysis and Recognition*, pages 1156–1160, 2015\.
- \[13\]Youngwan Lee, Soojin Jang, Yoorhim Cho, Seunghwan Lee, Yong\-Ju Lee, and Sung Ju Hwang\.MultihopSpatial: Multi\-hop compositional spatial reasoning benchmark for vision\-language model\.In*ECCV*, pages 580–598, 2026\.
- \[14\]Bohao Li, Yuying Ge, Yi Chen, Yixiao Ge, Ruimao Zhang, and Ying Shan\.SEED\-Bench\-2\-Plus: Benchmarking multimodal large language models with text\-rich visual comprehension\.*arXiv preprint arXiv:2404\.16790*, 2024\.
- \[15\]Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, Lei Zhang, Jianfeng Gao, and Chunyuan Li\.LLaVA\-Plus: Learning to use tools for creating multimodal agents\.In*ECCV*, pages 126–142, 2024a\.
- \[16\]Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang\.Grounding DINO: Marrying DINO with grounded pre\-training for open\-set object detection\.In*ECCV*, pages 38–55, 2024b\.
- \[17\]Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu\-Cheng Yin, Cheng\-Lin Liu, Lianwen Jin, and Xiang Bai\.OCRBench: On the hidden mystery of ocr in large multimodal models\.*Science China Information Sciences*, 67\(12\):220102, 2024c\.
- \[18\]Yuyang Liu, Meng Cao, Xinyuan Shi, and Xiaodan Liang\.COLT: Enhancing video large language models with continual tool usage\.*Transactions on Machine Learning Research*, 2026\.
- \[19\]Mao\-Lin Luo, Zi\-Hao Zhou, Yi\-Lin Zhang, Yuanyu Wan, Min\-Ling Zhang, and Tong Wei\.Keeplora: Continual learning with residual gradient adaptation\.In*ICLR*, 2026\.
- \[20\]Daniel Marczak, Simone Magistri, Sebastian Cygert, Bartłomiej Twardowski, Andrew D\. Bagdanov, and Joost van de Weijer\.No task left behind: Isotropic model merging with common and task\-specific subspaces\.In*Proceedings of the 42nd International Conference on Machine Learning*, pages 43177–43199\. PMLR, 2025\.
- \[21\]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y\. K\. Li, Y\. Wu, and Daya Guo\.DeepSeekMath: Pushing the limits of mathematical reasoning in open language models\.*arXiv preprint arXiv:2402\.03300*, 2024\.
- \[22\]Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach\.Towards vqa models that can read\.In*CVPR*, pages 8317–8326, 2019\.
- \[23\]Dídac Surís, Sachit Menon, and Carl Vondrick\.ViperGPT: Visual inference via python execution for reasoning\.In*ICCV*, pages 11888–11898, 2023\.
- \[24\]Haochen Wang, Xiangtai Li, Zilong Huang, Anran Wang, Jiacong Wang, Tao Zhang, Jiani Zheng, Sule Bai, Zijian Kang, Jiashi Feng, Zhuochen Wang, and Zhaoxiang Zhang\.Traceable evidence enhanced visual grounded reasoning: Evaluation and method\.In*ICLR*, 2026a\.
- \[25\]Qixun Wang, Yang Shi, Letian Cheng, Zhuoran Zhang, Yan He, Yuqi Tang, Qi Zhang, Xinlei Yu, Ruizhe Chen, Tianrun Xu, Yuanxing Zhang, Pengfei Wan, Haotian Wang, and Xianghua Ying\.Beacon: Knowing when and how to perform agentic visual reasoning\.*arXiv preprint arXiv:2607\.28595*, 2026b\.
- \[26\]Wenbin Wang, Liang Ding, Minyan Zeng, Xiabin Zhou, Li Shen, Yong Luo, Wei Yu, and Dacheng Tao\.Divide, conquer and combine: A training\-free framework for high\-resolution image perception in multimodal large language models\.In*AAAI*, pages 7907–7915, 2025\.
- \[27\]Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang\.Orthogonal subspace learning for language model continual learning\.In*Findings of EMNLP*, pages 10658–10671, 2023\.
- \[28\]Penghao Wu and Saining Xie\.V\*: Guided visual search as a core mechanism in multimodal LLMs\.In*CVPR*, pages 13084–13094, 2024\.
- \[29\]Yunqiu Xu, Linchao Zhu, and Yi Yang\.MC\-Bench: A benchmark for multi\-context visual grounding in the era of mllms\.In*ICCV*, pages 17675–17687, 2025\.
- \[30\]Shilin Yan, Jintao Tong, Hongwei Xue, Xiaojun Tang, Yangyang Wang, Kunyu Shi, Guannan Zhang, Ruixuan Li, and Yixiong Zou\.Act wisely: Cultivating meta\-cognitive tool use in agentic multimodal models\.*arXiv preprint arXiv:2604\.08545*, 2026\.
- \[31\]Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao\.Depth anything: Unleashing the power of large\-scale unlabeled data\.In*CVPR*, pages 10371–10381, 2024\.
- \[32\]Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan\.GPT4Tools: Teaching large language model to use tools via self\-instruction\.In*NeurIPS*, 2023\.
- \[33\]Xiaosong Yuan, Chen Shen, Shaotian Yan, Kaiyuan Liu, Xiaofeng Zhang, Sinan Fan, Liang Xie, Wenxiao Wang, Renchu Guan, Ying Wang, and Jieping Ye\.Differential fine\-tuning large language models towards better diverse reasoning abilities\.In*ICLR*, 2026\.
- \[34\]Ziwei Zheng, Minghao Yang, Jack Hong, Chenxiao Zhao, Guohai Xu, Le Yang, Chao Shen, and Xing Yu\.DeepEyes: Incentivizing “thinking with images” via reinforcement learning\.In*ICLR*, 2026\.
- \[35\]Xuan Zhou, Xuefeng Wei, Zhi Qu, Yusuke Sakai, Hidetaka Kamigaito, and Taro Watanabe\.A cross\-domain tool\-augmented vision–language framework for remote sensing image understanding\.*Remote Sensing*, 18\(10\):1613, 2026\.
- \[36\]Yiming Zuo, Karhan Kayan, Maggie Wang, Kevin Jeon, Jia Deng, and Thomas L\. Griffiths\.Towards foundation models for 3d vision: How close are we?In*International Conference on 3D Vision \(3DV\)*, pages 1285–1296, 2025\.

相似文章

TACO:面向智能体工具使用的工具增强信用优化

Hugging Face Daily Papers

TACO提出了一种针对代码工具智能体的新型信用优化方法,该方法使用差分奖励探针和结果门控优势路由来区分有用、冗余或误导性的工具调用,从而提升多模态智能体的性能。

Contract2Tool:学习前提与效果以实现可靠的工具增强型LLM代理

arXiv cs.AI

本文介绍了Contract2Tool,一个从工具元数据、文档和执行轨迹中自动推断轻量级工具契约(前提条件、效果、风险)的框架,为LLM代理实现可靠的因果工具过滤。实验表明,学习到的契约在下游多步骤代理任务中达到了接近黄金契约的性能,同时显著减少了token使用量。

SkillCAT:对比性评估与拓扑感知的LLM智能体技能自我进化

arXiv cs.CL

SkillCAT是一个无需训练的LLM智能体技能自我进化框架,通过三个阶段解决单轨迹偏差、未经验证的合并和全语料库加载等问题:对比因果提取、评估增强进化和拓扑感知任务执行,在基准测试上实现高达40.40%的提升。

CanvasAgent: 通过视觉工具编排实现复杂图像创建与编辑

Hugging Face Daily Papers

本文介绍了 CanvasCraft,一个用于复杂图像创建与编辑的大规模多模态工具使用数据集,以及 CanvasAgent,一个通过多轮交互和混合奖励优化来学习编排异构视觉工具的工具增强型多模态智能体。